"pluggable" data structure/object/interface/abstraction for DAG/unit (especially re: SQL/storage)

[thedavidmeister]

Taking #42 a few steps further...

This repo is intended to be a library for clients to use, but all the logic in the library is tightly coupled to SQL.

It's hard (for me) to see (upon casual inspection of the code) what code is implementing the ByteBall white paper so clients don't have to re-invent (and possibly screw up) key algorithms, and what code is just implementing SQL queries.

This causes a few issues:

• Every client must implement SQL
• It is harder for individual parts of the library to be split up and used independently for novel use-cases other than "wallet"
• Implementing key logic in languages/domains other than JavaScript becomes harder
• Writing unit tests for BB logic is much harder or impossible
• There is no obvious abstraction/interface/data structure for DAG/unit to optimise for performance/security/legibility without auditing/refactoring the whole library for every change
• Individual functions are just generally more complicated because they are forced to manage SQL on top of whatever other logic they need to handle

I think that if a goal was set to simply make the SQL storage backend pluggable and quarantine all SQL queries out of the "white paper logic", maybe even going as far as making the SQL storage backend a separate repo that is simply used "by default" here, then that would already help things a lot.

[Pascal-J]

One big benefit of this approach is that an SQL database is overkill for an append-only data structure. Decoupling would allow a change in backend. Lengthy syncing time is likely related to SQL inserts/appends of one transaction at a time.

[thedavidmeister]

@Pascal-J check out #42 for a discussion on validation (and hence sync times), specifically re: avoiding multiple serial round trips through SQL for every single unit being validated/synced :)

[maxmetagravity]

good advise， the SQL way is a fast implementation， "pluggable" backend will
force abstract the logic from the SQL ， it make it better scaleable， but this is a big job ：）， Hope do it earler

[thedavidmeister]

@iamliqiang a big job indeed, i think the current approach is to get more unit tests in place for existing functionality, then see what can be proposed

[tarmo888]

Obyte database is not strictly with "append-only" structure. It kind of is append-only if you take account only everything that is stable already, but all the data that is not stable gets updated a lot to figure out the next stability point.

Data feeds are append only, but Autonomous Agent state is not. Both of these are now moved to RocksDB because it makes more sense for key/value data.

Address definitions are not strictly append-only either, these are appended to separate table when they become public (when it is spent from them). Address definitions can be changed to different public key too.

Obyte full nodes have state and history of UTXOs (outputs table), which get updated with the is_spent=1 when they are spent. Archival nodes would be more strictly "append-only", but Obyte full nodes are not archival nodes because they have current state, not all history states.