RFC Storage/Primitives refactoring

[ilblackdragon]

I have bunch of comments:

• We need a way better solution then merging everything to primitives
• I think StateDb as an abstraction was a good level, where Trie is just an implementation of specific way to store data and merkelize it (alternative can be just a hash map for example, or may be there is a better data structure for such things).
• Storage IMO should not know about beacon/shard chains. That's more of a job of either specific chains or a blockchain object.
• Shard chains should not be stored/indexed by id, because we are going to be resharding and the same ID will point to a different chain. Genesis block is the primary index of each chain.
• Currently it seems like we don't have difference between core and node folders, so we should just move everything into node folder.

[MaksymZavershynskyi]

Regarding the ids for the shards

Conceptually it would be nice to have shards identified by their genesis, however not having a numerical id for the shards is much harder and not supported by some of our current algorithms. One of such places is authority, when we distribute the seats for the validators across the shards these shards are enumerated. Other place is column storage, the current solution (kvdb) identifies columns by numerical id only. I think it is a good idea to make each shard and beacon use separate set of columns because it allows parallelism and bug-protection (we already had a bug because we wrote beacon chain and shard chain into the same columns), so shards are going to be identified numerically anyway.

In general, I think it is way too early to address the resharding, and I was under impression that we all agreed that resharding is one of the last things that we will be solving post mainnet. @SkidanovAlex

[MaksymZavershynskyi]

Regarding having storage not know about anything like shards, chains, etc

I believe we need to have a data-layer separate from the domain layer that does the ORM. It prevents other components in our infra from being exposed to the implementation details of the storage. The reasons why we want this separation:

1. Efficiency. It is easier to write efficient code that does serialization/deserialization and caching in one place instead of throughout the infra. Right now serialization/deserialization and caching of the objects read from the column storage is spread through beacon chain, shard chain, and trie code;
2. Safety. If we allow various parts of infra operate directly on the columns then they can accidentally interfere with each other. It has already happened, our beacon chain and shard chain were writing block-index => block-hash mapping into the same column unintentionally overwriting each other. The only reason why it was not crashing us is because we had beacon blocks hash equal to its shard block hash;
3. ORM is a common practice. Separating data-layer from domain-layer is typical. Often it requires introduction of the classes/structs of data-layer that closely mirror classes/structs of the domain-layer. Often it bloats the number of classes. I suggest we do not do it for now, and instead use the same structs for data-layer and domain-layer, this will however make storage dependent on the beacon chain and shard chain -specific structs.
4. Forwards compatibility is easy to implement. Because column-id to object mapping is now located entirely inside the storage crate we can implement the code that allows forwards compatibility, e.g. it will continue working if we add more columns or shards. In contrast, if we allow various parts of our infra access storage directly by column id it might break if for instance we increase the number of shards.
5. No second-level column re-implementation. Current infra uses prefixes and suffixes for keys of the columns to allow storing different object in the same column. See COL_EXTRA and ExtraIndex, this is a column in which we store best block hash, transaction addresses and transaction results by encoding them differently. Having our virtual columns on the top of the real storage columns makes code more complex.

[ilblackdragon]

• Resharding is what we need to have before mainnet: otherwise, we won't be able to upgrade from one shard to few shards. We don't need to implement all of it, but we need to think through it otherwise we will need to refactor everything again.
• We do need shard id for authorities, but authorities are ephermal (e.g. computed at the moment), so they can rely on the current number of shards.
• Storage, on the other hand, is where proper managing of shard data is very important. I understand that you want to store it in different columns, but I think this is what is going to lead to issues down the line. At least in the current form.
  For example, you spin up a binary and receive a block for shard 3, but shard 3 is actually a different chain now. You actually need to finalize the previous chain (to get the final state) and then switch to a new chain. Or wipe the previous chain and create a new chain storage and re-fetch. I'm unclear how this process will be done in the current shard id -> column.
  The alternative, for example, would be to add a mapping somewhere in column 0, which maps shard genesis to column id.

For storage:

• Actually, ORM would be where you describe what you store in storage in your own module. E.g. chain, runtime and etc describe what they store and the just use storage/ORM for storing. E.g. how it was. If you think we had missing tools / leaked some details for ORM to work - that's fine to add them. But you just moved all the objects/logic into the ORM itself.
• For pretty much everything - I'm totally fine with hiding secondary indexes/columns; caching and etc in storage. Just prefer to not have Block/Chain/etc in it.

[SkidanovAlex]

Where do shard IDs get persisted today on the blockchain?
I.e. for resharding my understanding was that you can reshard via a hard fork, by just changing the boundaries of the shards.
Is the goal to not have shard IDs to avoid such hard fork, or are there some other places that break (such as persisting shard IDs somewhere)?

Dynamic resharding (without at least shutting down the client and restarting with a different config) is a nice-to-have, but by no means necessary?

[MaksymZavershynskyi]

• Regarding resharding before MainNet. I think any kind of resharding would be a stretch in terms of complexity, because it is hard to implement and we do not really have spare workforce to work on that. We have more crucial components of the system that are very far from finished, e.g. network;
• Regarding shard ids. We might be able to figure out a nice way to not have shard ids but instead rely on genesis. I like the idea, but suggest we do it as a part of the future iteration;
• Regarding storing in different columns. I feel like your concerns apply mostly when we use genesis hash as shard identifier and are the result of the API of the database engine that we are currently using, kvdb, being too rigid.
• Regarding storage and ORM. ORM objects almost always mirror the domain-layer objects. For now, I decided to use the same structs for ORM objects and domain-layer objects that created dependency of storage depending on beacon/shard-specific objects. We can split it into ORM-specific objects and domain-layer objects later if we don't like this dependency. We did not have ORM before, if we take a domain-layer object, serialize it into bytes and place directly into database column, it is not an ORM.

[ilblackdragon]

@SkidanovAlex
My idea for recharging was to "close" current shard chains and re-create new ones.
This requires one beacon chain block of time, when there is no progress on any shards, while new validators (and anybody else tracking particular shards) are syncing the state they are responsible.

There is no need to shut down anything, as this is just will happen pretty natural if we ID shards by genesis block. E.g. reshard happened:

• beacon chain marks one block that has new number of shards, checkpointing final state of the previous shards.
• you as validator is assigned to a new shard with different genesis block. Nobody gets assign to old ones. (a bit unclear yet who computes new genesis - may be validators from previous block should provide not final merkle root of the shard but actually parts of the tree such that new state roots for new shards are easily computable)
• you will fetch state from previous validators, in this case from multiple shards instead of one.

[ilblackdragon]

Closing at this point, we just going with the current architecture.