Implement ShardUId

[mzhangmzz]

This PR implements ShardUId as specified in near/NEPs#241 so that trie states are indexed by 'shard_uidinstead ofshard_id`.

The main change is to change the interface around trie storage to use shard_uid instead of shard_id. The database columns ColState and ColTrieChanges are now indexed by shard_uid instead of shard_id.

Changes that this PR does not include:

• change account_id_to_shard_id to account for changing shard layout
• gc

Test Plan:
The main source for errors is where shard_id and shard_uid is mixed up or the wrong shard_version is used for shard_uid, which may result in DB not found error. To test for the first type, we will change a wide range of existing unit and integration tests to use a shard layout with shard_version = 1 so shard_id and shard_uid cannot be mixed up.

[bowenwang1996]

I am afraid that sync_hash is not what you want here because sync_hash is the first block of the new block and I believe here the chunk could be included in a block in the previous epoch

[mzhangmzz]

Yeah, but we will disable state sync for epoch changes where resharding happens. I will add a comment here noting that.

[mzhangmzz]

I created an issue #4711 noting this

[bowenwang1996]

looks like CI is still failing

[bowenwang1996]

Given how invasive this change is, could you run nayduck to see that no test regresses?

[frol]

@mzhangmzz @bowenwang1996

That is fine if the config file does specify num_shards to be 1 through num_block_producer_seats_per_shard.len(), but the backward compatible test is actually testing for 4 shards.

#4569

[matklad]

Estimator changes looks good to me, leaving approval to unblock the PR. I haven't looked at the change as a whole.

[mzhangmzz]

@frol @bowenwang1996 hmm I realized that #4709 will not fix the backward_compatible test until that commit is included the in the latest release branch, i.e., the branch the test is using to make comparison.

I'm not sure when is the next time we will make the next release. Is there any way to bypass that?

[bowenwang1996]

@frol @bowenwang1996 hmm I realized that #4709 will not fix the backward_compatible test until that commit is included the in the latest release branch, i.e., the branch the test is using to make comparison.

I'm not sure when is the next time we will make the next release. Is there any way to bypass that?

Do you need just the branch or do you need a specific commit? If you just need it to be included in the branch, feel free to cherry-pick the commit on top of 1.21.0 branch.

[ailisp]

I mostly checked runtime part, and looks good!