feat: configure num threads for flat storage creation

[Longarithm]

Make number of threads for background flat storage creation configurable. In the node config we set it to 8, but we also allow node owners to modify it if they observe that BP slows down or creation is too slow until FS becomes a requirement. In the tests we limit number of threads to 1.

I think introducing some general structure as ChainConfig makes sense here, as we already use a configuration option save_trie_changes.

I'm open to other opinions how to organize this. In general NearConfig structure looks a bit suspicious to me, because it includes config taken from config.json and many other configs which are just derived from config.

Testing

Checking that parameter is taken correctly by the node.

[jakmeier]

LGTM

I wonder what our strategy is here for after the migration. Will we keep this config option, or remove it? If we keep it, we should probably add a an entry to the changelog to announce it.

I think introducing some general structure as ChainConfig makes sense here, as we already use a configuration option save_trie_changes.

Seems ok to me.

In general NearConfig structure looks a bit suspicious to me, because it includes config taken from config.json and many other configs which are just derived from config.

I think suspicious is a generous term here. I could think of other words to describe it.^^

This PR is obviously not gonna change anything on that. But we can discuss what our vision is. My gut feeling is we want a simple as possible struct that defines config.json. It should be structured how validators think about the config. The configs we use in nearcore code should not be tied to JSON format at all and be as detailed as we need them to be. Converting from the config.json struct to specific configurations such as ChainConfig should follow a consistent pattern, fn from_config(config: &NearConfig) -> Self for example.

I think the outline from a comment in an issue about documenting this stuff would be good strategy to really clean this up: #8022 (comment)

[Longarithm]

I wonder what our strategy is here for after the migration. Will we keep this config option, or remove it? If we keep it, we should probably add a an entry to the changelog to announce it.

I would keep it because I can imagine other such migrations in the future. Added it to the changelog under [unreleased]. Also renamed parameter to background_migration_threads because background_work_threads looks like more related to any work done in background :)

Please approve it again if you are okay with such naming.