Reducing data stored on tfchain

[brandonpille]

It is no news that we are storing too much data on tfchain, it is too centralized and not scalable. Let's discuss some ideas on how we can move the data out of the chain (not all data of course but most of it). Any ideas are welcome, we want to think outside of the box here.

The goal of this discussion is to find a better solution, to setup a plan of action, to setup a timeline and most important of all to reach consensus. It's important that we think this through and that we all believe in this new idea before we proceed. @DylanVerstraete, @maxux, @LeeSmet, @muhamadazmy, @coesensbert, @robvanmieghem, @delandtj, @renauter, @sabrinasadik can you all share your thoughts? Please tag other people that might have good ideas.

[brandonpille]

I'll start with my thoughts.

How I would want to see it is having minimal data on chain. I think we should introduce some "database" service that gathers this information from the nodes and that opens it up to the world. This service would ofcourse be replicated for scaling and safety purposes. The service doesn't have to use a pull mechanism, instead we could let the zos nodes push their information to the service. The zos nodes can ask tfchain for the location of the service(s). ZOS can then push it's information to the first service that is live.

This does raise some questions on minting though. Maybe the actual tokens to mint could be calculated outside the chain and we could introduce an extrinsic that will execute the transaction once multiple services send the same price. I'm just thinking out loud here, I might be completely on the wrong track.

[muhamadazmy]

I think the out of the box solution would be using IPFS or something similar where the node can make sure their data is first persisted somewhere to this distributed storage system and then only put the hash (which also an address to the required information) on the chain.

Ideally this ipfs (or ipfs like) system can be hosted on other zos nodes on the grid that has higher availability and granted to not shutdown.

This also raises few concerns since the data will have to be pushed first then committed to the chain how to prevent abuse of the storage (cost for persistence ? who is paying? etc...)

[brandonpille]

I like the idea. That would be a decentralized system. Though retrieving the data would be a bit harder. We would have to ping many nodes to get the information.

What do you mean by cost for persistence and who is paying?

[LeeSmet]

We know storing things on chain should only be done if it is actually needed for the chain/relevant. In that regard, one obvious issue is the kvstore pallet. This is essentially a pallet which has the sole purpose of allowing people to store arbitrary data under arbitrary keys on the chain, and serves no purpose whatsoever for the chain itself. Considering this is actively being used, we should deprecate it and disable the ability to insert new data (by returning an error notifying of deprecation, should anyone read the error), while allowing the data to still be read and deleted. This way we don't have an abrupt breaking change, and tools which utilize this functionality have time to transition to a new setup, while retaining access to existing data. In the version after this is introduced, the pallet can be deleted outright.

Aside from that, we currently need a couple of things on chain:

• Uptime reporting (for minting)
• Node information (needed for minting, some tools might use this as well). This needs to be looked at since there is some redundant data here, for instance node coordinates and node country/city. We could keep coords in case this gets relevant at some point, but city/country information should be removed as it is not needed for anything on chain (this is just human readable coords). We should also consider removing coords since they are not used for minting right now anyway.
• Farm info
• Contract info, which also tracks used resources. it should be noted that contracts have an arbitrary metadata field which contains user provided fields, which should probably also be removed.
• Payment details for the above contracts. This currently results in one call per hour where zos notifies the chain about used resources. Then there is an additional call to perform the actual payment. This is currently triggered by an off chain worker. That setup should also be looked at, to see if we can improve this. Note that it is actually this call which will cause funds to be locked on the users account.

Generally the chain should only contain data relevant for minting and payment, anything else should be removed. We also need to make sure we do not add any more data which is not strictly needed for these purposes.

Data needed for these purposes could also be stored off chain, and indeed a hash can be used on chain to verify the data. The chain does not really care where the data comes from, as long as it is deterministic. For this uses case, off chain workers can be used to compute the minting reward (if any), so long as this is easily verifiable by others.

[brandonpille]

Those are some good first points to work on. Could the off chain workers could also be used to get the data from outside the chain? We could then fetch the data and do the minting in the off chain workers? Then we would store almost no data on chain...

[renauter]

Generally the chain should only contain data relevant for minting and payment, anything else should be removed. We also need to make sure we do not add any more data which is not strictly needed for these purposes.

You stole my words, this is exactly my though.
Only data required for minting and billing purpose should remain.
Unfortunately I have no technical suggestion to solve the problem.

Meanwhile, a first task would be identifying which data could be off chain.
Actually bigger storage are located in tfgrid and smart-contract pallets.
I never understood what pallet kvstore before @LeeSmet explain so it definitively could be deprecated.

On the other side, I will emphasis that fact that we should also clean our existing storage first since there could be some inconsistencies.
See here what was found on pallet smart-contract.

And to prevent storage "leaking" we should generalize to all pallets the use of a live storage checking tool that can be run on a target network to make sure everything is as expected.
For now such tool is only available for pallet smart-contract (see issue here).

[brandonpille]

Yes it makes sense to clean the incosistencies. But we should also ask ourselves how they were introduced so that we can fix the source.
Regarding preventing the storage "leaking" I would propose we do these on a weekly basis maybe on CI?

[renauter]

Indeed the idea is to detect the inconsistencies, try to understand where it comes from, fix the code and clean the storage.
Last time we fixed some, others were unexplained (maybe result of old logic that went through some storage migrations...).
And yes, doing this check on CI on a weekly basis (and maybe also after each chain upgrade) would be nice.

[brandonpille]

Ok let's create an issue for it :)

[renauter]

#769

[robvanmieghem]

A standard solution: "We don't want it here, let's just throw it in another system".
What would the requirements for such an external storage system be?

• Ability to define strongly typed datastructures
• Needs signatures for the stored data that can be checked against the tfchain addresses
• Storage migrations for upgrades without breaking the signatures
• Replicated and always available as core components of the grid will depend on it
• Decentralized
• Cheap and payable with TFT
• Indexed or indexable
• Protected in a way that not everyone can dump shitloads of data in there
• Lower development maintenance and easier to adapt to change than the current solution

[brandonpille]

Some good points indeed! What do you mean by cheap and payable with TFT? Why would the system have to be paid for with TFT?

[robvanmieghem]

When thinking about it, we don't need a traditional storage system.
As long as the latest state is queryable and state updates are signed, we're good.
This makes me think of the Command Query Responsibility Segregation pattern which fits perfectly with the updates being events which are processed by the read storage.
In our case, the events can be emitted by the chain without defining on-chain datastructures but we keep the correctness protection, replication, ... from the chain.
Yes, the chain would become a write event log but no datastructures are defined and storage migrations are no longer needed.
The current graphql indexer infrastructure picks them up and provides the ability to query.
I think we've done this before ( or still do) for some updates.

[brandonpille]

I believe @DylanVerstraete has proposed this in the past. I do like that idea. We will need to have snapshots then no?

[robvanmieghem]

snapshots?

[brandonpille]

If we would have multiple instances of graphql, starting one up would have to go through all events to be up to date. In that case snapshots would be useful.

[robvanmieghem]

Indeed, unless the data for a specific subdomain is posted(updated) on a regular basis, we could shard the read infrastructure where a complete replay of all events is not needed.

[coesensbert]

Snapshots for current nets are already here: https://bknd.snapshot.grid.tf/
https://github.com/threefoldtech/grid_deployment/tree/development/snapshots