Template Sections

[YaronWittenstein]

Motivation

This PR rewrites the way a Template is represented and being used.
The main trigger for that change was the decision to store pieces of data that could have been otherwise stored off-chain.

Here are 2 examples:

Schema

Contains the high-level definitions of the persistent storage of a Template.
Each variable has an id, a name and a type (primitive or composite`) associated with it.

The Schema isn't required for the execution of a transaction.
From SVM's Runtime perspective - it only cares about the raw data.
This data contains the raw Layout of raw variables.

A raw variable has an id, offset and byte size. It has neither name nor a type.
Each raw variable represents a primitive. For example, an 5-items Array defined under the Schema will be translated
into 5 raw variables under the Layout.

API

Similarly, SVM's core doesn't need to hold the high-level API of a Template in order to execute a transaction.

It already has the Exports defined under a Wasm program.
Of course, an Export function is very low-level but that info is a non-concern for a user using smapp / similar

Additionally, each deployed Template should store its deployment content.
Details such as the layer of the deployment or the Account that sent the transaction.
This kind of information is a must-have for Transactions Historical Explorers but in theory part of this can be also used for core processes inside SVM (in future versions upon arithmetical changes).

Sections

Generally, SVM needs to retrieves different pieces of data in different contexts.
We can roughly categorize the content as Core and Non-Core.

It's pivotal to be able to fetch only the required data as fast as we can.
Secondly, it's crucial to make the format of a Template easy to evolve in the future.

The solution introduced in this PR is called the Template Sections.
Instead of looking at a Template as a single unit - from now on, we'll consider a Template
as a collection of independent Section. Each Section will serve as single purpose and belong to either the Core or Non-Core category.

The Template is an evolving entity. There is the pre-deploy stage and post-deploy stages.
Instead of having something such as PreDeployTemplate and PostDeployTemplate in the code base - deploying a Template means adding a Deploy Section to a collection of Section.

If in the future we'll add a removal feature for a Template - we could come up with a new Removal Section or similar.

Moving into the Sections design, we can now go further and store groups of Sections in different indexes very naturally.
Upon fetch, we should query the required indexes and get the corresponding Sections back.

This PR also implements a skip Section functionality when loading data.

Each Section is prefixed with a Section Preview, so when decoding the raw fetched data, we can ask to skip Sections that we don't want and only decode the ones we want.

To summarize, we get 2 persistence optimizations:

• Store different groups of Sections under the same key of the underlying key-value database
• Given a group of binary fetched Sections we can ask to decode only what the ones we're interested in and skip the rest.

Last notes:

• SVM will probably consider the Non-Core Sections as optional when retrieving them.
  It means that the client asking for them will have to support a fallback for obtaining that data (from some off-chain service).
  However, for Genesis it's seems fair that each Template will store each optional Section.

• A Future PR might be using the Sections building blocks for an App as well.

@avive @lrettig This PR is the outcome of our previous Node API changes talks

[neysofu]

Reviewing this now, let's see what I can learn.

[neysofu]

Awesome to see these many comments. I haven't looked at everything as it deals with a lot of stuff that I'm not familiar with, but it seems good to me

[neysofu]

I think we should keep this if possible. It'd still be relevant after updating it with the appropriate naming changes, right?