FRAME encapsulation vs configurability

[apopiak]

When writing pallets with FRAME it's great that I can tune the visibility of my pallet's types (including storage items) like plain old Rust types.
Unfortunately it's quite common to have to make adjustments to (potentially external) pallets when integrating them into a runtime. An example are custom storage migrations that need access to the innards of a pallet to do their job.

So going forward how do we want to trade off (or preferably enable both) encapsulation and configurability in FRAME?

[athei]

All the storage items and subsequently the types of the values in pallet_contracts are public. This decision was made long before my time and I now feel the weight of it: Basically every change to storage requires burning through a major version number. Even adding a new contract callable function requires this.

Because of this I am planning to make them all private with v4 and add functions to the public API when requested. Those can be added in minor version numbers.

[apopiak]

You would want the equivalent of Solidity's view functions then, though, right?

[gui1117]

I agree storage should better be private, so the API is clean

The only usecase I see for wanting to access storages is for migration where the pallet might not expose enough function to be able to write on_runtime_update code easily.

Though the proper way should be to have specific function to set those storage when doing migration.

[bkchr]

As @xlc said below, the frontend can already access all the storage values for reading. I have already run into multiple situations where I would at least like to have the keys of a particular storage item.
I also don't have any proper idea on how to solve this. I understand that this breaks semver, but it is also relative weird that we don't have access to this data from inside the runtime.

This is just frustrating 😬

[athei]

Then add a getter() that fetches the item for you. This way you can at least change the type of map or other properties without breaking semver.

[bkchr]

Yeah that works for the "pallet is used in a runtime" use case. However with parachains on the horizon we will require storage keys and the particular type to generate a storage proof. But yeah, this is maybe something we should solve in some different way.

[xlc]

Keep in mind that while storage may be private to other pallets, it is almost always public to frontend and usually the only way for frontend to fetch data from pallets.

[athei]

The frontend isn't covered by semver so breaking it in a minor version is OK, right?

[apopiak]

uhm, if you want you can be technically correct about that, but it sucks, it's code interacting with the pallet that will break due to changes
IMO all interfaces should be covered by semver

[xlc]

It depends on the goal of applying semvar. To not break just the Rust code or to not break any code?

We have custom RPC but it is no way near ready to replace direct storage access.

[athei]

uhm, if you want you can be technically correct about that, but it sucks, it's code interacting with the pallet that will break due to changes
IMO all interfaces should be covered by semver

Then it will be impossible to ever add any feature no matter how little to the contracts module without bumping the major version. That also sucks for users.

[apopiak]

Which is why I think we should consider a way to define an interface that we wouldn't break all the time