Add range to ReadonlyStorage

[ethanfrey]

Closes #53

This is working pretty well now. All tests pass and I am happy enough with the code structure.

TODO:

• Add CHANGELOG entries

Do #188 later

[ethanfrey]

Reponded

[ethanfrey]

This is not done, but close (the wasm integration is left).

But before I get into C types and all, I would like a look over the API I am creating. (And yes, I want this with a feature flag for more general blockchain support, even if this ends up as a default-on flag).

[ethanfrey]

Interesting... it seems that only the extern "C" functions that are reachable by rust code in the contract are required in the wasm build. I enabled the iterator feature and it didn't change the imports. I then modified code and it did quite a bit.

$ wasm-nm -i ./target/wasm32-unknown-unknown/release/queue.wasm 
i scan
i write_db
i next

$ wasm-nm -i ./target/wasm32-unknown-unknown/release/hackatom.wasm 
i read_db
i write_db
i canonicalize_address
i humanize_address

The feature flag seems unnecessary for building the wasm contracts, and we can always enable it there (default).

However, when working on the cosmwasm-vm and the functions it needs from the runtime, we need to clearly define this (as it is exposed or not exposed at link time, not when a contract is loaded)

[webmaster128]

Some random thoughts for now

[ethanfrey]

I think I've done all the plumbing, but now a SIGABRT when using the scan/next callbacks. I need to get into serious debug mode. (playing with unsafe code here).

One larger issue I found is when I write to two buffers, there is no clear way to return the length of both of them. In the write_db, we only wrote to one buffer and returned a single i32 as the length or error code.

Relevant lines to explain what I mean:

• Where the VM writes: https://github.com/CosmWasm/cosmwasm/blob/iterator/packages/vm/src/context.rs#L95-L105
• Where we read and truncate (value only) in the contact: https://github.com/CosmWasm/cosmwasm/blob/iterator/packages/std/src/imports.rs#L145-L152

@webmaster128 would it make sense to "upgrade" the buffer from {offset, len} to {offset, len, cap} like we do on the Go FFI boundary? Then we can always write the proper length in the Region object while writing data back from rust to wasm.

If you like the idea and have time, I'd be happy if you could work on a PR on that. Otherwise I look tomorrow

[webmaster128]

@webmaster128 would it make sense to "upgrade" the buffer from {offset, len} to {offset, len, cap} like we do on the Go FFI boundary? Then we can always write the proper length in the Region object while writing data back from rust to wasm.

If you like the idea and have time, I'd be happy if you could work on a PR on that. Otherwise I look tomorrow

That makes a lot of sense to me. Having two separate paths (data at pointer and return value) that need to be brought together in order to read data smaller than the whole region is no the nicest thing I can imagine.

The structure we end of with is by the way almost the same as a Rust vector

Most fundamentally, Vec is and always will be a (pointer, capacity, length) triplet. No more, no less.

https://doc.rust-lang.org/std/vec/struct.Vec.html#guarantees

I'm looking into this right now to understand the problem. Will at least start with something right now and push later.

[webmaster128]

Great stuff. Looking forward to finish this.

Do you have any idea what happens when you delete a database entry in the middle of a range query?

[ethanfrey]

Thank you, I'm glad to get this in and add a few follow up PRs, for lifetimes, renamings, and multiple iterators.

Do you have any idea what happens when you delete a database entry in the middle of a range query?

You cannot delete an entry in the middle of a range query.
The iterator holds &storage for the lifetime of the iterator.
Removing an item requires &mut storage, which requires that all other references have been released.

In this case the over-eager compiler helps us 😄

[ethanfrey]

Actually in this version (before #188), the iterators loads all data into memory, and you can delete while iterating, but the iterator will keep giving you a snapshot of previous data. I prefer the compiler prohibiting it.

[webmaster128]

🥇

Time to merge the big chunk