-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: build schema, inject into Wasm #112
Conversation
only for workspaces for now
All the `with_schema` versions end up being smaller than the originals, when testing with the `cw-plus` repo. Even though they have extra data injected into them?? Thanks, Walrus! ll target/wasm32-unknown-unknown/release/*.wasm -rwxr-xr-x 1 chadoh staff 331K Jan 23 11:51 target/wasm32-unknown-unknown/release/cw1_subkeys.wasm -rw-r--r-- 1 chadoh staff 314K Jan 23 11:47 target/wasm32-unknown-unknown/release/cw1_subkeys_with_schema.wasm -rwxr-xr-x 1 chadoh staff 217K Jan 23 11:51 target/wasm32-unknown-unknown/release/cw1_whitelist.wasm -rw-r--r-- 1 chadoh staff 205K Jan 23 11:48 target/wasm32-unknown-unknown/release/cw1_whitelist_with_schema.wasm -rwxr-xr-x 1 chadoh staff 368K Jan 18 14:08 target/wasm32-unknown-unknown/release/cw20_base.wasm -rw-r--r-- 1 chadoh staff 349K Jan 23 11:48 target/wasm32-unknown-unknown/release/cw20_base_with_schema.wasm -rwxr-xr-x 1 chadoh staff 391K Jan 18 14:08 target/wasm32-unknown-unknown/release/cw20_ics20.wasm -rw-r--r-- 1 chadoh staff 368K Jan 23 11:48 target/wasm32-unknown-unknown/release/cw20_ics20_with_schema.wasm -rwxr-xr-x 1 chadoh staff 343K Jan 23 11:51 target/wasm32-unknown-unknown/release/cw3_fixed_multisig.wasm -rw-r--r-- 1 chadoh staff 328K Jan 23 11:48 target/wasm32-unknown-unknown/release/cw3_fixed_multisig_with_schema.wasm -rwxr-xr-x 1 chadoh staff 439K Jan 23 11:51 target/wasm32-unknown-unknown/release/cw3_flex_multisig.wasm -rwxr-xr-x 1 chadoh staff 242K Jan 18 14:08 target/wasm32-unknown-unknown/release/cw4_group.wasm -rwxr-xr-x 1 chadoh staff 296K Jan 18 14:09 target/wasm32-unknown-unknown/release/cw4_stake.wasm Not sure if this way of finding the input/output filenames is brittle. Considered adding a couple other dependencies to build these paths, but it didn't seem worth it at this time.
Hey @chadoh. thank you for your proposal. This was discussed multiple times before and we don't want to embed the schema into the Wasm for various reasons:
It would be much better to either let wasmd store the schemas or have them off-chain.
It would be interesting to understand why this is the case. If you tested cw-plus then probably because you compared cosmwasm/workspace-optimizer:0.12.8 (Rust 1.63) with cosmwasm/workspace-optimizer:0.12.11 (Rust 1.66).
Historical reasons. I'd be very happy to explore if this can be consolidated to one builder image. |
and stop saying that the Wasm with injected schema is "compressed", since it is only slightly compressed. Walrus compresses it a _little_, but the wasm-opt step later (in the bash script part of the process) is a more complete compression.
Thanks for the quick response, @webmaster128! Even on a draft. And sorry to re-open a discussion that has happened before. I talked to a few people in the Cosmos ecosystem and none of them were aware of previous discussions about this, and they all liked the idea. Is there somewhere I can read the previous discussions? I actually realized some errors in my original post and have updated it. Re: "makes the generated Wasm files smaller, rather than larger" – that's only somewhat true. While testing locally, I was only running the
Yes, a little. Is it worth it, though? I'll get you numbers on how much space this adds. With optimized contract sizes on the order of 100-200kB, adding <10kB to a custom section containing the schema doesn't seem bad to me. And the benefits are huge! Building in the CosmWasm ecosystem will become 10x easier. 5-10% "bloat" for 10x dev experience seems worth it, imo. I can see the argument for storing them off-chain. How responsive is IPFS these days? That could be a solution. But it will slow down the optimize process, as well as the fetching-of-schemas. Given the small size of the embedded schema, I'm not sure it's worth it.
Not sure what you mean by this.
Yes, and I actually didn't use it here, either. I used The tongue-out face because I assume not every project implements a
Thanks again for the reminder! I added another TODO item for this. The embedded schema absolutely needs to be versioned, so that the schema format can continue to evolve, independently of the tooling that consumes it. |
Here's the current output when building the Found workspace member entries: ["packages/*", "contracts/*"]
Package directories: ["contracts/cw1-subkeys", "contracts/cw1-whitelist", "contracts/cw20-base", "contracts/cw20-ics20", "contracts/cw3-fixed-multisig", "contracts/cw3-flex-multisig", "contracts/cw4-group", "contracts/cw4-stake", "packages/controllers", "packages/cw1", "packages/cw2", "packages/cw20", "packages/cw3", "packages/cw4"]
Contracts to be built: ["contracts/cw1-subkeys", "contracts/cw1-whitelist", "contracts/cw20-base", "contracts/cw20-ics20", "contracts/cw3-fixed-multisig", "contracts/cw3-flex-multisig", "contracts/cw4-group", "contracts/cw4-stake"]
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
Compiling cw1-subkeys v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys)
Finished release [optimized] target(s) in 9.44s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
Compiling cw1-subkeys v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys)
Finished dev [unoptimized + debuginfo] target(s) in 1.25s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys/schema/cw1-subkeys.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys/schema/cw1-subkeys.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw1_subkeys.wasm
compressed schema size: 3kB
original Wasm size: 331kB
Wasm with injected schema: 314kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
Finished release [optimized] target(s) in 5.67s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
Finished dev [unoptimized + debuginfo] target(s) in 0.66s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist/schema/cw1-whitelist.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist/schema/cw1-whitelist.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw1_whitelist.wasm
compressed schema size: 2kB
original Wasm size: 216kB
Wasm with injected schema: 204kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.05s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base/schema/cw20-base.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base/schema/cw20-base.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw20_base.wasm
compressed schema size: 3kB
original Wasm size: 367kB
Wasm with injected schema: 349kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.05s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20/schema/cw20-ics20.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20/schema/cw20-ics20.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw20_ics20.wasm
compressed schema size: 2kB
original Wasm size: 390kB
Wasm with injected schema: 367kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.05s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
Finished dev [unoptimized + debuginfo] target(s) in 0.83s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig/schema/cw3-fixed-multisig.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig/schema/cw3-fixed-multisig.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw3_fixed_multisig.wasm
compressed schema size: 6kB
original Wasm size: 343kB
Wasm with injected schema: 327kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
Compiling cw3-flex-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig)
Finished release [optimized] target(s) in 16.43s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
Compiling cw3-flex-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig)
Finished dev [unoptimized + debuginfo] target(s) in 1.36s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig/schema/cw3-flex-multisig.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig/schema/cw3-flex-multisig.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw3_flex_multisig.wasm
compressed schema size: 6kB
original Wasm size: 438kB
Wasm with injected schema: 417kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.06s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group/schema/cw4-group.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group/schema/cw4-group.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw4_group.wasm
compressed schema size: 0kB
original Wasm size: 242kB
Wasm with injected schema: 226kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.05s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake/schema/cw4-stake.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake/schema/cw4-stake.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw4_stake.wasm
compressed schema size: 2kB
original Wasm size: 295kB
Wasm with injected schema: 278kB |
Another complication with using something like IPFS or other off-chain storage is that uploading contracts requires "logging in" / adding keys to two different systems: the target blockchain and the schema-storage location. |
is this already compressed with the best tool for compression? We were wondering if we can use
would there be an way to access this and/or make an API endpoint to get the schema so it's not hard to find?
maybe we can make it a part of the build so it's always run and developers don't have to manually create schemas. Seems that abstracting this step from rust devs could be a good thing.
Since we now have IDL versions, we can always detect the schema version. If it changes, the tooling can know how to parse the schema based on the IDL which is in the schemas now. cc @ethanfrey this is the PR I was mentioning |
This was now changed and will be shipped with the next release 0.15.0. Then we only have one image for x86_64 and one for ARM64. |
Thank you for this suggestion. We thought about the whole topic more internally over a couple of months. The new schema format we currently work on will allow you embedding right into the Wasm blob through custom Wasm sections, and does not require a change of the build system. Closing this for now. |
Wouldn't it be great to always have schemas available right on-chain?
Then tooling can plug right in. Tools like ts-codegen can build TS wrappers for on-chain contracts. Frontends like RAEN Admin can generate interactive docs. You can build interactive CLIs for specific deployed contracts. Imagine!
Perhaps this hasn't been done out of fear of bloating the size of deployed Wasm files. But if we compress the JSON with brotli, it ends up adding just a tiny number of kB for most contracts. The largest compressed schemas in the cw-plus repo weigh in at 6kB, for the multisig contracts. That's 6kB on (uncompressed) contracts weighing in at 343kB (fixed multisig) and 438kB (flex multisig). Not bad!
Since the point here is to only keep around schema'd Wasms, I just decided to update in-place. We could keep the old un-schema'd files around, though, if y'all prefer.
To do
Cargo.toml
definesmembers
or not).schema
binary / alias. In these cases, just don't add the schema.