New metadata format #296

ascjones · 2019-12-10T14:13:24Z

Closes #294.

Use externally tagged enums e.g. instead of "range.offset" use "range": { "offset" }
Convert field names to camelCase instead of snake_case
Selector bytes encoded as hex string e.g. instead of "[\"0x07\",\"0x5B\",\"0xCD\",\"0x15\"]" use "0x075bcd15"
Convert from type-metadata to https://github.com/paritytech/scale-info

The new type registry serialization depends on paritytech/scale-info#3, so only once that is merged will you get the full new format.

e.g. Incrementer:

{
  "registry": {
    "strings": [
      "Storage",
      "flipper",
      "__ink_private",
      "__ink_storage",
      "value",
      "Value",
      "ink_core",
      "storage",
      "cell",
      "SyncCell",
      "sync_cell",
      "Key",
      "ink_primitives",
      "key",
      "new",
      "init_value",
      "bool",
      "default",
      "flip",
      "get"
    ],
    "types": [
      {
        "composite": {
          "name": 1,
          "namespace": [
            2,
            2,
            3,
            4
          ],
          "fields": [
            {
              "name": 5,
              "type": 2
            }
          ]
        }
      },
      {
        "composite": {
          "name": 6,
          "namespace": [
            7,
            8,
            5
          ],
          "params": [
            3
          ],
          "fields": [
            {
              "name": 9,
              "type": 4
            }
          ]
        }
      },
      {
        "primitive": "bool"
      },
      {
        "composite": {
          "name": 10,
          "namespace": [
            7,
            8,
            9,
            11
          ],
          "fields": [
            {
              "name": 9,
              "type": 5
            }
          ]
        }
      },
      {
        "composite": {
          "name": 12,
          "namespace": [
            13,
            14
          ],
          "fields": [
            {
              "type": 6
            }
          ]
        }
      },
      {
        "array": {
          "len": 32,
          "type": 7
        }
      },
      {
        "primitive": "u8"
      }
    ]
  },
  "storage": {
    "struct": {
      "type": 1,
      "fields": [
        {
          "name": 5,
          "layout": {
            "struct": {
              "type": 2,
              "fields": [
                {
                  "name": 9,
                  "layout": {
                    "range": {
                      "offset": "0x0000000000000000000000000000000000000000000000000000000000000000",
                      "len": 1,
                      "elemType": 3
                    }
                  }
                }
              ]
            }
          }
        }
      ]
    }
  },
  "contract": {
    "name": 2,
    "constructors": [
      {
        "name": 15,
        "selector": "0x5ebd88d6",
        "args": [
          {
            "name": 16,
            "type": {
              "id": 3,
              "displayName": [
                17
              ]
            }
          }
        ],
        "docs": []
      },
      {
        "name": 18,
        "selector": "0x0222ff18",
        "args": [],
        "docs": []
      }
    ],
    "messages": [
      {
        "name": 19,
        "selector": "0x8c97db39",
        "mutates": true,
        "args": [],
        "returnType": null,
        "docs": []
      },
      {
        "name": 20,
        "selector": "0x25444afe",
        "mutates": false,
        "args": [],
        "returnType": {
          "id": 3,
          "displayName": [
            17
          ]
        },
        "docs": []
      }
    ],
    "events": [],
    "docs": []
  }
}

codecov-io · 2019-12-10T14:41:33Z

Codecov Report

❗ No coverage uploaded for pull request base (master@e020402). Click here to learn what that means.
The diff coverage is 98.86%.

@@            Coverage Diff            @@
##             master     #296   +/-   ##
=========================================
  Coverage          ?   72.45%           
=========================================
  Files             ?       76           
  Lines             ?     6604           
  Branches          ?        0           
=========================================
  Hits              ?     4785           
  Misses            ?     1819           
  Partials          ?        0

Impacted Files	Coverage Δ
abi/src/tests.rs	`100% <100%> (ø)`
abi/src/layout.rs	`50.72% <100%> (ø)`
abi/src/lib.rs	`63.63% <100%> (ø)`
abi/src/specs.rs	`70% <80%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e020402...49e5d32. Read the comment docs.

Robbepop · 2019-12-10T15:16:25Z

Superseedes #264

Robbepop

LGTM

I very much like that you actually added a wholesome test.
Let's merge this as soon as Polkadot JS gave their acknowledgement that their JSON parser can trivially parse the output of this PR.

jacogr · 2019-12-11T10:28:28Z

Is it possible to sneak in a version number into the InkProject struct? That would be beneficial going forward as well.

Robbepop · 2019-12-11T13:08:04Z

Is it possible to sneak in a version number into the InkProject struct? That would be beneficial going forward as well.

I was talking to Sean Young who implements the Solang Solidity to Wasm compiler and we want to share our metadata format for the contracts.
So for this purpose I think it would also be beneficial to include some kind of "language" property into the InkProject struct that is simply ink for ink! contracts for a start.

Also we though that some other metadata would be nice to have.
E.g. some hashes over the code to properly infer if the supplied code matches the metadata, the author of the contract to make it possible to contact maintainers, the used Rust (or other compiler) version to have "more deterministic" way to rebuild the contract etc.

What do you think?

cc @seanyoung

jacogr · 2019-12-11T14:33:07Z

@Robbepop It makes sense, the additional metadata is quite easy to add as a sub-structure, and is certainly very beneficial as we expand.

ascjones · 2019-12-11T16:31:23Z

Should this versioning/language field be done as a separate PR?

Robbepop · 2019-12-11T16:32:52Z

Should this versioning/language field be done as a separate PR?

Yes, also can you write an issue for that so we can track and discuss there what needs to be added exactly?

jacogr · 2019-12-12T09:02:07Z

Going through, think this is not correctly named/consistent -

"constructors": [
      {
        "name": 36,
        "selector": "0x5ebd88d6",
        "args": [
          {
            "name": 37,
            "type": {
              "ty": 4,
              "displayName": [
                38
              ]
            }
          }
        ],
        "docs": []
      },

ty above

ascjones · 2019-12-12T09:44:55Z

So should ty be type then? So it would be type.type

Robbepop · 2019-12-12T10:13:37Z

So should ty be type then? So it would be type.type

Well ty is an ID, so just call it id.

ascjones · 2019-12-12T11:44:35Z

Should this versioning/language field be done as a separate PR?

Yes, also can you write an issue for that so we can track and discuss there what needs to be added exactly?

#299

ascjones · 2019-12-16T13:02:19Z

Let's merge this as soon as Polkadot JS gave their acknowledgement that their JSON parser can trivially parse the output of this PR.

@jacogr are we good to go on your end?

# Conflicts: # .gitlab-ci.yml # abi/Cargo.toml # core/Cargo.toml # examples/delegator/Cargo.toml # examples/delegator/accumulator/Cargo.toml # examples/delegator/adder/Cargo.toml # examples/delegator/subber/Cargo.toml # examples/dns/Cargo.toml # examples/erc20/Cargo.toml # examples/erc721/Cargo.toml # examples/flipper/Cargo.toml # examples/incrementer/Cargo.toml # examples/multisig_plain/Cargo.toml # examples/runtime-storage/Cargo.toml # lang/macro/Cargo.toml # primitives/Cargo.toml

ascjones · 2020-03-24T12:14:21Z

bors try

bors · 2020-03-24T12:20:03Z

try

Build failed

ci/gitlab/gitlab.parity.io

ascjones · 2020-03-26T09:35:27Z

Now updated to use scale-info with new format. e.g. Incrementer:

{
  "registry": {
    "strings": [
      "Storage",
      "incrementer",
      "__ink_private",
      "__ink_storage",
      "value",
      "Value",
      "ink_core",
      "storage",
      "cell",
      "SyncCell",
      "sync_cell",
      "Key",
      "ink_primitives",
      "key",
      "new",
      "init_value",
      "i32",
      "default",
      "inc",
      "by",
      "get"
    ],
    "types": [
      {
        "composite": {
          "name": 1,
          "namespace": [
            2,
            2,
            3,
            4
          ],
          "fields": [
            {
              "name": 5,
              "type": 2
            }
          ]
        }
      },
      {
        "composite": {
          "name": 6,
          "namespace": [
            7,
            8,
            5
          ],
          "params": [
            3
          ],
          "fields": [
            {
              "name": 9,
              "type": 4
            }
          ]
        }
      },
      {
        "primitive": "i32"
      },
      {
        "composite": {
          "name": 10,
          "namespace": [
            7,
            8,
            9,
            11
          ],
          "fields": [
            {
              "name": 9,
              "type": 5
            }
          ]
        }
      },
      {
        "composite": {
          "name": 12,
          "namespace": [
            13,
            14
          ],
          "fields": [
            {
              "type": 6
            }
          ]
        }
      },
      {
        "array": {
          "len": 32,
          "type": 7
        }
      },
      {
        "primitive": "u8"
      }
    ]
  },
  "storage": {
    "struct": {
      "type": 1,
      "fields": [
        {
          "name": 5,
          "layout": {
            "struct": {
              "type": 2,
              "fields": [
                {
                  "name": 9,
                  "layout": {
                    "range": {
                      "offset": "0x0000000000000000000000000000000000000000000000000000000000000000",
                      "len": 1,
                      "elemType": 3
                    }
                  }
                }
              ]
            }
          }
        }
      ]
    }
  },
  "contract": {
    "name": 2,
    "constructors": [
      {
        "name": 15,
        "selector": "0x5ebd88d6",
        "args": [
          {
            "name": 16,
            "type": {
              "id": 3,
              "displayName": [
                17
              ]
            }
          }
        ],
        "docs": []
      },
      {
        "name": 18,
        "selector": "0x0222ff18",
        "args": [],
        "docs": []
      }
    ],
    "messages": [
      {
        "name": 19,
        "selector": "0xe7d0590f",
        "mutates": true,
        "args": [
          {
            "name": 20,
            "type": {
              "id": 3,
              "displayName": [
                17
              ]
            }
          }
        ],
        "returnType": null,
        "docs": []
      },
      {
        "name": 21,
        "selector": "0x25444afe",
        "mutates": false,
        "args": [],
        "returnType": {
          "id": 3,
          "displayName": [
            17
          ]
        },
        "docs": []
      }
    ],
    "events": [],
    "docs": []
  }
}

Robbepop

LGTM but we should merge this after type-info PR is merged into master and replace the explicit branches here.

abi/src/layout.rs

Robbepop · 2020-03-31T23:02:10Z

abi/src/lib.rs

@@ -83,3 +90,15 @@ impl InkProject {
        }
    }
 }
+
+fn hex_encode<S>(bytes: &[u8], serializer: S) -> Result<S::Ok, S::Error>


Could use some documentation with a simple example.

# Conflicts: # primitives/src/key.rs

codecov-commenter · 2020-06-11T07:55:54Z

Codecov Report

Merging #296 into master will increase coverage by 1.28%.
The diff coverage is 59.09%.

@@            Coverage Diff             @@
##           master     #296      +/-   ##
==========================================
+ Coverage   86.83%   88.11%   +1.28%     
==========================================
  Files         135      136       +1     
  Lines        5877     5924      +47     
==========================================
+ Hits         5103     5220     +117     
+ Misses        774      704      -70

Impacted Files	Coverage Δ
abi/derive/src/has_layout.rs	`74.19% <ø> (ø)`
abi/derive/src/impl_wrapper.rs	`100.00% <ø> (ø)`
core/src/env/types.rs	`23.52% <ø> (ø)`
core/src/storage/alloc/bump_alloc.rs	`88.88% <ø> (ø)`
core/src/storage/alloc/dyn_alloc.rs	`83.82% <ø> (ø)`
core/src/storage/cell/sync_cell.rs	`86.72% <0.00%> (-1.57%)`	⬇️
core/src/storage/chunk/sync_chunk/chunk.rs	`76.74% <0.00%> (-3.75%)`	⬇️
...orage/collections/binary_heap/duplex_sync_chunk.rs	`97.87% <ø> (ø)`
core/src/storage/collections/binary_heap/impls.rs	`97.47% <ø> (ø)`
core/src/storage/collections/bitvec/block.rs	`100.00% <ø> (ø)`
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13eb43e...a8f3ad6. Read the comment docs.

examples/flipper/.ink/abi_gen/metadata.json

examples/flipper/something/.rustc_info.json

ascjones · 2020-06-12T13:51:38Z

I know the current storage module is being replaced, but I have fixed the compilation errors anyway

ascjones · 2020-06-12T14:01:39Z

Latest incrementer metadata.json:

{
  "registry": {
    "strings": [
      "incrementer",
      "__ink_private",
      "__ink_storage",
      "Storage",
      "value",
      "ink_core",
      "storage",
      "Value",
      "cell",
      "sync_cell",
      "SyncCell",
      "ink_primitives",
      "key",
      "Key",
      "new",
      "init_value",
      "i32",
      "default",
      "inc",
      "by",
      "get"
    ],
    "types": [
      {
        "path": [
          1,
          1,
          2,
          3,
          4
        ],
        "composite": {
          "fields": [
            {
              "name": 5,
              "type": 2
            }
          ]
        }
      },
      {
        "path": [
          6,
          7,
          5,
          8
        ],
        "params": [
          3
        ],
        "composite": {
          "fields": [
            {
              "name": 9,
              "type": 4
            }
          ]
        }
      },
      {
        "primitive": "i32"
      },
      {
        "path": [
          6,
          7,
          9,
          10,
          11
        ],
        "composite": {
          "fields": [
            {
              "name": 9,
              "type": 5
            }
          ]
        }
      },
      {
        "path": [
          12,
          13,
          14
        ],
        "composite": {
          "fields": [
            {
              "type": 6
            }
          ]
        }
      },
      {
        "array": {
          "len": 32,
          "type": 7
        }
      },
      {
        "primitive": "u8"
      }
    ]
  },
  "storage": {
    "struct": {
      "type": 1,
      "fields": [
        {
          "name": 5,
          "layout": {
            "struct": {
              "type": 2,
              "fields": [
                {
                  "name": 9,
                  "layout": {
                    "range": {
                      "offset": "0x0000000000000000000000000000000000000000000000000000000000000000",
                      "len": 1,
                      "elemType": 3
                    }
                  }
                }
              ]
            }
          }
        }
      ]
    }
  },
  "contract": {
    "name": 1,
    "constructors": [
      {
        "name": 15,
        "selector": "0x5ebd88d6",
        "args": [
          {
            "name": 16,
            "type": {
              "id": 3,
              "displayName": [
                17
              ]
            }
          }
        ],
        "docs": []
      },
      {
        "name": 18,
        "selector": "0x0222ff18",
        "args": [],
        "docs": []
      }
    ],
    "messages": [
      {
        "name": 19,
        "selector": "0xe7d0590f",
        "mutates": true,
        "args": [
          {
            "name": 20,
            "type": {
              "id": 3,
              "displayName": [
                17
              ]
            }
          }
        ],
        "returnType": null,
        "docs": []
      },
      {
        "name": 21,
        "selector": "0x25444afe",
        "mutates": false,
        "args": [],
        "returnType": {
          "id": 3,
          "displayName": [
            17
          ]
        },
        "docs": []
      }
    ],
    "events": [],
    "docs": []
  }
}

Robbepop

LGTM

Robbepop · 2020-06-13T09:21:18Z

abi/src/specs.rs

-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn construct_selector_must_serialize_to_hex() {
-        // given
-        let name = "foo";
-        let cs: ConstructorSpec<MetaForm> = ConstructorSpec {
-            name,
-            selector: 123_456_789u32.to_be_bytes(),
-            args: Vec::new(),
-            docs: Vec::new(),
-        };
-        let mut registry = Registry::new();
-
-        // when
-        let json = serde_json::to_string(&cs.into_compact(&mut registry)).unwrap();
-
-        // then
-        assert_eq!(
-            json,
-            r#"{"name":1,"selector":"[\"0x07\",\"0x5B\",\"0xCD\",\"0x15\"]","args":[],"docs":[]}"#
-        );
-    }


Was this test moved or removed?

ascjones added 7 commits December 10, 2019 11:47

Move spec tests to separate file

6175cda

Add contract spec serialization test

8cd1c8b

Camel case field names

89df1a3

Encode selector as hex string

f1e75b1

Move specs module back to top level

65da756

Move layout test to tests.rs

7aeacca

Layout externally tagged enums & camelCase

d64facd

ascjones requested review from jacogr and Robbepop December 10, 2019 14:13

This was referenced Dec 10, 2019

Standardize JSON format type-metadata/type-metadata#36

Closed

Contract metadata format #294

Closed

Format code

73118f0

jacogr mentioned this pull request Dec 10, 2019

Support adjusted contracts ABI polkadot-js/api#1660

Closed

Robbepop previously approved these changes Dec 10, 2019

View reviewed changes

jacogr mentioned this pull request Dec 11, 2019

Updated ABI definitions & parsing polkadot-js/api#1662

Merged

Rename TypeSpec::ty -> TypeSpec::id

de1e257

ascjones dismissed Robbepop’s stale review via de1e257 December 12, 2019 11:31

ascjones mentioned this pull request Dec 12, 2019

Additional contract metadata #299

Closed

11 tasks

ascjones added 2 commits March 20, 2020 17:16

Convert layout to using scale-info

9e3c8f3

ascjones changed the title ~~Standardize metadata format~~ New metadata format Mar 23, 2020

Fix scale-info optional dependencies

43a38d2

bors bot added a commit that referenced this pull request Mar 24, 2020

Try #296:

830d2a6

Merge branch 'master' into aj-metadata

49e5d32

Fmt

557f45e

Merge branch 'master' into aj-metadata

66b0d3b

This was referenced Mar 30, 2020

Latest contract metadata polkadot-js/api#2118

Merged

Merge type id and definition paritytech/scale-info#3

Merged

Fix up straggling type-metadata dependency

df2bb60

Robbepop approved these changes Apr 1, 2020

View reviewed changes

Merge branch 'master' into aj-metadata

3301ce0

# Conflicts: # primitives/src/key.rs

ascjones added 2 commits June 11, 2020 09:00

scake-info master

aa27877

Use scale-info from crates.io

0dd1e32

Robbepop requested changes Jun 12, 2020

View reviewed changes

examples/flipper/.ink/abi_gen/metadata.json Outdated Show resolved Hide resolved

examples/flipper/something/.rustc_info.json Outdated Show resolved Hide resolved

ascjones added 3 commits June 12, 2020 14:23

oops

8e82661

Fix compilation errors

9d21bcd

Fmt

367ba7e

ascjones added 2 commits June 12, 2020 14:53

Fix examples manifests

1698489

Remove redundant into()

a8f3ad6

Robbepop approved these changes Jun 13, 2020

View reviewed changes

Robbepop merged commit 5a7b7e5 into master Jun 13, 2020

Robbepop deleted the aj-metadata branch June 13, 2020 09:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New metadata format #296

New metadata format #296

ascjones commented Dec 10, 2019 •

edited

codecov-io commented Dec 10, 2019 •

edited

Robbepop commented Dec 10, 2019

Robbepop left a comment

jacogr commented Dec 11, 2019

Robbepop commented Dec 11, 2019 •

edited

jacogr commented Dec 11, 2019

ascjones commented Dec 11, 2019

Robbepop commented Dec 11, 2019

jacogr commented Dec 12, 2019 •

edited

ascjones commented Dec 12, 2019

Robbepop commented Dec 12, 2019

ascjones commented Dec 12, 2019

ascjones commented Dec 16, 2019

ascjones commented Mar 24, 2020

bors bot commented Mar 24, 2020

ascjones commented Mar 26, 2020 •

edited

Robbepop left a comment

Robbepop Mar 31, 2020

codecov-commenter commented Jun 11, 2020 •

edited

ascjones commented Jun 12, 2020

ascjones commented Jun 12, 2020

Robbepop left a comment

Robbepop Jun 13, 2020

New metadata format #296

New metadata format #296

Conversation

ascjones commented Dec 10, 2019 • edited

codecov-io commented Dec 10, 2019 • edited

Codecov Report

Robbepop commented Dec 10, 2019

Robbepop left a comment

Choose a reason for hiding this comment

jacogr commented Dec 11, 2019

Robbepop commented Dec 11, 2019 • edited

jacogr commented Dec 11, 2019

ascjones commented Dec 11, 2019

Robbepop commented Dec 11, 2019

jacogr commented Dec 12, 2019 • edited

ascjones commented Dec 12, 2019

Robbepop commented Dec 12, 2019

ascjones commented Dec 12, 2019

ascjones commented Dec 16, 2019

ascjones commented Mar 24, 2020

bors bot commented Mar 24, 2020

try

Build failed

ascjones commented Mar 26, 2020 • edited

Robbepop left a comment

Choose a reason for hiding this comment

Robbepop Mar 31, 2020

Choose a reason for hiding this comment

codecov-commenter commented Jun 11, 2020 • edited

Codecov Report

ascjones commented Jun 12, 2020

ascjones commented Jun 12, 2020

Robbepop left a comment

Choose a reason for hiding this comment

Robbepop Jun 13, 2020

Choose a reason for hiding this comment

ascjones commented Dec 10, 2019 •

edited

codecov-io commented Dec 10, 2019 •

edited

Robbepop commented Dec 11, 2019 •

edited

jacogr commented Dec 12, 2019 •

edited

ascjones commented Mar 26, 2020 •

edited

codecov-commenter commented Jun 11, 2020 •

edited