Da 179/upgrade 5.0.0 #742

polymath-eric · 2022-04-29T02:05:32Z

Description

Upgrades the SDK for the 5.0.0 chain. The chain metatdata has made some changes. Notably the chain types are prefixed with PolymeshPrimitive or PalletSomePallet. While this makes the code more verbose, it does eliminate a lot of snake case in the Codecs which cleans up the eslint disable comments. Overall I think it reads cleaner, and the long names don't bother me that much besides the ones the stutter (e.g. Compliance Manager ones)

Breaking Changes

I think the external API hasn't changed, but would need to double check to be sure. For the most part the breaking changes are with the interaction with polkadot.js

JIRA Link

https://polymath.atlassian.net/browse/DA-260
https://polymath.atlassian.net/browse/DA-179

Checklist

Updated the Readme.md (if required) ?

polymath-eric

I found this to be a pretty challenging upgrade. I had to fill in some holes in my knowledge with how polkadot schemas work.

Test coverage isn't perfect, and I think some conversion and mock functions need to be touched up in their params, but overall this seems to be working.

I'll pick this up when I am back Thursday, but if someone else picks it up and has questions feel free to Slack me.

polymath-eric · 2022-04-29T02:11:21Z

package.json

-    "generate:defs": "ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --package polymesh-types --input ./src/polkadot",
-    "generate:meta": "ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --package polymesh-types --endpoint ws://localhost:9944 --output ./src/polkadot --strict",
+    "generate:defs": "ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --endpoint metadata.json --package polymesh-types --input ./src/polkadot/",
+    "generate:meta": "ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --endpoint metadata.json --package polymesh-types --output ./src/polkadot --strict",


It seemed like one of these commands didn't like the ws endpoint with the chain.

I uploaded a metatdata.json to use to drive: https://drive.google.com/file/d/1137jNjQZYdSNOkXtk42LYv8yD-kOkcHP/view?usp=sharing

I used: curl -H "Content-Type: application/json" -d '{"id":"1", "jsonrpc":"2.0", "method": "state_getMetadata", "params":[]}' http://localhost:9933 > metadata.json to get it.

What was the specific problem with using the endpoint?

Also, if it is unavoidable to use this approach, then we should add a script that fetches the metadata and saves it into the json file, and add that to the overall process, deleting it at the end (or just .gitignore it)

It kept on thinking it was a file

Error: Cannot find module '/Users/eric/polymesh-sdk/ws:/localhost:9944'

I had to check again since the error stripped a / it seems. the meta one seems fine though

Thats with:

"generate:defs": "ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --endpoint metadata.json --package polymesh-types --input ./src/polkadot/"

Created a task to track this

I ended up removing the task because there are other scripts that would benefit from having the metadata stored in a file. I created a task to automate fetching and storing the metadata as part of the typegen process

I have fetch-metadate as part of the typegen now, and it will get .gitignored

It relies on curl being present, but I think its reasonable to expect being in the path in this day and age on a dev machine.

Might be nice to use a node dependency though.

I'll create a task for that

package.json

polymath-eric · 2022-04-29T02:18:25Z

scripts/generateTxTags.js

@@ -27,7 +28,7 @@ websocket.onmessage = message => {
  const registry = new TypeRegistry();

  const metadata = new Metadata(registry, JSON.parse(message.data).result);
-  const modules = metadata.asLatest.modules;
+  const modules = metadata.asLatest.pallets;


There are other changes we should investigate: https://polymath.atlassian.net/browse/DA-305

src/api/entities/Venue/index.ts

polymath-eric · 2022-04-29T02:22:51Z

src/api/entities/common/namespaces/Authorizations.ts

+
+  /**
+   * @hidden
+   * Needed to handle snake case from rpc request


This took me a long time to figure out what was happening. I was in a circle of fixing things while breaking other things for longer than I'd like to admit.

This works, but really we should figure out why RPC stuff didn't get updated. I put the word basic in for copy pasta for snake case stuff.

I am hoping if we figure out the generate script stuff it will eliminate the need for these (more in conversion).

If we can't fully get rid of the schema then I think we should create some sort of manual mapping from schema types to metadata types, and then run a script that transforms the rpc portion of the schema with said mapping

The schema does play a role, but the actual types used are coming from augment-api-rpc and augment-api-query.

I think its weird the RPC one looks at polymesh-types/polymesh for its imports, the query one gets them from @polkadot/types/lookup.

There might be a reason for the divergence, but I'd think we can transform the rpc file like you are suggesting.

In that file I changed Authorization to PolymeshPrimitivesAuthorization by hand and that seemed to work.

I think changing that file is the right approach. I am going to give that a shot

src/api/procedures/setTransferRestrictions.ts

polymath-eric · 2022-04-29T02:37:24Z

src/base/Context.ts

  /**
   *  @hidden
   *
   * Creates an instance of a type as registered in the polymeshApi instance
   */
-  public createType<K extends keyof InterfaceTypes>(type: K, params: unknown): InterfaceTypes[K] {
+  public createType<K extends keyof InterfaceTypes>(type: K, params: unknown): any {


They made a change that was supposed to make the createType API smarter, but now InterfaceTypes[K] exceeds TypeScript stack size, and might be infinite recursion even.

You can copy the signature of the original createType to avoid this use of any

monitz87

GJ overall, this was a massive endeavor

monitz87 · 2022-05-04T03:50:34Z

package.json

-    "generate:defs": "ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --package polymesh-types --input ./src/polkadot",
-    "generate:meta": "ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --package polymesh-types --endpoint ws://localhost:9944 --output ./src/polkadot --strict",
+    "generate:defs": "ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --endpoint metadata.json --package polymesh-types --input ./src/polkadot/",
+    "generate:meta": "ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --endpoint metadata.json --package polymesh-types --output ./src/polkadot --strict",


Also, if it is unavoidable to use this approach, then we should add a script that fetches the metadata and saves it into the json file, and add that to the overall process, deleting it at the end (or just .gitignore it)

monitz87 · 2022-05-04T04:05:30Z

src/api/entities/Asset/Compliance/Requirements.ts

+import {
+  PolymeshPrimitivesComplianceManagerAssetCompliance,
+  PolymeshPrimitivesConditionTrustedIssuer,
+} from '@polkadot/types/lookup';
+import { AssetComplianceResult } from 'polymesh-types/types';


Why are some types being changed but others aren't?

We should try sticking with only the new types. Try removing the types part of the schema (since we still need the rpc part) and see how it goes. It should fix issues with mismatched types like VenueDetails which is Bytes in the metadata but Text in the schema.

Let me know how it goes. If it works we can modify the fetch-definitions script to remove the types

Anything coming from RPC wasn't updated. Will try out your suggestion and see how it goes.

After some experimenting it seems like we should be able to take out the types, but we'll need to figure out CountryCode generate section since that needs types.

We'll also need to go through all the createType calls and make sure none are using the old types either.

I added in a script that deals with Authorization and that seems to work.

The Result ones seem to live in src/polkadot/polymesh/types.ts. I will take a look in the morning to see if they can get similar treatment.

Seems like the script isn't good after all. The types and tests are happy, but when I tried it out it seems like the snake case type is returned (undefined errors). Using snake case for the field resolves it, so for now I am going to switch back to having another method for handling RPC authorization stuff.

The country code stuff should be able to be generated from the metadata. I'll create a task for that

scripts/generateTxTags.js

scripts/transactions.json

src/api/entities/Asset/TransferRestrictions/TransferRestrictionBase.ts

src/utils/conversion.ts

Adds a post process script that will transform the augment-api-rpc so that Authorization and IdentityId will be brough in from `@polkadot/types/lookup`

Also fix compilation errors that arise in sandbox

…data-v14 Adapt TxTag generating script to metadata v14

…ymesh-sdk into feat/DA-180-expose-tx-index

Also fixed documentHash conversion util tests

BREAKING CHANGE: 🧨 throw an error instead of a warning when the chain versions aren't within the expected ranges

Feat/da 180 expose tx index

monitz87 · 2022-05-26T14:07:22Z

scripts/fetchMetadata.sh

@@ -0,0 +1 @@
+curl -H "Content-Type: application/json" -d '{"id":"1", "jsonrpc":"2.0", "method": "state_getMetadata", "params":[]}' http://localhost:9933 > metadata.json


Suggested change

curl -H "Content-Type: application/json" -d '{"id":"1", "jsonrpc":"2.0", "method": "state_getMetadata", "params":[]}' http://localhost:9933 > metadata.json

curl -H "Content-Type: application/json" -d '{"id":"1", "jsonrpc":"2.0", "method": "state_getMetadata", "params":[]}' http://localhost:9933 > metadata.json

monitz87 · 2022-05-26T14:19:18Z

src/api/procedures/setTransferRestrictions.ts

@@ -1,32 +1,32 @@
 import {
  BTreeSetStatType,
-  BTreeSetStatUpdate,
-  BTreeSetTransferCondition,
+  PolymeshPrimitivesTransferComplianceTransferCondition,
 } from '@polkadot/types/lookup';
 import BigNumber from 'bignumber.js';
 import { TransferCondition, TxTag, TxTags } from 'polymesh-types/types';


TxTag and TxTags should be imported from ~/types

monitz87 · 2022-05-26T14:23:23Z

src/api/procedures/setTransferRestrictions.ts

+    if (type === TransferRestrictionType.Count) {
+      condition = { type, value: (r as CountTransferRestrictionInput).count };
+    } else {
+      condition = { type, value: (r as PercentageTransferRestrictionInput).percentage };
+    }


What you can do is

let value: BigNumber; if ('count' in inputCondition) { value = inputCondition.count; } else { value = inputCondition.percentage; }

You could stick this in a ternary as well but I'd rather not

* refactor: 💡 Eliminate RPC specific conversion functions Modify the schema object to make its types compatible with the rest of the api. This elminates snake case conversion functions to DRY the code base * fix: 🐛 Fix asset.settlement.canTransfer method Update types to use PolymeshPrimitivesIdentityId instead of IdentityId

…dation feat: 🎸 validate chain spec version when connecting

sonarcloud · 2022-05-30T14:53:45Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

100.0% Coverage
0.0% Duplication

monitz87 · 2022-05-30T20:49:53Z

🎉 This PR is included in version 14.0.0-alpha.15 🎉

The release is available on:

Your semantic-release bot 📦🚀

monitz87 · 2022-05-30T23:03:43Z

🎉 This PR is included in version 14.0.0-alpha.16 🎉

The release is available on:

Your semantic-release bot 📦🚀

monitz87 · 2022-05-31T15:01:30Z

🎉 This PR is included in version 15.0.0-alpha.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

monitz87 · 2022-07-12T19:12:14Z

🎉 This PR is included in version 15.0.0-beta.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

monitz87 · 2022-08-10T17:02:36Z

🎉 This PR is included in version 15.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

polymath-eric added 13 commits April 14, 2022 13:10

feat: 🎸 Upgrade polkadot SDK along with 5.0.0 chain types

2e9ee2c

feat: 🎸 Refactor code to use new trasnfer restrictions api

37b146a

feat: 🎸 Use names prefixed with PolymeshPrimitives and Pallet

7874b57

test: 💍 Make tests mostly compile

65490ec

test: 💍 Fix majority of broken tests

591227c

test: 💍 Fix transfer procedures

265c115

feat: 🎸 Update granular permissions results for new types

fe2b072

Merge remote-tracking branch 'origin/alpha' into DA-179/upgrade-5.0.0

173ebda

feat: 🎸 WIP Update transfer restrictions to create stats

32d7d94

test: 💍 Add test for addTransferRestriction

0afa552

refactor: 💡 Clean up syntax and comments

410c5aa

feat: 🎸 Use batching fully for add restriction

5bd29d5

feat: 🎸 Use proper batching for setTransferRestriction

f895ec8

polymath-eric requested review from monitz87 and VictorVicente as code owners April 29, 2022 02:05

polymath-eric changed the base branch from master to alpha April 29, 2022 02:05

polymath-eric marked this pull request as draft April 29, 2022 02:07

polymath-eric commented Apr 29, 2022

View reviewed changes

polymath-eric added 7 commits April 28, 2022 22:42

chore: 🤖 Add coverage flag back into package.json

3269643

refactor: 💡 Resolve minor code smells

33bf6bf

test: 💍 Increase conversion test coverage

194328c

test: 💍 Improve transfer restriction test coverage

2763d30

test: 💍 Remove .only in setTransferRestriction test

41d2c8e

test: 💍 Cover setTransferRestriction procedure

172ccb1

chore: 🤖 Fix lint errors

ac3f91d

monitz87 suggested changes May 9, 2022

View reviewed changes

polymath-eric added 4 commits May 9, 2022 15:42

refactor: 💡 Address PR comments

518b798

chore: 🤖 Add augmented rpc post process for Authroization types

a0fd403

Adds a post process script that will transform the augment-api-rpc so that Authorization and IdentityId will be brough in from `@polkadot/types/lookup`

test: 💍 Increase test coverage

9d6d3e3

refactor: 💡 Clean up code smells

fbf5d4c

monitz87 and others added 8 commits May 23, 2022 22:40

chore: 🤖 update ts-node

3aedadc

feat: 🎸 expose transaction index within block

e6c6983

Also fix compilation errors that arise in sandbox

fix: 🐛 disregard pallets without calls

af684ae

Merge pull request #750 from PolymathNetwork/chore/DA-305-txtags-meta…

0635f8b

…data-v14 Adapt TxTag generating script to metadata v14

Merge branch 'DA-179/upgrade-5.0.0' of github.com:PolymathNetwork/pol…

12a5d16

…ymesh-sdk into feat/DA-180-expose-tx-index

fix: 🐛 use proper type for document URIs

615782a

Also fixed documentHash conversion util tests

feat: 🎸 validate chain spec version when connecting

af76f93

BREAKING CHANGE: 🧨 throw an error instead of a warning when the chain versions aren't within the expected ranges

Merge pull request #751 from PolymathNetwork/feat/DA-180-expose-tx-index

a9a3f26

Feat/da 180 expose tx index

monitz87 suggested changes May 26, 2022

View reviewed changes

refactor: 💡 remove code smells

4c371f7

polymath-eric marked this pull request as ready for review May 26, 2022 21:33

polymath-eric and others added 7 commits May 26, 2022 18:22

refactor: 💡 Address PR comments

a69c9a1

feat: 🎸 log a warning for minor and patch version mismatch

30dd691

Merge pull request #752 from PolymathNetwork/feat/DA-324-version-vali…

a98d095

…dation feat: 🎸 validate chain spec version when connecting

test: 💍 Add test coverage for BTree conversion utils

9f055a6

chore: 🤖 Remove unused function createMockRpcDispatchable names

f2da942

test: 💍 Fix setTransferRestriction test

8c53ea9

monitz87 approved these changes May 30, 2022

View reviewed changes

polymath-eric merged commit 2450467 into alpha May 30, 2022

polymath-eric deleted the DA-179/upgrade-5.0.0 branch May 30, 2022 20:32

monitz87 added the released on @alpha label May 30, 2022

monitz87 added the released on @beta label Jul 12, 2022

monitz87 added the released label Aug 10, 2022

		@@ -0,0 +1 @@
		curl -H "Content-Type: application/json" -d '{"id":"1", "jsonrpc":"2.0", "method": "state_getMetadata", "params":[]}' http://localhost:9933 > metadata.json

Da 179/upgrade 5.0.0 #742

Da 179/upgrade 5.0.0 #742

Conversation

polymath-eric commented Apr 29, 2022

Description

Breaking Changes

JIRA Link

Checklist

polymath-eric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

polymath-eric May 5, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

monitz87 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented May 30, 2022 • edited

monitz87 commented May 30, 2022

monitz87 commented May 30, 2022

monitz87 commented May 31, 2022

monitz87 commented Jul 12, 2022

monitz87 commented Aug 10, 2022

polymath-eric May 5, 2022 •

edited

sonarcloud bot commented May 30, 2022 •

edited