Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mesh 1581: External Agents #978

Merged
merged 34 commits into from
May 11, 2021
Merged

Mesh 1581: External Agents #978

merged 34 commits into from
May 11, 2021

Conversation

Centril
Copy link
Contributor

@Centril Centril commented Apr 1, 2021 •
edited

changelog

new features

  • The ExternalAgents pallet has been added. The functionality is described more thoroughly in the code documentation. As a result, many things noted below, on a best-effort basis, have changed, including extrinsics, schema, events, error types, and some data types.
  • Types AgentGroup and AGId have been added.

modified API

  • SubsetRestriction (i.e. ExtrinsicPermissions & friends) is now redefined as either Whole, These(elems), and Except(elems).
  • TargetIdentity::PrimaryIssuanceAgent is now TargetIdentity::ExternalAgent, testing for if the identity is any of the external agents.
  • AuthorizationData::BecomeAgent(Ticker, AgentGroup) was added. This authorization is used for adding an external agent with a group to an asset. Similarly, AuthorizationType::BecomeAgent was added.
  • AuthIdentifier was removed. It was never actually in use.
  • Extrinsics CorporateActions::reset_caa, Asset::remove_primary_issuance_agent, and Asset::accept_primary_issuance_agent_transfer have been removed.
  • The error IrrelevantAuthorization (Portfolio pallet) was removed.
  • CorporateActions::Agent (storage item) was removed.
  • SecurityToken.primary_issuance_agent was removed.
  • SecondaryKeyNotAuthorizedForAsset was moved to the external agents pallet.

modified logic

  • Relevant extrinsics now check for agent permissions. Sometimes, this replaces other explicit checks such as from the explicit CAA or PIA concepts.
  • AuthorizationData::{TransferPrimaryIssuanceAgent, TransferCorporateActionAgent} are now always rejected by the chain. That is, these variants are deprecated awaiting future removal.

@Centril Centril added the release-notes Tag to use if PR needs to be reflected in release notes label Apr 1, 2021
@Centril Centril force-pushed the mesh-1581 branch 3 times, most recently from 0259f1d to f165612 Compare April 6, 2021 12:22
@Centril Centril force-pushed the mesh-1581 branch 5 times, most recently from 0ec2a65 to 13ac1ad Compare April 27, 2021 11:30
@Centril Centril force-pushed the mesh-1581 branch 4 times, most recently from 87253b6 to 84e5123 Compare May 4, 2021 10:09
@Centril Centril changed the title [WIP] Mesh 1581: External Agents Mesh 1581: External Agents May 5, 2021
Centril
Centril commented May 5, 2021
View reviewed changes
primitives/src/subset.rs Show resolved Hide resolved
primitives/src/subset.rs Show resolved Hide resolved
@Centril Centril requested a review from a team May 5, 2021 14:50
@Centril Centril added breaking-api-changes PR includes changes that break existing APIs storage-modification-changes PR includes changes that modify storage and requires additional logic to upgrade labels May 6, 2021
@CJP10
Copy link
Contributor

CJP10 commented May 7, 2021

LGTM! I'll do another pass over on Monday but so far it looks good.

maxsam4
maxsam4 suggested changes May 10, 2021
View reviewed changes
primitives/src/secondary_key.rs Show resolved Hide resolved
primitives/src/subset.rs Outdated Show resolved Hide resolved
primitives/src/subset.rs Outdated Show resolved Hide resolved
primitives/src/subset.rs Show resolved Hide resolved
primitives/src/subset.rs
pub struct SubsetRestriction<A: Ord>(pub Option<BTreeSet<A>>);
pub enum SubsetRestriction<A: Ord> {
/// No restrictions, the whole set.
Whole,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should get rid of Whole and have it always implied by Except(None)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per private discussion, we'll add a separate task to reconsider this since it probably requires storage migration...

Meanwhile, some notes:

  1. Using Except(BTreeSet::new()) is less storage efficient, as you need [0, 0], first encoding the variant of These and then the fact that the length is 0. Meanwhile, Whole just requires [0] since you encode the tag and then there's no data. There is no change in data size efficiency in Rust terms, as there aren't enough niches to optimize below 32 bytes in either case.
  2. Storage migration is likely needed as the data encoding, per above, has changed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed as https://polymath.atlassian.net/browse/MESH-1624.

@Centril Centril added the merge-after-ci PR can be merged after CI has completed label May 11, 2021
@poly-auto-merge poly-auto-merge bot merged commit 5291547 into develop May 11, 2021
@poly-auto-merge poly-auto-merge bot deleted the mesh-1581 branch May 11, 2021 11:02
adamdossa pushed a commit that referenced this pull request Jun 17, 2021
@Centril @adamdossa
Mesh 1581: External Agents (#978)
a7ed295 
* wip

* wip

* wip

* wip

* wip

* wip

* cargo fmt

* mesh-1581: improve docs

* wip

* wip

* wip

* wip

* move asset Trait to common

* mesh-1581: add benchmarks, mesh-1585

* fix rebase problems

* fix asset bench

* mesh-1581: add EA module docs

* mesh-1581: fix schema

* mesh-1581: reduce dependencies for EA module

* mesh-1618: add Except to SubsetRestriction

* mesh-1582: map groups to perms

* cargo fmt

* wip!

* wip

* tweak benchmarks

* tweak any EA prop logic

* fix owner migration

* EAs: use refcount for anti-orphaning

* fix cli tests

* test EA refcount more

* eas: improve internal docs

* fix cli tests more

* lattice_cmp: fix bug & add comments

* EA -> ExternalAgents
@adamdossa adamdossa mentioned this pull request Jun 17, 2021
Merged
adamdossa added a commit that referenced this pull request Jul 29, 2021
@adamdossa @CJP10
@Centril @fmiguelgarcia @vkomenda @JMoore96 @poly-auto-merge @maxsam4 @Neopallium @dependabot
Staging to v3.1.0 (#1062)
5b4a76c 
* MESH-1597: Split issuance and token creation (#981)

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* Update pallets/asset/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* WIP

* remove create_asset_and_mint

* remove create_asset_and_mint

* formatting

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* Mesh 1543/coverage bridge (#979)

* Refactor following TA

* Freeze/Unfreeze txs UT added

* Add `batch_propose_bridge_tx` UT

* CodeReview: Use constant

* CodeReview: Use `base_` prefix

* CodeREview: misc

* CodeREview: remove `box_syntax`

* MESH-1603: allow portfolio custodians unilaterally quit (#977)

* quit_portfolio_custody WIP

* refactoring fix

* fixed quit test

* fixed the quit benchmark

* update comment

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

* minor change

* minor change

Co-authored-by: Joel Moore <47923231+JMoore96@users.noreply.github.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: poly-auto-merge[bot] <65769705+poly-auto-merge[bot]@users.noreply.github.com>

* MESH-1598/ use expect/assert in benchmarks (#986)

* removed '?' in benchmarks and changed ensure to assert

* minor changes

* minor change

* fixed linting issue

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* comment correction

Co-authored-by: CJP10 <cjpet10@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* minor change

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* fixed linting and made minor changes

* minor changes and linting

* minor changes

* minor change

* minor changes

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: CJP10 <cjpet10@gmail.com>

* Fixed a bug preventing some operators from changing commission (#987)

* Fixed commission change bug

* Update pallets/runtime/tests/src/staking/mod.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: poly-auto-merge[bot] <65769705+poly-auto-merge[bot]@users.noreply.github.com>

* Add ITN details to README (#989)

Co-authored-by: poly-auto-merge[bot] <65769705+poly-auto-merge[bot]@users.noreply.github.com>

* MESH-1593/Fix `payout_stakers` benchmarks (#988)

* WIP

* WIP

* Add audit report (#995)

* MESH-1542/Improve code coverage for Identity module (#990)

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* Update pallets/runtime/tests/src/identity_test.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* WIP

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* Mesh 1581: External Agents (#978)

* wip

* wip

* wip

* wip

* wip

* wip

* cargo fmt

* mesh-1581: improve docs

* wip

* wip

* wip

* wip

* move asset Trait to common

* mesh-1581: add benchmarks, mesh-1585

* fix rebase problems

* fix asset bench

* mesh-1581: add EA module docs

* mesh-1581: fix schema

* mesh-1581: reduce dependencies for EA module

* mesh-1618: add Except to SubsetRestriction

* mesh-1582: map groups to perms

* cargo fmt

* wip!

* wip

* tweak benchmarks

* tweak any EA prop logic

* fix owner migration

* EAs: use refcount for anti-orphaning

* fix cli tests

* test EA refcount more

* eas: improve internal docs

* fix cli tests more

* lattice_cmp: fix bug & add comments

* EA -> ExternalAgents

* MESH-1587: move Alcyone & general genesis identity balances to the bridge (#966)

* updated general genesis

* missing instance fix

* changed develop runtime pallet order to fix 0 balance

* added CDD claims to fix a bridge genesis error

* typo

* fixed TooLong in benchmarks

* fixed ComplianceRequirementTooComplex in compliance benchmark

* updated testnet genesis config

* using complete_txs and README updates

* changed module boot order in testnet, removed dead code

* added testnet JSON genesis config

* fmt

* Apply suggestions from code review

Co-authored-by: Mudit Gupta <guptamudit@ymail.com>

* removed a comment

* removed testnet JSONs

* minor changes

* Give perms to genesis secondary keys

* Make genesis accounts rich

* Fun more genesis accounts

* minor refactoring

* removed unused import

Co-authored-by: Mudit Gupta <guptamudit@ymail.com>

* polymesh_schema: fix missing changes due to EA (#1005)

* polymesh_schema: fix missing changes due to EA

* Update polymesh_schema.json

* Fix bug in primary key (PK) rotation where key already being attached to DID was allowed. (#1006)

Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* Mesh 1196/refactor JS integration tests to TS (#915)

* Typed init.ts and modifying tsconfig

* modified init and create identities scripts

* added 1_poly_transfer.ts and helper files

* added 2_key_management .ts and helper files

* added 3_auth_join_did.ts and minor changes

* added 4_permission_management.ts, helper files, and formatted scripts

* added 5_claim_management.ts

* added 6_create_assets.ts

* minor change to fix complianceRequirement issue

* added 7_create_claim_compliance.ts and helper file

* added 8_asset_minting.ts

* added 9_bridge_transfer.ts and helper file

* added 10_governance_management.ts and help files

* added 11_A_settlement.ts and helper file

* added 11_B_settlement.ts

* general improvements

* improved error handling for identity script and helper

* minor changes

* improved error handling for poly_transfer script and asset helper

* improved error handling for key_management script and helper

* add catch to promises

* added @polkadot/typegen

* changes based around @polkadot/typegen

* minor change

* removed handle function

* minor changes

* Added Api Singleton Class and removed Api as a parameter from the scripts

* removed usage of unknown

* standardized on using signer and formatting

* removed .catch from lower levels

* added type safety to catch clause in scripts

* added offchain_worker_script.ts and made minor changes

* minor changes to comments and package.json

* minor change in asset_helper

* minor change

* removed old JS test scripts

Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* MESH-1623/add memo field to portfolio (#1004)

* portfolio needs `Ord` on `Memo`

* Add memo field to portfolio.move_portfolio_funds.

TODO: Add tests with a memo.
TODO: Update docs.

* Add missing `memo` field in corporate-actions pallet

* code cleanup.

* Remove TODO comment.

* cargo fmt - fixup corporate-actions.

* Fix schema for `MovePortfolioItem` struct.

* Tests for `move_portfolio_funds` with memo.

Also checks for `MovedBetweenPortfolios` event.

* Add note about memo to `move_portfolio_funds` docs.

* MESH-1620/refactor (#1011)

* Move stand-alone fn to methods.

* Move with_call_metadata into dispatch_call.

* Refactor batch and batch_atomic.

* Refactor for loop into filter_map.

Not sure if this is more readable then the for loop.

* cargo fmt

Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* ExternalAgents: Fix schema + `fn agent_permissions` (#1012)

Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* MESH-1619/Allow rescheduling failed Settlements (#996)

* first draft

* first draft

* Update pallets/settlement/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* Update pallets/settlement/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* Update pallets/settlement/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* WIP

* WIP

* WIP

* WIP

* Update pallets/settlement/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* WIP

* WIP

* WIP

* WIP

* Update pallets/settlement/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* Update pallets/settlement/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* Update pallets/runtime/tests/src/settlement_test.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* WIP

* WIP

* Update pallets/settlement/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* Update pallets/settlement/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* WIP

* WIP

* WIP

* WIP

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* MESH-1621/Script to create snapshot of all user balances  (#1000)

* created script to get user balances and store them in a csv file

* added secondary keys and progress bar

* removed CSV and duplication while adding new field isPrimaryKey

Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* MESH-1622/ create script to distribute POLYX (#1010)

* added script to distribute POLYX

* minor changes

* Bump lodash from 4.17.15 to 4.17.21 in /scripts/event-listener (#998)

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.15...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* MESH-1620/improve batch functions (EventCounts, ErrorAt) (#1003)

* API change - Return a vector of event counts from the batch functions.

* code cleanup.

* Add `EventCounts` and `ErrorAt` types to schema.

* Fix ident.

Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* Fix CI integration test. (#1020)

* MESH-1628/issue with duplicate secondary keys (#1018)

* Add more tests for `remove_secondary_key`

* Try adding the same key multiple times with different permissions.

* Add adding dup key with `add_secondary_keys_with_authorization`

* Improve `remove_secondary_keys_test`

* Add test for `set_permission_to_signer` with many tokens.

* cargo fmt

* Use reverse order for bulk permissions.

* Fix issue with remove_secondary_keys.

* Copy test from MESH-1640.

* Fix `PartialEq` impl for `SecondaryKey`

This fixes the dedup of the secondary_keys list.

* Only charge the fee after all checks.

* Check for duplicate secondary key before adding it.

This removes the need for the sort/dedup in `add_secondary_keys`.
Add tests for `join_identity`.

* cargo fmt

* Change `PartialEq` back.

The `PartialEq` impl is not used any more for preventing duplicate secondary keys.

* fix comment.

* Use default `PartialEq` impl.  Remove `Ord`

* code cleanup.

* Reduce redundant work.

* Some test cleanup.

* cargo fmt

Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* MESH-1637/SecondaryKeyPermissionsUpdated event emits new permissions in both params (#1017)

* WIP

* WIP

* WIP

* WIP

* Fix linting errors

Co-authored-by: Adam Dossa <adam.dossa@gmail.com>

* MESH-1642/Debug: Fuzz Settlement Tests (#1027)

* WIP

* WIP

* MESH-1638/Add a reverse map to allow an easy lookup from IdentityId to list of assets for which the identity is an EA (#1021)

* WIP

* WIP

* WIP

* Update pallets/external-agents/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* WIP

* WIP

* WIP

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>

* Update spec version and release version

* Fix migrations (#1061)

Co-authored-by: CJP10 <cjpet10@gmail.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Francisco Miguel García <fmiguelgarcia@users.noreply.github.com>
Co-authored-by: Vladimir Komendantskiy <komendantsky@gmail.com>
Co-authored-by: Joel Moore <47923231+JMoore96@users.noreply.github.com>
Co-authored-by: poly-auto-merge[bot] <65769705+poly-auto-merge[bot]@users.noreply.github.com>
Co-authored-by: Mudit Gupta <guptamudit@ymail.com>
Co-authored-by: Robert Gabriel Jakabosky <rjakabosky+github@neoawareness.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Robert Gabriel Jakabosky <rjakabosky+neopallium@neoawareness.com>
@adamdossa adamdossa mentioned this pull request Jul 29, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adamdossa adamdossa adamdossa approved these changes

@maxsam4 maxsam4 maxsam4 approved these changes

@CJP10 CJP10 CJP10 approved these changes

Assignees
No one assigned
Labels
breaking-api-changes PR includes changes that break existing APIs merge-after-ci PR can be merged after CI has completed release-notes Tag to use if PR needs to be reflected in release notes storage-modification-changes PR includes changes that modify storage and requires additional logic to upgrade
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@Centril @CJP10 @adamdossa @maxsam4