Add dense_pull_ledger_diff rpc method

[Dengjianping]

Signed-off-by: Dengjianping djptux@gmail.com

Description

With base64, the size of response can be reduced 30% around.

• CI should support dense_pull_ledger_diff(will support in PR Add pull-ledger-diff ci test for calamari #919)
• About the performance, which should be equal to pull_ledger_diff, or not worse too much.
  • pulling time
  • the performance on base64.encode
  • client decoding time

relates to OR closes: #XXXX

---

Before we can approve this PR for merge, please make sure that all the following items have been checked off:

• Linked to Github issue with discussion and accepted design OR have an explanation in the PR that describes this work.
• Added one label out of the L- group to this PR
• Added one or more labels from the A- and C- groups to this PR
• Explicitly labelled A-calamari, A-dolphin and/or A-manta if your changes are meant for/impact either of these (CI depends on it)
• This PR is targeted against the current Milestone ( otherwise discuss if it can be added in time)
• Re-reviewed Files changed in the Github PR explorer.

Situational Notes:

• If adding functionality, write unit tests!
• If importing a new pallet, choose a proper module index for it, and allow it in BaseFilter. Ensure every extrinsic works from front-end. If there's corresponding tool, ensure both work for each other.
• If needed, update our Javascript/Typescript APIs. These APIs are officially used by exchanges or community developers.
• If modifying existing runtime storage items, make sure to implement storage migrations for the runtime and test them with try-runtime. This includes migrations inherited from upstream changes, and you can search the diffs for modifications of #[pallet::storage] items to check for any.
• If runtime changes, need to update the version numbers properly:
  • authoring_version: The version of the authorship interface. An authoring node will not attempt to author blocks unless this is equal to its native runtime.
  • spec_version: The version of the runtime specification. A full node will not attempt to use its native runtime in substitute for the on-chain Wasm runtime unless all of spec_name, spec_version, and authoring_version are the same between Wasm and native.
  • impl_version: The version of the implementation of the specification. Nodes are free to ignore this; it serves only as an indication that the code is different; as long as the other two versions are the same then while the actual code may be different, it is nonetheless required to do the same thing. Non-consensus-breaking optimizations are about the only changes that could be made which would result in only the impl_version changing.
  • transaction_version: The version of the extrinsics interface. This number must be updated in the following circumstances: extrinsic parameters (number, order, or types) have been changed; extrinsics or pallets have been removed; or the pallet order in the construct_runtime! macro or extrinsic order in a pallet has been changed. You can run the metadata_diff.yml workflow for help. If this number is updated, then the spec_version must also be updated
• Verify benchmarks & weights have been updated for any modified runtime logics

[flame4]

GOOOOOOOOOD!!

Missing fields on DensePullLedgerDiffneed to be filled

[flame4]

LGTM!

[flame4]

From our experience in indexer, the pulling time and cpu usage gonna be better with this new api.

[ghzlatarev]

Nice.
As a follow up can we also add an integration test like for pull-ledger-diff. For reference I've updated the code here #919 since we need to use governance instead of sudo.

[Dengjianping]

Nice. As a follow up can we also add an integration test like for pull-ledger-diff. For reference I've updated the code here #919 since we need to use governance instead of sudo.

Thanks!