Ensure correct metadata for parsing (archive nodes, historical queries)

[OrmEmbaar]

There are several blocks throughout the chain that produce the following error when you try to fetch them:

2019-04-16 20:34:56        RPC-CORE: getBlock (hash: Hash): SignedBlock:: Source is too large
ExtError: getBlock (hash: Hash): SignedBlock:: Source is too large
    at call (/Users/self/dev/polkadot/statd/node_modules/@polkadot/rpc-core/index.js:134:15)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)

This code will recreate the problem:

    const hash = new Hash(
        '0x6f6f9bba0eed8e3ae9446c37eee763f93118b52a315a7b46090453ba6288da1f'
    );
    api.rpc.chain.getBlock(hash)

Also, you will notice there is no then or catch on the getBlock promise. That is because the above error does not actually land in the catch block. It gets handled and logged by the library itself, which makes it more difficult for me to handle the error in my code.

[jacogr]

It must be the runtime upgrade blocks. (Sudo + setCode) Cannot think off hand where the error originates, but def. an issue and it should just work.

(Strangely I have pulled a number of the upgrade blocks before without issues on multiple chains, so it feels like something new - or maybe I just got lucky earlier on and it just hit some limit lately)

[OrmEmbaar]

To add a little more information, the most recent instance of this happening is just prior to block 580,000. However, prior to that, there are perhaps another 100 or so errors of this type. If you need a larger sample of block hashes that can reproduce this issue I am happy to grab those for you.

[jacogr]

I am currently sync-ing with archive mode, so will be looking at this one as well as #846 ASAP. (They have been hanging for way too long)

[jacogr]

Ok, this one is problematic... basically the staking interfaces has changed radically. It seems to fail on this extrinsic -

 meta:
       FunctionMetadata [Map] {
         'id' => <BN: 2>,
         'name' => [String: 'unbond'],
         'arguments' => Vector [
           [FunctionArgumentMetadata],
           _Type: [Function: FunctionArgumentMetadata] ],
         'documentation' => Vector [
           [String: ' Schedule a portion of the stash to be unlocked ready for transfer out after the bond'],
           [String: ' period ends. If this leaves an amount actively bonded less than'],
           [String: ' T::Currency::existential_deposit(), then it is increased to the full amount.'],
           [String: ''],
           [String: ' Once the unlock period is done, you can call `withdraw_unbonded` to actually move'],
           [String: ' the funds out of management ready for transfer.'],
           [String: ''],
           [String: ' The dispatch origin for this call must be _Signed_ by the controller, not the stash.'],
           [String: ''],
           [String: ' See also [`Call::withdraw_unbonded`].'],
           _Type: [Function: Text] ],
         _jsonMap: Map {},
         _Types:
          { id: 'U16',
            name: 'Text',
            arguments: 'Vector',
            documentation: 'Vector' } } }

The issue at this point in time is that there was no actual extrinsic of type unbond.

Basically, the API gets the latest metadata, tries to use that to decode... and fails. All is not lost, at this point the support is just lacking - for these cases you need to use the metadata available at this specific block to decode.

Which is all fine and well, however doing this automatically is problematic since you effectively need to query the block, query for metadata at blockNumber - 1 (the query result is the data at the end of the block), inject it and use it.

TL;DR As it stands there is no "quick" API fix, however can be done to work around when using it as a trawler, something along the lines of

api/packages/api/src/Base.ts

Line 390 in 4e83a7f

this._runtimeMetadata = await this._rpcBase.state.getMetadata();

(where parentHash is passed into the getMetadata, or rather the last known good block)

Related to #449 (which goes the other way)

[jacogr]

Related to #449 (in reverse)

[nobrick]

I wrote a function to switch to metadata and types to the runtime version of a given block. Unfortunately it works only for switching from newer version to older version (eg. from 1058 to 1055).

When switching from older to newer (eg. from 1055 to 1058), event decoding returns bad deserialization error.

let apiSpecVersion = undefined;

export async function updateMetadataAndTypes(api, hash) {
  const runtimeVersion = await api.rpc.state.getRuntimeVersion(hash);
  const targetVersion = runtimeVersion.specVersion;

  if (apiSpecVersion && targetVersion.gt(apiSpecVersion)) {
    // FIXME: when injecting metadata and types with a newer runtime spec
    // version, event decoding does not work any more. For workaround, use
    // a supervisor to restart the process.
    process.exit(1);
  }

  if (targetVersion.eq(apiSpecVersion)) {
    return;
  }

  console.log(
    `Updating spec version from ${apiSpecVersion} to ${targetVersion}`
  );

  apiSpecVersion = targetVersion;

  const rpcMeta = await api.rpc.state.getMetadata(hash);

  api.registerTypes(
    getSpecTypes(
      api.registry,
      api._runtimeChain,
      runtimeVersion.specName,
      runtimeVersion.specVersion
    )
  );

  api.injectMetadata(rpcMeta, false);
  api._runtimeMetadata = rpcMeta;
  api._runtimeVersion = runtimeVersion;
  api._rx.runtimeVersion = runtimeVersion;

  const version = api.runtimeVersion.specVersion;
  console.log(`Metadata and types updated to version ${version}`);
}

For example:

const hash1055 = "0xe0d2fd5baf79aac0780bf706c1e22464a6f9190ba3cdd3295d10b5ec21b77160"
const hash1058 = "0x937d98e4bbd3908b9147db7798462f8cb0453593598754eb2225b118f0b83161"

updateMetadataAndTypes(api, hash1058)
await api.query.system.events.at(hash1058) // ok

updateMetadataAndTypes(api, hash1055)
await api.query.system.events.at(hash1055) // ok

updateMetadataAndTypes(api, hash1058)
await api.query.system.events.at(hash1058) // uncaught decoding error

Any idea on how to make this also works for switching back to metadata / types with newer versions?

[Swader]

Would this help?

https://github.com/paritytech/substrate-api-sidecar/blob/master/src/ApiHandler.ts

[xlc]

We also have some code dealing with different metadata versions https://github.com/open-web3-stack/open-web3.js/blob/02c9a1f0fd02cb5bc86a48a1037492c800e2f493/packages/scanner/src/Scanner.ts#L201

[polkadot-js-bot]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.