Add option to index genesis allocations.

[danbryan]

given this file genesis.tar.gz you can see there are several accounts that are funded, and several on chain events that happen. For example, with account dydx1qz30lp48lkdhcx2uw34v3mulc6se3wmw5cqn9t You can see it was given 100000000000dv4nt and it did a MsgCreateValidator tx. Both of these should be in the database, but they do not exist in block 1. I would like to see something like genesis = true.

    # index transactions
    index-chain = true
    genesis = true
    start-block = 2876768
    end-block = -1

The logic would be, query the chunked genesis via rpc. Look at all of the @type": "/cosmos.auth.v1beta1.BaseAccount and @type": "/cosmos.vesting.v1beta1.DelayedVestingAccount and any other type of accounts. Then look for all activity they did in the genesis file.

For example, this shows an account that has genesis funding. This should count as a Receive

      "balances": [
        {
          "address": "dydx1qz30lp48lkdhcx2uw34v3mulc6se3wmw5cqn9t",
          "coins": [
            {
              "denom": "dv4tnt",
              "amount": "100000000000"
            }
          ]
        },

and this shows an account that did the MsgCreateValidator tx , this should be treated like a normal MsgCreateValidator tx.

          "body": {
            "messages": [
              {
                "@type": "/cosmos.staking.v1beta1.MsgCreateValidator",
                "description": {
                  "moniker": "StakingCabin",
                  "identity": "C0522DF992B0C407",
                  "website": "https://stakingcabin.com",
                  "security_contact": "richard@stakingcabin.com",
                  "details": "StakingCabin"
                },
                "commission": {
                  "rate": "0.050000000000000000",
                  "max_rate": "0.200000000000000000",
                  "max_change_rate": "0.010000000000000000"
                },
                "min_self_delegation": "1",
                "delegator_address": "dydx1qz30lp48lkdhcx2uw34v3mulc6se3wmw5cqn9t",
                "validator_address": "dydxvaloper1qz30lp48lkdhcx2uw34v3mulc6se3wmw3xk9w6",
                "pubkey": {
                  "@type": "/cosmos.crypto.ed25519.PubKey",
                  "key": "0oTtF/f95IlIFJrQt0+tOoSHnVBN2E9Ccft/LFLLNf8="
                },
                "value": {
                  "denom": "dv4tnt",
                  "amount": "50000000000"
                }
              }
            ],
            "memo": "3b0e63739876dc66146453a6adcfcfbb2db66009@172.31.37.50:26656",
            "timeout_height": "0",
            "extension_options": [],
            "non_critical_extension_options": []
          },
          "auth_info": {
            "signer_infos": [
              {
                "public_key": {
                  "@type": "/cosmos.crypto.secp256k1.PubKey",
                  "key": "A2/RufqJhPYoVyi/1bGOqmkcT4TiUlIhIPUAUHNHDLXS"
                },
                "mode_info": {
                  "single": {
                    "mode": "SIGN_MODE_DIRECT"
                  }
                },
                "sequence": "0"
              }
            ],
            "fee": {
              "amount": [],
              "gas_limit": "200000",
              "payer": "",
              "granter": ""
            },
            "tip": null
          },
          "signatures": [
            "czrf8DmWjQYm35U4m0FENv5dPIfgAZLgTGalxU8z1TI7q+YvKTlOsW8Tmkuk3TBC+NAYiYgekFW5fHoKhrLsfA=="
          ]
        },

[pharr117]

Potential Worflow

The genesis_chunked endpoint works like so:

The chunks are returned in the following manner:

1. Each chunk is a base64 slice of the Genesis JSON file
2. Each chunk will end in invalid JSON
3. Each chunk will need to be decoded and concatenated together to produce valid JSON for parsing

The workflow would have to look like this:

1. Get initial chunk=0
2. Loop until chunk num == total - 1 to get all chunks
3. Base64 decode every chunk
4. Concatenate all base64 decoded chunks
5. Loop through parsed genesis results and index

Difficulties/Dangers

This workflow presents a few difficulties and dangers:

1. Most Nodes return a 403 Forbidden at this endpoint. I can assume this is due to the amount of data returned in the requests and to prevent abuse of Public Nodes
   
2. This will SIGNIFICANTLY increase the RAM required for the application. From testing, I have seen a single chunk taking up 21+ mb of memory, and we would need to load every chunk into the application at runtime
   

Suggestions

I would suggest 2 potential options for this feature:

1. Provide an entirely seperate index-genesis command instead of adding this into the base index command workflow
   • This would allow running this once in an expected environment to allow for workflows that increase RAM for this single process
2. Provide an option of loading the genesis from a pre-downloaded genesis file.
   • This would allow running the command without requiring a Node to allow the endpoint response

[danbryan]

yeah, i think we should have a separate index-genesis command. I believe spacebox has this capability too, might be worth look into seeing how they decided to solve it. I think chunked_genesis api endpoint is good, pending we can count on all chains having a cosmos-sdk version that supports this. An alternative is we could use the genesis url mentioned in the chain-registry for chains.