Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PFB results in panic: UnmarshalJSON cannot decode empty bytes #1363

Closed
rootulp opened this issue Feb 9, 2023 · 9 comments
Closed

PFB results in panic: UnmarshalJSON cannot decode empty bytes #1363

rootulp opened this issue Feb 9, 2023 · 9 comments
Assignees
Labels
bug Something isn't working

Comments

@rootulp
Copy link
Collaborator

rootulp commented Feb 9, 2023

Summary of Bug

recovered: UnmarshalJSON cannot decode empty bytes
stack:
goroutine 3717 [running]:
runtime/debug.Stack()	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/cosmos/cosmos-sdk/baseapp.newDefaultRecoveryMiddleware.func1({0x1f63960, 0xc0065ba800})	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/recovery.go:71 +0x27
github.com/cosmos/cosmos-sdk/baseapp.newRecoveryMiddleware.func1({0x1f63960?, 0xc0065ba800?})
	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/recovery.go:39 +0x30
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x1f63960, 0xc0065ba800}, 0xc001344b00?)
	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/recovery.go:28 +0x37
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x1f63960, 0xc0065ba800}, 0x2d32158?)
	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/recovery.go:33 +0x5e
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()
	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/baseapp.go:639 +0xf0
panic({0x1f63960, 0xc0065ba800})
	/usr/local/go/src/runtime/panic.go:838 +0x207
github.com/cosmos/cosmos-sdk/x/params/types.Subspace.Get({{0x2d31748, 0xc000f62bf0}, 0xc000122bd0, {0x2d0e170, 0xc000f62e50}, {0x2d0e1c0, 0xc000f62ef0}, {0xc000052600, 0x4, 0x12}, ...}, ...)
	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/x/params/types/subspace.go:110 +0x307
github.com/celestiaorg/celestia-app/x/blob/keeper.Keeper.GasPerBlobByte(...)
	/celestia-app/x/blob/keeper/params.go:36
github.com/celestiaorg/celestia-app/x/blob/keeper.Keeper.PayForBlobs({{0x2d31748, 0xc000f62bf0}, {0x2d0e170, 0xc000f62ea0}, {0x2d0e170, 0x0}, {{0x2d31748, 0xc000f62bf0}, 0xc000122bd0, {0x2d0e170, ...}, ...}}, ...)
	/celestia-app/x/blob/keeper/keeper.go:60 +0x3a8
github.com/celestiaorg/celestia-app/x/blob.NewHandler.func1({{0x2d24c58, 0xc000138020}, {0x2d32158, 0xc006a22680}, {{0xb, 0x0}, {0xc006eec050, 0x9}, 0x19ed, {0xb752ea5, ...}, ...}, ...}, ...)
	/celestia-app/x/blob/handler.go:20 +0x195
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x2d24c58, 0xc000138020}, {0x2d32158, 0xc006a22680}, {{0xb, 0x0}, {0xc006eec050, 0x9}, 0x19ed, ...}, ...}, ...)
	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/baseapp.go:800 +0x7a8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc000facfc0, 0x3, {0xc000540f20, 0x141, 0x141})
	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/baseapp.go:725 +0xc45
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc000facfc0, {{0xc000540f20?, 0x20?, 0xc00dc7a460?}})
	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/abci.go:283 +0x17a
github.com/tendermint/tendermint/abci/client.(*localClient).DeliverTxAsync(0xc0012e7740, {{0xc000540f20?, 0x0?, 0xc0012e7740?}})
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/abci/client/local_client.go:93 +0x105
github.com/tendermint/tendermint/proxy.(*appConnConsensus).DeliverTxAsync(0xc00def4bc0?, {{0xc000540f20?, 0x20?, 0xb?}})
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/proxy/app_conn.go:88 +0x26
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2d25e10?, 0xc003416cc0}, {0x2d2ffe8, 0xc0011e3b20}, 0xc00163e000, {0x2d329b0, 0xc000130978}, 0x19ec?)
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/state/execution.go:390 +0x822
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0033f33d0, 0x7}}, {0xc0033f33d7, 0x9}, 0x1, 0x19ec, {{0xc00c1e5cc0, ...}, ...}, ...}, ...)
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/state/execution.go:210 +0x171
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc00136aa80, 0x19ed)
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:1668 +0xb2d
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc00136aa80, 0x19ed)
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:1576 +0x2ff
github.com/tendermint/tendermint/consensus.(*State).handleCompleteProposal(0xc00136aa80, 0x454?)
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:1957 +0x399
github.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc00136aa80, {{0x2d06580?, 0xc003c70990?}, {0xc002e3f1d0?, 0x0?}})
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:834 +0x1c5
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc00136aa80, 0x0)
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:760 +0x419
created by github.com/tendermint/tendermint/consensus.(*State).OnStart
	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:379 +0x12d
: panic

Originally reported by @renaynay

Version

https://github.com/celestiaorg/celestia-app/releases/tag/v0.12.0-rc6

Steps to Reproduce

Submit a PFB on Arabica with

{"namespace_id": "0c204d39600fddd3", "data": "f1f20ca8007e910a3bf8b2e61da0f26bca07ef78717a6ea54165f5", "gas_limit": 700000, "fee": 2000}
@rootulp rootulp added the bug Something isn't working label Feb 9, 2023
@rootulp rootulp self-assigned this Feb 9, 2023
@rootulp
Copy link
Collaborator Author

rootulp commented Feb 9, 2023

We currently suspect that the GasPerBlobByte param was not defined in the genesis file for the current iteration of the Arabica testnet.

@evan-forbes
Copy link
Member

evan-forbes commented Feb 9, 2023

if we just didn't set the param, then the safest quickest thing in this case (single validator node on arabica), would be to hardfork and change the param manually via an upgrade handler esque script

either that, or we submit a proposal to update it via gov. iirc, we should be able to push a vote through that chain very quickly

@rootulp
Copy link
Collaborator Author

rootulp commented Feb 9, 2023

@Bidon15 confirmed that when Arabica was upgraded the param was not.

via an upgrade handler esque script

Are you referring to an x/upgrade plan?

or we submit a proposal to update it via gov

Seems possible via CLI commands draft proposal and submit proposal. I'll look into preparing a proposal.

What are the downsides to creating a new genesis file with the param populated and starting arabica-5 / shutting down arabica-4?

@evan-forbes
Copy link
Member

Are you referring to an x/upgrade plan?

we can hack it by just including a script in the binary then switch the only validator with that binary.

What are the downsides to creating a new genesis file with the param populated and starting arabica-5 / shutting down arabica-4?

that's also a very good option!!

@rootulp
Copy link
Collaborator Author

rootulp commented Feb 9, 2023

@Bidon15 @sysrex how feasible is it to start a new chain with an updated genesis file? The field that needs to be added to genesis.json is:

    "blob": {
      "params": {
        "min_square_size": 1,
        "max_square_size": 128,
+       "gas_per_blob_byte": 8
      }
    },

@rootulp
Copy link
Collaborator Author

rootulp commented Feb 9, 2023

Options:

  1. Stop arabica-4. Start arabica-5 with a genesis.json file that includes the param above.
  2. Draft and submit a governance proposal that sets the param on arabica-4. From the validator's account, vote yes on the proposal. Wait until it goes into effect.
  3. Write a manual script in the celestia-app repo that sets the param. Get the script reviewed, merged, and included in the celestia-appd binary. Cut a new release candidate (potentially v0.12.0-rc8). Upgrade the validator on arabica-4 to the new release candidate.

@rootulp
Copy link
Collaborator Author

rootulp commented Feb 9, 2023

@sysrex we explored option 2 with a legacy governance proposal:

{
   "title":"Parameter change: initialize gas_per_blob_byte",
   "description":" set the gas_per_blob_byte param to 8",
   "changes":[
      {
         "subspace":"blob",
         "key":"gas_per_blob_byte",
         "value":"8"
      }
   ],
   "deposit":"1000utia"
}

We submitted the governance proposal but it failed because we're trying to initialize a parameter that hasn't been registered yet. Ref:

  1. https://github.com/cosmos/cosmos-sdk/blob/8bd9288051d464a93d1101b189d7cb00c53beef1/x/params/types/subspace.go#L204-L214
  2. https://github.com/cosmos/cosmos-sdk/blob/8bd9288051d464a93d1101b189d7cb00c53beef1/x/params/proposal_handler.go#L37

So we're going to try option 1 now.

@rootulp
Copy link
Collaborator Author

rootulp commented Feb 9, 2023

We expect this to be resolved by deployment of arabica-5 which is scheduled for tomorrow morning.

@rootulp
Copy link
Collaborator Author

rootulp commented Feb 13, 2023

Submitting PFBs on arabica-5 appears to work.

$ curl -X POST -d '{"namespace_id": "0c204d39600fddd3",
  "data": "f1f20ca8007e910a3bf8b2e61da0f26bca07ef78717a6ea54165f5",
  "gas_limit": 140000, "fee": 2000}' http://localhost:26659/submit_pfb

Output:

{
  "height": 10573,
  "txhash": "E2CD12457D9263C98629B018C3F7C7D68B779FC90EA539462B94EEBB6D7191B3",
  "data": "122A0A282F63656C65737469612E626C6F622E76312E4D7367506179466F72426C6F6273526573706F6E7365",
  "raw_log": "[{\"msg_index\":0,\"events\":[{\"type\":\"celestia.blob.v1.EventPayForBlobs\",\"attributes\":[{\"key\":\"blob_size\",\"value\":\"1\"},{\"key\":\"namespace_ids\",\"value\":\"[\\\"DCBNOWAP3dM=\\\"]\"},{\"key\":\"signer\",\"value\":\"\\\"celestia1vdjkcetnw35kzvtv89ckvepcdpcx6mr2xd4nwur8dp6hydryxph8xdtyxdsnsumrwaekxumtwaksaphpv9\\\"\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"/celestia.blob.v1.MsgPayForBlobs\"}]}]}]",
  "logs": [
    {
      "msg_index": 0,
      "events": [
        {
          "type": "celestia.blob.v1.EventPayForBlobs",
          "attributes": [
            {
              "key": "blob_size",
              "value": "1"
            },
            {
              "key": "namespace_ids",
              "value": "[\"DCBNOWAP3dM=\"]"
            },
            {
              "key": "signer",
              "value": "\"celestia1vdjkcetnw35kzvtv89ckvepcdpcx6mr2xd4nwur8dp6hydryxph8xdtyxdsnsumrwaekxumtwaksaphpv9\""
            }
          ]
        },
        {
          "type": "message",
          "attributes": [
            {
              "key": "action",
              "value": "/celestia.blob.v1.MsgPayForBlobs"
            }
          ]
        }
      ]
    }
  ],
  "gas_wanted": 140000,
  "gas_used": 76252,
  "events": [
    {
      "type": "coin_spent",
      "attributes": [
        {
          "key": "c3BlbmRlcg==",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20=",
          "index": true
        },
        {
          "key": "YW1vdW50",
          "value": "MjAwMHV0aWE=",
          "index": true
        }
      ]
    },
    {
      "type": "coin_received",
      "attributes": [
        {
          "key": "cmVjZWl2ZXI=",
          "value": "Y2VsZXN0aWExN3hwZnZha20yYW1nOTYyeWxzNmY4NHoza2VsbDhjNWxwbmpzM3M=",
          "index": true
        },
        {
          "key": "YW1vdW50",
          "value": "MjAwMHV0aWE=",
          "index": true
        }
      ]
    },
    {
      "type": "transfer",
      "attributes": [
        {
          "key": "cmVjaXBpZW50",
          "value": "Y2VsZXN0aWExN3hwZnZha20yYW1nOTYyeWxzNmY4NHoza2VsbDhjNWxwbmpzM3M=",
          "index": true
        },
        {
          "key": "c2VuZGVy",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20=",
          "index": true
        },
        {
          "key": "YW1vdW50",
          "value": "MjAwMHV0aWE=",
          "index": true
        }
      ]
    },
    {
      "type": "message",
      "attributes": [
        {
          "key": "c2VuZGVy",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20=",
          "index": true
        }
      ]
    },
    {
      "type": "tx",
      "attributes": [
        {
          "key": "ZmVl",
          "value": "MjAwMHV0aWE=",
          "index": true
        },
        {
          "key": "ZmVlX3BheWVy",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20=",
          "index": true
        }
      ]
    },
    {
      "type": "tx",
      "attributes": [
        {
          "key": "YWNjX3NlcQ==",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20vMA==",
          "index": true
        }
      ]
    },
    {
      "type": "tx",
      "attributes": [
        {
          "key": "c2lnbmF0dXJl",
          "value": "eml2QnpZSy9mUWEwbklLUURBYzF2djVKZFNQUEFlK1RJQlJDVTd5dElCd2xpUGxRaVJLNFhKZUVoNXBxdWtaU3JTOHlsOVN4UUZXTWM0b293SFkySEE9PQ==",
          "index": true
        }
      ]
    },
    {
      "type": "message",
      "attributes": [
        {
          "key": "YWN0aW9u",
          "value": "L2NlbGVzdGlhLmJsb2IudjEuTXNnUGF5Rm9yQmxvYnM=",
          "index": true
        }
      ]
    },
    {
      "type": "celestia.blob.v1.EventPayForBlobs",
      "attributes": [
        {
          "key": "YmxvYl9zaXpl",
          "value": "MQ==",
          "index": true
        },
        {
          "key": "bmFtZXNwYWNlX2lkcw==",
          "value": "WyJEQ0JOT1dBUDNkTT0iXQ==",
          "index": true
        },
        {
          "key": "c2lnbmVy",
          "value": "ImNlbGVzdGlhMXZkamtjZXRudzM1a3p2dHY4OWNrdmVwY2RwY3g2bXIyeGQ0bnd1cjhkcDZoeWRyeXhwaDh4ZHR5eGRzbnN1bXJ3YWVreHVtdHdha3NhcGhwdjki",
          "index": true
        }
      ]
    }
  ]
}

Query the data that got posted:

$ curl -X GET 'http://localhost:26659/namespaced_data/0c204d39600fddd3/height/10573'
{"data":["8fIMqAB+kQo7+LLmHaDya8oH73hxem6lQWX1"],"height":10573}%

@rootulp rootulp closed this as completed Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants