Two nodes which have same configuration have different transaction records

[raphaellin]

We are from the imToken developer team.
We have two nodes deployed in our environment, they have the same startup parameters and configurations.
But we only can find one transaction in one node.
Attached please find the two records from those two nodes.

This is a very serious problem, some transactions will be lost for end-user. please help to provide a plan to fix it.
If you need any more information, you can send an email to me, raphael@token.im.

[raphaellin]

The node record which has specific transaction:
{"height":"5266739","txhash":"31938F53EB5FC2E95396EAF98211B7EF166C0F3B7C938CE7A4DC7418D82CFCA2","data":"0A060A0473656E64","raw_log":"[{\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"send\"},{\"key\":\"sender\",\"value\":\"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd\"},{\"key\":\"module\",\"value\":\"bank\"}]},{\"type\":\"transfer\",\"attributes\":[{\"key\":\"recipient\",\"value\":\"cosmos105hssfshxkdrj4ycpdd5yn00wmscjyfc6vk6zw\"},{\"key\":\"sender\",\"value\":\"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd\"},{\"key\":\"amount\",\"value\":\"1063237205uatom\"}]}]}]","logs":[{"events":[{"type":"message","attributes":[{"key":"action","value":"send"},{"key":"sender","value":"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd"},{"key":"module","value":"bank"}]},{"type":"transfer","attributes":[{"key":"recipient","value":"cosmos105hssfshxkdrj4ycpdd5yn00wmscjyfc6vk6zw"},{"key":"sender","value":"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd"},{"key":"amount","value":"1063237205uatom"}]}]}],"gas_wanted":"85000","gas_used":"62304","tx":{"type":"cosmos-sdk/StdTx","value":{"msg":[{"type":"cosmos-sdk/MsgSend","value":{"from_address":"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd","to_address":"cosmos105hssfshxkdrj4ycpdd5yn00wmscjyfc6vk6zw","amount":[{"denom":"uatom","amount":"1063237205"}]}}],"fee":{"amount":[{"denom":"uatom","amount":"2500"}],"gas":"85000"},"signatures":[{"pub_key":{"type":"tendermint/PubKeySecp256k1","value":"Auoah0sTBZ6XM8NekMsqWGWka2TtkNJVVHwHylI5J/xi"},"signature":"+VZl088AFicSJiYj9mU307+8ncE5ah3ZYOmG/zULITUePJ9vhnWnTnOaQSf82yZI+aY9Bss8WxLJNAqnLzSSzg=="}],"memo":"","timeout_height":"0"}},"timestamp":"2021-02-23T23:59:57Z"}

[raphaellin]

The node record which didn't has specific transaction
{"error":"tx (31938F53EB5FC2E95396EAF98211B7EF166C0F3B7C938CE7A4DC7418D82CFCA2) not found"}

[tac0turtle]

This may be because one of the nodes has block pruning while the other does not. Can you double-check this in your app.toml and your application state pruning settings? Did the second node use state sync to catch up? More information context is needed to understand how to help

Can you try searching for the transaction via the tx_search endpoint?

[alexanderbez]

Hi @raphaellin. Are you absolutely 100% certain both nodes had identical configurations from genesis? Did you perhaps tweak the settings of one node only to a later point in time revert? This is the only logical thing that makes sense.

In addition, can you provide your application pruning settings (found in app.toml) and the Tendermint block pruning settings (found in config.toml) please?

[raphaellintoken]

Thanks Marko. I think this is a standard practice which we will not miss. However, I will check with developer as well. Marko ***@***.***> 于2021年3月24日周三 下午5:50写道：

…

This may be because one of the nodes has block pruning while the other does not. Can you double-check this in your app.toml and your application state pruning settings? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#122 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATLYKUYLPL7JMNEI5VSPOADTFGYUVANCNFSM4ZUXJYEQ> .

[raphaellintoken]

This may be because one of the nodes has block pruning while the other does not. Can you double-check this in your app.toml and your application state pruning settings? Did the second node use state sync to catch up? More information context is needed to understand how to help

Can you try searching for the transaction via the tx_search endpoint?

@marbar3778

Got same problem when using 'tx_search' endpoint.
The result on issue node
/gaia curl "http://localhost:26657/tx_search?query=\"tx.height=5266739\"" { "jsonrpc": "2.0", "id": -1, "result": { "txs": [], "total_count": "0" }

The result on right node
/gaia # curl "http://localhost:26657/tx_search?query=\"tx.height=5266739\"" { "jsonrpc": "2.0", "id": -1, "result": { "txs": [ { "hash": "31938F53EB5FC2E95396EAF98211B7EF166C0F3B7C938CE7A4DC7418D82CFCA2", "height": "5266739", "index": 0, "tx_result": { "code": 0, "data": "CgYKBHNlbmQ=", "log": "[{\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"send\"},{\"key\":\"sender\",\"value\":\"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd\"},{\"key\":\"module\",\"value\":\"bank\"}]},{\"type\":\"transfer\",\"attributes\":[{\"key\":\"recipient\",\"value\":\"cosmos105hssfshxkdrj4ycpdd5yn00wmscjyfc6vk6zw\"},{\"key\":\"sender\",\"value\":\"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd\"},{\"key\":\"amount\",\"value\":\"1063237205uatom\"}]}]}]", "info": "", "gas_wanted": "85000", "gas_used": "62304", "events": [ { "type": "transfer", "attributes": [ { "key": "cmVjaXBpZW50", "value": "Y29zbW9zMTd4cGZ2YWttMmFtZzk2MnlsczZmODR6M2tlbGw4YzVsc2VycXRh", "index": true }, { "key": "c2VuZGVy", "value": "Y29zbW9zMXQ1dTBqZmczbGpzanJoMm05ZTQ3ZDRueTJoZWE3ZWVoeHJ6ZGdk", "index": true }, { "key": "YW1vdW50", "value": "MjUwMHVhdG9t", "index": true } ] }, { "type": "message", "attributes": [ { "key": "c2VuZGVy", "value": "Y29zbW9zMXQ1dTBqZmczbGpzanJoMm05ZTQ3ZDRueTJoZWE3ZWVoeHJ6ZGdk", "index": true } ] }, { "type": "message", "attributes": [ { "key": "YWN0aW9u", "value": "c2VuZA==", "index": true } ] }, { "type": "transfer", "attributes": [ { "key": "cmVjaXBpZW50", "value": "Y29zbW9zMTA1aHNzZnNoeGtkcmo0eWNwZGQ1eW4wMHdtc2NqeWZjNnZrNnp3", "index": true }, { "key": "c2VuZGVy", "value": "Y29zbW9zMXQ1dTBqZmczbGpzanJoMm05ZTQ3ZDRueTJoZWE3ZWVoeHJ6ZGdk", "index": true }, { "key": "YW1vdW50", "value": "MTA2MzIzNzIwNXVhdG9t", "index": true } ] }, { "type": "message", "attributes": [ { "key": "c2VuZGVy", "value": "Y29zbW9zMXQ1dTBqZmczbGpzanJoMm05ZTQ3ZDRueTJoZWE3ZWVoeHJ6ZGdk", "index": true } ] }, { "type": "message", "attributes": [ { "key": "bW9kdWxl", "value": "YmFuaw==", "index": true } ] } ], "codespace": "" }, "tx": "CpYBCpMBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnMKLWNvc21vczF0NXUwamZnM2xqc2pyaDJtOWU0N2Q0bnkyaGVhN2VlaHhyemRnZBItY29zbW9zMTA1aHNzZnNoeGtkcmo0eWNwZGQ1eW4wMHdtc2NqeWZjNnZrNnp3GhMKBXVhdG9tEgoxMDYzMjM3MjA1EmUKTgpGCh8vY29zbW9zLmNyeXB0by5zZWNwMjU2azEuUHViS2V5EiMKIQLqGodLEwWelzPDXpDLKlhlpGtk7ZDSVVR8B8pSOSf8YhIECgIIfxITCg0KBXVhdG9tEgQyNTAwEIiYBRpA+VZl088AFicSJiYj9mU307+8ncE5ah3ZYOmG/zULITUePJ9vhnWnTnOaQSf82yZI+aY9Bss8WxLJNAqnLzSSzg==" } ], "total_count": "1" }

[raphaellintoken]

Hi @raphaellin. Are you absolutely 100% certain both nodes had identical configurations from genesis? Did you perhaps tweak the settings of one node only to a later point in time revert? This is the only logical thing that makes sense.

In addition, can you provide your application pruning settings (found in app.toml) and the Tendermint block pruning settings (found in config.toml) please?

Hi @alexanderbez

Yes, the configuration of two nodes are absolutely the same.
and we used default value of pruning settings, can you provide a email so that I can send the configuration file to you.

[alexanderbez]

sure! bez@interchain.berlin :)

[alexanderbez]

@raphaellintoken I took a look at the configurations. They are indeed mainly identical.

However, you're using the default pruning settings. This means you're only going to be able to query the last 100 heights and every 100th after that.

This doesn't explain why one node returns a result and another doesn't, but I bet that the two nodes are just slightly out of sync and so one node prunes blocks at a slightly different rate than the other.

I recommend not using the default setting, but something custom. Perhaps consider keeping enough blocks within the unbonding period.

e.g. if your unbonding period is 3 weeks, then:

--pruning=custom --pruning-keep-recent=362880 --pruning-keep-every=0 --pruning-interval=100

[raphaellintoken]

Thanks @alexanderbez , we will try.
But I think it's better to find the root cause of this issue. other users may also meet this issue. Do you want to have more investigation？ we can supply more data what you need.

[alexanderbez]

Unfortunately not, as this is the only thing I can think of. Also, we have not seen any other reports like this.

Out of curiosity, the heights you are querying, how far back are they from the latest block? Less than 100?

[raphaellintoken]

When we found this issue, it is more than 100 from the latest block.