Sync issues on Cardano Node v8.7.3 #867
Comments
|
@Kammerlo Can you help me with this problem, thank you very much! |
|
@YuXiaoCoder I updated the configs so this should be working now. Currently I changed it to rely on the standard config files which are delivered by the ogmios container. You can change this (if you want to run it with different configs) by changing this within the docker-compose.yml |
|
@Kammerlo Ok, I'll try it and give a reply, thanks! |
|
@Kammerlo There may still be a problem, the Asset is still empty and the progress is zero. |
|
cardano-graphql-hasura startup error: table does not exist {"detail":{"info":{"objects":[{"definition":{"name":"Cardano","schema":"public"},"name":"table Cardano in source default","reason":"Inconsistent object: no such table/view exists in source: \"Cardano\"","type":"table"},{"definition":{"name":"ProtocolParams","schema":"public"},"name":"table ProtocolParams in source default","reason":"Inconsistent object: no such table/view exists in source: \"ProtocolParams\"","type":"table"},{"definition":{"name":"ReferenceInput","schema":"public"},"name":"table ReferenceInput in source default","reason":"Inconsistent object: no such table/view exists in source: \"ReferenceInput\"","type":"table"},{"definition":{"name":"Block","schema":"public"},"name":"table Block in source default","reason":"Inconsistent object: no such table/view exists in source: \"Block\"","type":"table"},{"definition":{"name":"Transaction","schema":"public"},"name":"table Transaction in source default","reason":"Inconsistent object: no such table/view exists in source: \"Transaction\"","type":"table"},{"definition":{"name":"ActiveStake","schema":"public"},"name":"table ActiveStake in source default","reason":"Inconsistent object: no such table/view exists in source: \"ActiveStake\"","type":"table"},{"definition":{"name":"Delegation","schema":"public"},"name":"table Delegation in source default","reason":"Inconsistent object: no such table/view exists in source: \"Delegation\"","type":"table"},{"definition":{"name":"RedeemerDatum","schema":"public"},"name":"table RedeemerDatum in source default","reason":"Inconsistent object: no such table/view exists in source: \"RedeemerDatum\"","type":"table"},{"definition":{"name":"StakePoolOwner","schema":"public"},"name":"table StakePoolOwner in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakePoolOwner\"","type":"table"},{"definition":{"name":"Epoch","schema":"public"},"name":"table Epoch in source default","reason":"Inconsistent object: no such table/view exists in source: \"Epoch\"","type":"table"},{"definition":{"name":"TransactionInput","schema":"public"},"name":"table TransactionInput in source default","reason":"Inconsistent object: no such table/view exists in source: \"TransactionInput\"","type":"table"},{"definition":{"name":"Utxo","schema":"public"},"name":"table Utxo in source default","reason":"Inconsistent object: no such table/view exists in source: \"Utxo\"","type":"table"},{"definition":{"name":"TokenMint","schema":"public"},"name":"table TokenMint in source default","reason":"Inconsistent object: no such table/view exists in source: \"TokenMint\"","type":"table"},{"definition":{"name":"SlotLeader","schema":"public"},"name":"table SlotLeader in source default","reason":"Inconsistent object: no such table/view exists in source: \"SlotLeader\"","type":"table"},{"definition":{"name":"CollateralInput","schema":"public"},"name":"table CollateralInput in source default","reason":"Inconsistent object: no such table/view exists in source: \"CollateralInput\"","type":"table"},{"definition":{"name":"Reward","schema":"public"},"name":"table Reward in source default","reason":"Inconsistent object: no such table/view exists in source: \"Reward\"","type":"table"},{"definition":{"name":"AdaPots","schema":"public"},"name":"table AdaPots in source default","reason":"Inconsistent object: no such table/view exists in source: \"AdaPots\"","type":"table"},{"definition":{"name":"Withdrawal","schema":"public"},"name":"table Withdrawal in source default","reason":"Inconsistent object: no such table/view exists in source: \"Withdrawal\"","type":"table"},{"definition":{"name":"CollateralOutput","schema":"public"},"name":"table CollateralOutput in source default","reason":"Inconsistent object: no such table/view exists in source: \"CollateralOutput\"","type":"table"},{"definition":{"name":"StakePool","schema":"public"},"name":"table StakePool in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakePool\"","type":"table"},{"definition":{"name":"StakePoolRetirement","schema":"public"},"name":"table StakePoolRetirement in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakePoolRetirement\"","type":"table"},{"definition":{"name":"StakeRegistration","schema":"public"},"name":"table StakeRegistration in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakeRegistration\"","type":"table"},{"definition":{"name":"StakeDeregistration","schema":"public"},"name":"table StakeDeregistration in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakeDeregistration\"","type":"table"},{"definition":{"name":"Script","schema":"public"},"name":"table Script in source default","reason":"Inconsistent object: no such table/view exists in source: \"Script\"","type":"table"},{"definition":{"name":"TokenInOutput","schema":"public"},"name":"table TokenInOutput in source default","reason":"Inconsistent object: no such table/view exists in source: \"TokenInOutput\"","type":"table"},{"definition":{"name":"Datum","schema":"public"},"name":"table Datum in source default","reason":"Inconsistent object: no such table/view exists in source: \"Datum\"","type":"table"},{"definition":{"name":"Redeemer","schema":"public"},"name":"table Redeemer in source default","reason":"Inconsistent object: no such table/view exists in source: \"Redeemer\"","type":"table"},{"definition":{"name":"TransactionOutput","schema":"public"},"name":"table TransactionOutput in source default","reason":"Inconsistent object: no such table/view exists in source: \"TransactionOutput\"","type":"table"},{"definition":{"comment":null,"name":"firstAppearedInBlock","source":"default","table":{"name":"Asset","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"firstAppearedInSlot":"slotNo"},"insertion_order":null,"remote_table":{"name":"Block","schema":"public"}}}},"name":"object_relation firstAppearedInBlock in table Asset in source default","reason":"Inconsistent object: table \"Block\" is not tracked","type":"object_relation"},{"definition":{"comment":null,"name":"stakePool","source":"default","table":{"name":"pool_relay","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"update_id":"update_id"},"insertion_order":null,"remote_table":{"name":"StakePool","schema":"public"}}}},"name":"object_relation stakePool in table pool_relay in source default","reason":"Inconsistent object: table \"StakePool\" is not tracked","type":"object_relation"},{"definition":{"comment":null,"name":"tokenMints","source":"default","table":{"name":"Asset","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"assetName":"assetName","policyId":"policyId"},"insertion_order":null,"remote_table":{"name":"TokenMint","schema":"public"}}}},"name":"array_relation tokenMints in table Asset in source default","reason":"Inconsistent object: table \"TokenMint\" is not tracked","type":"array_relation"},{"definition":{"comment":null,"name":"tokenMints","source":"default","table":{"name":"Asset","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"assetName":"assetName","policyId":"policyId"},"insertion_order":null,"remote_table":{"name":"TokenMint","schema":"public"}}}},"name":"array_relation tokenMints in table Asset in source default","reason":"Inconsistent object: table \"TokenMint\" is not tracked","type":"array_relation"},{"definition":{"comment":null,"name":"transaction","source":"default","table":{"name":"tx_metadata","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"tx_id":"id"},"insertion_order":null,"remote_table":{"name":"Transaction","schema":"public"}}}},"name":"object_relation transaction in table tx_metadata in source default","reason":"Inconsistent object: table \"Transaction\" is not tracked","type":"object_relation"}]},"message":"Inconsistent Metadata!"},"level":"warn","timestamp":"2024-04-27T10:14:10.689+0000","type":"metadata"} |
|
cardano-graphql-background logs: {"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"Db","msg":"Initializing...","time":"2024-04-28T04:51:30.520Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"HasuraBackgroundClient","msg":"Initializing","time":"2024-04-28T04:51:30.525Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"HasuraBackgroundClient","msg":"Initialized","time":"2024-04-28T04:53:09.729Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2024-04-28T04:53:09.729Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"MetadataFetchClient","msg":"Initialized","time":"2024-04-28T04:53:10.835Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Initializing","time":"2024-04-28T04:53:10.835Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Initialized","time":"2024-04-28T04:53:10.842Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"Worker","msg":"Starting","time":"2024-04-28T04:53:10.842Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"Worker","msg":"Started","time":"2024-04-28T04:53:10.875Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Starting","time":"2024-04-28T04:53:10.883Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Started","time":"2024-04-28T04:53:10.904Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Rolling back to genesis","time":"2024-04-28T04:53:10.949Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Deleted 0 assets","time":"2024-04-28T04:53:10.952Z","v":0}
/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:85
for (const entry of Object.entries(tx.mint.assets)) {
^
TypeError: Cannot convert undefined or null to object
at Function.entries (<anonymous>)
at ChainFollower.<anonymous> (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:85:64)
at Generator.next (<anonymous>)
at /app/packages/api-cardano-db-hasura/dist/ChainFollower.js:31:71
at new Promise (<anonymous>)
at __awaiter (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:27:12)
at Object.rollForward (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:80:62)
at handler (/app/node_modules/@cardano-ogmios/client/dist/ChainSynchronization/nextBlock.js:23:46)
at messageHandler (/app/node_modules/@cardano-ogmios/client/dist/ChainSynchronization/Client.js:17:43)
at asyncWrapper (/app/node_modules/fastq/queue.js:213:12)
Node.js v18.19.1
|
|
@YuXiaoCoder Hey, sorry for letting you wait! I'm currently working on this issue. I already found the problem why the chainfollower crashes and the assets were empty. I'm doing everything in this branch: fix/syncing-issues-assets-missing |
|
Thanks for doing this work! Was trying out your branch: fix/syncing-issues-assets-missing. All works fine except for very slow or stuck syncing of graphql. Not getting past Asset: 11% $ docker logs preprod-server-1 -f Let it run all night ~10hours . Maybe no metadata for preprod, hmm |
|
@Kammerlo Thank you very much for your help, looking forward to your new version |
|
@YuXiaoCoder So this fix solved your issue? @M2tec I'm currently resyncing to test it. I'll let you know. Did you find any other issues or is it just the 11% related? And did you check that your node + db-sync were at the tip? |
|
Yeah, works. { docker logs preprod-cardano-db-sync-1 -f I tested some graphQL queries with the hasura console via http://localhost:8091/console The service is trying to get metadata for the assets. But maybe this does not work for preprod. It puts 77850 assets in a retry state.
|
|
It will only crash for a fresh sync? I've already upgraded my mainnet with
But for preprod did a re-sync from scratch. The mainnet graphql-background doesn't crash because it already had the data for But mainnet Asset is stuck at 83%! How do I fix the mainnet once your code is merged? Will it be auto-fixed? |
|
@cshintov For mainnet a resync is needed as well, since we needed to upgrade the db and due to some changes of the schema. Did you just upgraded or the errors occur even after a resync? |
|
For the mainnet I didn't resync. But there were no errors when I did the upgrade. Eventhough the Assets is stuck at 83% But for preprod my node db was corrupted so I did resync. But when I started the graphql afterwards the above mentioned error happened. |
You mean I have to nuke |
|
db-sync and postgres. The node should be fine. It's just because the new version of db-sync needs a newer Postgres version and thats why a resync is needed |
|
Did the latest upgrade for 8.0.1 on preprod. Asset is stuck at 11%! been 8 hours! Current logs for background: |
|
@cshintov Fixed it in this branch: fix/assets-skipped. Could you check and verify if this solves your issue? Logs from server: If this solves your issue I will merge it as soon as possible. |
|
Will have to build the image. Can you point me towards the Dockerfile? |
|
you can checkout the branch and then start the docker compose with |
|
Tested fix/assets-skipped also. docker logs preprod-server-1 -f Slight nitpick, sorry Result mint is off by 1: But "cardanoDbMeta": { "initialized": true}. Have not seen this before. Many thanks! |
|
Perfect! |
|
Closing this issue since the bugfix is now in PreRelease 8.0.2 |
|
Yup. It worked. Thanks 🙂 I've another issue, but it's related to db-sync. Can't use db sync snapshot with v13.2.0.1 on mainnet. The Tried to build a new image but hitting error! The user has permission to access kvm, can run qemu directly, but nix build fails! Will raise an issue there! |
|
By the way, why there's no It doesn't matter? I can use |
Ask a question
Hi,
I am running a Cardano node on the mainnet with the following versions:
I am using the compatible configuration files located here LINK.
Due to the database upgrade, I reloaded the latest db-sync snapshot and cardano-node snapshot
After the service starts, everything works fine, but cardano-graphql-server is not populated with Asset data, which prevents the interface from being used
kubectl logs -f ada-2-0 ada-2-cardano-graphql-server --tail 10 {"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-04-25T07:13:11.178Z","v":0} {"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","tip":{"slot":122462877,"id":"6ba0e27077bd656e381af575faa0b21f104658f8eae804f192c07bdb60580205"},"msg":"[Object: null prototype] {\n slot: 122462877,\n id: '6ba0e27077bd656e381af575faa0b21f104658f8eae804f192c07bdb60580205'\n}","time":"2024-04-25T07:13:11.181Z","v":0} {"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 0%","time":"2024-04-25T07:13:24.157Z","v":0} {"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-04-25T07:13:24.158Z","v":0} {"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","tip":{"slot":122462905,"id":"843ccb435fd90c3cb35ecb4db28f42435d8fb93f198abf7fff082dfde6819a9d"},"msg":"[Object: null prototype] {\n slot: 122462905,\n id: '843ccb435fd90c3cb35ecb4db28f42435d8fb93f198abf7fff082dfde6819a9d'\n}","time":"2024-04-25T07:13:24.161Z","v":0} {"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-04-25T07:13:29.158Z","v":0}It would be great if you could help me with the node sync issue.
The text was updated successfully, but these errors were encountered: