Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent Ouroboros/ChainDB error when syncing on Testnet #471

Closed
oneEdoubleD opened this issue Jan 14, 2020 · 3 comments
Closed

Intermittent Ouroboros/ChainDB error when syncing on Testnet #471

oneEdoubleD opened this issue Jan 14, 2020 · 3 comments
Assignees
@mrBliss
Labels
bug Something isn't working byron Required for a Byron mainnet: replace the old core nodes with cardano-node. priority high issues/PRs that MUST be addressed. The release can't happen without this;
Milestone
S5 2020-01-30

Comments

@oneEdoubleD
Copy link
Contributor

oneEdoubleD commented Jan 14, 2020 •
edited
Loading

Issue:

Testnet nodes are intermittently crashing while syncing. When testing four nodes they crashed a total of 7 times in under an hour. There were no indications of memory issues in the process explorer while running these nodes. The error was:

cardano-node: ExceptionInLinkedThread "ThreadId 80" (ClosedDBError [("getEnvSTM",SrcLoc {srcLocPackage = "ouroboros-consensus-0.1.0.0-R0AVfeY78d7fCNQvHwSpA", srcLocModule = "Ouroboros.Storage.ChainDB.Impl", srcLocFile = "src/Ouroboros/Storage/ChainDB/Impl.hs", srcLocStartLine = 169, srcLocStartCol = 34, srcLocEndLine = 169, srcLocEndCol = 64})])

While most of the testing was completed on jlotoski/cardano-node-issue-461 I have also confirmed it occurs on Master.

This is a fatal error that crashes the node. Notably, this does not occur on Staging.

Steps to recreate:

  • Clone a node (or four, to speed up the process of having one crash as it's seemingly random)
  • Build with nix-build -A scripts.testnet.node
  • Run nodes

Edit:

To run nodes, run ./result in the nodes directory.
@nfrisby
Copy link
Contributor

nfrisby commented Jan 14, 2020 •
edited
Loading

Thanks! I think I've been investigating similar exceptions in the ouroboros-network test suite and am optimistic that I'll be able to address this relatively soon and simply.

@i-o-m i-o-m added the bug Something isn't working label Jan 16, 2020
@i-o-m i-o-m added this to the S5 2020-01-30 milestone Jan 16, 2020
@i-o-m i-o-m added the byron Required for a Byron mainnet: replace the old core nodes with cardano-node. label Jan 16, 2020
@vhulchenko-iohk vhulchenko-iohk added priority high issues/PRs that MUST be addressed. The release can't happen without this; and removed byron Required for a Byron mainnet: replace the old core nodes with cardano-node. labels Jan 16, 2020
@i-o-m i-o-m added the byron Required for a Byron mainnet: replace the old core nodes with cardano-node. label Jan 16, 2020
@mrBliss mrBliss self-assigned this Jan 21, 2020
@mrBliss
Copy link
Contributor

mrBliss commented Jan 21, 2020

Linking to IntersectMBO/ouroboros-network#1470

@oneEdoubleD
Copy link
Contributor Author

I have completely synced several nodes last Friday, and @ArturWieczorek ran 6 nodes for several hours without seeing this issue, so I think it's safe to say it is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrBliss mrBliss

Labels
bug Something isn't working byron Required for a Byron mainnet: replace the old core nodes with cardano-node. priority high issues/PRs that MUST be addressed. The release can't happen without this;
Projects
None yet
Milestone
S5 2020-01-30
Development

No branches or pull requests
5 participants
@mrBliss @nfrisby @oneEdoubleD @i-o-m @vhulchenko-iohk