Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CRASH: (monitor.ml.Error (Failure "Digit.split index out of bounds") #3143

Closed
jkrauska opened this issue Aug 8, 2019 · 7 comments · Fixed by #3199

Comments

@jkrauska
Copy link
Contributor

commented Aug 8, 2019

Crash on several nodes on testnet:

    (monitor.ml.Error (Failure "Digit.split index out of bounds")
  ("Raised at file \"stdlib.ml\", line 33, characters 17-33"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 173, characters 18-58"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 173, characters 18-58"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 195, characters 26-46"
    "Called from file \"lib/network_pool/f_sequence.ml\" (inlined), line 328, characters 32-77"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 580, characters 12-113"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 576, characters 28-74"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 640, characters 10-36"
    "Called from file \"lib/network_pool/indexed_pool.ml\", line 532, characters 10-138"
    "Called from file \"lib/network_pool/transaction_pool.ml\", line 384, characters 30-193"
    "Called from file \"lib/network_pool/network_pool_base.ml\", line 25, characters 15-65"
    "Called from file \"src/pipe.ml\", line 827, characters 4-56"
    "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5"
    "Called from file \"src/job_queue.ml\", line 170, characters 6-47"
    "Caught by monitor coda"))

@jkrauska jkrauska added the testnet label Aug 8, 2019

@jkrauska jkrauska added this to Needs triage in Testnet Blockers via automation Aug 8, 2019

@jkrauska jkrauska added this to Discuss in Work Prioritization - Protocol via automation Aug 8, 2019

@jkrauska

This comment has been minimized.

@jkrauska jkrauska changed the title CRASH CRASH: (monitor.ml.Error (Failure "Digit.split index out of bounds") Aug 8, 2019

@imeckler

This comment has been minimized.

Copy link
Contributor

commented Aug 8, 2019

I looked into this one -- couldn't get it to reproduce. F_sequence.split_at is throwing. I tried to get it to reproduce by

  • applying a transaction pool diff with 400 (> the max size of 300) transactions from one account
  • calling split_at with invalid indices

and couldn't.

@enolan could you investigate?

@garethtdavies

This comment has been minimized.

Copy link
Contributor

commented Aug 8, 2019

I just had this crash on the reinstated testnet coda version Commit ce3da11 on branch release/0.0.1-beta.2

(monitor.ml.Error (Failure "Digit.split index out of bounds")
  ("Raised at file \"stdlib.ml\", line 33, characters 17-33"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 173, characters 18-58"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 173, characters 18-58"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 195, characters 26-46"
    "Called from file \"lib/network_pool/f_sequence.ml\" (inlined), line 328, characters 32-77"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 580, characters 12-113"
    "Called from file \"lib/network_pool/f_sequence.ml\", line 640, characters 10-36"
    "Called from file \"lib/network_pool/indexed_pool.ml\", line 532, characters 10-138"
    "Called from file \"lib/network_pool/transaction_pool.ml\", line 384, characters 30-193"
    "Called from file \"lib/network_pool/network_pool_base.ml\", line 25, characters 15-65"
    "Called from file \"src/pipe.ml\", line 827, characters 4-56"
    "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5"
    "Called from file \"src/job_queue.ml\", line 170, characters 6-47"
    "Caught by monitor coda"))
@yourbuddyconner

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2019

I have seen this crash manifest 3-4 times on the faucet daemon overnight:

Version: 0.0.2-beta.2-ce3da112

2019-08-09 12:05:15 UTC [Info] Removing peer from peer set: [host : 116.203.20.122, discovery_port : 8303, communication_port : 8302]
2019-08-09 12:05:15 UTC [Info] Moving peer to disconnected peer set : ((host 116.203.20.122) (discovery_port 8303) (communication_port 8302))
☠ Coda Daemon crashed. The Coda Protocol developers would like to know why!
Please:
Open an issue:
https://github.com/CodaProtocol/coda/issues/new
Briefly describe what you were doing, and include the last 20 lines from .coda-config/coda.log.
And then paste the following:
(monitor.ml.Error (Failure "Digit.split index out of bounds")
("Raised at file \"stdlib.ml\", line 33, characters 17-33"
"Called from file \"lib/network_pool/f_sequence.ml\", line 173, characters 18-58"
"Called from file \"lib/network_pool/f_sequence.ml\", line 173, characters 18-58"
"Called from file \"lib/network_pool/f_sequence.ml\", line 195, characters 26-46"
"Called from file \"lib/network_pool/f_sequence.ml\" (inlined), line 328, characters 32-77"
"Called from file \"lib/network_pool/f_sequence.ml\", line 580, characters 12-113"
"Called from file \"lib/network_pool/f_sequence.ml\", line 640, characters 10-36"
"Called from file \"lib/network_pool/indexed_pool.ml\", line 532, characters 10-138"
"Called from file \"lib/network_pool/transaction_pool.ml\", line 384, characters 30-193"
"Called from file \"lib/network_pool/network_pool_base.ml\", line 25, characters 15-65"
"Called from file \"src/pipe.ml\", line 827, characters 4-56"
"Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5"
"Called from file \"src/job_queue.ml\", line 170, characters 6-47"
"Caught by monitor coda"))
☠ Coda Daemon crashed. The Coda Protocol developers would like to know why!
Please:
Open an issue:
https://github.com/CodaProtocol/coda/issues/new
Briefly describe what you were doing, and include the last 20 lines from .coda-config/coda.log.
And then paste the following:
(monitor.ml.Error (Failure "Digit.split index out of bounds")
("Raised at file \"stdlib.ml\", line 33, characters 17-33"
"Called from file \"lib/network_pool/f_sequence.ml\", line 173, characters 18-58"
"Called from file \"lib/network_pool/f_sequence.ml\", line 173, characters 18-58"
"Called from file \"lib/network_pool/f_sequence.ml\", line 195, characters 26-46"
"Called from file \"lib/network_pool/f_sequence.ml\" (inlined), line 328, characters 32-77"
"Called from file \"lib/network_pool/f_sequence.ml\", line 580, characters 12-113"
"Called from file \"lib/network_pool/f_sequence.ml\", line 576, characters 28-74"
"Called from file \"lib/network_pool/f_sequence.ml\", line 640, characters 10-36"
"Called from file \"lib/network_pool/indexed_pool.ml\", line 532, characters 10-138"
"Called from file \"lib/network_pool/transaction_pool.ml\", line 384, characters 30-193"
"Called from file \"lib/network_pool/network_pool_base.ml\", line 25, characters 15-65"
"Called from file \"src/pipe.ml\", line 827, characters 4-56"
"Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5"
"Called from file \"src/job_queue.ml\", line 170, characters 6-47"
"Caught by monitor coda"))

@imeckler imeckler moved this from Discuss to This release in Work Prioritization - Protocol Aug 9, 2019

@imeckler

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2019

@enolan I am reassigning this to you -- I have reproduced this, on the finger-tree-debug branch but am having trouble understanding the code where the error is happening. @bkase and @jkrauska say they are not going to do a redeploy of this network so let's do this

  1. Next week if you are in can you look at this?
  2. If it doesn't get fixed by the end of the week, we'll replace the finger tree with a list (the transaction pool is really small anyway, so the finger tree is not really necessary at the moment.)
@enolan

This comment has been minimized.

Copy link
Contributor

commented Aug 10, 2019

@o1pranay o1pranay added this to the Testnet Beta milestone Aug 12, 2019

enolan added a commit that referenced this issue Aug 14, 2019
Fix exception when splitting F_sequences.
Fixes #3143.

* Changed split predicate from > to >= to match paper
* Fixed up other things to match
* Improved finger tree tests
* Fix handling of measures when splitting a subtree of Nodes and
  the split point is in the left finger.
* Added a transaction pool test for maintaining the max size.
  This would not have caught the bug, but is nice to have.
ghost-not-in-the-shell pushed a commit that referenced this issue Aug 20, 2019
Fix exception when splitting F_sequences. (#3199)
* Fix exception when splitting F_sequences.

Fixes #3143.

* Changed split predicate from > to >= to match paper
* Fixed up other things to match
* Improved finger tree tests
* Fix handling of measures when splitting a subtree of Nodes and
  the split point is in the left finger.
* Added a transaction pool test for maintaining the max size.
  This would not have caught the bug, but is nice to have.

* fix warning
@enolan

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2019

should be fixed by #3199

@enolan enolan closed this Aug 21, 2019

Testnet Blockers automation moved this from Needs triage to Closed Aug 21, 2019

Work Prioritization - Protocol automation moved this from This release to Done Aug 21, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
You can’t perform that action at this time.