Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERROR validator: rpc error: code = DeadlineExceeded desc = context deadline exceeded #4673

Closed
GuillaumeOcculy opened this issue Jan 28, 2020 · 9 comments
Closed

ERROR validator: rpc error: code = DeadlineExceeded desc = context deadline exceeded #4673

GuillaumeOcculy opened this issue Jan 28, 2020 · 9 comments
Assignees
@prestonvanloon

Comments

@GuillaumeOcculy
Copy link

This is what it returns when I use this command :

docker run -it -d -v $HOME/prysm:/data --network="host"   gcr.io/prysmaticlabs/prysm/validator:latest   --beacon-rpc-provider=127.0.0.1:4000   --keymanager=keystore --keymanageropts='{"path":"/data","passphrase":"changeme"}'
[2020-01-27 23:42:45]  INFO validator: Waiting for beacon node to sync to latest chain head
[2020-01-27 23:43:01]  INFO validator: Waiting for validator to be activated in the beacon chain
[2020-01-27 23:43:01]  INFO validator: Validator activated pubKey=0xad4847f951aa
[2020-01-27 23:43:01] ERROR validator: rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2020-01-27 23:43:01] ERROR validator: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2020-01-27 23:47:24] ERROR validator: Could not report validator's rewards/penalties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded slot=129536
[2020-01-27 23:47:57]  INFO validator: New assignment attesterSlot=129553 committeeIndex=3 epoch=4048 pubKey=0xad4847f951aa status=ACTIVE validatorIndex=32146
[2020-01-27 23:50:48]  INFO validator: Submitted new attestations AggregatorIndices=[] AttesterIndices=[32146] BeaconBlockRoot=0x98c016047eac CommitteeIndex=3 Slot=129553 SourceEpoch=4047 SourceRoot=0xf915c7ae9d69 TargetEpoch=4048 TargetRoot=0x98c016047eac
[2020-01-27 23:53:48] ERROR validator: Could not report validator's rewards/penalties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded slot=129568
@clabornd
Copy link

Am having the same issue after sending the deposit. Ubuntu 16.04 LTS running from docker containers.

Stuff that might matter, I dunno:
I have two validatorprivatekeyxxxx in my $HOME/prysm
I re-submitted a deposit just yesterday after a failed attempt from a few days ago
I don't do any sort of cleanup between trying to activate on different deposits, I just follow the instructions on https://prylabs.net/participate

[2020-01-28 16:25:48] ERROR validator: rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2020-01-28 16:25:48] ERROR validator: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2020-01-28 16:25:48] ERROR validator: Could not report validator's rewards/penalties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded slot=134528

@Domintek
Copy link

I have experienced the same error validator: could not report validator's rewards/penalties error=rpc error: code = DeadlineExceeded .... multiple times over the cours of the last 24 hours and then the validator stalled. Running Docker in Win 10 and trying to follow the same instructions as Clabornd above.

@rauljordan
Copy link
Contributor

Hey all, this is due to several issues regarding beacon nodes being extremely inefficient at certain operations given the number of validators currently in the network. With some new changes that are now live in latest master, our beacon node is a lot more resilient and should minimize or eliminate these deadlines albeit in extreme scenarios. Closing for now unless it is still seen in latest master.

@GuillaumeOcculy
Copy link
Author

GuillaumeOcculy commented Feb 4, 2020 •
edited

@rauljordan I still have the error after updated the validator image :

2020/02/04 17:06:37 maxprocs: Leaving GOMAXPROCS=1: CPU quota undefined
[2020-02-04 17:06:37]  INFO node: Using custom parameter configuration
[2020-02-04 17:06:37]  WARN node: Using Demo Config
[2020-02-04 17:06:43]  INFO node: Found validator keys validators=1
[2020-02-04 17:06:43]  INFO node: Validating for public key pubKey=0x92612e5ca5ad9ca0f2d016bf7ee0f21ec3e96a314537d4f03522595379aa9ab65c2b39d6afc43a44f4678aa864a25468
[2020-02-04 17:06:43]  INFO node: Starting validator node version=Prysm/Git commit: 80248cd296d441121373b733d9d971f85057d8b5. Built at: 2020-02-04 12:20:20+00:00
[2020-02-04 17:06:43]  INFO registry: Starting 2 services: [*prometheus.Service *client.ValidatorService]
[2020-02-04 17:06:43]  INFO prometheus: Collecting metrics at endpoint endpoint=:8080
[2020-02-04 17:06:43]  WARN validator: You are using an insecure gRPC connection! Please provide a certificate and key to use a secure connection.
[2020-02-04 17:06:43]  INFO validator: Successfully started gRPC connection
[2020-02-04 17:06:43]  INFO validator: Waiting for beacon chain start log from the ETH 1.0 deposit contract
[2020-02-04 17:06:43]  INFO validator: Beacon chain started genesisTime=2020-01-10 00:00:00 +0000 UTC
[2020-02-04 17:06:43]  INFO validator: Waiting for validator to be activated in the beacon chain
[2020-02-04 17:06:43]  INFO validator: Validator activated pubKey=0x92612e5ca5ad
[2020-02-04 17:06:43] ERROR validator: rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2020-02-04 17:06:43] ERROR validator: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2020-02-04 17:07:24]  INFO validator: Submitted new attestations AggregatorIndices=[] AttesterIndices=[35211] BeaconBlockRoot=0xf1178bf64f83 CommitteeIndex=0 Slot=185136 SourceEpoch=5779 SourceRoot=0x6c6b6cca0dcb TargetEpoch=5785 TargetRoot=0xf1178bf64f83

@GuillaumeOcculy
Copy link
Author

GuillaumeOcculy commented Feb 4, 2020 •
edited

@rauljordan
By the way I have several times this error on updated beacon-chain image:

[2020-02-04 17:31:03]  WARN blockchain: Calculated head diffs from new block blkRoot=1478d331b0cc803c6a91f894b06b8a0711c00d404b049e685984728d5163ed6e blkSlot=185248 headRoot=f1178bf64f8394e3f61dd650b092cc388bfe47abe46e706a18d533c4cc7e84d5 headSlot=185024
[2020-02-04 17:31:04]  INFO forkchoice: Executing state transition on block root=0xeecf9caf... slot=185255
[2020-02-04 17:31:15]  INFO blockchain: Finished applying state transition attestations=15 deposits=0 slot=185255
[2020-02-04 17:31:23] ERROR blockchain: Could not receive attestation in chain service aggregationCount=1 beaconBlockRoot=0xffb9196b5d4d committeeIndex=5 error=signature did not verify
github.com/prysmaticlabs/prysm/beacon-chain/core/blocks.init
	beacon-chain/core/blocks/block_operations.go:39
runtime.doInit
	GOROOT/src/runtime/proc.go:5222
runtime.doInit
	GOROOT/src/runtime/proc.go:5217
runtime.doInit
	GOROOT/src/runtime/proc.go:5217
runtime.doInit
	GOROOT/src/runtime/proc.go:5217
runtime.doInit
	GOROOT/src/runtime/proc.go:5217
runtime.main
	GOROOT/src/runtime/proc.go:190
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1357
could not verify indexed attestation
github.com/prysmaticlabs/prysm/beacon-chain/blockchain/forkchoice.(*Store).verifyAttestation
	beacon-chain/blockchain/forkchoice/process_attestation.go:229
github.com/prysmaticlabs/prysm/beacon-chain/blockchain/forkchoice.(*Store).OnAttestation
	beacon-chain/blockchain/forkchoice/process_attestation.go:123
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).ReceiveAttestationNoPubsub
	beacon-chain/blockchain/receive_attestation.go:47
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).processAttestation
	beacon-chain/blockchain/receive_attestation.go:113
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1357
could not process attestation from fork choice service
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).ReceiveAttestationNoPubsub
	beacon-chain/blockchain/receive_attestation.go:49
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).processAttestation
	beacon-chain/blockchain/receive_attestation.go:113
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1357 slot=185216 targetRoot=0xffb9196b5d4d

@prestonvanloon prestonvanloon self-assigned this Feb 5, 2020
@prestonvanloon
Copy link
Member

I'm reopening this to try to capture a profile of a deadline exceeded on get attestation data or get block.

@prestonvanloon prestonvanloon reopened this Feb 5, 2020
@prestonvanloon
Copy link
Member

I have a feeling this happens when the user is requesting data which requires an epoch transition. Epoch transitions can often be expensive so I have been looking for ways to make it more efficient. I've submitted #4766 which might help reduce the frequency of this issue.

I still haven't captured a CPU profile of the timed out API call, but hopefully soon.

I'll keep hunting for bottlenecks...

@prestonvanloon prestonvanloon mentioned this issue Feb 6, 2020
Merged
@rauljordan
Copy link
Contributor

Can this be closed @prestonvanloon ?

@rauljordan
Copy link
Contributor

Has not been seen in a while and unable to reproduce through local testing - closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prestonvanloon prestonvanloon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@GuillaumeOcculy @rauljordan @prestonvanloon @Domintek @clabornd