Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Make more noise when the EL is broken #3986

Closed
wants to merge 3 commits into from

Conversation

michaelsproul
Copy link
Member

@michaelsproul michaelsproul commented Feb 16, 2023

Issue Addressed

Closes #3814, replaces #3818.

Proposed Changes

  • Add a WARN log for the case where we are attempting to sync chain segments but can't process them because they're building on an invalid parent. The most common case where we see this is when the execution node database is corrupt, causing sync to stall mysteriously (because we're currently logging the failure only at debug level).
  • Additionally I've bumped up the logging for invalid execution payloads to WARN. This may result in some duplicate logs as we log errors from the beacon_chain and then again from the beacon processor. Invalid payloads and corrupt DBs should be rare enough that this doesn't produce overwhelming log volume.

@michaelsproul michaelsproul added v4.0.0 Mainnet Capella release expected late March 2023 low-hanging-fruit Easy to resolve, get it before someone else does! labels Mar 14, 2023
Copy link
Member

@pawanjay176 pawanjay176 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just a minor suggestion

@pawanjay176 pawanjay176 added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Mar 15, 2023
@michaelsproul
Copy link
Member Author

bors r+

bors bot pushed a commit that referenced this pull request Mar 17, 2023
## Issue Addressed

Closes #3814, replaces #3818.

## Proposed Changes

* Add a WARN log for the case where we are attempting to sync chain segments but can't process them because they're building on an invalid parent. The most common case where we see this is when the execution node database is corrupt, causing sync to stall mysteriously (because we're currently logging the failure only at debug level).
* Additionally I've bumped up the logging for invalid execution payloads to `WARN`. This may result in some duplicate logs as we log errors from the `beacon_chain` and then again from the beacon processor. Invalid payloads and corrupt DBs _should_ be rare enough that this doesn't produce overwhelming log volume.
@bors bors bot changed the title Make more noise when the EL is broken [Merged by Bors] - Make more noise when the EL is broken Mar 17, 2023
@bors bors bot closed this Mar 17, 2023
@michaelsproul michaelsproul deleted the louder-errors branch March 17, 2023 03:11
Woodpile37 pushed a commit to Woodpile37/lighthouse that referenced this pull request Jan 6, 2024
Closes sigp#3814, replaces sigp#3818.

* Add a WARN log for the case where we are attempting to sync chain segments but can't process them because they're building on an invalid parent. The most common case where we see this is when the execution node database is corrupt, causing sync to stall mysteriously (because we're currently logging the failure only at debug level).
* Additionally I've bumped up the logging for invalid execution payloads to `WARN`. This may result in some duplicate logs as we log errors from the `beacon_chain` and then again from the beacon processor. Invalid payloads and corrupt DBs _should_ be rare enough that this doesn't produce overwhelming log volume.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request low-hanging-fruit Easy to resolve, get it before someone else does! ready-for-merge This PR is ready to merge. UX-and-logs v4.0.0 Mainnet Capella release expected late March 2023
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants