Skip to content

Fix Lighthouse v8.1.0 SSE race condition and reward calculation bugs#219

Merged
leobago merged 15 commits into
masterfrom
dev
Feb 17, 2026
Merged

Fix Lighthouse v8.1.0 SSE race condition and reward calculation bugs#219
leobago merged 15 commits into
masterfrom
dev

Conversation

@Zyra-V21
Copy link
Copy Markdown
Collaborator

@Zyra-V21 Zyra-V21 commented Feb 16, 2026

Summary

This PR merges all recent dev changes into master, addressing two main areas:

Lighthouse v8.1.0 SSE timing fix

Lighthouse v8.1.0 (commit cd8049a69) emits the Head SSE event before the block and state are fully committed. This causes GotEth to fetch stale data when following the chain head, resulting in f_proposed=false, f_timestamp=0, and f_attestations=0 for affected epochs.

Changes:

  • Cache state root from Head SSE events and use root-based state lookup with retry at epoch boundaries (8e21b77, f5db50c, 0e35d96, ecaa493, 8a7ba8e, 5e995b9)
  • Retry block downloads on 404 instead of immediately marking blocks as missing (66d1730)

Validated on a test machine running Lighthouse v8.1.0 + GotEth against mainnet: all blocks correctly show f_proposed=true with proper f_timestamp and f_attestations values after applying the fix.

Reward calculation and overflow fixes

These commits were merged to dev prior to the Lighthouse fix and address several issues found in reward/fee calculations:

  • eba4883fix: use NextState for consolidated amounts in EpochReward calculation
    Consolidated validator balances were being read from the current state instead of the next state. Since consolidations take effect at the start of the next epoch, the amounts must be sourced from NextState to reflect the actual post-consolidation balances.

  • d74d19bfix: change aggregated_rewards from UInt64 to Int64 in t_pool_summary
    Aggregated rewards can be negative (e.g. penalties exceeding rewards in a given epoch). The column type was UInt64 which caused underflow wrapping negative sums to huge values near 2^64.

  • 63bb131chore: add math import for overflow protection in BlockGasFees
    Adds the math package import required by the overflow protection logic.

  • 51f6713fix: add overflow protection in BlockGasFees calculation
    Large gas prices or unusual fee values could cause uint64 overflow when computing (gasPrice - baseFee) * gasUsed, corrupting EL fee data. Added validation that gasPrice >= baseFee, a cap on maximum reasonable transaction fee, and overflow checks before accumulating totals.

  • 563ad4cfix: adjust maxReasonableTxFee to fit uint64 range
    The initial maxReasonableTxFee constant exceeded uint64 range. Adjusted to ~184 ETH which still catches anomalous fees while fitting within bounds.

  • ab95a42fix: correct consolidation timing in EpochReward calculation
    Consolidations are processed at the start of the epoch, not at the end. The reward calculation was applying consolidation adjustments at the wrong point, causing incorrect balance diffs for validators involved in consolidations.

  • 1aa0a77fix: prevent double-counting consolidations in reward calculation
    When a source validator consolidates into a target, both the source's balance decrease and the target's balance increase were being counted separately. This fix ensures consolidation transfers are only accounted for once.

  • 5461500fix: add deposit tracking for reward calculation and fix bounds check
    External deposits were not being subtracted from the balance diff, causing deposited ETH to appear as earned rewards. Added processDepositsForRewardCalculation() to filter external deposits. Also fixed an array bounds check.

Closes

Files changed

File Change
pkg/analyzer/chain_analyzer.go Add epochBoundaryStateRoots sync.Map
pkg/analyzer/epoch_boundary_state_root.go New: cache helpers for epoch boundary state roots
pkg/analyzer/download.go Root-based state download with retry
pkg/analyzer/routines.go Cache state root from Head SSE events
pkg/clientapi/block.go Retry block download on 404
pkg/clientapi/state.go Add RequestBeaconStateByRoot
pkg/spec/block.go Overflow protection in BlockGasFees
pkg/spec/metrics/standard.go Bounds check for deposits
pkg/spec/metrics/state_electra.go Consolidation/deposit processing
pkg/db/migrations/000001_init_schema.up.sql Int64 for aggregated_rewards

Testing

  • Reproduced the bug: GotEth + Lighthouse v8.1.0 → all head-mode blocks incorrectly marked as f_proposed=false
  • Verified fix: same setup → all blocks correctly marked as f_proposed=true with valid metrics
  • Backwards compatible with Lighthouse v8.0.1 (slot-based fallback remains for non-cached states)
  • Goteth changes has been tested during months in testing site without any anomaly. --OK

Breaking Changes

None. Internal behavior change only.

Zyra-V21 and others added 15 commits January 22, 2026 13:53
- Fixes negative rewards for post-Electra validators with >32 ETH
- ConsolidatedAmounts should be fetched from NextState, not CurrentState
- Resolves GitHub issue #215
- Prevents UInt64 underflow when aggregating negative rewards
- Allows proper handling of negative validator rewards
- Resolves GitHub issue #214
Required for math.MaxUint64 to check for uint64 overflow before
accumulating transaction fees. Part of fix for issue #216.
- Check for underflow when GasPrice < baseFeePerGas
- Skip transactions with priority fee > 1e21 (>1000 ETH, likely corrupted)
- Check for uint64 overflow before accumulating fees
- Add logging for debugging suspicious values

Fixes #216
Go 1.25.6 stricter type checking revealed 1e21 exceeds uint64 max (2^64-1).
Changed to 1e19 Wei (~10 ETH) as practical sanity cap for tx fees.
Original 1e21 (1000 ETH) was impossible to represent in uint64 anyway.
The previous fix (eba4883) incorrectly used NextState.ConsolidatedAmounts,
which contains consolidations for the NEXT epoch (not yet applied to balances).

This fix:
- Reverts EpochReward to use CurrentState.ConsolidatedAmounts
- Adds processPendingConsolidations(CurrentState) in PreProcessBundle
- CurrentState consolidations = those processed at START of current epoch,
  which already affected NextState.Balances

Tested locally: Validator 2170445 epoch 424365 now shows correct reward
(~9k gwei) instead of corrupt -64 ETH.
The previous fix (ab95a42) incorrectly assumed CurrentState.ConsolidatedAmounts
would contain the consolidations affecting the current epoch's reward. However,
state objects are reused between iterations, causing processPendingConsolidations
to accumulate values across epochs.

Changes:
- Clear CurrentState.ConsolidatedAmounts at start of PreProcessBundle
- Add processConsolidationsForRewardCalculation() to identify consolidations
  processed at the START of NextState's epoch by comparing pending lists
- Only subtract consolidations that actually affected NextState.Balances

This fixes validators receiving consolidations showing negative rewards.
- Add processDepositsForRewardCalculation() to identify external deposits
  processed at the START of NextState's epoch
- Skip internal deposits (slot = 0) created by queue_excess_active_balance
  as these are balance restructuring, not new funds
- Clear DepositedAmounts at start of PreProcessBundle (state reuse fix)
- Add bounds check for NextState.Deposits array access

Note: Validator 1111619 issue (negative reward during switch_to_compounding)
is a deeper timing issue with how goteth reads beacon state balances, not
a reward calculation bug. The beacon node returns post-epoch-processing
balances instead of pre-processing balances.
Store a lightweight mapping slot -> stateRoot for epoch-boundary slots,
populated from Head SSE events. This avoids the racy slot-based state
resolution in Lighthouse v8.1.0+.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add setEpochBoundaryStateRoot() and takeEpochBoundaryStateRoot() to
isolate the caching mechanics for epoch-boundary state roots from
Head SSE events.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Member

@leobago leobago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@leobago leobago merged commit a848734 into master Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants