Skip to content

Instrument attestation signing. #8508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mergify merged 3 commits into from
Dec 2, 2025
Merged

Instrument attestation signing. #8508

mergify merged 3 commits into from
Dec 2, 2025
+77 −66

Conversation

@jimmygchen
Copy link
Member

@jimmygchen jimmygchen commented Dec 1, 2025 •
edited
Loading

Issue Addressed

We noticed attestation signing taking 2+ seconds on some of our hoodi nodes and the current traces doesn't provide enough details. This PR adds a few more spans to the attestation_duty_cycle code path in the VC.

Before:
image

After:
image

@jimmygchen jimmygchen added val-client Relates to the validator client binary ready-for-review The code is ready for review labels Dec 1, 2025
pawanjay176
pawanjay176 reviewed Dec 1, 2025
View reviewed changes
validator_client/signing_method/src/lib.rs Outdated
"local_keystore_signer",
)
.ok_or(Error::ShuttingDown)?
.spawn_blocking_with_rayon_async(RayonPoolType::HighPriority, move || {
Copy link
Member

@pawanjay176 pawanjay176 Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why we are using rayon here? 🤔

Copy link
Member Author

@jimmygchen jimmygchen Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh just trying to isolate the possibility of spwan blocking handle causing the slowness. Going to deploy and see if there’s any differences in performance

Copy link
Member Author

@jimmygchen jimmygchen Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching to rayon seems to have improved the p50 numbers quite a bit:

image

with much higher CPU usage:
image

but p99 still looks pretty bad
image

Copy link
Member Author

@jimmygchen jimmygchen Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This trace is quite interesting, looks like check_and_insert_attestation is happening sequentially:

image

Copy link
Member Author

@jimmygchen jimmygchen Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've identified the bottleneck, and will make the fix in a separate PR. I'll revert the rayon change here

@jimmygchen jimmygchen added work-in-progress PR is a work-in-progress optimization Something to make Lighthouse run more efficiently. and removed ready-for-review The code is ready for review labels Dec 1, 2025
@michaelsproul michaelsproul added ready-for-merge This PR is ready to merge. and removed work-in-progress PR is a work-in-progress labels Dec 2, 2025
@mergify
Copy link

mergify bot commented Dec 2, 2025 •
edited
Loading

Merge Queue Status Beta

✅ The pull request has been merged

This pull request spent 38 minutes 53 seconds in the queue, including 37 minutes 20 seconds waiting for CI.
The checks were run on draft #8515.

Required conditions to merge
  • check-success=local-testnet-success
  • check-success=test-suite-success

@mergify mergify bot added the queued label Dec 2, 2025
mergify bot added a commit that referenced this pull request Dec 2, 2025
@mergify
Merge of #8508
a6cb6bb
@mergify mergify bot mentioned this pull request Dec 2, 2025
Closed
6 tasks
@michaelsproul michaelsproul mentioned this pull request Dec 2, 2025
Open
@mergify mergify bot merged commit 7ef9501 into sigp:unstable Dec 2, 2025
36 checks passed
@mergify mergify bot removed the queued label Dec 2, 2025
@jimmygchen jimmygchen deleted the instrument-attestation-signing branch December 2, 2025 05:40
kevaundray added a commit to eth-act/lighthouse that referenced this pull request Dec 13, 2025
@kevaundray @jimmygchen
@0xMushow @chong-he @michaelsproul @macladson @eserilev @pawanjay176
Merge unstable into optional-proofs (#2)
0fe4341 
* Update local testnet scripts for the fulu fork (sigp#8489)

* Remove `fulu-devnet-3` testing on CI
* Delete `scripts/local_testnet/network_params_das.yaml` and consolidate it into the main `network_params.yaml` file we use on CI
* Delete enclave before building image, so it doesn't cause slow image building.


  


Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>

* Fix data columns sorting when reconstructing blobs (sigp#8510)

Closes sigp#8509


  


Co-Authored-By: Antoine James <antoine@ethereum.org>

* Instrument attestation signing. (sigp#8508)

We noticed attestation signing taking 2+ seconds on some of our hoodi nodes and the current traces doesn't provide enough details. This PR adds a few more spans to the `attestation_duty_cycle` code path in the VC.

Before:
<img width="842" height="154" alt="image" src="https://github.com/user-attachments/assets/cfc5c8c0-e6f2-4f56-a8e4-65001af4a005" />

After:
<img width="496" height="217" alt="image" src="https://github.com/user-attachments/assets/c91cfa88-af1b-456e-8c64-625809eb6881" />


  


Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>

* Always use committee index 0 when getting attestation data (sigp#8171)

* sigp#8046


  Split the function `publish_attestations_and_aggregates` into `publish_attestations` and `handle_aggregates`, so that for attestations, only 1 task is spawned.


Co-Authored-By: Tan Chee Keong <tanck@sigmaprime.io>

Co-Authored-By: chonghe <44791194+chong-he@users.noreply.github.com>

Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>

Co-Authored-By: Michael Sproul <michael@sigmaprime.io>

* Move deposit contract artifacts to /target (sigp#8518)

Alternative to:

- sigp#8488


  Refactors deposit_contract crate to comply with Rust build conventions by placing generated artifacts in the build output directory.


Co-Authored-By: Michael Sproul <michael@sigmaprime.io>

* Move beacon state endpoints to a separate module. (sigp#8529)

Part of the http api refactor to move endpoint handlers to separate modules.

This should improve code maintainability, incremental compilation time and rust analyzer performance.


  


Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>

* Refactor `consensus/types` (sigp#7827)

Organize and categorize `consensus/types` into modules based on their relation to key consensus structures/concepts.
This is a precursor to a sensible public interface.

While this refactor is very opinionated, I am open to suggestions on module names, or type groupings if my current ones are inappropriate.


Co-Authored-By: Mac L <mjladson@pm.me>

* Move validator http endpoints to a separate module (sigp#8536)

Continuation of:
* sigp#8529

Moving `/validator` endpoints out of `http_api` to a separation module. This should improve code maintainability, incremental compilation time and rust analyzer performance.

This is a tedious but straight forward change, so we're going with a pair & insta-merge approach to avoid painful & slow async review. @michaelsproul and I paired on the first commit - I believe we are almost done, will pair with @pawanjay176 tomorrow to wrap it up and merge tomorrow. (cc @macladson )


  


Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>

* Move beacon pool http api to its own separate module (sigp#8543)

Continuation of:
* sigp#8536

Moving `/beacon/pool` endpoints out of `http_api` to a separation module. This should improve code maintainability, incremental compilation time and rust analyzer performance.

This is a tedious but straight forward change, so we're going with a pair & insta-merge approach to avoid painful & slow async review


  


Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>

* Reduce `eth2` dependency space  (sigp#8524)

Remove certain dependencies from `eth2`, and feature-gate others which are only used by certain endpoints.

| Removed | Optional | Dev only |
| -------- | -------- | -------- |
| `either` `enr` `libp2p-identity` `multiaddr` | `protoarray` `eth2_keystore` `eip_3076` `zeroize` `reqwest-eventsource` `futures` `futures-util` | `rand` `test_random_derive` |

This is done by adding an `events` feature which enables the events endpoint and its associated dependencies.
The `lighthouse` feature also enables its associated dependencies making them optional.

The networking-adjacent dependencies were removed by just having certain fields use a `String` instead of an explicit network type. This means the user should handle conversion at the call site instead. This is a bit spicy, but I believe `PeerId`, `Enr` and `Multiaddr` are easily converted to and from `String`s so I think it's fine and reduces our dependency space by a lot. The alternative is to feature gate these types behind a `network` feature instead.


Co-Authored-By: Mac L <mjladson@pm.me>

* Clarify `alloy` dependencies (sigp#8550)

Previously, we had a pinned version of `alloy` to fix some crate compatibility issues we encountered during the migration away from `ethers`. Now that the migration is complete we should remove the pin. This also updates alloy crates to their latest versions.


Co-Authored-By: Mac L <mjladson@pm.me>

* Remove `consensus/types` re-exports (sigp#8540)

There are certain crates which we re-export within `types` which creates a fragmented DevEx, where there are various ways to import the same crates.

```rust
// consensus/types/src/lib.rs
pub use bls::{
AggregatePublicKey, AggregateSignature, Error as BlsError, Keypair, PUBLIC_KEY_BYTES_LEN,
PublicKey, PublicKeyBytes, SIGNATURE_BYTES_LEN, SecretKey, Signature, SignatureBytes,
get_withdrawal_credentials,
};
pub use context_deserialize::{ContextDeserialize, context_deserialize};
pub use fixed_bytes::FixedBytesExtended;
pub use milhouse::{self, List, Vector};
pub use ssz_types::{BitList, BitVector, FixedVector, VariableList, typenum, typenum::Unsigned};
pub use superstruct::superstruct;
```

This PR removes these re-exports and makes it explicit that these types are imported from a non-`consensus/types` crate.


Co-Authored-By: Mac L <mjladson@pm.me>

* Fix testnet script (sigp#8557)

Fix an issue where a kurtosis testnet script was failing because no supernodes were provided


```
There was an error interpreting Starlark code
Evaluation error: fail: Fulu fork is enabled (epoch: 0) but no supernodes are configured, no nodes have 128 or more validators, and perfect_peerdas_enabled is not enabled. Either configure a supernode, ensure at least one node has 128+ validators, or enable perfect_peerdas_enabled in network_params with 16 participants.
at [github.com/ethpandaops/ethereum-package/main.star:83:57]: run
at [github.com/ethpandaops/ethereum-package/src/package_io/input_parser.star:377:17]: input_parser
at [0:0]: fail
```


  


Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>

Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>

* Do not request attestation data when attestation duty is empty (sigp#8559)

Co-Authored-By: Tan Chee Keong <tanck@sigmaprime.io>

* Rust 1.92 lints (sigp#8567)

Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>

---------

Co-authored-by: Jimmy Chen <jimmy@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
Co-authored-by: 0xMushow <105550256+0xMushow@users.noreply.github.com>
Co-authored-by: Antoine James <antoine@ethereum.org>
Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <michaelsproul@users.noreply.github.com>
Co-authored-by: Mac L <mjladson@pm.me>
Co-authored-by: Eitan Seri-Levi <eserilev@ucsc.edu>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Tan Chee Keong <tanck@sigmaprime.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pawanjay176 pawanjay176 pawanjay176 left review comments

@michaelsproul michaelsproul michaelsproul approved these changes

Assignees

No one assigned

Labels

optimization Something to make Lighthouse run more efficiently. ready-for-merge This PR is ready to merge. val-client Relates to the validator client binary

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jimmygchen @michaelsproul @pawanjay176