smite-scenarios: warm up and freeze Eclair's JIT before the snapshot - #161
smite-scenarios: warm up and freeze Eclair's JIT before the snapshot#161erickcestari wants to merge 2 commits into
Conversation
morehouse
left a comment
morehouse
left a comment
There was a problem hiding this comment.
I had a couple big issues when testing this locally:
- Eclair actually rejects concurrent
open_channelmessages, so ~90% of theopen_channelswe send during warmup get immediately rejected. - After snapshot Eclair is in a state where all its pending channel slots are occupied. Depending on timing of the first
SendOpenChannelafter that, Eclair may reject it due to "rate limit reached". I think we're warming up and snapshotting too fast, so Eclair doesn't have time to process the queue ofopen_channels and then remove pending channels after disconnect.
Probably we should pause after each open_channel message and wait for Eclair's accept_channel, so we can be sure that Eclair's processing is done. We then also probably want a sleep or ping-pong after each disconnect so that Eclair has time to tear the actors down and free up pending slots. This will be slower, but maybe we'll need fewer iterations if we actually wait to make sure Eclair fully processes everything before moving on.
I also have some ideas to improve further, but those are for follow-up work.
- Measure the individual effects of JIT freeze and warmup in isolation
- Try warming up more of the funding flow (e.g., through
funding_signed, mining blocks,channel_ready) - Try warming up gossip paths
| let directive = std::env::temp_dir().join("smite-jit-exclude-all.json"); | ||
| fs::write(&directive, "[ { match: [\"*.*\"], Exclude: true } ]\n")?; | ||
| match self.run_jcmd(&["Compiler.directives_add", &directive.to_string_lossy()]) { |
There was a problem hiding this comment.
I'd be curious how much difference the JIT freeze makes on its own -- is this the load-bearing change or is it the warmup?
There was a problem hiding this comment.
I'll try to also bench it to see the results.
There was a problem hiding this comment.
I've done the bench. You can see the results below.
| // Sync so Eclair processes (and JIT-compiles) the whole batch before conn | ||
| // drops at the end of this iteration, releasing its pending-channel slots. | ||
| ping_pong(&mut conn)?; |
There was a problem hiding this comment.
Eclair actually processes open_channel messages async, so the only way to be sure it has processed the whole batch is to wait for all the accept_channel messages. Eclair also rejects concurrent open_channels, so we actually need to send them one at a time and wait for each response before sending the next one. A few other benefits of doing this:
- More thorough warmup, potentially requiring much fewer iterations.
- We can detect if our
open_channelmessages ever start getting rejected (e.g., due to Eclair update) and raise the alarm.
There was a problem hiding this comment.
Nice, I've fixed that. Now it waits for the accept_channel and then sends an error message for eclair be able to clean the state.
| if let Ok(opts) = std::env::var("SMITE_ECLAIR_JAVA_OPTS") { | ||
| cmd.env("JAVA_OPTS", opts); | ||
| } |
There was a problem hiding this comment.
This ends up overriding the JAVA_OPTS set in init.sh, which includes our coverage agent and the C1 compiler flag. I think we probably want to append to JAVA_OPTS here instead.
There was a problem hiding this comment.
The idea is to override even the JAVA_OPTS that exists at init.sh so it's easier to test different options without the need to build a new docker image.
There was a problem hiding this comment.
I have updated the docs to make it more clear.
Sorry. 🤦 That explains why it was so fast. I'm working on fixing it, applying your suggestions and trying new approaches to improve the exec/s even more. |
337789d to
cc6c54e
Compare
|
Applied an input that is able to finish the before: after: after without jit_freeze: |
|
The second commit is needed since reducing the Nyx VM from 4096 MB to 2048 MB halves the tmpfs cap on /tmp, leaving too little space to unpack |
Eclair's channel path is still interpreted when the Nyx snapshot is taken, so every restored execution re-pays the cold-JVM cost. Drive open_channel exchanges before the snapshot so HotSpot compiles that path, then freeze the JIT with a catch-all Exclude directive via jcmd, so no compiler threads run during fuzzing. The image ships the full JDK for jcmd, and the JVM runs with -XX:TieredStopAtLevel=1: C2 measured worse here, since its speculation is tuned to the warmup path and varied fuzz inputs trip uncommon traps and deoptimize on every restore. Opens are strictly sequential and each is retired with an error for its temporary channel id. Eclair evaluates opens in a single interceptor actor per peer and rejects anything arriving while it is busy, so pipelined opens warm nothing; the error closes the channel actor and frees the pending-channel slot, which is otherwise only released on disconnect. A rejected open aborts the setup rather than being retried: it means the channel path never ran, and the cause is not transient. Worth roughly 2x execs/sec against no warmup, measured with smitebot bench-exec.
The Nyx guest has no disk: /tmp is a tmpfs that must hold the container tarball, the tree extracted from it, and every byte the target writes at runtime. Its default cap is half the VM's RAM, so unpacking a 600 MB image needs a 3 GB VM; below that, tar fails with ENOSPC and the agent aborts with an empty message that says nothing about what went wrong. Raise the cap to 90% and drop the tarball once it is unpacked, so NYX_MEM_MB is the only knob that governs how much memory the target actually gets. The default 2048 MB now boots and snapshots Eclair with JIT warmup enabled, where it previously required 4096.
cc6c54e to
5acee05
Compare
2 participants
Snapshot-restore fuzzing of Eclair started every iteration on a cold JVM, so the channel path ran interpreted. Add an Eclair-specific SnapshotSetup that, before the snapshot is taken:
Keeps C1 only (-XX:TieredStopAtLevel=1): C2 measured ~28% slower here as its speculation deopts on varied fuzzing inputs. The runtime image switches to the full JDK for jcmd.
Warming the JVM before taking the snapshot increased steady-state fuzzing throughput from 2.3 to 3.9 execs/sec for or the following input, a 1.70× (69.6%) improvement. (I've used #160 to benchmark it).
We can see the JVM logs optimizing the code with:
Logs:
Input
Before
After
Target: eclair
1. Summary Statistics
Median Coverage Over Time
Distribution Comparisons