Effectiveness Evaluation Framework for Different Mutator Configurations

[Chand-ra]

Use this statistical framework to rigorously compare two IR‑mutator configurations (e.g., “full stack” vs. “full stack + new operator”) across Lightning implementations. The primary metric is edge coverage growth over 24‑hour campaigns.

The framework is largely inspired by Evaluating Fuzz Testing (Klees et al., 2018).

1. What You Compare

Both configurations run the IR fuzzing scenario against the same Lightning targets, starting from the same snapshot. The only difference is the set of mutators.

	Configuration A	Configuration B
Mutators	e.g., current Full Stack	e.g., Full Stack + your new operator
Scenario	IR scenario (IrScenario)	Same
Snapshot	PostInitSetup (or advanced, but identical for both)	Same
Generators	Existing suite	Same
Seeds	Empty corpus, or a small set of simple programs (see Section 2)	Same
Engine	AFL++ (latest stable) + Nyx, fast schedule	Same

Why everything except the mutators is identical?

We want to isolate the effect of the mutator changes. If we also changed the scenario, snapshot, or seeds, any observed difference in coverage growth could be caused by those factors rather than the mutators themselves. Keeping all other variables constant lets us attribute differences directly to the mutation strategy.

2. Seed Corpus

The IR fuzzer includes generators that produce valid programs at runtime, an empty initial corpus is acceptable and often simplest. The generators will populate the queue with structurally sound seeds.

Alternatively, you may start with a small set of simple IR programs (e.g., ping/pong, a minimal open_channel). These give a slight head start but are not strictly necessary.

Important: If the IR format changes (e.g., a new instruction set or a new postcard schema), all seeds must be regenerated from the new IR. Never mix seeds produced by different IR versions in the same comparison: that would introduce a confounding variable.

• Same seeds for all trials: Use the identical seed set for every trial (empty or not) and every configuration to ensure fairness.

3. Trial Structure

• Number of trials: 30 independent trials per configuration, per target. This number is recommended by Klees et al. (2018) to achieve reasonable statistical power for detecting meaningful differences.
• Duration: Each trial runs for 24 hours (wall‑clock). This is long enough to observe substantial coverage growth and plateaus.
• Parallelism:
  • Each trial is a single AFL++ process, so it uses exactly one CPU core.
  • You do not need to give each trial a dedicated machine: multiple trials can share a multi‑core machine by running in parallel, each pinned to a physical core to avoid cache contention.
  • For example, on a 30‑core server you can run all 30 trials in parallel, each pinned to a physical core. The entire campaign for one configuration per target finishes in slightly over 24 hours.

4. Data to Collect

For each trial, collect from AFL++’s output directory:

• plot_data – time series of edges_found. We extract two metrics:

  • Final coverage (the number of edges found at the 24‑hour mark). This is the primary metric.
  • AUC (Area Under the edge‑Coverage Curve) as a secondary metric that summarises early exploration speed.

  Why AUC? It summarises the entire coverage growth trajectory: a higher AUC means the fuzzer discovered more edges, sooner.

• fuzzer_stats – we only care about execs_per_sec (to check if one configuration trades coverage quality for raw speed). We ignore other stats (e.g., pending_favs, cycles_done, stability) because they are either redundant with coverage data or not directly relevant to our comparison of coverage growth.

• fuzz_bitmap – Used to calculate union coverage (see section 5.8).

5. Analysis (per target)

For each target separately:

1. Extract metrics from each trial:

   • Primary metric: The final coverage value (the edges_found at the 24‑hour mark).
   • Secondary metric (AUC): Compute the Area Under the edge‑Coverage Curve from plot_data using trapezoidal integration. AUC summarises early exploration speed and is reported alongside final coverage for additional context.

2. Sanity‑check visualisation (coverage‑over‑time):

   For each configuration, plot the median edges_found over time with IQR bands. If the two configurations' bands overlap for most of the campaign and only diverge near hour 24, the final‑coverage result should be interpreted cautiously: the advantage may be driven by late, noisy differences rather than sustained improvement.

3. Visualise: Box‑plots of final-coverage values for Configuration A and B.

4. Statistical test: Use the Mann‑Whitney U test (also called Wilcoxon rank‑sum).

   • Why not a t‑test or just compare means? Fuzzing metrics like final-coverage are almost never normally distributed. The Mann‑Whitney U test is non‑parametric: it doesn’t assume normality, and instead tests whether one configuration tends to produce higher final-coverage values more often than the other.
   • Why not simply compare medians? Medians alone don’t tell you if the difference is statistically reliable. The Mann‑Whitney test assesses whether the observed difference could be due to random chance, and it works on the full distributions, not just a central value.
   • The test gives a p‑value: the probability of seeing a difference at least as extreme as observed, assuming the two configurations are equally effective.

5. Measure the magnitude of difference: We use the Vargha‑Delaney $\hat{A}_{12}$ statistic.

   It measures the probability that a randomly chosen trial from Configuration B will have a higher final-coverage than a randomly chosen trial from Configuration A.

   • $\hat{A}_{12} = 0.5$ → no difference (both equally likely to win).
   • $\hat{A}_{12} > 0.5$ → B tends to outperform A.
   • For example, $\hat{A}_{12} = 0.68$ means that in about 68% of random pairings, B’s final-coverage is larger than A’s.

   This is straightforward to compute: it’s the Mann‑Whitney U statistic divided by the product of the two sample sizes ($n_A \times n_B$).

6. Report:

   • Median final-coverage and AUC for each configuration.
   • Inter‑quartile range (IQR) = difference between the 75th and 25th percentiles.
     • Why IQR? The IQR captures the spread of the middle 50% of the data. A small IQR means the fuzzer’s performance is consistent across trials; a large IQR indicates high variability. If Configuration B has a higher median but also a much wider IQR, the improvement might be less reliable: some runs may perform exceptionally well while others lag. Reporting both median and IQR gives a fuller picture than a single number.
   • Raw p‑value from the Mann‑Whitney test.
   • Effect size $\hat{A}_{12}$.

7. Adjusting for multiple comparisons:

   When you test multiple targets (up to 4), the chance of getting at least one false‑positive (p < 0.05) simply by luck increases. This is easiest to see with a short calculation.

   If you run an independent test on each of $m$ targets and use the standard $\alpha = 0.05$ threshold, the probability that all tests correctly fail to reject the null hypothesis (no false positives) is $(1 - 0.05)^m = 0.95^m$.

   For $m = 4$, that is $0.95^4 \approx 0.8145$.

   Thus, the probability of at least one false positive is $1 - 0.8145 = 0.1855$, or about 18.6%. That is, you have a nearly one‑in‑five chance of concluding a difference exists when in fact none does. That’s too high for a rigorous evaluation.

   The Holm‑Bonferroni correction controls this family‑wise error rate so that the overall probability of any false positive remains ≤ 0.05, regardless of how many targets you test. It works by adjusting each raw p‑value according to its rank, making it harder to reach significance when many tests are performed.

   The procedure:

   • Sort the p‑values from smallest to largest: $p_1 \le p_2 \le p_3 \le p_4$.
   • Compute adjusted values:
     $$p_{\text{adj},i} = \min(1, ; p_i \times (m - i + 1))$$, where $m$ is the number of targets (e.g., 4).
   • Compare each $p_{\text{adj},i}$ to 0.05. If the smallest adjusted p‑value is < 0.05, that target’s difference is significant; then proceed to the next, and so on. Stop at the first non‑significant one.

8. Union coverage (descriptive only):

   For each configuration, perform a byte‑wise maximum (logical OR) across all 30 bitmaps, then count the number of non‑zero bytes. This gives the union of edges covered by any trial, representing the coverage ceiling for a multi‑core deployment. This is a single scalar per configuration, so it cannot be statistically tested. Report it purely as a practical upper‑bound estimate.

6. Interpretation

• Why 0.05? It’s the conventional significance level in empirical sciences, meaning there’s only a 5% risk of concluding a difference exists when in fact it doesn’t. It’s not a magic number, but a widely accepted trade‑off between sensitivity and false positives.

• If Configuration B gives a higher median final-coverage, an adjusted p‑value < 0.05, and $\hat{A}_{12} &gt; 0.5$ (say > 0.56):

  The new mutator likely improves coverage growth. The effect is statistically significant, and the effect size indicates a meaningful (not just trivial) advantage. Check the IQR: a smaller or comparable IQR relative to A suggests the improvement is consistent.

• If adjusted p‑value < 0.05 but $\hat{A}_{12}$ is very close to 0.5 (e.g., 0.51):

  The difference is statistically significant but practically tiny. The new mutator provides a marginal, possibly negligible, improvement. Consider whether the extra code complexity is worth it.

• If adjusted p‑value > 0.05 but $\hat{A}_{12}$ is large (e.g., > 0.6):

  The effect is substantial, but your trial count might be too low to reach significance. Consider increasing the number of trials (e.g., to 50) before drawing firm conclusions. Always look at the effect size alongside the p‑value.

• If adjusted p‑value > 0.05 and $\hat{A}_{12}$ is very close to 0.5:

  There is no evidence that the new mutator changes coverage. The observed difference is well within the range of random noise. The mutator likely has no practical impact; consider whether it might still be useful in niche protocol states not captured by the default snapshot, or whether a larger experiment (more trials) could reveal a subtle effect.

• If $\hat{A}_{12}$ < 0.5 (Configuration B tends to produce lower final-coverage than A):

  The new mutator might degrade coverage. If this trend is statistically significant (adjusted p < 0.05), the mutator is harmful and should be rejected or redesigned. If not significant, the negative trend is weak and could be due to chance; more trials can clarify.

• IQR interpretation:

  • A configuration with higher median and smaller IQR is clearly superior: it consistently achieves better coverage.
  • A configuration with higher median but larger IQR may have a few excellent runs masking many mediocre ones; the advantage is less reliable.
  • If the IQRs overlap substantially, the difference in medians may not be meaningful.

• Check execution speed: Report median execs_per_sec. If Configuration B achieves equal or better coverage at a lower execution speed, the mutations are more efficient. If speed is much lower without coverage gain, the mutator may be too expensive.

7. Quick Sanity Checks

• Throughput (execs_per_sec): compare medians; large discrepancies should be discussed.
• Hardware noise: If you run many parallel trials, monitor whether execution speed varies across cores. You can group trials by throughput and re‑run the analysis on each group to check if results hold.

8. What to Do with the Results

• Coverage improves significantly → consider merging the new mutator into the default set.
• No significant difference → the mutator might still be useful for specific protocol corners; test on advanced snapshots (e.g., PostChannelOpenSetup) or run the full ground‑truth evaluation to see if it helps find real bugs.
• Coverage degrades → examine whether the mutations are too destructive, too slow, or interfere with other operators.

9. Walkthrough Examples

Single Target Example

9.1 Single Target (CLN)

Configurations:

• A (Baseline): Current full mutator stack (InstructionReorder, InputSwap, OperationParam, Splice, GeneratorInsertion).
• B (Experimental): Baseline + a new InstructionDelete mutator.

We run 30 trials of each configuration on the same machine (30 cores, each trial pinned to a physical core), all starting from the same 5 IR program seeds and the PostInitSetup snapshot.

9.1.1 Final-coverage from plot_data

For each trial, AFL++ writes plot_data as a table of timestamps (Unix seconds) and cumulative edge counts. Consider one trial from Configuration A, whose plot_data (abbreviated) looks like:

Time (hours)	edges_found
0	0
1	100
2	150
4	170
8	175
24	180

Thus, this trial’s final-coverage = 180.

For Configuration A, suppose the 30 resulting final-coverage values (after rounding) are:

152, 148, 155, 147, 150, 153, 149, 151, 148, 154, 150, 149, 152, 147, 153, 148, 151, 150, 149, 152, 154, 148, 150, 151, 149, 153, 150, 148, 152, 151

For Configuration B:

160, 162, 158, 163, 159, 161, 164, 160, 162, 161, 159, 163, 160, 162, 164, 161, 163, 160, 162, 164, 161, 159, 163, 160, 162, 164, 161, 163, 160, 162

9.1.2 Medians and IQRs

• A: Median (15th/16th average of sorted values) = 150.5; Q1 (25th percentile) = 149, Q3 (75th percentile) = 152 → IQR = 152 - 149 = 3.
• B: Median = 162. Q1 = 160, Q3 = 163 → IQR = 3.

Both IQRs are small and equal, indicating consistent performance in each configuration.

9.1.3 Mann‑Whitney U test

We rank all 60 final-coverage values together. Every value from B (min 158) is larger than every value from A (max 155). Therefore, the Mann‑Whitney U statistic for B vs. A is $30 \times 30 = 900$, the maximum possible.

The exact p‑value can be calculated using an online calculator, and is < 0.0001. This means the probability of such an extreme separation happening by chance is essentially zero.

9.1.4 Effect size

$\hat{A}_{12} = U / (n_A \times n_B) = 900 / 900 = 1.0$.

A randomly chosen trial from Configuration B always has a higher final-coverage than a randomly chosen trial from A. A massive effect.

9.1.5 Execution speed

From fuzzer_stats:

• Median execs_per_sec: A = 125, B = 122 (slight drop).

9.1.6 Interpretation

• Statistically significant: p < 0.05 (no correction needed for a single target).
• Effect size $\hat{A}_{12} = 1.0$ – overwhelming.
• Performance is consistent (identical IQRs).
• The negligible speed loss (125 → 122 execs/sec) is far outweighed by the coverage gain.

Conclusion: The InstructionDeleteMutator substantially improves coverage on CLN. Merge it into the default mutator set.

Note: The separation in this example is artificially clean for illustration. Real‑world coverage distributions typically show overlap and noise, as seen in the multi‑target example below.

The same procedure applies when using AUC as the metric: we’d compute medians, IQRs, and p‑values.

Multiple Targets Example

9.2 Multiple Targets (All Four Implementations)

Now suppose we repeat the same experiment on LND, LDK, and Eclair in addition to CLN. All other parameters (30 trials, 24 hours, identical seeds and snapshot) remain the same. We obtain:

Target	Median A	Median B	IQR A	IQR B	raw p‑value	$\hat{A}_{12}$
CLN	150.5	162	3	3	< 0.0001	1.00
LND	210	215	12	14	0.032	0.61
LDK	180	182	5	6	0.410	0.53
Eclair	90	105	8	9	0.007	0.66

9.2.1 Holm‑Bonferroni correction

We have $m = 4$ tests. Sort the raw p‑values from smallest to largest:

1. CLN: < 0.0001
2. Eclair: 0.007
3. LND: 0.032
4. LDK: 0.410

Apply the formula $p_{\text{adj},i} = \min(1, ; p_i \times (m - i + 1))$:

• CLN: $p_{\text{adj}} = 0.0001 \times 4 \approx 0.0004$
• Eclair: $p_{\text{adj}} = 0.007 \times 3 = 0.021$
• LND: $p_{\text{adj}} = 0.032 \times 2 = 0.064$
• LDK: $p_{\text{adj}} = 0.410 \times 1 = 0.410$

Compare each to 0.05:

• CLN: significant (0.0004 < 0.05)
• Eclair: significant (0.021 < 0.05)
• LND: not significant (0.064 > 0.05) – stop here; LDK is automatically not significant.

9.2.2 Interpretation per target

• CLN: Huge, significant effect ($\hat{A}_{12}=1.0$). Clear win for B.
• Eclair: Significant with a medium‑large effect ($\hat{A}_{12}=0.66$). IQRs comparable → consistent improvement.
• LND: Raw p = 0.032 looked promising, but after correction it is 0.064 (not significant). The effect size (0.61) suggests a moderate advantage; with more trials (e.g., 50) it might become significant.
• LDK: No significant difference ($\hat{A}_{12}=0.53$ nearly 0.5). The mutator has essentially no effect here.

9.2.3 Overall conclusion

The new InstructionDeleteMutator clearly and significantly improves coverage on CLN and Eclair, shows a promising but unconfirmed trend on LND, and has no impact on LDK. The variability suggests the mutator’s benefit may depend on implementation‑specific code patterns. We recommend merging it, while considering a follow‑up experiment with more trials on LND. The same evaluation pipeline can be re‑run if the mutator is later refined.

10. Future Work

• Automated evaluation script: A single script that runs the full per‑target analysis pipeline from a directory of plot_data and fuzzer_stats files and outputs a summary table like the one in Section 9.2, alongside box-plots for visualization.

For the formal evaluation that performs ground truth evaluation with real historical vulnerabilities using survival analysis, see the artifact at [WIP].

[Chand-ra]

I've tried to explain each statistical tool used in relevant detail, and the links lead to further reading if interested, but it is very possible that I under-explained something because it was obvious to me. Let me know, and I'll add more clarifications.

Other feedback is also very welcomed.

Also, this framework isn't necessarily limited to mutators. Anything that might have an effect on the fuzzer's coverage-over-time capabilities, like a custom scheduler for mutators, can be compared against a suitable baseline using this framework. The only prerequisite is to isolate the effect of whatever we're testing by making every other configuration parameter same in: our chosen baseline and baseline + new feature.

[morehouse]

Good work -- this is well thought out.

The overall framework LGTM, but I think we may want to change some of the evaluation metrics. See comments inline.

2. Seed Corpus

* **Why not just an empty input?**
  An empty input provides no structural guidance. As _Klees et al. (2018)_ show, a well‑chosen seed corpus can dramatically improve fuzzing effectiveness, and a completely empty corpus often results in poor initial coverage growth. Using a small set of valid, simple IR programs gives the fuzzer a realistic foothold without favouring any particular code path.

This is true, but we also have generators that basically create seeds during execution. So I think it may also be defensible to start from an empty corpus if we want.

* **What seeds to use?**
  Start with 5 simple IR programs (e.g., a `ping`/`pong` exchange, a minimal `open_channel`). _Alternatively_, you can run a short pilot campaign with the current mutator set, collect the queue, minimise it, and sample a few inputs as a trimmed corpus. This is fully automated and may better reflect what a real fuzzer sees.

If we do either approach, it will be important to regenerate the seeds any time the IR changes since the postcard format is generally not backwards compatible. (And of course we wouldn't want to compare trials across different IRs either since that would be a confounding variable).

3. Trial Structure

* **Number of trials:** 30 independent trials per configuration, per target. This number is recommended by _Klees et al. (2018)_ to achieve reasonable statistical power for detecting meaningful differences.

* **Duration:** Each trial runs for 24 hours (wall‑clock). This is long enough to observe substantial coverage growth and plateaus.

* **Parallelism:**
  
  * Each trial is a single AFL++ process, so it uses exactly one CPU core.
  * You do _not_ need to give each trial a dedicated machine: multiple trials can share a multi‑core machine by running in parallel, each pinned to a physical core to avoid cache contention.
  * For example, on a 30‑core server you can run all 30 trials in parallel, each pinned to a physical core. The entire campaign for one configuration per target finishes in slightly over 24 hours.

If we have limited resources, we might consider evaluating only a single target for each configuration. It's less thorough but may be good enough to demonstrate that a new mutator is valuable.

4. Data to Collect

For each trial, collect from AFL++’s output directory:

* **`plot_data`** – time series of `edges_found`. We will compute the **Area Under the edge‑Coverage Curve (AUC)**.
  _Why AUC?_ It summarises the entire coverage growth trajectory: a higher AUC means the fuzzer discovered more edges, sooner.

AUC is interesting and worth calculating, but I wouldn't consider it the primary metric we care about. A configuration that reaches high coverage initially and then plateaus would have a higher AUC than a configuration that increases coverage slowly but eventually achieves higher coverage, yet we would consider the second configuration to generally be better for finding bugs.

I think a better primary metric is the median coverage value after 24h.

* **`fuzzer_stats`** – we only care about **`execs_per_sec`** and **`stability`**.
  
  * **Stability:** Stability monitors target non-determinism. Persistent stability below 90% indicates the snapshot or target environment has non-deterministic behavior that should be investigated before trusting coverage measurements.
  * **`execs_per_sec`:** To check if one configuration trades coverage quality for raw speed.

We do care about stability and exec speed, but they are secondary to the total coverage achieved. For example, we could achieve 100% stability and millions of execs/sec by creating a configuration that effectively does nothing, but it would be impossible to actually find bugs.

Also note that stability values are currently very bad for most of the supported fuzzing targets (~40%) due to non-deterministic thread scheduling, RNG, etc. in the targets, and we basically ignore the metric. This is due to the targets themselves, and I wouldn't expect new mutator configs to have a significant effect.

5. Analysis (per target)

IIUC, we should be able to use the same statistical framework but replace AUM with the final coverage metric instead.

5. **Report:**
   
   * **Median AUC** for each configuration.
   * [**Inter‑quartile range (IQR)**](https://en.wikipedia.org/wiki/Interquartile_range) = difference between the 75th and 25th percentiles.
     
     * **Why IQR?** The IQR captures the spread of the middle 50% of the data. A small IQR means the fuzzer’s performance is consistent across trials; a large IQR indicates high variability. If Configuration B has a higher median but also a much wider IQR, the improvement might be less reliable: some runs may perform exceptionally well while others lag. Reporting both median and IQR gives a fuller picture than a single number.
   * Raw **p‑value** from the Mann‑Whitney test.
   * **Effect size** 

I would be very interested in seeing median coverage-over-time plots with IQR bands, along with box plots for the final coverage values.

It also would be very useful to have the union of achieved coverage across all trials, as this would indicate what kind of coverage we could expect to achieve from multi-core fuzzing campaigns (how we use smite in practice). Probably the simplest way to calculate this value is to compute bitwise-AND of all the fuzz_bitmap files across a config's trials, then count the number of bytes that are not 0xff.

[Chand-ra]

This is true, but we also have generators that basically create seeds during execution. So I think it may also be defensible to start from an empty corpus if we want.

* **What seeds to use?**
  Start with 5 simple IR programs (e.g., a `ping`/`pong` exchange, a minimal `open_channel`). _Alternatively_, you can run a short pilot campaign with the current mutator set, collect the queue, minimise it, and sample a few inputs as a trimmed corpus. This is fully automated and may better reflect what a real fuzzer sees.

If we do either approach, it will be important to regenerate the seeds any time the IR changes since the postcard format is generally not backwards compatible. (And of course we wouldn't want to compare trials across different IRs either since that would be a confounding variable).

Hmm, it is true that Klees et al. (2018) primarily worked with byte-level fuzzers so their suggestion is primarily structured around such. It is also true that we don't want to compare seeds across different IR's.

Since we generate seeds on-the-fly with Generators, I think it's fine to start with an empty corpus, but we definitely want the same Generator and IR combination for each mutator. I'll add this to the framework.

If we have limited resources, we might consider evaluating only a single target for each configuration. It's less thorough but may be good enough to demonstrate that a new mutator is valuable.

True, and that is shown with the single-target example below. However, I don't want to suggest it here because it is very possible to have a config that doesn't produce much advantage on one target but a lot with some other, so it's best to run experiments on every target that we care about.

AUC is interesting and worth calculating, but I wouldn't consider it the primary metric we care about. A configuration that reaches high coverage initially and then plateaus would have a higher AUC than a configuration that increases coverage slowly but eventually achieves higher coverage, yet we would consider the second configuration to generally be better for finding bugs.

True.

I think a better primary metric is the median coverage value after 24h.

Sure, I'll make AUC the secondary metric and median coverage the primary.

We do care about stability and exec speed, but they are secondary to the total coverage achieved. For example, we could achieve 100% stability and millions of execs/sec by creating a configuration that effectively does nothing, but it would be impossible to actually find bugs.

For sure, these are secondary and sort of sanitary-metrics to complement the initial coverage metric.

Also note that stability values are currently very bad for most of the supported fuzzing targets (~40%) due to non-deterministic thread scheduling, RNG, etc. in the targets, and we basically ignore the metric. This is due to the targets themselves, and I wouldn't expect new mutator configs to have a significant effect.

Right, mutators aren't supposed to generate invalid programs so they don't really contribute to stability. Since it is already pretty low, I guess we can get rid of it in the evaluation framework for now.

IIUC, we should be able to use the same statistical framework but replace AUM with the final coverage metric instead.

Yup.

I would be very interested in seeing median coverage-over-time plots with IQR bands, along with box plots for the final coverage values.

It also would be very useful to have the union of achieved coverage across all trials, as this would indicate what kind of coverage we could expect to achieve from multi-core fuzzing campaigns (how we use smite in practice). Probably the simplest way to calculate this value is to compute bitwise-AND of all the fuzz_bitmap files across a config's trials, then count the number of bytes that are not 0xff.

Sure, sounds informative. However, union coverage across 30 trials is a single scalar per configuration, we can't apply Mann-Whitney or compute IQR on it. Hence, this metric should be descriptive only (a practical upper-bound estimate for multi-core deployment).

Also, "bitwise-AND and count bytes not 0xff" is not union coverage. It would instead measure something like "common coverage" (edges where all trials saturated). Union-coverage should be computable by:

1. Take all fuzz_bitmap files from the trials of one configuration.

2. Perform a byte-wise maximum (or logical OR) across all bitmaps.

3. Any byte > 0 indicates that edge was covered by at least one trial.

4. Count the number of non-zero bytes → union edge coverage.

I'll add it to the framework.

[morehouse]

This all sounds good.

Also, "bitwise-AND and count bytes not 0xff" is not union coverage. It would instead measure something like "common coverage" (edges where all trials saturated). Union-coverage should be computable by:

1. Take all `fuzz_bitmap` files from the trials of one configuration.

2. Perform a byte-wise maximum (or logical OR) across all bitmaps.

3. Any byte > 0 indicates that edge was covered by at least one trial.

4. Count the number of non-zero bytes → union edge coverage.

I believe the format for AFL++'s fuzz_bitmap files are a bit unusual: 0xff bytes indicate no coverage, while 0x00 indicates the edge was covered 1 time and other values indicate other "buckets" of edge-covered counts. So I think bitwise-AND is actually correct for union. But definitely double-check before implementing.

[Chand-ra]

Here are the results for the very first mutator configuration (full stack vs. full stack minus splice). The SpliceMutator is currently unmerged and the version at #135 is used.

The PR for this mutator already details one campaign, but that one was run with about a month older version of Smite IR, so it did not have a lot of paths to explore. This campaign (and the ones that will follow) are run on commit a9183ef (dated 29 July, 2026) of the master branch.

Note that ir-splice is the baseline configuration and is meant to represent "all available mutators minus splice" and ir-full-stack is meant to represent "all the available mutators" (op-param + input-swap + delete + reorder + gen-insert + splice) in the report that follows.

---

Fuzzing Evaluation Report

Configuration A (Baseline): ir-splice
Configuration B (Experimental): ir-full-stack

1. Summary Statistics

Target	Duration (h)	n (Baseline)	n (Exp.)	Median Cov. (Baseline)	Median Cov. (Exp.)	Adj. p-value (Cov.)	Â12 (Cov.)	Median AUC (Baseline)	Median AUC (Exp.)	Adj. p-value (AUC)	Â12 (AUC)	Union Cov. (Baseline)	Union Cov. (Exp.)	Execs/s (Baseline)	Execs/s (Exp.)
cln	24	20	20	8351	8367	1	0.48125	196078	196231	1	0.5125	15127	15841	10.22	6.035
eclair	24	20	20	11286.5	11342.5	1	0.555	267829	269387	1	0.565	12473	11856	5.075	3.805
ldk	24	20	20	27021.5	15690	0.0361686	0.2575	412019	370391	0.00936509	0.2175	35767	16460	26.64	9.28
lnd	24	20	20	26216	26236.5	0.135616	0.68625	628228	628096	1	0.435	26719	26399	27.72	11.025

A comprehensive version of this table including raw P-values and Interquartile Ranges (IQRs) is available in evaluation_metrics.csv.

2. Interpretation Guide

Use the generated matrix above to objectively evaluate the experimental configuration. For full methodology, see the Smite Fuzzing Evaluation Framework.

Key Metrics

• Adj. p-value: Mann-Whitney U test corrected for multiple targets via Holm-Bonferroni. Controls false-positive rate to ≤ 5% across all targets.
• Â12: Probability that a random B trial outperforms a random A trial. 0.5 = no difference; 0.7 = B wins 70% of pairings. Always read alongside the p-value.
• IQR: Spread of the middle 50% of trials. A much larger IQR in B suggests a few outlier runs may be inflating the median.
• AUC: Coverage speed — how much was discovered and how early. Useful when final coverage is similar between configurations.
• Union Coverage: Union of all trial bitmaps; the coverage ceiling for a multi-core deployment. Descriptive only, cannot be statistically tested.
• Execs/s: A large drop in B without a coverage gain means the new feature is too expensive.

Reading the Results

Adj. p	Â12	Conclusion
< 0.05	> 0.5	Meaningful improvement. Check IQRs are comparable, then merge.
< 0.05	~0.5	Significant but negligible. Check if worth the added complexity.
> 0.05	> 0.6	Promising but underpowered. Re-run with more trials (e.g., 50).
> 0.05	~0.5	No effect. Try an advanced snapshot or ground-truth evaluation.
any	< 0.5	B underperforms A. If significant, reject or redesign the feature.

Time-series caveat: If the IQR bands overlap for most of the campaign and only diverge near the end, treat the final-coverage result cautiously — late divergence may reflect noise rather than a sustained advantage.

3. Visualizations

Note: In the box plots below, the central box represents the Interquartile Range (IQR, the middle 50% of trials), demonstrating the consistency of the fuzzer's performance. The internal line represents the median.

Target: cln

Median Coverage Over Time

Distribution Comparisons

Final Edge Coverage	Area Under Curve (Speed)

---

Target: eclair

Median Coverage Over Time

Distribution Comparisons

Final Edge Coverage	Area Under Curve (Speed)

---

Target: ldk

Median Coverage Over Time

Distribution Comparisons

Final Edge Coverage	Area Under Curve (Speed)

---

Target: lnd

Median Coverage Over Time

Distribution Comparisons

Final Edge Coverage	Area Under Curve (Speed)

---

First thoughts

A very negative result for the SpliceMutator, definitely not mergeable as it currently stands. The only statistically significant result in the campaign is for LDK, which shows a very steep degradation with the experimental configuration, probably due to the extreme reduction in execs/sec.

For CLN and Eclair, the adjusted p-values of 1.0 indicate that the SpliceMutator had absolutely no statistically measurable effect on coverage, rendering it dead weight given the computational overhead.

It is also interesting to see that LND showed a tiny betterment in final edge coverage despite a similar dip in execs/sec to LDK. It has been quite an outlier across various other campaigns that I've run and I cannot quite reason why.