Conformance Tools

This guide details how to use Firedancer's conformance tooling on Agave changes.

This guide is intended for all contributors making changes to Agave who wish to ensure their changes:

• Do not break consensus
• Do not introduce a regression
• Produce the expected results for the components changed, beyond any included unit tests

Conformance testing is designed to leverage artifacts - such as fixtures and ledgers - to ensure the end-to-end protocol implementation for a given component still produces the expected result for a given set of inputs. The specific details of how each form of conformance testing works are provided below, as well as the necessary instructions to run them yourself.

Thank you to the Firedancer team for spearheading this effort and enabling it for Agave contributions.

Entrypoints

Entrypoint	Responsibility	Primary Crates Exercised
sol_compat_gossip_decode_v1	Deserialize a binary gossip wire message and decode it into structured effects; validates CRDS/gossip protocol parsing.	solana-gossip
sol_compat_shred_parse_v1	Ingest serialized shreds through the blockstore ingest/recovery pipeline and replay validation; exercises shred parsing and blockstore-owned handling.	solana-ledger
sol_compat_block_execute_v1	Reconstruct block state, commit the block's transactions through the Bank, track cost, and return bank and leader-schedule effects.	solana-runtime, solana-accounts-db
sol_compat_txn_cost_v1	Compute cost-model accounting for a transaction (signature, write-lock, data-bytes, execution, and loaded-accounts-data-size costs).	solana-cost-model, solana-runtime-transaction
sol_compat_txn_execute_v1	Execute a full transaction through the real Bank load-and-execute path (fees, nonce, rent, account loading, program execution). Runtime-level transaction processing.	solana-runtime, solana-svm, solana-accounts-db
sol_compat_svm_txn_execute_v1	Execute a sanitized transaction message (all instructions) directly through the SVM invoke context — no bank fee/nonce/loader machinery. SVM-level message processing.	solana-svm, solana-program-runtime
sol_compat_instr_execute_v1	Execute a single instruction against the SVM via InvokeContext::process_message, resolving the builtin/BPF program from the program cache.	solana-program-runtime, solana-bpf-loader-program, solana-system-program, agave-precompiles
sol_compat_vm_serialize_execute_v1	Serialize an instruction's program input parameters into VM memory; reports the serialized-memory hash, input memory regions, and per-account metadata.	solana-program-runtime
sol_compat_vm_syscall_execute_v1	Set up an sBPF VM and invoke a single syscall, checking memory mapping, CU consumption, and post-syscall effects.	solana-syscalls, solana-program-runtime, solana-sbpf
sol_compat_elf_loader_v1	Load and verify an sBPF program ELF under a program-runtime environment; reports rodata hash, entry PC, text section layout, and call destinations.	solana-sbpf, solana-syscalls

Vectors

Fixtures live in the test-vectors repository, grouped by entrypoint into per-category directories (elf_loader/, instr/, txn/, ...), each with a fixtures/ directory of .fix files. A .fix is a Protobuf-encoded <Category>Fixture holding the recorded input and expected output for a single entrypoint invocation.

Fixtures are published as release tarballs according to each commit SHA of the test-vectors repository.

You can see one such release here: 0074649d

Conformance CLI

solana-conformance is a CLI for running fixtures against one or more clients and diffing the results. It is the main tool for debugging conformance mismatches.

Setup instructions live in the solfuzz wiki. Note: the solfuzz repository is currently private, so the link requires access.

Building the Agave Target

solana-conformance does not run Agave code directly. It loads a shared object (.so) that exports the sol_compat_* entrypoints.

Each conformance crate builds its own .so, covering different fixture sets:

Package	Library	Fixture sets
agave-conformance	libagave_conformance.so	block, cost, gossip, shred, txn
solana-svm-conformance	libsolana_svm_conformance.so	elf_loader, instr, syscall, vm_serialization

Build one from this repository:

./cargo stable build --release --manifest-path dev-bins/Cargo.toml -p <package> --lib

This produces dev-bins/target/release/<library>.

Claude Skills

claude-skills is a collection of Claude Code skills for Firedancer's conformance-fuzzing workflow, such as debug-fuzz-mismatch and regenerate-test-vectors. See its README for installation. Note: the repository is currently private, so the link requires access.

Workflows

Inspecting a Failed CI Run

If a conformance CI job has failed, you can inspect the regression.

Download the job's fixtures and run them exactly as CI does:

ci/download-test-vectors.sh txn
ci/test-conformance.sh agave-conformance test_exec_txn fd-vectors/txn

Debugging a Mismatch

If ci/test-conformance.sh is showing failures, you may need to debug them one at a time. Pick a failing fixture from the FAIL: lines in the output.

Simply placing eprintln! throughout the code, rebuilding the target, and running the fixture standalone works just fine:

ci/test-conformance.sh agave-conformance test_exec_txn fd-vectors/txn/fixtures/<hash>.fix

However, solana-conformance offers some more powerful debugging tools.

To see the transaction, accounts, and features a fixture feeds in, decode it:

solana-conformance decode-protobufs -i <fixture> -o decoded

To see only your own prints, build the Agave target and execute the fixture with -n, which hides the effects:

solana-conformance execute -n -i <fixture> -t dev-bins/target/release/libagave_conformance.so

To step through in a debugger, build the runner with debug info and run it under gdb:

./cargo stable build --profile release-with-debug --manifest-path dev-bins/Cargo.toml -p agave-conformance --bin test_exec_txn
gdb --args dev-bins/target/release-with-debug/test_exec_txn <fixture>

To check whether your change caused a failure, compare against a master build (e.g. from a separate worktree). Give the two .so files different names, since results are written per file name:

cp <master-worktree>/dev-bins/target/release/libagave_conformance.so master.so
cp dev-bins/target/release/libagave_conformance.so pr.so
solana-conformance run-tests -i <fixture or dir> -s master.so -t pr.so -o results
git diff --no-index results/master.so.txt results/pr.so.txt

Pass Firedancer's libfd_exec_sol_compat.so as -t instead to check whether Firedancer agrees with Agave.

Regenerating Fixtures

If a failure is expected, such as an error code change or something unreachable, you can regenerate fixtures.

Warning: Regenerating fixtures should be rare. Fixtures are the reference for correct behavior, so regenerating them rewrites what "correct" means. Only do it once you've confirmed the breakage is intentional using the inspection and debugging workflows above.

Build the Agave target from your branch, pick a failing fixture from the FAIL: lines, and regenerate it with solana-conformance. create-fixtures reruns the fixture's input against the target and records the new output:

solana-conformance create-fixtures -i fd-vectors/txn/fixtures/<hash>.fix \
  -t dev-bins/target/release/libagave_conformance.so -o regenerated

create-fixtures deletes its output directory before writing, so never point -o at a directory you want to keep. To regenerate several fixtures at once, copy them into one directory and pass that to -i instead.

Check that the regenerated fixture passes, and that only the fields you expect changed:

ci/test-conformance.sh agave-conformance test_exec_txn regenerated/<hash>.fix
solana-conformance decode-protobufs -i fd-vectors/txn/fixtures/<hash>.fix -o decoded-old
solana-conformance decode-protobufs -i regenerated/<hash>.fix -o decoded-new
git diff --no-index decoded-old decoded-new

Finally, copy the regenerated fixtures over the originals in a test-vectors checkout (e.g. cp regenerated/<hash>.fix <test-vectors>/txn/fixtures/) and open a PR there. Once it merges, test-vectors publishes a release tagged with the first 8 characters of the merge commit. Point your Agave PR at it by updating TEST_VECTORS_COMMIT and TEST_VECTORS_SHA256SUMS in ci/download-test-vectors.sh, where the latter is the output of curl -fsSL <release-url>/SHA256SUMS | sha256sum.

Generating Fixtures for New Features

You're adding a new feature or behavior, and test-vectors has no fixtures for it yet. Start by deciding which cases the fixtures should cover: the success path, each failure path, and edge cases such as boundary values. Each case becomes a fixture input.

Generating new fixtures is still a work in progress. The approaches below work, but none of them is smooth yet.

kunorpus generators. The primary approach is kunorpus, the corpus generator in solfuzz. Modify or add a generator so it produces inputs for your cases, then run them through the pipeline that turns them into fixtures with Agave's output. The pipeline has several steps; the generate-new-test-vectors skill in claude-skills walks through it.

Rust tests. Alternatively, write the cases as Rust tests in the conformance crate, and have them write each input and its effects out as a .fix file. conformance/tests/gossip/fixtures.rs does this for gossip: an #[ignore]d test that writes fixtures to FIXTURE_DIR when run manually. This duplicates each case between the tests and the fixtures, which we intend to remove later.

Fuzzing. You can also run the solfuzz fuzzer against your change until it builds up a corpus, then derive new fixtures from that corpus. If the fuzzer finds a crash or mismatch along the way, it saves the input that caused it, which can become a fixture too. The run-fuzzer skill in claude-skills covers running it.

Whichever approach you use, add the new fixtures to test-vectors as described in Regenerating Fixtures.