Conformance Testing

This guide details how to test your changes to Agave for conformance with the Solana protocol.

This guide is intended for all contributors making changes to Agave who wish to ensure their changes:

• Do not break consensus
• Do not introduce a regression
• Produce the expected results for the components changed, beyond any included unit tests

Conformance testing is designed to leverage artifacts - such as fixtures and ledgers - to ensure the end-to-end protocol implementation for a given component still produces the expected result for a given set of inputs. The specific details of how each form of conformance testing works are provided below, as well as the necessary instructions to run them yourself.

Thank you to the Firedancer team for spearheading this effort and enabling it for Agave contributions.

Fixture-Based Testing

First identify the entrypoints you wish to invoke with test vectors (fixtures). This will depend on which Agave components you've modified, and can include more than one.

The table below details each vector entrypoint and its associated coverage.

Entrypoint	Responsibility	Primary Crates Exercised
sol_compat_gossip_decode_v1	Deserialize a binary gossip wire message and decode it into structured effects; validates CRDS/gossip protocol parsing.	solana-gossip
sol_compat_shred_parse_v1	Ingest serialized shreds through the blockstore ingest/recovery pipeline and replay validation; exercises shred parsing and blockstore-owned handling.	solana-ledger
sol_compat_block_execute_v1	Reconstruct block state, commit the block's transactions through the Bank, track cost, and return bank and leader-schedule effects.	solana-runtime, solana-accounts-db
sol_compat_txn_cost_v1	Compute cost-model accounting for a transaction (signature, write-lock, data-bytes, execution, and loaded-accounts-data-size costs).	solana-cost-model, solana-runtime-transaction
sol_compat_txn_execute_v1	Execute a full transaction through the real Bank load-and-execute path (fees, nonce, rent, account loading, program execution). Runtime-level transaction processing.	solana-runtime, solana-svm, solana-accounts-db
sol_compat_svm_txn_execute_v1	Execute a sanitized transaction message (all instructions) directly through the SVM invoke context — no bank fee/nonce/loader machinery. SVM-level message processing.	solana-svm, solana-program-runtime
sol_compat_instr_execute_v1	Execute a single instruction against the SVM via InvokeContext::process_message, resolving the builtin/BPF program from the program cache.	solana-program-runtime, solana-bpf-loader-program, solana-system-program, agave-precompiles
sol_compat_vm_serialize_execute_v1	Serialize an instruction's program input parameters into VM memory; reports the serialized-memory hash, input memory regions, and per-account metadata.	solana-program-runtime
sol_compat_vm_syscall_execute_v1	Set up an sBPF VM and invoke a single syscall, checking memory mapping, CU consumption, and post-syscall effects.	solana-syscalls, solana-program-runtime, solana-sbpf
sol_compat_elf_loader_v1	Load and verify an sBPF program ELF under a program-runtime environment; reports rodata hash, entry PC, text section layout, and call destinations.	solana-sbpf, solana-syscalls

The rest of this section walks through running fixtures for one entrypoint, using the ELF loader (sol_compat_elf_loader_v1) as the worked example. The same steps apply to every entrypoint — only the category name changes.

Getting the Test Vectors

Fixtures live in the test-vectors repository, grouped by entrypoint into per-category directories (elf_loader/, instr/, txn/, ...), each with a fixtures/ directory of .fix files. A .fix is a Protobuf-encoded <Category>Fixture holding the recorded input and expected output for a single entrypoint invocation.

Until fixtures are published as release tarballs, clone the repository and point a runner directly at the relevant <category>/fixtures/ directory:

git clone https://github.com/firedancer-io/test-vectors.git

Running the Vectors

Each entrypoint has a runner binary in the solana-svm-conformance crate (dev-bins workspace), named test_exec_<category> — here, test_exec_elf_loader. It decodes each .fix, runs it through Agave, and compares the result against the expected output.

Build the desired runner:

cargo build --manifest-path dev-bins/Cargo.toml -p solana-svm-conformance

Run it against its fixtures:

dev-bins/target/debug/test_exec_elf_loader -- test-vectors/elf_loader/fixtures/*.fix

Each fixture prints OK: <file> or FAIL: <file> (with the expected and actual effects on failure); the process exits with the number of failures, so 0 means every vector passed. For the tally alone:

dev-bins/target/debug/test_exec_elf_loader -- test-vectors/elf_loader/fixtures/*.fix \
  | grep -c '^OK:'

Debugging Mismatches with solana-conformance

solana-conformance is a CLI for running fixtures against one or more clients and diffing the results. It is the main tool for debugging conformance mismatches.

Setup instructions live in the solfuzz wiki. Note: the solfuzz repository is currently private, so the link requires access.

Building the Agave Target

solana-conformance does not run Agave code directly. It loads a shared object (.so) that exports the sol_compat_* entrypoints.

Build it from this repository:

cargo build --manifest-path dev-bins/Cargo.toml -p solana-svm-conformance --lib

This produces dev-bins/target/debug/libsolana_svm_conformance.so.

Usage Examples

Set a variable to keep the commands short:

AGAVE_SO=dev-bins/target/debug/libsolana_svm_conformance.so

Check fixtures against their expected output. Each .fix file embeds the expected effects. run-fixtures executes the input and diffs the result against them:

solana-conformance run-fixtures -i test-vectors/instr/fixtures/system -t $AGAVE_SO -o results

A summary line reports the counts, e.g. Results: 7400 passed, 0 failed, 0 skipped.

Inspect a fixture. Decode it into readable text:

solana-conformance decode-protobufs -i <fixture or dir> -o decoded

Execute a fixture and print its effects. Best for one fixture or a small directory. This is also how you see log output (e.g. eprintln!) from the target:

solana-conformance execute -i <fixture or dir> -t $AGAVE_SO

Pass -n to hide the effects and only show logs.

Diff two clients. run-tests runs each fixture through a reference target (-s) and one or more test targets (-t), then diffs the outputs:

solana-conformance run-tests -i <fixture or dir> \
  -s $AGAVE_SO \
  -t <firedancer>/build/native/clang/lib/libfd_exec_sol_compat.so \
  -o results

Each target's effects are written under results/, one text file per target:

diff results/libsolana_svm_conformance.so.txt results/libfd_exec_sol_compat.so.txt

Debugging a Failing Fixture

The general loop:

1. Run run-fixtures (or run-tests) to find failing fixtures.
2. Re-run on a single failing .fix — -i accepts one file.
3. Run execute against your target to see its effects.
4. Add debug prints (e.g. eprintln!) to the Agave code in question.
5. Rebuild the .so and re-run execute.
6. Repeat until the divergence has been identified.

Tips:

• run-tests hides target stderr. Use execute to see your log output.
• -l 2 raises the target log level. Firedancer's FD_LOG_NOTICE / FD_LOG_WARNING prints only show at -l 2 or lower. The default (-l 5) hides them.
• -d runs single-threaded, which is easier to follow while debugging.