TurboJam

[sierkov]

Hi @davxy,

I've prepared a Docker image with the fuzzing target of TurboJam, an open-source C++ implementation of JAM. The image is publicly available on DockerHub as r2rationality/turbojam-fuzz:20250821-000. It targets GP version 0.6.7.

Below is a quick usage guide along with a few open questions. Please let me know if this packaging approach works for you.

Usage

The image expects an externally mounted volume at /fuzzer, where it will create:

• fuzzer.sock - the UNIX socket for listening for incoming connections
• fuzzer.log - a diagnostic log file

Example command (creates both in ./jam-fuzz on the host):

docker run --rm -it -v ./jam-fuzz:/fuzzer r2rationality/turbojam-fuzz:20250821-000

Open Questions

1. Behavior of GetState with an unknown header hash Can you confirm that closing the connection is the expected behavior in the following scenario?
   • The client submits a block that is rejected.
   • The client then requests the posterior state of that block by header hash.
   • The target immediately closes the connection, per: “Receiving an unexpected or malformed message results in immediate session termination.”
2. PVM traces for three failing tests The current implementation passes all conformance tests except three, which show minor divergences in gas consumption. Could you provide PVM traces for the following cases?
   • preimages/00000070
   • preimages/00000091
   • preimages/00000092.
3. Including PVM traces in fuzz reports Would it be feasible to automatically include PVM traces in fuzzing reports? This would speed up debugging of subtle issues (e.g., identifying misbehaving host calls).

[davxy]

Hi @sierkov .

First, some issues to be fixed

• I can't run the container as the current machine user (see logs below). It only works with superuser.

  • For reference, boka provides a Docker image that runs as a normal user. Please verify this (you can test with ./get_target boka and ./run_target boka). I’ve already added your target to the scripts so you can check yours as well.

• Please add an argument to specify the socket name. I use the same socket name for all targets. You can refer to boka for an example.

• Please include a latest tag in your Docker image, so I don’t need to update the download script with every release. Again, see boka as reference.

❯ ./run_target.sh turbojam --help
Run turbojam via Docker
/bin/bash: /home/dev/turbojam/start-fuzzer.sh: Permission denied

[davxy]

Also, @sierkov, are you on any of the JAM Matrix channels?
I recommend joining the new "JAM Conformance" room: #jam-conformance:matrix.org.
People have been asking questions about your implementation there.

[sierkov]

@davxy thank you. I've published an updated image: r2rationality/turbojam-fuzz:latest.
This version supports alternative user ids and custom socket paths. Also, the image size has been reduced.

You can find suggested adjustments to get_target.sh and run_target.sh in #27

Let me know if you have any further feedback.

P.S. I'll check messages in the "JAM Conformance" channel later today.

[davxy]

Now your target starts and waits for connection.
Unfortunately it disconnects as soon as I send the PeerInfo message :-/ (I don't receive anything back)

[sierkov]

@davxy Thanks! I've published an updated image. It fixes an issue where the API was incorrectly expecting a peer_info message with jam_version 0.6.6. I had forgotten to bump the version number, and since my test client had the same mismatch, the problem went unnoticed during testing.

BTW, I’d like to expose the API’s log file to make it easier to analyze communication-level issues. Do you have any suggestions on the best way to approach this?

[davxy]

Answers to the questions above

Behavior of GetState with an unknown header hash

If you fail to import, you should return the last correctly imported state root.
If the fuzzer expects this failure then the next block is sent, otherwise it will ask for the state at the block just imported.

• if you want a report with a diff then you return the state at the best block you have.
• if you don't care of the diff you can close the connection

PVM traces for three failing tests

you talk aboutnthe test vectors right? I will provide you the traces ( do you still need these?)

Including PVM traces in fuzz reports

Not currently, perhaps you can also try ask for traces to other teams as well in the jam conformance chan :-). For us generating these traces is not sonething that we have scripted/automated and currently involves some manual intervention

[davxy]

Your reports are published in your dedicated folder. Table upsated as well: https://github.com/davxy/jam-conformance/tree/main/fuzz-reports

[sierkov]

@davxy Thank you! Yes, three PVM traces for traces test-vectors are still needed:

• preimages/00000070
• preimages/00000091
• preimages/00000092

On the idea of requesting traces from other teams: to be honest, I’m a bit reluctant. My main concern is keeping the implementation as clean as possible and avoiding any potential claims of collusion.

I was thinking there might be a very simple way to generate traces instead, for example:

• Forking a new process with an adjusted environment that forces the PVM to produce its trace in the location of a new report.
• Executing the PVM program inside that process.

I would have gladly contributed an initial implementation if the relevant parts of the Fuzzer’s source code were public. In my view, automatic availability of PVM traces could noticeably accelerate progress for many teams.

[davxy]

main concern is keeping the implementation as clean as possible and avoiding any potential claims of collusion

I don't understand how asking for traces can lead to this?

You should also consider that PolkaJam team is not different from any other team and we should not be treated as some kind of reference

BTW I'll try to provide the traces you need tomorrow (afk for the next 3 days)

[sierkov]

@davxy Thanks again. I’ve prepared an updated 0.6.7 image that resolves most reported issues, with the exception of two:

1. Occasional mismatch in accumulate_gas_used. I’ve added some notes on this below.
2. Code hash mismatch for services “upgraded” via the upgrade host calls This occurs because the argument memory buffer has four non-zero leading bytes. My hunch is that this may be a side effect of execution taking a non-standard path, which also seems consistent with the observed differences in accumulate_gas_used.

Before diving deeper into the gas issue, I’d like your input on one question: 👉 Would you prefer that turbojam first cleans up all 0.6.7 reports before moving to 0.7.0, or should it switch sooner to reduce the overhead of testing multiple JAM versions in parallel?

Another idea I’d like to float (before we talk about PVM traces): would it be feasible to modify the bootstrap service to log gas usage after each host call? My suspicion is that divergences in gas usage primarily come from host-call results. If I could pinpoint which host call misbehaves, it might already provide substantial debugging value.

Regarding PVM traces, it’s not my intention to overcomplicate matters by asking only for Parity’s PVM traces. That said, with a paranoid hat on, I worry that analyzing artifacts (such as logs) from other implementations outside well-defined protocols (traces, fuzzer, jamsnp) could inadvertently reveal their approaches and be perceived as collusion. My reluctance is based on two main reasons:

1. Parity’s implementation is the only one where the team is not competing for the JAM prize, and thus benefits from other implementations being developed “clean room.” In contrast, prize-eligible implementations are incentivized to move fast, and may unintentionally expose details in development artifacts such as logs.
2. Since what counts as collusion is subjective and will be decided later by Parity and the Polkadot Fellows, I’d personally prefer to stay on the safe side and avoid using artifacts from other implementations until explicit guidance says it’s acceptable. Creating a JAM implementation is a significant investment, so I’d like to minimize potential risks.

[davxy]

For the 0.6.7 we're not going to test new traces.
All the traces are the ones contained here. New stuff will be proposed for 0.7.0

[davxy]

Curious. Why TurboJam is not in clients list yet? Is it under a different name?