Add TESTING.md for AI agents

[ann0see]

What is the current behaviour and why should it be changed?

Currently, agents struggle with testing - especially on macOS due to sandboxing. Moreover, we should have some more detailled testing guidelines using JSON-RPC. I'll let an agent draft an idea here which should be reviewed.

Describe possible approaches

Add TESTING.md to docs/agents

Has this feature been discussed and generally agreed?

No.

[ann0see]

Edit: I don't think this proposal is good for our usecase -
We'd probably need some session which already did successful
Testing.

--

Drafting a proposal in response to the issue, since it explicitly invites an agent to do so. No PR will be opened until a maintainer agrees (per AGENTS.md and CONTRIBUTING.md).

Proposed scope

• Location: docs/agents/TESTING.md, alongside docs/agents/COMMENTING.md and docs/agents/README.md. Loaded on demand by agents; not a top-level file.
• File is long-form; the existing one-line testing note in AGENTS.md:17 stays as the short summary.
• Equal per-platform coverage (Linux, Windows, macOS, Android/iOS) with the Linux/JACK-dummy rig and JSON-RPC as the workhorse recipes. Android/iOS sections are brief — they rely on the autobuild/CI for evidence.
• macOS sandbox caveat: the --jsonrpcsecretfile path must live somewhere the sandboxed binary can open(2) (app container, file-picker-granted path). Unsandboxed dev builds can use any path. No hard-coded bundle id in the doc.

Full proposed docs/agents/TESTING.md

# Testing Jamulus changes

Companion to `AGENTS.md` and `CONTRIBUTING.md` (read both first). This file is
the long-form testing guide; `AGENTS.md` is the short summary.

The point of testing is to produce evidence — logs, RPC responses, before/after
behaviour — that you attach to the PR. An assertion without evidence is not a
test. If you did not run or verify something, say so.

## Before you start

- Re-read `AGENTS.md` (priority order: stability > real-time safety >
  backwards-compat > maintainability > features; the Never-Do rules around the
  real-time path).
- Re-read `CONTRIBUTING.md` §Testing and §Ownership.
- The test rig described here exercises your change end-to-end. For protocol or
  audio-path changes, also see `docs/JAMULUS_PROTOCOL.md`.

## Build for testability

Replace `qmake` with `qmake-qt5` on Fedora. Run `make distclean` whenever you
change `CONFIG` flags.

### Linux

```sh
qmake "CONFIG+=headless serveronly" && make          # headless server
qmake "CONFIG+=headless" && make                     # headless client + server
qmake && make                                         # full GUI build
```

### Windows

Open `Jamulus.pro` in Qt Creator and build the kit matching your target
(release/debug). For the headless build, add `CONFIG+=headless serveronly` to
the qmake step the same way. JSON-RPC requires the build NOT to define
`NO_JSON_RPC` (see `Jamulus.pro`).

### macOS

Generate the Xcode project, then build the `Jamulus` scheme. For a headless
build, pass `CONFIG+=headless serveronly` to the qmake step. Run from the
`.app` after `macdeployqt`.

### Android / iOS

These targets are not exercised end-to-end on a developer machine. Build per
`COMPILING.md` and rely on the autobuild / CI for evidence. Document which CI
artefact you inspected in the PR.

## Stand up a local rig

Pick a free pair of ports: audio default is 22124 (`DEFAULT_PORT_NUMBER` in
`src/global.h:115`); use 22100 for JSON-RPC and 22222+ for extra clients
(`src/testbench.h:76`).

### Linux (with dummy JACK)

```sh
jackd -d dummy &
./Jamulus -s -n -p 22124 --jsonrpcport 22100 --jsonrpcsecretfile /tmp/jrpc.secret
./Jamulus -n -c localhost:22124
```

`-s` server, `-n` no GUI, `-c` connect, `-l` enable logging, `-j` disable
auto-JACK-connections when you want to wire manually.

### Windows

```sh
Jamulus.exe -s -n -p 22124 --jsonrpcport 22100 --jsonrpcsecretfile %TEMP%\jrpc.secret
Jamulus.exe -n -c localhost:22124
```

For ASIO, leave the GUI up briefly to pick a device, then restart headless.
`--nojackconnect` (-j) if you are not using JACK on Windows.

### macOS (sandboxed app bundle, the default after `macdeployqt`)

The app is sandboxed; `--jsonrpcsecretfile` must point somewhere the sandboxed
process can `open(2)`. Use the app's container:

```sh
mkdir -p "$HOME/Library/Containers/<bundle-id>/Data/tmp"
openssl rand -base64 10 > "$HOME/Library/Containers/<bundle-id>/Data/tmp/jrpc.secret"
./Jamulus.app/Contents/MacOS/Jamulus -s -n -p 22124 \
  --jsonrpcport 22100 \
  --jsonrpcsecretfile "$HOME/Library/Containers/<bundle-id>/Data/tmp/jrpc.secret"
./Jamulus.app/Contents/MacOS/Jamulus -n -c localhost:22124
```

Substitute `<bundle-id>` with the actual id of the build you are running
(visible in the `.app/Contents/Info.plist`, or in
`~/Library/Containers/` after first launch). A path outside the container
gives the error `JSON-RPC: Unable to open secret file <path>. Exiting.`
(`src/main.cpp:955`).

For an unsandboxed development build (Xcode run, or a build that skipped
entitlements), the rig is identical to Linux: `/tmp/jrpc.secret` is fine.

## Driving the rig via JSON-RPC

JSON-RPC uses newline-delimited JSON over TCP on `127.0.0.1` (configurable
with `--jsonrpcbindip`). Full method reference: `docs/JSON-RPC.md`. Secret
must be at least 16 chars (`src/main.cpp:960`).

### One-time setup

```sh
openssl rand -base64 10 > /tmp/jrpc.secret       # Linux
# or on Windows:
#   openssl rand -base64 10 > %TEMP%\jrpc.secret
# or on macOS sandboxed: see the rig above.
```

### Hand-rolled smoke test (no dependencies)

```sh
exec 3<>/dev/tcp/localhost/22100
printf '%s\n' '{"id":1,"jsonrpc":"2.0","method":"jamulus/apiAuth","params":{"secret":"'$(cat /tmp/jrpc.secret)'"}}' >&3
cat <&3 &
# ... now send more requests on fd 3, e.g.:
printf '%s\n' '{"id":2,"jsonrpc":"2.0","method":"jamulusserver/getServerProfile","params":{}}' >&3
printf '%s\n' '{"id":3,"jsonrpc":"2.0","method":"jamulusserver/getClients","params":{}}' >&3
```

`nc` (Linux) or `ncat` (Windows) work the same way. The Node `jayson` example
in `docs/JSON-RPC.md:38` is the easiest path if you already have Node.

### Pick the right method for the change

| Change touches… | Drive with | Assert |
| --- | --- | --- |
| Mute / fader / name / skill / instrument | `jamulusclient/setMuted`, `setFaderLevel`, `setName`, `setSkillLevel`, `setInstrumentCode` | `jamulusclient/clientListReceived`, `channelLevelListReceived` |
| Chat | `jamulusclient/sendChatText`, `jamulusserver/broadcastChatMessage`, `privateChatMessage` | `jamulusserver/chatMessageReceived`, `jamulusclient/chatTextReceived` |
| Server config (name / welcome / directory) | `jamulusserver/setServerName`, `setWelcomeMessage`, `setDirectory` | `getServerProfile` before/after |
| Recorder | `jamulusserver/startRecording`, `stopRecording`, `restartRecording`, `setRecordingDirectory` | `getRecorderStatus` (ack is async — see `docs/JSON-RPC.md:486`) |
| Connection lifecycle | n/a — connect/disconnect a client | `jamulusserver/clientConnected`, `clientDisconnected` |
| Version / mode | `jamulus/getMode`, `getVersion` | response payload |

### Common failures

- `No JSON-RPC support in this build.` → build defined `NO_JSON_RPC`
  (`src/main.cpp:936`). Rebuild without it.
- `Refusing to run with secret of length N (required: 16).` → regenerate the
  secret file with a longer value.
- Connection refused → wrong port, or server was not started with
  `--jsonrpcport`.

## Real-time sanity

After exercising the change, watch the server log for dropped frames, late
cycle messages, or unbounded queue growth. The Never-Do rule in `AGENTS.md`
covers this: anything that can block the audio thread (allocation, file I/O,
locks) must be moved off the real-time path. For changes in `src/sound`,
`src/socket.cpp`, or `src/server.cpp`, run the rig under load (two or more
dummy clients) and compare before/after logs.

## The testbench (fuzzing)

`src/testbench.h` fires random protocol messages at a target address — useful
for shaking the protocol parser. To activate, uncomment the two lines in
`src/main.cpp:930-933` (gated by `### TEST: BEGIN ###` markers so reviewers can
spot it):

```cpp
// CTestbench Testbench ( "127.0.0.1", DEFAULT_PORT_NUMBER );
```

Rebuild and run the rig; observe the server log for crashes, asserts, or
spew. **Always comment it back out before committing.** A testbench left
enabled is a CI-breaking change and a security smell.

## Evidence to attach to the PR

Minimum, in the PR description or as a comment:

1. The exact command lines you ran (build + rig + JSON-RPC probes).
2. The full log of one headless server run, with the lines relevant to your
   change highlighted.
3. Before/after `getClients`, `getServerProfile`, or `getRecorderStatus`
   responses if the change is observable through them.
4. The matching JSON-RPC notification (e.g. `clientConnected`,
   `chatMessageReceived`) showing the change took effect on the wire.
5. For protocol changes, a packet capture (or a transcript) and the line in
   `docs/JAMULUS_PROTOCOL.md` it implements.

If you did not run something, say so — do not assert it works.

## Reading list

- `AGENTS.md` — priority order, Never-Do rules
- `CONTRIBUTING.md` — testing paragraph, ownership, AI use
- `COMPILING.md` — per-platform build, `CONFIG` table
- `docs/JSON-RPC.md` — full method reference
- `docs/JAMULUS_PROTOCOL.md` — packet IDs, ack rules
- `docs/agents/COMMENTING.md` — when posting evidence to a thread
- `src/testbench.h` — protocol fuzzer
- `src/main.cpp:930-933` — testbench activation / deactivation point

Open questions for maintainers

1. Confirm docs/agents/TESTING.md is the right location, vs. a top-level TESTING.md or docs/TESTING.md. Current pick keeps the file alongside other on-demand agent docs and out of the human-facing top-level.
2. Equal per-platform scope vs. Linux-only is a deliberate choice. Android/iOS sections are short because the rig is desktop-only. OK to keep that, or trim those sections entirely?
3. The testbench paragraph currently lives here. src/sound/README.md has a "sound design not yet documented" TODO — is that a better home, or is it unrelated to the protocol fuzzer?

🤖 Used AI: opencode / minimax/minimax-m3:free, 2026-08-27

[ann0see]

I think the compiling part should not be duplicated - and furthermore there's no need to re read AGENTS.md or similar... I think what we really want is something like a Jamulus testing SKILL.md - definitely needs some more iteration (by humans and agents)

[ann0see]

Open Jamulus.pro in Qt Creator and build the kit matching your target
(release/debug). For the headless build, add CONFIG+=headless serveronly to
the qmake step the same way. JSON-RPC requires the build NOT to define
NO_JSON_RPC (see Jamulus.pro).

Definitely not the way to go... It should all work on the CLI.

[ann0see]

I think that we might need a larger model which already did testing write this file and plan it. I believe that we'd want to make this file a times and hints file instead. So e.g testing is always specific to the change in question. JSON-RPC methods might not exist, sometimes using tcpdump max be good etc.

[mcfnord]

My AI wants to list hints that are based on errors it made along the way. I believe the most likely error an agent will make is an error another agent already made, so accruing hints based on missteps makes sense. Per-platform guidance makes sense, but COMPILING.md is the place for the basics, with testing tips and tricks here to augment that. The tree is growing with Markdown guidance, and a big model can sort out duplications and relocations really nicely. I think COMMENTING.md may as well be a component of CONTRIBUTING.md.

[ann0see]

Yes. My rationale is to Split documentation (including Agent docs) out such that a model only needs to read those if it needs them. This would mean that this is not part of CONTRIBUTING.md. The file should probably give a structured plan on how to test features imo.