Testing

cmake -S . -B build
cmake --build build
cd build && ctest -j8

There are five groups of tests (175 without reference programs, about 240 with all reference programs installed):

Group	Prefix	What it tests
Unit tests	unit-	The library in memory: encoder → decoder loopback of all modems and modes, sample rates, input formats, chunk sizes, noise, different clocks, frequency offsets, echoes, amplitude, many transmissions in a row, the codecs (Baudot, Caller-ID, UIC, Varicode, MFSK Varicode, AX.25 frames, KISS, FX.25, IL2P, the Olivia FEC, the QPSK convolutional code, the CSS frames, the Morse code tables and timing, Reed-Solomon), the packets, the streams (also with a receive task), and the AudioTools pipeline. Source: tests/unit/unit-tests.cpp
CLI tests	cli-	tinymodem transmits a WAV file and decodes it again. For FSK, this is the original minimodem test suite ported to tests/cli-test.sh; the other tests cover MFSK, PSK (also QPSK and 8PSK), OFDM, AX.25 (also FX.25 and IL2P), ggwave, Olivia, Contestia, CSS, Morse code, DTMF, KISS and packets.
minimodem interop tests	interop-tx-, interop-rx-	The FSK CLI tests with the original minimodem on one side: interop-tx means we transmit and the original receives, interop-rx the reverse.
direwolf interop tests	interop-ax25-	AX.25 with direwolf: direwolf decodes our frames (tx), we decode direwolf's frames (rx), and we decode direwolf's 100 frames with increasing noise (noise, at least 60 must be decoded). The same in both directions with FX.25 (fx25-tx, fx25-rx) and IL2P (il2p-tx, il2p-rx): the tests check that direwolf reports the frames as FX.25 or IL2P, and that we decoded them as such. Source: tests/ax25-interop.sh
ggwave interop tests	interop-ggwave-	All 9 protocols with ggwave's tools: ggwave-from-file decodes our messages (tx), we decode the messages of ggwave-to-file (rx). Source: tests/ggwave-interop.sh

MFSK, PSK, OFDM, Olivia, CSS, Morse code, DTMF and the packets are tested with TinyModem on both sides: there is no reference program for them in the tests. The PSK Varicode tables, the QPSK and 8PSK constellations and the Olivia/Contestia waveform and FEC follow fldigi's source code, but the interoperability with fldigi itself is not tested.

Run a single test with ctest -R unit-noise --output-on-failure, or call the programs directly:

./build/tests/unit-tests mfsk-robust
bash tests/cli-test.sh amplitude ./build/tinymodem /usr/bin/minimodem

To check the stream classes with a receive task for data races, build the unit tests with ThreadSanitizer. On newer Linux kernels, setarch -R is needed to disable address randomization:

g++ -std=c++17 -g -O1 -fsanitize=thread -DIS_MIN_DESKTOP -DNO_MAIN -Wno-format \
  -DTEST_DATA_DIR=\"$PWD/tests\" -I../arduino-audio-tools/src -Isrc \
  tests/unit/unit-tests.cpp -o unit-tsan -lpthread
setarch -R ./unit-tsan stream-rtos
setarch -R ./unit-tsan mfsk-stream

Interoperability tests

The minimodem interop tests run only if the original minimodem is found in the PATH (sudo apt install minimodem), or if it is given explicitly:

cmake -S . -B build -DMINIMODEM_REFERENCE=/path/to/minimodem

The original passes the verify-perfect test with confidence=inf. Our float Goertzel filters leave a tiny amount of noise, so with our receiver that test checks only for (rate perfect).

The direwolf interop tests run if atest and gen_packets are found (sudo apt install direwolf). To use an unpacked direwolf package, define the programs and the location of its shared libraries:

cmake -S . -B build -DDIREWOLF_ATEST=/path/atest -DDIREWOLF_GEN_PACKETS=/path/gen_packets \
  -DDIREWOLF_LIBRARY_PATH=/path/lib

The ggwave interop tests run if ggwave-to-file and ggwave-from-file are found. They are examples of ggwave and are not packaged, so build them yourself. They don't need SDL2 or the git submodules, so they can be compiled directly:

git clone https://github.com/ggerganov/ggwave.git
cd ggwave
for t in ggwave-to-file ggwave-from-file; do
  g++ -O2 -std=c++17 -Iinclude -Iexamples examples/$t/main.cpp examples/ggwave-common.cpp \
    src/ggwave.cpp -o ~/.local/bin/$t -ldl -lpthread
done

Or with cmake: git clone --recursive and cmake -DGGWAVE_SUPPORT_SDL2=OFF. Use -DGGWAVE_TO_FILE=/path -DGGWAVE_FROM_FILE=/path if they are not in the PATH.

The CI installs both minimodem and direwolf.

Arduino

The examples are compiled in CI with arduino-cli for ESP32 and RP2040. The modem-stream-rtos example is compiled for the RP2040 with FreeRTOS (rp2040:rp2040:rpipico:os=freertos). To check them locally (the --library options aren't needed if both libraries are installed in your Arduino libraries folder):

arduino-cli compile -b esp32:esp32:esp32 --library . --library ../arduino-audio-tools examples/loopback

On a board, the benchmark example shows whether the modems run fast enough. The results on the ESP32-S3 and ESP32-P4 are in Performance.

The modems were also tested over the air on a Waveshare ESP32-P4-WIFI6: the board sends each mode through its speaker and receives it with its onboard microphone. DTMF, PSK31, QPSK63, 8PSK125, AX.25, FX.25, IL2P, Olivia, Contestia, CSS and MFSK were received without errors, but the small speaker needs moderate levels (0.4 - 0.6 of full scale) and attenuates frequencies below about 1.2 kHz. The results depend on the placement of the board.