Conformance Testing

Conformance Testing

veetee is tested at four levels: unit tests for each control function, scripted runs of vttest compared with golden screens, esctest2 with a list of explained differences, and recordings of real OpenVMS sessions replayed through the emulator. All of them run in CI on every push, and the unit tests also run on Windows.

cargo test --workspace         # unit and integration tests
cargo xtask vttest             # vttest suite (needs curl, a C compiler and make)
cargo xtask esctest            # esctest2 (needs git and python3)
cargo xtask openvms            # replay the OpenVMS recordings

Unit tests

crates/vt-core/tests/ holds tests grouped by terminal generation (terminal.rs, vt220_vt320.rs, vt420.rs, vt520.rs). Each feeds host output to a Terminal and checks the screen, cursor, modes or replies. Test comments cite the DEC manual section they follow.

vttest

cargo xtask vttest downloads a pinned vttest release, verifies its SHA-256 checksum, builds it under target/conformance/, and runs every script in tests/conformance/vttest/MODEL/.

Scripts drive vttest's menus through vt-headless, which runs the program on a pseudo-terminal against the emulator and compares screens with snapshots/*.screen:

cargo xtask vttest vt420/menu11-3            # only scripts whose name contains this
cargo xtask vttest --bless vt420/menu11-3    # re-record after checking the change is right

Coverage by menu is in the compatibility matrix. Only bless a snapshot after checking the new screen against the manual or against what vttest says should appear.

Screen snapshots

page 24x80 cursor 3,17 visible
modes decawm decom
 1  |Test of autowrap
 2 W|DOUBLE WIDE
    ~bb..88

Each row is NN S|text, where S is the line size (blank single, W double width, T/B double height top/bottom). A ~ line gives renditions per cell (bold 1, underline 2, blink 4, reverse 8), a p line marks protected cells, f and b lines give foreground and background colour indexes (. for the default), soft-font characters show as ▯. A memory line appears when the cursor is off page 1 or the screen shows part of a longer page.

Session script commands

Command	Meaning
model vt420, size 24 80, autowrap on, answerback TEXT	Terminal configuration before spawn
extensions xterm-compat utf8 xterm-sgr	Opt-in non-DEC behaviour
spawn PROGRAM ARGS…	Start the program on a PTY (${VAR} expands environment variables)
send "TEXT"	Raw bytes to the program, with \r, \n, \t, \e, \xHH escapes
type "TEXT", key NAME	Typed text; DEC keys such as pf1, kp5, find, do, udk6
prompt "TEXT"	Wait until TEXT is just before the cursor
wait-text "TEXT", settle, sleep MS, timeout SECONDS	Waiting
snapshot NAME	Compare the screen with NAME.screen
step NAME	Snapshot at a Push <RETURN> prompt, then press Return
walk PREFIX "MENU PROMPT"	Snapshot every Push <RETURN> screen until the menu prompt returns
send-answerback, expect-exit	Answerback key; wait for the program to exit

esctest2

cargo xtask esctest fetches esctest2 at a pinned commit and runs it against the VT525 model with xterm compatibility enabled, at --max-vt-level=5. esctest2 describes xterm, so some tests fail by design. Each expected failure is listed in tests/conformance/esctest/expected-failures.txt together with its reason — xterm-only features, modes esctest expects xterm to lack, or a documented DEC behaviour.

The task fails if a test fails that is not on the list, if a listed test starts passing, or if an entry has no reason. cargo xtask esctest --update rewrites the list, keeping existing reasons and marking new entries TODO: triage. esctest2 is GPL-licensed and is only downloaded and run, never included in veetee.

Session recordings

veetee --record FILE.vtrec records what a host sends, the terminal's replies and checkpoints (see Connections). vt-headless replay FILE.vtrec feeds the recording through a terminal with the recorded configuration and compares the screen at each checkpoint, and at the end, with FILE/NAME.screen (in the snapshot format above):

vt-headless replay edt.vtrec --bless          # write the screens after checking them
vt-headless replay edt.vtrec                  # compare
vt-headless replay edt.vtrec --golden DIR     # screens somewhere else

cargo xtask openvms replays every recording in tests/conformance/openvms/.

Windows

The CI windows job builds the workspace with MSYS2's GTK 4 and libadwaita, runs the tests, bundles the release zip, and launches it with Mesa's software OpenGL to capture a screenshot of a Command Prompt session (uploaded as a build artifact).

Parser traces and fuzzing

cargo run -p vt-headless -- trace --8bit capture.bin   # one parser action per line
cargo +nightly fuzz run parser                         # requires cargo-fuzz