QA Testing
Panel plugins talk to servers, so most of what can go wrong needs a server to go wrong against. The repository carries a sandbox for that: Docker environments that build mc from the working tree and run it against a host serving the same files over sftp, ssh, ftp and samba -- by hand, or with the keys pressed from a script.
Everything below lives in tests/misc/docker.
tests/misc/docker/sandbox.sh debian-12 up
That builds two images, starts the server and builds mc -- a few minutes the first time, seconds afterwards. Then:
tests/misc/docker/sandbox.sh debian-12 mc # mc in the container, by hand
tests/misc/docker/sandbox.sh debian-12 test # the keys pressed from a script
tests/misc/docker/sandbox.sh ui # the same, chosen from a menu
mc is built from your working tree. Nothing is written into the tree itself:
the sources are mounted read-only, copied inside, and the build lives in a
Docker volume. The environment name may be left out; debian-12 is the
default, or whatever $MC_SANDBOX says.
After editing the sources, rebuild without leaving the sandbox:
tests/misc/docker/sandbox.sh debian-12 build
The rest of the commands:
| command | what it does |
|---|---|
up |
images, server and mc, from nothing |
mc |
run mc against the environment |
build |
rebuild mc, keeping the object files: build [-f profiles]
|
test |
press the keys in the cases: test [-c subject] [-w transports] [-l locale] [-o key=value]... [-k keymap] [dir...]
|
ui |
the same, chosen from a menu |
check |
ask every protocol for a listing, without a terminal |
shell |
a shell next to mc: ssh, curl, smbclient are there |
remote |
a shell on the server |
logs |
what the server has to say |
down |
stop the containers |
clean |
stop them and throw the build away |
list |
the environments, subjects, transports, profiles and keymaps there are |
What is chosen is chosen apart:
-
the environment (
envs/<name>/): the image mc is built and run in.debian-12is the everyday one;debian-12-minimalhas nofile,unzip,zip,7z,bzip2;alpine-remotekeeps mc on Debian and puts the far end on Alpine (musl, busybox);debian-10is the oldest GLib the project takes, 2.58, with libarchive 3.3, curl 7.64 and libssh2 1.8 around it. Base images are pinned by digest. -
the subject (
cases/<name>/): the files and thecases.tsvof what to press on them. One so far,archives. -
how mc is run there: the transport (
local,sftp,ftp,smb,shfor shell-link), the locale (-l ru_RU.KOI8-R), ini values written before mc starts (-o old_esc_mode=true), a keymap fromcommon/keymaps/(-k shift-tab-complete), and a build profile fromcommon/features.ini(build -f all,ncurses,-f asan; each set of features keeps its own build directory).
Before blaming the code, ask the sandbox whether it is alive:
$ tests/misc/docker/sandbox.sh debian-12 check
sftp ok
ftp ok
ssh ok
smb ok
mc ok
plugins: arcmc docker ftp git hello k8s panelize s3 samba sftp shell-link sqlite systemd
This needs no terminal, so it also works from a script. A protocol that says
FAILED here is a server that never came up, not a plugin that misbehaves.
The plugins: line says what configure left out: on debian-10 there is no
sftp (libssh2 1.8 is older than the 1.9 configure asks for) and no s3
(curl 7.64).
One host, one tree, four ways in. The user is mc and the password is mc
everywhere.
| panel | host / UNC | port | remote path |
|---|---|---|---|
| SFTP | remote |
22 | /home/mc/cases/archives |
| Shell link | remote |
22 | empty, or the same |
| FTP | remote |
21 | /cases/archives |
| SMB | //remote/cases |
-- | archives |
localhost will not do: mc runs in its own container and the server is the
one next to it, reachable by the name remote.
The same tree is in /work/local/archives inside the mc container, for
whatever needs no server at all. test writes the connections into the
plugins' ini files itself; by hand, make one with S-F4 in the plugin's panel.
One directory per situation, and in each a cases.tsv saying what its files
are for -- read it as a checklist, or let test walk it:
$ cat 03-nested/cases.tsv
file key expect why transports
outer.tar Enter archive panel then Enter on small.zip inside it
outer.tar Enter,on small.zip,Enter,..,.. the panel it came from twice: inner archive, outer archive, then sftp or ftp
zip-in-zip.zip cd zip-in-zip.zip/uzip:// extfs panel utar:// is gone, uzip:// is the filesystem left to try local
zip-in-zip.zip cd zip-in-zip.zip/uzip://,on small.zip,Enter extfs panel a file inside an mc filesystem is left to mc.ext.ini local
| directory | what it is for |
|---|---|
01-formats |
tar, zip and 7z, including one past libarchive's buffer |
02-content |
archives with no extension, and plain text named as one |
03-nested |
an archive inside an archive, and one inside uzip://
|
04-non-ascii |
Cyrillic and spaces in names, inside the archives and out |
The keys go comma separated, in order: Enter, F3, F5, C-o, .. (up
one level), on <name> (the cursor goes there), cd <path> (the Quick cd
box), type <text>. The expectations: archive panel, listing, error dialog, nothing, no error, the panel it came from, extfs panel, copy to the other panel (the file is then in /tmp, as big as mc said), the name as written (the shell printed it). The last column names the
transports a row is for; empty means all.
The archives are generated from a seed, so they are the same bytes wherever they are made, and screens can be compared between runs.
The interesting part is not the file, it is which panel you are standing in when you press the key.
sftp and shell link hand the archive over as a stream, so it opens
without being downloaded first. 01-formats/big.7z is the case that only
works because that stream can seek: a 7z keeps its directory at the end of
the file.
ftp and samba cannot supply a stream yet, so mc fetches a local copy first and you will see the pause.
A local panel in /work/local/archives covers the same ground with no
server. 03-nested/zip-in-zip.zip is the odd one: cd zip-in-zip.zip/uzip://
puts the panel inside an mc filesystem, where a file is not something
open(2) can reach, so Enter on the archive inside it goes to mc.ext.ini
rather than to a plugin, and opens as uzip:// again.
Anywhere, 02-content is what happens when the name does not say:
magic.ini knows archives by extension, so noext is left alone everywhere,
and notanarchive.tar.gz -- plain text with a lying name -- gets an error
box from the operation that was asked to open it. Silence there would hide
the mistake.
tests/misc/docker/sandbox.sh debian-12 test -w local,sftp,ftp,smb,sh
tests/misc/docker/sandbox.sh debian-12 test -w sh 01-formats
tests/misc/docker/sandbox.sh debian-12 test -l ru_RU.KOI8-R -o old_esc_mode=true -k shift-tab-complete
tests/misc/docker/sandbox.sh alpine-remote test -w sftp,sh,ftp,smb
run-cases.sh starts mc under tmux in the case directory (through the
plugin's connection list for a remote one), finds the file by quick search,
presses the keys and reads what came of it: an Arcmc: panel title, the
viewer's button bar, an Error box, the file in /tmp. mc's stderr is
read as well; an assertion or a critical warning fails the case whatever the
screen shows. Each line comes out as ok, FAIL (with the screen mc showed),
known, FIXED or skip; the exit status is 0 when nothing failed.
An environment that expects something else says so in envs/<name>/expect.tsv:
dir/file, key, expectation, and optionally the transports it applies to;
* stands for any case or any key. An expectation known: <why> marks a
failure that is understood -- the ftp plugin cannot fetch a name with
spaces, shell-link against a busybox host shows non-ASCII names as question
marks, there is no sftp plugin on Debian 10 -- so it stays out of the
failures, and is reported as FIXED the day it passes.
Every run writes reports/<stamp>-<env>/: index.md with a table per
transport and the failures, and under <transport>/ a results.tsv (case,
keys, expectation, verdict, milliseconds, reason), the screen of every
failure and mc's stderr per case. index.md is what goes into a release
issue.
sandbox.sh ui draws the whole choice on one screen -- environment,
subject, transports, locale, build profile, ini values, keymap, case
directories -- with the mouse or the arrows, shows the command it adds up
to, runs it and opens the screens of what failed. It holds nothing the
command line cannot do; the last choice is kept in reports/last.ui and the
last command in reports/last.cmd.
Add a directory under envs/ with a docker-compose.yml in it. That is the
whole of it: sandbox.sh finds environments by looking for that file, and
each one is a separate compose project with its own network, containers and
build volume. An existing environment is never edited, never rebuilt, and
never fights another one for a port.
One that only changes what is installed -- an older distribution, fewer
tools, another shell -- is a Dockerfile.mc with a different FROM and the
same COPY lines from common/; build-mc.sh does not care which
distribution it is on. One that changes the far end reuses the mc image and
points remote at another Dockerfile, as alpine-remote does. Copy
envs/debian-12/docker-compose.yml, change the project name, the
SANDBOX_ENV value and the dockerfile: lines, add a README.md saying
what the environment is meant to catch and an expect.tsv (the header line
is enough), and:
tests/misc/docker/sandbox.sh <name> up
Pin the base image by digest: a run should be the same run next month.
tests/misc/docker/sandbox.sh debian-12 remote # a shell on the server
tests/misc/docker/sandbox.sh debian-12 shell # a shell next to mc
tests/misc/docker/sandbox.sh debian-12 clean # containers and build gone
The unit tests run in there as well, against the same build:
tests/misc/docker/sandbox.sh debian-12 shell
cd /work/build-all/tests && make check