QA Testing
Panel plugins talk to servers, so most of what can go wrong needs a server to go wrong against. The repository carries a sandbox for that: Docker environments that build mc from the working tree and run it against a host serving the same files over sftp, ssh, ftp and samba -- by hand, or with the keys pressed from a script.
Everything below lives in tests/misc/docker.
tests/misc/docker/sandbox.sh debian-12 up
That builds two images, starts the server and builds mc -- a few minutes the first time, seconds afterwards. Then:
tests/misc/docker/sandbox.sh debian-12 mc # mc in the container, by hand
tests/misc/docker/sandbox.sh debian-12 test # the keys pressed from a script
tests/misc/docker/sandbox.sh ui # the same, chosen from a menu
mc is built from your working tree. Nothing is written into the tree itself:
the sources are mounted read-only, copied inside, and the build lives in a
Docker volume. The environment name may be left out; debian-12 is the
default, or whatever $MC_SANDBOX says.
After editing the sources, rebuild without leaving the sandbox:
tests/misc/docker/sandbox.sh debian-12 build
The rest of the commands:
| command | what it does |
|---|---|
up |
images, server and mc, from nothing |
mc |
run mc against the environment |
build |
rebuild mc, keeping the object files: build [-f profiles]
|
test |
press the keys in the cases: test [-c subject] [-w transports] [-l locale] [-o key=value]... [-k keymap] [-g] [dir...]
|
ui |
the same, chosen from a menu |
check |
can every plugin be loaded, and does every protocol answer |
shell |
a shell next to mc: ssh, curl, smbclient are there |
remote |
a shell on the server |
logs |
what the server has to say |
down |
stop the containers |
clean |
stop them and throw the build away |
list |
the environments, subjects, transports, profiles and keymaps there are |
What is chosen is chosen apart:
-
the environment (
envs/<name>/): the image mc is built and run in.debian-12is the everyday one;debian-12-minimalhas nofile,unzip,zip,7z,bzip2;alpine-remotekeeps mc on Debian and puts the far end on Alpine (musl, busybox);debian-10is the oldest GLib the project takes, 2.58, with libarchive 3.3, curl 7.64 and libssh2 1.8 around it. Base images are pinned by digest. -
the subject (
cases/<name>/): the files and thecases.tsvof what to press on them.archivesopens archives over every transport;editorpresses keys in mcedit (the line filter, the search dialog);terminalpresses them in the embedded terminal (a command, Ctrl-L against Ctrl-Alt-L, the keybar, Ctrl-F1, insert mode in the shell's line);structopens a u-boot image and an MBR through the mcstruct plugin;panelpresses the quick filter, quick cd and the find dialog;luaopens a dBase table, an ELF and a picture through the viewers written in Lua. The last five are local only: they press keys on mc itself, not on a file a server holds. A picture is read as chafa's characters, which is what the script draws where the terminal has no sixel; sixel itself is not checked and cannot be, since it reaches the terminal as a DCS the screen library never sees. -
how mc is run there: the transport (
local,sftp,ftp,smb,shfor shell-link), the locale (-l ru_RU.KOI8-R), ini values written before mc starts (-o old_esc_mode=true), a keymap fromcommon/keymaps/(-k shift-tab-complete), and a build profile fromcommon/features.ini(build -f all,ncurses,-f asan; each set of features keeps its own build directory).
Before blaming the code, ask the sandbox whether it is alive:
$ tests/misc/docker/sandbox.sh debian-12 check
sftp ok
ftp ok
ssh ok
smb ok
mc ok
plugins: arcmc docker ftp git hello k8s panelize s3 samba sftp shell-link sqlite systemd
This needs no terminal, so it also works from a script. A protocol that says
FAILED here is a server that never came up, not a plugin that misbehaves.
The plugins: line says what configure left out: on debian-10 there is no
sftp (libssh2 1.8 is older than the 1.9 configure asks for) and no s3
(curl 7.64).
One host, one tree, four ways in. The user is mc and the password is mc
everywhere.
| panel | host / UNC | port | remote path |
|---|---|---|---|
| SFTP | remote |
22 | /home/mc/cases/archives |
| Shell link | remote |
22 | empty, or the same |
| FTP | remote |
21 | /cases/archives |
| SMB | //remote/cases |
-- | archives |
localhost will not do: mc runs in its own container and the server is the
one next to it, reachable by the name remote.
The same tree is in /work/local/archives inside the mc container, for
whatever needs no server at all. test writes the connections into the
plugins' ini files itself; by hand, make one with S-F4 in the plugin's panel.
One directory per situation, and in each a cases.tsv saying what its files
are for -- read it as a checklist, or let test walk it:
$ cat 03-nested/cases.tsv
file key expect why transports
outer.tar Enter archive panel then Enter on small.zip inside it
outer.tar Enter,on small.zip,Enter,..,.. the panel it came from twice: inner archive, outer archive, then sftp or ftp
zip-in-zip.zip cd zip-in-zip.zip/uzip:// extfs panel utar:// is gone, uzip:// is the filesystem left to try local
zip-in-zip.zip cd zip-in-zip.zip/uzip://,on small.zip,Enter extfs panel a file inside an mc filesystem is left to mc.ext.ini local
| directory | what it is for |
|---|---|
01-formats |
tar, zip and 7z, including one past libarchive's buffer |
02-content |
archives with no extension, and plain text named as one |
03-nested |
an archive inside an archive, and one inside uzip://
|
04-non-ascii |
Cyrillic and spaces in names, inside the archives and out |
The keys go comma separated, in order: Enter, F3, F5, C-o, .. (up
one level), on <name> (the cursor goes there), cd <path> (the Quick cd
box), type <text>. The expectations: archive panel, listing, error dialog, nothing, no error, the panel it came from, extfs panel, copy to the other panel (the file is then in /tmp, as big as mc said), the name as written (the shell printed it). The last column names the
transports a row is for; empty means all.
The archives are generated from a seed, so they are the same bytes wherever they are made, and screens can be compared between runs.
The interesting part is not the file, it is which panel you are standing in when you press the key.
sftp and shell link hand the archive over as a stream, so it opens
without being downloaded first. 01-formats/big.7z is the case that only
works because that stream can seek: a 7z keeps its directory at the end of
the file.
ftp and samba cannot supply a stream yet, so mc fetches a local copy first and you will see the pause.
A local panel in /work/local/archives covers the same ground with no
server. 03-nested/zip-in-zip.zip is the odd one: cd zip-in-zip.zip/uzip://
puts the panel inside an mc filesystem, where a file is not something
open(2) can reach, so Enter on the archive inside it goes to mc.ext.ini
rather than to a plugin, and opens as uzip:// again.
Anywhere, 02-content is what happens when the name does not say:
magic.ini knows archives by extension, so noext is left alone everywhere,
and notanarchive.tar.gz -- plain text with a lying name -- gets an error
box from the operation that was asked to open it. Silence there would hide
the mistake.
tests/misc/docker/sandbox.sh debian-12 test -w local,sftp,ftp,smb,sh
tests/misc/docker/sandbox.sh debian-12 test -w sh 01-formats
tests/misc/docker/sandbox.sh debian-12 test -l ru_RU.KOI8-R -o old_esc_mode=true -k shift-tab-complete
tests/misc/docker/sandbox.sh alpine-remote test -w sftp,sh,ftp,smb
run-cases.sh starts mc under tmux in the case directory (through the
plugin's connection list for a remote one), finds the file by quick search,
presses the keys and reads what came of it: an Arcmc: panel title, the
viewer's button bar, an Error box, the file in /tmp, or plainly
text: <what the screen must show> and no text: <what it must not>. A key
is Enter, F3, F5, C-o, .., on <name>, cd <path>, type <text>,
or key <name> for anything tmux has a name for: F4, M-S, C-M-l,
Escape. What mc keeps between runs -- the cursor position per file, the
history of every dialog -- is thrown away before each case, so one case does
not hand the next its state. mc's stderr is
read as well; an assertion or a critical warning fails the case whatever the
screen shows. Each line comes out as ok, FAIL (with the screen mc showed),
known, FIXED or skip; the exit status is 0 when nothing failed.
An environment that expects something else says so in envs/<name>/expect.tsv:
dir/file, key, expectation, and optionally the transports it applies to;
* stands for any case or any key. An expectation known: <why> marks a
failure that is understood -- the ftp plugin cannot fetch a name with
spaces, shell-link against a busybox host shows non-ASCII names as question
marks, there is no sftp plugin on Debian 10 -- so it stays out of the
failures, and is reported as FIXED the day it passes.
Every run writes reports/<stamp>-<env>/: index.md with a table per
transport and the failures, and under <transport>/ a results.tsv (case,
keys, expectation, verdict, milliseconds, reason), the screen of every
failure and mc's stderr per case. index.md is what goes into a release
issue.
sandbox.sh ui draws the whole choice on one screen -- environment,
subject, transports, locale, build profile, ini values, keymap, case
directories -- with the mouse or the arrows, shows the command it adds up
to, runs it and opens the screens of what failed. It holds nothing the
command line cannot do; the last choice is kept in reports/last.ui and the
last command in reports/last.cmd.
The same cases, with mc running under memcheck:
tests/misc/docker/sandbox.sh debian-12 build -f all,debug
tests/misc/docker/sandbox.sh debian-12 test -g -w local 01-formats
The debug profile builds with -O0 -g3 -fno-omit-frame-pointer, so a stack
in the log reads as source lines instead of addresses. -g starts mc under
valgrind --tool=memcheck.
It costs time: about 28 seconds a case against 3 without it, three minutes for
a directory of five cases. So a run under memcheck is one directory at a time,
not the whole subject over every transport. Every wait in the script is
multiplied by six for it, and $SLOW sets another factor for a slow machine.
mc is asked to quit with F10 at the end of a case instead of being killed,
because a killed process writes no summary; confirm_exit is turned off for
the run so that F10 does not stop at a question. The screen of the case is
taken before that.
What fails a case is an invalid read, write or free, or a jump on an uninitialised value with mc in the stack -- whatever the screen shows. What is definitely lost is written down and left to a person: mc frees little on the way out by design.
common/valgrind.supp holds what the libraries never free -- glib, ncurses,
S-Lang, dlopen, iconv, libmagic, samba, curl -- and names no frame of mc's
own. One thing it cannot hold is the regex engine: pcre2 compiles a pattern to
machine code of its own and reads the subject a word at a time past its end,
which memcheck reports as a jump on uninitialised bytes with a stack of bare
addresses. A suppression that matches an unnamed frame matches every error
there is, so those are counted apart instead, as opaque, and fail nothing.
A panel drawn with file highlighting on gives seven of them, the viewer about
a hundred.
The log of each case is kept next to its screen as <case>.<key>.valgrind,
<transport>/valgrind.tsv counts them, and index.md gains a table:
| transport | case | key | errors | invalid | opaque | definitely lost |
|-----------|-----------------------|-------|--------|---------|--------|-----------------|
| local | 01-formats/small.zip | Enter | 7 | 0 | 7 | 0 |
Only debian-12 has valgrind in its image; -g in another environment says so
and stops. sandbox.sh ui offers it as Memory check.
Keys pressed while mc is still coming up are dropped -- the shell in the terminal has not printed its prompt yet -- and memcheck stretches that window from a fraction of a second to several seconds, which is why the script waits before it presses anything.
tests/misc/docker/sandbox.sh debian-12 check
Before it asks the protocols for a listing, check asks whether every plugin
the build installed can be loaded: it reads the symbols each .so needs and
looks for them in the mc binary and in the libraries the process carries.
This is not a formality. mc links lib/ as a static archive unless
--enable-mclib is given, and a linker takes out of an archive only what the
program itself refers to. A widget that only a plugin uses is left behind, and
the plugin then fails at g_module_open with an undefined symbol -- which is
how the mcstruct plugin came not to load in a clean build, while it worked in
a tree configured with --enable-mclib. Nothing else catches that until
someone presses the key.
Add a directory under envs/ with a docker-compose.yml in it. That is the
whole of it: sandbox.sh finds environments by looking for that file, and
each one is a separate compose project with its own network, containers and
build volume. An existing environment is never edited, never rebuilt, and
never fights another one for a port.
One that only changes what is installed -- an older distribution, fewer
tools, another shell -- is a Dockerfile.mc with a different FROM and the
same COPY lines from common/; build-mc.sh does not care which
distribution it is on. One that changes the far end reuses the mc image and
points remote at another Dockerfile, as alpine-remote does. Copy
envs/debian-12/docker-compose.yml, change the project name, the
SANDBOX_ENV value and the dockerfile: lines, add a README.md saying
what the environment is meant to catch and an expect.tsv (the header line
is enough), and:
tests/misc/docker/sandbox.sh <name> up
Pin the base image by digest: a run should be the same run next month.
tests/misc/docker/sandbox.sh debian-12 remote # a shell on the server
tests/misc/docker/sandbox.sh debian-12 shell # a shell next to mc
tests/misc/docker/sandbox.sh debian-12 clean # containers and build gone
The unit tests run in there as well, against the same build:
tests/misc/docker/sandbox.sh debian-12 shell
cd /work/build-all/tests && make check