Tracking: the 14 stems skipped in test_file_corpus, and the ten causes behind them

[ESultanik]

Summary

tests/test_magic.py::test_file_corpus excludes 14 stems from the upstream libmagic corpus behind a single blanket comment:

if testfile.stem not in (
        "JW07022A.mp3", "gedcom", "cmd1", "cmd2", "cmd3", "cmd4", "jpeg-text", "jsonlines1",
        "multiple", "osm", "pnm1", "pnm2", "pnm3", "utf16xmlsvg"
):
    # The files we skip fail because there is a bug in our implementation that we have not yet fixed

The comment records that bugs exist but not which ones, so the list is not actionable and there is no way to tell when one of them is fixed. Running each skipped stem shows the 14 failures are a handful of distinct causes rather than 14 separate problems.

Inventory

Measured on master (commit eed1f27) with the libmagic 5.48 corpus, using the same normalization the test applies:

stem	expected	PolyFile reports
pnm1	Netpbm image data, size = 2 x 2, greymap, ASCII text	, greymap
pnm2	Netpbm image data, size = 2 x 2, rawbits, greymap	, rawbits, greymap
pnm3	Netpbm image data, size = 10 x 20, pixmap, ASCII text	, pixmap
gedcom	GEDCOM genealogy text version 5.5, ASCII text	GEDCOM genealogy text version 5.5 plus unmerged continuation lines
osm	OpenStreetMap XML data, ASCII text	OpenStreetMap XML data
cmd1	a /usr/bin/cmd1 script, ASCII text executable	a /usr/bin/cmd1 script executable (binary data)
cmd2	a /usr/bin/cmd2 script, ASCII text executable	ascii text
cmd3	a /usr/bin/cmd3 script executable (binary data)	data
cmd4	a /usr/bin/cmd4 script executable (binary data)	data
jpeg-text	ASCII text, with no line terminators	ascii text
jsonlines1	New Line Delimited JSON text data	ascii text
multiple	Viva File 2.0, then - RTF1.0, - Test File 1.0, - ABCD File, ASCII text, with no line terminators	ascii text
JW07022A.mp3	Audio file with ID3 version 2.2.0, contains: MPEG ADTS, layer III, ...	Audio file with ID3 version 2.2.0, contains: data
utf16xmlsvg	SVG Scalable Vector Graphics image, Unicode text, UTF-16, ...	, EFI variable 3997439, total size: 6881395 bytes

These group into roughly four causes:

1. The text-encoding suffix is never appended. libmagic classifies the encoding of a matched buffer and appends it, so OpenStreetMap XML data becomes OpenStreetMap XML data, ASCII text. PolyFile stops at the type description. Accounts for osm, gedcom, pnm1, pnm3, cmd1, cmd2, and utf16xmlsvg.
2. A parent test's description is dropped when a child continues it. The three pnm results lose Netpbm image data, size = 2 x 2 and keep only the child's , greymap, leaving output that begins with a comma. This looks like the same area as cause 1 but is a separate defect: the text is missing from the middle of the description rather than the end.
3. Custom .magic sidecars are mishandled. cmd1 through cmd4 each ship a .magic script that the test parses in place of the default matcher. cmd3 and cmd4 produce only data, meaning nothing in the sidecar matched.
4. Multiple matches are not combined. multiple expects libmagic's \n- -joined list of every match. PolyFile reports them as separate matches, which the test's assertIn cannot express.

JW07022A.mp3 and jsonlines1 do not obviously fall into any of the four and probably want individual investigation.

A related detail

PolyFile reports the plain-text classification in lower case, ascii text, where libmagic reports ASCII text:

import polyfile.magic
polyfile.magic.local_date = polyfile.magic.utc_date
from polyfile.magic import MagicMatcher, MAGIC_DEFS

matcher = MagicMatcher.parse(*MAGIC_DEFS)
with open("file/tests/jpeg-text.testfile", "rb") as f:
    print({str(m) for m in matcher.match(f.read())})

{'ascii text'}

test_file_corpus hides this by lower-casing both sides, and carries a dedicated escape hatch for the exact string ASCII text. Since PolyFile's default output mode mimics file, the casing is user-visible even though no test sees it.

Suggested outcome

Replace the blanket comment with per-stem entries naming the cause and linking the issue that tracks it, then split causes 1 through 4 into their own issues. Cause 1 alone would unskip seven of the fourteen.

Environment

• PolyFile at commit eed1f27
• libmagic 5.48 corpus
• Python 3.14

[ESultanik]

One caveat on the four causes above: the level-0 match order is currently close to arbitrary because of #3477, so a few of these may be ordering artifacts rather than description bugs. Worth re-measuring this table after that lands, before splitting the causes into separate issues.

[ESultanik]

Re-measured against the libmagic 5.48 definitions

Re-ran all 14 stems on master at 36602cb, after the 5.48 definition update. None of the new
patterns fixed any of them: all 14 still fail. The reference oracle is the submodule's own
binary, which is built at file/src/file (5.48) and reproduces every expected .result string:

TZ=UTC ./file/src/file -b -m file/magic/magic.mgc file/tests/<stem>.testfile   # strongest match
TZ=UTC ./file/src/file -b -k -m file/magic/magic.mgc file/tests/<stem>.testfile # all matches
TZ=UTC ./file/src/file -m file/magic/magic.mgc -l                               # test strengths

Two changes since the original table:

• cmd1 now produces a second match, a /usr/bin/cmd1 script text executable, alongside
  a /usr/bin/cmd1 script executable (binary data).
• JW07022A.mp3 now also produces a tar archive (old) match.

Three of the four stated causes are wrong

Cause 3, "custom .magic sidecars are mishandled", is false. cmd1 through cmd4 ship no
.magic sidecar in the FILE5_48 corpus:

$ ls file/tests/*.magic
file/tests/multiple-A.magic  file/tests/multiple-B.magic
file/tests/regex-eol.magic   file/tests/searchbug.magic

They are plain shebang files matched against the default definitions, driven by
polyfile/magic_defs/varied.script:8-14. The only sidecar gap is multiple, which ships two
sidecars and a .flags file that the harness does not read at all — now #3487.

Cause 2, "a parent test's description is dropped when a child continues it", is the wrong
mechanism. use/NamedTest message propagation works correctly: UseTest._match
(polyfile/magic.py:2517) re-yields the named test's entire result stream into the invoking test's
flat result list, and Match.message (polyfile/magic.py:2974) concatenates it in order. Verified
both with a synthetic named test and by fixing the real defect.

The real cause is that RegexType.parse_expected (polyfile/magic.py:1887) never strips libmagic's
= equality prefix, so polyfile/magic_defs/images:192 compiles to
re.compile(b'=^[0-9]{1,50}[ \t\x0c\r\n]+[0-9]{1,50}') — a literal = followed by ^, which cannot
match. StringTest.parse (polyfile/magic.py:1327-1329) does strip it, so only regex is affected;
40 =-prefixed regex tests ship across 9 definition files. Now #3482.

The tar archive (old) match on JW07022A.mp3 is not a defect. Real libmagic 5.48 finds it too:

$ TZ=UTC ./file/src/file -b -k -m file/magic/magic.mgc file/tests/JW07022A.mp3.testfile
Audio file with ID3 version 2.2.0, contains: MPEG ADTS, layer III, v1, 96 kbps, 44.1 kHz, Monaural\012- \012-  tar archive (old), type 'C' ID3\002, ...\012- data

It is weaker than the ID3 match, so plain file does not print it. PolyFile reports every match, so
seeing it is correct behavior.

The actual causes

Stem	Blocked on
pnm2	#3482
pnm1, pnm3	#3482, then #3488
cmd3, cmd4	#3483
gedcom	#3483, then #3488
cmd1, cmd2	#3483, #3488, #3490
JW07022A.mp3	#3485
jsonlines1	#3486
jpeg-text, osm	#3488
utf16xmlsvg	#3484, #3488, #3489
multiple	#3487, #3491 (and #3477)

Each cause is now its own issue, and this one becomes the tracking issue:

• regex tests ignore the = operator, report the wrong %s extent, and drop the s flag #3482 — regex ignores the = operator, reports the wrong %s extent, and drops the s flag
• The string w flag never reaches the matcher, and string relative offsets use the consumed length #3483 — the string w flag never reaches the matcher, and string relative offsets use the
  consumed length
• A use test succeeds even when the named test list it references matched nothing #3484 — a use test succeeds even when the named list matched nothing
• Indirect offsets of type .i and .I are not decoded as ID3 synchsafe integers #3485 — indirect offsets of type .i and .I are not ID3 synchsafe-decoded
• Newline-delimited JSON is reported as plain text #3486 — newline-delimited JSON is reported as plain text
• test_file_corpus ignores per-test .magic sidecars and .flags files #3487 — test_file_corpus ignores per-test .magic sidecars and .flags files
• No equivalent of libmagic's file_ascmagic, so the text-encoding description is never appended #3488 — no file_ascmagic equivalent, so the text-encoding description is never appended
• Text tests run against raw bytes, so no text definition matches a UTF-16 file #3489 — text tests run against raw bytes, so no text definition matches a UTF-16 file
• The t and b flags do not select the text or binary soft-magic pass #3490 — the t and b flags do not select the text or binary pass (supersedes search flags b and t are mapped to the wrong behavior or dropped #3478's scope)
• No MAGIC_CONTINUE equivalent, so matches are never joined with libmagic's separator #3491 — no MAGIC_CONTINUE equivalent (blocked on Test strength ignores libmagic's type and relation terms, so 96% of tests tie at 20 #3477)

Measured effect of the small fixes

Prototyped #3482, #3483 and #3485 as monkeypatches and ran the full 88-stem corpus:

NEWLY PASSING (of the 14 skipped): 4 -> ['JW07022A.mp3', 'cmd3', 'cmd4', 'pnm2']
STILL FAILING:                    10 -> ['cmd1', 'cmd2', 'gedcom', 'jpeg-text', 'jsonlines1',
                                         'multiple', 'osm', 'pnm1', 'pnm3', 'utf16xmlsvg']
REGRESSIONS among the other 74:    0

With #3486 that is 5 of 14. The remaining 9 all need #3488, the encoding trailer.

One trap worth recording: the declared-length relative-offset rule in #3483 applies to string
only. Applying it to search as well turns gedcom's message into
GEDCOM genealogy text version 2 VERS 2.x, because libmagic's moffset uses the found extent for
FILE_SEARCH (file/src/softmagic.c:912-914) and the declared vallen for FILE_STRING
(softmagic.c:904-905).

Replacing the skip list

#3487 also replaces the blanket tuple with a mapping from each still-failing stem to the issue that
explains it, and asserts that a mapped stem still fails, so that a fix cannot land unnoticed and the
list shrinks on its own.

[ESultanik]

Retired. KNOWN_FAILURES at tests/test_magic.py:28-35 is now empty, and every stem of the
libmagic 5.48 corpus matches what file reports.

All ten cause-issues this tracked have merged: #3482, #3483, #3484, #3485, #3486, #3487, #3488,
#3489, #3490, and #3491.

The comment that replaced the blanket skip list points back here as the record of how the original
fourteen were retired, so the history stays discoverable:

KNOWN_FAILURES: Dict[str, int] = {
    # Empty, and worth keeping that way: every stem of the libmagic corpus now matches what
    # libmagic reports. `test_file_corpus` asserts that a stem listed here still FAILS, so an entry
    # is a deliberate record of a filed bug rather than a way to quiet the suite. Add one only
    # alongside the issue that explains it, mapped to that issue number, and delete it in the
    # change that fixes it. Issue #3480 tracks how the previous fourteen were retired.
}

Closing as completed.