Why we don't parse the compiled magic.mgc format

[unclesp1d3r]

Why we don't parse the compiled magic.mgc format

Short answer: magic.mgc isn't really a file format. It's libmagic's in-memory C structs written straight to disk. That ties it to one build of one version of libmagic on one platform. We're not willing to depend on that.

What's in the file

The compiler (file -C, or magic_compile()) parses the text magic files into an array of struct magic, then writes those structs out raw:

for (i = 0; i < MAGIC_SETS; i++) {
    len = m * map->nmagic[i];
    if (write(fd, map->magic[i], len) != (ssize_t)len) {
        file_error(ms, errno, "error writing `%s'", dbname);
    }
}

There's a small header first (a 4-byte magic number 0xF11E041C, a version number, per-set entry counts), then the struct array. Each entry is FILE_MAGICSIZE bytes, currently 432, with fixed-size arrays for the test value, the description, the MIME type, and the Apple/extension metadata.

Everything awkward about the format falls out of that one decision to dump the structs. Byte order is whatever the machine was. There are no markers and no conversion, so a .mgc built big-endian is unreadable little-endian. The compiler's padding choices are part of the on-disk layout too, and nobody wrote them down.

There's also no spec to work from. apprentice.c is the specification: about 3,000 lines of C that's also handling version migration and mmap.

The version churn

Since the layout is the struct layout, any change to struct magic breaks every .mgc in existence. New field, resized field, reordered field, doesn't matter. libmagic deals with this by bumping a version number and refusing to load anything that doesn't match exactly:

if (m->h.version != VERSIONNO) {
    file_error(ms, 0, "version mismatch (%d != %d)",
               m->h.version, VERSIONNO);
    return NULL;
}

Hence the error everyone has seen at least once:

File 5.45 supports only version 18 magic files. magic.mgc is version 16

Across the releases we tracked, the version went 8, 12, 14, 16, 18, 20. Supporting .mgc means writing a parser for each of those layouts and adding another one every time upstream touches the struct. What you get for that is faster startup. That's it. It's a bad trade and it doesn't get better over time.

What we do instead

We parse the text magic files, the same ones file compiles from. That format is documented in magic(5) and stable across libmagic releases. You can read it when a rule misbehaves. It's BSD-2 licensed, so we can bundle it.

We're not out on a limb here either. OpenBSD rewrote file from scratch and kept using the upstream Magdir files. Trail of Bits' PolyFile parses text. So does arcana. Every independent reimplementation we looked at made the same call, and OpenBSD's has been shipping for years.

If startup time ever turns into a real problem, the fix is our own compiled format built from our AST. We version it, so it stays stable. Adopting someone else's unstable binary layout to save a few milliseconds is how you sign up for permanent maintenance work.

What this means for you

Some distributions ship only the compiled database. Ubuntu installs magic.mgc and leaves the source directory empty, for example. On those systems, point --magic at a text magic file or directory, or use --use-builtin for the rules compiled into the binary.

Handing rmagic a .mgc file gives you an unsupported-format error rather than quietly wrong answers, which is the behavior we want.

The longer write-up, with the full version table and the licensing analysis, is at docs/research/magic-mgc-format-analysis.md.

If you've got a use case that actually needs .mgc support, say so here. We'd rather hear the argument than assume there isn't one.