Repository navigation
v0.4.0: a compressed database, and Windows and macOS fixes
The magic database is now embedded compressed, and every program that imports magic is about 10.4 MB smaller. This release also fixes the Windows and macOS archives of v0.3.1.
Breaking change
MagicDB was a []byte variable. It is now a function, MagicDB() ([]byte, error), that expands the database into a new slice of about 10 MiB on each call. A program that read the variable must call the function and check the error. A program that only identifies files does not use it, because New loads the database itself.
What changed
- The database is embedded as
libmagic/magic.mgc.zst, 345,182 bytes fromzstd -19, in place of the 10,793,952-bytemagic.mgc.Newexpands it with thelibzstd.athat libmagic already links, straight into C memory. The Go heap never holds the database. - Measured on one machine:
cmd/filewent from 14,554,368 to 4,111,496 bytes.Newwent from about 0.27 ms to about 1.7 ms. A program that opens one handle and shares it, which is safe, does not notice. - A test checks the expanded database against a SHA-256 that the build records, and another feeds damaged input and expects an error.
Fixed: v0.3.1 did not link on Windows, and probably not on macOS
The archives of v0.3.1 came from zig 0.17.1, whose C libraries made the build of libmagic use two functions that a user's C library does not always have:
- Windows:
libmagic.acalledstrndupfrom the C library, so a link with MinGW gcc failed withundefined reference to strndup. CI found it, and MinGW gcc 16.2 reproduces it. - macOS:
libmagic.acalledpipe2, which is in the macOS 27 SDK of zig and not in older macOS. This was not tested, because no Mac and no macOS CI was available.
The four archives are rebuilt so that libmagic uses its own strndup and plain pipe, as in v0.3.0. Those modules are now v5.48.2. Do not use v0.3.1 on Windows or macOS.
How it was tested
- CI passed on linux/amd64, linux/arm64, windows/amd64 and windows/arm64. macOS has no CI, because GitHub gives no runner for its jobs.
- On the build host, the tests passed on linux/amd64, on linux/arm and linux/arm64 under qemu, and on windows/amd64 under wine, with zig and with MinGW gcc.
- freebsd/amd64 links, and nothing ran it. The darwin archives compile, and were not linked.
- A program in an empty module ran
go getfor v0.4.0, named a GIF, and checked the hash ofMagicDB().
Known fault, not new
A program that links the Linux archives with glibc gets wrong descriptions where a rule prints part of a regular expression match, such as for Netpbm files. The archives are built against musl, where regmatch_t is 16 bytes, and it is 8 bytes in glibc. The MIME types and the tests are not affected. A program that links with musl gives the same output as file.
Versions
- libmagic 5.48 (file 5.48)
- zlib 1.4, bzip2 1.0.8, xz 5.8.4, zstd 1.5.7
- libgnurx 2.5.1 on Windows, which is under LGPL 2.1
- Root
v0.4.0. Target modules:v5.48.2for windows_amd64, windows_arm64, darwin_amd64 and darwin_arm64, andv5.48.1for the others.