Skip to content

Releases

v0.7.0: the decoders and the Compound Files for ivy

Choose a tag to compare

@kenshaw kenshaw released this 17 Nov 06:15

This release adds the decoders and the reader of Compound Files that ivy needs, and moves the walker to binwalk-ng. The package magic gives the same answers as v0.6.1.

go get github.com/xo/magic@v0.7.0

The packages for ivy

  • compress/palmdoc decodes the compression of PalmDOC, compress/huffcdic decodes the Huffman compression of Mobipocket, and compress/lzx decodes LZX, as a port of lzxd. They do not register with compress, because libmagic does not use them (D61).
  • cfb reads the directory and the streams of a Compound File, the format of Office 97 to 2003, and the property sets of MS-OLEPS, such as the thumbnail of a document (D62). magic does not import it. Its engine keeps its own reader, which follows libmagic.

The walker follows binwalk-ng

binwalk-ng is the maintained fork of binwalk. The walker follows it at the commit 02835ae0f4, with each fix that needs no new decoder (D60).

  • The scan skips runs of zero bytes, as binwalk-ng does.
  • The output follows the Chroot of binwalk-ng. It cleans each path and writes relative links.
  • A carved extraction that fails runs again on the data to the end of the input.
  • The formats take the changes of binwalk-ng to their parsers, descriptions and extractors. eva, program_store and dlink_fw are new, and the walker now extracts srecord, srecord_generic and dms.

The three gates of the test module give no difference from binwalk-ng on 2301 inputs.

compress/xz and compress/zstd did not change, and stay at v0.6.0.

v0.6.1: find an xz stream at any offset

Choose a tag to compare

@kenshaw kenshaw released this 15 Nov 10:16

This release fixes the walker for xz data, and changes no answer of the package magic.

  • The walker finds an xz stream at any offset. v0.6.0 missed a stream that started further into the data than its own length, because it copied a bug of binwalk: since a commit after its release 3.1.0, binwalk compares the absolute offset of the next stream with the bytes that follow the match. The bug is reported to binwalk as ReFirmLabs/binwalk#982.
  • The oracle of the test module carries the same fix as a patch, so the gates still compare the walker with binwalk (D59).
  • cmd/walk says that it has no decoder of xz, lzma or zstd. Its module requires no other module, so it cannot import compress/xz and compress/zstd, as a program of its own does.

compress/xz and compress/zstd did not change, and stay at v0.6.0.

v0.6.0: the binary walker

Choose a tag to compare

@kenshaw kenshaw released this 15 Nov 09:13

This release adds the binary walker: the package walk finds the files that are embedded in a binary file, such as a firmware image, as binwalk 3 does, and extracts them. The package magic does not change.

The walker

The walker has the 111 signatures of binwalk 3, with the same names and descriptions, and gives the same matches, at the commit of binwalk that docs/upstream/upstreams.tsv names.

for e, err := range walk.WalkFile(ctx, "firmware.bin", walk.WithDepth(8)) {
	if err != nil {
		return err
	}
	fmt.Printf("%*s%#x %s\n", 2*e.Depth, "", e.Offset, e.Description)
}
  • Walk, WalkFile and WalkBytes return an iterator of entries, depth first.
  • Entry.Members gives the files of a match in memory, and walk.Extract writes them under an os.Root. No path or link leaves the root, and no device is made.
  • WithDepth walks the extracted files, as binwalk -M does. WithMaxOutput limits what one extraction writes.
  • walk.Entropy gives the entropy of each block, as binwalk -E.
  • cmd/walk is an example program.

A program imports the formats that it needs, or all of them:

import _ "github.com/xo/magic/walk/format/all"

The formats xz, lzma and zstd need the decoders of compress/xz and compress/zstd. The keys of the firmware of D-Link are in walk/format/encrypted, which a program imports to decrypt it.

Extraction

The walker extracts the formats that binwalk extracts with its own code, and gives the same files. It also extracts most of the formats that binwalk extracts with programs, and writes what those programs write. It runs no program:

  • SquashFS 1.x to 4.0, both byte orders, with gzip, lzma (and the LZMA of vendors), lzo, xz, lz4 and zstd, as sasquatch.
  • cramfs and ISO 9660, as 7-Zip; JFFS2, as jefferson; YAFFS, as unyaffs; UBI and UBIFS, as ubireader; FAT and ext, as tsk_recover.
  • zip, tar, cpio, 7-Zip (with PPMd and the filters), cab (stored and MSZIP), lzma, zstd, lz4, lzop and compress'd.

docs/BACKLOG.md lists the formats that the walker finds and does not extract yet.

How it is tested

The test module builds binwalk 3 from its source, and three gates compare the walker with it on 1881 inputs: the matches, the extracted files, and the walk of the extracted files. A test for each file system and archive compares the files with the tool that made the image, or with the tree that made it. FuzzWalk fuzzes every format.

The decoders

  • compress has a stream decoder, compress.Stream, which also gives the size of the stream. compress/xz and compress/zstd register one, at compress/xz/v0.6.0 and compress/zstd/v0.6.0.

Size

The walker is in packages of its own, so a program that imports only magic does not grow. walk/format/all adds about 0.9 MB.

License

The walker translates binwalk, under the MIT license, in LICENSE.binwalk, and the crate delink of its author, under the MIT license. Two of its decoders are copies from the Go standard library, under the BSD license of Go.

The formats build only where an int has 64 bits, as binwalk does.

v0.5.1: the README links the related xo projects

Choose a tag to compare

@kenshaw kenshaw released this 15 Nov 04:39

This release changes no code of the package. It gives the same answers as v0.5.0.

  • The README opens with a bar of links and the badges of the other xo projects: the tests, the Go reference, the latest release and Discord.
  • The README has a new section, Related projects, which lists the other xo projects, and iv, an image viewer for the terminal that uses magic.
  • The test module requires magic v0.5.0, as the decoder modules do.
  • CI runs on the macOS runners that GitHub still has, macos-15-intel and macos-latest.

compress/xz and compress/zstd did not change, and stay at v0.5.0.

v0.5.0: a port of libmagic to pure Go

Choose a tag to compare

@kenshaw kenshaw released this 15 Nov 04:26

magic is now a port of libmagic to pure Go. It needs no cgo, no C compiler and no libmagic, and a program that imports it builds for every target of Go.

The port gives the output of libmagic 5.48, with the commits of file after the release, to bbffb78d. The rules of file are compiled into the package.

A new API

The API is new, so this is a new minor version (D39). Detect, DetectFile, DetectReader and DetectReaderAt take a context.Context and return a Result. A Result holds every form of the answer at once: Description, MIME, Charset, Extensions, Apple and Matches. Options such as WithAll, WithDecompress and WithLimit take the place of the flags of libmagic. The package has no handle to close.

The README has a table that maps each call of v0.4.1 to the port. A file that cannot be opened is now an error, where libmagic gave the text "cannot open".

Compressed data

WithDecompress decodes a format only when the program imports its decoder, as a program imports the driver of database/sql:

import (
	_ "github.com/xo/magic/compress/bzip2"
	_ "github.com/xo/magic/compress/xz"
	_ "github.com/xo/magic/compress/zlib"
	_ "github.com/xo/magic/compress/zstd"
)

compress/xz and compress/zstd are modules of their own, tagged compress/xz/v0.5.0 and compress/zstd/v0.5.0. magic never runs a program.

Size and speed

A program that prints the MIME type of a file grows by about 980 KB, stripped, on linux/amd64. Most of it is the compiled rules.

The engine is about as fast as libmagic on most files, and slower on text. On linux/amd64, one walk for the MIME type took, with v0.4.1 and with the port:

Input v0.4.1 (cgo) v0.5.0
Word document 5.1 µs 4.3 µs
GIF image 20 µs 26 µs
XLSX file 35 µs 37 µs
random bytes 252 µs 334 µs
UTF-8 text 334 µs 1.09 ms

Most of the time on text is in Go's regexp. docs/BACKLOG.md lists the work that can make it faster.

How it is tested

The test module builds libmagic from the source of file at bbffb78d and compares the port with it: 2,471 files and damaged copies of them, 14 combinations of flags, as bytes and as files. Every answer is the same, except 50 where a decoder of Go reads damaged compressed data in another way than the library of C. All 113 tests of file pass.

License

The port translates file, which is under its own BSD license, in LICENSE.file. A program that ships magic must reproduce the notices that the README lists (D29).

Removed

The cgo package, MagicDB, the static archives and the modules of the targets are gone. The tags of those modules, such as libmagic/linux_amd64/v5.48.2, stay published.

v0.4.1: fix a crash on Linux with glibc

Choose a tag to compare

@kenshaw kenshaw released this 15 Nov 01:38

This release fixes a crash on Linux. Upgrade if you build on Linux with gcc, which is the default.

Fixed: a crash on Linux with glibc

Every release up to v0.4.0 can crash with a segmentation fault on Linux when the program links glibc, which is what gcc does on most Linux systems. A script of 17 bytes that starts with #! /usr/bin/cmd1 is enough. Other files get wrong descriptions, such as the size of a Netpbm image.

The Linux archives were compiled against the headers of musl, where regmatch_t has 8-byte offsets, and glibc writes 4-byte ones. libmagic then read the positions of a regular expression match wrongly.

The Linux archives now carry their own regex engine, libgnurx, which the Windows archives already use, under renamed names. libmagic never calls the regex functions of the C library, so the output is the same with glibc and with musl.

The new harness of the pure Go port found the crash. It builds libmagic 5.48 from the source of file as an oracle, and compares the package with it on 1,544 inputs and 11,571 questions. They now agree when the program links glibc, and when it links musl.

Versions

  • Root v0.4.1. The API does not change.
  • Target modules: v5.48.2 for linux_amd64, linux_arm64 and linux_arm, which hold the new archives. The others do not change.
  • libmagic 5.48, zlib 1.4, bzip2 1.0.8, xz 5.8.4, zstd 1.5.7.
  • libgnurx 2.5.1 is now in the Linux archives as well as the Windows ones. It is under LGPL 2.1.

Known limits

  • A program for linux/arm (32-bit) cannot link the archives with glibc, because they call the names that musl gives to its 64-bit time functions. Every release has this. linux/arm64 needs glibc 2.38 or later for strlcpy.
  • macOS has no CI, so the darwin archives are compiled and not run.

v0.4.0: a compressed database, and Windows and macOS fixes

Choose a tag to compare

@kenshaw kenshaw released this 15 Nov 00:55

The magic database is now embedded compressed, and every program that imports magic is about 10.4 MB smaller. This release also fixes the Windows and macOS archives of v0.3.1.

Breaking change

MagicDB was a []byte variable. It is now a function, MagicDB() ([]byte, error), that expands the database into a new slice of about 10 MiB on each call. A program that read the variable must call the function and check the error. A program that only identifies files does not use it, because New loads the database itself.

What changed

  • The database is embedded as libmagic/magic.mgc.zst, 345,182 bytes from zstd -19, in place of the 10,793,952-byte magic.mgc. New expands it with the libzstd.a that libmagic already links, straight into C memory. The Go heap never holds the database.
  • Measured on one machine: cmd/file went from 14,554,368 to 4,111,496 bytes. New went from about 0.27 ms to about 1.7 ms. A program that opens one handle and shares it, which is safe, does not notice.
  • A test checks the expanded database against a SHA-256 that the build records, and another feeds damaged input and expects an error.

Fixed: v0.3.1 did not link on Windows, and probably not on macOS

The archives of v0.3.1 came from zig 0.17.1, whose C libraries made the build of libmagic use two functions that a user's C library does not always have:

  • Windows: libmagic.a called strndup from the C library, so a link with MinGW gcc failed with undefined reference to strndup. CI found it, and MinGW gcc 16.2 reproduces it.
  • macOS: libmagic.a called pipe2, which is in the macOS 27 SDK of zig and not in older macOS. This was not tested, because no Mac and no macOS CI was available.

The four archives are rebuilt so that libmagic uses its own strndup and plain pipe, as in v0.3.0. Those modules are now v5.48.2. Do not use v0.3.1 on Windows or macOS.

How it was tested

  • CI passed on linux/amd64, linux/arm64, windows/amd64 and windows/arm64. macOS has no CI, because GitHub gives no runner for its jobs.
  • On the build host, the tests passed on linux/amd64, on linux/arm and linux/arm64 under qemu, and on windows/amd64 under wine, with zig and with MinGW gcc.
  • freebsd/amd64 links, and nothing ran it. The darwin archives compile, and were not linked.
  • A program in an empty module ran go get for v0.4.0, named a GIF, and checked the hash of MagicDB().

Known fault, not new

A program that links the Linux archives with glibc gets wrong descriptions where a rule prints part of a regular expression match, such as for Netpbm files. The archives are built against musl, where regmatch_t is 16 bytes, and it is 8 bytes in glibc. The MIME types and the tests are not affected. A program that links with musl gives the same output as file.

Versions

  • libmagic 5.48 (file 5.48)
  • zlib 1.4, bzip2 1.0.8, xz 5.8.4, zstd 1.5.7
  • libgnurx 2.5.1 on Windows, which is under LGPL 2.1
  • Root v0.4.0. Target modules: v5.48.2 for windows_amd64, windows_arm64, darwin_amd64 and darwin_arm64, and v5.48.1 for the others.

v0.3.1: rebuild with zlib 1.4

Choose a tag to compare

@kenshaw kenshaw released this 11 Nov 21:18

Do not use this release on Windows or macOS. It does not link with MinGW gcc (undefined reference to strndup), and its macOS archives call pipe2, which older macOS does not have. v0.4.0 fixes both.

This release rebuilds all archives with zlib 1.4. The Go code and the API do not change.

What changed

  • zlib is now 1.4. It fixes two CVEs: a buffer overflow in a non-blocking gzwrite(), and an out-of-bounds pointer in contrib/infback9. libmagic reads compressed files through zlib. We did not check whether either code path is reachable through libmagic.
  • Every target module has the new version v5.48.1, such as libmagic/linux_amd64/v5.48.1. The version of file stays 5.48, so the patch number grows, as D18 in docs/PLAN.md says. The root tag is v0.3.1.
  • The archives were built with zig 0.17.1. The archives of v0.3.0 came from zig 0.16, so all five libraries differ in bytes.
  • The Windows targets pass -Wno-incompatible-pointer-types. file 5.48 passes a long * where it wants a const time_t *, and the sizes differ on Windows. zig 0.16 warned about this and zig 0.17 stops with an error. The flag builds the same code as before and does not fix the mismatch.

To update, run go get github.com/xo/magic@v0.3.1.

How this release was tested

GitHub Actions was down when this release was made, so no CI run exists for it. The tests ran on the build host:

  • The package tests passed on linux/amd64, on linux/arm and linux/arm64 under qemu, and on windows/amd64 under wine.
  • windows/arm64 and freebsd/amd64 link into a program, and nothing ran them. The darwin archives compiled and were not linked.
  • A program in an empty module ran go get for v0.3.1 and printed the version and a MIME type.

No Go file changed since v0.3.0, which passed CI on all runnable jobs, including windows/arm64 with llvm-mingw.

Known fault, not new

A program that links the Linux archives with glibc gets a wrong description for Netpbm files (.pbm, .pgm, .ppm) on linux/amd64. The archives are built for musl. A program that links them with musl gives the same output as file. The MIME types and the tests are not affected. v0.2.1 and v0.3.0 have the same fault. The README and docs/BACKLOG.md describe it, and open question 9 in docs/PLAN.md asks how to fix it. The release notes of v0.3.0 said that the archives work under both libcs. That was true for linking and for MIME types only.

Versions

  • libmagic 5.48 (file 5.48)
  • zlib 1.4, bzip2 1.0.8, xz 5.8.4, zstd 1.5.7
  • libgnurx 2.5.1 on Windows, which is under LGPL 2.1

v0.3.0: a module for each platform

Choose a tag to compare

@kenshaw kenshaw released this 13 Oct 22:45

Each platform is now a Go module of its own. A build downloads the archives of your platform and no other.

What changed

  • Each target is a module in libmagic/<goos>_<goarch>/, such as github.com/xo/magic/libmagic/linux_amd64/v5. The root module imports the module of your platform and holds no archives. The root module is now about 12 MB.
  • Before this release, every user downloaded the archives of every platform. After it, a go build downloads the root module and one target module (6.7 to 9.6 MB). We tested this on linux/amd64 from an empty module cache.
  • go get and go mod tidy still fetch every target module once, because they read the imports for all platforms. The next build downloads one.
  • The import path github.com/xo/magic and the API do not change. Run go get github.com/xo/magic@v0.3.0.

New platforms

magic now builds the same platforms as resvg:

  • linux: amd64, arm64, arm (armv7)
  • darwin: amd64, arm64
  • windows: amd64, arm64
  • freebsd: amd64

The new targets are windows/arm64 and freebsd/amd64. The Linux archives are static and built for musl, so a program links them under glibc and under musl.

How well each platform is tested

  • The tests pass in CI on linux/amd64, linux/arm64, macOS arm64, windows/amd64 and windows/arm64. On windows/arm64, the C compiler is clang from llvm-mingw.
  • linux/arm passed under qemu on the build host, and has no CI job.
  • freebsd/amd64 links into a program. Nothing has run it, because GitHub has no hosted FreeBSD runner.
  • macOS amd64 has no CI result, because its CI job never gets a runner.

Versions

  • libmagic 5.48 (file 5.48)
  • zlib 1.3.2, bzip2 1.0.8, xz 5.8.4, zstd 1.5.7
  • libgnurx 2.5.1 on Windows, which is under LGPL 2.1

Tags

  • The root tag is v0.3.0.
  • Each target tag is the path of its folder and the version of file, such as libmagic/linux_amd64/v5.48.0.

The decisions are D16 to D19 in docs/PLAN.md.

magic v0.2.0

Choose a tag to compare

@kenshaw kenshaw released this 11 Sep 22:05

Full Changelog: v0.1.0...v0.2.0