Skip to content

v0.6.0: the binary walker

Choose a tag to compare

@kenshaw kenshaw released this 15 Nov 09:13
· 20 commits to main since this release

This release adds the binary walker: the package walk finds the files that are embedded in a binary file, such as a firmware image, as binwalk 3 does, and extracts them. The package magic does not change.

The walker

The walker has the 111 signatures of binwalk 3, with the same names and descriptions, and gives the same matches, at the commit of binwalk that docs/upstream/upstreams.tsv names.

for e, err := range walk.WalkFile(ctx, "firmware.bin", walk.WithDepth(8)) {
	if err != nil {
		return err
	}
	fmt.Printf("%*s%#x %s\n", 2*e.Depth, "", e.Offset, e.Description)
}
  • Walk, WalkFile and WalkBytes return an iterator of entries, depth first.
  • Entry.Members gives the files of a match in memory, and walk.Extract writes them under an os.Root. No path or link leaves the root, and no device is made.
  • WithDepth walks the extracted files, as binwalk -M does. WithMaxOutput limits what one extraction writes.
  • walk.Entropy gives the entropy of each block, as binwalk -E.
  • cmd/walk is an example program.

A program imports the formats that it needs, or all of them:

import _ "github.com/xo/magic/walk/format/all"

The formats xz, lzma and zstd need the decoders of compress/xz and compress/zstd. The keys of the firmware of D-Link are in walk/format/encrypted, which a program imports to decrypt it.

Extraction

The walker extracts the formats that binwalk extracts with its own code, and gives the same files. It also extracts most of the formats that binwalk extracts with programs, and writes what those programs write. It runs no program:

  • SquashFS 1.x to 4.0, both byte orders, with gzip, lzma (and the LZMA of vendors), lzo, xz, lz4 and zstd, as sasquatch.
  • cramfs and ISO 9660, as 7-Zip; JFFS2, as jefferson; YAFFS, as unyaffs; UBI and UBIFS, as ubireader; FAT and ext, as tsk_recover.
  • zip, tar, cpio, 7-Zip (with PPMd and the filters), cab (stored and MSZIP), lzma, zstd, lz4, lzop and compress'd.

docs/BACKLOG.md lists the formats that the walker finds and does not extract yet.

How it is tested

The test module builds binwalk 3 from its source, and three gates compare the walker with it on 1881 inputs: the matches, the extracted files, and the walk of the extracted files. A test for each file system and archive compares the files with the tool that made the image, or with the tree that made it. FuzzWalk fuzzes every format.

The decoders

  • compress has a stream decoder, compress.Stream, which also gives the size of the stream. compress/xz and compress/zstd register one, at compress/xz/v0.6.0 and compress/zstd/v0.6.0.

Size

The walker is in packages of its own, so a program that imports only magic does not grow. walk/format/all adds about 0.9 MB.

License

The walker translates binwalk, under the MIT license, in LICENSE.binwalk, and the crate delink of its author, under the MIT license. Two of its decoders are copies from the Go standard library, under the BSD license of Go.

The formats build only where an int has 64 bits, as binwalk does.