Repository navigation
v0.6.0: the binary walker
This release adds the binary walker: the package walk finds the files that are embedded in a binary file, such as a firmware image, as binwalk 3 does, and extracts them. The package magic does not change.
The walker
The walker has the 111 signatures of binwalk 3, with the same names and descriptions, and gives the same matches, at the commit of binwalk that docs/upstream/upstreams.tsv names.
for e, err := range walk.WalkFile(ctx, "firmware.bin", walk.WithDepth(8)) {
if err != nil {
return err
}
fmt.Printf("%*s%#x %s\n", 2*e.Depth, "", e.Offset, e.Description)
}Walk,WalkFileandWalkBytesreturn an iterator of entries, depth first.Entry.Membersgives the files of a match in memory, andwalk.Extractwrites them under anos.Root. No path or link leaves the root, and no device is made.WithDepthwalks the extracted files, as binwalk -M does.WithMaxOutputlimits what one extraction writes.walk.Entropygives the entropy of each block, as binwalk -E.cmd/walkis an example program.
A program imports the formats that it needs, or all of them:
import _ "github.com/xo/magic/walk/format/all"The formats xz, lzma and zstd need the decoders of compress/xz and compress/zstd. The keys of the firmware of D-Link are in walk/format/encrypted, which a program imports to decrypt it.
Extraction
The walker extracts the formats that binwalk extracts with its own code, and gives the same files. It also extracts most of the formats that binwalk extracts with programs, and writes what those programs write. It runs no program:
- SquashFS 1.x to 4.0, both byte orders, with gzip, lzma (and the LZMA of vendors), lzo, xz, lz4 and zstd, as sasquatch.
- cramfs and ISO 9660, as 7-Zip; JFFS2, as jefferson; YAFFS, as unyaffs; UBI and UBIFS, as ubireader; FAT and ext, as tsk_recover.
- zip, tar, cpio, 7-Zip (with PPMd and the filters), cab (stored and MSZIP), lzma, zstd, lz4, lzop and compress'd.
docs/BACKLOG.md lists the formats that the walker finds and does not extract yet.
How it is tested
The test module builds binwalk 3 from its source, and three gates compare the walker with it on 1881 inputs: the matches, the extracted files, and the walk of the extracted files. A test for each file system and archive compares the files with the tool that made the image, or with the tree that made it. FuzzWalk fuzzes every format.
The decoders
compresshas a stream decoder,compress.Stream, which also gives the size of the stream.compress/xzandcompress/zstdregister one, atcompress/xz/v0.6.0andcompress/zstd/v0.6.0.
Size
The walker is in packages of its own, so a program that imports only magic does not grow. walk/format/all adds about 0.9 MB.
License
The walker translates binwalk, under the MIT license, in LICENSE.binwalk, and the crate delink of its author, under the MIT license. Two of its decoders are copies from the Go standard library, under the BSD license of Go.
The formats build only where an int has 64 bits, as binwalk does.