Repository navigation
PolyFile 1.0 expands recursive analysis of archives and compressed streams, adds a terminal
explorer, and updates the HTML byte viewer. Structured output identifies the original file or
decoded buffer for each byte range. See the JSON output format.
The tested Python range is 3.11–3.14. On Python 3.15, the pinned google-re2 dependency lacks
compatible wheels, and clean installations fail while building it from source.
Track Python 3.15 installation support.
Added
- Ship
py.typedfor Python type checkers and document the public analysis and extension APIs. - Breaking. Identify GGUF models as
model/x-ggufand map their structure automatically.
The refreshed Kaitai gallery also adds a TP-Link MiniFS parser and updates the RPM and U-Boot
uImage parsers. Kaitai gallery update. - Breaking: Add
--explore FILE, a Textual terminal inspector with source-aware structure,
virtualized hex and UTF-8 views, byte selection, coverage, navigation, cancellable search, and
exact byte extraction. Textual is installed by default. Explorer mode opens after analysis and
requires a named file with stdin, stdout, and stderr connected to an interactive terminal. - Breaking: Recursively classify and parse PNG images in ICO files and architecture members
in supported universal Mach-O binaries. Preserve source coordinates, directory fields, PNG
previews, and shared analysis limits; report invalid and overlapping member ranges explicitly. - Breaking: Analyze 7z archive structure and validated members with
py7zr==1.1.4in
isolated helpers. Solid/non-solid archives, compressed headers, checksums, decoded source
coordinates, and optional recursive/helper resource limits have regression coverage. - Breaking: HTML reports show embedded image previews beside the ASCII column as you scroll
their byte ranges. Nearby previews share browsing controls, decoded images identify their encoded
containers, and narrow layouts use a strip below the bytes. Activate a preview to enlarge it
or reveal its structure node; the existing tree previews remain available. - Breaking: PDF tokens and mapped elements retain source-qualified input dependencies through
supported arithmetic, conversions, selection, and reconstruction. Computed values use zero-size
owner anchors, and disjoint or cross-source inputs remain separate. Untagged prefixes no longer
invent lexical positions. Hex and escaped string payloads recurse in decoded coordinates. - Breaking: Qualify PDF instrumentation against
pdfminer.six==20260107, isolate private API
hooks, and report unsupported status for unqualified releases. Preserve raw lexeme extents,
dictionary names and values, compressed-object sources, encrypted arrays, and wrapper metadata. - Document the PDFminer upgrade qualification procedure and supported provenance boundaries.
- Breaking: Unencrypted LRZIP 0.6 and 0.7 archives are recursively analyzed through an
installed LRZIP 0.7.3 or newer helper, preserving physical framing and decoded source
coordinates. Missing helpers and unsupported archive variants report explicit status; declared
sizes, integrity, optional output limits, and helper resource controls gate recursive analysis. - Breaking: Lzip version 1 payloads are recursively analyzed with native header, raw-LZMA,
CRC32, decoded-size, and member-size validation. Concatenated members preserve physical
structure and share optional resource limits. Older versions report unsupported status;
the decoder uses the Python standard library and requires no external helper. - Breaking: Decode zlib streams and recursively map validated content, preserving source-aware
offsets and reporting checksum, dictionary, trailing-data, and resource-limit failures (#3625). - Breaking: Modern LZ4 payloads are recursively analyzed after validating frame headers,
block/content checksums, and concatenated/skippable frames. The requiredlz4==4.4.5backend
honors optional output, frame, depth, and native-buffer limits. Legacy frames and declared
external dictionaries report unsupported status. - Breaking: Zstandard payloads are recursively analyzed after validating concatenated and
skippable frames. Optional output, frame, depth, and decoder-window limits preserve explicit
failure status. Python 3.11 through 3.13 requiresbackports.zstd==1.8.0; Python 3.14 uses its
standard-library backend. External dictionaries and historical frames report unsupported status. - Breaking: Bzip2 streams are decoded and recursively analyzed using Python's standard
library. Concatenated streams retain their physical framing and produce one validated decoded
buffer; integrity errors, truncation, and configured resource limits report explicit status. - Breaking: Identify SIMH tape images with native record checks, map their records and
markers, and recursively analyze independently recognizable record payloads (#3619). - Breaking: ELF identification descriptions now include validated linkage, interpreter,
GNU/Go build IDs, ABI versions, stripping/debug information, and supported core-file details.
The command line, Python identification API, and SBuD root values share these descriptions;
malformed metadata retains the ELF classification and reports a diagnostic. - Breaking: LZMA-alone files are decoded and recursively analyzed using the standard
library, preserving their physical header and decoded coordinate source. Known and streamed
sizes, supported properties, malformed endings, and optional resource limits are validated. - Breaking: XZ streams are decoded and recursively analyzed, with complete integrity checks,
concatenated-stream and four-byte padding semantics, source-aware archive recursion, and
optional native memory, decoded output, depth, and stream-count limits. - Breaking: UNIX pack streams now emit
application/x-unix-packand recursively analyze
decoded content through a bounded native canonical Huffman decoder. Each concatenated member
retains physical header/tree/body offsets and is size-checked and counted before nested analysis.
Historical1f 1fpack files keep their description and report unsupported decoding status. - Breaking: Freeze 1.x and gzip 0.5 streams now emit
application/x-freezeand recursively
analyze validated decoded content without external helpers. Native freeze decoding handles
adaptive Huffman rebuilding and its space-filled 4 KiB dictionary. Gzip 0.5 candidates require
valid framing, CRC32, and decoded size before selection; both formats preserve physical fields
and honor optional analysis limits. - Breaking: SCO compress -H (LZH) streams now emit
application/x-sco-compressand recursively
analyze decoded content using the supported installed GNU gzip helper. Physical signature/body
regions and decoded coordinates remain distinct; unavailable helpers and configured limits
report explicit analysis status. - Breaking: UNIX compress (
.Z) payloads are recursively analyzed through an automatically
discovered GNU gzip 1.15 or newer, preserving physical framing and decoded source coordinates.
Missing or incompatible helpers report unsupported status. Optional--helper-timeout/
AnalysisLimits.helper_timeoutlimits each external decoder invocation; helper memory ceilings
use Linux address-space limits and report unsupported status on other platforms when requested. - Optional recursive analysis controls through
AnalysisLimitsand--max-depth,
--max-member-bytes,--max-total-bytes,--max-members, and--max-codec-memory.
Defaults are unlimited; skipped regions report their reason in JSON output. - Native checksum-based TAR detection and recursive analysis of regular TAR members, including
nested archives. Headers, PAX/GNU name extensions, padding, and end markers retain physical
offsets. Links remain metadata, and sparse reconstruction reports unsupported status.
Breaking: valid TAR headers that soft magic misses now reportapplication/x-tar, and
TAR JSON/HTML output includes member structure and nested classifications. Native TAR verdicts
replace generic soft-magic TAR aliases to avoid repeated recursive analysis. Specialized
GPKG verdicts also supersede redundant generic TAR aliases. - Breaking: Gzip members are decoded and recursively analyzed, with validated framing,
checksums, concatenated-member semantics, and optional resource limits. Raw member fields
remain available alongside decoded content.
Changed
- Breaking. Libmagic regex tests use the native
google-re2dependency. Custom definitions
that use unsupported syntax such as lookaround assertions or backreferences produce an error
with the definition filename and line number.
Custom magic compatibility. - Pin PDFminer to the qualified 20260107 release, isolate document-local xref instrumentation,
and qualify future updates through locked dependencies and PDF contract tests. Unqualified
installations retain identification and report unsupported PDF instrumentation. - Breaking: Terminal logging uses Rich 15.0.0 on stderr, with live matcher-pass progress,
a bounded file-type tree during analysis, and a complete tree and summary at completion.
Redirected stderr retains plain warnings, errors, and explicitly requested debug/trace logs;
status and ordinary informational messages are suppressed. Quiet mode suppresses all logging.
MIME descriptions move from interleaved stderr fragments into the match display. Scan scope
and result formats are unchanged. - Modernize handwritten Python annotations for Python 3.11 with native generics, unions, and
subclass-preservingSelfreturns, and enforce modernization rules with Ruff (#3670). - Debugger and command callbacks, generated command subclasses, and command-line output state
have explicit Python 3.11 type contracts while retaining existing behavior (#3673). - Build helper annotations describe compiler results, parser manifests, and optional progress
output while retaining standard-library-only initialization (#3673). - PolyMerge annotations describe trace sources, structured documents, and graph nodes. JSON
inputs use context-managed streams after argument parsing (#3673). - Core Python annotations describe nullable analysis caches, callable parsers, generic matchers,
search and serialization traversal, and native decoder boundaries without changing their
runtime behavior (#3671). - Remove compatibility imports and runtime bases for Python versions older than 3.11.
- Breaking: JSON elements identify their physical or decoded
source; offsets stop at
decoded-buffer boundaries. ZIP and PDF decoded content uses explicitdecoded_contentnodes.
Use(source, offset)to locate a byte in JSON. HTML highlighting, downloads, and coverage use
the selected source, and the viewer renders decoded text as plain text. - Breaking: Require Python 3.11 or later and pin patched filelock 4.2.7 and Pillow 13.0.0.
CI tests Python 3.11 through 3.14 independently so a failing job preserves other results.
Fixed
- Breaking. Reject output paths that alias the input or another output, and report file open
failures as CLI errors. MIME filters honor globs, and each text output honors--max-matches.
Output path checks
· MIME filters
· Text output. - Breaking. Map ZIP members with conflicting local and central directory compression methods
and report the conflict without decoding the member.
ZIP method conflicts. - Breaking: Label NES 2.0 header fields according to their format and report mapper,
memory, timing, console, and expansion device values in structured output (#3860). - Correct the README and development guide to describe implemented 7z parsing and recursive
analysis of supported members. Clarify that future recursive coverage uses format parsers and
exhaustive scanning is not planned (#3780). Thanks to @bfs2021 for submitting an earlier proposed
documentation update in PR #3781. - Breaking: Apply PDF member and decoded-byte limits before stream expansion, including
filter chains and structural xref/object streams. Preserve known physical structure when
decoding is refused, and reuse stream budgets across repeated lookups (#3779). - Breaking: Identify HWP 5.0 from its compound-document
FileHeaderstream with the
standalone libmagic description andapplication/x-hwp. Validate sector chains and stream
bounds before replacing the definitions' OLE-prefixedapplication/hwp+zipresult (#3535). - Breaking: Register relaxed PDF identification independently of lazy PDF parser imports,
restoring PDF matches in binary polyglots and files with long preambles (#3763). - Breaking: DER high-tag-number decoding follows libmagic's 32-bit wraparound, avoiding
quadratic tag arithmetic when recursively analyzing long continuation-byte runs (#3761). - Breaking: PDF byte parsing yields byte and PNG predictor containers before their contents.
Predictor parameters and original-byte structure are traversed once (#3754). - Breaking: Decrypted PDF arrays retain their original container spans, including empty
and nested arrays. Decrypted strings remain in separate decoded sources (#3744). - Breaking: Indirect objects in PDF object streams now appear under their decoded buffer,
preserving token spans, names, arrays, and references. Cached lookups reuse the same source,
including when the catalog itself is compressed (#3743). - Breaking: PDF strings, hex strings, and escaped names retain their complete raw spans.
Non-UTF-8 dictionary names preserve their original bytes, and dictionary pairs, values, and
containing objects map their complete source extents (#3740, #3741, #3746). - Breaking: PDF string conversions retain token spans and render floating-point values
without debug metadata. Byte conversions use native count/error behavior, copying and
pickling preserve wrapper state, and index-protocol selection preserves the selected span
(#3742, #3747, #3748, #3749). - Breaking: HTTP AST spans use original UTF-8 byte positions rather than character
positions, including multibyte headers and bodies within nested or decoded sources (#3712).
Request-start cursors count bytes and reject positions inside a UTF-8 character. - Breaking: Nested BrainFu loops use their immediate parent's source coordinates, keeping
loop and command offsets aligned in bounded stream views (#3711). - Breaking: PDF concatenation retains nonlinear source mappings and combines the full spans
of contiguous instrumented operands, including when a later slice or split consumes the result
(#3666). - Breaking: Prefix concatenation preserves nonlinear PDF token spans. Aligned values retain
the existing inferred positions for native prefixes, and contiguous token prefixes retain
their full recorded source spans (#3666). - Breaking: Text and byte wrappers constructed from numeric PDF tokens retain conservative
whole-token spans instead of inferring an exact byte map from matching value lengths (#3666). - Breaking: PDF codec conversions retain exact byte mappings only when a native codec and
representable input establish one-to-one positions. Lossy and custom conversions keep whole
source spans even when their input and output lengths match. Codec registry changes during
conversion and custom codecs with altered metadata preserve conservative source spans (#3666). - Breaking: PDF token splitting rejects incompatible separator families, accepts contiguous
byte-buffer separators, and checks platform-sized integer overflow formaxsplit(#3666). - Breaking: PDF slices, split parts, and stripped values report bounded source spans.
Byte-aligned selections identify their source interval; nonlinear values without a byte map
retain the whole original token span. Structured selections preserve recorded element spans
(#3666). - Breaking: PDF token splitting respects finite split limits, empty separator fields, and
integer-index protocol arguments without losing unsplit suffixes (#3666). - Breaking: PDF numeric protocols return native values without conversion errors or
deprecation warnings. Useas_int()andas_float()to retain byte provenance on converted
tokens. Text encoding and stripping support built-in defaults (#3666). - Breaking: Dominator forests retain isolated roots, allowing PolyMerge to associate
nonempty file structures with tainted functions that have no callers or callees (#3729). - Breaking: PolyMerge returns an empty association map for an empty PolyFile structure,
including when the program trace contains tainted bytes (#3682). - Breaking: Aborting PDB while stepping through parser matches stops that debugging run
without printing or yielding an absent match (#3678). - Breaking: PolyMerge handles commands without
--diffand compares the functions shared
by each actual input group. Empty legacy traces and function source iteration work;
unavailable optional demangling reports an argument error (#3675). - Breaking: Lazy sequences support negative indexes and slices over their complete input
while retaining lazy positive-index access and the Sequence interface. - Breaking: Loading a serialized multi-sequence search returns its result without printing
or terminating the process, and rejects serialized values that are not search roots. - Breaking: Saved multi-sequence searches serialize trie structure and rebuild derived links
when loaded, so ordinary search tries round-trip without a serialization loop (#3728). - Breaking: Retained search utilities handle empty patterns, generic sequence symbols, and
deep tries without recursive lookup limits. Preserve the originalACNodeconstructor and
five-field cache layout, and document library usage and cache compatibility. Binary encoding
rejects cycles, negative integers, and malformed records with contextual errors instead of
looping, silently corrupting values, or accepting incomplete input (#3728). - Breaking: FileStream seeks use bounded view-relative positions and return the new
position. Negative view origins and lengths are rejected, and past-EOF origins consistently
produce empty views at EOF, including when views are nested (#3664). - Breaking: FileStream integer indexes and unit-step slices normalize negative and omitted
bounds consistently with bytes. Missing integer indexes raiseIndexError(#3664). - Breaking: FileStream accepts readable binary inputs and rejects text streams and destructive
or text path modes before opening a file. Unnamed binary buffers are supported, and public
docstrings specify parent, view, backing-buffer, cursor, and ownership semantics (#3664). - Breaking: PDF, Kaitai, and HTTP AST offsets stay within their sources across nested stream
views and decoded data. Parser debugger locations distinguish backing-file, stream, and
analysis-source coordinates (#3665). - Breaking:
--max-matches 0no longer starts recursive analysis while preparing structured
output. - Breaking: The CLI restores terminal, logging, and signal state after scanning. The new
Analyzer.close()method releases unfinished scan iterators. - Breaking: Profiler pauses return after pausing each parent once, restoring pause/resume
and unprofiled debugger operations (#3727). - Breaking: Brainfuck matching returns successful results after optional arithmetic noop
validation, while retaining cancelling-pair rejection and existing thresholds (#3668). - Breaking: JPEG2000 image previews encode PNG bytes into the data URL without raising
TypeError(#3667). - Breaking: HTTP chunk decoding accepts full hexadecimal sizes and extensions, counts each
decoded byte once, and retains available prefixes with their actual length on incomplete input.
Trailer fields stop at the terminating empty line, so subsequent pipelined request headers
cannot populate or overwrite trailer metadata (#3674). - Bounded file regions now limit reads before allocating data, including
read()and
read(-1), so archive members cannot expose neighboring bytes or copy the remaining archive.
Security
- Pin build and runtime setuptools to 84.2.1, excluding releases affected by the Windows
legacy wheel path escape in GHSA-xmj3-9gf7-h52w (#3752).