Skip to content

PolyFile 1.0.0

Latest

Choose a tag to compare

@ESultanik ESultanik released this 19 Nov 22:18
Immutable release. Only release title and notes can be modified.
35b6e5c

PolyFile 1.0 expands recursive analysis of archives and compressed streams, adds a terminal
explorer, and updates the HTML byte viewer. Structured output identifies the original file or
decoded buffer for each byte range. See the JSON output format.

The tested Python range is 3.11–3.14. On Python 3.15, the pinned google-re2 dependency lacks
compatible wheels, and clean installations fail while building it from source.
Track Python 3.15 installation support.

Added

  • Ship py.typed for Python type checkers and document the public analysis and extension APIs.
  • Breaking. Identify GGUF models as model/x-gguf and map their structure automatically.
    The refreshed Kaitai gallery also adds a TP-Link MiniFS parser and updates the RPM and U-Boot
    uImage parsers. Kaitai gallery update.
  • Breaking: Add --explore FILE, a Textual terminal inspector with source-aware structure,
    virtualized hex and UTF-8 views, byte selection, coverage, navigation, cancellable search, and
    exact byte extraction. Textual is installed by default. Explorer mode opens after analysis and
    requires a named file with stdin, stdout, and stderr connected to an interactive terminal.
  • Breaking: Recursively classify and parse PNG images in ICO files and architecture members
    in supported universal Mach-O binaries. Preserve source coordinates, directory fields, PNG
    previews, and shared analysis limits; report invalid and overlapping member ranges explicitly.
  • Breaking: Analyze 7z archive structure and validated members with py7zr==1.1.4 in
    isolated helpers. Solid/non-solid archives, compressed headers, checksums, decoded source
    coordinates, and optional recursive/helper resource limits have regression coverage.
  • Breaking: HTML reports show embedded image previews beside the ASCII column as you scroll
    their byte ranges. Nearby previews share browsing controls, decoded images identify their encoded
    containers, and narrow layouts use a strip below the bytes. Activate a preview to enlarge it
    or reveal its structure node; the existing tree previews remain available.
  • Breaking: PDF tokens and mapped elements retain source-qualified input dependencies through
    supported arithmetic, conversions, selection, and reconstruction. Computed values use zero-size
    owner anchors, and disjoint or cross-source inputs remain separate. Untagged prefixes no longer
    invent lexical positions. Hex and escaped string payloads recurse in decoded coordinates.
  • Breaking: Qualify PDF instrumentation against pdfminer.six==20260107, isolate private API
    hooks, and report unsupported status for unqualified releases. Preserve raw lexeme extents,
    dictionary names and values, compressed-object sources, encrypted arrays, and wrapper metadata.
  • Document the PDFminer upgrade qualification procedure and supported provenance boundaries.
  • Breaking: Unencrypted LRZIP 0.6 and 0.7 archives are recursively analyzed through an
    installed LRZIP 0.7.3 or newer helper, preserving physical framing and decoded source
    coordinates. Missing helpers and unsupported archive variants report explicit status; declared
    sizes, integrity, optional output limits, and helper resource controls gate recursive analysis.
  • Breaking: Lzip version 1 payloads are recursively analyzed with native header, raw-LZMA,
    CRC32, decoded-size, and member-size validation. Concatenated members preserve physical
    structure and share optional resource limits. Older versions report unsupported status;
    the decoder uses the Python standard library and requires no external helper.
  • Breaking: Decode zlib streams and recursively map validated content, preserving source-aware
    offsets and reporting checksum, dictionary, trailing-data, and resource-limit failures (#3625).
  • Breaking: Modern LZ4 payloads are recursively analyzed after validating frame headers,
    block/content checksums, and concatenated/skippable frames. The required lz4==4.4.5 backend
    honors optional output, frame, depth, and native-buffer limits. Legacy frames and declared
    external dictionaries report unsupported status.
  • Breaking: Zstandard payloads are recursively analyzed after validating concatenated and
    skippable frames. Optional output, frame, depth, and decoder-window limits preserve explicit
    failure status. Python 3.11 through 3.13 requires backports.zstd==1.8.0; Python 3.14 uses its
    standard-library backend. External dictionaries and historical frames report unsupported status.
  • Breaking: Bzip2 streams are decoded and recursively analyzed using Python's standard
    library. Concatenated streams retain their physical framing and produce one validated decoded
    buffer; integrity errors, truncation, and configured resource limits report explicit status.
  • Breaking: Identify SIMH tape images with native record checks, map their records and
    markers, and recursively analyze independently recognizable record payloads (#3619).
  • Breaking: ELF identification descriptions now include validated linkage, interpreter,
    GNU/Go build IDs, ABI versions, stripping/debug information, and supported core-file details.
    The command line, Python identification API, and SBuD root values share these descriptions;
    malformed metadata retains the ELF classification and reports a diagnostic.
  • Breaking: LZMA-alone files are decoded and recursively analyzed using the standard
    library, preserving their physical header and decoded coordinate source. Known and streamed
    sizes, supported properties, malformed endings, and optional resource limits are validated.
  • Breaking: XZ streams are decoded and recursively analyzed, with complete integrity checks,
    concatenated-stream and four-byte padding semantics, source-aware archive recursion, and
    optional native memory, decoded output, depth, and stream-count limits.
  • Breaking: UNIX pack streams now emit application/x-unix-pack and recursively analyze
    decoded content through a bounded native canonical Huffman decoder. Each concatenated member
    retains physical header/tree/body offsets and is size-checked and counted before nested analysis.
    Historical 1f 1f pack files keep their description and report unsupported decoding status.
  • Breaking: Freeze 1.x and gzip 0.5 streams now emit application/x-freeze and recursively
    analyze validated decoded content without external helpers. Native freeze decoding handles
    adaptive Huffman rebuilding and its space-filled 4 KiB dictionary. Gzip 0.5 candidates require
    valid framing, CRC32, and decoded size before selection; both formats preserve physical fields
    and honor optional analysis limits.
  • Breaking: SCO compress -H (LZH) streams now emit application/x-sco-compress and recursively
    analyze decoded content using the supported installed GNU gzip helper. Physical signature/body
    regions and decoded coordinates remain distinct; unavailable helpers and configured limits
    report explicit analysis status.
  • Breaking: UNIX compress (.Z) payloads are recursively analyzed through an automatically
    discovered GNU gzip 1.15 or newer, preserving physical framing and decoded source coordinates.
    Missing or incompatible helpers report unsupported status. Optional --helper-timeout /
    AnalysisLimits.helper_timeout limits each external decoder invocation; helper memory ceilings
    use Linux address-space limits and report unsupported status on other platforms when requested.
  • Optional recursive analysis controls through AnalysisLimits and --max-depth,
    --max-member-bytes, --max-total-bytes, --max-members, and --max-codec-memory.
    Defaults are unlimited; skipped regions report their reason in JSON output.
  • Native checksum-based TAR detection and recursive analysis of regular TAR members, including
    nested archives. Headers, PAX/GNU name extensions, padding, and end markers retain physical
    offsets. Links remain metadata, and sparse reconstruction reports unsupported status.
    Breaking: valid TAR headers that soft magic misses now report application/x-tar, and
    TAR JSON/HTML output includes member structure and nested classifications. Native TAR verdicts
    replace generic soft-magic TAR aliases to avoid repeated recursive analysis. Specialized
    GPKG verdicts also supersede redundant generic TAR aliases.
  • Breaking: Gzip members are decoded and recursively analyzed, with validated framing,
    checksums, concatenated-member semantics, and optional resource limits. Raw member fields
    remain available alongside decoded content.

Changed

  • Breaking. Libmagic regex tests use the native google-re2 dependency. Custom definitions
    that use unsupported syntax such as lookaround assertions or backreferences produce an error
    with the definition filename and line number.
    Custom magic compatibility.
  • Pin PDFminer to the qualified 20260107 release, isolate document-local xref instrumentation,
    and qualify future updates through locked dependencies and PDF contract tests. Unqualified
    installations retain identification and report unsupported PDF instrumentation.
  • Breaking: Terminal logging uses Rich 15.0.0 on stderr, with live matcher-pass progress,
    a bounded file-type tree during analysis, and a complete tree and summary at completion.
    Redirected stderr retains plain warnings, errors, and explicitly requested debug/trace logs;
    status and ordinary informational messages are suppressed. Quiet mode suppresses all logging.
    MIME descriptions move from interleaved stderr fragments into the match display. Scan scope
    and result formats are unchanged.
  • Modernize handwritten Python annotations for Python 3.11 with native generics, unions, and
    subclass-preserving Self returns, and enforce modernization rules with Ruff (#3670).
  • Debugger and command callbacks, generated command subclasses, and command-line output state
    have explicit Python 3.11 type contracts while retaining existing behavior (#3673).
  • Build helper annotations describe compiler results, parser manifests, and optional progress
    output while retaining standard-library-only initialization (#3673).
  • PolyMerge annotations describe trace sources, structured documents, and graph nodes. JSON
    inputs use context-managed streams after argument parsing (#3673).
  • Core Python annotations describe nullable analysis caches, callable parsers, generic matchers,
    search and serialization traversal, and native decoder boundaries without changing their
    runtime behavior (#3671).
  • Remove compatibility imports and runtime bases for Python versions older than 3.11.
  • Breaking: JSON elements identify their physical or decoded source; offsets stop at
    decoded-buffer boundaries. ZIP and PDF decoded content uses explicit decoded_content nodes.
    Use (source, offset) to locate a byte in JSON. HTML highlighting, downloads, and coverage use
    the selected source, and the viewer renders decoded text as plain text.
  • Breaking: Require Python 3.11 or later and pin patched filelock 4.2.7 and Pillow 13.0.0.
    CI tests Python 3.11 through 3.14 independently so a failing job preserves other results.

Fixed

  • Breaking. Reject output paths that alias the input or another output, and report file open
    failures as CLI errors. MIME filters honor globs, and each text output honors --max-matches.
    Output path checks
    · MIME filters
    · Text output.
  • Breaking. Map ZIP members with conflicting local and central directory compression methods
    and report the conflict without decoding the member.
    ZIP method conflicts.
  • Breaking: Label NES 2.0 header fields according to their format and report mapper,
    memory, timing, console, and expansion device values in structured output (#3860).
  • Correct the README and development guide to describe implemented 7z parsing and recursive
    analysis of supported members. Clarify that future recursive coverage uses format parsers and
    exhaustive scanning is not planned (#3780). Thanks to @bfs2021 for submitting an earlier proposed
    documentation update in PR #3781.
  • Breaking: Apply PDF member and decoded-byte limits before stream expansion, including
    filter chains and structural xref/object streams. Preserve known physical structure when
    decoding is refused, and reuse stream budgets across repeated lookups (#3779).
  • Breaking: Identify HWP 5.0 from its compound-document FileHeader stream with the
    standalone libmagic description and application/x-hwp. Validate sector chains and stream
    bounds before replacing the definitions' OLE-prefixed application/hwp+zip result (#3535).
  • Breaking: Register relaxed PDF identification independently of lazy PDF parser imports,
    restoring PDF matches in binary polyglots and files with long preambles (#3763).
  • Breaking: DER high-tag-number decoding follows libmagic's 32-bit wraparound, avoiding
    quadratic tag arithmetic when recursively analyzing long continuation-byte runs (#3761).
  • Breaking: PDF byte parsing yields byte and PNG predictor containers before their contents.
    Predictor parameters and original-byte structure are traversed once (#3754).
  • Breaking: Decrypted PDF arrays retain their original container spans, including empty
    and nested arrays. Decrypted strings remain in separate decoded sources (#3744).
  • Breaking: Indirect objects in PDF object streams now appear under their decoded buffer,
    preserving token spans, names, arrays, and references. Cached lookups reuse the same source,
    including when the catalog itself is compressed (#3743).
  • Breaking: PDF strings, hex strings, and escaped names retain their complete raw spans.
    Non-UTF-8 dictionary names preserve their original bytes, and dictionary pairs, values, and
    containing objects map their complete source extents (#3740, #3741, #3746).
  • Breaking: PDF string conversions retain token spans and render floating-point values
    without debug metadata. Byte conversions use native count/error behavior, copying and
    pickling preserve wrapper state, and index-protocol selection preserves the selected span
    (#3742, #3747, #3748, #3749).
  • Breaking: HTTP AST spans use original UTF-8 byte positions rather than character
    positions, including multibyte headers and bodies within nested or decoded sources (#3712).
    Request-start cursors count bytes and reject positions inside a UTF-8 character.
  • Breaking: Nested BrainFu loops use their immediate parent's source coordinates, keeping
    loop and command offsets aligned in bounded stream views (#3711).
  • Breaking: PDF concatenation retains nonlinear source mappings and combines the full spans
    of contiguous instrumented operands, including when a later slice or split consumes the result
    (#3666).
  • Breaking: Prefix concatenation preserves nonlinear PDF token spans. Aligned values retain
    the existing inferred positions for native prefixes, and contiguous token prefixes retain
    their full recorded source spans (#3666).
  • Breaking: Text and byte wrappers constructed from numeric PDF tokens retain conservative
    whole-token spans instead of inferring an exact byte map from matching value lengths (#3666).
  • Breaking: PDF codec conversions retain exact byte mappings only when a native codec and
    representable input establish one-to-one positions. Lossy and custom conversions keep whole
    source spans even when their input and output lengths match. Codec registry changes during
    conversion and custom codecs with altered metadata preserve conservative source spans (#3666).
  • Breaking: PDF token splitting rejects incompatible separator families, accepts contiguous
    byte-buffer separators, and checks platform-sized integer overflow for maxsplit (#3666).
  • Breaking: PDF slices, split parts, and stripped values report bounded source spans.
    Byte-aligned selections identify their source interval; nonlinear values without a byte map
    retain the whole original token span. Structured selections preserve recorded element spans
    (#3666).
  • Breaking: PDF token splitting respects finite split limits, empty separator fields, and
    integer-index protocol arguments without losing unsplit suffixes (#3666).
  • Breaking: PDF numeric protocols return native values without conversion errors or
    deprecation warnings. Use as_int() and as_float() to retain byte provenance on converted
    tokens. Text encoding and stripping support built-in defaults (#3666).
  • Breaking: Dominator forests retain isolated roots, allowing PolyMerge to associate
    nonempty file structures with tainted functions that have no callers or callees (#3729).
  • Breaking: PolyMerge returns an empty association map for an empty PolyFile structure,
    including when the program trace contains tainted bytes (#3682).
  • Breaking: Aborting PDB while stepping through parser matches stops that debugging run
    without printing or yielding an absent match (#3678).
  • Breaking: PolyMerge handles commands without --diff and compares the functions shared
    by each actual input group. Empty legacy traces and function source iteration work;
    unavailable optional demangling reports an argument error (#3675).
  • Breaking: Lazy sequences support negative indexes and slices over their complete input
    while retaining lazy positive-index access and the Sequence interface.
  • Breaking: Loading a serialized multi-sequence search returns its result without printing
    or terminating the process, and rejects serialized values that are not search roots.
  • Breaking: Saved multi-sequence searches serialize trie structure and rebuild derived links
    when loaded, so ordinary search tries round-trip without a serialization loop (#3728).
  • Breaking: Retained search utilities handle empty patterns, generic sequence symbols, and
    deep tries without recursive lookup limits. Preserve the original ACNode constructor and
    five-field cache layout, and document library usage and cache compatibility. Binary encoding
    rejects cycles, negative integers, and malformed records with contextual errors instead of
    looping, silently corrupting values, or accepting incomplete input (#3728).
  • Breaking: FileStream seeks use bounded view-relative positions and return the new
    position. Negative view origins and lengths are rejected, and past-EOF origins consistently
    produce empty views at EOF, including when views are nested (#3664).
  • Breaking: FileStream integer indexes and unit-step slices normalize negative and omitted
    bounds consistently with bytes. Missing integer indexes raise IndexError (#3664).
  • Breaking: FileStream accepts readable binary inputs and rejects text streams and destructive
    or text path modes before opening a file. Unnamed binary buffers are supported, and public
    docstrings specify parent, view, backing-buffer, cursor, and ownership semantics (#3664).
  • Breaking: PDF, Kaitai, and HTTP AST offsets stay within their sources across nested stream
    views and decoded data. Parser debugger locations distinguish backing-file, stream, and
    analysis-source coordinates (#3665).
  • Breaking: --max-matches 0 no longer starts recursive analysis while preparing structured
    output.
  • Breaking: The CLI restores terminal, logging, and signal state after scanning. The new
    Analyzer.close() method releases unfinished scan iterators.
  • Breaking: Profiler pauses return after pausing each parent once, restoring pause/resume
    and unprofiled debugger operations (#3727).
  • Breaking: Brainfuck matching returns successful results after optional arithmetic noop
    validation, while retaining cancelling-pair rejection and existing thresholds (#3668).
  • Breaking: JPEG2000 image previews encode PNG bytes into the data URL without raising
    TypeError (#3667).
  • Breaking: HTTP chunk decoding accepts full hexadecimal sizes and extensions, counts each
    decoded byte once, and retains available prefixes with their actual length on incomplete input.
    Trailer fields stop at the terminating empty line, so subsequent pipelined request headers
    cannot populate or overwrite trailer metadata (#3674).
  • Bounded file regions now limit reads before allocating data, including read() and
    read(-1), so archive members cannot expose neighboring bytes or copy the remaining archive.

Security

  • Pin build and runtime setuptools to 84.2.1, excluding releases affected by the Windows
    legacy wheel path escape in GHSA-xmj3-9gf7-h52w (#3752).