Architecture and Design

Architecture and Design

This page describes how RE-Toolkit is built and why. It covers the component model, the data flow through a run, the stage contract, the dispatch and scoring models, and the design decisions that a maintainer would otherwise have to reconstruct by reading the source.

Contents

• Design principles
• Component model
• Run lifecycle
• Type detection and dispatch
• The stage contract
• Stage numbering versus execution order
• Tool execution model
• Summary synthesis and scoring
• Aggregation layer
• Installer architecture
• Extension points
• Design decisions

Design principles

Separate provisioning from analysis. The installer and the analyzer are independent programs with independent lifecycles. A workstation is provisioned rarely and analysis runs constantly. Coupling them would mean a tool-inventory change forced an analyzer release, and an analyzer fix required re-provisioning.

One stage, one file, one function. Every analysis stage is a single file defining a single function. A stage can be read, tested, reasoned about, and disabled in isolation. Adding analysis capability means adding a file and one dispatch line, not editing a monolith.

Structured findings, not files on disk. A tool that writes output nobody parses has not contributed anything. Stages produce files, but the deliverable is the structured findings those files feed into _summary.json. The HTML report renders that JSON; it is not an independent source of truth.

Explainability over scoring. A severity number without justification cannot be acted on or disputed. Every signal contributing to a verdict is recorded with its name, weight, and supporting evidence, so any verdict can be traced to the specific observation that produced it.

Assume the target is hostile. Input is sandboxed before anything touches it, tool invocations are bounded, and nothing executes the target unless the operator explicitly opts into a dynamic tier.

Fail visibly, continue anyway. One tool failing must not end a run. Failures are captured, recorded in the ledger, and surfaced in the report, while the remaining stages proceed. The installer follows the same rule: it deliberately does not use set -e, so a single package failure cannot abort provisioning.

Component model

graph TB
    subgraph provision["Provisioning, run rarely"]
        INST["install-retoolkit.sh<br/>Layered provisioner"]
        INST --> APT["System packages"]
        INST --> VENV["Python virtual environment"]
        INST --> GHID["Ghidra"]
        INST --> RULES["capa and YARA rules"]
    end

    subgraph runtime["Analysis, run constantly"]
        DRV["analyze-binaries.sh<br/>Driver and CLI"]

        subgraph libs["lib/ shared modules"]
            COM["common.sh<br/>Logging, paths, sandbox"]
            TR["tool-runner.sh<br/>Bounded execution, ledger"]
            DT["detect-type.sh<br/>Type detection"]
            DISP["dispatch.sh<br/>Pipeline orchestration"]
            AGG["aggregate.sh<br/>Cross-target rollup"]
            GH["ghidra-helper.sh<br/>Ghidra discovery"]
            VIZ["viz-helper.sh<br/>Visualization emitters"]
        end

        STG["stages/static/<br/>46 stage files"]
        GD["GhidraDump.py<br/>Ghidra postscript"]
    end

    subgraph out["Artifacts"]
        SUM["_summary.json"]
        REP["_report.html"]
        IDX["index.html"]
    end

    DRV --> COM
    DRV --> DT
    DRV --> DISP
    DISP --> STG
    STG --> TR
    STG --> GD
    TR --> APT
    TR --> VENV
    GH --> GHID
    STG --> SUM
    SUM --> REP
    AGG --> IDX
    VIZ --> REP

Loading

Responsibilities are deliberately narrow:

Component	Owns	Does not own
analyze-binaries.sh	Argument parsing, run setup, target iteration	Tool invocation, type logic
common.sh	Logging, path handling, sandboxing	Anything tool-specific
tool-runner.sh	Timeouts, capture, ledger, output validation	Which tool to run
detect-type.sh	Classifying a file	What to do about the classification
dispatch.sh	Which stages run, in what order	How any stage works
stages/static/*.sh	One analysis concern each	Orchestration, aggregation
aggregate.sh	Cross-target synthesis	Per-target analysis

The rule that keeps this honest: a stage never decides whether it should run. The dispatcher decides. A stage only decides how to do its job when asked.

Run lifecycle

sequenceDiagram
    participant Op as Operator
    participant Drv as Driver
    participant Com as common.sh
    participant Det as detect-type.sh
    participant Dis as dispatch.sh
    participant Stg as Stages
    participant Agg as aggregate.sh

    Op->>Drv: analyze-binaries.sh -t target -o out
    Drv->>Drv: Parse arguments, resolve skip controls
    Drv->>Com: Set up logging and output root
    Drv->>Drv: Enumerate targets

    loop For each target
        Drv->>Com: prepare_sandboxed_target()
        Com-->>Drv: Path to sandboxed copy
        Drv->>Det: detect_type(sandboxed copy)
        Det-->>Drv: Primary type and runtime
        Drv->>Dis: analyze_one(target, outdir)

        Dis->>Stg: Universal triage
        Dis->>Stg: Type-specific static stages
        Dis->>Stg: Cross-cutting static stages
        opt Dynamic enabled
            Dis->>Stg: Dynamic tiers 1 to 4
        end
        Dis->>Stg: Summary synthesis
        Dis->>Stg: Visualization
        Dis->>Stg: Report rendering

        Drv->>Com: verify_input_untouched()
    end

    Drv->>Agg: write_run_json_and_index()
    Agg-->>Op: index.html and per-target reports

Loading

Two steps in that sequence are safety-critical and bracket everything else. prepare_sandboxed_target() runs before any analysis, and verify_input_untouched() runs after, proving the original was not modified.

Type detection and dispatch

Detection is ordered so more specific signatures win. A UPX-packed PE matches both the UPX signature and the generic PE signature, so UPX is tested first.

flowchart TD
    START([Target file]) --> UPX{UPX packed?}
    UPX -->|Yes| RUPX[upx-packed]
    UPX -->|No| PE{PE?}
    PE -->|Yes| CLR{CLR header?}
    CLR -->|Yes| RNET[pe-dotnet]
    CLR -->|No| RPE[pe-native]
    PE -->|No| ELF{ELF?}
    ELF -->|Yes| RELF[elf]
    ELF -->|No| OTHER{Other signature}
    OTHER --> RMACH[macho]
    OTHER --> RWASM[wasm]
    OTHER --> RPYC[pyc]
    OTHER --> RJAR[jar]
    OTHER --> RPDF[pdf]
    OTHER --> ROLE[ole]
    OTHER --> RAPK[apk]
    OTHER --> RDEX[dex]
    OTHER --> RCFG[config-xml]
    OTHER --> RUNK[unknown]

    RPE --> SUB{Runtime probe}
    RELF --> SUB
    RMACH --> SUB
    SUB -->|Go| GO[Go handling added]
    SUB -->|Rust| RUST[Rust handling added]
    SUB -->|Neither| PLAIN[Primary type only]

Loading

Runtime sub-classification composes with the primary type instead of replacing it. A Go ELF receives the full ELF stage set plus Go-specific handling, because it is still an ELF and every ELF question remains valid.

upx-packed is a transient classification. The UPX stage unpacks the image and the unpacked result is re-analyzed under its real type, so a packed PE is ultimately analyzed as a PE.

Each primary type maps to a dispatch branch:

Type	Representative stages
pe-native	Triage, PE, LIEF, pev, Manalyze, peframe, Ghidra, disassembly
pe-dotnet	PE stages plus .NET disassembly, decompilation, deobfuscation
elf	Triage, ELF, LIEF, Ghidra, disassembly, hardening checks
macho	Triage, Mach-O, LIEF, Ghidra, disassembly
wasm	Triage, WebAssembly validation, disassembly, decompilation
pyc	Triage, multi-decompiler bytecode recovery
jar	Triage, archive listing, Java decompilation
pdf	Triage, PDF structure and active content
ole	Triage, OLE and OOXML macro extraction
apk	Triage, container extraction, manifest, signature, DEX
dex	Triage, DEX decompilation
config-xml	Triage, structural inspection
unknown	Triage and format-agnostic stages only

Cross-cutting stages run for every type: IOC extraction, fuzzy hashing, cryptographic key extraction, summary, visualization, and reporting.

The stage contract

Every stage file honors the same contract, which is what makes stages interchangeable and independently testable.

A stage:

1. Accepts $1 as the sandboxed target path and $2 as the output directory.
2. Creates its own output subdirectory and writes only inside it.
3. Invokes tools exclusively through run_tool or run_shell.
4. Honors its skip control, checked by the driver or the stage itself.
5. Skips cleanly with a logged reason when a required tool is unavailable.
6. Returns rather than calling exit, leaving the driver in control.
7. Never modifies the target.

The contract is what allows the dispatcher to treat 46 heterogeneous analyses uniformly. Violating item 6 is the most damaging failure mode, because a stage calling exit terminates the whole run and discards every other target.

Stage numbering versus execution order

Stage filenames are numbered by where their output belongs in a directory listing, not by when they run. Summary (85), visualization (89), and report (90) carry lower numbers than the dynamic stages (92 through 98) but execute after them.

flowchart LR
    subgraph filesystem["Filename order, how output reads"]
        F1["00-triage"] --> F2["10-pe"] --> F3["30-ghidra"]
        F3 --> F4["85-summary"] --> F5["89-viz"] --> F6["90-report"]
        F6 --> F7["92-98 dynamic"]
    end

    subgraph execution["Execution order, how data flows"]
        E1["00-triage"] --> E2["Type-specific static"]
        E2 --> E3["Cross-cutting static"]
        E3 --> E4["92-98 dynamic"]
        E4 --> E5["85-summary"]
        E5 --> E6["89-viz"]
        E6 --> E7["90-report"]
    end

Loading

The reason is data dependency. Summary synthesis needs every upstream result, static and dynamic, before it can compute a verdict. Visualization reads the summary. The report consumes both. So those three must run last regardless of their numbers.

The numbers stay low because an analyst opening the output directory should see triage first, tool output in the middle, and the summary, visualization, and report grouped where attention naturally lands.

This separation is deliberate but not self-evident. A run log showing stages 85, 89, and 90 executing after 92 through 98 looks like a misordering bug and has been reported as one. It is correct behavior.

Tool execution model

No stage invokes a tool directly. Everything goes through tool-runner.sh, which provides four guarantees a raw invocation cannot.

flowchart TD
    S["Stage requests a tool"] --> AVAIL{Tool installed?}
    AVAIL -->|No| SKIP["Log skip reason<br/>Record in ledger<br/>Return cleanly"]
    AVAIL -->|Yes| RUN["Execute under timeout"]
    RUN --> TIMEOUT{Timed out?}
    TIMEOUT -->|Yes| KILL["Terminate<br/>Record timeout"]
    TIMEOUT -->|No| EXIT{Exit status}
    EXIT -->|Non-zero| FAIL["Capture stderr<br/>Record failure"]
    EXIT -->|Zero| SHAPE{Output shape valid?}
    SHAPE -->|No| SILENT["Record silent failure"]
    SHAPE -->|Yes| OK["Record success"]
    KILL --> LEDGER["Run ledger"]
    FAIL --> LEDGER
    SILENT --> LEDGER
    OK --> LEDGER
    SKIP --> LEDGER

Loading

Bounded execution. Every invocation has a timeout, so a malformed or hostile input cannot hang a run indefinitely.

Nothing suppressed. stdout and stderr are captured to per-tool logs rather than discarded. A failure remains diagnosable after the run.

The ledger. Each invocation is recorded with its outcome, which lets the report state which tools ran, which were skipped, and why, instead of leaving the analyst to infer coverage from which files happen to exist.

Output-shape validation. This addresses the failure mode that ordinary error handling misses: a tool that exits zero while producing unusable output. A real example is an invocation using a flag the installed version does not recognize, which prints usage text to stdout and exits successfully. The exit status says success, the output file exists and is non-empty, and the parser silently extracts nothing. Shape validation catches that class by checking output against what the parser expects.

Summary synthesis and scoring

Summary synthesis reads every stage's output and produces _summary.json, the authoritative artifact for a target. Severity uses a weighted, explainable model rather than simple accumulation.

Each signal contributes a named tuple:

Field	Meaning
name	Stable identifier for the signal
weight	Points contributed to the risk score
evidence	The specific observation supporting it

Total score maps to a band:

Band	Score
Critical	100 and above
High	60 to 99
Medium	30 to 59
Low	10 to 29
Informational	Below 10

Two properties matter more than the numbers.

Composite suppression. Related signals describing the same underlying observation are not allowed to compound. When a composite indicator fires, the narrower signals it subsumes are suppressed, so a single concept such as "packed" cannot inflate a score by being counted several ways.

Evidence ordering. Signals are sorted by weight when rendered, so the explanation of a verdict leads with what drove it rather than with whichever check happened to run first.

Because the scoring logic lives inside a bash heredoc and cannot be imported, it is covered by a golden-sample regression test that deliberately mirrors the implementation, including exact band boundaries. Changing a weight or threshold requires updating the mirror in the same commit.

Aggregation layer

Per-target analysis produces one _summary.json per binary. Aggregation works a level above, synthesizing artifacts that only exist across a run:

Artifact	Purpose
_run.json	Run manifest: what was analyzed, with which toolchain
index.html	Codebase-wide index across all targets
Similarity matrix	Pairwise comparison from per-target fuzzy hashes
Cluster graph	Related samples in a batch, rendered as a graph
Threat intelligence export	Indicators in portable formats

Similarity is where batch analysis pays off: fuzzy hashes computed per target become meaningful only when compared, which cannot happen until every target has been processed.

Installer architecture

The installer provisions in numbered layers, each independently skippable and individually logged to /var/log/retoolkit/.

flowchart TD
    L0["LAYER 0<br/>Analyzer source to /opt/retoolkit"]
    L1["LAYER 1<br/>System packages via apt"]
    L2["LAYER 2<br/>.NET SDK"]
    L2B["LAYER 2B to 2G<br/>Vendor and release tools"]
    L2H["LAYER 2H<br/>Source-build fallbacks"]
    L3["LAYER 3<br/>Python virtual environment"]
    L4["LAYER 4<br/>Ghidra"]
    L4B["LAYER 4B to 4E<br/>Opt-in analysis tools"]
    L5["LAYER 5<br/>capa and YARA rules"]
    L8["LAYER 8 to 11<br/>Dynamic tiers, opt-in"]
    L12["LAYER 12<br/>Post-install verification"]

    L0 --> L1 --> L2 --> L2B --> L2H --> L3 --> L4 --> L4B --> L5 --> L8 --> L12
    L1 -.failed packages.-> L2H

Loading

Three properties define the design:

apt first, source second. Distribution packages are preferred. Source builds exist as a fallback for tools that a rolling distribution may not carry at a given moment, which is why LAYER 2H runs after LAYER 1 and specifically attempts recovery of packages apt could not resolve. An apt miss is therefore not necessarily a failure, and the installer does not report it as one until after LAYER 2H has had its chance.

Idempotent. Existing installs are detected and skipped, so re-running is safe and is the normal way to repair a partial install.

No set -e. This is deliberate. The installer must survive an individual package failure, record it, and continue, then report everything unresolved in a verification table at the end. Aborting on first failure would leave a half-provisioned system and no summary of what went wrong.

Layers 6 and 7 are intentionally unused. Layer 6 was renumbered and layer 7 was reserved and never allocated. The gap is preserved rather than closed, because renumbering would invalidate every log and document referring to a layer by number.

Extension points

To add	Touch
An analysis stage	New file in stages/static/, one line in dispatch.sh, a skip control in the driver, a consumer in 85-summary.sh
A tool to an existing stage	The stage file, via run_tool
A file type	detect-type.sh for detection, dispatch.sh for a branch, stages for handling
A scoring signal	85-summary.sh via add_signal, plus the mirror in the scoring test
A visualization	89-viz.sh, using emitters from viz-helper.sh
An installer tool	The appropriate layer, plus the verification table

Step-by-step instructions are in Development.

Design decisions

Bash for orchestration, Python for analysis. Orchestration is process management, and shell is a good fit for that on a platform where every analysis tool is a command-line program. Parsing and synthesis are data work, so they happen in Python, embedded as heredocs where they need stage context. The tradeoff is real: heredoc-embedded Python cannot be imported or unit tested directly, which is why the scoring model is covered by a deliberate mirror test.

Sandbox by copy rather than by permission. Making the original read-only would be weaker: some tools fail confusingly on read-only input, and a sufficiently privileged process could restore write access. Copying is unconditional and cheap relative to analysis cost, and it holds even against a future tool with an undiscovered destructive flag, because that tool can only reach the copy.

Static analysis is always the foundation. There is no dynamic-only mode. Dynamic stages cross-reference the strings, imports, signatures, and indicators that static analysis produces, so running dynamic alone would degrade the dynamic results themselves.

Self-contained output. Reports and visualizations use inline SVG and inline styling with no external references. An analyst is frequently working on an isolated network, and a report that requires fetching a script from a CDN is a report that does not render where it is most needed. It also means viewing a report generates no network traffic that could signal analysis to an adversary.

Per-version history is not kept in source. Header blocks describe current behavior. Release history lives in CHANGELOG.md. Embedded changelogs in source headers grew to over a thousand lines and pushed the actual documentation below the point where anyone would read it.