Pipeline architecture by 416rehman · Pull Request #2 · 416rehman/DeepZero

416rehman · 2026-04-16T04:45:58Z

No description provided.

…elper, add exit codes - fix B110: replace bare except:pass with specific exceptions + debug logging - fix B104: default API host 0.0.0.0 -> 127.0.0.1 - fix B324: add usedforsecurity=False to MD5 hash - fix B701: add jinja2 autoescape - extract _build_runner helper to deduplicate CLI setup - add SystemExit(1) to all CLI error paths - remove 19 unused imports across 10 files - strict score: 18.7 -> 70.9

…y types in stage/runner/cli

… (loldrivers_filter, pe_ingest) - test_providers_llm: LLMProvider properties, completion, retry, rate limiting, token errors, import guard - test_providers_decompiler: analyzeHeadless finder, cache hit/corrupt, ghidra execution - test_loldrivers_filter: DB loading, process logic, skip/pass verdicts - test_pe_ingest: directory discovery, extensions, recursion, subdirs, metadata - also: add _load_env helper for dotenv ImportError guard in cli.py

… fix pe_ingest B110 - test_stages_llm: 13 tests covering process flow, caching, classification, template vars, artifact loading - test_ghidra_decompile: 7 tests covering error paths, successful/failed decompilation, cache skip - fix imphash extraction B110: add debug logging and noqa annotation - total: 129 tests (up from 74)

…tages_llm.py

…inner, add --model to resume - extract run_subprocess_with_kill and kill_process_tree to engine/process.py (design coherence) - rename _run_inner to _execute_pipeline_stages (naming quality) - add --model/-m option to resume command for API coherence with run - add network exposure warning when serve --host is non-localhost

…les without suppressions

…test suite

…state abstractions

…essions

… legacy ghidra_runner.py

…tive asyncio execution

…eouts

…limits

…chitecture and workflow execution schema

…scope correctly

…eam repository config defaults

…gainst Github Actions environment containers

Copilot

Pull request overview

This PR introduces a new “DeepZero” pipeline architecture centered around filesystem-backed state (“state ledger”), adds built-in + external processors for driver-analysis workflows, and replaces/removes the previous byovd_agent implementation.

Changes:

Add core engine modules (pipeline loading/validation, runner, registry, state store, process utilities, UI) and built-in processors (ingest/filter/sort/top-k/command/LLM).
Add external processors for Ghidra decompilation, LOLDrivers filtering, and bulk Semgrep scanning + a reference pipelines/loldrivers pipeline.
Add extensive test coverage, CI workflow, packaging/metadata updates, and remove legacy byovd_agent code.

Reviewed changes

Copilot reviewed 81 out of 88 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_state.py	Adds tests for namespaced sample state + state store persistence/atomic writes.
tests/test_stages_llm.py	Adds tests for `GenericLLM` processing, caching, template vars, classification.
tests/test_stage_sort.py	Adds a basic unit test for `Sort` reduce processor.
tests/test_stage.py	Tests processor resolution, base classes, dataclasses, registry behavior.
tests/test_sort.py	Adds more complete tests for sorting behavior + validation edge cases.
tests/test_semgrep_scanner.py	Smoke test for Semgrep scanner processor construction.
tests/test_runner.py	Tests `PipelineRunner` execution, resume, failure isolation, parallel map, limits.
tests/test_registry.py	Tests `engine.registry` registry + resolving built-ins and file processors.
tests/test_providers_llm.py	Tests `LLMProvider` model parsing and retry/backoff behavior via mocks.
tests/test_process.py	Tests subprocess execution with timeout and cross-platform kill behavior.
tests/test_pipeline_e2e.py	End-to-end tests for ingest/map/batch/reduce interactions and caching/abort flows.
tests/test_pipeline.py	Tests pipeline YAML loading/validation behavior (env vars, ordering, work dir).
tests/test_pe_ingest.py	Tests external PE ingest processor behavior (hashes, recursion, filters).
tests/test_loldrivers_filter.py	Tests LOLDrivers filter DB loading + filtering by SHA256.
tests/test_ghidra_decompile.py	Tests external Ghidra decompile processor validation, caching, and execution paths.
tests/test_extract_dispatch.py	Tests Ghidra post-script helper extraction with fully mocked Ghidra modules.
tests/test_engine_ui.py	Tests `PipelineDashboard` stats and layout generation.
tests/test_engine_registry.py	Additional registry tests for register/resolve behavior.
tests/test_engine_pipeline.py	Tests `load_pipeline`/`validate_pipeline` APIs.
tests/test_engine_llm.py	Basic tests for `LLMProvider` init/complete/import guard.
tests/test_engine_context.py	Tests `generate_context()` markdown output/truncation and artifacts listing.
tests/test_context.py	Additional context generation test using `SampleState` history.
tests/test_cli.py	Tests CLI command registration/help and basic behaviors for validate/init/status/run/resume/serve.
tests/test_builtin_ingest.py	Tests built-in `FileDiscovery` ingest behavior.
tests/test_builtin_filters.py	Tests built-in `MetadataFilter`, `HashExclude`, and `TopKSelector`.
tests/test_builtin_external.py	Tests built-in `GenericCommand` behavior using a mocked async runner.
tests/test_api_server.py	Tests Starlette API endpoints against filesystem state store.
src/deepzero/stages/top_k.py	Adds `TopKSelector` reduce processor.
src/deepzero/stages/sort.py	Adds `Sort` reduce processor.
src/deepzero/stages/llm.py	Adds `GenericLLM` map processor (jinja prompts, caching, classification, artifact context).
src/deepzero/stages/ingest.py	Adds `FileDiscovery` ingest processor.
src/deepzero/stages/hash_filter.py	Adds `HashExclude` map processor.
src/deepzero/stages/filter.py	Adds `MetadataFilter` map processor.
src/deepzero/stages/command.py	Adds `GenericCommand` map processor for running external commands.
src/deepzero/stages/init.py	Registers built-in processors in the registry.
src/deepzero/engine/ui.py	Adds Rich-based pipeline dashboard TUI.
src/deepzero/engine/types.py	Adds shared enums for verdict/status types.
src/deepzero/engine/registry.py	Adds processor registry + resolution (built-ins, processors/ file refs, dotted refs).
src/deepzero/engine/process.py	Adds subprocess runner with timeout + process-tree killing.
src/deepzero/engine/pipeline.py	Adds pipeline definition/YAML loader, env-var expansion, processor validation.
src/deepzero/engine/llm.py	Adds `LLMProvider` wrapper with adaptive retry/backoff and import guards.
src/deepzero/engine/context.py	Adds `generate_context()` to write sample context markdown via atomic writes.
src/deepzero/engine/init.py	Defines engine package.
src/deepzero/api/server.py	Adds Starlette API server for run/sample listing and artifact viewing.
src/deepzero/api/init.py	Defines api package.
src/deepzero/main.py	Adds module entrypoint to CLI.
src/deepzero/init.py	Defines package metadata/version.
src/byovd_agent/translate/ghidra_runner.py	Removes legacy BYOVD Ghidra runner implementation.
src/byovd_agent/translate/init.py	Removes legacy translate package init.
src/byovd_agent/prompts.py	Removes legacy prompt content.
src/byovd_agent/models.py	Removes legacy pydantic models.
src/byovd_agent/knowledge/vuln_patterns.py	Removes legacy vuln-pattern knowledge module.
src/byovd_agent/knowledge/loldrivers.py	Removes legacy LOLDrivers DB wrapper.
src/byovd_agent/knowledge/init.py	Removes legacy knowledge package init.
src/byovd_agent/ingest/triage.py	Removes legacy ingest/triage implementation.
src/byovd_agent/ingest/scraper.py	Removes legacy scraper implementation.
src/byovd_agent/ingest/init.py	Removes legacy ingest package init.
src/byovd_agent/identify/semgrep_scanner.py	Removes legacy semgrep formatting/scanning utilities.
src/byovd_agent/identify/init.py	Removes legacy identify package init.
src/byovd_agent/config.py	Removes legacy dotenv-based config.
src/byovd_agent/agent.py	Removes legacy DeepAgents-based orchestrator agent.
src/byovd_agent/init.py	Removes legacy package init.
pyproject.toml	Renames project to `deepzero`, updates dependencies/extras, pytest config, scripts.
processors/semgrep_scanner/semgrep_scanner.py	Adds external bulk Semgrep scanner processor (batch stage).
processors/loldrivers_filter/loldrivers_filter.py	Adds external LOLDrivers filter processor (auto-download + cache).
processors/ghidra_decompile/ghidra_decompile.py	Adds external Ghidra decompile processor (headless, caching, timeouts).
pipelines/loldrivers/rules/msr_access.yaml	Adds Semgrep rules for MSR access patterns.
pipelines/loldrivers/rules/method_neither.yaml	Updates METHOD_NEITHER rule text (contains a mojibake issue).
pipelines/loldrivers/rules/buffer_overflow.yaml	Adds Semgrep rules for overflow/info-leak patterns.
pipelines/loldrivers/rules/arbitrary_rw.yaml	Adds Semgrep rules for arbitrary R/W and related primitives.
pipelines/loldrivers/pipeline.yaml	Adds reference LOLDrivers pipeline definition wiring external+built-in processors.
pipelines/loldrivers/assessment.j2	Adds LLM assessment prompt template.
func.txt	Adds an unshown auxiliary file.
copy_reports.ps1	Removes legacy report-copying script.
README.md	Rewrites documentation to match the new DeepZero pipeline/orchestrator focus.
LICENSE	Adds MIT license file.
.gitignore	Updates ignore patterns for DeepZero workspace/cache/build/test artifacts.
.github/workflows/ci.yml	Adds CI for linting (ruff), security scan (bandit), and pytest on 3.11/3.12.
.env.example	Replaces env template to match litellm + ghidra configuration.

Comments suppressed due to low confidence (1)

pipelines/loldrivers/rules/method_neither.yaml:9

The message contains mojibake (ΓÇö) instead of an em dash, which will show up incorrectly in Semgrep output and reports. Replace it with a proper — (or -) character.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-16T04:50:09Z

+class StageOutput:  # helper
+    def __init__(self, **kwargs):
+        for k, v in kwargs.items():
+            setattr(self, k, v)
+


The StageOutput helper class defined here is unused in this test module (no references found), and it shadows the real deepzero.engine.state.StageOutput name used elsewhere. Removing it avoids confusion and keeps the tests focused on the runner behavior.

Copilot · 2026-04-16T04:50:10Z

+from deepzero.engine.stage import (
+    Sample,
+    BulkMapProcessor,
+    ProcessorResult,
+    ProcessorContext,
+)
+
+
+class SemgrepScanner(BulkMapProcessor):
+    description = (
+        "runs semgrep batch scan against decompiled source across all active samples"
+    )
+
+    def validate(self, ctx: ProcessorContext) -> list[str]:
+        errors = []
+        if not shutil.which("semgrep"):
+            errors.append(
+                "semgrep CLI not found in PATH - install with: pip install semgrep"
+            )
+
+        rules_dir = self.config.get("rules_dir")
+        if not rules_dir:
+            errors.append("semgrep_scanner requires 'rules_dir' in config")
+        else:
+            rules_path = (Path.cwd() / rules_dir).resolve()
+            if not rules_path.exists():
+                rules_path = (ctx.pipeline_dir / rules_dir).resolve()
+            if not rules_path.exists():
+                errors.append(f"rules_dir does not exist: {rules_dir}")
+
+        return errors
+
+    def process(
+        self, ctx: ProcessorContext, entries: list[Sample]
+    ) -> list[ProcessorResult]:
+        rules_dir = self.config.get("rules_dir", "")


SemgrepScanner subclasses BulkMapProcessor, whose process() contract expects list[ProcessorEntry]. This implementation types the argument as list[Sample] and then accesses entry.sample_dir, which Sample does not have, causing an AttributeError at runtime when the runner calls this batch stage. Update the signature/imports to use ProcessorEntry (and keep the list order aligned with the input entries).

Copilot · 2026-04-16T04:50:10Z

+    def validate(self, ctx: ProcessorContext) -> list[str]:
+        errors = []
+
+        prompt_ref = self.config.get("prompt")
+        if not prompt_ref:
+            errors.append("GenericLLM requires 'prompt' template mapping in config")
+        else:
+            prompt_path = (Path.cwd() / prompt_ref).resolve()
+            if not prompt_path.exists():
+                prompt_path = (ctx.pipeline_dir / prompt_ref).resolve()
+            if not prompt_path.exists():
+                errors.append(f"Prompt template does not exist: {prompt_ref}")
+
+        # structurally validate LLM bindings early
+        model = ctx.global_config.get("model")
+        if model:
+            try:
+                import litellm
+
+                env_state = litellm.validate_environment(model=model)
+                if not env_state.get("keys_in_environment", True):
+                    missing_keys = env_state.get("missing_keys", [])
+                    if missing_keys:
+                        errors.append(
+                            f"LLM backend '{model}' missing credentials in environment. Need: {missing_keys}"
+                        )
+            except ImportError:
+                errors.append(
+                    "LLM configured, but 'litellm' framework is not installed"
+                )
+
+        return errors
+
+    def process(self, ctx: ProcessorContext, entry: ProcessorEntry) -> ProcessorResult:
+        if ctx.llm is None:
+            return ProcessorResult.fail("no llm provider configured for generic_llm")
+
+        prompt_ref = self.config.get("prompt", "")
+        prompt_text = self._render_prompt(prompt_ref, ctx, entry)
+
+        output_file = self.config.get("output_file", "assessment.md")
+        output_path = entry.sample_dir / output_file
+        if output_path.exists():
+            self.log.info("output already cached: %s", output_path.name)
+            content = output_path.read_text(encoding="utf-8", errors="replace")
+            return self._make_result(content, output_file)
+
+        max_retries = self.config.get("max_retries", 3)
+        backoff_config = self.config.get("backoff", {})
+
+        messages = [{"role": "user", "content": prompt_text}]
+
+        response = ctx.llm.complete(
+            messages,
+            max_retries=max_retries,
+            initial_backoff=backoff_config.get("initial", 2.0),
+            max_backoff=backoff_config.get("max", 60.0),
+            backoff_decay=backoff_config.get("decay", 0.7),
+        )
+
+        tmp = output_path.with_suffix(".tmp")
+        tmp.write_text(response, encoding="utf-8")
+        os.replace(tmp, output_path)
+        self.log.info("response written to %s (%d chars)", output_file, len(response))
+
+        return self._make_result(response, output_file)
+
+    def _make_result(self, content: str, output_file: str) -> ProcessorResult:
+        data: dict[str, Any] = {"llm_output_file": output_file}
+
+        classify_by = self.config.get("classify_by", "")
+        if classify_by:
+            import re
+
+            match = re.search(classify_by, content[:200], re.IGNORECASE)
+            if match:
+                verdict_text = match.group(0).strip("[]").lower()
+                data["classification"] = verdict_text
+
+        return ProcessorResult.ok(
+            artifacts={"llm_output": output_file},
+            data=data,
+        )
+
+    def _render_prompt(
+        self, prompt_ref: str, ctx: ProcessorContext, entry: ProcessorEntry
+    ) -> str:
+        template_path = self._resolve_template(prompt_ref)
+
+        if template_path is not None:
+            raw = template_path.read_text(encoding="utf-8")
+            template_vars = self._build_template_vars(ctx, entry)
+
+            env = jinja2.Environment(
+                loader=jinja2.FileSystemLoader(str(template_path.parent)),
+                undefined=jinja2.Undefined,
+                autoescape=jinja2.select_autoescape(),
+            )
+            template = env.from_string(raw)
+            return template.render(**template_vars)
+
+        return prompt_ref


GenericLLM.validate() resolves prompt templates relative to both Path.cwd() and ctx.pipeline_dir, but _render_prompt() / _resolve_template() only resolve via Path.cwd() (and absolute paths). This means a template that exists relative to the pipeline directory can pass validation but be treated as a literal prompt string at runtime. Consider resolving relative refs against ctx.pipeline_dir as well (or pass ctx into _resolve_template).

Copilot · 2026-04-16T04:50:10Z

+    spec = importlib.util.spec_from_file_location(
+        f"deepzero.custom.{file_path.stem}", file_path
+    )
+    if spec is None or spec.loader is None:
+        raise ImportError(f"cannot load processor from {file_path}")
+
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+


_load_specific_class() executes the module without first registering it in sys.modules (unlike _load_processor_from_file()). This can lead to duplicate module instances on repeated loads and can break imports/state that rely on module identity. Insert the module into sys.modules[spec.name] before exec_module().

Copilot · 2026-04-16T04:50:10Z

+    try:
+        hist = history
+    except NameError:
+        hist = {}
+


This helper uses try/except NameError around variables that are always defined in this scope (e.g., history), which is dead code and makes the fixture harder to understand/maintain. Simplify by assigning directly (or parameterizing the fixture) and remove the unreachable exception paths.

416rehman · 2026-04-16T04:55:53Z

@copilot apply changes based on the comments in this thread

… template resolution Agent-Logs-Url: https://github.com/416rehman/DeepZero/sessions/4d129d7b-9c62-4794-a11a-285adce9fb26 Co-authored-by: 416rehman <15690606+416rehman@users.noreply.github.com>

Copilot · 2026-04-16T05:00:38Z

@copilot apply changes based on the comments in this thread

Applied all 5 review changes in commit 221aebf:

tests/test_runner.py – Removed the unused StageOutput helper class that shadowed deepzero.engine.state.StageOutput.
processors/semgrep_scanner/semgrep_scanner.py – Updated process() and all helper method signatures (_build_bulk_dir, _run_and_distribute, _distribute_findings) to use ProcessorEntry instead of Sample, fixing the AttributeError on entry.sample_dir at runtime.
src/deepzero/stages/llm.py – Updated _resolve_template() to accept an optional ctx parameter and resolve relative template refs against ctx.pipeline_dir when Path.cwd()-relative resolution fails, matching the validation logic.
src/deepzero/engine/registry.py – Added sys.modules[spec.name] = module before exec_module() in _load_specific_class(), consistent with _load_processor_from_file().
tests/test_stages_llm.py – Removed the three dead try/except NameError blocks (for hist, sample_path_val, and sample_dir_val) and simplified to direct assignments. Also added ghidra_result.json to .gitignore.

Agent-Logs-Url: https://github.com/416rehman/DeepZero/sessions/ff83a24c-e0b7-4091-9be5-080ebcff2236 Co-authored-by: 416rehman <15690606+416rehman@users.noreply.github.com>

416rehman added 30 commits April 8, 2026 01:01

breadth-first core engine

1f7e5b8

type safety: add LLMProtocol and GlobalConfig TypedDict to replace An…

9bdaca8

…y types in stage/runner/cli

cleanup: fix unused imports in test files, add _load_env dotenv guard

d015fcd

cleanup: remove unused imports in test_ghidra_decompile.py and test_s…

71e5e0f

…tages_llm.py

cleanup: remove unused imports in runner.py after process.py extraction

08d743e

desloppify: Refactored runner broad exceptions

28eaad4

desloppify: resolve security vulnerabilities per strict structural ru…

dae7ceb

…les without suppressions

chore: add root dir to pytest pythonpath to resolve tools imports in …

91015ea

…test suite

ci: add github actions workflow for testing and linting

370bd44

feat: max concurrency pooling and rich progress bars

65397dd

fix: globally silence tool info logs during progress loop

fae8541

Refactor engine APIs: introduce generic pipeline enums and processor …

98a17fa

…state abstractions

chore: resolve static analysis vulnerabilities strictly without suppr…

a5b1049

…essions

chore: finalize pipeline architecture and stabilize CI

5ab9892

Merge main into pipeline-architecture, resolving conflict by deleting…

37483c1

… legacy ghidra_runner.py

desloppify: fix unused imports and security issues

e20a9e0

desloppify: ruff autofixes for unused imports

aeb2a08

refactor: drop #nosec tag and switch ghidra decompile processor to na…

8413638

…tive asyncio execution

refactor: resolve pipeline coupling and technical debt

8764640

docs: overhaul README with updated features and architecture

6afd696

docs: add .env.example template for quick setup

2948862

fix(ghidra): sync internal timeout to prioritize global StageSpec tim…

b8cd205

…eouts

Refactoring processor validate lifecycle configurations

8b8c9be

Enforce Ruff formatting constraints rigidly inside GitHub Actions CI …

4cdb38f

…limits

docs: Completely overwrite README to precisely standardize project ar…

49a749f

…chitecture and workflow execution schema

docs: strip explicitly opinionated terminology and genericize README …

4382889

…scope correctly

416rehman added 2 commits April 16, 2026 00:44

chore: include explicitly formatted MIT License aligning with downstr…

82a451c

…eam repository config defaults

chore: remove hanging local ghidra output from version control

1515228

Copilot AI review requested due to automatic review settings April 16, 2026 04:45

Copilot started reviewing on behalf of 416rehman April 16, 2026 04:46 View session

fix: explicit internal dev bindings mapping pytest-asyncio directly a…

3de5cba

…gainst Github Actions environment containers

Copilot AI reviewed Apr 16, 2026

View reviewed changes

Copilot started work on behalf of 416rehman April 16, 2026 04:56 View session

Apply review feedback: fix types, dead code, module registration, and…

221aebf

… template resolution Agent-Logs-Url: https://github.com/416rehman/DeepZero/sessions/4d129d7b-9c62-4794-a11a-285adce9fb26 Co-authored-by: 416rehman <15690606+416rehman@users.noreply.github.com>

Copilot finished work on behalf of 416rehman April 16, 2026 05:01

Copilot started work on behalf of 416rehman April 16, 2026 05:10 View session

Fix ruff/bandit/format issues and add tool config to pyproject.toml

27873c0

Agent-Logs-Url: https://github.com/416rehman/DeepZero/sessions/ff83a24c-e0b7-4091-9be5-080ebcff2236 Co-authored-by: 416rehman <15690606+416rehman@users.noreply.github.com>

Copilot finished work on behalf of 416rehman April 16, 2026 05:14

416rehman merged commit bfbbf3e into main Apr 16, 2026
2 checks passed

416rehman deleted the pipeline-architecture branch April 16, 2026 05:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipeline architecture#2

Pipeline architecture#2
416rehman merged 35 commits intomainfrom
pipeline-architecture

416rehman commented Apr 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

416rehman commented Apr 16, 2026

Uh oh!

Copilot AI commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

416rehman commented Apr 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

416rehman commented Apr 16, 2026

Uh oh!

Copilot AI commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants