Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The V1 demo proves public cloud-storage exposure in local sandboxes, applies Ter
8. Validate the attack path is blocked.
9. Write case-study-ready evidence and metrics.

Hackathon V1 includes live LocalStack runs for AWS S3 and Azure Blob scenarios, plus offline deterministic demos for Kubernetes, Docker Compose, on-prem baselines, and generic plan review.
Hackathon V1 includes a live-validated LocalStack AWS S3 path, a manifest-backed Azure Blob probe path that still needs live LocalStack Azure entitlement to validate, plus offline deterministic demos for Kubernetes, Docker Compose, on-prem baselines, and generic plan review.

## Why this exists

Expand Down Expand Up @@ -280,7 +280,7 @@ Users do not need to write prompts. `nullstate` sends internal red-team and blue

| Scenario | Backend | Status |
|---|---|---|
| `azure-public-blob` | `localstack-azure` | live LocalStack demo available |
| `azure-public-blob` | `localstack-azure` | manifest-backed/offline demo available; live LocalStack Azure validation pending emulator entitlement |
| `aws-public-s3` | `localstack-aws` | live LocalStack demo available |
| `k8s-privileged-pod` | `kind-kubernetes` | offline demo available; live kind pending |
| `compose-exposed-admin` | `docker-compose` | offline demo available; live Docker probe pending |
Expand Down Expand Up @@ -430,6 +430,6 @@ cosign verify-blob $wheel.FullName --bundle "$($wheel.FullName).sigstore.json" -

## Status

Working now: live LocalStack AWS/Azure storage scenarios, offline deterministic demos for all listed scenarios, constrained red attack command execution, deterministic remediation, sandbox registry, report artifacts, model metrics artifacts, branded CLI output, and DevSecOps repo structure.
Working now: live LocalStack AWS storage validation, manifest-backed Azure Blob probe generation pending live LocalStack Azure entitlement, offline deterministic demos for all listed scenarios, constrained red attack command execution, deterministic remediation, sandbox registry, report artifacts, model metrics artifacts, branded CLI output, and DevSecOps repo structure.

Experimental: richer scenario-specific attack scripts, live Kubernetes/Compose/on-prem adapters, richer resolved-dependency SBOMs, and broader artifact redaction coverage.
67 changes: 52 additions & 15 deletions docs/handoff.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Nullstate Project Handoff

Last updated: 2026-06-18
Last updated: 2026-06-19

## Read this first

Expand All @@ -14,6 +14,7 @@ The hackathon freeze rule was lifted on 2026-06-18 by the project owner. PR #24
Recent merged checkpoint:

```text
f621b1b docs: record PR 24 merge
1e7b1c6 feat: add constrained red attack runner
```

Expand All @@ -26,6 +27,39 @@ git log --oneline -8

Do not rely on MCP state, chat memory, or remote PR metadata. Use local files and Git only unless the user explicitly provides other tooling.

## Fresh agent checklist

When this project is opened on the main device:

1. Confirm branch and cleanliness:

```powershell
git status --short --branch
git log --oneline -8
```

2. Confirm local secret files are present only locally and ignored by Git:

```powershell
git check-ignore .env .env.local
```

3. Recreate the local Python environment if needed:

```powershell
python -m pip install -e .
```

4. Run the verification gate before taking new work:

```powershell
python -m ruff check src tests
python -m mypy src
python -m unittest discover -s tests -v
```

5. Start a new feature branch for follow-up product work. Keep `main` protected and use PR checks.

## Project goal

`nullstate` is becoming an open-core DevSecOps product:
Expand Down Expand Up @@ -154,7 +188,11 @@ docs/plans/2026-06-01-real-sandbox-red-team-commands.md

## Last verification run

The last full verification passed:
The last full verification passed on 2026-06-19 after reinstalling the project with dev extras:

```powershell
python -m pip install -e .[dev]
```

```powershell
python -m ruff check src tests
Expand All @@ -167,15 +205,15 @@ Result:
```text
Ruff passed
mypy passed
69 tests OK
146 tests OK
```

Smoke run also passed:
Smoke run also passed on 2026-06-19:

```powershell
python -m nullstate run examples/aws-public-s3 --offline --mock-agents --runs-dir runs/platform-smoke
python -m nullstate bundle --runs-dir runs/platform-smoke
python -m nullstate dashboard --runs-dir runs/platform-smoke
python -m nullstate run examples/aws-public-s3 --offline --mock-agents --runs-dir runs/handback-smoke-20260619-195333
python -m nullstate bundle --runs-dir runs/handback-smoke-20260619-195333
python -m nullstate dashboard --runs-dir runs/handback-smoke-20260619-195333
```

Generated:
Expand Down Expand Up @@ -238,7 +276,7 @@ Recommended next tests:
- Live LocalStack Azure run where available.
- Offline run still passes.
- Report classifies runtime evidence as observed/inconclusive/simulated.
- Remaining enterprise hardening: future live-cloud approval gate and live Azure emulator validation after LocalStack Azure entitlement is available.
- Remaining enterprise hardening: live Azure emulator validation after LocalStack Azure entitlement is available.

If Azure LocalStack support is unavailable or unreliable, do not overclaim Azure runtime exploitation. Prefer clear report language such as `runtime probe inconclusive; deterministic IaC validation still blocked the configured exposure`.

Expand All @@ -259,7 +297,7 @@ Run bundle is the key contract between:
- support tickets
- future enterprise dashboards

Current productization checkpoints on `feature/red-agent-runner` also include:
Current productization checkpoints merged into `main` include:

- provider presets for Google, Claude, custom, and generic OpenAI-compatible endpoints
- SARIF export and GitHub Actions code-scanning workflow
Expand Down Expand Up @@ -300,11 +338,10 @@ Do not jump straight to full SaaS before stabilizing:

## Branch and release guidance

Until the user says the freeze is over:
The hackathon freeze is over, but `main` remains protected:

- Do not merge PR #24.
- Do not push or merge to `main`.
- Feature branch checkpoint pushes are allowed.
- Start substantive work on a feature branch.
- Open PRs into `main` and wait for required checks.
- Direct pushes to `main` are blocked by branch protection.
- After applying CodeRabbit review feedback, do not push solely to trigger another CodeRabbit review loop; fold those fixes into the next substantive batch unless the user explicitly approves a review-response push.
- Do not tag releases.
- Do not update `main`.
- Do not tag releases or publish product releases without explicit owner approval.
16 changes: 9 additions & 7 deletions docs/plans/2026-06-01-real-sandbox-red-team-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

> **For Agent:** Use executing-plans skill to implement this plan task-by-task.

> **Status as of 2026-06-19:** This plan is mostly implemented and was merged into `main` through PR #24. Do not replay Tasks 1-6 from scratch. Use this file as historical context and continue only with the remaining live LocalStack Azure validation work after the account has Azure Emulator entitlement.

**Goal:** Replace shallow LocalStack health probes with scenario-specific red-team probes that attempt real sandbox reads before and after remediation while preserving strict command boundaries.

**Architecture:** Keep the existing constrained `attack.py` runner. Move exploit logic into scenario templates that receive a target URL, stage, and a generated evidence manifest. The deterministic detector remains the source of truth, but the report distinguishes between configuration validation and real runtime exploit evidence.
Expand All @@ -12,18 +14,19 @@

## Current State

The current red-team execution feature is safe but shallow:
The current red-team execution feature is safe and partly runtime-backed:

- `src/nullstate/attack_runner.py` executes only generated `attack.py` inside the run directory.
- `events.jsonl` records command, stdout, stderr, return code, target URL, stage, timestamps, and duration.
- AWS/Azure `attack.py` scripts currently call `/_localstack/health` when online.
- The before/after `success` and `blocked` verdict still comes from `simulate_attack()`.
- AWS has a live-validated LocalStack object-read probe path.
- Azure has a manifest-backed blob-read probe path, but live LocalStack Azure validation is still blocked by Azure Emulator entitlement.
- The before/after final verdict still relies on deterministic IaC validation for reliability; reports distinguish observed, inconclusive, and simulated runtime evidence.

This is a strong security boundary, but not yet a full enterprise exploit validation engine.

## Progress Status

Updated 2026-06-09:
Updated 2026-06-19:

- Completed locally: Task 1 attack evidence manifest.
- Completed locally: Task 2 safe runner manifest argument.
Expand Down Expand Up @@ -58,9 +61,8 @@ Updated 2026-06-09:
- Google AI Studio / Gemini users can provide only `NULLSTATE_LLM_PROVIDER=google` and `NULLSTATE_LLM_API_KEY`
- Claude is routed through Anthropic's OpenAI SDK compatibility endpoint and documented as experimental
- self-hosted AMD/vLLM/SGLang users can still provide explicit shared or role-specific base URLs
- Verified locally with Ruff, mypy, full unittest discovery, and an offline Azure smoke run.
- Not pushed: local feature-branch work should remain local unless the user explicitly asks to push.
- Freeze rule: do not merge to `main`, do not push unless the user explicitly asks.
- Merged into `main` through PR #24 after review blockers and required checks passed.
- Verified locally on 2026-06-19 with Ruff, mypy, full unittest discovery, and an offline AWS handback smoke run.

Next task to execute:

Expand Down
14 changes: 9 additions & 5 deletions docs/progress.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,17 +66,17 @@ This file tracks roadmap progress after the hackathon freeze. Keep updates brief
- Added `nullstate policy validate`.
- Writes optional `policy-validation.json` for CI evidence.
- Exits with code `2` when the policy file is malformed or invalid.
- `fix: harden policy and evidence verification` (local checkpoint)
- `fix: harden policy and evidence verification` (merged in PR #24)
- Reviewed the latest CodeRabbit findings and accepted the verified fail-closed issues.
- `nullstate policy-result` now fails closed when `findings.json` is missing, malformed, or not a list.
- `nullstate evidence-verify` now fails copied/wrong manifests whose declared run identity does not match the target run.
- Malformed evidence manifests now surface as CLI parameter errors instead of Python tracebacks.
- This checkpoint is intentionally local until the next substantive batch push, to avoid triggering a CodeRabbit review loop over review-response fixes.
- `fix: make SARIF upload and CodeQL fixtures CI-clean` (local checkpoint)
- Batched with the next productization push to avoid a CodeRabbit review loop over review-response fixes.
- `fix: make SARIF upload and CodeQL fixtures CI-clean` (merged in PR #24)
- Investigated PR #24 failing checks with `gh`.
- Fixed GitHub SARIF upload validation by moving finding logical locations under SARIF result locations.
- Reworked scrubber test fixture values/naming so CodeQL does not treat the test artifact as clear-text secret storage.
- GitHub has not re-run these checks yet because the fixes remain local on this branch.
- GitHub checks later passed on PR #24 before merge.
- `feat: add scenario policy presets` (this checkpoint)
- Added `nullstate policy init --scenario <name>`.
- Scenario presets narrow generated `allowed_scenarios` and `allowed_backends` to one known scenario/backend pair.
Expand Down Expand Up @@ -136,7 +136,7 @@ This file tracks roadmap progress after the hackathon freeze. Keep updates brief
- `docs: add first release verification checklist` (this checkpoint)
- Added a runbook checklist for the first tagged release.
- Captures PR check review, manual release dry-run rehearsal, release inspection, GitHub attestation verification, and Sigstore bundle verification.
- Keeps the freeze rule explicit: no tagging, release publishing, or `main` updates until approved.
- Keeps release approval explicit: no tagging or release publishing until approved.
- `feat: add versioned remediation metadata` (this checkpoint)
- Added `remediation.json` with remediation ruleset version, scenario, changed files, and applied deterministic rule IDs.
- Included remediation metadata in reports, run bundles, and blue-team remediation events.
Expand Down Expand Up @@ -190,6 +190,10 @@ This file tracks roadmap progress after the hackathon freeze. Keep updates brief
- `feat: add constrained red attack runner` (merged PR #24)
- Squash-merged the productization branch into `main` after all review threads and remote checks were green.
- Merge commit: `1e7b1c6`.
- `docs: prepare main-device handoff` (this checkpoint)
- Refreshed handoff, runbook, README, progress, and active plan docs for the post-freeze `main` state.
- Corrected Azure runtime wording so it does not overclaim live LocalStack Azure validation before emulator entitlement is available.
- Reinstalled local dev extras and reran Ruff, mypy, 146 unit tests, and an offline AWS handback smoke run.

## In Progress

Expand Down
2 changes: 1 addition & 1 deletion docs/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ python -m nullstate scrub 20260509-200601 --runs-dir runs --output-dir scrubbed-

## First Tagged Release Checklist

Use this checklist before the first product tag. Do not tag, publish a release, merge to `main`, or push to `main` until the freeze is explicitly lifted.
Use this checklist before the first product tag. The hackathon merge freeze was lifted on 2026-06-18, but product tags and release publishing still need explicit owner approval. Keep normal changes on feature branches and merge through protected PR checks.

Confirm the release candidate PR checks are green:

Expand Down