feat: infer scenarios and probe sandbox status#13
Merged
Conversation
📝 WalkthroughWalkthroughThis PR implements automatic scenario and sandbox backend inference for the ChangesAuto Scenario/Target Inference & Sandbox Probing
Sequence Diagram(s)sequenceDiagram
participant User
participant CLI
participant ResolveScenario as _resolve_scenario
participant InferScenario as infer_scenario
participant GetScenario as get_scenario
participant ResolveBackend as _resolve_backend
participant GetBackend as get_backend
participant RunLogic as run_logic
User->>CLI: nullstate run /path/to/iac --offline
CLI->>ResolveScenario: _resolve_scenario(iac_dir, "auto")
ResolveScenario->>InferScenario: infer_scenario(iac_dir)
InferScenario->>InferScenario: read & pattern-match IaC files
InferScenario-->>ResolveScenario: Scenario object or None
ResolveScenario->>GetScenario: get_scenario(matched_name)
GetScenario-->>ResolveScenario: Scenario
ResolveScenario-->>CLI: resolved_scenario
CLI->>ResolveBackend: _resolve_backend("auto", scenario.backend)
ResolveBackend->>GetBackend: get_backend(scenario.backend)
GetBackend-->>ResolveBackend: SandboxBackend
ResolveBackend-->>CLI: resolved_backend
CLI->>RunLogic: execute red/blue agents with inferred config
RunLogic-->>User: findings.json with inferred scenario/target
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/nullstate/cli.py`:
- Around line 341-362: The two helpers _resolve_scenario and _resolve_backend
lack return type annotations, preventing mypy from validating callers; add
explicit return types matching what get_scenario and get_backend return by
importing and using Scenario from .scenarios for _resolve_scenario and Backend
from .sandbox for _resolve_backend, then annotate the functions as def
_resolve_scenario(terraform_dir: Path, scenario: str) -> Scenario: and def
_resolve_backend(target: str, scenario_backend: str) -> Backend: while leaving
existing logic and error handling unchanged.
In `@src/nullstate/scenario_detection.py`:
- Around line 46-47: The function _looks_like_k8s_privileged_pod currently
treats any file containing "hostPath:" as a privileged-k8s signal; change the
logic so hostPath is only considered when Kubernetes markers are present:
require both "apiVersion:" and "kind:" to be in text, and then check for either
"privileged: true" or "hostPath:" (i.e., gate the hostPath check behind the
apiVersion/kind check) so non-Kubernetes files with hostPath do not trigger the
detector.
- Around line 50-54: The helper _looks_like_compose_exposed_admin currently
returns true for any compose file because it only checks filename and
"services:"; change it to detect admin-specific signals by parsing the compose
content (or using regex) to look for admin service names (e.g., "adminer",
"phpmyadmin", "portainer", "grafana", "kibana") and for explicit port/expose
mappings that bind typical admin ports (e.g., 80, 443, 8080, 3000, 9000, 5601)
under a service's "ports:" or "expose:" entries; only return True when at least
one admin service name or an exposed/admin port mapping is present, otherwise
return False. Ensure you update _looks_like_compose_exposed_admin to inspect
each service block in text_by_name values and match on service name keys and
port lines rather than just "services:".
- Around line 42-43: The helper _looks_like_aws_public_s3 is too permissive;
replace the broad substring check for "aws_s3_bucket" with a tighter heuristic:
return true if text contains "aws_s3_bucket_public_access_block" OR contains an
aws_s3_bucket resource declaration plus explicit public indicators (e.g.,
'resource "aws_s3_bucket"' together with 'acl = "public-read"' or 'acl =
"public-read-write"', a grant/acl that mentions "AllUsers" or
"AuthenticatedUsers", or an explicit public_access_block configuration set to
false). Update the function _looks_like_aws_public_s3 to scan for those combined
patterns instead of plain "aws_s3_bucket" so only buckets that are likely
publicly accessible trigger the aws-public-s3 scenario.
In `@tests/test_offline_scenario_runs.py`:
- Around line 118-120: The test uses next(runs_dir.iterdir()) and directly
indexes findings[0], which can raise StopIteration/FileNotFoundError or
IndexError instead of an assertion failure; change to collect the runs into a
list (e.g., runs = list(runs_dir.iterdir())) and assert its length equals the
expected count before selecting run_dir (e.g., run_dir = runs[0]), and add an
assertion that findings is non-empty (e.g., self.assertTrue(findings) or
self.assertGreater(len(findings), 0)) before accessing findings[0] so failures
produce clear assertion messages; update references to runs_dir and findings
accordingly.
In `@tests/test_scenario_detection.py`:
- Around line 26-28: The test currently accesses inferred.name without checking
for None; update the test around infer_scenario(demo_dir) to first assert that
inferred is not None (e.g., using self.assertIsNotNone(inferred,
f"infer_scenario returned None for {demo_dir} expected {expected}")), then
perform the existing self.assertEqual(inferred.name, expected); reference the
infer_scenario call and the inferred variable so the assertion failure will
clearly indicate which demo_dir/expected pair failed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 9a199d43-e2a6-4434-b6fd-38a0250f90de
📒 Files selected for processing (10)
CHANGELOG.mdREADME.mddocs/architecture.mddocs/runbook.mdsrc/nullstate/cli.pysrc/nullstate/sandbox.pysrc/nullstate/scenario_detection.pytests/test_offline_scenario_runs.pytests/test_sandbox.pytests/test_scenario_detection.py
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_cli_model_endpoint.py`:
- Line 82: The test uses a relative scenario path string
"examples/azure-public-blob" which can break when tests run from a different
CWD; resolve that path to an absolute path before passing it to the CLI. Update
the test in tests/test_cli_model_endpoint.py to compute the scenario path from
the test file location (e.g. using __file__ or Path(__file__).resolve().parent
and joining the repo/testroot up to the examples directory) and pass the
resulting absolute path to the CLI invocation instead of the raw relative
string.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: b1abba09-85b6-4a5a-bda7-b5525cde9679
📒 Files selected for processing (5)
CHANGELOG.mdREADME.mddocs/runbook.mdsrc/nullstate/cli.pytests/test_cli_model_endpoint.py
This was referenced May 9, 2026
Owner
Author
|
all checks pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nullstate runto--scenario autoand--target autonullstate sandbox status, including LocalStack Docker and HTTP health checksEvidence to capture before merge
python -m nullstate run examples/aws-public-s3 --offlineshowing inferred scenario/targetpython -m nullstate sandbox status localstack-azureshowing Docker running and HTTP 200, with license/session IDs redacted from any separate Docker logsVerification
python -m unittest discover -s tests -vpython -m ruff check src testspython -m mypy srcpython -m pip_audit . --skip-editablepython -m nullstate run examples/aws-public-s3 --offline --runs-dir $env:TEMP\nullstate-auto-aws-runpython -m nullstate sandbox status localstack-azurepython -m nullstate run examples/azure-public-blob --offline --target localstack-azure --runs-dir $env:TEMP\nullstate-auto-azure-runCloses #12
Summary by CodeRabbit
New Features
nullstate run, removing the need for explicit--scenario/--targetin most cases.Documentation
--scenario auto/--target auto, clarified--offlinevs model usage, and documented--mock-agentsdeterministic fallback.Tests