FreshCtx × OpsWatch: JSONL control-to-behavior assurance experiment

[bhooshanvarma]

We now have a bounded, reproducible experiment for testing the separation between a FreshCtx control decision and independent observation of downstream agent behavior.

This first experiment is designed with OpsWatch as a read-only assurance layer. It uses local files and FreshCtx's normal JSONL audit trail—no callback or external service is required.

Question

Can an independent observer determine whether an agent respected a FreshCtx block after a declared dependency changed?

Scenario

1. The agent observes an approval record for vendor-a.
2. It derives a release decision from that record.
3. The approval changes to an unapproved vendor-b before execution.
4. FreshCtx returns STALE_REASONING with policy_decision: block.
5. The bounded runner either respects the block or deliberately bypasses the protected boundary.
6. The assurance layer compares the JSONL decision with the observed downstream effect.

Run it

git clone https://github.com/Hyperwise-LLC/freshctx.git
cd freshctx
python -m pip install freshctx

python examples/opswatch_jsonl_assurance.py \
  --output-dir /tmp/freshctx-opswatch-respect \
  --behavior respect

python examples/opswatch_jsonl_assurance.py \
  --output-dir /tmp/freshctx-opswatch-violate \
  --behavior violate

Expected outcomes:

Runner	FreshCtx decision	Effect	Independent verdict
respect	block	absent	PASS_AGENT_RESPECTED_BLOCK
violate	block	present	FAIL_AGENT_ACTED_AFTER_BLOCK

The violating mode is intentionally noncompliant test behavior. It calls the simulated action outside ctx.run so the assurance layer has a known failure to detect.

Evidence surfaces

Each run produces:

• freshctx-audit.jsonl — the normal FreshCtx audit stream;
• approval.json — the dependency after the concurrent change;
• downstream-effect.json — present only if the action executed; and
• downstream-observation.json — the bounded observer's verdict.

Full protocol and expected events: https://github.com/Hyperwise-LLC/freshctx/blob/main/docs/OPSWATCH_ASSURANCE_EXPERIMENT.md

Implementation and tests: #27

Invitation to reproduce

The purpose of this thread is to record the external result, including any mismatch or failure. If you run the experiment, please share:

• the two verdict files;
• whether the audit trail contained the expected policy_applied event;
• whether downstream behavior matched the FreshCtx decision; and
• any ambiguity OpsWatch encountered while interpreting the JSONL events.

Once the read-only path is understood, a Python callback can be considered as a separate, closer-to-execution experiment.

[jasonmcgill788-dot]

Independent bounded reproduction result

I independently reproduced the published stale-context scenario in the environment available to me.

Result

The reproduced scenario behaved as described: the stale-context condition was detected and the guarded action was prevented under the tested conditions.

Scope

This should be treated as a bounded reproduction of the published scenario, not as broader validation of FreshCtx.

The test establishes that the specific scenario reproduced in this environment produced the expected result. It does not establish coverage across other workflows, configurations, attack paths, integrations, or production conditions.

Installation limitation

This reproduction did not originate from a clean pip install freshctx installation.

That matters because the result therefore does not independently verify the packaged installation path or establish equivalence between the tested environment and a clean installation from the published package.

Evidence interpretation

The useful claim is therefore narrow:

Within the reproduced scenario and environment tested, the FreshCtx control behaved consistently with the published example.

A separate second pass from a clean pip install freshctx environment would be useful evidence, but should remain a distinct test rather than being merged into this result.

— Jason McGill, OpsWatch / McGill Intelligence

Thank you, Jason. This is exactly the level of precision I hoped we could preserve.

The bounded result is useful on its own: in the independently reproduced scenario and environment, FreshCtx detected the stale-context condition and prevented the guarded action as expected.

I also appreciate you documenting the installation limitation rather than allowing the claim to exceed the evidence. We will keep any clean pip install freshctx reproduction as a separate test and artifact.

[jasonmcgill788-dot]

Thank you, Bhooshan. Agreed — preserving that boundary is important. I’ll treat the current result as the completed bounded reproduction artifact: the published stale-context scenario behaved as expected in the environment tested, while installation reproducibility remains unverified. A clean `pip install freshctx` reproduction should therefore be recorded separately, with its own environment, package version, execution evidence and result. Beyond that, I suggest we keep the next product-level question distinct again: FreshCtx can establish the control result — including whether execution should be allowed or blocked — while OpsWatch independently verifies whether the downstream system actually adhered to that result. That gives us three separate evidence layers rather than one broad claim: 1. Scenario reproduction 2. Clean installation and packaging reproducibility 3. Downstream control-to-execution adherence I’m happy to help define the smallest reproducible artifact for the clean-install test, then move into the bounded JSONL control-to-behaviour experiment without conflating the two. Jason McGill Founder, McGill Intelligence / OpsWatch