feat: add bounded autoresearch and context telemetry - #44
Conversation
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_9ea580f9-485a-4646-be1d-3409b3aaf28a) |
|
Tick the box to add this pull request to the merge queue (same as
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29f8261f92
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let mut ledger = if resume { | ||
| load_ledger(&ledger_path).await? |
There was a problem hiding this comment.
When --resume is run from any clean commit other than ledger.best_commit—including after an accepted commit succeeds but saving the ledger fails—the recorded best metric is loaded without verifying the code that produced it. Subsequent candidates are therefore measured from an unrelated checkpoint while being compared against the stale metric, corrupting acceptance decisions and experiment history; verify or restore best_commit before continuing.
Useful? React with 👍 / 👎.
| git_command(workspace, &["reset", "--hard", checkpoint]).await?; | ||
| git_command(workspace, &["clean", "-fd"]).await?; |
There was a problem hiding this comment.
When an iteration modifies an existing Git-ignored file that affects validation or the metric, this reset leaves that modification in place, so later iterations no longer start from the recorded best state; an accepted result can likewise depend on ignored changes that git add -A never commits. git clean -h explicitly describes -x as “remove ignored files, too,” meaning the current -fd does not restore them, so ignored state must be protected or restored without deleting the ignored spec and ledger.
Useful? React with 👍 / 👎.
| let baseline = run_with_budget( | ||
| started, | ||
| self.config.max_duration_secs, | ||
| measure_metric(&self.config, &self.workspace), | ||
| ) |
There was a problem hiding this comment.
When the initial workspace fails validation_command but still emits a numeric metric, the new-run path records that invalid result as the best baseline without running the configured quality gate. Valid later candidates can then be rejected because they are compared with a metric produced by broken code, even though the configuration says validation must pass before a metric is considered; run validation and fail fast before measuring the baseline.
Useful? React with 👍 / 👎.
| .with_workspace(workspace.to_path_buf()) | ||
| .with_memory_ideas(cfg.memory.clone()) | ||
| .with_group_chat(cfg.group_chat.clone()) | ||
| .with_skills(discovered_skills) | ||
| .await; | ||
| #[cfg(feature = "zkr-memory")] | ||
| { | ||
| runner = runner.with_zkr(zkr_store, cfg.zkr.clone()); |
There was a problem hiding this comment.
When the default memory.inject_context, ZKR recall, or ZKR self-improvement settings are enabled, this restricted runner still forwards those configurations, so handle_message_with_model can inject stored personal context and ZKR recall into the experiment prompt and persist reflections. Disabling the memory tool group therefore does not keep memory out of the model's ambient context as claimed, potentially sending unrelated private memory to the autoresearch provider; use disabled memory/ZKR settings or an isolated runner for this mode.
AGENTS.md reference: AGENTS.md:L45-L47
Useful? React with 👍 / 👎.
| id: uuid::Uuid::new_v4().to_string(), | ||
| sender_id: "autoresearch".to_string(), | ||
| sender_name: Some("Autoresearch".to_string()), | ||
| chat_id: "autoresearch".to_string(), | ||
| text: prompt, |
There was a problem hiding this comment.
When a new objective is started with --resume unset in a workspace that has run autoresearch before, every invocation still uses the fixed autoresearch chat ID. AgentRunner consequently loads the prior run's persisted conversation history and supplies stale objectives, metrics, and rejected hypotheses to the new experiment, even though a fresh ledger was requested; derive the chat ID from the ledger or objective while retaining it only for resumes of that run.
AGENTS.md reference: AGENTS.md:L45-L47
Useful? React with 👍 / 👎.
| let result = run_with_budget(started, self.config.max_duration_secs, turn).await; | ||
|
|
||
| let hypothesis = result | ||
| .as_ref() | ||
| .map(|response| first_line(response).unwrap_or_else(|| "agent experiment".into())) | ||
| .unwrap_or_else(|error| format!("agent error: {error}")); | ||
|
|
||
| let (decision, metric, reason) = match result { |
There was a problem hiding this comment.
When the experiment agent uses its available shell tool to commit, reset, or check out another branch despite the prompt, the controller evaluates the candidate without verifying that HEAD and the current branch still match the checkpoint. On rejection, restore_checkpoint then runs git reset --hard on whichever branch is currently selected—git reset -h says this resets HEAD, the index, and the working tree—so it can rewrite an unrelated branch; on acceptance, an agent-created commit can instead make commit_experiment abort with no changes. Capture and enforce both the branch and HEAD after the turn.
Useful? React with 👍 / 👎.
| let commit = if accepted { | ||
| Some(commit_experiment(&self.workspace, iteration).await?) |
There was a problem hiding this comment.
When a repository has a slow or hung pre-commit or commit-msg hook, an accepted iteration calls git commit outside run_with_budget, so max_duration_secs and command_timeout_secs no longer bound the run. git commit -h identifies --no-verify as the option that bypasses those hooks, and this invocation does not use it; therefore a hook can keep autoresearch running indefinitely after the metric was accepted. Execute the commit under the remaining run budget and recover the candidate on timeout.
Useful? React with 👍 / 👎.
| if ledger.objective != self.config.objective | ||
| || ledger.direction != self.config.direction.trim().to_ascii_lowercase() | ||
| { | ||
| bail!("autoresearch ledger does not match the current spec; use a new ledger path"); |
There was a problem hiding this comment.
When the ignored autoresearch spec is edited between invocations while keeping the same objective and direction, --resume accepts the old ledger even if metric_command now measures a different unit or workload. The recorded best_metric is then compared directly with values from the new command, producing meaningless acceptance decisions despite HEAD still matching best_commit; persist and verify the metric and validation definition, or a fingerprint of all result-affecting settings, before resuming.
Useful? React with 👍 / 👎.
| .kill_on_drop(true) | ||
| .spawn() | ||
| .with_context(|| format!("starting autoresearch command: {command}"))?; | ||
| tokio::time::timeout(Duration::from_secs(timeout_secs), child.wait_with_output()) | ||
| .await |
There was a problem hiding this comment.
When a validation or metric command launches child processes, such as sh -c cargo test, timing out drops or kills only the direct sh child; its cargo/compiler descendants can remain alive and continue modifying the workspace while the controller rejects, restores, or evaluates the next iteration. kill_on_drop(true) is not process-tree cleanup, so these supposedly bounded commands need an isolated process group or equivalent descendant termination.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| /// Record the estimated shape of a provider request for harness telemetry. | ||
| pub async fn record_context(&self, snapshot: ContextSnapshot) { |
There was a problem hiding this comment.
In a long-running chat or server process, every provider attempt appends another ContextSnapshot to this vector, but the only exposed operation aggregates the entire vector and no per-request context history is available. Memory therefore grows without bound and each summary becomes progressively slower even though running counters would provide identical results; store aggregate totals rather than retaining every snapshot.
AGENTS.md reference: AGENTS.md:L13-L16
Useful? React with 👍 / 👎.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_a2ff5a06-aea9-402b-9cd4-4310ec491258) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ce86947ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let policy = Arc::new(ExecutionPolicy::from_config(&cfg.policy)); | ||
| let memory = build_memory_backend(workspace, &cfg).await?; | ||
| let embedding_provider = build_embedding_provider(&cfg)?; | ||
| let system_prompt = prompt::build_system_prompt(workspace).await; |
There was a problem hiding this comment.
When the workspace contains USER.md or MEMORY.md, this restricted runner still calls prompt::build_system_prompt, whose prompt-file list loads both files and sends their contents to the configured provider. Fresh evidence shows the prior ambient-memory issue remains despite the new memory_enabled guards, because those guards only cover conversation, DB, and ZKR paths; construct an autoresearch-specific prompt that excludes personal-memory files.
AGENTS.md reference: AGENTS.md:L45-L47
Useful? React with 👍 / 👎.
| if !run_with_budget( | ||
| started, | ||
| self.config.max_duration_secs, | ||
| run_validation(&self.config, &self.workspace), | ||
| ) |
There was a problem hiding this comment.
When a validation or metric command generates or reformats a tracked file, the sole cleanliness check has already run, so the ledger records the pre-command HEAD as best_commit while the first experiment starts from an uncommitted, different baseline. A later rejection then restores the recorded commit rather than the measured state, while an acceptance can commit evaluator side effects; verify that trusted baseline commands leave the tracked tree unchanged or restore it before recording the baseline.
Useful? React with 👍 / 👎.
| let remaining = Duration::from_secs(max_duration_secs).saturating_sub(started.elapsed()); | ||
| tokio::time::timeout(remaining, future) | ||
| .await | ||
| .with_context(|| "autoresearch wall-clock budget exhausted")? |
There was a problem hiding this comment.
When the remaining max_duration_secs is shorter than command_timeout_secs, this outer timeout cancels run_validation or measure_metric before run_process reaches its own timeout branch. Fresh evidence is this nested timeout path: dropping wait_with_output invokes kill_on_drop only for the direct shell and bypasses terminate_process_group, so cargo/compiler descendants can survive and keep modifying the workspace during cleanup or a later iteration.
Useful? React with 👍 / 👎.
| files.push(IgnoredFile { | ||
| relative, | ||
| contents: Some(tokio::fs::read(&path).await?), | ||
| symlink_target: None, |
There was a problem hiding this comment.
In workspaces with large ignored trees other than top-level target—for example node_modules, .venv, or generated datasets—this reads every ignored regular file into IgnoredWorkspaceState and retains all bytes for the whole iteration. A routine autoresearch run can therefore allocate gigabytes or fail before contacting the agent; snapshot only protected experiment inputs or otherwise bound/spool this state instead of buffering every ignored file.
AGENTS.md reference: AGENTS.md:L13-L18
Useful? React with 👍 / 👎.
| let temporary = path.with_extension("tmp"); | ||
| tokio::fs::write(&temporary, content).await?; | ||
| tokio::fs::rename(&temporary, path).await?; |
There was a problem hiding this comment.
On Windows, renaming a file over an existing destination fails, so the first post-baseline save_ledger attempts to rename the temporary file onto the already-present ledger and aborts every run after an iteration. If the candidate was accepted, its commit has already landed while the ledger remains stale; use a cross-platform atomic replacement primitive rather than relying on Unix overwrite semantics.
Useful? React with 👍 / 👎.
| async fn ensure_ledger_path_safe(workspace: &Path, ledger_path: &Path) -> anyhow::Result<()> { | ||
| let Ok(relative) = ledger_path.strip_prefix(workspace) else { | ||
| return Ok(()); |
There was a problem hiding this comment.
When the workspace is supplied as a relative path such as . and ledger_path is an absolute path inside that same workspace, this lexical strip_prefix fails and treats the ledger as external. A non-ignored ledger then bypasses the safety check, gets created after the initial cleanliness check, and can be staged by the next accepted git add -A, after which every ledger save leaves tracked changes behind; normalize or canonicalize both paths before deciding containment.
Useful? React with 👍 / 👎.
| async fn restore_checkpoint(workspace: &Path, checkpoint: &str) -> anyhow::Result<()> { | ||
| // The clean-workspace precondition makes these scoped resets recoverable: | ||
| // only changes made by the rejected iteration can exist at this point. | ||
| git_command(workspace, &["reset", "--hard", checkpoint]).await?; |
There was a problem hiding this comment.
When an experiment modifies files in an initialized submodule, this reset restores only the superproject because it omits recursive submodule updating, and the following git clean -fd does not reset the tracked submodule working tree. Local git reset -h confirms that --recurse-submodules is the option controlling recursive submodule updates, so rejected changes otherwise survive into later measurements and can corrupt acceptance decisions.
Useful? React with 👍 / 👎.
| configure_process_group(command); | ||
| command | ||
| .stdout(std::process::Stdio::piped()) | ||
| .stderr(std::process::Stdio::piped()); | ||
| let child = command | ||
| .kill_on_drop(true) | ||
| .spawn() |
There was a problem hiding this comment.
After the model has edited the workspace, validation, metric, and Git commands all reach this helper without clearing the controller environment, which includes credentials loaded by dotenvy. An injected change can therefore alter a referenced metric script or install a Git hook and have the trusted controller execute it with provider and channel tokens that the ordinary shell tool deliberately scrubs; apply the same secret-filtered environment here and disable repository-controlled hooks for acceptance commits.
AGENTS.md reference: AGENTS.md:L110-L114
Useful? React with 👍 / 👎.
| if let (Some(tracker), Some(usage)) = (&self.cost_tracker, response.usage.as_ref()) { | ||
| let _ = tracker | ||
| .record( | ||
| model, | ||
| TokenUsage { | ||
| input_tokens: usage.input_tokens as usize, | ||
| output_tokens: usage.output_tokens as usize, | ||
| total_tokens: usage.input_tokens as usize + usage.output_tokens as usize, | ||
| }, | ||
| ) | ||
| .await; |
There was a problem hiding this comment.
When the configured model is absent from CostTracker's hard-coded price table—as the default gpt-5.5 currently is—this newly wired usage recording calls record, whose unknown-model fallback assigns zero input and output prices. /cost and the HTTP summary consequently report paid default-model calls as costing $0; represent unknown pricing explicitly or ensure every selectable default and alias resolves to a real price rather than silently recording free usage.
Useful? React with 👍 / 👎.
1 participant
Summary
apollo autoresearchloop with baseline measurement, median sampling, validation retries/timeouts, strict improvement gates, rollback, local commits, TOML ledger persistence, resume, and wall-clock/iteration budgets.Audit notes
sh -cconfiguration and are documented as such.Validation
cargo fmt --all -- --checkcargo test --workspace --all-features(sequential, green)cargo test -p apollo-agent autoresearch --lib -- --nocapturecargo test -p apollo-agent --all-features --lib(302 tests, 297 passed, 5 ignored)cargo clippy -p apollo-agent --lib --all-features -- -D warningscargo build --release -p apollo-agent -p apollo-tuiNext steps
Open workspace in Conductor
Note
Medium Risk
Autoresearch runs trusted
sh -cmetric/validation commands and usesgit reset --hardon rejected iterations; safety relies on clean-worktree preconditions, branch/HEAD guards, and the restricted tool/memory surface rather than sandbox isolation.Overview
Adds
apollo autoresearch, a Git-backed loop that optimizes a numeric metric from a TOML spec: baseline measurement, one hypothesis per iteration, validation with retries/timeouts, median sampling, strict improvement gates, rollback on rejection, local commits on acceptance, and a durable TOML ledger with--resumechecks (branch, HEAD, spec fingerprint).Autoresearch agents are built through a new restricted automation path: only runtime and filesystem tools, plus
AgentRunner::with_memory_enabled(false)so history, personal context, ZKR, and turn persistence stay out of prompts and stores. Sharedbuild_automation_agentwires this up for the new command.The rx4 rotary bridge now feeds the existing cost tracker with per-request context-shape estimates (system/history/tool character counts and ~4 chars/token) and records provider usage from responses;
CostSummaryincludes aggregated context telemetry for comparing harness configs.Reviewed by Cursor Bugbot for commit 4ce8694. Bugbot is set up for automated code reviews on this repo. Configure here.