Skip to content

feat(api): add avg tool calls and avg model calls - #5

Merged
JinnanDuan merged 2 commits into
openBitFun:mainfrom
Peanut-Puff:feat/subagent-metadata-backfill
Jun 10, 2026
Merged

feat(api): add avg tool calls and avg model calls#5
JinnanDuan merged 2 commits into
openBitFun:mainfrom
Peanut-Puff:feat/subagent-metadata-backfill

Conversation

@Peanut-Puff

@Peanut-Puff Peanut-Puff commented Jun 10, 2026

Copy link
Copy Markdown

Summary by CodeRabbit

  • New Features
    • Added trajectory statistics (average tool calls and model calls) to job summary display
    • Enabled visualization of nested subagent trajectories within observation results in the trajectory viewer

@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f62cb26f-ea87-4bea-9985-6a192184ebc7

📥 Commits

Reviewing files that changed from the base of the PR and between d43f13f and 09efe55.

📒 Files selected for processing (7)
  • apps/viewer/app/lib/api.ts
  • apps/viewer/app/lib/types.ts
  • apps/viewer/app/routes/job.tsx
  • apps/viewer/app/routes/trial.tsx
  • src/harbor/agents/installed/bitfun_cli.py
  • src/harbor/viewer/server.py
  • tests/unit/agents/installed/test_bitfun_cli.py

📝 Walkthrough

Walkthrough

The PR introduces support for tracking and displaying subagent trajectory relationships. It extends the data model to include subagent references and metadata provenance, backfills those references from child session metadata on the backend, aggregates trajectory statistics, and displays both metrics and embedded subagent traces throughout the frontend.

Changes

Subagent Trajectory References and Metrics

Layer / File(s) Summary
Data Model Extensions for Subagent References
apps/viewer/app/lib/types.ts
SubagentTrajectoryRef interface is introduced with optional trajectory, session, and path identifiers plus metadata. ToolCall, Step, TrajectoryAgent, and ObservationResult each gain optional extra metadata fields. ObservationResult adds optional subagent_trajectory_ref[] array. Trajectory adds optional trajectory_id and recursive subagent_trajectories[] fields.
Backend Metadata Backfill for Subagent References
src/harbor/agents/installed/bitfun_cli.py, tests/unit/agents/installed/test_bitfun_cli.py
_attach_subagent_refs_from_metadata() scans child subagent session metadata files, extracts parent tool-call identifiers from relationship fields, matches those identifiers to parent session steps, and populates SubagentTrajectoryRef entries in observation results. Deduplicates against already-referenced subagent ids and annotates both parent Step.extra and inserted refs with provenance flags. Unit test validates backfill behavior.
Backend Trajectory Statistics Aggregation
src/harbor/viewer/server.py
New GET endpoint /api/jobs/{job_name}/trajectory-stats scans trial directories for agent/trajectory.json files, aggregates tool-call and model-call counts across trajectory steps, and returns per-job trajectory count and average metrics (rounded to 1 decimal place) or None when no valid trajectories exist.
Frontend Job Page Statistics Display
apps/viewer/app/lib/api.ts, apps/viewer/app/routes/job.tsx
TrajectoryStats client-side type and fetchTrajectoryStats() function call the backend endpoint. Job page fetches trajectory stats via React Query and conditionally renders avg_tool_calls and avg_model_calls metrics in the job header row.
Frontend Subagent Trace Rendering in Trials
apps/viewer/app/routes/trial.tsx
Adds findSubagentTrajectory() helper, SubagentTraceList accordion component, and SubagentTrace component to resolve and render embedded subagent traces within trial steps. Updates StepContent to accept optional subagentTrajectories prop and rewires observation result rendering to include subagent trace lists alongside content renderers.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • JinnanDuan/bitfun-harbor#4: Includes the same subagent metadata backfill logic and corresponding type/rendering/test updates for subagent trajectory references.

Poem

🐰 A rabbit hops through trajectories deep,
Finding lost trails where subagents creep.
With metadata whispers and stats to compile,
The viewer now shows every nested beguile! 🌿

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@JinnanDuan
JinnanDuan merged commit 5419b20 into openBitFun:main Jun 10, 2026
1 check was pending
@Peanut-Puff
Peanut-Puff deleted the feat/subagent-metadata-backfill branch June 10, 2026 08:24
Messimeimei pushed a commit to Messimeimei/bitfun-harbor that referenced this pull request Jul 7, 2026
* Adding Cline V2

* Adding Cline Core which is just V2 with just bash tool

* Adding Cline Core extra logic

* Remove cline core and keep cline v2

* Revert modal changes from cline v2 branch

* feat(cline-v2): ATIF trajectories + output cleanup (openBitFun#5)

* feat(cline): emit ATIF trajectories

Adds ATIF-v1.6 trajectory support for the Cline CLI agent. The agent now
copies its session directory from ~/.cline/data/sessions/ into
/logs/agent/cline-sessions/ at the end of a run, then converts the
native messages.json into a Trajectory and writes it to
logs_dir/trajectory.json.

Tool-result user messages are folded into the preceding agent step's
observation so tool_call <-> result correlation matches the spec.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(cline): preserve tool_results in mixed-content user messages

A user message containing both text and tool_result blocks silently
dropped the tool_results: the first guard required `not text_parts`
and the second required `not message_text`, so neither branch fired
and the step was emitted with only the text.

Always attempt to attach tool_results to the prior agent step's
observation, fold any unmatched orphans into the message text, and
skip emitting the step only when nothing is left.

Adds tests for both the mixed-content and orphan-with-text cases.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* cleanup(cline): drop redundant sort and scope session copy to latest only

- _find_session_messages_file: remove sorted() before max(); max alone does
  the selection.
- run() cleanup: copy only the most recently modified session subdirectory
  into /logs/agent/cline-sessions/ instead of the entire sessions/ tree,
  so long-lived or reused environments don't bloat the log artifact.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(cline-v2): emit ATIF trajectories and scope session copy

The cline-v2 agent is a sibling class to cline-cli and was missing the
same trajectory emission + scoped session copy applied in earlier
commits. Mirror the changes here so runs of cline-v2 produce
trajectory.json and a cline-sessions/ dir under logs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* revert(cline-cli): drop trajectory changes from cline.py

cline-v2 is the agent that actually runs; keep the PR focused on v2.py
and the shared trajectory.py module. cline-cli can be wired up in a
follow-up if needed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* cleanup(cline-v2): drop redundant agent output artifacts

Harbor only consumes trajectory.json from an agent's logs_dir. The rest
of what cline-v2 was emitting was cline-specific observability with no
downstream reader:

- prompt-meta.pre.json  (full content already logged verbatim in job.log
                         via the heredoc command)
- prompt.debug.txt      (one int; duplicated in meta)
- system_prompt.latest.md, prompt_manifest.latest.json
                        (copies of the sole task-*.{system_prompt.md,
                         manifest.json} files — trivially redundant since
                         each trial runs exactly one task)
- taskHistory.json      (superseded by cline-sessions/*.messages.json)
- api_history/          (legacy pre-v2 task format; messages.json covers it)
- artifacts.index.txt   (ls -lah dump; reproducible)

Cleanup command is now just the scoped cline-sessions/ copy. Also drop
_build_pre_run_metadata and _infer_task_id_from_trial_id which became
dead after the prompt-meta write went away.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(cline-v2): persist install step stdout/stderr to setup/<label>.log

Harbor's BaseInstalledAgent creates an empty logs_dir/setup/ for every
installed agent; the directory is meant for install-time artifacts but
cline-v2 was leaving it empty. Write one log file per setup step keyed
by retry_label (install-root-prereqs.log, install-agent-runtime.log)
with the command's captured stdout/stderr, so failures and version
output are persisted to disk instead of only reaching the trial-level
debug logger.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(cline-v2): bookend setup logs with start/end timestamps

Add ISO-8601 start/end timestamps and duration to each setup/<label>.log
header. Makes it easy to see which install step hung or took longest
without grepping the trial-level debug logger.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* cleanup(cline-v2): rename cline-sessions/ to sessions/

Match the source path name (~/.cline/data/sessions). The defensive
"cline-" prefix guarded against a namespace collision that doesn't
exist.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(cline-v2): drop --json, decouple usage extraction from trajectory

Stop passing --json to cline so cline.txt becomes a human-readable
summary instead of a 380K machine-parseable event stream. Usage
(tokens/cost) now comes from messages.json directly via a dedicated
_populate_usage_from_session method, decoupled from the trajectory
converter so observability keeps working if conversion trips on an
edge case.

Removes ~240 lines of now-dead cline.txt parsers:
_load_cline_json_records, _extract_task_id_from_records,
_load_task_history_entries, _extract_usage_from_run_result,
_extract_usage_from_agent_events, _extract_usage_from_task_history,
_coerce_int, _coerce_float.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* address gemini review: write setup logs on failure; harden glob race

- _exec_with_setup_retries now writes a per-attempt setup log on
  timeout and generic exception paths, not only on success, so
  failed install attempts are diagnosable under logs_dir/setup/.
- _find_session_messages_file wraps the mtime max() in try/except
  OSError to handle the theoretical case where a globbed file is
  removed before stat().

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

* Fix cline-v2 flag handling

* Update src/harbor/agents/installed/cline/v2.py

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Fix committed modal conflict markers

* Replace cline-v2 with cline-cli

* Handle Cline CLI local binary fallback

---------

Co-authored-by: Robin Newhouse <robin@cline.bot>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants