💬 Feedback on Proposal and Harbor Agent Skills

[Zhuofeng-Li]

Thank you for trying our skills! We appreciate your feedback. 🙌

Note

• All contributors (with or without GPU access): proposal-agent, harbor-task-agent
• Contributors with GPU access: harbor-task-validator

Please leave a comment below using the following template:

## Proposal Agent

<!-- Your feedback -->

## Harbor Task Agent

<!-- Your feedback -->

## Harbor Task Validator

<!-- Your feedback -->

[Django-Jiang]

Feedback on proposal-agent and the end-to-end contribution workflow

Contribution: Discussion #16 → RSI-Index/agentdojo-attack-panel-d16 (targeted prompt-injection attacks against AgentDojo's fixed defense panel, domain-expertise route). Driven end-to-end by Claude Code with a human contributor. The proposal reached Accept in 5 rubric rounds; task construction then took 6 /task repair rounds after /task confirm. Sharing where the time actually went, honestly including our own mistakes.

On the "under 1 hour" expectation

The estimate looks realistic for the path it describes — a coauthor whose baseline, evaluation protocol, and action space are already fixed in their head, on a task that hits none of the system's weak spots. That was not our path, for reasons partly ours and partly the workflow's:

• We designed while reading the codebase, so Rounds 1–5 were real scientific decisions, not transcription.
• Almost all elapsed time was after proposal Accept, in task construction. The "1 hour" framing (and the README) largely stops at Accept, so the construction phase sets no expectation at all.

What consumed the time, and whose fault it was

Proposal phase (5 rounds) — mostly well spent. The rubric reviewer caught three real defects we had missed: candidate code being imported into the shared Judge process, infrastructure-error cells being droppable from the scoring denominator, and an unenforceable "hidden workspace suite" claim. That is the reviewer doing its job. No complaint.

Construction phase (6 rounds) — mostly avoidable. Blockers in order: (1) source-pin syntax + a relative model identifier in instructions; (2) the four no-attack utility references needing GPU calibration the read-only builder cannot do; (3) the Docker RUN block being split instead of single; (4) first-run-then-freeze calibration requiring cross-submission state the execution model does not provide; (5) the HF model source judged mutable because a git-LFS checkout pins only pointer files; (6) git clone source-pin syntax not recognized.

Four of those six (1, 3, 5, 6) came from one root cause on our side: we modeled the task on tasks/ (Terminal-Bench) instead of rsi-tasks/. The correct forms for every one of them already exist verbatim in the two accepted RSI tasks — rsi-tasks/learnability-cot's git init + fetch --depth=1 <sha> + checkout --detach FETCH_HEAD + test rev-parse single-RUN block and its requirements.lock, and rsi-tasks/molmo2-pointing-refined's digest-pinned FROM ...@sha256: base plus per-file SHA-256 model manifest. Reading those two Dockerfiles first would have collapsed four rounds to zero.

Concrete suggestions

1. Point contributors at rsi-tasks/ as the implementation reference before /task. The single highest-value fix. One line — "implementation syntax follows the accepted tasks in rsi-tasks/, not tasks/" — or having the builder template directly from them, removes the majority of our construction churn. proposal-agent never surfaced this distinction.

2. Expose the construction validator's rules. The validator lives in the private RSI-Index/RSI-Skills repo (404 for contributors), so each syntax rule — single RUN, --detach, git init over git clone, per-file digest manifest for LFS-backed models — had to be reverse-engineered by submit→fail→read. Error text like "rejects the immutable source-checkout syntax" states the rejection without the accepted shape. Publishing these rules (or the check scripts) turns a ~20-minute-per-iteration black-box loop into a static checklist.

3. Deepen the repair budget and return all diagnostics at once. The builder stops at its repair limit exposing one blocker at a time; Round 4 cited two defects ("fixed-runtime integrity and retry-path defects") without naming them until asked. Serial single-blocker exposure multiplies wall-clock by (#blockers × ~20 min). Reporting every failing check per round — with the concrete file/value/path each concerns — would let a contributor fix them in one pass.

4. Document the no-GPU-path boundary for calibration-dependent tasks. Our reward contract needed a per-defense utility floor → needed measured references → the builder has no GPU → deadlock. It took three rounds (build-time? cross-submission freeze? in-run recompute?) to land on computing the references deterministically inside each Judge run. The README's no-GPU path and this class of task (any reward needing empirical calibration) have an unstated conflict worth documenting, ideally with the in-run-recompute pattern as the sanctioned resolution.

What worked well

• The rubric reviewer's adversarial evaluation-integrity checks were genuinely sharp and caught real reward-hacking paths we had not seen.
• proposal-agent's repository-first discipline and its refusal to invent the baseline/metric kept the proposal honest.
• The eight-gate breakdown returning every gate's status (not stopping at the first failure) made each iteration's target unambiguous.

Net: the system is strong at catching real defects and weak at transparency and up-front guidance. The single change with the most leverage is directing contributors to rsi-tasks/ as the implementation model before construction begins.