Replies: 1 comment
Here is CC generated proposal with example mode (evolve mode for circle packing problem)Proposal: CycleAnalyzer — Outer Loop Observability Problem An outer-loop optimizer needs to understand what happened in an inner-loop cycle: what agents ran, in what order, what each produced, what the evaluator said. Today this is scattered across .factory/ in different formats. Nothing assembles it into one consumable record. What we need per cycle For a circle-packing evolve cycle, the optimizer needs to see: Cycle 7: What was tried, what ran, what each produced, did it help, and where to find the evaluator's raw output. Where this data lives today Design: CycleAnalyzer Mode-agnostic. Doesn't understand evaluator output — just traces what the DAG did and points at eval artifacts. @dataclass
class AgentStep:
"""One agent's execution within a cycle."""
order: int
node_id: str # DAG node that triggered this
role: str # researcher, builder, etc.
duration_s: float
cost_usd: float | None
succeeded: bool
produced: list[str] # files written (from node.writes)
@dataclass
class CycleRecord:
"""What the optimizer sees after one inner-loop cycle."""
cycle_number: int
mode: str
# What ran, in order
steps: list[AgentStep]
# Bottom line
verdict: str # keep / revert / error
score_before: float | None # from eval.completed events or results.tsv
score_after: float | None
score_delta: float | None
hypothesis: str | None
# Eval artifacts — opaque pointers, optimizer knows what's inside
eval_artifacts: list[str]
# Totals
total_cost_usd: float
total_duration_s: float
# Trajectory context
score_trajectory: list[float]
consecutive_reverts: int
plateau_detected: boolThe optimizer reads score_delta and verdict for quick decisions. When it needs detail, it reads the eval artifacts directly — it knows its own evaluator's output format. API analyzer = CycleAnalyzer(
factory_dir=Path(".factory"),
workflow=evolve_workflow(),
eval_artifacts_glob="experiments/*/eval_result.json" # or explicit paths
)
cycle = analyzer.latest()
cycle.steps # what ran, in order, what each produced
cycle.eval_artifacts # pointers to evaluator output — optimizer reads these
cycle.verdict # keep or revert
cycle.score_delta # did it help
analyzer.trajectory() # [0.62, 0.66, 0.72, 0.78, 0.82]
analyzer.analyze() # all CycleRecordsHow it assembles a CycleRecord
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
As first step towards implementing or reusing existing logs for optimization we have a summary of all the files and what modes generate them. As rule of thumb we want files/logs with dense information with wider coverage across modes. In future, we can bring in more systematic observability like modes always emitting certain logs, step. Evaluation that is written as part of mode always giving out log in certain format etc.
Here is summary of all the files emitted by modes and their usefullness to outloop optimizer
Here is summary of table summarizing what mode emit what files
All reactions