Skip to content

orchestrate.py uses global started_at for per-kernel time budget, causing premature termination of later kernels #12

@yu200512

Description

@yu200512

Description

In orchestrate.py, cmd_record() computes time_spent_minutes for each kernel using state["started_at"] -- the timestamp when Phase B began (the entire optimization run):

# orchestrate.py line 534-546 (current upstream)
started = state.get("started_at")
if started:
    start_dt = datetime.fromisoformat(started)
    ...
    delta = now_dt - start_dt
    target["time_spent_minutes"] = round(delta.total_seconds() / 60.0)

This means kernel #5's elapsed time includes ALL time spent on kernels #1-#4. With a max_minutes_per_kernel of 120, kernel #5 can inherit 100+ minutes before running its first experiment, hitting the time budget almost immediately.

Reproduce

Run optimization on any model with 5+ kernels:

uv run python auto_optimizer.py --optimize-only ... --iterations 300

Observe:

Expected

Per-kernel time budget should start counting from when the kernel begins optimization, not from Phase B start. Kernel #5 should get as much exploration time as kernel #1.

Root Cause

Single global started_at used for all kernels. No per-kernel start timestamp.

Fix

Add kernel_started_at field to each kernel entry, set on transition. Compute time_spent_minutes from kernel_started_at instead of started_at.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions