Should AutoReclab also have a Termination mechanism for budget overruns

[dylanfandio]

Should AutoRecLab also have a termination mechanism for budget overruns?

Context

AutoRecLab currently focuses on iterative experimentation, debugging, and improvement. That is useful for exploration, but it can also lead to long-running runs when the task is difficult, the search space is large, or model/tool calls become expensive. It is already in some parts included in config.toml but there is still a lot which can be expanded.

This raises a practical question: should AutoRecLab be able to stop itself when a predefined budget is exceeded?

Motivation

A expansion of the termination mechanism could help in several situations:

• Prevent excessive API costs
• Avoid wasting time on unproductive runs
• Make experiments more predictable and easier to reproduce
• Give users better control over runtime and spending
• Support safer default behavior for longer autonomous jobs

Possible budget types

A termination rule could be based on one or more of the following:

• Token budget
• Monetary cost budget
• Wall-clock time budget
• Maximum number of failed retries or debug attempts

Open design questions

• Should AutoRecLab stop immediately (hard stop) or finish the current iteration (soft stop) first?
• Should the user be able to define budgets in config files and CLI arguments?
• Should the system produce a summary report when it stops due to budget limits?
• Should there be different defaults for development and production-like runs?

Potential implementation idea

A simple first version could be:

• Check the budget before starting a new node or LLM call
• Stop the tree search once a limit is exceeded
• Save the current best result and write a clear reason for termination

Discussion

We think this would be a useful safety feature, especially for autonomous runs that can otherwise keep going without a strong stopping condition.

What would be the best budget signal for AutoRecLab:

• time
• tokens
• money
• iteration count
• or a combination of these?