trajectories

Trajectories

The trajectories/ folder is the default location that experiment results (invocations of run.py) will be written to.

At a high level, the experiments folder is organized in the following manner:

trajectories
├── <user 1> 👩‍💻
│ ├── <experiment 1> 🧪
│ │ ├── all_preds.jsonl
│ │ ├── args.yaml
│ │ ├── *.html (Webpage Files)
│ │ └── *.traj (Trajectories)
│ └── <experiment 2> 🧪
│   ├── all_preds.jsonl
│   ├── args.yaml
│   ├── *.html (Webpage Files)
│   └── *.traj (Trajectories)
├── <user 2> 👨‍💻
│ ├── <experiment 1> 🧪
│ │ └── ...
│ └── <experiment 2> 🧪
│   └── ...
...

Where every experiment follows the pattern trajectories/<user name>/<experiment name>. The <user name> is automatically inferred from your system, and the experiment name is inferred from the arguments of the run.py.

How an Experiment Folder is Generated

Each call to run.py produces a single trajectories/<user name>/<experiment name> folder containing the following assets:

• all_preds.jsonl: A single file containing all of the predictions generated for the experiment (1 prediction per task instance), where each line is formatted as:

{
    "instance_id": "<Unique task instance ID>",
    "model_patch": "<.patch file content string>",
    "model_name_or_path": "<Model name here (Inferred from experiment configs)>",
}

• args.yaml: A summary of the configurations for the experiment run.
• <instance_id>.traj: A .json formatted file containing the (thought, action, observation) turns generated by SWE-agent towards solving <instance_id>.
• <instance_id>.html: An .html single webpage render of the trajectory, which can be directly opened in the browser for easier viewing of the trajectory.

⚠️ Notes

• Evaluation is not completed by run.py, it is a separate step.
• all_preds.jsonl can be referenced directly into evaluation/run_eval.sh to run evaluation.