SETA (Scaling Environments for Terminal Agents) is an ORS environment for evaluating terminal-based coding and system administration capabilities, developed by CAMEL-AI. It contains 1376 tasks covering software engineering (622 tasks), system administration (605 tasks), DevOps (33 tasks), debugging (29 tasks), configuration (35 tasks), security (20 tasks), networking (15 tasks), and other categories. Each task is validated by a pytest test suite with weighted scoring.
- Terminal-based task completion
- Software engineering automation
- System administration and user management
- DevOps, security, and networking tasks
Agents are given a sandboxed environment with CLI tools (bash, read, write, edit, grep, glob, ls). Uses custom Docker image with Ubuntu 24.04.
There is one split in this environment:
- train: 1376 tasks
Difficulty distribution: Easy (25), Medium (629), Hard (722). Tasks span software engineering, system administration, DevOps, security, networking, debugging, and configuration categories.
This is a multi-turn environment with pytest-based validation. The agent uses CLI tools to complete terminal tasks, then calls submit_solution to run tests. Each pytest test has a specific weight, and the final reward is the weighted sum of passed tests (0.0 to 1.0).
Data consists of the SETA Dataset directory sourced from GitHub camel-ai/seta-env. Each task includes task.yaml, weights.json, tests/test_outputs.py, and required data files. Data is stored on the OpenReward platform.
| Tool | Description |
|---|---|
submit_solution |
Run pytest tests and get weighted score. Ends the episode. |
bash |
Execute shell commands in sandbox. |
read |
Read file contents. |
write |
Write files. |
edit |
Edit existing files. |
grep |
Search file contents. |
glob |
Find files by pattern. |
ls |
List directory contents. |
Multi-turn. Agents explore files, write scripts, execute commands, then submit for test validation.
[Put environment difficulty information here]
None.
Agents in SETA operate within sandboxed environments. Commands are executed in isolated containers with controlled filesystem access.
@misc{seta,
author = {Qijia Shen and Jay Rainton and Aznaur Aliev and Ahmed Awelkair and Boyuan Ma and Zhiqi (Julie) Huang and Yuzhen Mao and Wendong Fan and Philip Torr and Bernard Ghanem and Changran Hu and Urmish Thakker and Guohao Li},
title = {{SETA: Scaling Environments for Terminal Agents}},
year = {2026},
month = jan,
url = {https://github.com/camel-ai/seta},
note = {Blog: \url{https://eigent-ai.notion.site/SETA-Scaling-Environments-for-Terminal-Agents-2d2511c70ba280a9b7c0fe3e7f1b6ab8}}
}