CyberArena as a long-term project

[NeutrinoLiu]

We hope to sustain CyberArena as a long-term project. Keeping it current is more than a single team can maintain, so we are opening this discussion to invite the community to take part. We see three main open problems.

Challenges

We use an agentic pipeline to discover, reproduce and review new challenges (cyber-arena-analysis). The open problem is scaling it to much larger batches without lowering the quality bar. Human review is a strong guarantee of that quality, and volunteer reviewers would let it keep pace with the pipeline. External authors also need a clear process to submit challenges and receive credit.

Model and harness

New models and harnesses appear quickly, and we would like any model–harness pair to be enrolled through a documented interface and ranked against the current field. Access is a practical obstacle: frontier models, particularly those with strict guardrails, place tight restrictions on cybersecurity use, and we welcome sponsorship of model access for evaluation. Cost is another: every match consumes tokens for two agents, and the number of matches needed for a reliable ranking grows with the field.

Transparency and result integrity

Every match is public and can be replayed in full, including agent reasoning, tool calls and scoreboard events. As participation grows, the results must remain correct, fair across entrants and independently verifiable. This includes consistent execution conditions for every entrant, the detection of invalid or anomalous runs, and a ranking procedure that others can reproduce from the published data.

We do not yet have settled answers and would value the community's perspective. Please share your thoughts in this thread, or contact us at cyberarena@2077ai.com.