SecQue is an environment for evaluating financial analysis capabilities on SEC filings. It contains 565 expert-written questions covering comparison analysis, ratio calculation, risk assessment, and financial insight generation. Agents receive questions with relevant SEC filing context and must provide detailed analytical answers.
- Financial document analysis
- SEC filing interpretation
- Ratio calculation and comparison analysis
- Risk assessment and financial insight generation
Agents are given a standard environment with no sandbox or file system access.
MIT.
There is one split in this environment:
- test: 565 tasks
Questions cover companies including Discover Financial Services, NextEra Energy, Apple Inc., and NVIDIA. Context is provided as markdown-formatted SEC filing excerpts.
This is a single-turn environment. The agent submits an answer via the answer tool. An LLM grader (gpt-5-mini) evaluates the response against expert reference answers, allowing for reasonable numerical rounding and minor formatting variations. Reward is binary: 1.0 if correct, 0.0 if incorrect.
Data consists of a Parquet file (secque.parquet) sourced from HuggingFace nogabenyoash/SecQue. Each row contains a question, SEC filing context in markdown format, and expert reference answer. Data is stored on the OpenReward platform.
| Tool | Description |
|---|---|
answer |
Submit your detailed financial analysis answer. Ends the episode. |
Single-turn. The agent reads the question and SEC filing context, then submits one answer.
Model performance on SECQUE:
| Model | Accuracy |
|---|---|
| GPT-4o | 69% |
| Llama-3.3-70B-Instruct | 65% |
| GPT-4o-mini | 64% |
| Qwen2.5-32B-Instruct | 61% |
| Phi-4 | 56% |
| Meta-Llama-3.1-8B-Instruct | 48% |
Ratio analysis and analyst insights are the most challenging categories, requiring complex numerical reasoning combined with financial understanding.
OpenAI API key required for LLM-based grading. Pass via secrets={"openai_api_key": "..."}.
Agents in SecQue analyze SEC filings in a standard environment. The environment does not present direct safety risks.
@article{benyoash2025secque,
title={SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities},
author={Ben Yoash, Noga and Brief, Meni and Ovadia, Oded and Levin-Schwartz, Yael and Eldan, Ronen},
journal={arXiv preprint arXiv:2504.04596},
year={2025}
}