BigBang-v1

BigBang-v1 is a 35B-A3B agentic model built on Qwen3.6-35B-A3B, with 35B total parameters and 3B parameters activated during inference. We evaluate it on long-horizon search, coding, scientific research, and AI research.

We release the reported results, a lightweight general-agent harness with search, visit, and code_exec, and download support for the official SWE-Bench Pro and SciCode-Verified datasets.

Main results

BigBang-V1 on eight representative benchmarks spanning long-horizon search, software engineering, scientific research, and AI research. BigBang-V1 obtains the highest reported score among the selected 35B models on all eight benchmarks. It even exceeds DeepSeek V4 Pro Preview (1.6T) on FrontierScience Research, Humanity's Last Exam, PaperBench(Code-Dev) and BioMysteryBench-HD.

BigBang-v1 results on eight representative benchmarks.

Benchmark results

Comparison of BigBang-V1 with representative closed- and open-source frontier models, together with models at the 35B scale, across benchmarks for long-horizon search, coding, scientific research, and AI research. The "-" indicates the score is not publicly available or not tested.

Benchmark	Closed-Source Models			Open-Source Models				35B Models
	Claude Opus 4.8	Gemini 3.1 Pro	GPT 5.5	GLM 5.2	DeepSeek V4 Flash Preview	DeepSeek V4 Pro Preview	Step-3.7 Flash	Qwen3.6 35B-A3B	Nex-N2 mini	Agents A1	Apodex 1.0-mini	BigBang V1
Long-horizon Search
BrowseComp	84.3	85.9	84.4	68.7	73.2	83.4	75.8	67.9	74.1	48.5	73.9	76.5
xbench	61.4	65.0	72.4	65.8	62.2	64.8	50.8	32.6	57.2	52.4	61.8	58.4
Coding Tasks
SWE-Bench Pro	69.2	54.2	58.6	62.1	52.6	55.4	56.3	43.6	50.2	42.3	38.7	54.2
SciCode-V-Sub	92.3	-	95.1	84.3	83.7	90.2	-	56.5	39.0	64.1	-	68.6
SciCode-V-Main	78.1	-	90.6	70.3	68.6	78.1	-	26.6	15.6	50.0	-	50.0
Scientific Research
FS-R	45.2	24.8	58.3	52.4	37.7	40.7	37.2	11.9	36.8	38.4	29.6	46.2
HLE	57.9	51.4	52.2	54.7	45.1	48.2	47.2	36.2	38.4	46.3	45.3	50.3
BioMystery-HS	88.5	-	76.7	75.3	68.0	64.4	57.5	44.8	42.9	48.9	50.2	57.5
BioMystery-HD	42.4	-	23.5	21.6	23.5	13.7	11.8	2.0	5.9	2.0	5.9	15.7
AI Research
MLE-Bench(Lite)	63.6	-	59.1	72.7	40.9	59.1	40.9	31.8	18.2	27.3	27.3	59.1
PaperBench(Code-Dev)	-	-	64.2	63.6	40.4	50.4	36.7	30.7	14.8	17.3	20.5	53.6

We use a strict evaluation-time decontamination policy to prevent direct retrieval of benchmark artifacts or reference answers. We reject queries that explicitly target Hugging Face, filter search results from huggingface.co, hf.co, and their subdomains, block direct visit calls to those domains, and never expose reference answers in agent-visible messages or tool outputs. Comparison scores may therefore differ from official reports that use other harnesses or web-access policies.

The table matches the complete result table on the BigBang-v1 Hugging Face model card. BrowseComp, xbench, FrontierScience-Research, and HLE use the general-agent runner in this repository. SWE-Bench Pro and SciCode-Verified use their benchmark-specific official harnesses, as described below.

Supported evaluations

Script identifier	Evaluation	Official data source	Evaluation protocol
browsecomp	BrowseComp	OpenAI encrypted CSV	5× discard-all; force a final answer at the fifth-attempt limit
xbench	xbench-DeepSearch-2510	xbench encrypted CSV	Avg@5, aligned with the official default
frontierscience	FrontierScience-Research	openai/frontierscience research split	Avg@30, aligned with the official default
hle	Humanity's Last Exam, text-only	Gated cais/hle dataset	One score per text-only item
swebench-pro	SWE-Bench Pro	ScaleAI/SWE-bench_Pro test split	Evaluated with the benchmark's official mini-SWE-agent harness and evaluator
scicode-verified	SciCode-Verified	SciCode-Verified v2 release	Evaluated with the official SciCode-Verified harness; pass@1 with cumulative subproblems and all-or-nothing main-problem scoring

For BrowseComp, we use five discard-all attempts and force a final answer when the fifth-attempt limit is reached. We evaluate xbench with Avg@5 and FrontierScience-Research with Avg@30, matching their official defaults.

SWE-Bench Pro and SciCode-Verified are not run through the general search/visit/code_exec loop. SWE-Bench Pro uses the official public test split, official mini-SWE-agent harness, and official evaluator. SciCode-Verified uses its official with-background pass@1 protocol: subproblems are cumulative, and a main problem passes only when all of its scored subproblems pass.

BrowseComp and xbench are MIT licensed through their official repositories. FrontierScience is Apache-2.0. HLE is marked MIT but gated, and its dataset card asks users not to publicly share, re-upload, or distribute the dataset. The SWE-Bench Pro reference repository and mini-SWE-agent fork are MIT licensed; SciCode-Verified is distributed under Apache-2.0. We provide download scripts rather than copies of benchmark data. See THIRD_PARTY.md for source and license links.

General-agent tool policy

For BrowseComp, xbench, FrontierScience-Research, and HLE, we run each item in an OpenAI-compatible tool-calling loop with:

• search: Google Search results through Serper;
• visit: webpage reading through Jina Reader followed by goal-conditioned extraction using an explicitly configured summary-model endpoint;
• code_exec: a persistent, per-item E2B shell sandbox.

Each trajectory is capped at 500 tool calls. During evaluation, we remove Hugging Face pages from search results and block direct visit calls to those pages. The separate data-download script is not subject to this restriction.

Installation

Python 3.10 or later is required.

python -m venv .venv
source .venv/bin/activate
pip install -e .
cp .env.example .env

The package does not install a console command. Run the scripts in scripts/ directly.

Set OPENAI_BASE_URL, OPENAI_API_KEY, and OPENAI_MODEL for the evaluated model's OpenAI-compatible endpoint. SERPER_API_KEY, JINA_API_KEY, and E2B_API_KEY enable the three tools. VISIT_SUMMARY_BASE_URL, VISIT_SUMMARY_API_KEY, and VISIT_SUMMARY_MODEL configure the extraction stage in visit. Judge settings are separate and are needed only with --judge.

Serve BigBang-v1 with SGLang as follows:

python -m sglang.launch_server \
  --model-path endless-frontier/BigBang-v1 \
  --host 0.0.0.0 \
  --port 8000 \
  --context-length 262144 \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Download benchmark data

Download the three non-gated benchmarks:

python scripts/download_data.py \
  --benchmarks browsecomp xbench frontierscience

For HLE, accept the access conditions on Hugging Face, set HF_TOKEN, and run:

python scripts/download_data.py --benchmarks hle

Download the official SWE-Bench Pro test split:

python scripts/download_data.py --benchmarks swebench-pro

Download the SciCode-Verified v2 manifest, JSONL, and grading HDF5. The HDF5 is approximately 1.1 GB; all three files are checked against the release manifest and pinned MD5 values.

python scripts/download_data.py --benchmarks scicode-verified

Run the general-agent evaluations

Run one BrowseComp item once:

python scripts/run_eval.py \
  --benchmark browsecomp \
  --limit 1 \
  --repeats 1 \
  --output outputs/browsecomp-smoke.jsonl

Omit --limit for a full run. Without --repeats, the runner uses Avg@5 for xbench and Avg@30 for FrontierScience-Research.

python scripts/run_eval.py \
  --benchmark xbench \
  --output outputs/xbench.jsonl

Add --judge to score answers with an OpenAI-compatible judge. Configure the judge endpoint separately from the evaluated-model endpoint.

python scripts/run_eval.py \
  --benchmark frontierscience \
  --output outputs/frontierscience.jsonl \
  --judge

The runner writes one JSON record per item and repeat. Each record contains the benchmark, item identifier, repeat index, final response, tool-call count, and conversation trajectory. With --judge, it also contains the parsed score and raw judge response. The runner rejects an existing output path unless --overwrite is supplied.

Keep credentials in an untracked .env file or secret manager. Do not commit downloaded benchmark data, generated trajectories, reference answers, or service keys.

License

The harness is released under the Apache License 2.0. Benchmark datasets and attributed upstream components remain subject to their respective terms; see THIRD_PARTY.md.