RAAS: Robust Architecture Adaptive Search

Rethinking LLM Agentic Architecture Search: A Robust Evaluation View

RAAS is a robust evaluation framework for automated LLM agentic architecture search. Instead of learning from a single absolute score for one sampled workflow, RAAS builds more reliable search signals by comparing peer architectures on the same query and aggregating multiple independent executions.

Highlights

• Robust architecture search. RAAS searches over an Agentic Supernet and adapts the architecture distribution using stable merit signals.
• Contextual Architecture Orchestration (CAO). Candidate architectures are evaluated as a cohort on the same query, producing peer-normalized merit signals that reduce task-difficulty bias.
• Multi-Trial Assessment Synthesis (MTAS). Multiple independent executions are synthesized to reduce stochastic workflow variance.
• Broad evaluation. The paper evaluates RAAS on mathematical reasoning, code generation, and multi-step tool-use benchmarks including GSM8K, MATH, MultiArith, HumanEval, MBPP, and GAIA.

Method overview

RAAS addresses two common instabilities in automated agentic workflow optimization:

1. Task-difficulty entanglement: absolute scores mix architecture quality with query difficulty.
2. Execution variance: one stochastic run may not represent the true capability of an architecture.

RAAS therefore uses:

• CAO to evaluate multiple candidate architectures on an identical query and compute relative, context-aware advantages.
• MTAS to execute each candidate multiple times and aggregate the outcomes into a more stable capability estimate.
• Merit-weighted adaptation to reinforce architecture patterns that consistently outperform their peer group under the same query context.

Implementation note: this early code release builds on the MaAS-style Agentic Supernet codebase. Some scripts and folders still retain maas names from the upstream baseline, while the package namespace in this repository is raas.

Repository structure

.
├── config/                       # Example model and runtime configuration
├── examples/maas/                # Optimization entry points and example assets
│   ├── optimize.py               # Baseline Agentic Supernet optimizer
│   └── optimize-grpo.py          # Robust/group-normalized optimizer entry point
├── raas/
│   ├── actions/                  # Agent actions and workflow primitives
│   ├── configs/                  # Configuration loaders
│   ├── ext/raas/                 # RAAS/MaAS benchmark and optimizer extensions
│   │   ├── benchmark/            # Dataset wrappers and experiment configs
│   │   ├── models/               # Controller and robust sampling utilities
│   │   └── scripts/              # Optimizers, evaluators, optimized graphs
│   ├── provider/                 # LLM provider integrations
│   ├── rag/                      # RAG components
│   └── tools/                    # Tool-use utilities
└── GRPO_USAGE_GUIDE.md           # Additional command examples

Installation

git clone https://github.com/ridlog/raas.git
cd raas

conda create -n raas python=3.8 -y
conda activate raas
pip install -U pip

Install the dependencies required by your target backend and benchmark. The included requirements.txt is an environment export from the development environment, so you may use it as a reference when recreating the full environment.

Configuration

Copy the example configuration and fill in the model credentials/endpoints you plan to use:

cp config/config2.example.yaml config/config2.yaml
# edit config/config2.yaml

At minimum, configure the optimization model and execution model used by the search scripts, for example OpenAI-compatible or local model endpoints.

Quick start

Run baseline Agentic Supernet optimization

cd examples/maas
python optimize.py --dataset GSM8K --sample 4 --batch_size 4

Run robust group-normalized optimization

cd examples/maas
python optimize-grpo.py --dataset MATH \
  --sample 4 \
  --max_sampling_rounds 8 \
  --success_threshold 2 \
  --noise_scale 0.03 \
  --cost_weight 2.0

Test an optimized workflow

cd examples/maas
python optimize-grpo.py --dataset HumanEval --is_test

The bundled experiment configuration currently exposes MATH, GSM8K, and HumanEval in raas/ext/raas/benchmark/experiment_configs.py. Add new benchmark wrappers and update this config to reproduce additional paper settings.

Custom robust-search configuration

You can place robust sampling and normalization hyperparameters in a YAML file:

grpo:
  max_sampling_rounds: 8
  success_threshold: 3
  architecture_noise_scale: 0.03
  cost_weight: 2.0
  normalization_strategy: group_mean
  track_architecture_diversity: true
  diversity_penalty: 0.1

Then run:

cd examples/maas
python optimize-grpo.py --dataset MATH --grpo_config path/to/grpo_config.yaml

Citation

If you find this repository useful, please consider citing:

@misc{yang2026raas,
  title  = {Rethinking LLM Agentic Architecture Search: A Robust Evaluation View},
  author = {Yang, Jiayi and Wan, Guancheng and Zhang, Man and Ye, Mang},
  year   = {2026}
}

Acknowledgements

This repository builds on the Agentic Supernet / MaAS codebase and related LLM-agent workflow optimization research. We thank the authors and maintainers of the open-source agent frameworks and benchmarks used in this project.