Quick Start

The Code Repo is for Paper Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training

Before running any experiment, make the following two changes:

1. The scripts in these folder use /home/ubuntu/CoEA_exp as the default project path. Replace it with the absolute path to the project on the target machine.
2. The scripts in these folder use 10.16.51.39:8000 for the local search service and 10.16.51.39:9000 for the Python Sandbox. Update these endpoints if the Tool services are deployed elsewhere.

1. Install the Environments

The project was validated with NVIDIA Driver 580.159.04 and CUDA 13.0. Ensure that GPU packages such as PyTorch and vLLM in the CoEA and GRPO environments are built for CUDA 13.0 rather than cu128.

export PROJECT_ROOT=/absolute/path/to/CoEA_exp

# Install the Tool environment shared by local search and the Sandbox broker
conda create -n CoEA_search python=3.12 -y
conda activate CoEA_search
python -m pip install --upgrade pip
python -m pip install -r "$PROJECT_ROOT/requirements-coea-search.txt"
python -m pip install hf_transfer
conda install -c pytorch -c nvidia faiss-gpu=1.8.0 -y

# Install the CoEA environment
conda create -n CoEA python=3.11 -y
conda activate CoEA
python -m pip install --upgrade pip
python -m pip install -r "$PROJECT_ROOT/requirements-coea-train.txt"

# Install the GRPO environment
conda create -n GRPO python=3.11 -y
conda activate GRPO
python -m pip install --upgrade pip
python -m pip install -r "$PROJECT_ROOT/requirements-grpo-train.txt"
python -m pip install -r "$PROJECT_ROOT/requirements-grpo-vllm.txt"
python -m pip install flash-attn --no-build-isolation

# Verify the driver, PyTorch version, and CUDA build used by PyTorch
nvidia-smi
python -c "import torch; print(torch.__version__, torch.version.cuda)"

2. Prepare Data and Start the Tool Services

Download the base models and the Wiki-18 corpus, E5 retriever, and FAISS index required by local search. The preprocessed dataset splits used in the paper are already included under data.

The Local Search service uses the code from Search-R1

export PROJECT_ROOT=/absolute/path/to/CoEA_exp

conda activate CoEA_search
cd "$PROJECT_ROOT"

# Download the Qwen3.5 base models
bash src/download_scripts/download_llm.sh

# Download Wiki-18, the E5 retriever, and the FAISS index
bash src/download_scripts/download_local_search_data.sh

# Build the Docker image used by the Python Sandbox
bash src/tools/sandbox/build_python_sandbox_image.sh

Start the local search service in the first dedicated terminal:

conda activate CoEA_search

export PROJECT_ROOT=/absolute/path/to/CoEA_exp
export DOWNLOAD_DIR="$PROJECT_ROOT/downloads"
export PYTHONPATH="$PROJECT_ROOT/src:${PYTHONPATH:-}"
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1

cd "$PROJECT_ROOT/src/tools/local_search_service/search_r1/search"

python retrieval_server.py \
  --index_path "$DOWNLOAD_DIR/e5_Flat.index" \
  --corpus_path "$DOWNLOAD_DIR/wiki-18.jsonl" \
  --topk 5 \
  --retriever_name e5 \
  --retriever_model "$DOWNLOAD_DIR/e5-base-v2" \
  --faiss_gpu

Start the Python Sandbox broker in a second dedicated terminal:

conda activate CoEA_search

export PROJECT_ROOT=/absolute/path/to/CoEA_exp
export PYTHONPATH="$PROJECT_ROOT/src:${PYTHONPATH:-}"
export SANDBOX_DOCKER_IMAGE=coea-python-sandbox:3.11
export SANDBOX_CPUS=4
export SANDBOX_MEMORY=10g
export SANDBOX_PIDS_LIMIT=64
export SANDBOX_USER=10001:10001
export SANDBOX_TMPFS_SIZE=10g
export TMPDIR="$PROJECT_ROOT/tmp"

mkdir -p "$TMPDIR"
cd "$PROJECT_ROOT/src"

uvicorn tools.sandbox.sandbox_broker:app \
  --host 0.0.0.0 \
  --port 9000

3. Run the Experiments

Keep both Tool services running, then run the required scripts from a new terminal:

export PROJECT_ROOT=/absolute/path/to/CoEA_exp
export PYTHONPATH="$PROJECT_ROOT/src:${PYTHONPATH:-}"

# The release scripts default to /home/ubuntu/CoEA_exp. Replace it once with
# the actual absolute project path before the first run.
sed -i "s#/home/ubuntu/CoEA_exp#$PROJECT_ROOT#g" "$PROJECT_ROOT"/cmd/*.sh

cd "$PROJECT_ROOT/src"

conda activate GRPO

# Train the full-parameter GRPO model on the mathematics task
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_grpo_math_with_kl.sh"

# Train the LoRA GRPO model on the mathematics task and export the merged HF model
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_grpo_math_with_kl_lora.sh"

# Train the full-parameter GRPO model on the QA task
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_grpo_qa_with_kl.sh"

conda activate CoEA

# Train the subset=4 OpenAI-ES model on the mathematics task
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_oai_es_coea_math_subset4_chunk_split_update_together.sh"

# Train the standard OpenAI-ES model on the mathematics task
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_oai_es_coea_math.sh"

# Run the subset=2 OpenAI-ES ablation on the mathematics task
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_oai_es_coea_math_subset2_chunk_split_update_together.sh"

# Run the subset=8 OpenAI-ES ablation on the mathematics task
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_oai_es_coea_math_subset8_chunk_split_update_together.sh"

# Run the subset=4 OpenAI-ES ablation with independent z-score normalization
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_oai_es_coea_math_subset4_chunk_split_update_together_no_joint_z_score.sh"

# Run the OpenAI-ES ablation with sigma=2e-3
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_oai_es_coea_math_sigma2e-3.sh"

# Train the subset=4 OpenAI-ES model on the QA task
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_oai_es_coea_qa_subset4_chunk_split_update_together.sh"

# Train the standard OpenAI-ES model on the QA task
bash "$PROJECT_ROOT/cmd/qwen3_5_4b_oai_es_coea_qa.sh"

# Evaluate the Qwen3.5-4B base model on the mathematics task
bash "$PROJECT_ROOT/cmd/eval_math_qwen_3_5_4b.sh"

# Evaluate the Qwen3.5-9B base model on the mathematics task
bash "$PROJECT_ROOT/cmd/eval_math_qwen_3_5_9b.sh"

# Evaluate the full-parameter GRPO model on the mathematics task
bash "$PROJECT_ROOT/cmd/eval_math_grpo.sh"

# Evaluate the merged LoRA GRPO model on the mathematics task
bash "$PROJECT_ROOT/cmd/eval_math_grpo_lora.sh"

# Evaluate the subset=4 OpenAI-ES model on the mathematics task
bash "$PROJECT_ROOT/cmd/eval_math_oai_es_subset4.sh"

# Evaluate the standard OpenAI-ES model on the mathematics task
bash "$PROJECT_ROOT/cmd/eval_math_oai_es.sh"

# Evaluate the subset=2 OpenAI-ES ablation on the mathematics task
bash "$PROJECT_ROOT/cmd/eval_math_oai_es_subset2.sh"

# Evaluate the subset=8 OpenAI-ES ablation on the mathematics task
bash "$PROJECT_ROOT/cmd/eval_math_oai_es_subset8.sh"

# Evaluate the subset=4 ablation with independent z-score normalization
bash "$PROJECT_ROOT/cmd/eval_math_oai_es_no_joint_z_score.sh"

# Evaluate the OpenAI-ES ablation with sigma=2e-3
bash "$PROJECT_ROOT/cmd/eval_math_oai_es_sigma2e-3.sh"

# Evaluate the full-parameter GRPO model on the QA task
bash "$PROJECT_ROOT/cmd/eval_qa_grpo.sh"

# Evaluate the Qwen3.5-4B base model on the QA task
bash "$PROJECT_ROOT/cmd/eval_qa_qwen_3_5_4b.sh"

# Evaluate the subset=4 OpenAI-ES model on the QA task
bash "$PROJECT_ROOT/cmd/eval_qa_oai_es_subset4.sh"

# Evaluate the standard OpenAI-ES model on the QA task
bash "$PROJECT_ROOT/cmd/eval_qa_oai_es.sh"

For environment setup, data preparation, Tool service configuration, and the complete parameter reference, see full_workflow_usage_en.md.