Qwen3 ALFWorld Harness Scaling

Private research repo for ALFWorld text-only harness experiments with Qwen3.

This repo contains:

• H1-H4 harness prompt builders and parsers.
• Random/mock/OpenAI-compatible/local-transformers policies.
• JSONL episode, step, and parser-error logging.
• Remote setup scripts for /data/fanzhaoxin/harnessscale.
• Phase-0 and pilot result summaries.

Large artifacts are intentionally excluded:

• Qwen model weights
• ALFWorld data zips / extracted datasets
• venvs
• raw long-running run directories

Key Results

See ALFWORLD_HARNESS_SCALE_RESULTS.md.

Completed pilot:

• Qwen3-1.7B x H1-H4 x 20 episodes
• Qwen3-4B x H1-H4 x 20 episodes

Main finding: Qwen3 must run with thinking disabled for this harness benchmark. Qwen3-4B is much more format-stable than Qwen3-1.7B, but both have 0.000 ALFWorld task success under the current zero-shot action policy.

Local Parser Tests

python -m unittest tests/test_alfworld_harness_parsers.py

Remote Smoke

ssh HRWL-4090 'bash /data/fanzhaoxin/harnessscale/src/scripts/run_remote_mock_smoke.sh /data/fanzhaoxin/harnessscale'

Qwen Smoke

Serve a model:

ssh HRWL-4090 'SKIP_INSTALL=1 GPU_MEMORY_UTILIZATION=0.35 CUDA_VISIBLE_DEVICES=5 bash /data/fanzhaoxin/harnessscale/src/scripts/serve_qwen_vllm.sh /data/fanzhaoxin/harnessscale /data/fanzhaoxin/harnessscale/models/Qwen3-1.7B'

Run a smoke:

ssh HRWL-4090 'MODEL_NAME=Qwen3-1.7B EPISODES=5 bash /data/fanzhaoxin/harnessscale/src/scripts/run_remote_qwen_smoke.sh /data/fanzhaoxin/harnessscale'

Run a pilot:

ssh HRWL-4090 'MODEL_NAME=Qwen3-4B EPISODES=20 bash /data/fanzhaoxin/harnessscale/src/scripts/run_remote_qwen_smoke.sh /data/fanzhaoxin/harnessscale'

Repo Hygiene

Before pushing:

git status --short
git ls-files | grep -E 'safetensors|\\.env|alfworld_data|runs|tmp' && echo "check ignore rules"