ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving

📁 Project Structure

Code

/fs/scratch/rb-bd-dlp-rng-dl01-cr-tfx/special/tfx-901/shz1syv/codes/ExploreVLA/

Datasets

• NavSim (RGB + Depth) /fs/scratch/rb-bd-dlp-rng-dl01-cr-tfx/special/tfx-901/shz1syv/navsim

• nuScenes (RGB) /fs/scratch/rb-bd-dlp-rng-dl01-cr-tfx/special/tfx-901/archive/aid_005/nuScenes

• nuScenes (Depth) /fs/scratch/rb-bd-dlp-rng-dl01-cr-tfx/special/tfx-901/shz1syv/nuscenes/

• PWM Dataset /fs/scratch/rb-bd-dlp-rng-dl01-cr-tfx/special/tfx-901/shz1syv/pwm_datasets

• NavSim v1 Training Metric Cache /fs/scratch/rb-bd-dlp-rng-dl01-cr-tfx/special/tfx-901/shz1syv/hf/ReCogDrive_Metric_Cache

• NavSim v1/v2 Test Metric Cache /fs/scratch/rb-bd-dlp-rng-dl01-cr-tfx/special/tfx-901/shz1syv/navsim/exp

Model Checkpoints

• Stage 1 (Imitation Learning / SFT) .../codes/ExploreVLA/resume_checkpoint_sft

• Stage 2 (Reinforcement Learning) .../codes/ExploreVLA/resume_checkpoint

---

🚀 Training

Single A100 (Debug / Development)

Request an A100 GPU, load environment, and start training:

bsub -Is -q inter_a100 -n 4 -M 32000 -W 12:00 -gpu "num=1:mig=7" /bin/bash
module load conda/4.11.0 cuda/12.4 gcc/10.2.0
conda activate /fs/scratch/rb-bd-dlp-rng-dl01-cr-tfx/special/tfx-901/shz1syv/.conda/envs/showo

bash scripts/finetune/navsim/run_sft_navsim_baseline_1a100.sh

The conda environment is based on Show-o.

---

🐞 Debugging (Remote Attach)

Update the host according to your assigned node host: rng-dl01-w0xx:

        {
            "name": "Python: Remote Attach",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "rng-dl01-w089",
                "port": 9501
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "${workspaceFolder}"
                }
            ]
        }

Enable debug mode in config:

experiment:
    is_debug: true

---

🖥️ Cluster Training

cd bsub

# Stage 1: Imitation Learning
bsub < navsim.bsub  

# Stage 2: Reinforcement Learning
bsub < navsim_rl.bsub  

---

⚙️ Configuration

Use my_navsim.yaml for Stage 1.

Stage 1: Pre-training

Task: image + action → image

experiment:
    stage: 'pretrain'
    eval_only: false
    is_debug: false

---

Stage 1: Supervised Fine-Tuning (SFT)

Task: image → action + image

experiment:
    stage: 'sft'
    eval_only: false
    is_debug: false

---

📊 Evaluation

NAVSIM v1 (PDMS)

bash scripts/finetune/navsim/run_sft_navsim_rl_eval_predefine_std.sh

⚠️ Make sure to load the Stage 2 RL checkpoint from: .../resume_checkpoint/...

---

NAVSIM v2 (EPDMS)

1. Rename original navsim folder to avoid conflicts:

   navsim → navsim_v1
   

2. Optional (already downloaded): Download NavSim v2.2 repo and place only the navsim folder here

3. Optional (already recomputed): Recompute metric cache (v1 and v2 caches are NOT compatible)

4. Run evaluation:

python my_run_pdm_score_v2.py

⚠️ After evaluation, rename the folder back if needed.

---

🙏 Acknowledgements

• Show-o
• Policy World Model (PWM)