Evaluation results are significantly lower than reported in the paper

[Eku127]

Hi, thank you for sharing the model weights. I evaluated the model StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln on the val_unseen split (R2R_VLNCE_v1-3 preprocessed) using 4 × L20 GPUs and followed your evaluation pipeline closely.

However, I observed that the results are significantly lower than those reported in your paper. Please see the attached screenshots for comparison:

Interestingly, I ran the same evaluation setup for two other baselines, and their results are consistent with the paper. I was wondering if the released checkpoints for StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln are the same as those used to produce the final results reported in the paper?

I have attached my final results and looking forward to your advice and clarification—thank you!

result.json

[zouc15]

I'm encountering the same issue

[kellyiss]

Hi @Eku127 @zouc15, thanks again for raising this — we did a deeper check and found two main issues that likely explain the performance gap:

1. Bug in config:
   We discovered a bug where the num_history argument wasn’t properly passed into the model, so it defaulted to None during evaluation — this had a noticeable impact on performance. We’ve now fixed it by explicitly setting:

model.model.num_history = args.num_history

here: https://github.com/OpenRobotLab/StreamVLN/blob/2f2f8ecd77c4e4fa8ebd4555f9534f28157dc88b/streamvln/streamvln_eval.py#L531

2. Dataset version mismatch:
   We trained and evaluated our model on R2R_VLNCE_v1, so all reported results are based on the v1 test set.
   According to https://jacobkrantz.github.io/vlnce/data
   v1 contains some episodes with less reasonable initial headings, which were corrected in v1-3. Unfortunately, we overlooked that update and have been using v1 consistently for both training and evaluation.
   If you want to evaluate on v1-3, we recommend also switching your training data to match — the differences in initial headings between versions can lead to performance shifts. We’ll update the checkpoints to support R2R_VLNCE_v1-3 when things are a bit less busy.

You can follow the instructions here https://github.com/OpenRobotLab/StreamVLN/blob/main/README.md#-data-preparation to download the R2R_VLNCE_v1 val episodes.
We just re-ran evaluation with the fixed config and confirmed that our reported results are reproducible on v1.

Let us know if you have any further questions!

[kellyiss]

Note: The current evaluation script does not include voxel pruning. The success rate (SR ≈ 55.7%) reported in Table 5 of our paper excludes voxel pruning and is the result that should be reproduced

[Eku127]

Thanks a lot for the prompt clarification! I’ll rerun the evaluation with the new configuration you provided. I have a few follow-up questions I want to confirm:

1. I noticed that the model repo has also been updated — should I pull the latest version for proper evaluation?
2. It seems that results from other methods in Table 1 were originally run on v1-3 (based on my test result). If that’s the case, would you consider sharing the ckpts on v1-3 and updating the table so all methods are evaluated consistently on v1-3? (Also, many people currently default to using v1-3 for R2R Val-Unseen evaluation I think.)
3. Is it possible to provide the evaluation code with voxel prunning?

Really appreciate your quick and detailed reply— thank you again!

[kellyiss]

1. Either updating the model config or modify the evaluation code can fix the bug.

2. We've confirmed that v1-3 gives more favorable initial headings, so training there likely produces better models. The performance gap should not be large, but we agree it’s better to evaluate all methods consistently on v1-3. Since our pipeline involves two-stage training and DAgger re-collection, updating all results will take some time. For now, note that RXR is not affected by this gap — you should be able to reproduce our RXR results directly.

3. The voxel pruning code still needs some cleanup before release — we’ll share it once it’s ready.

[zouc15]

1. Either updating the model config or modify the evaluation code can fix the bug.
2. We've confirmed that v1-3 gives more favorable initial headings, so training there likely produces better models. The performance gap should not be large, but we agree it’s better to evaluate all methods consistently on v1-3. Since our pipeline involves two-stage training and DAgger re-collection, updating all results will take some time. For now, note that RXR is not affected by this gap — you should be able to reproduce our RXR results directly.
3. The voxel pruning code still needs some cleanup before release — we’ll share it once it’s ready.

@kellyiss Thank you so much for your response! @Eku127 , I used the R2R-V1 dataset, which I think explains why my results are slightly better than yours.

[Eku127]

I launched a task yesterday using the modified version of the codebase, running on the R2R-v1-preprocessed dataset, and encountered a strange but familiar issue. It seems that the agent gets confused at a certain point in the environment and doesn’t proceed correctly.

I came across this related discussion: #3 (comment), and i think following the instructions here will be a temporary fix.

At the moment, I haven’t had the chance to dig deeply into the code, so I’d appreciate any insights into what might be causing this behavior (output only two actions).

[22:07:52.235085] 256 You are an autonomous navigation assistant. Your task is to Turn around and exit the door into the workout room. Walk to the end of the first machine and turn left. Walk forward until you exit through the arch way into the living room
. Stand so you are in the living room even with the right side of the archway .  Devise an action sequence to follow the instruction using the four actions: TURN LEFT (←) or TURN RIGHT (→) by 15 degrees, MOVE FORWARD (↑) by 25 centimeters, or STOP. These a
re your historical observations <memory>.
[22:07:53.220130] <|im_start|>assistant
←→<|im_end|>
[22:07:53.220261] actions [2, 3]
[22:07:54.047105] <|im_start|>assistant
←→<|im_end|>
[22:07:54.047270] actions [2, 3]
[22:07:54.885067] <|im_start|>assistant
←→<|im_end|>
[22:07:54.885197] actions [2, 3]
[22:07:55.721102] <|im_start|>assistant
←→<|im_end|>
[22:07:55.721246] actions [2, 3]
[22:07:56.566053] <|im_start|>assistant
←→<|im_end|>
[22:07:56.566197] actions [2, 3]
[22:07:57.410130] <|im_start|>assistant
←→<|im_end|>
[22:07:57.410249] actions [2, 3]
[22:07:58.249022] <|im_start|>assistant
←→<|im_end|>
[22:07:58.249136] actions [2, 3]
[22:07:59.078242] <|im_start|>assistant
←→<|im_end|>
[22:07:59.078354] actions [2, 3]
[22:07:59.916063] <|im_start|>assistant
←→<|im_end|>
[22:07:59.916168] actions [2, 3]
[22:08:00.813957] <|im_start|>assistant
→→→→<|im_end|>
[22:08:00.814124] actions [3, 3, 3, 3]
[22:08:01.933141] <|im_start|>assistant
→→↑↑<|im_end|>
[22:08:01.933270] actions [3, 3, 1, 1]
[22:08:03.086283] <|im_start|>assistant
↑↑↑←<|im_end|>
[22:08:03.086390] actions [1, 1, 1, 2]
                                                                                                                                [22:08:04.227280] <|im_start|>assistant██████████████████████████████████████▌                   | 46/63 [20:38<06:00, 21.23s/it]
↑←←←<|im_end|>
[22:08:04.227389] actions [1, 2, 2, 2]
[22:08:04.717675] 290 You are an autonomous navigation assistant. Your task is to Turn around and exit the door into the workout room. Walk to the end of the first machine and turn left. Walk forward until you exit through the arch way into the living room. Stand so you are in the living room even with the right side of the archway .  Devise an action sequence to follow the instruction using the four actions: TURN LEFT (←) or TURN RIGHT (→) by 15 degrees, MOVE FORWARD (↑) by 25 centimeters, or STOP. These are your historical observations <memory>.
Traceback (most recent call last):
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 611, in <module>
    eval()
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 561, in eval
    evaluate(model, tokenizer, args)
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 580, in evaluate
    sucs, spls, oss, ones, ep_num = evaluator.eval_action(get_rank())
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 347, in eval_action
    outputs = self.model.generate(**input_dict, do_sample=False, num_beams=1, max_new_tokens=10000, use_cache=True, return_dict_in_generate=True, past_key_values=past_key_values)
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/home//workspace/StreamVLN/streamvln/model/stream_video_vln.py", line 379, in generate
    ) = self.prepare_inputs_labels_for_multimodal(
  File "/data/home//workspace/StreamVLN/streamvln/model/stream_video_vln.py", line 224, in prepare_inputs_labels_for_multimodal
    cur_memory_feature = memory_features[batch_idx][cur_mem_id]
TypeError: 'NoneType' object is not subscriptable

[kellyiss]

Outputting only two actions is not a bug. Although we explicitly train the model to predict 4-step actions, the output of large language models can be unstable sometimes. The model may occasionally fail to follow the expected pattern—especially when dealing with some hard cases.
This kind of behavior is not uncommon when finetuning large language models: even if we force them to follow a fixed template during training, they may still occasionally produce off-pattern or nonsensical outputs.

[Eku127]

@kellyiss Thanks so much for your guidance! Based on your instructions, I’ve obtained results consistent with those in Table 1 of the paper. Really appreciate the help!

If possible, could you update this issue with the results from training on R2R v1-3 preprocessed data as well as the corresponding checkpoint releases? If Stage 1 ckpt is also released that will be even greater 😆

[kellyiss]

If we update the results and checkpoints, we’ll make sure to update the README accordingly. I’ll go ahead and close this issue for now.