TypeError: 'NoneType' object is not subscriptable

[missTL]

After a period of testing, the error message is as follows:

[10:30:18.360321] <|im_start|>assistant                                                                                
↑↑←←<|im_end|>                                                                                                         
[10:30:18.360425] actions [1, 1, 2, 2]                                                                                 
[rank7]: Traceback (most recent call last):                                                                           
[rank7]:   File "/home/zengshuang.zs/github/StreamVLN/streamvln_eval.py", line 582, in <module>                       
[rank7]:     eval()                                                                                                   
[rank7]:   File "/home/zengshuang.zs/github/StreamVLN/streamvln_eval.py", line 532, in eval                           
[rank7]:     evaluate(model, tokenizer, args)                                                                         
[rank7]:   File "/home/zengshuang.zs/github/StreamVLN/streamvln_eval.py", line 551, in evaluate                       
[rank7]:     sucs, spls, oss, ones, ep_num = evaluator.eval_action(get_rank())                                        
[rank7]:   File "/home/zengshuang.zs/github/StreamVLN/streamvln_eval.py", line 330, in eval_action                    
[rank7]:     outputs = self.model.generate(**input_dict, do_sample=False, num_beams=1, max_new_tokens=10000, use_cache=
True, return_dict_in_generate=True, past_key_values=past_key_values)                                                  
[rank7]:   File "/home/zengshuang.zs/anaconda3/envs/lhvln/lib/python3.9/site-packages/torch/utils/_contextlib.py", line
 116, in decorate_context                                                                                             
[rank7]:     return func(*args, **kwargs)                                                                             
[rank7]:   File "/home/zengshuang.zs/github/StreamVLN/streamvln/model/stream_video_vln.py", line 379, in generate     
[rank7]:     ) = self.prepare_inputs_labels_for_multimodal(                                                           
[rank7]:   File "/home/zengshuang.zs/github/StreamVLN/streamvln/model/stream_video_vln.py", line 224, in prepare_inputs
_labels_for_multimodal                                                                                                
[rank7]:     cur_memory_feature = memory_features[batch_idx][cur_mem_id]                                              
[rank7]: TypeError: 'NoneType' object is not subscriptable                                                            
scene QUCTc6BB5sX:   0%|                                                                        | 0/31 [00:42<?, ?it/s]
[10:30:19.727267] <|im_start|>assistant                                                                               
↑↑→↑<|im_end|>                                                                                                              "localhost.localdomain" 10:33 11-7月-25
[10:30:19.727371] actions [1, 1, 3, 1]                                                                                
                                                                                                                      [
10:30:21.062158] <|im_start|>assistant                                                  | 1/32 [00:39<20:15, 39.19s/it]
↑↑←↑<|im_end|>                                                                                                        
[10:30:21.062278] actions [1, 1, 2, 1]                                                                                
W0711 10:30:21.719878 140136153896768 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 97786 closi
ng signal SIGTERM                                                                                                     
W0711 10:30:21.721962 140136153896768 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 97787 closi
ng signal SIGTERM                                                                                                     
W0711 10:30:21.724307 140136153896768 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 97788 closi
ng signal SIGTERM
W0711 10:30:21.726102 140136153896768 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 97789 closi
ng signal SIGTERM
W0711 10:30:21.726994 140136153896768 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 97790 closi
ng signal SIGTERM
W0711 10:30:21.728477 140136153896768 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 97791 closi
ng signal SIGTERM
W0711 10:30:21.730464 140136153896768 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 97792 closi
ng signal SIGTERM
E0711 10:30:24.018149 140136153896768 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_
rank: 7 (pid: 97793) of binary: /home/zengshuang.zs/anaconda3/envs/lhvln/bin/python3.9

[kellyiss]

Thanks for catching this issue.
This occurs when the LLM occasionally outputs fewer than four valid actions. While simply rerunning the process typically resolves it, we can implement a more robust solution.

Suggested Fix:
Please add the following code after
https://github.com/OpenRobotLab/StreamVLN/blob/75a7da78b7513a97ffdf9227aed93a4613373314/streamvln_eval.py#L335

if len(action_seq) < 4 and 0 not in action_seq:
    if len(action_seq) == 0:
        action_seq = [2, 3, 2, 3]  
    elif len(action_seq) == 1:
        action_seq += [2, 2, 3]     
    elif len(action_seq) == 2:
        action_seq += [2, 3]       
    elif len(action_seq) == 3:
        action_seq += [2]          

[missTL]

It seems not to be this issue. Even if the previous actions were less than 4, no error would occur. But after adding more, the same problem persists.

[14:04:59.166037] actions [2, 2, 2, 1]
[rank7]: Traceback (most recent call last):
[rank7]: File "/home/zengshuang.zs/StreamVLN/streamvln/streamvln_eval.py", line 584, in
[rank7]: eval()
[rank7]: File "/home/zengshuang.zs/StreamVLN/streamvln/streamvln_eval.py", line 534, in eval
[rank7]: evaluate(model, tokenizer, args)
[rank7]: File "/home/zengshuang.zs/StreamVLN/streamvln/streamvln_eval.py", line 553, in evaluate
[rank7]: sucs, spls, oss, ones, ep_num = evaluator.eval_action(get_rank())
[rank7]: File "/home/zengshuang.zs/StreamVLN/streamvln/streamvln_eval.py", line 332, in eval_action
[rank7]: outputs = self.model.generate(**input_dict, do_sample=False, num_beams=1, max_new_tokens=10000, use_cache=True, return_dict_in_gene$
ate=True, past_key_values=past_key_values)
[rank7]: File "/home/zengshuang.zs/anaconda3/envs/lhvln/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank7]: return func(*args, **kwargs)
[rank7]: File "/home/zengshuang.zs/StreamVLN/streamvln/model/stream_video_vln.py", line 379, in generate
[rank7]: ) = self.prepare_inputs_labels_for_multimodal(
[rank7]: File "/home/zengshuang.zs/StreamVLN/streamvln/model/stream_video_vln.py", line 224, in prepare_inputs_labels_for_multimodal
[rank7]: cur_memory_feature = memory_features[batch_idx][cur_mem_id]
[rank7]: TypeError: 'NoneType' object is not subscriptable
[14:05:00.965509] <|im_start|>assistant
↑←↑→<|im_end|>
[14:05:00.965679] actions [1, 2, 1, 3]
scene QUCTc6BB5sX: 52%|█████████████████████████████████████████████▉ | 16/31 [21:49<20:27, 81.84s/it$
[14:05:02.920821] <|im_start|>assistant
→→↑→<|im_end|>
[14:05:02.920971] actions [3, 3, 1, 3]

[chchnii]

Can you keep testing on a single GPU and send me the model's action output for the episode that failed?

[Eku127]

I got the same inference problem.

[16:50:25.727923] actions [1, 2, 2, 1]
Traceback (most recent call last):
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 590, in <module>
    eval()
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 540, in eval
    evaluate(model, tokenizer, args)
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 559, in evaluate
    sucs, spls, oss, ones, ep_num = evaluator.eval_action(get_rank())
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 332, in eval_action
    outputs = self.model.generate(**input_dict, do_sample=False, num_beams=1, max_new_tokens=10000, use_cache=True, return_dict_in_generate=True, past_key_values=past_key_values)
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/home//workspace/StreamVLN/streamvln/model/stream_video_vln.py", line 379, in generate
    ) = self.prepare_inputs_labels_for_multimodal(
  File "/data/home//workspace/StreamVLN/streamvln/model/stream_video_vln.py", line 224, in prepare_inputs_labels_for_multimodal
    cur_memory_feature = memory_features[batch_idx][cur_mem_id]
TypeError: 'NoneType' object is not subscriptable
scene QUCTc6BB5sX:  52%|█████████████████████████████████████▋                                  | 33/63 [27:39<25:08, 50.28s/it]
[16:50:26.842349] <|im_start|>assistant
↑↑↑STOP<|im_end|>
[16:50:26.842490] actions [1, 1, 1, 0]

s[16:50:27.191750] scene_episode QUCTc6BB5sX_1114 success: 1.0, spl: 1.0, os: 1.0, ne: 1.4105442762374878[27:21<09:41, 24.23s/it[16:50:27.191918] episode start face four kitchen bar stools, turn left, turn right after the last kitchen bar stool and walk straight through kitchen, stop in the middle of the stone arch doorway.
[16:50:29.171400] 0 You are an autonomous navigation assistant. Your task is to face four kitchen bar stools, turn left, turn right after the last kitchen bar stool and walk straight through kitchen, stop in the middle of the stone arch doorway.  Devise an action sequence to follow the instruction using the four actions: TURN LEFT (←) or TURN RIGHT (→) by 15 degrees, MOVE FORWARD (↑) by 25 centimeters, or STOP.
[16:50:29.812581] <|im_start|>assistant
←←←←<|im_end|>
[16:50:29.812692] actions [2, 2, 2, 2]
[16:50:30.919578] <|im_start|>assistant
←←←←<|im_end|>
[16:50:30.919695] actions [2, 2, 2, 2]
[2025-07-16 16:50:30,932] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 255037 closing signal SIGTERM
[2025-07-16 16:50:30,933] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 255038 closing signal SIGTERM
[2025-07-16 16:50:30,934] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 255039 closing signal SIGTERM
[2025-07-16 16:50:31,453] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 3 (pid: 255040) of binary: /data/home//miniconda3/envs/streamvln/bin/python3.9
Traceback (most recent call last):
  File "/data/home//miniconda3/envs/streamvln/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

[missTL]

Can you keep testing on a single GPU and send me the model's action output for the episode that failed?

The following is the error message for the single GPU:

[17:11:42.287480] actions [1, 3, 1, 1]
[17:11:44.256731] <|im_start|>assistant
↑→↑←<|im_end|>
[17:11:44.256857] actions [1, 3, 1, 2]
[17:11:46.304733] <|im_start|>assistant
←←↑↑<|im_end|>
[17:11:46.304902] actions [2, 2, 1, 1]
[17:11:47.210876] 33 You are an autonomous navigation assistant. Your task is to Walk past the counters and
exit the kitchen. Wait next to the landscape painting on the wall. Devise an action sequence to follow the
instruction using the four actions: TURN LEFT (←) or TURN RIGHT (→) by 15 degrees, MOVE FORWARD (↑) by 25 ce
ntimeters, or STOP. These are your historical observations .
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/zengshuang.zs/StreamVLN/streamvln/streamvln_eval.py", line 593, in
[rank0]: eval()
[rank0]: File "/home/zengshuang.zs/StreamVLN/streamvln/streamvln_eval.py", line 543, in eval
[rank0]: evaluate(model, tokenizer, args)
[rank0]: File "/home/zengshuang.zs/StreamVLN/streamvln/streamvln_eval.py", line 562, in evaluate
[rank0]: sucs, spls, oss, ones, ep_num = evaluator.eval_action(get_rank())
[rank0]: File "/home/zengshuang.zs/StreamVLN/streamvln/streamvln_eval.py", line 332, in eval_action
[rank0]: outputs = self.model.generate(**input_dict, do_sample=False, num_beams=1, max_new_tokens=10000,
use_cache=True, return_dict_in_generate=True, past_key_values=past_key_values)
[rank0]: File "/home/zengshuang.zs/anaconda3/envs/lhvln/lib/python3.9/site-packages/torch/utils/contextli
b.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/zengshuang.zs/StreamVLN/streamvln/model/stream_video_vln.py", line 379, in generate ", line 264, in launch
[rank0]: ) = self.prepare_inputs_labels_for_multimodal(
[rank0]: File "/home/zengshuang.zs/StreamVLN/streamvln/model/stream_video_vln.py", line 224, in prepare_i$puts_labels_for_multimodal pr
[rank0]: cur_memory_feature = memory_features[batch_idx][cur_mem_id]
[rank0]: TypeError: 'NoneType' object is not subscriptable
scene QUCTc6BB5sX: 2%|█▏ | 4/255 [$ 94:22<4:34:51, 65.70s/it]

[chchnii]

“[17:11:47.210876] 33 You are an autonomous navigation assistant. ...... .”

The number before the instruction (33 in your case) is the current step_id. It should be a multiple of 4, so 33 shouldn't appear. It's likely that some earlier output actions weren't in groups of four. Please double-check.

Also, a reminder: when testing with A100 GPUs, the simulator renders mosaic-like corrupted images during simulation – this must be avoided. If A100 GPUs are used, the incorrect action output is likely caused by the mosaic rendering artifacts.

Thanks!

[Eku127]

“[17:11:47.210876] 33 You are an autonomous navigation assistant. ...... .”

The number before the instruction (33 in your case) is the current step_id. It should be a multiple of 4, so 33 shouldn't appear. It's likely that some earlier output actions weren't in groups of four. Please double-check.

Also, a reminder: when testing with A100 GPUs, the simulator renders mosaic-like corrupted images during simulation – this must be avoided. If A100 GPUs are used, the incorrect action output is likely caused by the mosaic rendering artifacts.

Thanks!

I get the same 33 on my test case. The case is that one step mis-ouput 5 actions.

[19:46:54.508996] scene_episode QUCTc6BB5sX_1003
[19:46:56.488942] 0 You are an autonomous navigation assistant. Your task is to Walk past the counters and exit the kitchen. Wait next to the landscape painting on the wall.  Devise an action sequence to follow the instruction using the four actions: TURN LEFT (←) or TURN RIGHT (→) by 15 degrees, MOVE FORWARD (↑) by 25 centimeters, or STOP.
[19:46:57.132869] <|im_start|>assistant
→→↑↑<|im_end|>
[19:46:57.132999] actions [3, 3, 1, 1]
[19:46:58.222012] <|im_start|>assistant
↑↑↑↑<|im_end|>
[19:46:58.222120] actions [1, 1, 1, 1]
[19:46:59.311306] <|im_start|>assistant
↑↑→↑<|im_end|>
[19:46:59.311399] actions [1, 1, 3, 1]
[19:47:00.403438] <|im_start|>assistant
↑←↑→<|im_end|>
[19:47:00.403530] actions [1, 2, 1, 3]
[19:47:01.527453] <|im_start|>assistant
↑↑↑→↑<|im_end|>
[19:47:01.527552] actions [1, 1, 1, 3, 1]
[19:47:02.727484] <|im_start|>assistant
↑→↑↑<|im_end|>
[19:47:02.727576] actions [1, 3, 1, 1]
[19:47:03.819639] <|im_start|>assistant
↑→↑←<|im_end|>
[19:47:03.819731] actions [1, 3, 1, 2]
[19:47:04.910324] <|im_start|>assistant
←←↑↑<|im_end|>
[19:47:04.910417] actions [2, 2, 1, 1]
[19:47:05.377065] 33 You are an autonomous navigation assistant. Your task is to Walk past the counters and exit the kitchen. Wait next to the landscape painting on the wall.  Devise an action sequence to follow the instruction using the four actions: TURN LEFT (←) or TURN RIGHT (→) by 15 degrees, MOVE FORWARD (↑) by 25 centimeters, or STOP. These are your historical observations <memory>.
Traceback (most recent call last):
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 593, in <module>
    eval()
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 543, in eval
    evaluate(model, tokenizer, args)
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 562, in evaluate
    sucs, spls, oss, ones, ep_num = evaluator.eval_action(get_rank())
  File "/data/home//workspace/StreamVLN/streamvln/streamvln_eval.py", line 340, in eval_action
    outputs = self.model.generate(**input_dict, do_sample=False, num_beams=1, max_new_tokens=10000, use_cache=True, return_dict_in_generate=True, past_key_values=past_key_values)
  File "/data/home/jiangjiajun/miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/home//workspace/StreamVLN/streamvln/model/stream_video_vln.py", line 379, in generate
    ) = self.prepare_inputs_labels_for_multimodal(
  File "/data/home//workspace/StreamVLN/streamvln/model/stream_video_vln.py", line 224, in prepare_inputs_labels_for_multimodal
    cur_memory_feature = memory_features[batch_idx][cur_mem_id]
TypeError: 'NoneType' object is not subscriptable
scene QUCTc6BB5sX:  53%|██████████████████████████████████████████████████████████████████████████████████████████████████████▋                                                                                           | 135/255 [1:29:38<1:19:40, 39.84s/it]
[2025-07-16 19:47:08,315] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 441895) of binary: /data/home//miniconda3/envs/streamvln/bin/python3.9
Traceback (most recent call last):
  File "/data/home//miniconda3/envs/streamvln/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/data/home//miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

And I find out this bug can be reproduced while running the episode QUCTc6BB5sX_1003 . I used one L20 for inference.

[kellyiss]

Although we haven’t encountered cases where action_seq exceeds a length of 4 in our testing, the fix would be:

if len(action_seq) > 4:  
    action_seq = action_seq[:4]  

[Eku127]

Although we haven’t encountered cases where action_seq exceeds a length of 4 in our testing, the fix would be:

if len(action_seq) > 4:  
    action_seq = action_seq[:4]  

With this change, evaluation on the entire envdrop val_unseen split completes without errors.

Notably, the issue appears isolated to the episode QUCTc6BB5sX_1003 . If you have time, feel free to take a look and verify :p