Resume training not working (WandB error + wrong pretrained policy path)

[nicolas-duque]

System Info

- `lerobot` version: 0.1.0
- Platform: Linux-5.15.0-131-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.29.1
- Dataset version: 3.3.2
- Numpy version: 2.2.3
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Cuda version: 12040
- Using GPU in script?: yes

Information

• One of the scripts in the examples/ folder of LeRobot
• My own task or dataset (give details below)

Reproduction

Resuming training (as mentioned in LeRobot example 4):

python lerobot/scripts/train.py \ --config_path=outputs/train/act_test/checkpoints/last/pretrained_model/ \ --resume=true

Expected behavior

results in two different errors:

1. If WandB is enabled, the following error appears:
   RuntimeError: Couldn't get the previous WandB run ID for run resumption.
   Seems like the wandb/latest-run/run-* is not existing in the output directory.

2. If WandB is disabled (--wandb.enable=false), the following error appears later:
   Traceback (most recent call last): File "/home/duque/lerobot/lerobot/scripts/train.py", line 286, in <module> train() File "/home/duque/lerobot/lerobot/configs/parser.py", line 120, in wrapper_inner response = fn(cfg, *args, **kwargs) File "/home/duque/lerobot/lerobot/scripts/train.py", line 139, in train policy = make_policy( File "/home/duque/lerobot/lerobot/common/policies/factory.py", line 143, in make_policy policy = policy_cls.from_pretrained(**kwargs) File "/home/duque/lerobot/lerobot/common/policies/pretrained.py", line 101, in from_pretrained policy = cls._load_as_safetensor(instance, model_file, map_location, strict) File "/home/duque/lerobot/lerobot/common/policies/pretrained.py", line 138, in _load_as_safetensor safetensors.torch.load_model(model, model_file, strict=strict, device=map_location) File "/home/duque/anaconda3/envs/lerobot5/lib/python3.10/site-packages/safetensors/torch.py", line 202, in load_model state_dict = load_file(filename, device=device) File "/home/duque/anaconda3/envs/lerobot5/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file with safe_open(filename, framework="pt", device=device) as f: FileNotFoundError: No such file or directory: "outputs/train/act_test/checkpoints/last/model.safetensors"
   Looking into the path, I found out that the path is not correct since it should be outputs/train/act_test/checkpoints/last/pretrained_model/model.safetensors

I actually got to solve this error by changing the policy_path in the TrainPipelineConfig class, however I am not sure of how correct this is or if it may be breaking something else:

Although not strictly necessary, I would appreciate any input on how to resume training with WandB enabled as well.

[srsaito]

I've independently encountered and fixed this same issue. When attempting to resume training using a command as documented:

python lerobot/scripts/train.py \
    --config_path=outputs/train/run_name/checkpoints/last/pretrained_model/ \
    --resume=true

The code incorrectly sets paths, causing it to look for files in the wrong locations.

Instead of modifying policy_path, though, I found that changing checkpoint_path is a more targeted solution:

# Original code
policy_path = Path(config_path).parent
self.policy.pretrained_path = policy_path
self.checkpoint_path = policy_path.parent

# My proposed fix
policy_path = Path(config_path).parent
self.policy.pretrained_path = policy_path
self.checkpoint_path = policy_path  # Only this line changes

I recommend this approach because checkpoint_path is only used in the resumption logic (specifically in train.py when cfg.resume is true), while policy_path is used in multiple places throughout the codebase. This change maintains the separation between model loading (policy_path) and training state loading (checkpoint_path)

Successfully tested fix by:

1. Making the change
2. Interrupting training with Ctrl+C
3. Resuming with documented command line
4. Confirming training continued from the correct checkpoint

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity (6 months). It will be closed if no further activity occurs. Any change, comment or update to this issue will reset this count. Thank you for your contributions.

[github-actions]

This issue was closed because it has been stalled for 14 days with no activity. Feel free to reopen if is still relevant, or to ping a collaborator if you have any questions.