torch.load() - OSError on a file larger than 2GB

[Roffild]

🐛 Bug

In version 1.2, torch.load() does not work with files larger than 2GB.

import torch
import pathlib
torch.save(torch.rand(int((2 * 1024**3) / 4), dtype=torch.float32, device="cpu"),
           pathlib.Path("pytorch_test_load.temp"))
torch.load(pathlib.Path("pytorch_test_load.temp"))

 File "torch\serialization.py", line 581, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
OSError: [Errno 22] Invalid argument

This is probably due to #20270 or #20900

Windows 10
Python 3.7
Pytorch 1.2.0

cc @ezyang @gchanan @zou3519 @peterjc123

[peterjc123]

Since these PRs are merged, could you please try whether the nightly one solves this issue?

[Akamight]

Nope. It doesn't fix it. Running torch-1.3.0.dev20190924 on Python 3.6.8 still yields the error.

[Akamight]

Looks like force setting f_should_read_directly = False before this block in serialization.py

    offset = f.tell() if f_should_read_directly else None
    for key in deserialized_storage_keys:
        assert key in deserialized_objects
        #print(deserialized_objects[key])
        deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
        if offset is not None:
            offset = f.tell()

seems to solve the problem

[philippslang]

The long cast in #20900 limits the file offset to a 32 bit integer, hence the >2G issues.

[ezyang]

The fix is getting reverted as it breaks Windows CI