Certain .pyi files are not encoded as UTF-8 in Windows

[marovira]

🐛 Describe the bug

On Windows platforms, PyTorch 2.3.0 causes mypy to fail with the following error messages:

mypy: can't decode file '.venv\Lib\site-packages\torch\_C\_VariableFunctions.pyi': 'utf-8' codec can't decode byte 0x93 in position 290995: invalid start byte
mypy: can't decode file '.venv\Lib\site-packages\torch\_VF.pyi': 'utf-8' codec can't decode byte 0x93 in position 290995: invalid start byte

Inspecting _VariableFunctions.pyi and _VF.pyi reveals that the files are encoded with Latin1 encoding instead of UTF-8 like the rest of the pyi files. Looking deeper at the contents of the wheel obtained from https://download.pytorch.org/whl/cu121/torch-2.3.0%2Bcu121-cp311-cp311-win_amd64.whl shows that the files are incorrectly encoded there as well. It is important to note that PyTorch 2.2.2 does not have this problem. It was introduced in 2.3.0.

As a workaround, the files can be re-encoded to UTF-8, which allows mypy to work correctly.

Versions

PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Enterprise
GCC version: Could not collect
Clang version: 17.0.1
CMake version: version 3.27.4
Libc version: N/A

Python version: 3.11.3 (tags/v3.11.3:f3909b8, Apr 4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22631-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1650
Nvidia driver version: 552.22
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=4701
DeviceID=CPU0
Family=107
L2CacheSize=12288
L2CacheSpeed=
Manufacturer=AuthenticAMD
MaxClockSpeed=4701
Name=AMD Ryzen 9 7900X 12-Core Processor
ProcessorType=3
Revision=24834

Versions of relevant libraries:
[pip3] mypy==1.10.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] onnx==1.16.0
[pip3] onnxruntime==1.17.3
[pip3] torch==2.3.0+cu121
[pip3] torchvision==0.18.0+cu121
[conda] Could not collect

cc @seemethere @malfet @osalpekar @atalman @ezyang @xuzhao9 @gramster

[malfet]

I wonder how it happened, as I don't see the problem on trunk right now

[marovira]

I have no idea. I double-checked the files in the source tree just to ensure that no changes had been made, and I double-checked the encoding (which is set to UTF-8) so it's not from the source files. My guess is it's coming from the process that fills in those files.

[malfet]

@marovira do you mind sharing the line around the offending characters? As at least on Mac builds all sequences are valid unicode ones

import torch
import os
print(torch.__version__)
x=open(os.path.join(os.path.dirname(torch.__file__), "_VF.pyi"), "rb").read()
y=x.decode("utf-8")
print(len(x), len(y))

Does not fail and produces (1137705, 1137683) for me

[marovira]

Sure. I'll post them after I've had dinner.

[malfet]

Same here, I can get to a Windows machine but probably after dinner...

[marovira]

I dug around the file a bit and found where the characters are coming from. In the docstring for cov(input: Tensor, *, ...), jump to line 6320. The offending text is this one:

            These relative weights are typically large for observations considered �important� and smaller for
            observations considered less �important�. Its numel must equal the number of columns of :attr:`input`.
            Must have floating point dtype. Ignored if ``None``. Defaults to ``None``.

The same issue is present in _VariableFunctions.pyi in the exact same line (6320).

[marovira]

Looking through the PyTorch source code, I think I know what's going on. The docstring comes from torch/_torch_docs.py. Reading through the docstring for torch.cov which starts at 2263, it appears that the issue is coming from this:

        These relative weights are typically large for observations considered “important” and smaller for
        observations considered less “important”. Its numel must equal the number of columns of :attr:`input`.

Note the double-quotes around important. It seems that for some reason those characters are getting mangled, which results in invalid UTF-8 characters. No idea why this would be happening, but that's where the problem is coming from. Also important to note that the file itself is valid UTF-8, so the issue is coming from whatever system takes that docstring and sets it in the pyi files.

[malfet]

Above-mentioned changes were introduced by #58311 and there hasn't been any significant changes to neither torchgen/utils.py nor to tools/pyi/gen_pyi.py that can warrant that change, that probably it has something to do with the build environment itself.

But let's fix it two way:

• Get rid of unicode characters in _torch_docs.py
• Specify file encoding in torhcgen to be ascii (or utf-8?)
• Figure out why flake8 didn't raise the error, because torch_docs is missing encoding magic

[fingoldo]

Wondering if there's some flag for mypy to skip such files.