Skip to content

gh-146393 bytecodes.c parsing fails with different encodings #148609

@Wulian233

Description

@Wulian233

Bug report

Bug description:

I’m not sure whether this is a general issue or something specific to my environment, but today I’m unable to build JIT using PCbuild\build.bat --experimental-jit.

After bisecting, I found that the problem appears after this commit was merged:
95cbd4a
Below are the error logs and a screenshot. (My system language is Chinese, so MSVC does not output English messages sorry about that)

Build log
PS E:\cpython> PCbuild\build.bat --experimental-jit
Using "E:\cpython\PCbuild\\..\externals\pythonx86\tools\python.exe" (found in externals directory)
Fetching external libraries...
bzip2-1.0.8 already exists, skipping.
mpdecimal-4.0.0 already exists, skipping.
sqlite-3.50.4.0 already exists, skipping.
xz-5.8.1.1 already exists, skipping.
zlib-ng-2.2.4 already exists, skipping.
zstd-1.5.7 already exists, skipping.
Fetching external binaries...
libffi-3.4.4 already exists, skipping.
openssl-bin-3.5.5 already exists, skipping.
tcltk-8.6.15.0 already exists, skipping.
llvm-21.1.4.0 already exists, skipping.
Finished.
Using "D:\Microsoft Visual Studio\MSBuild\Current\Bin\msbuild.exe" (found in the Visual Studio installation)
Using "E:\cpython\PCbuild\\..\externals\pythonx86\tools\python.exe" (found in externals directory)

E:\cpython>"D:\Microsoft Visual Studio\MSBuild\Current\Bin\msbuild.exe" "E:\cpython\PCbuild\pcbuild.proj" /t:Build /m /nologo /v:m /clp:summary /p:Configuration=Release /p:Platform=x64 /p:IncludeExternals=true /p:IncludeCTypes=true /p:IncludeSSL=true /p:IncludeTkinter=true /p:DisableGil= /p:UseTestMarker= /p:GIT="C:\Program Files\Git\cmd\git.exe" /p:UseJIT=true /p:UseTIER2=1 /p:PyStats= /p:UseTailCallInterp= /p:DisableRemoteDebug= /p:StackRefDebug= /p:UsePymallocHugepages=   

  _freeze_module.vcxproj -> E:\cpython\PCbuild\win32\_freeze_module.exe
  zlib-ng.vcxproj -> E:\cpython\PCbuild\amd64\zlib-ng.lib
  Killing any running python.exe instances...
  Regenerate Global Objects
  # not changed: E:\cpython\Include\internal\pycore_global_strings.h
  # not changed: E:\cpython\Include\internal\pycore_runtime_init_generated.h
  # not changed: E:\cpython\Include\internal\pycore_unicodeobject_generated.h
  # not changed: E:\cpython\Include\internal\pycore_global_objects_fini_generated.h

  =============================================================
  JIT support for x86_64-pc-windows-msvc is still experimental!
             Please report any issues you encounter.
  =============================================================

  Regenerate cases
  Traceback (most recent call last):
    File "E:\cpython\Tools\cases_generator\opcode_id_generator.py", line 61, in <module>
      data = analyze_files(args.input)
    File "E:\cpython\Tools\cases_generator\analyzer.py", line 1424, in analyze_files
      return analyze_forest(parser.parse_files(filenames))
                            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
    File "E:\cpython\Tools\cases_generator\parser.py", line 46, in parse_files
      src = file.read()
  UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 33327: illegal multibyte sequence

E:\cpython\PCbuild\regen.targets(97,5): error MSB3073: Command ""E:\cpython\PCbuild\\..\externals\pythonx86\tools\python.exe"
 Tools\cases_generator\opcode_id_generator.py Python\bytecodes.c" exited with code 1. [E:\cpython\PCbuild\pythoncore.vcxproj]

Build FAILED.

E:\cpython\PCbuild\regen.targets(97,5): error MSB3073: Command ""E:\cpython\PCbuild\\..\externals\pythonx86\tools\python.exe"
 Tools\cases_generator\opcode_id_generator.py Python\bytecodes.c" exited with code 1. [E:\cpython\PCbuild\pythoncore.vcxproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:07.25
Image

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions