Basic math operations produce a "floating point exception"

[timothygebhard]

🐛 Describe the bug

When I try to run the following simple piece of code:

import numpy as np
import torch

np.random.seed(42)

x = torch.from_numpy(np.random.rand(100)).float()
print(x)

exp_x = torch.exp(x)
print(exp_x)

I get a floating point exception that kills my Python interpreter:

(venv) [tgebhard@g108] ~ % python test.py                                                                                                                                                                   
tensor([0.3745, 0.9507, 0.7320, 0.5987, 0.1560, 0.1560, 0.0581, 0.8662, 0.6011,
        0.7081, 0.0206, 0.9699, 0.8324, 0.2123, 0.1818, 0.1834, 0.3042, 0.5248,
        0.4319, 0.2912, 0.6119, 0.1395, 0.2921, 0.3664, 0.4561, 0.7852, 0.1997,
        0.5142, 0.5924, 0.0465, 0.6075, 0.1705, 0.0651, 0.9489, 0.9656, 0.8084,
        0.3046, 0.0977, 0.6842, 0.4402, 0.1220, 0.4952, 0.0344, 0.9093, 0.2588,
        0.6625, 0.3117, 0.5201, 0.5467, 0.1849, 0.9696, 0.7751, 0.9395, 0.8948,
        0.5979, 0.9219, 0.0885, 0.1960, 0.0452, 0.3253, 0.3887, 0.2713, 0.8287,
        0.3568, 0.2809, 0.5427, 0.1409, 0.8022, 0.0746, 0.9869, 0.7722, 0.1987,
        0.0055, 0.8155, 0.7069, 0.7290, 0.7713, 0.0740, 0.3585, 0.1159, 0.8631,
        0.6233, 0.3309, 0.0636, 0.3110, 0.3252, 0.7296, 0.6376, 0.8872, 0.4722,
        0.1196, 0.7132, 0.7608, 0.5613, 0.7710, 0.4938, 0.5227, 0.4275, 0.0254,
        0.1079])
zsh: floating point exception  python test.py
(venv) [tgebhard@g108] ~ %

The problem also occurs for other mathematical operations such as torch.log() or torch.cos(). It seems like it only happens if the size of the input tensor is at least 100, though.

Moreover, the issue only occurs on some machines, under some specific circumstances: My local machine will run the code above without any problem, but one of the machines at work reproducibly gives the error above, but only if I request at least 14 CPU cores (it's a batch queue system based on HTCondor). It might, therefore, be the case that only this particular machine has a problem. Any pointers for debugging this are greatly appreciated! 🙂

Versions

Information about the Python environment:

PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun  2 2021, 10:49:15)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No devices found.
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] pytorch-lightning==1.8.3.post0
[pip3] torch==1.13.0
[pip3] torchmetrics==0.10.3
[conda] Could not collect

Information about the machine where the problem occurs (output of lscpu):

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          256
On-line CPU(s) list:             0,1,9-16,26-31
Off-line CPU(s) list:            2-8,17-25,32-255
Thread(s) per core:              0
Core(s) per socket:              64
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           49
Model name:                      AMD EPYC 7662 64-Core Processor
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         1499.941
CPU max MHz:                     2000.0000
CPU min MHz:                     1500.0000
BogoMIPS:                        3999.98
Virtualization:                  AMD-V
L1d cache:                       2 MiB
L1i cache:                       2 MiB
L2 cache:                        32 MiB
L3 cache:                        256 MiB
NUMA node0 CPU(s):               0-63,128-191
NUMA node1 CPU(s):               64-127,192-255
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpui
                                 d extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dn
                                 owprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2
                                  cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock n
                                 rip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

cc @VitalyFedyunin @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

[jgong5]

@timothygebhard Seems we are not able to repro it locally on Intel CPU. To further narrow down the issue, are you able to try sanitizer build and run your test with sanitizer?

pytorch/CMakeLists.txt

Line 183 in 8226a5d

option(USE_ASAN "Use Address+Undefined Sanitizers" OFF)

[timothygebhard]

@jgong5 I tried all day to build PyTorch with ASAN as described here (the guide might need an update, BTW), but I did not succeed. Ultimately, it always crashes with some clang: error: linker command failed with exit code 1, and the undefined messages that it keeps complaining about seem related to OpenMP (e.g., /usr/bin/ld: lib/libtorch_cpu.so: undefined reference to omp_in_parallel'), even though the build_with_asan() command from the guide above sets USE_OPENMP=0 🤷‍♂️

When talking to our cluster admin at work, he suspected that the problem might be related to MKL, possibly in combination with an AMD CPU? In that case, I guess I should try to make sure to compile the sanitizer build with the same MKL version as the PyTorch version that's being shipped via pip?

[zadaianchuk]

Similar issue seems to occur more than a year ago:
#66247 (comment)

Are there any plans to update MKL version that potentially doesn't have this bug in combination with an AMD CPU?

[timothygebhard]

Running echo 'run' | gdb --args python test.py seems to confirm that it's indeed an MKL issue:

Thread 1 "python" received signal SIGFPE, Arithmetic exception.
0x000015551042ec47 in mkl_vml_serv_GetMinN () from /lustre/home/tgebhard/.virtualenvs/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so

[piraka9011]

I have also seen the same issue in #66247 and had to resort to building my own PyTorch container with USE_MKL=OFF in order to resolve this issue.
My stack trace was also similar to what was reported there.

This does not happen on Intel CPUs and occurs on specific AMD CPUs.
For ex. it works on my local AMD Ryzen 7 3700X.
But when I deploy it to a cluster with Epyc Rome or Threadripper CPUs, I get a SIGFPE.

[jgong5]

@timothygebhard You mentioned that it is related to the MKL version that pytorch release package is built with. May I know what particular MKL version it is? Have you tried with a newer version of MKL?

[mseitzer]

This issue happens for me with pre-built Pytorch packages on versions 1.11, 1.12, 1.13, 2.0 (nightly). I have not tried earlier versions.

The relevant line from torch.__config__.show() (the same for all tested Pytorch versions):

  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications

According to #66247 (comment), the issue disappears with MKL 2021.4, although I have not tried this.

It would be great to upgrade the MKL standard Pytorch is built with, as this essentially renders Pytorch unusable on machines with the affected CPUs.

[timothygebhard]

@jgong5 The MKL version is the one mentioned by @mseitzer. I haven't tried to build PyTorch myself using a newer version of MKL yet; however, I seem to recall that when I did build it without MKL (using OpenBLAS instead), the error was gone.

[jingxu10]

I cannot reproduce this issue on neither AWS M5a nor M6a 4xlarge EC2 instances with the nightly built
pip3 install --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cpu
Would you let us know which instance type could reproduce this issue?

Alternatively, could you try compiling PyTorch from source? Instructions are available at https://github.com/pytorch/pytorch#from-source
conda install commands should install the latest version of MKL.

[piraka9011]

Here is the output of lscpu from an example machine of where this happens

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         43 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  32
  On-line CPU(s) list:   0,6,11-15,22,27-31
  Off-line CPU(s) list:  1-5,7-10,16-21,23-26
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC 7F52 16-Core Processor
    CPU family:          23
    Model:               49
    Thread(s) per core:  2
    Core(s) per socket:  16
    Socket(s):           1
    Stepping:            0
    Frequency boost:     enabled
    CPU max MHz:         3500.0000
    CPU min MHz:         2500.0000
    BogoMIPS:            6999.96
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc 
                         cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3d
                         nowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cq
                         m rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm
                         _lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
Virtualization features: 
  Virtualization:        AMD-V
Caches (sum of all):     
  L1d:                   512 KiB (16 instances)
  L1i:                   512 KiB (16 instances)
  L2:                    8 MiB (16 instances)
  L3:                    256 MiB (16 instances)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-31
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; LFENCE, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
  Srbds:                 Not affected
  Tsx async abort:       Not affected

I don't think this kind of machine explicitly is available on AWS, but it is on coreweave.com for example.

[jingxu10]

could you try compiling PyTorch from source? Instructions are available at https://github.com/pytorch/pytorch#from-source
conda install commands should install the latest version of MKL. If the latest MKL solves this issue, compiling from source by yourself should work. Could you have a try on your environment?

[soumith]

This issue happens for me with pre-built Pytorch packages on versions 1.11, 1.12, 1.13, 2.0 (nightly)

@malfet any reason we are holding back on MKL 2020 for these packages, instead of just upgrading to the latest MKL?
You touched it last in 2020 haha, but these are the relevant lines: https://github.com/pytorch/builder/blob/main/common/install_mkl.sh#L6-L13
They build our docker image that builds the wheels, and the docker image in-turn statically links to this MKL. Upgrading this and rebuilding the docker image would likely solve this.

[malfet]

@malfet any reason we are holding back on MKL 2020 for these packages, instead of just upgrading to the latest MKL?

MKL 2020 is the last one that allows AMD CPU users to opt into AVX2 accelerated kernels using environment variable. Later ones require injecting a symbol in global namespace.

Here is an old issue (dating back to 2020) that discusses perf problems associated with updating to newer MKL: pytorch/builder#504

But stability is more important than perf, so imo we should update to the one that does not crash.

[malfet]

Here is the output of lscpu from an example machine of where this happens

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         43 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  32
  On-line CPU(s) list:   0,6,11-15,22,27-31
  Off-line CPU(s) list:  1-5,7-10,16-21,23-26
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC 7F52 16-Core Processor
    CPU family:          23
    Model:               49
    Thread(s) per core:  2
    Core(s) per socket:  16

I can not repro it on AWS C5a.4xlarge, which has the same CPU family and model:

$ lscpu 
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          16
On-line CPU(s) list:             0-15
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           49
Model name:                      AMD EPYC 7R32

@piraka9011 what is the type of instance you've allocated on coreweave.com? (As I could not repro it on 16 vCPU EPYC Rome instance, i.e. running AMD EPYC 7402P 24-Core Processor )

[piraka9011]

@jingxu10 Ive been compiling from source with the USE_MKL=OFF flag.
I will try compiling with the flag removed and the latest mkl installed via Conda and report back.

@malfet so I don't think there's an easy way to get the same exact config on CoreWeave using their standard virtual server offering, but I will try to find a config to repro the instance exactly.

[malfet]

@piraka9011 I'm going to switch to 2022.2.1 in the nightlies and will run CPU performance tests afterwards (on Intel CPUs it showed moderate perf gains, and smoke tests I've run on AMD ones showed they they are no worse, but much better if LD_PRELOAD trick is used)

[timothygebhard]

I can confirm that the problem is still present in the current nightly built (2.0.0.dev20221216+cpu, which still uses MKL Version 2020.0.0 Product Build 20191122).

I also finally got around to building PyTorch from source (with conda)¹, which gave me version 2.0.0a0+git212873c, including a newer version of MKL:

Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications

Unfortunately, even with this version, the SIGFPE issue is still present on my machine.

Footnotes

1. As a side node, building PyTorch from source took well over 24 hours for me (on a machine with 32 cores), so even if that had fixed the issue, I am not sure how feasible that would be for most people who just want to use PyTorch? ↩

[jingxu10]

@timothygebhard The latest MKL is 2022.2.1, but the one you got was 2021.4. Since @malfet had enabled the 2022.2.1 into nightly builds, you can probably try the ones later than 20221218? @malfet Any idea which one has 2022.2.1 enabled?

[malfet]

Unfortunately, even with this version, the SIGFPE issue is still present on my machine.

@timothygebhard would you be comfortable attaching gdb/lldb to the process running pytorch and sharing exact instruction/FPU registers when exception occurs? (I can provide all the instruction on how to do it if needed)

1. As a side node, building PyTorch from source took well over 24 hours for me (on a machine with 32 cores), so even if that had fixed the issue, I am not sure how feasible that would be for most people who just want to use PyTorch?

Hmm, this shouldn't be the case, even if you build with CUDA support for all possible architectures. I.e. CPU build of PyTorch takes less than 30 min on my rather aniquated laptop. Have you installed ninja? If not, have you passed MAX_JOBS parameter to ensure build is running in parallel rather than sequetnially?

[timothygebhard]

@jingxu10 I just installed the 2.0.0.dev20221218+cpu nightly, which indeed was built with the 2022.2 version of MKL. I am happy to report that with this version, the problem seems resolved — my minimal examples run without causing a floating point exception 🙂

@malfet Sure, I can give it a try! I just ran gdb -ex r --args python test.py for a PyTorch version using MKL 2020, and it resulted in:

Thread 1 "python" received signal SIGFPE, Arithmetic exception.
0x0000155488495c47 in mkl_vml_serv_GetMinN () from /lustre/home/tgebhard/.virtualenvs/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so

Is this helpful / what you were looking for, or what else should I run?

Regarding the long build time: I did install ninja via conda (as described in the "Install dependencies" command here), but I did not explicitly set MAX_JOBS. I can try again if you think that was the issue?

[malfet]

@timothygebhard can you run following commands x/4i $rip and info all-registers and share the output here.

As for build time, ninja should auto-detect the number of CPU cores and do parallelization automatically. Do you mind sharing .ninja_log file from build folder, that would give me an idea about parallelism/built times you've observed
I believe 16 core builders are used for nightly builds of PyTorch and it finishes in under 4 hours for all GPU architectures. If one builds for themselves it's not necessary to build for all arches, and this way builds are much faster.

[timothygebhard]

Here is the output from the those two commands (I hope I understood correctly how to use them):

(gdb) x/4i $rip
=> 0x15548827bc47 <mkl_vml_serv_GetMinN+135>:   idiv   %esi
   0x15548827bc49 <mkl_vml_serv_GetMinN+137>:   cvtsi2sd %eax,%xmm5
   0x15548827bc4d <mkl_vml_serv_GetMinN+141>:   mov    %ecx,%eax
   0x15548827bc4f <mkl_vml_serv_GetMinN+143>:   pxor   %xmm0,%xmm0

(gdb) info all-registers
rax            0x3b                59
rbx            0x7ffffffdc30c      140737488208652
rcx            0x3b                59
rdx            0x0                 0
rsi            0x0                 0
rdi            0x7ffffffdc30c      140737488208652
rbp            0x7ffffffdc360      0x7ffffffdc360
rsp            0x7ffffffdc2b0      0x7ffffffdc2b0
r8             0x0                 0
r9             0x15549ace7d20      23453118659872
r10            0x501ca5f           84003423
r11            0x56                86
r12            0x501ca00           84003328
r13            0x3e8               1000
r14            0x7fffffffc430      140737488340016
r15            0x3e8               1000
rip            0x15548827bc47      0x15548827bc47 <mkl_vml_serv_GetMinN+135>
eflags         0x10206             [ PF IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
st0            1.490116119384765625e-08 (raw 0x3fe58000000000000000)
st1            0                   (raw 0x00000000000000000000)
st2            0                   (raw 0x00000000000000000000)
st3            1.00000000000000000005e-31 (raw 0x3f9881ceb32c4b43fcf5)
st4            -31                 (raw 0xc003f800000000000000)
st5            10000000000000000   (raw 0x40348e1bc9bf04000000)
st6            nan(0xc000000000000000) (raw 0x7fffc000000000000000)
st7            4                   (raw 0x40018000000000000000)
fctrl          0x37f               895
fstat          0x20                32
ftag           0xffff              65535
fiseg          0x0                 0
fioff          0x0                 0
foseg          0x0                 0
fooff          0x0                 0
fop            0x0                 0
mxcsr          0x1fa2              [ DE PE IM DM ZM OM UM PM ]
ymm0           {v16_bfloat16 = {0x1440, 0x1e, 0x0 <repeats 14 times>}, v8_float = {0x1e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0xa2, 0x45, 0xf3, 0x41, 0x0 <repeats 28 times>}, v16_int16 = {0x45a2, 0x41f3, 0x0 <repeats 14 times>}, v8_int32 = {0x41f345a2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x41f345a2, 0x0, 0x0, 0x0}, v2_int128 = {0x41f345a2, 0x0}}
ymm1           {v16_bfloat16 = {0x0 <repeats 16 times>}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x0, 0x0}}
ymm2           {v16_bfloat16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_float = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x7fffffffffffffff, 0x7fffffffffffffff, 0x0, 0x0}, v32_int8 = {0xff <repeats 16 times>, 0x0 <repeats 16 times>}, v16_int16 = {0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0x0}, v2_int128 = {0xffffffffffffffffffffffffffffffff, 0x0}}
ymm3           {v16_bfloat16 = {0x0, 0x0, 0x0, 0x0 <repeats 13 times>}, v8_float = {0xca000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0xa0, 0xbc, 0x4d, 0x55, 0x55, 0x15, 0x0 <repeats 26 times>}, v16_int16 = {0xbca0, 0x554d, 0x1555, 0x0 <repeats 13 times>}, v8_int32 = {0x554dbca0, 0x1555, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x1555554dbca0, 0x0, 0x0, 0x0}, v2_int128 = {0x1555554dbca0, 0x0}}
ymm4           {v16_bfloat16 = {0x0 <repeats 16 times>}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x0, 0x0}}
ymm5           {v16_bfloat16 = {0x0 <repeats 16 times>}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x0, 0x0}}
ymm6           {v16_bfloat16 = {0x1440, 0x1e, 0x0 <repeats 14 times>}, v8_float = {0x1e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0xa2, 0x45, 0xf3, 0x41, 0x0 <repeats 28 times>}, v16_int16 = {0x45a2, 0x41f3, 0x0 <repeats 14 times>}, v8_int32 = {0x41f345a2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x41f345a2, 0x0, 0x0, 0x0}, v2_int128 = {0x41f345a2, 0x0}}
ymm7           {v16_bfloat16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x30, 0x2e, 0x37, 0x39, 0x39, 0x34, 0x2c, 0x20, 0x0 <repeats 16 times>}, v16_int16 = {0x2020, 0x2020, 0x2020, 0x2020, 0x2e30, 0x3937, 0x3439, 0x202c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x20202020, 0x20202020, 0x39372e30, 0x202c3439, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x2020202020202020, 0x202c343939372e30, 0x0, 0x0}, v2_int128 = {0x202c343939372e302020202020202020, 0x0}}
ymm8           {v16_bfloat16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x30, 0x2e, 0x36, 0x38, 0x38, 0x34, 0x2c, 0x20, 0x30, 0x2e, 0x34, 0x33, 0x34, 0x38, 0x2c, 0x20, 0x0 <repeats 16 times>}, v16_int16 = {0x2e30, 0x3836, 0x3438, 0x202c, 0x2e30, 0x3334, 0x3834, 0x202c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x38362e30, 0x202c3438, 0x33342e30, 0x202c3834, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x202c343838362e30, 0x202c383433342e30, 0x0, 0x0}, v2_int128 = {0x202c383433342e30202c343838362e30, 0x0}}
ymm9           {v16_bfloat16 = {0x0, 0x0, 0x2, 0x4, 0x0 <repeats 12 times>}, v8_float = {0x0, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x3e8, 0x0, 0x0, 0x0}, v32_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x40, 0x8f, 0x40, 0x0 <repeats 24 times>}, v16_int16 = {0x0, 0x0, 0x4000, 0x408f, 0x0 <repeats 12 times>}, v8_int32 = {0x0, 0x408f4000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x408f400000000000, 0x0, 0x0, 0x0}, v2_int128 = {0x408f400000000000, 0x0}}
ymm10          {v16_bfloat16 = {0x0, 0x0, 0x0, 0x1, 0x0 <repeats 12 times>}, v8_float = {0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0, 0x0, 0x0, 0x0, 0xe0, 0x7f, 0xad, 0x3f, 0x0 <repeats 24 times>}, v16_int16 = {0x0, 0x0, 0x7fe0, 0x3fad, 0x0 <repeats 12 times>}, v8_int32 = {0x0, 0x3fad7fe0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x3fad7fe000000000, 0x0, 0x0, 0x0}, v2_int128 = {0x3fad7fe000000000, 0x0}}
ymm11          {v16_bfloat16 = {0x0 <repeats 16 times>}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x0, 0x0}}
ymm12          {v16_bfloat16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x10, 0xba, 0x90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x10, 0xba, 0x90, 0x0 <repeats 21 times>}, v16_int16 = {0xba10, 0x90, 0x0, 0x0, 0xba10, 0x90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x90ba10, 0x0, 0x90ba10, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x90ba10, 0x90ba10, 0x0, 0x0}, v2_int128 = {0x90ba10000000000090ba10, 0x0}}
ymm13          {v16_bfloat16 = {0x0, 0x0, 0x0 <repeats 14 times>}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x10, 0xba, 0x90, 0x0 <repeats 29 times>}, v16_int16 = {0xba10, 0x90, 0x0 <repeats 14 times>}, v8_int32 = {0x90ba10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x90ba10, 0x0, 0x0, 0x0}, v2_int128 = {0x90ba10, 0x0}}
ymm14          {v16_bfloat16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x10, 0xba, 0x90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x10, 0xba, 0x90, 0x0 <repeats 21 times>}, v16_int16 = {0xba10, 0x90, 0x0, 0x0, 0xba10, 0x90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x90ba10, 0x0, 0x90ba10, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x90ba10, 0x90ba10, 0x0, 0x0}, v2_int128 = {0x90ba10000000000090ba10, 0x0}}
ymm15          {v16_bfloat16 = {0x0, 0x0, 0x0 <repeats 14 times>}, v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x10, 0xba, 0x90, 0x0 <repeats 29 times>}, v16_int16 = {0xba10, 0x90, 0x0 <repeats 14 times>}, v8_int32 = {0x90ba10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x90ba10, 0x0, 0x0, 0x0}, v2_int128 = {0x90ba10, 0x0}}

—

Regarding the long build time: I realized I made a dumb error on my side — I ran the build on a machine using a shared file system, which was a bad idea. When I re-run the build using local storage, it finishes in about 35 minutes 🙂

[malfet]

(gdb) x/4i $rip
=> 0x15548827bc47 <mkl_vml_serv_GetMinN+135>:   idiv   %esi

This looks like a good old integral division by zero (easily reproducible by the following code: https://godbolt.org/z/rh7d8MdEE ), so I'm confused why python runtime reports it as floating point exception.... (And even more confused what MKL is trying to do here.)

[piraka9011]

Using the nightly version works for me too.
Also FWIW, sharing how CoreWeave resolved this for nodes/pods running in their k8s cluster.

...applying lxcfs-admission-webhook: disabled has done the trick here for prior clients seeking the same solution.
There is a layer (lxcfs) that exposes /proc/cpuinfo /proc/meminfo /proc/stat etc to look like the limits section of the resources, so applications that read those to determine ie how many number of threads to spin up don’t spin up more than the limit. Seems like something pytorch reads there messes it up sometime. Doesn’t make sense to me, but you also don’t really need that feature when doing most ML tasks so I disabled it for your namespace.

[levmckinney]

Will there be a patch for torch 1.13 to resolve this? I'm still running into this issue using the 1.13.1-cuda11.6-cudnn8-runtime container.

[malfet]

@levmckinney no, there are usually no back ports to previous releases, but it should be fixed in PyTorch-2.0