Different Dropout behavior on macOS and Linux

[rasbt]

🐛 Describe the bug

Even when using a fixed random seed, the dropout masks in macOS and Linux are different. Here is some code to reproduce this:

import torch

print(torch.__version__)

t = torch.tensor([
        [1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.5517, 0.4483, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.3800, 0.3097, 0.3103, 0.0000, 0.0000, 0.0000],
        [0.2758, 0.2460, 0.2462, 0.2319, 0.0000, 0.0000],
        [0.2175, 0.1983, 0.1984, 0.1888, 0.1971, 0.0000]])

torch.manual_seed(123)
dropout = torch.nn.Dropout(0.5)

print(dropout(t))

Below are the results. Note that this is consistent across PyTorch versions (I tested 2.1.0 and 2.2.1) and whether the code is run on the CPU or GPU:

macOS torch 2.1.0, CPU

2.1.0
tensor([[2.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.7600, 0.6194, 0.6206, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.4920, 0.4924, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.3966, 0.0000, 0.3776, 0.0000, 0.0000],
        [0.0000, 0.3326, 0.3332, 0.3084, 0.3332, 0.0000]])

macOS torch 2.2.1, CPU

tensor([[2.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.7600, 0.6194, 0.6206, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.4920, 0.4924, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.3966, 0.0000, 0.3776, 0.0000, 0.0000],
        [0.0000, 0.3326, 0.3332, 0.3084, 0.3332, 0.0000]])

Google Colab (Linux) torch 2.1.0, CPU

2.1.0+cu121
tensor([[2.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.8966, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.6206, 0.0000, 0.0000, 0.0000],
        [0.5516, 0.4920, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4350, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.3326, 0.0000, 0.0000, 0.0000, 0.0000]])

Google Colab (Linux) torch 2.2.1, CPU

2.2.1+cu121
tensor([[2.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.8966, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.6206, 0.0000, 0.0000, 0.0000],
        [0.5516, 0.4920, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4350, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.3326, 0.0000, 0.0000, 0.0000, 0.0000]])

Google Colab (Linux) torch 2.1.0, CPU

2.1.0+cu121

tensor([[2.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.8966, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.7600, 0.6194, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.4920, 0.0000, 0.4638, 0.0000, 0.0000],
        [0.4350, 0.3966, 0.3968, 0.0000, 0.0000, 0.0000],
        [0.3870, 0.3326, 0.3332, 0.3084, 0.3332, 0.0000]], device='cuda:0')

Versions

MacOS environment

Collecting environment information...
PyTorch version: 2.2.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.3.1 (arm64)
GCC version: Could not collect
Clang version: 12.0.5 (clang-1205.0.22.9)
CMake version: version 3.26.1
Libc version: N/A

Python version: 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:41:52) [Clang 15.0.7 ] (64-bit runtime)
Python platform: macOS-14.3.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1

Versions of relevant libraries:
[pip3] numpy==1.26.0
[pip3] torch==2.2.1
[conda] numpy                     1.26.0                   pypi_0    pypi
[conda] torch                     2.2.1                    pypi_0    pypi

Linux environment

Collecting environment information...
PyTorch version: 2.1.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.27.9
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.1.58+-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 535.104.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.6
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             8
On-line CPU(s) list:                0-7
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family:                         6
Model:                              79
Thread(s) per core:                 2
Core(s) per socket:                 4
Socket(s):                          1
Stepping:                           0
BogoMIPS:                           4399.99
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          128 KiB (4 instances)
L1i cache:                          128 KiB (4 instances)
L2 cache:                           1 MiB (4 instances)
L3 cache:                           55 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-7
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Mitigation; PTE Inversion
Vulnerability Mds:                  Vulnerable; SMT Host state unknown
Vulnerability Meltdown:             Vulnerable
Vulnerability Mmio stale data:      Vulnerable
Vulnerability Retbleed:             Vulnerable
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:           Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Vulnerable

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] torch==2.1.0+cu121
[pip3] torchaudio==2.1.0+cu121
[pip3] torchdata==0.7.0
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.16.0
[pip3] torchvision==0.16.0+cu121
[pip3] triton==2.1.0
[conda] Could not collect

cc @pbelevich

[albanD]

I'm afraid this is expected, see the first line in the reproducibility doc: https://pytorch.org/docs/stable/notes/randomness.html

[rasbt]

Your code is incomplete, missing closing ')' and ']', so I use my code :
...
the results are same on MacOS and Linux with CPU backend.
2.3.0a0+git660ec3d
tensor([[ 0.0000, -0.0000, -0.6071, -0.0000, 0.6972],
[ 0.0000, -0.0000, -0.7583, 1.5342, -0.0000],
[ 1.3914, -3.6121, 3.7919, -0.0000, 0.0000],
[-3.2065, -0.0000, -2.8191, -0.8153, 0.0000],
[ 1.9971, 0.0000, 3.6638, -0.6757, 1.7611]])

Can you update the code and try latest build?

Sorry about the missing ]). I just fixed that. I also just reran it on the latest nightly, and the bug still persists:

Linux (CPU)

2.3.0.dev20240311+cu121
tensor([[2.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.8966, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.6206, 0.0000, 0.0000, 0.0000],
        [0.5516, 0.4920, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4350, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]])

macOS (CPU)

2.3.0.dev20240311
tensor([[2.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.7600, 0.6194, 0.6206, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.4920, 0.4924, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.3966, 0.0000, 0.3776, 0.0000, 0.0000]])

[rasbt]

Thanks for the response @albanD , which section from the doc do you mean in particular so that I can quote it with an explanation.

I am still surprised that it produces different results though. Other scientific packages like NumPy, SciPy, Scikit-Learn etc are consistent on macos and Linux.

Operations like random weight initialization in PyTorch are also consistent across macOS and Linux. It seems that only Dropout is impacted.

E.g.,

import torch

torch.manual_seed(123)
torch.nn.Linear(4, 4).weight

returns

Parameter containing:
tensor([[-0.2039,  0.0166, -0.2483,  0.1886],
        [-0.4260,  0.3665, -0.3634, -0.3975],
        [-0.3159,  0.2264, -0.1847,  0.1871],
        [-0.4244, -0.3034, -0.1836, -0.0983]], requires_grad=True)

on both macOS and Linux.

[albanD]

The first paragraph:
"
Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds.
"
In general, I do think that this is the kind of expectation you should have.

That being said, we do try to be consistent but it is not always easy and is hard to enforce.
For example, my local linux machine gives me the same result as your "macOS CPU".

I'm afraid this is not going to be very high priority as, as you pointed out, it has been like that for a while.
If you do find where the discrepancy comes from, I'm happy to review a PR fixing it though!

[rasbt]

Completely reproducible results are not guaranteed across ... different platforms.

Thanks. For some reason my brain skipped over this and focused on the "between CPU and GPU executions" part.

If you do find where the discrepancy comes from, I'm happy to review a PR fixing it though!

Sounds fair!

[LamForest]

@albanD @rasbt

I try to dig little deeper to this issue. It seems to be a consistency problem between MKL and non-MKL implemation of bernoulli_.float.

Dropout use bernoulli_.float to produce mask, which has 2 versions of implemention: MKL and non-MKL, controlled by C++ macro AT_MKL_ENABLED().

1. non-MKL implementation:
   

   pytorch/aten/src/ATen/native/cpu/DistributionKernels.cpp

   Line 45 in d1510e0

   void bernoulli_scalar_kernel(const TensorBase &self, double p, c10::optional<Generator> gen) {

Using this code:

import torch
a = torch.ones(10)
torch.manual_seed(123)
a.bernoulli_(0.5)
print(a)

the output is: tensor([1., 1., 0., 1., 1., 0., 0., 0., 0., 1.])

2. MKL implementation:
   

   pytorch/aten/src/ATen/native/cpu/DistributionKernels.cpp

   Line 49 in d1510e0

   void bernoulli_scalar_kernel(const TensorBase &self, double p, c10::optional<Generator> gen) {

With the same code, the output is tensor([1., 1., 1., 1., 1., 1., 0., 1., 0., 0.])

---

We can see the discrepancy between two versions of bernoulli_.float. Anyone can build pytorch with or without MKL to reproduce the above result.

@rasbt IMO, if you are using mac silicon, it lacks MKL support but official linux pytorch release has MKL support. This is the reason why linux and mac cpu results differ.

@albanD This is not across platform or device, is this expected?

[malfet]

IMO this still fits expected behavior pattern. According to #69967 MKL implementation is much faster, and still statistical criteria for the RNG. But perhaps one can disable it if say deterministic mode is enabled

[LamForest]

IMO this still fits expected behavior pattern. According to #69967 MKL implementation is much faster, and still statistical criteria for the RNG. But perhaps one can disable it if say deterministic mode is enabled

If MKL generates rng parellelly to speed up, just like CUDA, then the result inevitable will be different since RNG algorithm is different.

As you suggest, can we use non-MKL implementaion when use_deterministic_algorithms(True)?

[albanD]

We could do that but it won't allow you to rely on the fact that the generated number will be the same accross version and platform as other things might lead to the same behavior in the future.

I don't think we want to do anything here since the discrepancy is from build-time availability and lead to significant speedup.

[LamForest]

@albanD I agree. At least can I add this behaviour to bernoulli and dropout's document? IMO, this behaviour is out of most of users' expectations.

[albanD]

I don't think bernoulli and dropout are special here. If we do something, we could add a comment to every single random function pointing to https://pytorch.org/docs/stable/notes/randomness.html .
cc @svekars do you think that's an ok pattern to add to our doc?