Segmentation fault when doing backward for loss func #443
Description
Describe the bug
I tried to test if the following code works in the dev clouds, but it crashed almost immediately after the backward() was called.
#!/usr/bin/env python
# encoding: utf-8
'''
==============================================================
Copyright © 2019 Intel Corporation
SPDX-License-Identifier: MIT
==============================================================
'''
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import intel_extension_for_pytorch as ipex
'''
BS_TRAIN: Batch size for training data
BS_TEST: Batch size for testing data
EPOCHNUM: Number of epoch for training
'''
BS_TRAIN = 50
BS_TEST = 10
EPOCHNUM = 2
'''
TestDataset class is inherited from torch.utils.data.Dataset.
Since data for training involves data and ground truth, a flag "train" is defined in the initialization function. When train is True, instance of TestDataset gets a pair of training data and label data. When it is False, the instance gets data only for inference. Value of the flag "train" is set in __init__ function.
In __getitem__ function, data at index position is supposed to be returned.
__len__ function returns the overall length of the dataset.
'''
class TestDataset(Dataset):
def __init__(self, train = True):
super(TestDataset, self).__init__()
self.train = train
def __getitem__(self, index):
if self.train:
return torch.rand(3, 112, 112), torch.rand(6, 110, 110)
else:
return torch.rand(3, 112, 112)
def __len__(self):
if self.train:
return 100
else:
return 20
'''
TestModel class is inherited from torch.nn.Module.
Operations that will be used in the topology are defined in __init__ function.
Input data x is supposed to be passed to the forward function. The topology is implemented in the forward function. When perform training/inference, the forward function will be called automatically by passing input data to a model instance.
'''
class TestModel(nn.Module):
def __init__(self):
super(TestModel, self).__init__()
self.conv = nn.Conv2d(3, 6, 3)
self.norm = nn.BatchNorm2d(6)
self.relu = nn.ReLU()
def forward(self, x):
x = self.conv(x)
x = self.norm(x)
x = self.relu(x)
return x
'''
Perform training and inference in main function
'''
def main():
'''
The following 3 components are required to perform training.
1. model: Instantiate model class
2. optim: Optimization function for update topology parameters during training
3. crite: Criterion function to minimize loss
'''
model = TestModel()
optim = torch.optim.SGD(model.parameters(), lr=0.01)
crite = nn.MSELoss(reduction='sum')
'''
1. Instantiate the Dataset class defined before
2. Use torch.utils.data.DataLoader to load data from the Dataset instance
'''
train_data = TestDataset()
trainLoader = DataLoader(train_data, batch_size=BS_TRAIN)
test_data = TestDataset(train=False)
testLoader = DataLoader(test_data, batch_size=BS_TEST)
'''
Apply Intel Extension for PyTorch optimization against the model object and optimizer object.
'''
model, optim = ipex.optimize(model, optimizer=optim)
'''
Perform training and inference
Use model.train() to set the model into train mode. Use model.eval() to set the model into inference mode.
Use for loop with enumerate(instance of DataLoader) to go through the whole dataset for training/inference.
'''
for i in range(0, EPOCHNUM - 1):
'''
Iterate dataset for training to train the model
'''
model.train()
for batch_index, (data, y_ans) in enumerate(trainLoader):
optim.zero_grad()
y = model(data)
loss = crite(y, y_ans)
print(loss)
loss.backward()
optim.step()
'''
Iterate dataset for validation to evaluate the model
'''
model.eval()
for batch_index, data in enumerate(testLoader):
y = model(data)
if __name__ == '__main__':
main()
print('[CODE_SAMPLE_COMPLETED_SUCCESFULLY]')```
### Versions
PyTorch version: 2.0.1a0+cxx11.abi
PyTorch CXX11 ABI: Yes
IPEX version: 2.0.110+xpu
IPEX commit: ba7f6c127
Build type: Release
OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: N/A
IGC version: 2023.2.0 (2023.2.0.20230721)
CMake version: version 3.16.3
Libc version: glibc-2.31
Python version: 3.9.16 (main, Jun 15 2023, 02:33:25) [GCC 13.1.0] (64-bit runtime)
Python platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.31
Is XPU available: False
DPCPP runtime version: 2023.2.1
MKL version: 2023.2.0
GPU models and configuration:
Intel OpenCL ICD version: N/A
Level Zero version: N/A
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
Stepping: 4
CPU MHz: 1200.172
CPU max MHz: 3700.0000
CPU min MHz: 1200.0000
BogoMIPS: 6800.00
Virtualization: VT-x
L1d cache: 384 KiB
L1i cache: 384 KiB
L2 cache: 12 MiB
L3 cache: 38.5 MiB
NUMA node0 CPU(s): 0-5,12-17
NUMA node1 CPU(s): 6-11,18-23
Vulnerability Itlb multihit: KVM: Vulnerable
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities
Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.0.110+xpu
[pip3] numpy==1.24.3
[pip3] torch==2.0.1a0+cxx11.abi
[pip3] torchvision==0.15.2a0+cxx11.abi
[conda] intel-extension-for-pytorch 2.0.110 py39_xpu_0 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] mkl 2023.2.0 intel_49495 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] mkl-dpcpp 2023.2.0 intel_49495 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] mkl-service 2.4.0 py39hae59892_35 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] mkl_fft 1.3.6 py39h173b8ae_56 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] mkl_random 1.2.2 py39h1595b48_76 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] mkl_umath 0.1.1 py39hd987cd3_86 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] numpy 1.24.3 py39hed7eef7_0 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] numpy-base 1.24.3 py39he88ecf9_0 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] pytorch 2.0.1 py39_xpu_0 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel
[conda] torchvision 0.15.2 py39_0 file:///glob/development-tools/versions/oneapi/2023.2.0.1/oneapi/conda_channel