No outputs with the IPEX optimized model on Colab

[sayakpaul]

I am on Colab and here's the CPU stat:

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              2
On-line CPU(s) list: 0,1
Thread(s) per core:  2
Core(s) per socket:  1
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               79
Model name:          Intel(R) Xeon(R) CPU @ 2.20GHz
Stepping:            0
CPU MHz:             2199.998
BogoMIPS:            4399.99
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            56320K
NUMA node0 CPU(s):   0,1
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities

I installed the extension with

!python -m pip install intel_extension_for_pytorch -f https://software.intel.com/ipex-whl-stable

When I am running:

import torch
import torchvision.models as models

model = models.resnet50(pretrained=True)
model.eval()
data = torch.rand(1, 3, 224, 224)

model = model.to(memory_format=torch.channels_last)
data = data.to(memory_format=torch.channels_last)

#################### code changes ####################
import intel_extension_for_pytorch as ipex
model = ipex.optimize(model)
######################################################

with torch.no_grad():
  print(model(data))

It doesn't do anything. In fact when run in the interactive mode, it makes Colab restart the runtime. But when run from the Colab terminal (or from shell using python <script_name>.py) it doesn't print anything.

[EikanWang]

Thanks for your information. We are working on this issue and will keep you posted if there is any finding.

[jgong5]

This issue should have the same root cause as #235. @sayakpaul Please check my comment there: #235 (comment)

[EikanWang]

@sayakpaul , we released the latest IPEX to fix the issue. Could you please try the latest IPEX and check whether the issue has been fixed?

[EikanWang]

https://pypi.org/project/intel-extension-for-pytorch/1.12.100/

[sayakpaul]

Will do.

[rahulunair]

I just checked and it works now with ipex version 1.12.100, I can see output when we print the result using:

print(model(data)

          9.0761e-01,  2.1421e+00,  3.2412e-01,  1.0827e+00,  1.9377e+00,
          2.4419e-01,  7.5711e-01, -3.6044e+00,  2.2269e-01, -5.7394e-03,
          1.2909e+00,  3.4097e+00,  1.4625e+00, -9.2349e-01,  2.2971e-01,
          1.3845e+00,  1.9893e-02, -2.5797e-02,  1.0002e+00,  1.7227e+00,
          4.5895e-02,  1.7255e+00,  7.7651e-01,  1.1131e+00, -2.2579e+00,
          3.2223e+00,  1.2480e+00,  4.5337e-01,  6.0977e-01, -1.5743e+00,
          9.6624e-01,  1.8643e+00,  6.0268e-01, -3.0862e-01,  4.0406e-01,
         -4.9450e-01,  7.4416e-01,  3.7605e-01,  2.1783e+00,  2.1417e+00,
          1.6291e+00, -2.4393e+00,  9.1662e-01,  2.3993e+00,  4.4553e-01,
         -1.0598e+00, -7.0069e-02, -1.8448e+00, -7.3401e-01,  8.3961e-01,
          3.4538e+00,  1.5848e+00, -2.6664e+00,  3.7892e+00,  3.1986e+00,
          6.6627e-01, -1.6759e+00,  2.2567e+00, -1.2109e+00,  2.9752e+00,
          4.7682e-01,  2.1077e+00,  3.5374e+00,  1.4244e+00,  7.9581e-01,
         -1.1950e+00,  8.4212e-01,  3.1894e+00,  1.6200e+00, -1.4924e+00,
         ....

Please find the colab link for verifying: https://colab.research.google.com/drive/1HZFMT5IcHFykk0MafsNBAS2uJcGW2am1?usp=sharing

[sayakpaul]

@rahulunair thanks for providing the Colab.

I further extended it to include the following:

• An assertion check to compare if the PT model outputs and the Intel-optimized model outputs are the same (with a given tolerance).
• A benchmark to compare their runtimes.

@EikanWang I can confirm that with IPEX (version 1.12.100) it works.

With the vanilla PT model, the average latency is 0.1710120957200047 secs and with the optimized model it's 0.13993526895998912 secs.

Here's my extended Colab: https://colab.research.google.com/gist/sayakpaul/2d31764c53da3c959c9294d457a3c0eb/scratchpad.ipynb.

I believe with an AVX512-capable machine, the runtime boosts will be more significant. Colab (free version) doesn't provide a machine equipped with AVX512 I believe.