Upgrade MKL-DNN commit #15116

hshen14 · 2018-12-29T03:26:50Z

The commit includes the optimizations for:

INT8 depthwise convolution on CLX
INT8 1x1 convolution on CLX
FP32/INT8 gemm convolution (group convolution)

test=develop

…convolutions; test=develop

hshen14 · 2018-12-29T04:41:36Z

http://ci.paddlepaddle.org/downloadBuildLog.html?buildId=43515&plain=true

[03:43:23]W: [Step 1/1] /paddle/build/third_party/ngraph/src/extern_ngraph/src/ngraph/runtime/cpu/mkldnn_emitter.cpp:810:51: error: expected type-specifier
[03:43:23]W: [Step 1/1] size_t primitive_index = insert_primitive(new mkldnn::relu_forward(

@baojun-nervana We want to upgrade MKL-DNN commit, which requires relu primitive should be changed to elementwise. Could you please update the code accordingly and include this PR? Thanks.

@luotao1

luotao1 · 2019-01-07T04:43:57Z

@hshen14 PR #15175 is merged, and the latest error is

[04:31:45][Step 1/1] /paddle/build/third_party/install/mkldnn/lib/libmkldnn.so: undefined reference to `cblas_sgemm_pack_get_size'

http://ci.paddlepaddle.org/viewLog.html?buildId=45858&tab=buildLog&buildTypeId=Paddle_PrCi&logTab=tree&filter=all&_focus=11035

hshen14 · 2019-01-07T05:15:20Z

@luotao1 Will try to reproduce locally first.

hshen14 · 2019-01-07T05:50:32Z

Confirmed with MKL-DNN team, MKL-DNN breaks compilation due to new packed BLAS API change (which is not available in earlier versions of Intel MKL). So, we have to upgrade MKLML to:

https://github.com/intel/mkl-dnn/releases/download/v0.17.2/mklml_lnx_2019.0.1.20181227.tgz

Is it possible to upgrade MKLML first? @luotao1

luotao1 · 2019-01-07T05:56:59Z

@hshen14 I will upgrade MKLML at first.

luotao1 · 2019-01-08T10:59:17Z

@hshen14 MKLML updated in #15190 is merged, you can update MKLDNN commit id.

hshen14 · 2019-01-09T01:09:14Z

@luotao1

PR_Windows_CI fail:
http://ci.paddlepaddle.org/viewLog.html?buildId=47144&buildTypeId=PaddleWindows_PrWindowsCi&tab=buildLog

[01:05:12] : [Step 2/5] C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: “cmd.exe”已退出，代码为 1。 [D:\home\BuildAgent\work\a9b0372f0aea0a80\build\python\paddle_python.vcxproj]

Another GPU failure?
http://ci.paddlepaddle.org/viewLog.html?buildId=47138&buildTypeId=Paddle_PrCiPython35&tab=buildLog

local_stderr: b'W0109 01:57:43.922690 30317 init.cc:121] AVX is available, Please re-compile on local machine\nW0109 01:57:49.484200 30317 device_context.cc:257] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 9.0, Runtime API Version: 8.0\nW0109 01:57:49.484375 30317 device_context.cc:265] device: 0, cuDNN Version: 7.0.\n\n\n\n\n\n\nTraceback (most recent call last):\n File "dist_se_resnext.py", line 258, in \n runtime_main(DistSeResneXt2x2)\n File "/paddle/build/python/paddle/fluid/tests/unittests/test_dist_base.py", line 204, in runtime_main\n model.run_trainer(args)\n File "/paddle/build/python/paddle/fluid/tests/unittests/test_dist_base.py", line 164, in run_trainer\n feed=feeder.feed(get_data()))\n File "/paddle/build/python/paddle/fluid/tests/unittests/test_dist_base.py", line 151, in get_data\n origin_batch = next(reader_generator)\n File "/paddle/build/python/paddle/batch.py", line 35, in batch_reader\n for instance in r:\n File "/paddle/build/python/paddle/reader/decorator.py", line 52, in reader\n for e in map(func, *rs):\n File "/paddle/build/python/paddle/dataset/flowers.py", line 130, in reader\n data = batch['data']\nKeyError: 'data'\n'
[01:57:50] test_dist_se_resnext failed

hshen14 · 2019-01-09T05:15:53Z

The rest result seems fail with different symptom after several re-runs. Is there any way to make the infrastructure more stable? @luotao1

luotao1 · 2019-01-09T06:55:48Z

Error log on local windows machine:

It seems that the output directory changes from lib to bin:

So we should update as follows:

Does mkldnn.dll decide to move from lib to bin?

hshen14 · 2019-01-09T07:15:46Z

Need help from @yinghu5

yinghu5 · 2019-01-10T03:44:20Z

Yes, MKL DNN team decide to move the destination folder from lib to bin/ as it is aligned with general binary libraries distribution. The change will be done in current master and later version (after V.17.2)

luotao1 · 2019-01-10T03:54:14Z

Thanks @yinghu5!
@hshen14, could you update lib to bin to pass the windows CI?

hshen14 · 2019-01-10T08:45:30Z

@luotao1

Seems Windows test fail:

http://ci.paddlepaddle.org/viewLog.html?buildId=48051&buildTypeId=PaddleWindows_PrWindowsCi&tab=buildLog

[08:30:43]W: [Step 4/5] Errors while running CTest
[08:30:43] : [Step 4/5]
[08:30:43] : [Step 4/5] 79% tests passed, 78 tests failed out of 372
[08:30:43] : [Step 4/5]
[08:30:43] : [Step 4/5] Total Test time (real) = 253.21 sec
[08:30:43] : [Step 4/5]
[08:30:43] : [Step 4/5] The following tests FAILED:
[08:30:43] : [Step 4/5] 1 - system_allocator_test (Exit code 0xc0000135
[08:30:43] : [Step 4/5] )
[08:30:43] : [Step 4/5] 2 - allocator_facade_test (Exit code 0xc0000135
......

luotao1 · 2019-01-10T08:52:18Z

Seems Windows test fail:

@wopeizl adds the MKLDNN bin/ into CI configure path and reruns the windows CI.

luotao1 · 2019-01-10T09:35:52Z

[08:42:44][Step 1/1] /paddle/paddle/fluid/inference/tests/api/tester_helper.h:91: Failure
[08:42:44][Step 1/1] The difference between pdata_ref[j] and pdata[j] is 0.0026035308837890625, which exceeds FLAGS_accuracy, where
[08:42:44][Step 1/1] pdata_ref[j] evaluates to 0.74058622121810913,
[08:42:44][Step 1/1] pdata[j] evaluates to 0.74318975210189819, and
[08:42:44][Step 1/1] FLAGS_accuracy evaluates to 0.001.
[08:42:44][Step 1/1] [  FAILED  ] Analyzer_dam.compare (2522 ms)

It seems that MKL has random diff.
http://ci.paddlepaddle.org/viewLog.html?buildId=47988&tab=buildLog&buildTypeId=Paddle_PrCi&logTab=tail

hshen14 · 2019-01-10T09:48:00Z

As mentioned by @luotao1, there is random failure in MKL. Could you please help on that? @yinghu5 Thanks.

luotao1 · 2019-01-15T03:28:55Z

The earliest MKL diff is in Dec 15th. @yinghu5 @bingyanghuang @yihuaxu

[21:24:34][Step 1/1] /paddle/paddle/fluid/inference/tests/api/tester_helper.h:87: Failure
[21:24:34][Step 1/1] The difference between pdata_ref[j] and pdata[j] is 0.0028614401817321777, which exceeds 1e-3, where
[21:24:34][Step 1/1] pdata_ref[j] evaluates to 0.74058622121810913,
[21:24:34][Step 1/1] pdata[j] evaluates to 0.74344766139984131, and
[21:24:34][Step 1/1] 1e-3 evaluates to 0.001.
[21:24:34][Step 1/1] [  FAILED  ] Analyzer_dam.compare (542 ms)

http://ci.paddlepaddle.org/viewLog.html?tab=buildLog&buildTypeId=Paddle_PrCiNight&buildId=37730&_focus=21309#_state=65

luotao1 · 2019-01-15T05:40:16Z

@hshen14 I rerun the PR_CI again, and the new error is:

[04:54:15][Step 1/1] 328/525 Test #349: test_conv2d_int8_mkldnn_op ......................***Failed    2.31 sec
[04:54:15][Step 1/1] *** Aborted at 1547528055 (unix time) try "date -d @1547528055" if you are using GNU date ***
[04:54:15][Step 1/1] PC: @                0x0 (unknown)
[04:54:15][Step 1/1] *** SIGSEGV (@0x7fcb44d19e80) received by PID 71119 (TID 0x7fcb4ba6a700) from PID 1154588288; stack trace: ***
[04:54:15][Step 1/1]     @     0x7fcb4b649390 (unknown)
[04:54:15][Step 1/1]     @     0x7fcb06366f7b _ZN6mkldnn4impl3cpu32_gemm_x8s8s32x_convolution_fwd_tIL18mkldnn_data_type_t6ELS3_6EE8pp_ker_tclEPhPKiPKcPKffffimm
[04:54:15][Step 1/1]     @     0x7fcb0636c2bb _ZNK6mkldnn4impl3cpu32_gemm_x8s8s32x_convolution_fwd_tIL18mkldnn_data_type_t6ELS3_6EE19execute_forward_thrEiiPKhPKaPKcPhRKNS0_15memory_tracking9grantor_tE
[04:54:15][Step 1/1]     @     0x7fcb0636c60b _ZNK6mkldnn4impl3cpu32_gemm_x8s8s32x_convolution_fwd_tIL18mkldnn_data_type_t6ELS3_6EE15execute_forwardEv
[04:54:15][Step 1/1]     @     0x7fcb05c6abc9 _ZNK6mkldnn4impl3cpu32_gemm_x8s8s32x_convolution_fwd_tIL18mkldnn_data_type_t6ELS3_6EE7executeEPNS0_7event_tE
[04:54:15][Step 1/1]     @     0x7fcb05c5c183 mkldnn::impl::cpu::cpu_engine_t::submit()
[04:54:15][Step 1/1]     @     0x7fcb065bc096 mkldnn::impl::stream_eager_t::submit_impl()
[04:54:15][Step 1/1]     @     0x7fcb065bb481 mkldnn_stream::submit()
[04:54:15][Step 1/1]     @     0x7fcb065bb698 mkldnn_stream_submit
[04:54:15][Step 1/1]     @     0x7fcb079a2273 mkldnn::stream::submit()
[04:54:15][Step 1/1]     @     0x7fcb07f561ee paddle::operators::ConvMKLDNNOpKernel<>::ComputeINT8()
[04:54:15][Step 1/1]     @     0x7fcb07f58d17 _ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelRegistrarFunctorINS0_8platform8CPUPlaceELb0ELm0EJNS0_9operators18ConvMKLDNNOpKernelIhfEEEEclEPKcSE_iEUlS4_E_E9_M_invokeERKSt9_Any_dataS4_
[04:54:15][Step 1/1]     @     0x7fcb092a2850 paddle::framework::OperatorWithKernel::RunImpl()
[04:54:15][Step 1/1]     @     0x7fcb0929cdfd paddle::framework::OperatorBase::Run()
[04:54:15][Step 1/1]     @     0x7fcb0789280e paddle::framework::Executor::RunPreparedContext()
[04:54:15][Step 1/1]     @     0x7fcb07894eaa paddle::framework::Executor::Run()
[04:54:15][Step 1/1]     @     0x7fcb07755188 _ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL18pybind11_init_coreERNS_6moduleEEUlRNS2_9framework8ExecutorERKNS6_11ProgramDescEPNS6_5ScopeEibbE85_vJS8_SB_SD_ibbEJNS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESV_
[04:54:15][Step 1/1]     @     0x7fcb077aaf0d pybind11::cpp_function::dispatcher()
[04:54:15][Step 1/1]     @           0x4c5326 PyEval_EvalFrameEx
[04:54:15][Step 1/1]     @           0x4b9b66 PyEval_EvalCodeEx
[04:54:15][Step 1/1]     @           0x4c1f56 PyEval_EvalFrameEx
[04:54:15][Step 1/1]     @           0x4b9b66 PyEval_EvalCodeEx
[04:54:15][Step 1/1]     @           0x4c17c6 PyEval_EvalFrameEx
[04:54:15][Step 1/1]     @           0x4b9b66 PyEval_EvalCodeEx
[04:54:15][Step 1/1]     @           0x4c17c6 PyEval_EvalFrameEx
[04:54:15][Step 1/1]     @           0x4b9b66 PyEval_EvalCodeEx
[04:54:15][Step 1/1]     @           0x4c17c6 PyEval_EvalFrameEx
[04:54:15][Step 1/1]     @           0x4b9b66 PyEval_EvalCodeEx
[04:54:15][Step 1/1]     @           0x4c1f56 PyEval_EvalFrameEx
[04:54:15][Step 1/1]     @           0x4b9b66 PyEval_EvalCodeEx
[04:54:15][Step 1/1]     @           0x4d57a3 (unknown)
[04:54:15][Step 1/1]     @           0x4a587e PyObject_Call
[04:54:15][Step 1/1] Segmentation fault

http://ci.paddlepaddle.org/viewLog.html?buildId=49403&tab=buildLog&buildTypeId=Paddle_PrCi&logTab=tree&filter=all&_focus=22857#_state=23303

hshen14 · 2019-01-15T06:33:38Z

@xiaolil1 will take a look. BTW, is it reproducible 100% or a random issue? @luotao1

luotao1 · 2019-01-15T06:49:59Z

@hshen14 @xiaolil1 Though it's a random fail, it has the risk in conv2d_int8_mkldnn.

hshen14 · 2019-01-15T07:08:57Z

@hshen14 @xiaolil1 Though it's a random fail, it has the risk in conv2d_int8_mkldnn.

We will try reproducing locally and investigate the reason. From the crash message, it shows the potential issue of GEMM INT8, so conv INT8 kernel itself should be okay.

luotao1 · 2019-01-16T08:40:17Z

http://ci.paddlepaddle.org/viewType.html?buildTypeId=Paddle_PrCi&branch_Paddle=15116&tab=buildTypeStatusDiv
All the machines of failedconv2d_int8_mkldnn are 5117. @hshen14 @bingyanghuang

hshen14 · 2019-01-18T01:34:37Z

@luotao1 @yinghu5 INT8 UT failure was caused by a potential regression from MKL-DNN as similar symptom was reported from other DL frameworks. We had an internal bug tracking the issue and will keep you updated.

hshen14 · 2019-02-01T06:52:24Z

@luotao1 One Windows failure - rerun now:
[03:39:45] : [Step 2/5] C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB4018: “CustomBuild”任务意外失败。 [D:\home\BuildAgent\work\a9b0372f0aea0a80\build\paddle\fluid\operators\ctc_align_op.vcxproj]
[03:39:45] : [Step 2/5] C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB4018: System.IO.IOException: 文件“D:\home\BuildAgent\temp\buildTmp\tmp9fda56ecc91a40e6988890b6b8543a61.cmd”正由另一进程使用，因此该进程无法访问此文件。 [D:\home\BuildAgent\work\a9b0372f0aea0a80\build\paddle\fluid\operators\ctc_align_op.vcxproj]

http://ci.paddlepaddle.org/viewLog.html?buildId=55875&buildTypeId=PaddleWindows_PrWindowsCi&tab=buildLog

@panyx0718 Need your approval for cmake external change introduced by MKL-DNN change
[03:44:37] : [Step 1/1] You must have panyx0718 approval for the api change! cmake/external
[03:44:37]W: [Step 1/1] + APPROVALS=FALSE
[03:44:37]W: [Step 1/1] + echo 'current pr 15116 got approvals: FALSE'
[03:44:37]W: [Step 1/1] + '[' FALSE == FALSE ']'
[03:44:37]W: [Step 1/1] + echo 'You must have panyx0718 approval for the api change! cmake/external'
[03:44:37]W: [Step 1/1] + exit 1

http://ci.paddlepaddle.org/viewLog.html?buildId=55852&buildTypeId=Paddle_PrCi&tab=buildLog

hshen14 · 2019-02-15T08:26:03Z

@luotao1 @kbinias @jianhang-liu @yinghu5 After discussion, we agreed to have two-stage MKL-DNN upgrade. @kbinias's team was working on validation with MKL-DNN v0.18rc and will prepare a PR after successful validation. I will keep this PR open for reference and close it when the new PR with v0.18rc is submitted.

hshen14 · 2019-02-22T00:38:13Z

The new MKL-DNN upgrade was WIP: #15861

Close this intermediate PR.

Upgrade MKL-DNN commit with optmizations on depthwise, 1x1, and gemm …

b0e508e

…convolutions; test=develop

luotao1 added the Intel label Dec 30, 2018

baojun-nervana mentioned this pull request Jan 4, 2019

Update ngraph to resolve issue with mkldnn upgrade #15175

Merged

Merge branch 'public-dev' into develop

179ec9e

luotao1 mentioned this pull request Jan 7, 2019

update mklml version #15190

Merged

Merge branch 'public-dev' into develop

b22af7e

Merge from develop and resolve conflicts

038dff8

hshen14 added 2 commits January 10, 2019 12:45

Merge branch 'public-dev' into develop

2e4e6b6

Adapt the new binary folder after MKL-DNN v0.17.2

94fdd35

luotao1 mentioned this pull request Jan 15, 2019

test_analyzer_* random crashs at compare_mkldnn #15032

Closed

hshen14 added 2 commits February 1, 2019 09:38

Merge remote-tracking branch 'public-paddle-dev/develop' into develop

69bb9be

Try to upgrade with a new MKL-DNN version (0.17.2 tag); test=develop

0962be9

hshen14 closed this Feb 22, 2019

jianhang-liu mentioned this pull request Feb 22, 2019

Upgrade MKLDNN to v0.18 and fix issue caused by lib/lib64 #15861

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade MKL-DNN commit #15116

Upgrade MKL-DNN commit #15116

hshen14 commented Dec 29, 2018 •

edited

hshen14 commented Dec 29, 2018

luotao1 commented Jan 7, 2019

hshen14 commented Jan 7, 2019

hshen14 commented Jan 7, 2019

luotao1 commented Jan 7, 2019

luotao1 commented Jan 8, 2019

hshen14 commented Jan 9, 2019 •

edited

hshen14 commented Jan 9, 2019

luotao1 commented Jan 9, 2019

hshen14 commented Jan 9, 2019

yinghu5 commented Jan 10, 2019

luotao1 commented Jan 10, 2019

hshen14 commented Jan 10, 2019

luotao1 commented Jan 10, 2019

luotao1 commented Jan 10, 2019 •

edited

hshen14 commented Jan 10, 2019

luotao1 commented Jan 15, 2019

luotao1 commented Jan 15, 2019

hshen14 commented Jan 15, 2019 •

edited

luotao1 commented Jan 15, 2019

hshen14 commented Jan 15, 2019

luotao1 commented Jan 16, 2019

hshen14 commented Jan 18, 2019 •

edited

hshen14 commented Feb 1, 2019

hshen14 commented Feb 15, 2019

hshen14 commented Feb 22, 2019

Upgrade MKL-DNN commit #15116

Upgrade MKL-DNN commit #15116

Conversation

hshen14 commented Dec 29, 2018 • edited

hshen14 commented Dec 29, 2018

luotao1 commented Jan 7, 2019

hshen14 commented Jan 7, 2019

hshen14 commented Jan 7, 2019

luotao1 commented Jan 7, 2019

luotao1 commented Jan 8, 2019

hshen14 commented Jan 9, 2019 • edited

hshen14 commented Jan 9, 2019

luotao1 commented Jan 9, 2019

hshen14 commented Jan 9, 2019

yinghu5 commented Jan 10, 2019

luotao1 commented Jan 10, 2019

hshen14 commented Jan 10, 2019

luotao1 commented Jan 10, 2019

luotao1 commented Jan 10, 2019 • edited

hshen14 commented Jan 10, 2019

luotao1 commented Jan 15, 2019

luotao1 commented Jan 15, 2019

hshen14 commented Jan 15, 2019 • edited

luotao1 commented Jan 15, 2019

hshen14 commented Jan 15, 2019

luotao1 commented Jan 16, 2019

hshen14 commented Jan 18, 2019 • edited

hshen14 commented Feb 1, 2019

hshen14 commented Feb 15, 2019

hshen14 commented Feb 22, 2019

hshen14 commented Dec 29, 2018 •

edited

hshen14 commented Jan 9, 2019 •

edited

luotao1 commented Jan 10, 2019 •

edited

hshen14 commented Jan 15, 2019 •

edited

hshen14 commented Jan 18, 2019 •

edited