test_analyzer_* random crashs at `compare_mkldnn` #15032

luotao1 · 2018-12-25T04:35:35Z

test_analyzer_small_dam random crashes at compare_mkldnn in three nightly CI at a same machine (5117).

test_analyzer_small_dam (NUMERICAL)

http://ci.paddlepaddle.org/viewLog.html?buildId=40622&tab=buildLog&buildTypeId=Paddle_PrCiNight&logTab=tree&filter=all&_focus=21635
http://ci.paddlepaddle.org/viewLog.html?tab=buildLog&buildTypeId=Paddle_PrCiNight&buildId=40460&_focus=22113
http://ci.paddlepaddle.org/viewLog.html?buildId=40324&tab=buildLog&buildTypeId=Paddle_PrCiNight&logTab=tree&filter=all&_focus=21764

test_analyzer_dam random crashes at compare_mkldnn in nightly CI
http://ci.paddlepaddle.org/viewLog.html?buildId=44600&tab=buildLog&buildTypeId=Paddle_PrCiNight&logTab=tree&filter=all&_focus=22935
test_analyzer_mobilenet_depthwise_conv crashes at compare_mkldnn in nightly CI
http://ci.paddlepaddle.org/viewLog.html?buildId=44596&tab=buildLog&buildTypeId=Paddle_PrCiNight&logTab=tree&filter=all&_focus=23004

The text was updated successfully, but these errors were encountered:

luotao1 · 2018-12-25T04:42:15Z

Is the same problem like #14174?

jianhang-liu · 2019-01-03T02:33:27Z

@jczaja Could you help to have a check at this? Thanks!

luotao1 · 2019-01-03T02:35:46Z

It seems that mkldnn hangs from the log, is a openmp problem?

jczaja · 2019-01-03T08:17:30Z

@luotao1 We will look to that and get back to You

jczaja · 2019-01-03T16:52:04Z

@luotao1 I did not get to reproduce problem.

Could you make a changes so that : compare_mkldnn works without mkl_dnn eg. compare(false) and then run experiments. So we know if MKL-DNN is actual problem ?
How did you observe that mkldnn hangs on logging? Did you attach debugger and observe where execution is looping over?
Do you use parallel mode of execution of tests of ctest? What is ctest commandline to run tests?

luotao1 · 2019-01-04T02:47:18Z

Could you make a changes so that : compare_mkldnn works without mkl_dnn eg. compare(false) and then run experiments. So we know if MKL-DNN is actual problem ?

We have compare UT which works without mkldnn, and it all runs successfully. You can see these UTs in url.

[22:59:02][Step 1/1] [       OK ] Analyzer_dam.compare (553 ms)

How did you observe that mkldnn hangs on logging? Did you attach debugger and observe where execution is looping over?

All the logs like as follows. Since there is no error log, I guess mkldnn hangs. But I didn't attach debugger and observe where execution is looping over.

[22:59:02][Step 1/1] I1223 22:59:01.768762 59580 analysis_predictor.cc:345] == optimize end ==
[22:59:02][Step 1/1] I1223 22:59:01.771080 59580 tester_helper.h:181] Warm up run...
[22:59:02][Step 1/1] W1223 22:59:01.771121 59580 naive_executor.cc:43] The NaiveExecutor can not work properly if the cmake flag ON_INFER is not set.
[22:59:02][Step 1/1] W1223 22:59:01.771126 59580 naive_executor.cc:45] Unlike the training phase, all the scopes and variables will be reused to save the allocation overhead.
[22:59:02][Step 1/1] W1223 22:59:01.771132 59580 naive_executor.cc:48] Please re-compile the inference library by setting the cmake flag ON_INFER=ON if you are running Paddle Inference
[22:59:02][Step 1/1]

[23:54:36][Step 1/1] 	138 - test_analyzer_dam (Timeout)

Since some hangs in our business job due to openmp library (you can ask @jianhang-liu for more details), I guess that maybe openmp hangs in our CI as well.

Do you use parallel mode of execution of tests of ctest? What is ctest commandline to run tests?

Yes, we use parallel mode of execution of tests of ctest. We use

Paddle/paddle/scripts/paddle_build.sh

Lines 415 to 418 in bf518ec

    
           if [ ${TESTING_DEBUG_MODE:-OFF} == "ON" ] ; then 
        
               ctest -V 
        
           else 
        
               ctest --output-on-failure

The parallel level is

luotao1 · 2019-01-04T02:51:08Z

@jczaja http://ci.paddlepaddle.org/viewLog.html?tab=buildLog&buildTypeId=Paddle_PrCiNight&buildId=45100&_focus=23067#_state=23743 test_analyzer_small_dam crashs again. Why it crashs more frequently than test_analyzer_mobilenet_depthwise_conv? And test_analyzer_resnet50 also has compare_mkldnn, but it works normally.

jczaja · 2019-01-04T15:03:20Z

@luotao1

5117 seems to be virtual machine of 16 sockets and 1 thread per socket (1 core in 1 socket). So 16 logical threads. Could you please tell me how many physical cores/sockets where dedicated to that virtual machine? For example if 5117 is virtual machine with 16 logical threads, which underneath is using 4 physical threads then we would have threads oversubscription when using MKL-DNN four tasks at once. PaddlePaddle is having timeout set to 600s , when there is oversubscription of threads, it may happen that tasks running in parallel will have much longer execution time(as they are fighting for resources) Please check settings of this virtual machine and let me know how many physical threads is in use for 5117.
What is target num threads value that is to be used for each test? How to you set omp num threads
so it take into consideration number of threads in 5117 divided by number of ctest parallel tasks running?
Could you please run on this machine same build but with ctest executing test in sequential order (no parallel mode execution) ?
If execution of previous points gives us no information then, could you please disable ctest timeout and run tests so we see if this is just very log execution or hang

jianhang-liu · 2019-01-06T13:10:53Z

@luotao1 From the log you indicated, I found below:
19:19:29] : [Step 1/1] -- Do not have AVX2 intrinsics and disabled MKL-DNN
Is this a correct build option?

jianhang-liu · 2019-01-06T13:47:46Z

@luotao1 Please ignore my above comment. Just found there are two builds in this log. The second one should be valid one with all features including MKL turn on.

jianhang-liu · 2019-01-06T14:02:46Z

@luotao1 @jczaja One interesting point is that "dam" and "small" dam share same test app so they has exactly same test cases, i.e. "dam" also has compare_mkldnn test case. But according to log, no issue to run all test case of "dam" (including "compare_mkldnn" of course).
Line 22514: [19:50:36] : [Step 1/1] 154/514 Test #138: test_analyzer_dam ............................... Passed 71.89 sec

jianhang-liu · 2019-01-06T14:26:02Z

Another hint is that "small dam" is tested immediately after "dam". This means those two models will be tested concurrently due to parallel mode of ctest. This may make over-subscription of OMP core even worse.
[19:49:24] : [Step 1/1] Start 138: test_analyzer_dam
...
[19:49:26] : [Step 1/1] Start 139: test_analyzer_small_dam

luotao1 · 2019-01-07T05:39:27Z

But according to log, no issue to run all test case of "dam" (including "compare_mkldnn" of course).

@jianhang-liu test_analyzer_dam also have compare_mkldnn, see http://ci.paddlepaddle.org/viewLog.html?buildId=44600&tab=buildLog&buildTypeId=Paddle_PrCiNight&logTab=tree&filter=all&_focus=22935

If test_analyzer_dam runs successfully, there is no detailed log (containing each part of UT) in the CI log.

luotao1 · 2019-01-07T06:27:54Z

@jczaja

5117 is a virtual machine of 16 sockets and 1 thread per socket (1 core in 1 socket). The result of cat /proc/cpuinfo is :

Is the number of physical threads 16?

The default openmp num threads is 1.

Paddle/paddle/fluid/inference/api/paddle_api.h

Lines 199 to 201 in 5ee596c

    
           // number of cpu math library (such as MKL, OpenBlas) threads for each 
        
           // instance. 
        
           int cpu_math_library_num_threads_{1};

I could try run test_analyzer_small_dam, test_analyzer_dam, test_analyzer_resnet50 and test_analyzer_mobilenet_depthwise_conv in sequential order. Does change to sequential order will solve the issue?

jianhang-liu · 2019-01-07T06:56:21Z

@luotao1 "I could try run test_analyzer_small_dam, test_analyzer_dam, test_analyzer_resnet50 and test_analyzer_mobilenet_depthwise_conv in sequential order". It somehow prove our suspect: the hang in "small dam" is possibly due to "oversubscription" of OMP cores since it's running concurrently with "dam". I think change the order of those tests could be a acceptable workaround for now. @jczaja What's your idea?

jczaja · 2019-01-07T10:26:42Z

@luotao1 Yes, if problem is with threads over subscription then sequential execution of ctest should help.
I experimented with MKL-DNN unit tests running with ctest -j4 (parallel mode execution) and there is threads oversubscription and total time of execution of all unit tests is 3 times longer and specific times of tests execution are much longer. So this situation may be similar to what we observe at 5117.
So there are two experiments that I'm interested to check:

Make ctest running in sequential mode
Remove timeout(temporary) for parallel execution of ctest . just to confirm that is due to slow execution and not due to actual hang.

Why I wrote that sequential execution should help? Because I'm suspecting that it could be that virtual machine that of 5117 is causing threads over-subscription . I'm not expert on VM , but
as we can see from logs (and @luotao1 confirmed) there is 16 threads on VM. The thing is that
when setting up Virtual machine it is should be possible to dedicate some resources (cores, sockets and memory) for VM. So it is posible for example to dedicate 4 HW (physical) threads to execute VM with 16 threads. By such a setting we would have threads oversubscription. It is unlikely , as overall performance of sucha VM would be very poor, but possible. Hence:

Please tell us how do you set up VM (what kind of VM software is used?) and check if possible how many physicall resources where dedicated to create VM of 16 sockets of 1 CPU with 1 thread .
How many VMs are running on 5117 . The other reasons of threads over-subscription could be that
more than one VMs are sharing 5117 , and there is not enough threads to be shared among them?

luotao1 · 2019-01-07T12:04:36Z

@jczaja Thanks very much for so detailed explanation!

Make ctest running in sequential mode

#15196 make related ctest running in sequential mode.

Remove timeout(temporary) for parallel execution of ctest . just to confirm that is due to slow execution and not due to actual hang.

It's difficult to do this experiment since nightly stress testing runs on the develop branch.

Please tell us how do you set up VM (what kind of VM software is used?)

I'm not expert on VM as well. We will observe nightly stress testing after #15196 merged several days at first.

luotao1 · 2019-01-11T03:49:36Z

@jczaja @jianhang-liu @yihuaxu
test_analyzer_small_dam timeouts at compare_mkldnn again. Does the mkldnn kernel of conv3d have problem?
http://ci.paddlepaddle.org/viewLog.html?buildId=48242&tab=buildLog&buildTypeId=Paddle_PrCiNight&logTab=tree&filter=all&_focus=23274

luotao1 · 2019-01-11T04:37:50Z

@jczaja The detail machine configuration: http://ci.paddlepaddle.org/viewLog.html?buildId=48242&tab=buildLog&buildTypeId=Paddle_PrCiNight&logTab=tree&filter=all&state=65&_focus=75#_state=65,34

[22:56:29][Step 1/1] 架构：              x86_64
[22:56:29][Step 1/1] CPU op-mode(s):        32-bit, 64-bit
[22:56:29][Step 1/1] 字节序：           Little Endian
[22:56:29][Step 1/1] CPU(s):                16
[22:56:29][Step 1/1] On-line CPU(s) list:   0-15
[22:56:29][Step 1/1] Thread(s) per core:    1
[22:56:29][Step 1/1] Core(s) per socket:    1
[22:56:29][Step 1/1] 座：                 16
[22:56:29][Step 1/1] NUMA 节点：         1
[22:56:29][Step 1/1] 厂商 ID：           GenuineIntel
[22:56:29][Step 1/1] CPU 系列：          6
[22:56:29][Step 1/1] 型号：              85
[22:56:29][Step 1/1] 型号名称：        Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz
[22:56:29][Step 1/1] 步进：              4
[22:56:29][Step 1/1] CPU MHz：             2000.031
[22:56:29][Step 1/1] BogoMIPS：            4016.87
[22:56:29][Step 1/1] L1d 缓存：          32K
[22:56:29][Step 1/1] L1i 缓存：          32K
[22:56:29][Step 1/1] L2 缓存：           4096K
[22:56:29][Step 1/1] NUMA 节点0 CPU：    0-15

jczaja · 2019-01-11T16:36:33Z

@luotao1 When We look at specification of Xeon Gold 5117:
https://ark.intel.com/products/122460/Intel-Xeon-Gold-5117-Processor-19-25M-Cache-2-00-GHz-

There is 14 cores within single socket. And specification you sent/from logs is 16 sockets with 1 core in each socket. So Please tell me:

Why those specifications are different?
How virtualization was done? (what virtualization software used)
How many virtual machines are running on 5117 ?
how many docker containers in running simultaneously on 5117 ?

luotao1 · 2019-01-15T02:55:57Z

@jczaja @jianhang-liu
Discussed with @tianshuo78520a

How virtualization was done? (what virtualization software used)

We use OpenStack.

How many virtual machines are running on 5117 ?

One or Two virtual machines run on 5117.

how many docker containers in running simultaneously on 5117 ?

Only One docker container runs at the same time.

jczaja · 2019-01-15T11:24:28Z

@luotao1
5117 is providing AVX-512. Some other platforms that I have seen only provided AVX2. My question is do you have in CI other platform that has AVX-512 ? I understand that problem(timeout) is manifesting itself only on 5117 , is there any other platform in CI that CI works fine and this platform is having AVX-512 ?

luotao1 · 2019-01-15T12:50:51Z

@jczaja #15335 disable conv3d mkldnn in dam to check whether conv3d mkldnn is the real reason.

luotao1 · 2019-01-15T13:45:48Z

I understand that problem(timeout) is manifesting itself only on 5117 , is there any other platform in CI that CI works fine and this platform is having AVX-512

I check that all the fail log is on 5117 only. We have Intel Xeon Processor (Skylake) with AVX-512. It works OK on this issue.
http://ci.paddlepaddle.org/viewLog.html?tab=buildLog&buildTypeId=Paddle_PrCiNight&buildId=37730&_focus=75#_state=65

[21:01:39][Step 1/1] Architecture:          x86_64
[21:01:39][Step 1/1] CPU op-mode(s):        32-bit, 64-bit
[21:01:39][Step 1/1] Byte Order:            Little Endian
[21:01:39][Step 1/1] CPU(s):                26
[21:01:39][Step 1/1] On-line CPU(s) list:   0-25
[21:01:39][Step 1/1] Thread(s) per core:    1
[21:01:39][Step 1/1] Core(s) per socket:    1
[21:01:39][Step 1/1] Socket(s):             26
[21:01:39][Step 1/1] NUMA node(s):          1
[21:01:39][Step 1/1] Vendor ID:             GenuineIntel
[21:01:39][Step 1/1] CPU family:            6
[21:01:39][Step 1/1] Model:                 85
[21:01:39][Step 1/1] Model name:            Intel Xeon Processor (Skylake)
[21:01:39][Step 1/1] Stepping:              4
[21:01:39][Step 1/1] CPU MHz:               1999.993
[21:01:39][Step 1/1] BogoMIPS:              3999.98
[21:01:39][Step 1/1] Hypervisor vendor:     KVM
[21:01:39][Step 1/1] Virtualization type:   full
[21:01:39][Step 1/1] L1d cache:             32K
[21:01:39][Step 1/1] L1i cache:             32K
[21:01:39][Step 1/1] L2 cache:              4096K
[21:01:39][Step 1/1] L3 cache:              16384K
[21:01:39][Step 1/1] NUMA node0 CPU(s):     0-25

jczaja · 2019-01-15T13:57:06Z

@luotao1 The log you mentioned shows that on other SKX also test_analyzer_small_dam fails it just it is not timeout , but computational problem (diff in results). Perhaps it just different outcome of the same problem

luotao1 · 2019-01-15T14:00:57Z

Yes, it is the MKL diff problem, same with #15116 (comment)

jczaja · 2019-01-18T12:44:01Z

@luotao1 Since couple of people are looking at that issue, I'm sharing my current status. Hopefully it will be helpful.

I got reproduction of problem When running in a loop(upto hundred times) test_analyzer_dam_small .We can get either crash(segfault) or hang(timeout when running under ctest). CI on 5117 is building Paddle WITHOUT ON_INFER=ON and at that situation there can be hang of test_analyzer_small_dam test. If ON_INFER=ON is specified then randomly test_analyzer_small_dam will result in Segmentation Fault.

Hang:
Execution hangs on mutex when Var is to be get from here:

Paddle/paddle/fluid/framework/operator.cc

Line 1064 in 7e651a3

auto* trans_var = new_scope->Var(var_name);

Crash:
Access to new_scope::vars_ cause crash as vars_ contains invalid data.

Workaround:
When we disable Caching of scopes: TryAndCreateTransferScope, No hang or crash is observed
eg.

Paddle/paddle/fluid/framework/operator.cc

Line 1057 in 7e651a3

new_scope = TryCreateTransferScope(kernel_type_for_var,

So currently We can see that there is a problem with TransferScope, that randomly manifest as hang, crash depending on ON_INFER being set or not.

jianhang-liu · 2019-01-18T12:55:22Z

@jczaja Great! Actually this afternoon we just found build withOUT ON_INFER=ON is the key to reproduce. We can easily reproduce two types of CI error now (Timeout, compare fail due to diff) in our local 6151 server. For timeout (hang) issue, we positioned the error almost same as you (but not as detail as you). We wonder whether it's caused by SCOPE_XXX_LOCK definition.

jianhang-liu · 2019-01-21T08:42:50Z

@luotao1 @jczaja We confirmed the same root cause as Jacek, i.e. current code in TryCreateTransferScope (use thread_local as cache to avoid creating transfer scope) may have random failure which cause this CI failure (hangs as timeout, segfault as crash). By simply comment out it (i.e. don't use cache; always create "new_scope"), we won't run into any error anymore. We trial with several times to enable/disable this cache and proved no issue occur when cache is disabled.
@luotao1 Could you please check in framework side for a more clean fix? Thanks.

luotao1 · 2019-01-21T09:04:35Z

By simply comment out it (i.e. don't use cache; always create "new_scope")

@jianhang-liu Could you create a simple PR to show which lines are comment out?

@Superjomn Could you help see the cache in TryCreateTransferScope?

luotao1 · 2019-01-21T09:11:50Z

Paddle/paddle/fluid/framework/transfer_scope_cache.cc

Lines 30 to 46 in 6597ccb

    
           Scope* TryCreateTransferScope(OpKernelType type0, OpKernelType type1, 
        
                                         const Scope* scope) { 
        
             Scope* new_scope{nullptr}; 
        
             size_t infer_cache_key = 
        
                 CombineHash(OpKernelType::Hash()(type0), OpKernelType::Hash()(type1)); 
        
             infer_cache_key = 
        
                 CombineHash(infer_cache_key, std::hash<const Scope*>()(scope)); 
        
             auto it = global_transfer_data_cache().find(infer_cache_key); 
        
             if (it != global_transfer_data_cache().end()) { 
        
               new_scope = global_transfer_data_cache()[infer_cache_key]; 
        
             } else { 
        
               new_scope = &scope->NewScope(); 
        
               global_transfer_data_cache()[infer_cache_key] = new_scope; 
        
             } 
        
             global_transfer_scope_cache().insert(new_scope); 
        
             return new_scope;

@Superjomn Is line45 unused?

luotao1 · 2019-01-21T10:54:30Z

#15032 (comment) is hot-fixed in #15450

luotao1 · 2019-07-10T02:23:00Z

http://ci.paddlepaddle.org/viewLog.html?buildId=126022&buildTypeId=Paddle_PrCiNight&tab=buildLog&_focus=20439
http://ci.paddlepaddle.org/viewLog.html?buildId=126027&buildTypeId=Paddle_PrCiNight&tab=buildLog&_focus=20449
http://ci.paddlepaddle.org/viewLog.html?buildId=125989&buildTypeId=Paddle_PrCiNight&tab=buildLog&_focus=20433

[21:16:22]	[ RUN      ] Analyzer_MM_DNN.mkldnn_cache_clear

hang on Analyzer_MM_DNN.mkldnn_cache_clear currently. (three times on Nightly CI)
@jianhang-liu @LeoZhao-Intel

LeoZhao-Habana · 2019-07-10T02:30:10Z

do we have any conclusion why it hangs with TryCreateTransferScope()? are there any race conditions or multi-instances case?

luotao1 · 2019-07-10T08:13:47Z

English Description

Discussed with @LeoZhao-Intel and @jianhang-liu, we have some common views:

TryCreateTransferScope() is added for fixing GPU inference memory leak problem, which is not required for CPU inference. See the notes by @Superjomn

Paddle/paddle/fluid/framework/operator.cc

Lines 1106 to 1107 in e9c7e21

// batches, so the `new_scope` here will result in GPU memroy explosion

// over the running of operators.
remove TryCreateTransferScope() will fix the hang problem, see Disable cache for scope creation to workaround CI failure #15450
However, remove TryCreateTransferScope() will cause CPU inference memory leak problem, the reason is that we don't release the transfer_scope in OperatorWithKernel::RunImpl

Paddle/paddle/fluid/framework/operator.cc

Lines 923 to 924 in e9c7e21

auto* transfer_scope =

PrepareData(scope, *kernel_type_, &transfered_inplace_vars, runtime_ctx);

TODO：

remove TryCreateTransferScope() and release the transfer_scope in OperatorWithKernel::RunImpl when using NaiveExecutor in CPU inference. @LeoZhao-Intel
test the memory leak problem in detect model [DO NOT MERGE] detect model test2 for dynamic shape #18372 @LeoZhao-Intel @luotao1

Chinese Description

简单说下原因：

为了解决GPU预测内存泄漏，@春伟加了TryCreateTransferScope函数，同时不释放transfer_scope。
TryCreateTransferScope函数对CPU预测是不需要的，去掉该函数后，就不会有随机timeout的问题（已验证）。
目前去掉TryCreateTransferScope函数后，CPU预测会出现内存涨的问题。原因是没有释放transfer_scope。

TODO （@intel @luotao）：

在CPU预测的时候，不使用TryCreateTransferScope函数，同时释放transfer_scope。
验证CPU预测时，内存泄漏问题、随机timeout问题和预测性能问题。

luotao1 added the Intel label Dec 25, 2018

luotao1 changed the title ~~test_analyzer_small_dam random crashs at compare_mkldnn~~ test_analyzer_* random crashs at compare_mkldnn Jan 3, 2019

luotao1 assigned jczaja Jan 3, 2019

luotao1 mentioned this issue Jan 7, 2019

run analyzer_tester serial in multi-thread #15196

Merged

luotao1 mentioned this issue Jan 9, 2019

reduce threads number to avoid analyzer_rnn1_tester hang in CI #15245

Merged

bingyanghuang mentioned this issue Jan 15, 2019

Disable conv3d mkldnn in dam #15335

Merged

jczaja mentioned this issue Jan 15, 2019

minor memory leak in op creation #15339

Closed

luotao1 mentioned this issue Jan 21, 2019

gemm acc issue on AVX512 and AVX #15447

Closed

luotao1 mentioned this issue Feb 21, 2019

disable dam temporarily #15860

Merged

chengduoZH closed this as completed in #15860 Feb 21, 2019

luotao1 reopened this Feb 21, 2019

luotao1 mentioned this issue Mar 13, 2019

[MKL-DNN] Fully Connected #15226

Merged

luotao1 mentioned this issue Apr 4, 2019

test_analyzer_small_dam random fails on accuracy #16473

Closed

luotao1 mentioned this issue Apr 17, 2019

disable test_elementwise_mul_mkldnn_op case #16824

Merged

luotao1 mentioned this issue Jun 29, 2019

[DO NOT MERGE] detect model test2 for dynamic shape #18372

Closed

LeoZhao-Habana mentioned this issue Jul 10, 2019

not use transferscope cache in cpu case #18578

Merged

luotao1 closed this as completed in #18578 Jul 12, 2019

test_analyzer_* random crashs at compare_mkldnn #15032

test_analyzer_* random crashs at compare_mkldnn #15032

Comments

luotao1 commented Dec 25, 2018 • edited

luotao1 commented Dec 25, 2018

jianhang-liu commented Jan 3, 2019

luotao1 commented Jan 3, 2019

jczaja commented Jan 3, 2019

jczaja commented Jan 3, 2019

luotao1 commented Jan 4, 2019

luotao1 commented Jan 4, 2019

jczaja commented Jan 4, 2019

jianhang-liu commented Jan 6, 2019

jianhang-liu commented Jan 6, 2019

jianhang-liu commented Jan 6, 2019

jianhang-liu commented Jan 6, 2019

luotao1 commented Jan 7, 2019

luotao1 commented Jan 7, 2019

jianhang-liu commented Jan 7, 2019 • edited

jczaja commented Jan 7, 2019 • edited by luotao1

luotao1 commented Jan 7, 2019

luotao1 commented Jan 11, 2019

luotao1 commented Jan 11, 2019

jczaja commented Jan 11, 2019 • edited

luotao1 commented Jan 15, 2019

jczaja commented Jan 15, 2019 • edited

luotao1 commented Jan 15, 2019

luotao1 commented Jan 15, 2019

jczaja commented Jan 15, 2019 • edited

luotao1 commented Jan 15, 2019

jczaja commented Jan 18, 2019 • edited

jianhang-liu commented Jan 18, 2019

jianhang-liu commented Jan 21, 2019

luotao1 commented Jan 21, 2019

luotao1 commented Jan 21, 2019

luotao1 commented Jan 21, 2019

luotao1 commented Jul 10, 2019

LeoZhao-Habana commented Jul 10, 2019

luotao1 commented Jul 10, 2019 • edited

English Description

Chinese Description

test_analyzer_* random crashs at `compare_mkldnn` #15032

test_analyzer_* random crashs at `compare_mkldnn` #15032

luotao1 commented Dec 25, 2018 •

edited

jianhang-liu commented Jan 7, 2019 •

edited

jczaja commented Jan 7, 2019 •

edited by luotao1

jczaja commented Jan 11, 2019 •

edited

jczaja commented Jan 15, 2019 •

edited

jczaja commented Jan 15, 2019 •

edited

jczaja commented Jan 18, 2019 •

edited

luotao1 commented Jul 10, 2019 •

edited