runtime_context_cache_pass breaks accuracy of FP32 C-API inference of ResNet50 and MobileNet-v1

[wojtuss]

The runtime_context_cache_pass breaks accuracy of inference of (at least) ResNet50 and MobileNet-v1 if the inference requires more than 1 iteration.

INT8 inference works fine.

Can be verified using the test in PR #16559.
With a single iteration, the test passes. With multiple iterations, the test fails.

[luotao1]

With multiple iterations, the test fails.

Could you tell me how to reproduce it? Could you paste the command to me directly?

[wojtuss]

It is already reproduced on CI (http://ci.paddlepaddle.org/viewLog.html?buildId=79052&buildTypeId=Paddle_PrCi) for build PR_CI (Paddle) in #16559.

Otherwise build #16559 and run the test

ctest -R test_analyzer_int8_resnet50 -V

or

ctest -R test_analyzer_int8_mobilenet -V

I have modified the test so it runs 2 iterations. With the runtime_context_cache_pass it fails, without the pass it passes.
Internally, I can also reproduce this bug using our standalone C-API application.

[luotao1]

I reproduce it, and I am debugging now.

[luotao1]

Do you have any observation why with runtime_context_cache_pass it fails?
I try two ways, but it doesn't help me:

• cfg->SwitchSpecifyInputNames();, set the PaddleTensor' name correct
• move runtime_context_cache_pass to the end of passes

https://github.com/PaddlePaddle/Paddle/compare/develop...luotao1:int8_cache?expand=1

128: I0403 08:05:28.432641 18927 tester_helper.h:508] --- comparing outputs ---
128: I0403 08:05:28.432653 18927 tester_helper.h:449] Avg top1 INT8 accuracy: 0.7100
128: I0403 08:05:28.432658 18927 tester_helper.h:451] Avg top1 FP32 accuracy: 0.3700
128: I0403 08:05:28.432662 18927 tester_helper.h:453] Accepted accuracy drop threshold: 0.01
128: F0403 08:05:28.432680 18927 tester_helper.h:454] Check failed: std::abs(avg_acc1_quant - avg_acc1_ref) <= FLAGS_quantized_accuracy (0.34 vs. 0.01)
128: *** Check failure stack trace: ***
1/1 Test #128: test_analyzer_int8_mobilenet .....***Exception: Other 58.45 sec

[luotao1]

Some observation:

• When I move runtime_context_cache_pass before fc_fuse_pass: accuracy normal.
• When I move runtime_context_cache_pass before fc_fuse_pass, and then delete fc_fuse_pass: accuracy fails.