Maximize Mask-RCNN (TF, Keras) inference time performance on AKS with CPU

[Larleyt]

Hello!
The aim: run inference (no training needed) of a custom Mask-RCNN at CPU VM as an AksWebservice as fast as possible. (CPU is chosen mainly because it's cheaper.)

Default TF build from pip on inference sent warnings like:
Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA

So I tried to install provided CPU-optimized build using:
CondaDependencies.add_tensorflow_conda_package(core_type='cpu', version="1.15")

It successfully installs TF 1.15 with all the needed instructions but only for MKL-DNN operations:

This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags. 

And it slowed down the inference by x2. Why could that be? I've seen quite a lot of similar issues ([1], [2], [3] about performance degradation when using MKL. Maybe, it is somehow threads related.
I also tried pip intel-tensorflow==1.15.2. Same performance.
Installing CondaDependencies.add_tensorflow_pip_package(core_type='cpu', version="1.15")
leads to no optimizations at all (pip installs usual TF binary).

So I decided to build my own TF1.15.3 with AVX2 AVX512F FMA but without MKL (correct me if I'm wrong and that won't change the performance):

$ bazel build -c opt --copt=-march=native --copt=-mfpmath=both //tensorflow/tools/pip_package:build_pip_package

It's been successfully compiled and installed without errors. No warnings about unsupported instructions or MKL-DNN-only operations occurred. But no performance boost has been noticed.

So, why optimized TF builds works the same (if not worse for MKL-DNN part)? Am I using wrong type of VM for this type of task? (Right now I'm using Fs-v2 series.)

And also few side questions if you please:

1. How to delete an Environment from a Workspace? So it wouldn't appear on Environment.list(ws) anymore?
2. Why could it be that the AksWebservice doesn't create all the replicas but only few of them? Say, 2/10. Others are "unavailable" on a Kubernetes Services page on portal.azure.com. Autoscaling is set to False so this should not be the reason.

[SnehaAgrawal-MSFT]

@Larleyt Could you Please share the document link you are referring to help you better on this.

[Larleyt]

Could you Please share the document link you are referring to help you better on this.

I'm not sure, what document are you talking about? If you mean Mask-RCNN itself here it is, but we've changed its last layers and some configs to detect necessary objects.

Also, forgot to mention: right now I'm using Fs-v2 tier cluster.

[Larleyt]

Is it okay that I'm getting drastically different inference times at F32s-v2, F16s-v2, F8s-v2, F4s-v2 , although, setting explicitly the same number of cores and RAM per replica of AksWebservice? Say, I'm setting 1 replica, 1 core, 2 gb RAM per replica, and the results are:

**Mean / min / max** (inferencing 30 images for 3 times) 

F32s-v2   0.73 / 0.69 / 0.8
F16s-v2   1.04 / 0.98 / 1.08
F8s-v2    1.67 / 1.63 / 1.75
F4s-v2    2.76 / 2.67 / 2.93

It seems like a bug to me, cause all of them have the same Intel Xeon Platinum 8168 (Skylake) and should be the same when have the same amount of resources.