Problem about use_mkldnn flag

[luotao1]

This issue is from Intel sides.

The problem we have with use_mkldnn flag is that we need to modify all models to enable newly integrated MKL-DNN ops. It would be great, if we can only add an integration and in case when Fluid is run with use mkldnn parameter, all available Ops are used without modification of model scripts, to pass the option.

[Superjomn]

I think that makes sense.

[luotao1]

How about using MKLDNNPlace() to call all available OPs?

place=fluid.MKLDNNPlace()

[kbinias]

It would be great to have global use_mkldnn flag. Now it is very difficult to fallback this flag to some ops, e.g.: elementwise_add, because the python code is auto generate with generate_layer_fn(_OP). Also it would be nice to have another flag not_use_mkldnn on layer level, to switch off MKL-DNN from some layers/ops.

[mozga-intel]

@luotao1 Thank you for this issue.

I will use the sum operator as an example to describe this problem. So, during the implementation of this operator I have encountered a few difficulties. The current version of the sum operator of MKLDNN algorithm is using inside of the other operators. Therefore the flag can not be uses in this way. For instance:
Backward.py

To that end I agree with @kbinias.

[Superjomn]

it would be nice to have another flag not_use_mkldnn on layer level, to switch off MKL-DNN from some layers/ops.

That makes sense, the user can

• globally set a default config, ON or OFF
• set MKLDNN config separately for specific operator

For example, such use case (the name of APIs is not important)

set_default_use_mkldnn(True)

v1 = op0(v0) # use mkldnn by default if supported
# ...
v2 = op1(v1, use_mkldnn=False) # turn off MKLDNN for this operator
# ...

The above function is better.

[luotao1]

We will have a discussion about how to add this global flag function in our framework ASAP.

[mrysztow]

@luotao1 Any ETA for this?

[luotao1]

@mrysztow Discussed with @Superjomn and @tensor-tang, the newest design is:

• use an environment variable export FLAGS_use_mkldnn=True to control it.
• add EnableMKLDNN function (maybe rename it) in paddle/fluid/platform/mkldnn_helper.h
  

  Paddle/paddle/fluid/inference/tests/test_helper.h

  Lines 136 to 146 in 406c1dd

  	void EnableMKLDNN(
  	const std::unique_ptr<paddle::framework::ProgramDesc>& program) {
  	for (size_t bid = 0; bid < program->Size(); ++bid) {
  	auto* block = program->MutableBlock(bid);
  	for (auto* op : block->AllOps()) {
  	if (op->HasAttr("use_mkldnn")) {
  	op->SetAttr("use_mkldnn", true);
  	}
  	}
  	}
  	}

• call EanbleMKLDNN function in Executer:Run (paddle/fluid/framework/executer.cc) to transpiler the program.

We will finish it ASAP before next meeting.

[tpatejko]

@luotao1 What's the reason for using an environment variable? Does it mean that some parameters will be passed as script's arguments and some of them will be passed as environment variables?

[Superjomn]

Does it mean that some parameters will be passed as script's arguments and some of them will be passed as environment variables?

I think an environment variable might be more general for both python scripts/c++ bin.

Or do you have any suggestion?
@tpatejko

[mrysztow]

@Superjomn I think it is a good idea, as now we need to change all model scripts to add this option. But will it be consistent with focing usage of other backends, like gpu?

[Superjomn]

That depends, the MKLDNN might be the first one that has a global switch in runtime.

I suggest we use the environment variable first, with two singleton functions in both python and C++ to wrap it, called static bool DoUseGlobalMKLDNN() and def do_use_global_mkldnn()(name need to futher considered), and use this way first.

If the environment variable's name or this plan is changed, just update those two function is easier and flexible.
@mrysztow

[luotao1]

@mrysztow You can use FLAGS_use_mkldnn=true to control it now, see #11319 for details.

[tpatejko]

@Superjomn My main concern was some sort of consistency and intuition behind passing parameters to the models scripts. I mean, a user will have to set an environment variable if they want to run something on CPU with MKLDNN, which is different from running something on GPU, that requires passing an argument to a model script.

However, it's true what you're saying, this approach is more flexible, as it doesn't require changing scripts with models.

[Superjomn]

Yeah, I understand your concern.

the MKLDNN might be the first one that has a global switch in runtime.

The environment variable is a low-level API and easy to hide in many ways.

We can concentrate on the implementation of MKLDNN functions first, and have another discussion about this API after MKLDNN functions are ready and release to users.