using C++ interface on CPU to extract feature vector, delete NDArray::WaitAll() generate incorrect result

[WANG-MengJiao]

I am using a c++ api on CPU to extract a 512-dim feature vector from images. The feature vector will be printed out.
I encountered with the problem that with/without NDArray::WaitAll() generate totally different outputs. Among the results, with NDArray::WaitAll() can get correct vector.
However, the NDArray::WaitAll() is very time consuming. According to the comment in feature_extract.cpp, the NDArray::WaitAll() is needed only when copy between GPU and main memory. So I am wondering why the removal of NDArray::WaitAll() will cause incorrect output when I run the program on CPU. Is there any other tricks I should do?
The code I used is as follows:

using namespace std;
using namespace mxnet::cpp;

Context global_ctx(kCPU,0);

class FeatureExtractor {
 private:
  /*the mean image, get from the pretrained model*/
  NDArray mean_img;
  /*the following two maps store all the paramters need by the model*/
  map<string, NDArray> args_map;
  map<string, NDArray> aux_map;
  Symbol net;
  Executor *executor;
  /*Get the feature layer we want to extract*/
  void GetFeatureSymbol() {
    net = Symbol::Load("/mymodelpath/model-r50-symbol.json")
              .GetInternals()["fc1_output"];
  }
  /*Fill the trained paramters into the model, a.k.a. net, executor*/
  void LoadParameters() {
    map<string, NDArray> paramters;
    NDArray::Load("/mymodelpath/model-r50-0104.params", 0, &paramters);
    for (const auto &k : paramters) {
      if (k.first.substr(0, 4) == "aux:") {
        auto name = k.first.substr(4, k.first.size() - 4);
        aux_map[name] = k.second;
      }
      if (k.first.substr(0, 4) == "arg:") {
        auto name = k.first.substr(4, k.first.size() - 4);
	    args_map[name] = k.second;
      }
    }
  }

 public:
  FeatureExtractor() {
    GetFeatureSymbol();
    LoadParameters();
  }

  void Extract(NDArray data) {
    data -= 127.5;
    data = data * 1.0 / 127.5;
    args_map["data"] = data;
    /*bind the executor*/
    executor = net.SimpleBind(global_ctx, args_map, map<string, NDArray>(), map<string, OpReqType>(), aux_map); 
    executor->Forward(false);    
    NDArray array = executor->outputs[0];
    NDArray::WaitAll();
    /*print out the features*/
    for (int i = 0; i < 512; ++i) {
      cout << array.At(0, i) << ",";
    }
    cout << endl;
  }
};

[leleamol]

Hi @WANG-MengJiao Thanks for submitting the issue. I have submitted a PR #12930 that demonstrates the usage of C++ API in inference workflows. This might provide some help.
I will add the labels so that MXNet community members can provide help

[leleamol]

@mxnet-label-bot add [C++, Question]

[ThomasDelteil]

@WANG-MengJiao NDArray::WaitAll() makes you wait for the current operations to complete in the engine. If you don't use that, you would be reading the outputs buffer out of sync with what you expect.

[WANG-MengJiao]

@ThomasDelteil Thank you for your response. Let me to get this straight. Do you mean without NDArray::WaitAll(), I should read the outputs from sync instead of using NDArray array = executor->outputs[0]? Since I am a newbie to MxNet, could you give me some example about how to do that?

Also, from the comments in feature_extract.cpp, NDArray::WaitAll() is only needed when copying between GPU and main memory. However, I did all the work on CPU. To my best understanding, there would be no copying between memories. Why the NDArray::WaitAll() is still required in my case?

[leleamol]

@ThomasDelteil can you please respond to @WANG-MengJiao

[konioyxgq]

@WANG-MengJiao @leleamol I have the same problem, have you solved it? Is there any trick I do?
Thank you so much

[konioyxgq]

@WANG-MengJiao Please！！！！