Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to run mxnet(C++) in single-thread mode? #15275

Closed
xianyujie opened this issue Jun 19, 2019 · 13 comments
Closed

How to run mxnet(C++) in single-thread mode? #15275

xianyujie opened this issue Jun 19, 2019 · 13 comments

Comments

@xianyujie
Copy link

@xianyujie xianyujie commented Jun 19, 2019

Hey, I want to run mxnet in single-thread mode, but I find my programs have many threads when using mxnet.

###Using mxnet

ubuntu@ubuntu-MW50-SV0:~$ top -H -b -n1 | grep validate11 | wc -l
33

###Without mxnet

ubuntu@ubuntu-MW50-SV0:~$ top -H -b -n1 | grep validate11 | wc -l
2

So, when I compile mxnet, if there are some options to prevent multi-threads?

@xianyujie

This comment has been minimized.

Copy link
Author

@xianyujie xianyujie commented Jun 19, 2019

@pengzhao-intel do you have some tips? many thanks.

@xianyujie

This comment has been minimized.

Copy link
Author

@xianyujie xianyujie commented Jun 19, 2019

I want to take part in the FRVT competition. Here are some requirements:

Implementations must run in single-threaded mode, because NIST will parallelize the test 
by dividing the workload across many cores and many machines. Implementations must ensure 
that there are no issues with their software being parallelized via the fork() function. 
Developers should take caution with checking threading when using third-party frameworks 
(e.g., TensorFlow, MXNet, etc.).
@pengzhao-intel

This comment has been minimized.

Copy link
Contributor

@pengzhao-intel pengzhao-intel commented Jun 19, 2019

Could you try below setting?

export KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0
export OMP_NUM_THREADS=1
numactl --physcpubind=0 --membind=0 python ... 
@xianyujie

This comment has been minimized.

Copy link
Author

@xianyujie xianyujie commented Jun 19, 2019

@pengzhao-intel I tried, and it's still multi-threads.
By recompiling mxnet with OPENMP=0, the number of threads reduced from 33 to 11. It does not completely prohibit multi-threads.

@pengzhao-intel

This comment has been minimized.

Copy link
Contributor

@pengzhao-intel pengzhao-intel commented Jun 19, 2019

Could you help provide the reproducible case?

BTW, please try naive engine :)
export MXNET_ENGINE_TYPE=NaiveEngine

@leleamol

This comment has been minimized.

Copy link
Contributor

@leleamol leleamol commented Jun 19, 2019

@mxnet-label-bot add [Question, Pending Requester Info]

@ElaineBao

This comment has been minimized.

Copy link
Contributor

@ElaineBao ElaineBao commented Jun 20, 2019

Hi @xianyujie, I tried the solution mentioned in your email, and confirm that with:

compile mxnet with OPENMP=0
export MXNET_ENGINE_TYPE=NaiveEngine

mxnet is running with minimum number of threads.

@xianyujie

This comment has been minimized.

Copy link
Author

@xianyujie xianyujie commented Jun 20, 2019

@ElaineBao yes, it works, but sometimes the number of threads increased, I'm retesting if it was an error caused by my test.

@xianyujie

This comment has been minimized.

Copy link
Author

@xianyujie xianyujie commented Jun 20, 2019

compile mxnet with OPENMP=0 
export OMP_NUM_THREADS=1
export MXNET_ENGINE_TYPE=NaiveEngine

solve it! Other environment variables are as follows:
http://mxnet.incubator.apache.org/versions/master/faq/env_var.html?highlight=naive
thanks @pengzhao-intel

@pengzhao-intel

This comment has been minimized.

Copy link
Contributor

@pengzhao-intel pengzhao-intel commented Jun 20, 2019

compile mxnet with OPENMP=0 
export OMP_NUM_THREADS=1
export MXNET_ENGINE_TYPE=NaiveEngine

solve it! Other environment variables are as follows:
http://mxnet.incubator.apache.org/versions/master/faq/env_var.html?highlight=naive
thanks @pengzhao-intel

Cool!

Thanks, @ElaineBao as well, and I am going to close this issue :)
Feel free to let us know if you encounter any issue in the future.

@pengzhao-intel

This comment has been minimized.

Copy link
Contributor

@pengzhao-intel pengzhao-intel commented Jun 20, 2019

Resolved and closing now.

@igor-byel

This comment has been minimized.

Copy link

@igor-byel igor-byel commented Nov 6, 2019

compile mxnet with OPENMP=0 
export OMP_NUM_THREADS=1
export MXNET_ENGINE_TYPE=NaiveEngine

solve it! Other environment variables are as follows:
http://mxnet.incubator.apache.org/versions/master/faq/env_var.html?highlight=naive
thanks @pengzhao-intel

we used such solution also for frvt
setting those variable in code

std::shared_ptr<Interface>
Interface::getImplementation() {
//set environemnt variable for mxnet to run single thread
    try {
        char mxnetConfig1[] = "MXNET_ENGINE_TYPE=NaiveEngine";
        putenv(mxnetConfig1);

        char mxnetConfig2[] = "OMP_NUM_THREADS=1";
        putenv(mxnetConfig2);
    }
    catch (const std::exception& e)
    {

    }

it worked for frvt test on our local machine
but when we send it to frvt for check they respond that our lib is threading.
Did you succeed with mxnet configuration?
Thank you.

@cyrusbehr

This comment has been minimized.

Copy link

@cyrusbehr cyrusbehr commented Jan 16, 2020

@igor-byel I experienced the same issue you are facing, and here is my fix. The issue is likely due to the way you are using the putenv function. From the linux manual:

The putenv() function adds or changes the value of environment
variables. The argument string is of the form name=value. If name
does not already exist in the environment, then string is added to
the environment. If name does exist, then the value of name in the
environment is changed to value. The string pointed to by string
becomes part of the environment, so altering the string changes the
environment.

In your above implementation, your mxnetConfig1 and mxnetConfig2 variables are no longer defined once they go out of scope (which is at the end of the getImplementation function). You therefore have a dangling pointer in your environment. Make mxnetConfig1 mxnetConfig2 global vars, member vars, or allocate on heap for the changes to persist.

For more info, refer to this stack overflow question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.