Skip to content
This repository

all processes run on one CPU core #840

Closed
rth opened this Issue October 06, 2011 · 12 comments

4 participants

Roman Yurchak Fernando Perez Min RK Zach Dwiel
Roman Yurchak
rth commented October 06, 2011

When using IPython parallel capabilities on a Ubuntu server, I experience that all engines launched with ipcluster are run in one CPU core, regardless that several are available.

The script I am calling is roughly given bellow:

      $ipcluster start -n 8

      from IPython.parallel import Client
      rc = Client(profile='default')
      dv  = rc[:]
      dv.block = True

      def f(args):
           subprocess.call('a bash program with custom arguments', shell=True)

      dv.map(f, argument_list)

With htop, I can see that each ipengine has done it's subprocess.call as the corresponding process was started. However, for some reason all processes share the same logical core/thread.

I thought that once a process was started, it was up to the linux kernel to do load-balancing between available cpu cores? Is there something in the way the processes were started by IPython that might explain this behaviour?

I am using trunk (0.12.dev) version of IPython, and python 2.6 on Ubuntu 9.10

Min RK
Owner

That's very interesting. I don't know how IPython could exert control over such things.

When you use ipcluster, it is effectively:

subprocess.Popen(['ipcontroller', *args])
for i in range(n):
    subprocess.Popen(['ipengine', *args])

And ipcontroller itself starts 4 subprocesses via multiprocessing.

Is the behavior any different if you start ipengine and ipcontroller separately, rather than using ipcluster?

I'll try to run a similar test on my machines today.

Min RK
Owner

I just ran this test on my Ubuntu machine (11.04):

$>ipcontroller start -n 8

from IPython import parallel
rc = parallel.Client()

def burn(t):
    import time
    from subprocess import Popen, PIPE
    p = Popen(["bash", "-c", "while true; do :; done"], stdout=PIPE, stderr=PIPE)
    time.sleep(t)
    p.terminate()

rc[:].apply_sync(burn, 10)

And it reliably uses as many cores as engines.

Fernando Perez
Owner

There's nothing IPython does to control process cpu affinity, we leave that up to the kernel. There is an API for explicit control of cpu affinity under linux, using the taskset command, but we don't go anywhere near that.

My wild guess is that your system was pretty busy otherwise, and all engines got put on one cpu by the kernel because the linux kernel scheduler decided there were other tasks more deserving of their own cpu. But if your engines start doing truly cpu-intensive things, they should be quickly moved over to their own cpu each.

You can't really make any long-term conclusions about cpu affinity from only seeing the cpu they start at, as at that point they are really not doing much. Now, in your case, if the subprocess your call is executing is itself cpu-intensive, then it's the right thing to shove all the engines into a single cpu, since the load will be consumed by your subprocesses, not by ipython itself. So in that case, the kernel seems to be doing the right thing: moving the engines over to one cpu, and leaving one separate cpu for each of the subprocesses to work as efficiently as possible.

I'm closing this as a bug since it's really not an ipython issue, but we can continue discussing the topic if you have further questions. I'm just trying to keep our bug triage moving.

Fernando Perez fperez closed this October 06, 2011
Fernando Perez fperez reopened this October 07, 2011
Fernando Perez
Owner

I got the following by email, so I'm reopening the issue. For some odd reason we are getting the affinity mask set in our subprocesses...

---------- Forwarded message ----------
From: Roman Yurchak roman.yurchak@crans.org
Date: Fri, Oct 7, 2011 at 10:15
Subject: Re: [ipython] all processes run on one CPU core (#840)
To: MinRK benjaminrk@gmail.com, fernando.perez@berkeley.edu

Hello,

On 06/10/11 22:00, Fernando Perez wrote:

There is an API for explicit control of cpu affinity under linux,
using the taskset command
Thanks to point this out. I checked, and problematic processes had
affinity mask: 1
while all normal processes have
affinity mask: ffff
so the problem is definitely with cpu affinity.

My wild guess is that your system was pretty busy otherwise,
It wasn't the case: there were 7/8 cores free.

Now, in your case, if the subprocess your
call is executing is itself cpu-intensive, then it's the right
thing
to shove all the engines into a single cpu, since the load
will be consumed by your subprocesses, not by ipython itself.
Sure, but the problem was that cpu-intensive processes were permanently
run in one core.

On 06/10/11 20:58, Min RK wrote:

When you use ipcluster, it is effectively:

subprocess.Popen(['ipcontroller', *args])
for i in range(n):
    subprocess.Popen(['ipengine', *args])

And ipcontroller itself starts 4 subprocesses via multiprocessing.

Is the behavior any different if you start ipengine and ipcontroller >
separately, rather than using ipcluster?
Actually, starting ipengines by hand does solve the problem!

It seems that for some reason, ipengines launched with ipcluster had
affinity mask set to 1 which explains cpu affinity of sub-processes
launched within.

Min RK, I have tried your test code (test1.py atteched) on the server
and it fails (cpu affinity = 1) when ipengines were started by
ipcluster and succeeds (cpu affinity = ffff) otherwise.

Just to check that it's not related to process spawning with subprocess,
I have rewritten test1.py with two layers of Popen (see test2.py) and
subsequently without using IPython. This works fine.

While I don't have this problem on my laptop [python 2.7], it is
systematic on every node of the network cluster I have access to, which
has following configurations:

Linux kernel [ 2.6.31-22-server, 2.6.32-33-generic]
Ubuntu [9.10, 10.04.3 LTS ]
Python [2.6.4, 2.6.5]
zeromq-2.1.9
pyzmq-2.1.9
IPython 0.12.dev

Best regards,

Roman Yurchak

Min RK
Owner

Since IPython certainly doesn't touch affinity itself, it seems like there is something peculiar about your systems.

I do not see a change in the affinity under Ubuntu 10.04 or 11.04, using the System Python on each, and htop shows that all CPUs are being used on both.

Have you done any configuration of IPython, or are you running everything default?

Also, your mail to the list didn't actually include the test scripts you mentioned, can you post them to gist/pastebin?

Roman Yurchak
rth commented October 07, 2011

I agree that there is probably some kind of misconfiguration of the systems I use. Anyway it's not a substantial problem, launching ipengines manually still works. It's just weird...

I have tried to remove ~/.ipython to use default config and it's doesn't change the result.

Yes, sorry I forgot to attach files to my previous mail:
http://perso.crans.org/yurchak/i/ipython/test1.py
http://perso.crans.org/yurchak/i/ipython/test2.py

Min RK
Owner

What if you run test2 without shell=True? I don't think we do that when launching engines.

Roman Yurchak
rth commented October 07, 2011

It gives the same result => affinity mask: ffff

Roman Yurchak
rth commented October 07, 2011

After a while, I finally managed to track down the point where ipcontoller gets it's cpu affinity changed :

bin/ipcluster:  from IPython.parallel.apps.baseapp import BaseParallelApplication
IPython/parallel/apps/ipclusterapp.py: from IPython.parallel.apps.baseapp import BaseParallelApplication
IPython/parallel/__init__.py: from .client.client import Client
IPython/parallel/client/client.py: from IPython.parallel import util
IPython/parallel/util.py: from IPython.utils.newserialized import serialize, unserialize
IPython/utils/newserialized.py: try: import numpy

Indeed it seems that on my system import numpy sets the parent process cpu affinity mask to 1. I'll write to numpy mailing-list to ask if it's an expected behaviour...

For the record I am using numpy 1.6.1 with GOTO BLAS implementation and numpy.test('full') returns no errors.

Fernando Perez
Owner

OK, I'm closing it again as indeed it seems it's outside our range to fix. We can always reopen if need be, but I'm trying to be aggressive with triage so we don't drown in open bugs.

Fernando Perez fperez closed this October 07, 2011
Fernando Perez
Owner

Posting @rth's private reply that has useful info here, thanks for following up!

Hello,

I just wanted to let you know the end point of this story, in case
someone else has the same problem using ipcluster.

Actually, the default GotoBLAS2 installation from sources comes with

# If you want to disable CPU/Memory affinity on Linux.
# NO_AFFINITY = 1

or in other words NO_AFFINITY = 0, which sets parent process cpu
affinity mask when either numpy or scipy are imported.

Compiling GotoBlas2 with NO_AFFINITY = 1, solves the problem.

Best regards,

Roman

Zach Dwiel

For more details on the OpenBLAS stuff Roman is talking about, see here: http://stackoverflow.com/a/15641148/2093984.

A quick fix is to run os.system("taskset -p 0xff %d" % os.getpid()) after all of your imports and before you start starting new processes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.