First attempt to use more gpus

[tfarago]

A collaborative PR for making UFO work on more GPUs from concert.

[tfarago]

This is a deep WIP so far...

[tfarago]

@matze so I tried this:

shape = 2048, 4096
image = np.random.normal(1000, 10, size=shape).astype(np.float32)


def generate(num_images=10):
    image = np.random.normal(1000, 10, size=shape).astype(np.float32)

    for i in range(num_images):
        print 'yielding', id(image), i + 1
        yield image
        print 'yielded', id(image), i + 1


@async
def process(backproject, arch, gpu, result, num_images=10):
    try:
        inject(generate(num_images=num_images), backproject(result(), arch=arch, gpu=gpu)) 
    finally:
        backproject.wait()


def run(num_images=10):
    arch = Ufo.ArchGraph()
    gpus = arch.get_gpu_nodes()
    resources = {}
    for i, gpu in enumerate(gpus):
        resources[gpu] = (Backproject(image.shape[1] / 2), Result())

    futures = []
    st = time.time()
    for gpu, (bp, result) in resources.iteritems():
        futures.append(process(bp, arch, gpu, result, num_images=num_images))

    wait(futures)
    print 'duration: {} s'.format(time.time() - st)

    return zip(*resources.values())[1]

But unfortunately I got no speedup plus the memory was not released.

[tfarago]

Even though I call wait the resources are not released. But when I type exit in the session I see the release messages from the framework, so it actually does the job but a little late. Maybe we need to release resources manually with this scheduler?

[tfarago]

And funnily, those yielding and yielded printouts look like the stuff gets parallelized... weird.

[matze]

There is a nasty race condition for me in this example. Waiting for the futures begins before the processes started which results in an AttributeError because there is no Backproject.thread object yet created.

[matze]

Actually I have problems to get this running at all. Do you made any changes to either ufo-core or Concert?

[tfarago]

Did you install this branch when you tried?

[matze]

Yes, of course.

[tfarago]

Can you re-check? I tried with master and got the same error you mentioned above

AttributeError: 'Backproject' object has no attribute 'thread'

[tfarago]

I just logged in, did no changes to ufo-* and it works.

[matze]

As I said, it's a race condition. If it starts executing before waiting on the futures begins it should be fine. But I am not sure how to solve this generally at the moment.

[tfarago]

It is not a race condition because when you wait for the futures (if you mean the ones just before the duration printout) you are waiting for the process function which executes these commands serially (pseudo):

try:
    backproject.start() # inside the __call__, backproject.thread exists from now on
    while True:
        # crunch
finally:
    backproject.wait()

Try this process function instead:

@async
def process(backproject, arch, gpu, result, num_images=10):
    inject(generate(num_images=num_images), backproject(result(), arch=arch, gpu=gpu)) 
    backproject.wait()

Does it by any chance say:

TypeError: __call__() got an unexpected keyword argument 'gpu'

[matze]

Does it by any chance say:

No, it never did so because I told you already that I am using the correct branch. After removing the try/finally blocks it kind of works now.

[tfarago]

Hm, but then I have no idea what the problem is. Especially since for me it works every time. Actually I experienced some random hangings and segfaults but that was very rare and before your final python multithreading fix, now I have no idea if it still happens.

[matze]

I release the GIL in the output task now which let's all schedulers run fully in parallel. I haven't checked the memory consumption yet though.

[tfarago]

Should I try? Do I need to update/do smth. special on the ufo server?

[matze]

Yeah, try what you wanted to try. I see the speed up and I guess you have to update ufo-core if you don't use a local installation.

[tfarago]

After a fresh pull it doesn't build...

EDIT: All OK, make clean rules.

[tfarago]

Nope, it builds on my machine but not on the server...

[matze]

Nope, it builds on my machine but not on the server...

I ran it on the server ...

[tfarago]

In /opt/ufo-core:

[ 47%] Building C object ufo/CMakeFiles/ufo.dir/ufo-output-task.c.o
/opt/ufo-core/ufo/ufo-output-task.c: In function ‘ufo_output_task_get_output_buffer’:
/opt/ufo-core/ufo/ufo-output-task.c:116:5: error: ‘buffer’ may be used uninitialized in this function [-Werror=uninitialized]
cc1: all warnings being treated as errors

make[2]: *** [ufo/CMakeFiles/ufo.dir/ufo-output-task.c.o] Error 1
make[1]: *** [ufo/CMakeFiles/ufo.dir/all] Error 2
make: *** [all] Error 2

How did you manage locally?

[matze]

I have no idea why it didn't complain in my case. Anyway, I pushed a change that removes that warning, compiled and installed it. Have fun.

[tfarago]

Thanks, I will!

[tfarago]

I am trying to make this work with CopyTask but no luck. The code from here crashes with the following awfully:

f = FooProcess()
im = np.empty((512, 512), dtype=np.float32)
result = Result()
inject((im,), f(result()))

Can @matze take a look please?

[tfarago]

This has been around for long enough.