Output Buffer Sharing between GPU and CPU

[OliverScherf]

Hello,

I'm working on a program that should use both CPU and GPU. I already opened an Issue where a similar question was answered. #229 The use case changed a bit, so I just open this issue.

I have the following scenario. I want to split up work between the CPU and GPU. They work mostly on the same input data, yet independent of each other. After the computation I want to merge the result of both CPU and GPU. But because CPU and GPU share the same physical memory, I want to avoid any unnecessary copies.
Note that the output values are independent of other output values.

The GPU would write to output[0 .. m], while the CPU would write to output[m .. n]
The result after the computation would be like this:
Output array: [0------GPU output------m---CPU output---n]

Is such a behavior possible? If yes, what do I need to consider while doing it?

Any help is much appreciated!

[alalek]

Regardless of buffer sharing topic, overcomlicated task here is selecting of "m" value which should guarantee optimal load of both CPU/GPU devices.
You may start this research activity with dedicated buffers for each GPU/CPU parts.

[OliverScherf]

Thanks for your reply.

My plan to determine these values is by trying it out on the target machine.

I think it's also important to notice that the "merged" output is needed for the next step as input. The next step should do similar/the same things as the first one.

More precisly: I'm planning to execute a Convolution across the CPU and GPU. A part of the filters should be executed on the GPU, the others on the CPU.

Help is much appreciated. :)

[JacekDanecki]

@OliverScherf Can you start your research with "m" aligned to page size (4096 bytes)?

[MichalMrozek]

I think you can start by creating one big buffer.
Then partition it with page granularity (4096u) between CPU / GPU using clCreateSubBuffer.

Then do the following:

• for GPU part, use GPU sub buffer, call clEnqueueNDRange, return event1.

• for CPU part , call mapBuffer for CPU sub buffer (you may use another queue for that to not interfere with GPU part), this will give you pointer on the CPU, do the CPU part and then call clEnqueueUnmapMemObject, returning event2.

• now if you want to go further on the GPU, synchronize further calls with event 1 + event2

• if you want to go further on the CPU, then call clEnqueueMapBuffer on the main buffer with those events above and blocking flag set to true

[OliverScherf]

@JacekDanecki The problem is, that I can't really set "m" aligned to page size, because the data would most likely overlap the page size boundaries. But yes, researching on how much faster the GPU is, is also mandatory.

@MichalMrozek Thanks for the explanation. I think I understood the procedure. However, I don't think I can't really get around the copying. I can't split up the subresults in 4096u boundaries. I need to merge them later by copying anyway I guess. Or do I miss out something?

I would propose the following procedures to accomplish the task:
For the simplification lets say our output has a size of 1.000.000. The first half (0-499.999) will be computed by the GPU and second half by the CPU (500.000-1.000.000)

MEMORY ALLOCATION:
I would like to allocate the memory for both the input and output buffer on my own. I will pass CL_MEM_USE_HOST_PTR to clCreateBuffer. I also allocate the required memory page aligned.
E.g. with the outputBuffer:

unsigned int  OUTPUT_BUFFER_SIZE = 1000000;
unsigned int  ALIGNMENT = 4096;
float* output_ptr;
posix_memalign((void**) (&output_ptr), ALIGNMENT, OUTPUT_BUFFER_SIZE);
cl::Buffer outputBuffer = cl::Buffer(ocl->context, CL_MEM_READ_WRITE | CL_MEM_USE_HOST_PTR, OUTPUT_BUFFER_SIZE, (void*) output_ptr);

START CPU CODE
Start the CPU Side in a different thread. The CPU Code will allocate different, additional output memory for it's output, so that the GPU outputBuffer is not touched.

thread.create(my_kernel_on_cpu());

START CL KERNEL
Now I would load the kernel and set the arguments:

cl::Kernel kernel = cl::Kernel(program, "my_kernel");
kernel.setArg(0, inputBuffer);
kernel.setArg(1, outputBuffer);

And enqueue the NDRange:

queue.enqueueNDRangeKernel(kernel, 0, cl::NDRange(OUTPUT_BUFFER_SIZE / 2)); //note that I only work half of the work-items with GPU

SYNC AND MERGE
sync OpenCL and CPU Thread and merge outputs

queue.finish();
thread.join();
// at this point I can safely operate on the output_ptr and merge the result
memcpy(output_ptr[OUTPUT_BUFFER_SIZE / 2], cpu_output, OUTPUT_BUFFER_SIZE / 2);

As you can see, I dont use mapBuffer. For my understandig I would need to use mapBuffer only if I would use CL_MEM_ALLOC_HOST_PTR instead of CL_MEM_USE_HOST_PTR right?

Would this procedure work and be efficient?

[MichalMrozek]

Please align your buffer size to page size , otherwise buffer will not share the same physical memory.
You may use Intercept Layer:
https://github.com/intel/opencl-intercept-layer
with
ContextHintLevel to 0xff to show driver diagnostics about when transfers happen and when physical storage is reused.

With your solution you are doing a copy which can actually be avoided.
On Integrated Devices we can re-use the same storage and that will happen if you properly create buffers and sub-buffers. (then map and unmap calls will be doing nothing, driver diagnostics will tell you that)

Map and UnMaps are still needed if you want CPU access to the buffer, do not skip those.

[OliverScherf]

OK thanks for the hints.
There is one thing I don't get with the subbuffers:
If I partition two subbuffers with page granularity (4096u) using clCreateSubBuffer, how am I end up having a continuous array?

E.g.:
big_buffer: 10 * 4096bytes (4K) total size
gpu_subbuffer: first half of big_buffer (5 * 4K)
cpu_subbuffer: second half of big_buffer (5 * 4K)

The kernel who works on gpu_subbuffer will write 4.5 * 4K of values.
The CPU who works on cpu_subbuffer will fill the whole buffer --> 5 * 4K of values

The result of the operation would be big_buffer to have 0.5 * 4K values nonsense/uninilized memory in between the gpu_subbuffer and cpu_subbuffer. Hence, I would have to move the values from the cpu_subbuffer to the end of the meaningful values of gpu_subbuffer.
I need a continuous array without unused parts in between.

Or do I miss out something? Thanks for any further tips!! :)

[MichalMrozek]

You are correct, if you have hole then this will require additional copy.
You may try with more fine grained alignment to cache line ( 64 bytes instead of page size ), however this raises the risk of having data corruption.
Please make sure there are no accesses that will share a cache line. Please also make sure there is no false cache line sharing.
The part of the buffer that is potentially shared is the most dangerous.

[OliverScherf]

Okay Thanks a lot! My question is answered. Now that I know the potential risks and restrictions I will consider them in my application.

I will close the issue for now.

[OliverScherf]

I followed your steps @MichalMrozek and it works like a charm.

I found a way to deal with the non-continuous data as well. Many thanks again for your support.

[MichalMrozek]

Awesome ! Very glad to hear that.