-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python control of CUDA streams #167
Conversation
…ags to enable new stream creation.
…to Python (if they exist).
… I know works with cupy.
Codecov Report
@@ Coverage Diff @@
## master #167 +/- ##
==========================================
+ Coverage 66.76% 67.00% +0.24%
==========================================
Files 69 69
Lines 7416 7464 +48
==========================================
+ Hits 4951 5001 +50
+ Misses 2465 2463 -2
Continue to review full report at Codecov.
|
Interesting. Is it worthwhile to code something like the |
I was thinking about adding in some cupy tests yesterday. It would be good from a coverage perspective to be able to test that we can make |
Sort of related to this, if you look in |
I did some reading on pycuda vs cupy today and I think the answer to my previous interoperability question is "yes". They are targeted to two different audiences with cupy being an easy way to move onto the GPU if you have a lot of numpy-based codes and pycuda is more about building things from scratch in a very CUDA way. In that regard how you can interface pycuda and Bifrost is limited to being able to move data between the array formats since I can't find an interface in pycuda that allows you to use an external thread. cupy, on the other hand, does have that feature and I use it in the example. |
Assuming the tests pass, this all looks great to me. (I can probably try it later today.) And it makes a great case for interoperability with other frameworks. Will maybe want to tout that in release notes and documentation next time. Could it be worthwhile to put |
(Maybe if it's in the library, |
Maybe the thing to do is use the terminology of cupy and call it something like |
I'm holding off on this to see if I can get a self-hosted runner going on the test machine. It sort of works but there are many mysteries. |
Those really slow builds on self-hosted/Python3.[68] are all for this. |
One thing we could do is run two self-hosted runner at once, one per GPU. |
Also maybe it's not necessary to include the CuPy tests for both 3.6 and 3.8? |
The cupy-cuda112 wheel is helping. It dropped the "Software Install - Python, part 2" time down to ~1 min from ~12 min. If we keep using the wheel then we need to make sure we are using the correct one for whatever version of cuda we have in the Dockerfile. |
We should be running two runners now, one on each GPU. I should test that. |
Two runners gets it down to ~30 minutes. Another way to save time would be to drop the Py2.7 self-hosted test. That would mean giving up Py2.7 testing for the GPU related code. |
I think I'm happy with that is in here now that there is the "per-thread default stream" flag in
|
I haven't been able to reproduce that failure. Oh well. |
@league do you think this one is ready to merge? I can't think of anything else I'd like to see as part of it. |
I hand-patched ./configure along with update to configure.ac
With bbd4b67, I thought it might be useful for the config summary to show what happened with the stream-model setting, so it's not just buried in the scroll. I thought about adding it to Trying to test on cetus, I discovered another thing: permission denied errors on This happens on every CUDA test:
|
This problem seems familiar (as in something I've fixed in the past) but looking back through the history the only thing I see is f2691ff. That's a different problem of cleaning up stale contents but has similar "stuff from other users" roots as this. I'm kind of surprised that it is even trying to change the permissions since world writable is what we want. Maybe we need to only try to make The UUID thing is interesting. What are you thinking with this? That there would be a per-user |
Regarding What's the number right beneath |
Ooh! Exciting security-hole opportunity, haha.
Combine that with the fact that I was just proposing pulling an identifier out of a config file and use it in the filename passed to |
Yeah, that number is the PID. Having everything in one place makes it easier to at least attempt to cleanup after a process died before it could do so itself. I guess that would be true of |
Looks like the permission denied issue is limited to
but |
Trying this, which is probably closer to what bool created = std::filesystem::create_directories(path);
if(created) {
std::filesystem::permissions(path, (std::filesystem::perms) perms,
std::filesystem::perm_options::replace);
} |
Using |
Before, it led to permission denied exceptions when /dev/shm/bifrost had been created by someone else. (We create it 0777 so can just proceed to use it even if owner is different.)
This PR exposes the
bfStreamGet
andbfStreamSet
methods through the Python API. This can help with interfacing Bifrost with other CUDA-aware packages, such ascupy
. For example:This PR also adds
set_stream
andget_stream
methods toBifrostObject
s that support them.