make conds use non-blocking transfers#7152
Conversation
|
This needs proper testing, there are sometimes weird side effects when enabling the non blocking transfers like this. |
|
At least for the portion of testing I am able to do, I have been using this change on my local copy for I think months at this point, it works completely fine on CUDA. I assume that the existing code for gating use of non-blocking transfers covers most of the known problematic backends, but if needed it could be gated to be CUDA-only |
|
I'm running this with my 7900 XTX (after force-enabling non-blocking support since it seems to be globally disabled) and so far it seems to work fine here too. I don't know if it did anything for performance though. |
|
Changes here need more evaluation before merging because I am noticing after removing all blocking calls on a path that performance is sometimes inexplicably slower, particularly on smaller image sizes (768x768 for example) -- 1024x1024 has a measurable performance boost though. From comparing profiles of it on and off there is no obvious part that is being slowed down but I would want to figure it out before this gets merged. |
|
Changed it to copy the conds to device before applying repeat, and I also changed memory pinning so that it should only happen once and avoids unnecessary copies every step (we're trying it every time but it'll become a no-op if the tensor is already pinned). Copying the tensor to device first means it's smaller when copied over and also should mean less memory usage because it'll probably be created as a view. This is probably as good as things will get unless it's changed to where the conds go to the GPU once and are kept there unless changed. |
Test Evidence CheckIf this PR modifies behavior that requires testing, a test explanation is required. PRs lacking applicable test explanations may not be reviewed until added. Please add test explanations to ensure code quality and prevent regressions. If this PR changes user-facing behavior, visual proof (screen recording or screenshot) is required. PRs without applicable visual documentation may not be reviewed until provided. You can add it by:
|
4 participants
This change should make use of non-blocking transfers for conds, avoiding unnecessary syncs. Should improve speed, as long as other things are not causing syncs between steps. It is also set up to use existing functions from the model management class to avoid using non blocking on unsupported platforms.