You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I was comparing Nsight systems profiling output for the GeoModel-Tilecal integration. In addition to the GPU utilization being low with a single thread because of all the processing done on the CPU to report to Geant4 after each step, there is a shift in which action takes the most time.
The top picture has 1 thread, the bottom has 32 threads. With 32 threads, most of the time is spent in memcpyasync in the extend-from-secondaries (and step-gather) actions which are not async and require a device synchronization, preventing kernels of other streams from executing and leading to many small gaps in GPU utilization. These copies come from thrust calls return values, e.g., remove_if_alive. I suspect than refactoring to have asynchronous memcopy to pinned memory would help scale GPU utilization with many host threads. I don't know of a way to tell Thrust where to copy the return value of the function so if we want to verify that this leads to performance improvements we might have to implement these functions.
The text was updated successfully, but these errors were encountered:
@esseivaju I think you've done a great job addressing this. Could you please add links to the relevant PRs, take a new snapshot of the timing, and close this out?
sethrj
changed the title
Performance scaling with host threads
Improve performance scaling with host threads
Oct 30, 2023
Comparison of scaling with host threads between Celeritas v0.3.2 and 2332351. Measured on Perlmutter using the Atlas Tilecal example, vecGeom 1.2.5 and Geant4 11.0.1. Relevant PRs:
Add unified memory support #965 (as of this comment, only helps if track sorting is enabled, which was turned off in this benchmark)
As mentioned in the first comment, the only async copy to pageable memory remaining are return values from thrust calls. There is no way to change that (see this discussion) and the gains would most likely be small since we have to synchronize each stream at the end of each step anyway so it's not a top priority to fix.
I was comparing Nsight systems profiling output for the GeoModel-Tilecal integration. In addition to the GPU utilization being low with a single thread because of all the processing done on the CPU to report to Geant4 after each step, there is a shift in which action takes the most time.
The top picture has 1 thread, the bottom has 32 threads. With 32 threads, most of the time is spent in memcpyasync in the extend-from-secondaries (and step-gather) actions which are not async and require a device synchronization, preventing kernels of other streams from executing and leading to many small gaps in GPU utilization. These copies come from thrust calls return values, e.g., remove_if_alive. I suspect than refactoring to have asynchronous memcopy to pinned memory would help scale GPU utilization with many host threads. I don't know of a way to tell Thrust where to copy the return value of the function so if we want to verify that this leads to performance improvements we might have to implement these functions.
![image](https://private-user-images.githubusercontent.com/7202402/272850518-64f9879a-9ed3-445f-9d25-d83d0909b916.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MzE4MzQsIm5iZiI6MTcyNDUzMTUzNCwicGF0aCI6Ii83MjAyNDAyLzI3Mjg1MDUxOC02NGY5ODc5YS05ZWQzLTQ0NWYtOWQyNS1kODNkMDkwOWI5MTYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjRUMjAzMjE0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZDg0NDJmNGI0Zjk4MDMzMmZiNTQ2YTEzNjkzOGFiOTBkYmM4MDFiYjk1YTFmMjFkOWU3MmExNjI5YzU1YWY4ZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.he-sIPCwvqcxhas1QqBFOafS6JIKr96eg27SZQ3ZPRM)
The text was updated successfully, but these errors were encountered: