Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to put the 3D array in pinned memory so it stays on the host #143

Merged
merged 3 commits into from
Sep 2, 2020

Conversation

MaxThevenet
Copy link
Member

@MaxThevenet MaxThevenet commented Sep 2, 2020

This PR adds option to allocate the 3D field array on pinned memory, so it stays on the host. This should generate H2D and D2H copies of slices, which is slower, but would allow to run a large problem (that doesn't fit in GPU memory) on GPU.

NOTE: This has not been tested for accuracy yet. For all slices, in verbose = 1 mode, I get messages like islice: 6 n_iter: 1 relative B field error: 0, so I am not sure this is correct. Hence, this would need more testing before merging.

  • Small enough (< few 100s of lines), otherwise it should probably be split into smaller PRs
  • Tested (describe the tests in the PR description)
  • Runs on GPU (basic: the code compiles and run well with the new module)
  • Contains an automated test (checksum and/or comparison with theory)
  • Documented: all elements (classes and their members, functions, namespaces, etc.) are documented
  • Constified (All that can be const is const)
  • Code is clean (no unwanted comments, )
  • Style and code conventions are respected at the bottom of https://github.com/Hi-PACE/hipace
  • Proper label and GitHub project, if applicable

@MaxThevenet MaxThevenet added component: fields About 3D fields and slices, field solvers etc. GPU Related to GPU acceleration performance optimization, benchmark, profiling, etc. labels Sep 2, 2020
@SeverinDiederichs
Copy link
Member

This looks great! Your input script must have been bad, if the error is 0 everywhere it basically means that nothing is happening and all quantities should be 0.

I tested it on my GPU and

  1. it gives the correct result
  2. it uses indeed less memory on the GPU:
    Using development:
Total GPU global memory (MB) spread across MPI: [7973 ... 7973]
Free  GPU global memory (MB) spread across MPI: [5162 ... 5162]

Using host_device

Total GPU global memory (MB) spread across MPI: [7973 ... 7973]
Free  GPU global memory (MB) spread across MPI: [6058 ... 6058]

Copy link
Member

@SeverinDiederichs SeverinDiederichs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we found out, there is an issue with IO, but without IO it works great. This is a big step towards high-resolution simulations on a single GPU.

src/fields/Fields.cpp Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: fields About 3D fields and slices, field solvers etc. GPU Related to GPU acceleration performance optimization, benchmark, profiling, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants