Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve device initialization for eventual CMSSW use #1037

Open
Tracked by #1143
sethrj opened this issue Nov 28, 2023 · 2 comments
Open
Tracked by #1143

Improve device initialization for eventual CMSSW use #1037

sethrj opened this issue Nov 28, 2023 · 2 comments
Labels
core Software engineering infrastructure enhancement New feature or request
Milestone

Comments

@sethrj
Copy link
Member

sethrj commented Nov 28, 2023

We've never had a completely satisfactory way of handling device initialization:

  • celeritas::device is a global variable that's allowed to be initialized exactly once
  • The CUDA runtime API device context is stored with threadlocal
  • Disabling the device (basically) has to be done with an environment variable

Important features we want to retain from the current implementation:

  • Manage a "stream" abstraction (so that the state's stream ID can be converted to a CUDA/HIP stream pointer without requiring CUDA API inclusion from host code)
  • Let params objects determine whether to allocate on device or not
  • Save diagnostic data about the device being used

It's really unclear to me how in practice we'd want a single process to share multiple GPUs.

@sethrj sethrj added enhancement New feature or request core Software engineering infrastructure labels Nov 28, 2023
@sethrj sethrj added this to the v1.0.0 milestone Nov 28, 2023
@sethrj sethrj mentioned this issue Mar 8, 2024
9 tasks
@drbenmorgan
Copy link
Contributor

I think the option to enable/disable device use is also required to cleanly implement drbenmorgan/celer-adept#10. I don't know how tightly coupled this is to the other items here, but perhaps we could split it out into a subitem? At least in terms of the celer-adept use case it probably only needs a parameter in SetupOptions and some way to pass that other than an env var into Device?

@sethrj
Copy link
Member Author

sethrj commented Jul 19, 2024

Yeah, we can split it into a separate option. For what it's worth, it is possible to set Celeritas "environment variables" programmatically—celeritas::environment().insert({"CELER_DISABLE_DEVICE", "1"}). But there should be a better way than that to disable the device when integrating externally.

The biggest problem is that lots of the code assumes this setting is set exactly once. If we allow it to change then it could result in an inconsistent state. The best option I can think of is to define a more general Config class (perhaps a wrapper to std::map<std::string, std::any>), pass it into the highest level classes (perhaps sharing it with a global config as a default?) and replace static data with class data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Software engineering infrastructure enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants