nvptx64-nvidia-cuda target does not support self-referential statics

[FractalFir]

When compiling this code for the nvptx64-nvidia-cuda target, an LLVM error will be thrown:

#![no_std]
struct Bar(&'static Bar);
static FOO: Bar = Bar(&FOO);

rustc-LLVM ERROR: Circular dependency found in global variable set
Compiler returned: 101

This appears to be a fundamental limitation of PTX(self-referential globals are not allowed). This issue seems to be present in all versions of rust which support this target(from the newest nightly back to at least 1.45.2).

I am opening this issue mostly to document this limitation, and I have no idea how it could get fixed.

[workingjubilee]

uh... cc @rust-lang/opsem If you want an excuse to call PTX a nonconforming implementation of Rust, we've got one, I guess?

[RalfJung]

I would be very surprised if this is the only non-conformity of that target. ;) This one is fairly harmless since it just reliably errors.

[workingjubilee]

We need to patch this to obtain a post-monomorphization error for this target so that we error before we reach LLVM, as LLVM may change whether or not this errors for reasons that may not be related to whether or not it is correct. It would be best if we did not produce IR with LLVM or hardware-level UB on a target.

I guess the hardware is virtual, but

[FractalFir]

We need to patch this to obtain a post-monomorphization error for this target so that we error before we reach LLVM, as LLVM may change whether or not this errors for reasons that may not be related to whether or not it is correct.

What would be the best way to detect this? And would people mind some kind of fix-up for this instead?

For Rust-CUDA(which I am currently working on), we are fine generating some kind of lazy-initialized static here(if that is possible, and does not violate the semantics of the Rust language).

Of course, I'd prefer if I could contribute something like that to the upstream(so the issue is fixed there too). The bug affects both the ordinary cg_llvm & cg_nvvm, so fixing it generally would be ideal.

[FractalFir]

Moreover, I must note this issue is caused by any kind of cyclical reference - not just a self-reference.

[workingjubilee]

For Rust-CUDA(which I am currently working on), we are fine generating some kind of lazy-initialized static here(if that is possible, and does not violate the semantics of the Rust language).

Oh dear. "Lazily-initialized" how, exactly?

[FractalFir]

For Rust-CUDA(which I am currently working on), we are fine generating some kind of lazy-initialized static here(if that is possible, and does not violate the semantics of the Rust language).

Oh dear. "Lazily-initialized" how, exactly?

Before starting execution(all CUDA entrypoints are clearly marked), we check an atomic variable. If it is the first time this kernel is excecuted, we execute a fixup function.

Nothing crazy(like recreating LazyLock in PTX), all statics get fixed up at once. Perhaps "Lazy-initialized" is not the best term for it. Just a fixup on first execution of a loaded program.

Ideally, we would not need this whole song and dance, but it seems like some crates(like bumpalo) use self-referential statics. bumpalo is decently common, so it would be neat if it worked here.

[LegNeato]

/cc @kjetilkjeka as you may have opinions here.

[kjetilkjeka]

This is a bit of an annoyance. I guess it's one of things I would hope that LLVM or the CUDA runtime handled, but it's also not that much of a surprise when they don't. I'm not an expert on solving these kind of problems, but here's my view of things.

I guess it's possible to also call a ptx function from a different ptx module (e.g. by using cuLinkAddFile to link them) . I think to produce correct code it would require us to emit the "fixup code" in all ptx functions? But we might also want to avoid the code for this impossible branch when the functions are being inlined (which is practically always in ptx) as they are already run on the "outer function" or kernels the functions are being inlined into.

This makes me think the init code should probably be added later than (llvm) codegen. Ideally at ptx loading/jitting. Since jittign is done by the CUDA driver, practically speaking it needs to happen somewhere in the linking/lto stages of things. Do we have sufficient control of things in cg_nvvm and what comes after to make that feasible?

Is this analog to the things going on with wasm with __wasm_apply_global_relocs? If so, can we learn something from the approach taken there?

Unless this materializes incredibly quickly then emitting the kind of error that @workingjubilee suggests will be an important first step.