[SER] Passing a payload by value to HitObject::Invoke asserts in CodeGen

[damyanp]

Description

Passing a ray payload to dx::HitObject::Invoke via an "in" (by value) function parameter crashes the compiler in CodeGen. Invoke's payload parameter is inout, so a by-value parameter is not a valid argument for it, but instead of being diagnosed in Sema this reaches CodeGen and asserts while emitting the call.

In a release build the assert is compiled out, an invalid bitcast is emitted, and the user instead sees an unhelpful validation failure.

This is not related to payload access qualifiers - it reproduces with -disable-payload-qualifiers too. It also appears specific to HitObject::Invoke: the same shader using TraceRay instead compiles fine.

This was originally reported as a comment on #6464, but that issue is about a different (PAQ analysis) crash, so splitting it out here. It is also adjacent to #7761, which covered a const payload and was made a Sema error by #7797, but that change does not cover this by-value case.

Steps to Reproduce

struct [raypayload] Payload {
  float value : write(caller, closesthit, miss) : read(caller, closesthit, miss);
};

// 'p' is an "in" (by value) parameter. Passing it to Invoke's inout payload
// parameter asserts in CodeGen.
void Function(Payload p) {
  RayDesc ray = (RayDesc)0;
  dx::HitObject obj = dx::HitObject::MakeMiss(0, 0, ray);
  dx::HitObject::Invoke(obj, p);
}

[shader("raygeneration")]
void RayGen() {
  Payload p;
  p.value = 0;
  Function(p);
}

Command line: -T lib_6_9

Changing Function(Payload p) to Function(inout Payload p) compiles successfully, which is the workaround.

Actual Behavior

With an assert-enabled build:

Internal compiler error: LLVM Assert

assert(castIsValid(op, S, Ty) && "Invalid cast!")

dxcompiler!llvm_assert
dxcompiler!llvm::CastInst::Create
dxcompiler!llvm::IRBuilder<...>::CreateCast
dxcompiler!llvm::IRBuilder<...>::CreateBitCast
dxcompiler!clang::CodeGen::CodeGenFunction::EmitCall
dxcompiler!clang::CodeGen::CodeGenFunction::EmitCall
dxcompiler!clang::CodeGen::CodeGenFunction::EmitCallExpr
dxcompiler!`anonymous namespace'::ScalarExprEmitter::VisitCallExpr
dxcompiler!clang::CodeGen::CodeGenFunction::EmitScalarExpr
dxcompiler!clang::CodeGen::CodeGenFunction::EmitAnyExpr
dxcompiler!clang::CodeGen::CodeGenFunction::EmitIgnoredExpr
dxcompiler!clang::CodeGen::CodeGenFunction::EmitStmt

A preceding assert also fires:

assert(type->isReferenceType() == E->isGLValue() && "reference binding to unmaterialized r-value!")

With a release build the same source instead produces:

error: validation errors

Function: ?RayGen@@YAXXZ: error: Instructions must be of an allowed type.
note: at 'unreachable' in block '#0' of function '?RayGen@@YAXXZ'.
Validation failed.

Expected Behavior

This should be rejected in Sema with a diagnostic explaining that the payload argument to dx::HitObject::Invoke must be an inout-compatible lvalue, rather than asserting in CodeGen or producing a validation failure.

Environment

• DXC version: built from source at 7676b1f90 (current main), x64, Visual Studio, both Debug (assert build) and Release
• Host Operating System: Windows 11