Fix ABI problem when marshalling X509VerifyStatusCode

[uweigand]

The X509VerifyStatusCode type is defined as an enum in C code, but as
a struct (with a single member) in C# code. This means that marshalling
this type only works correctly on platforms where the ABI treats these
two types as equivalent. This fails e.g. on Linux on s390x.

Fixed by changing all native functions that take X509VerifyStatusCode
as argument or return type to use a plain "int" instead.

[ghost]

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The X509VerifyStatusCode type is defined as an enum in C code, but as
a struct (with a single member) in C# code. This means that marshalling
this type only works correctly on platforms where the ABI treats these
two types as equivalent. This fails e.g. on Linux on s390x.

Fixed by changing all native functions that take X509VerifyStatusCode
as argument or return type to use a plain "int" instead.

Author:	uweigand
Assignees:	-
Labels:	area-System.Security
Milestone:	-

[bartonjs]

I'm not fundamentally opposed here, but I am confused... what ends up happening on s390x?

[uweigand]

I'm not fundamentally opposed here, but I am confused... what ends up happening on s390x?

For return values, an integer is returned in general purpose register %r2. However, all structs, including a struct that contains only a single integer, are returned on the stack via implicit reference. So if one side of the marshalling assumes we need an implicit reference, and the other side doesn't, the arguments end up all confused.

(This is different on x64 and arm, because there is an ABI special case where structs that contain only a single integer are not returned via implicit reference, but rather in the same register where a plain integer would be returned.)

[bartonjs]

Ah, so the native library (which sees the type as an enum/integer) still does single register, but since the P/Invoke Marshaller sees the type as a struct it tries unpacking it per struct rules, resulting in a mismatch?

[uweigand]

Ah, so the native library (which sees the type as an enum/integer) still does single register, but since the P/Invoke Marshaller sees the type as a struct it tries unpacking it per struct rules, resulting in a mismatch?

Exactly. The marshaller prepares a buffer on the stack, and passes a pointer to that buffer to the native routine, expecting the native routine to fill (what it thinks to be) the struct to be returned in. But native code thinks it returns an integer, and so doesn't expect a buffer pointer but simply returns the integer in register. (Plus the presence of the return pointer shifts the other arguments by one slot, which native code also doesn't expect, so the incoming arguments are already wrong.)