LibraryImport was working properly with NET8 but not anymore with NET9

[RomainAn]

Description

Hi everyone.

Big congratulation for NET9 release ! We appreciate your hard work and we are looking forward for the future of dotnet.

We recently updated our stack from NET8 to NET9, and most of the transition went well.
However one service that we have is now failing to work and the only difference is the version of .NET.

        [LibraryImport(DllName, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(AnsiStringMarshaller)), SuppressUnmanagedCodeSecurity()]
        [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)]
        [UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
        internal static partial int tpcall(string svc, IntPtr idata, long ilen, out IntPtr odata, out long olen, long flag);

     int tpcall(char *svc, char *idata, long ilen, char **odata, long *olen, long flags);

The service is a wrapper around a C library (Oracle Tuxedo).

We are now getting InvalidOperation Exception : TPEINVAL error 4 (which is about parameters we feed the method invocation.
We did check breaking changes about interop but we only found this :
https://learn.microsoft.com/en-us/dotnet/core/compatibility/interop/9.0/cet-support

But this doesn't seems to be applying to us.

Is there any other changes that might affect the Interop part between NET8/9 ?

Best regards.

Reproduction Steps

I don't think it's possible to provide any minimal repro code.

Expected behavior

The method call should be working as expected.

Actual behavior

The Interop call is now failing

Regression?

This was working fine in NET8 but is now failing in NET9

Known Workarounds

No response

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[huoyaoyuan]

You can inspect the generated code by LibraryImport generator to see if there's any difference, and use a minimal C library to dump the values accepted at native side.

[karakasa]

The generated code:

.NET 8

// <auto-generated/>
internal static unsafe partial class PInvoke
{
    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Interop.LibraryImportGenerator", "8.0.11.1707")]
    [global::System.Runtime.CompilerServices.SkipLocalsInitAttribute]
    internal static partial int tpcall(string svc, nint idata, long ilen, out nint odata, out long olen, long flag)
    {
        global::System.Runtime.CompilerServices.Unsafe.SkipInit(out odata);
        global::System.Runtime.CompilerServices.Unsafe.SkipInit(out olen);
        byte* __svc_native = default;
        int __retVal = default;
        // Setup - Perform required setup.
        scoped global::System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller.ManagedToUnmanagedIn __svc_native__marshaller = new();
        try
        {
            // Marshal - Convert managed data to native data.
            __svc_native__marshaller.FromManaged(svc, stackalloc byte[global::System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller.ManagedToUnmanagedIn.BufferSize]);
            // Pin - Pin data in preparation for calling the P/Invoke.
            fixed (long* __olen_native = &olen)
            fixed (nint* __odata_native = &odata)
            {
                // PinnedMarshal - Convert managed data to native data that requires the managed data to be pinned.
                __svc_native = __svc_native__marshaller.ToUnmanaged();
                __retVal = __PInvoke(__svc_native, idata, ilen, __odata_native, __olen_native, flag);
            }
        }
        finally
        {
            // CleanupCallerAllocated - Perform cleanup of caller allocated resources.
            __svc_native__marshaller.Free();
        }

        return __retVal;
        // Local P/Invoke
        [global::System.Runtime.InteropServices.DllImportAttribute("a.dll", EntryPoint = "tpcall", ExactSpelling = true)]
        [global::System.Runtime.InteropServices.UnmanagedCallConvAttribute(CallConvs = new global::System.Type[] { typeof(global::System.Runtime.CompilerServices.CallConvCdecl) }), global::System.Runtime.InteropServices.DefaultDllImportSearchPathsAttribute((global::System.Runtime.InteropServices.DllImportSearchPath)4096)]
        static extern unsafe int __PInvoke(byte* __svc_native, nint __idata_native, long __ilen_native, nint* __odata_native, long* __olen_native, long __flag_native);
    }
}

.NET 9

// <auto-generated/>
internal static unsafe partial class PInvoke
{
    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Interop.LibraryImportGenerator", "9.0.11.2809")]
    [global::System.Runtime.CompilerServices.SkipLocalsInitAttribute]
    internal static partial int tpcall(string svc, nint idata, long ilen, out nint odata, out long olen, long flag)
    {
        odata = default;
        olen = default;
        byte* __svc_native = default;
        int __retVal = default;
        // Setup - Perform required setup.
        scoped global::System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller.ManagedToUnmanagedIn __svc_native__marshaller = new();
        try
        {
            // Marshal - Convert managed data to native data.
            __svc_native__marshaller.FromManaged(svc, stackalloc byte[global::System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller.ManagedToUnmanagedIn.BufferSize]);
            // Pin - Pin data in preparation for calling the P/Invoke.
            fixed (long* __olen_native = &olen)
            fixed (nint* __odata_native = &odata)
            {
                // PinnedMarshal - Convert managed data to native data that requires the managed data to be pinned.
                __svc_native = __svc_native__marshaller.ToUnmanaged();
                __retVal = __PInvoke(__svc_native, idata, ilen, __odata_native, __olen_native, flag);
            }
        }
        finally
        {
            // CleanupCallerAllocated - Perform cleanup of caller allocated resources.
            __svc_native__marshaller.Free();
        }

        return __retVal;
        // Local P/Invoke
        [global::System.Runtime.InteropServices.DllImportAttribute("a.dll", EntryPoint = "tpcall", ExactSpelling = true)]
        [global::System.Runtime.InteropServices.UnmanagedCallConvAttribute(CallConvs = new global::System.Type[] { typeof(global::System.Runtime.CompilerServices.CallConvCdecl) }), global::System.Runtime.InteropServices.DefaultDllImportSearchPathsAttribute((global::System.Runtime.InteropServices.DllImportSearchPath)4096)]
        static extern unsafe int __PInvoke(byte* __svc_native, nint __idata_native, long __ilen_native, nint* __odata_native, long* __olen_native, long __flag_native);
    }
}

[MichalPetryka]

C long corresponds to CLong, not C# long.

[jtschuster]

It looks like odata should be a ref parameter instead of out since tpcall() requires that *odata points to a buffer. In .NET 9, the generated code now clears the out parameters rather than leaving them uninitialized, which seems to be causing the issue.

From https://docs.oracle.com/cd/E13203_01/tuxedo/tux71/html/rf3c31.htm:

odata is the address of a pointer to the buffer where a reply is read into, and olen points to the length of that reply. *odata must point to a buffer originally allocated by tpalloc().

With the suggestion to use CLong rather than long, your signature would look like this:

internal static partial int tpcall(string svc, IntPtr idata, CLong ilen, ref IntPtr odata, out CLong olen, CLong flag);

Please let me know if this fixes the problem and we can close this issue.

[AaronRobinsonMSFT]

@RomainAn One more piece of data that would be helpful, target OS and bitness. Statements about C long like #109946 (comment) are correct, but only insomuch as they represent an attempt to create a fully cross-platform solution. There are scenarios where a C# long and a C long are the same size so it is possible you've been running on those platforms and that specific aspect of the signature isn't relevant, although should be fixed at some point.

Understanding not just the .NET version but also the machine environment (OS and bitness) where this failure is occurring is important for interop related questions.

[RomainAn]

Hello everyone,

Thank you all for your kind help.

We found the issue is now solved as @jtschuster mentioned, it was related to this : #101076

Wish you all a great day !