Error in C# binding - Attempted to Read or Write Protected Memory

[kalengi]

Make sure you have read the documentation, and have put forth a reasonable effort to find an existing answer.

Expected behaviour

Processing an audio file should produce a transcript on each and every call.

var transcript = leopardTranscriber.ProcessFile(audioFilePath);

Actual behaviour

The transcription requests are being handled by a server and work very well when the requests come from a single client. However, when I send requests simultaneously from two clients:

1. The requests get serviced properly for a while
2. Then after some unpredictable time the server crashes with the error:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at Pv.Leopard.pv_leopard_process_file(IntPtr, IntPtr, IntPtr ByRef, Int32 ByRef, IntPtr ByRef)
--------------------------------
   at Pv.Leopard.ProcessFile(System.String)
   at AudioTranscriber.Services.TranscriptionService.GetTranscript(AudioTranscriber.AudioRequest, Grpc.Core.ServerCallContext)

After some research, I came across this plausible explanation for the behavior: https://stackoverflow.com/a/42382470/212076

• The problem may be due to mixed build platforms DLLs in the project. i.e You build your project to Any CPU but have some DLLs in the project already built for x86 platform. These will cause random crashes because of different memory mapping of 32bit and 64bit architecture. If all the DLLs are built for one platform the problem can be solved.

I examined the DLLs that ship with the Leopard Nuget package:

• Leopard.dll: 32-bit
• libpv_leopard.dll: 64-bit

Since my project targets 64-bit architecture, then the issue must be triggered by the involvement of the 32-bit Leopard.dll. Under normal circumstances, it plays nicely with it's 64-bit counterparts. However, under conditions of increased load from multiple simultaneous requests, it causes memory access issues arising from the different memory mapping.

Is it possible to have a 64-bit version of Leopard.dll available for testing to verify this assertion?

Steps to reproduce the behaviour

1. Create a C# gRPC Server that

   • Initializes the Leopard library
     • var leopardTranscriber = Leopard.Create(accessKey);
   • Waits for requests from clients
   • Handles each client request and returns the transcript
     • var transcript = leopardTranscriber.ProcessFile(audioFilePath);
     • return Task.FromResult(transcript);

2. Create a C# gRPC Client that

   • Initializes the gRPC connection

     • var port = 3050;
     • var serverUrl = $"http://localhost:{port}";
     • var channel = GrpcChannel.ForAddress(serverUrl);
     • var client = new AudioTranscriber.AudioTranscriberClient(channel);
     • var audioRequest = new AudioRequest { AudioFilePath = audioFilePath };

   • Sends a request to the server for a transcript

     • var transcript = await client.GetTranscript(audioRequest);

3. Start two instances of the Client and have them repeatedly send requests to the Server. Exactly as you would to stress test the Server.

After a while of getting proper transcripts you get the AccessViolationException and the server crashes.

[laves]

Hi @kalengi, thank you for your detailed report. One question: does your server handle the requests asynchronously? I see you are using a single instance of Leopard to process all server calls. If a call to ProcessFile is made and then another before the first finishes, I would expect there to be an issue. If this is a possible situation, I'd recommend having one Leopard instance per request or a pool of pre-initialized Leopards to handle requests without conflicting. If this is not the case, we can look into your idea that it may be caused by conflicting dll architectures.

[kalengi]

Hi @laves,

Thanks very much for these insights. The Server requests are indeed asynchronous, and your solution may sort out the memory access issue.

I shall implement the Leopard pool and report back on the result.

[kalengi]

I can confirm that implementing a Leopard Pool eliminated the AccessViolationException.

[laves]

Great!