Get Word Graph - catastrophic failure

[johnml1135]

We called word graph and machine-engine crashed.

2023-11-06 09:01:09 | serval-api: 2023-11-06T09:01:09.287587063Z info: Grpc.Net.Client.Internal.GrpcCall[3]       Call failed with gRPC error status. Status code: 'Unavailable', Message: 'Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. IOException: The response ended prematurely while waiting for the next frame from the server.'.
-- | --
  |   | 2023-11-06 09:01:09 | serval-api: 2023-11-06T09:01:09.287432009Z fail: Grpc.Net.Client.Internal.GrpcCall[6]       Error starting gRPC call.       System.Net.Http.HttpRequestException: An error occurred while sending the request.        ---> System.IO.IOException: The request was aborted.        ---> System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.          at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame\|58_1()          at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)          at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()          --- End of inner exception stack trace ---          at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)          at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()          at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()          at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationTo
  |   | 2023-11-06 09:01:09 | serval-api: 2023-11-06T09:01:05.838346253Z info: System.Net.Http.HttpClient.SmtTransfer.ClientHandler[100]       Sending HTTP request POST http://machine-engine/serval.translation.v1.TranslationEngineApi/GetWordGraph
  |   | 2023-11-06 09:01:06 | machine-engine: 2023-11-06T09:01:02.85937085Z info: System.Net.Http.HttpClient.ClearML-NoRetry.LogicalHandler[101]       End processing HTTP request after 48.2537ms - 200
  |   | 2023-11-06 09:01:05 | serval-api: 2023-11-06T09:01:05.838295592Z info: System.Net.Http.HttpClient.SmtTransfer.LogicalHandler[100]       Start processing HTTP request POST http://machine-engine/serval.translation.v1.TranslationEngineApi/GetWordGraph
  |   | 2023-11-06 09:01:05 | serval-api: 2023-11-06T09:00:49.80766581Z fail: Serval.Shared.Controllers.ServiceUnavailableExceptionFilter[0]       A user tried to access an unavailable service. See health-check logs for more details.
  |   | 2023-11-06 09:01:03 | machine-engine: 2023-11-06T09:01:02.8593362Z info: System.Net.Http.HttpClient.ClearML-NoRetry.ClientHandler[101]       Received HTTP response headers after 48.1815ms - 200

[johnml1135]

From the rancher logs:

If I got the right one... I see it was terminated with Reason: Error, Exit Code: 139. And on the hosts syslog, I see several messages like this:
[5532514.274044] .NET ThreadPool[3711789]: segfault at 10 ip 00007f168c2468c0 sp 00007f166d7f7320 error 4 in libthot.so[7f168c179000+1b2000]

[ddaspit]

If we can find out which project, then we can reproduce it in a dev environment.

[johnml1135]

Can we at least wrap the C++ code code in a overall try/catch to not have the unhandled seg fault, such as with https://github.com/Plaristote/segvcatch?

[ddaspit]

Once a segmentation fault occurs, any further execution of code in the process has undefined behavior. There just isn't a safe way to continue running the process. I'm guessing that is why the library hasn't been touched in 5 years.

[ddaspit]

I did some investigation. The SMT model is corrupted. It looks like SF crashed while the model was being written to disk. I will update Machine and Thot, so that the corrupted model is handled more gracefully. The user will need to retrain the model to fix it. Maybe SF could prompt the user to retrain the model if there is an error while inferencing?