Skip to content

Commit

Permalink
fix: correct shutdown sequence for http server (ENG-87) (#847)
Browse files Browse the repository at this point in the history
Please read the full report [here](https://www.notion.so/voiceflow/general-runtime-MongoDB-issues-5dcc5295b80a4a39916b6b4a6691cf91).

Put simply, we were seeing this Mongo Error: `'MongoNotConnectedError: Client must be connected before running operations`
However, it was always occurring AFTER a `SIGTERM` event:
![image](https://github.com/voiceflow/general-runtime/assets/5643574/35c030d4-3738-4134-a72c-f978a8e37282)

This is because we were kill all client connections first (Mongo, Redis, etc), THEN we were stopping the server from accepting new connections.

What this means is that it was possible for a existing request to be executed, the mongo client to get closed, then for that request to try and make a call to mongo.

To reproduce the error locally, just checkout this commit/branch:
18f076c
mesh the service and try to make a API call that uses mongo to it within 30 seconds.
<img width="1406" alt="Screenshot 2024-06-14 at 7 12 03 PM" src="https://github.com/voiceflow/general-runtime/assets/5643574/9f9ba100-5fc9-4647-89cc-a237ac861529">



Co-authored-by: Tyler <tylerhan97@gmail.com>
  • Loading branch information
DecathectZero and DecathectZero committed Jun 14, 2024
1 parent 79b48ad commit 647a4d6
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,16 +62,16 @@ class Server {

/**
* Stop server
* - Stops services first, then server
* - stops accepting new connections, wait for all existing ones to drain, then stop services
*/
async stop() {
// Stop services
await this.serviceManager.stop();

if (this.server) {
this.server.close();
await once(this.server, 'close');
}

// Stop services
await this.serviceManager.stop();
}
}

Expand Down

0 comments on commit 647a4d6

Please sign in to comment.