fix: make shutdown more graceful

[aaron-steinfeld]

Description

Couple existing issues here:

• clients were being shutdown before the server, meaning we were unable to drain pending requests at the moment of shutdown
• if the service fails to start, the shutdown hook was throwing a NPE because not all variables had values yet. No functional impact here, but commonly caused log confusion.

[github-actions]

Test Results

32 tests  ±0   32 ✅ ±0   8s ⏱️ ±0s
 9 suites ±0    0 💤 ±0 
 9 files   ±0    0 ❌ ±0 

Results for commit 8e0acda. ± Comparison against base commit fc0f3c2.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.25%. Comparing base (fc0f3c2) to head (8e0acda).

Additional details and impacted files

@@            Coverage Diff            @@
##               main      #97   +/-   ##
=========================================
  Coverage     67.25%   67.25%           
  Complexity      107      107           
=========================================
  Files            17       17           
  Lines           620      620           
  Branches         32       32           
=========================================
  Hits            417      417           
  Misses          184      184           
  Partials         19       19           

Flag	Coverage Δ
unit	67.25% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tim-mwangi]

clients were being shutdown before the server, meaning we were unable to drain pending requests at the moment of shutdown

Shouldn't clients be shut down before the server?

[aaron-steinfeld]

clients were being shutdown before the server, meaning we were unable to drain pending requests at the moment of shutdown

Shouldn't clients be shut down before the server?

Nope - it's a little unintuitive. Shutdown server (the graceful path) basically stops accepting requests and then waits for any pending request to complete before shutting down and returning. It's hard to complete those pending requests when we've already shut down all our clients. So shutting down the clients should really be the last thing we do once we know they're no longer in use.