Improve Graph updaters shutdown

[vpaturet]

Summary

This PR ensures that graph updaters stop execution gracefully when the OTP server shuts down.
This is particularly important in a cloud environment where OTP instances are routinely started and stopped.

More specifically:

• all updaters should exit their initialization sequence when receiving an interruption signal.
• polling updaters should exit their polling loop when receiving an interruption signal,
• listening updaters that perform retry logic when disconnected should exit their retry loop when receiving an interruption signal.

Additionally:

• log messages during the shutdown sequence are clarified.
• in a couple of places in the code Throwables were caught. Now Exceptions are caught instead, to prevent hiding memory issues.

Issue

None

Unit tests

✅

Documentation

No

[codecov]

Codecov Report

Patch coverage: 31.64% and project coverage change: +0.06 🎉

Comparison is base (ad3520f) 64.51% compared to head (668ca8f) 64.57%.

Additional details and impacted files

@@              Coverage Diff              @@
##             dev-2.x    #5092      +/-   ##
=============================================
+ Coverage      64.51%   64.57%   +0.06%     
- Complexity     14031    14064      +33     
=============================================
  Files           1725     1729       +4     
  Lines          67222    67374     +152     
  Branches        7200     7208       +8     
=============================================
+ Hits           43365    43507     +142     
- Misses         21434    21447      +13     
+ Partials        2423     2420       -3     

Impacted Files	Coverage Δ
...anner/ext/legacygraphqlapi/LegacyGraphQLUtils.java	24.56% <0.00%> (ø)
...ygraphqlapi/datafetchers/LegacyGraphQLLegImpl.java	58.58% <0.00%> (+0.42%)	⬆️
...raphqlapi/generated/LegacyGraphQLDataFetchers.java	0.00% <ø> (ø)
...pplanner/ext/siri/SiriTimetableSnapshotSource.java	0.00% <0.00%> (ø)
...er/ext/siri/updater/SiriETGooglePubsubUpdater.java	0.00% <0.00%> (ø)
...pentripplanner/ext/siri/updater/SiriSXUpdater.java	0.00% <0.00%> (ø)
...t/siri/updater/azure/AbstractAzureSiriUpdater.java	0.00% <0.00%> (ø)
...tripplanner/ext/transmodelapi/TransmodelGraph.java	0.00% <0.00%> (ø)
...er/ext/transmodelapi/TransmodelGraphQLPlanner.java	0.00% <0.00%> (ø)
.../support/OTPProcessingTimeoutGraphQLException.java	0.00% <0.00%> (ø)
... and 29 more

... and 48 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[t2gran]

Only minor readability issues.

We should however, refactor this code a bit. Thread handling is difficult, and almost impossible to unit-test (often just creating a sense of false-security). So, my suggestion is to refactor the Updater interfaces, and break up the different phases in methods and move all iteration and thread handling logic to the "updater framework".

[leonardehrenfried]

Ouch!

[leonardehrenfried]

Do we really want to catch RuntimeException? Aren't those meant to stop the JVM?

[jtorin]

It's fine. RuntimeException is the base-class for all unchecked exceptions, for which lots of instances can occur during quite normal operations (like NoSuchElementException for example).

On this subject, while I agree that there are restrictions to its application, I don't think catching Throwable is automatically bad form. I tend to catch all errors on the top-level of the module or thread and log with as much pertinent information as possible from the state. If the alternative is to let the error propagate upwards to either the JVM or the execution framework (like Spring or an executor service) I rather log it myself to make sure the information is sent through the logging system in use. (Propagation of stdout/stderr is not always setup correctly.)

Catching Throwable and then continue execution is a no-no though, we'll agree on that.

[leonardehrenfried]

I realised that we caught Exception before which is a superclass of RuntimeException so this is catching fewer exceptions, not more.

[jtorin]

Indeed it is. Unless there is a specific reason Exception would actually be better.

(That RuntimeException inherits from Exception always trips me up.)

[vpaturet]

This is always safe to catch RuntimeException instead of Exception since compilation would fail if the block threw any checked exception. In this part of the code catching Exception is problematic since it could swallow InterruptedException and prevent graceful shutdown. And this is exactly what happened in this method.

But I can also replace :

        catch (RuntimeException e) {
            LOG.error(e.getMessage(), e);
          } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            otpIsShuttingDown = true;
            LOG.info("OTP is shutting down, cancelling wait for updaters readiness.");
          }

by:

         catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            otpIsShuttingDown = true;
            LOG.info("OTP is shutting down, cancelling wait for updaters readiness.");
          } catch (Exception e) {
            LOG.error(e.getMessage(), e);
          }

which is equivalent and maybe less confusing.
What do you think?

[jtorin]

@vpaturet I think the suggested second form is indeed clearer.