Allow Instance Health Check interval to be configured

[hardillb]

Description

As a FlowFuse admin
I would like to be able to set the timeout on the healthcheck
So that I can run large blocking workloads without nr-launcher killing the process.

Which customers would this be available to

Everyone - CE/Starter/Team/Enterprise

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

Requested by: https://app-eu1.hubspot.com/contacts/26586079/record/0-3/10928370628/

• Allow Instance Health Check interval to be configured nr-launcher#227
• Update drivers
  • Allow Instance Health Check interval to be configured driver-localfs#123
  • Allow Instance Health Check interval to be configured driver-docker#91
  • Allow Instance Health Check interval to be configured driver-k8s#149
• Implement health check timeout via Env Vars #3713
• Implement health check timeout via UI #3715

[Steve-Mcl]

Questions

1. As an MVP, would it be suitable/desirable to support this via ENV Vars initially?
2. There are multiple constants that define the hang detection. Would we expose all or just HEALTH_POLL_INTERVAL?
   • HEALTH_POLL_INTERVAL: Interval between status polls of Node-RED used to detect a hung runtime (default 7.5sec)
   • HEALTH_POLL_MAX_STARTUP_ERROR_COUNT: The number of consecutive timeouts during startup phase before considering a NR hang (default 10)
   • HEALTH_POLL_MAX_ERROR_COUNT: The number of consecutive timeouts before considering a NR hang (default 3)
   • HEALTH_POLL_TIMEOUT: Timeout to apply to health polling requests (defaults to 500 less than HEALTH_POLL_INTERVAL)

[knolleary]

@Steve-Mcl I think env var approach would be a good quick iteration in terms of implementation. A second iteration that adds options in the settings for this could then just cause those env vars to be set under the covers.

If we don't do the UI piece, it will be critical to have clear, discoverable documentation about the setting. I don't think adding the UI piece is a huge overhead here, so I'd suggest see if you can timebox that piece to a couple hours at most once the base env var parts are done.

In terms of what settings, HEALTH_POLL_INTERVAL is the main one we care about here and I think is sufficient.

[Steve-Mcl]

In the design phase, I tried the env var approach and discovered the UX is sub optimal in that the container needs to be suspended (destroyed) in order for it to pick up any change of value. This is actually not necessary since we pass settings to Launcher.start (and that is where the health check timer is started). Therefore, for improved UX and simplified implementation I am going back on the idea of using ENV VARs and actually passing the timer value along with other settings like userDir and port in the drivers async settings call.

[knolleary]

@Steve-Mcl I've taken a second look at this and I don't see any need to update all the drivers. We can centralise it in the /settings api endpoint in the forge app - https://github.com/FlowFuse/flowfuse/blob/main/forge/routes/api/project.js#L798 - which takes the settings from the driver and adds in a bunch of standard settings (which this would be one).

The driver settings are intended to be the driver-specific things that need setting differently depending on the needs of the driver.

[Steve-Mcl]

@Steve-Mcl I've taken a second look at this and I don't see any need to update all the drivers. We can centralise it in the /settings api endpoint in the forge app - https://github.com/FlowFuse/flowfuse/blob/main/forge/routes/api/project.js#L798 - which takes the settings from the driver and adds in a bunch of standard settings (which this would be one).

Hmmm, good point. i think since this work started out as a Env Vars I got trapped in that route.

Will pull everything tomorrow and revisit.

[joepavitt]

@Steve-Mcl status report please? Is #3715 the only outstanding item here?

[Steve-Mcl]

@Steve-Mcl status report please? Is #3715 the only outstanding item here?

#3715 (PR #3716) is the only item to be merged. There was some re-work as requested and carried out on the drivers.

Ready for final review/merge at any time.