Reload configuration when included files are changed

[hpcsc]

Feature Suggestion

I'm looking into deploying backstage to Kubernetes environment and use Hashicorp Vault dynamic database credentials feature (https://www.vaultproject.io/docs/secrets/databases/postgresql) to generate database credentials. These credentials will be generated and injected into backstage container filesystem using a sidecar. Backstage container will use $include to read those credentials

It's working fine however when database credentials are rotated by the sidecar, backstage is not able to reload configuration from filesystem yet. This is a request to look into some way to reload files ($include or $file) when files are updated

Possible Implementation

The simplest implementation that I can think of is using combination of filesystem watch and retry when failure: watching files when starting up and refresh in memory configuration when changed (I'm not too familiar with backstage code yet so not sure how the configuration is handled at the moment). In the edge case where backstage somehow misses filesystem change (.e.g. when container is under load and OS decides to drop change notification?), connection to database will fail with invalid credentials, a config reload and retry might work

Context

See Feature Suggestion above

[Rugvip]

We're pretty close to supporting this! Most of it was implemented in #6754. The missing pieces are watching of $include and $file references, as well as the implementation in the DB connection end. We could either re-read the secrets when failing to connect to the DB, or use config.subscribe() to listen for updates.

Is this something you want to work on @hpcsc? Otherwise we'll call it up for grabs for now

[hpcsc]

thanks @Rugvip. I'm afraid I can't work on this at the moment so it would be great if someone else can help

[vinzscam]

Closed by #9608