Changing database details on multi-instance isn't applied to other instance until reboot

[flamber]

Describe the bug
Changing database connection details, like "Database name" or "Connection String" on multi-instance (HA/horizontal scaling) isn't applied to other instance until all instances are reboot.
Example when switching from the selected database from staging to production, or a different user (different privileges).
Or using Serialization, which will then cause all instances not to use the updated details.

There's no workaround like using sticky-connections on the load balancer, or even dropping the connections on the database.
It requires restarting all instances after details has been changed to ensure all instances are connecting using updated details.
Or manually saving settings for each instance (if the instances are accessible individually).

To Reproduce

1. Setup two instances to the same application database - we'll call them instanceA and instanceB
2. instanceA: Setup a new database source in Admin > Databases (I used MariaDB, but anything goes)
3. instanceA: Run a query like SELECT DATABASE(), CURRENT_USER(); result would be something like:

| DATABASE | CURRENT_USER |
| -------- | ------------ |
| mydb     | myuser       |

4. instanceB: Run step 3 on this instance, results are the same.
5. instanceB: Admin > Database > change some details like Username or Database name, and run step 3 again, now results are:

| DATABASE | CURRENT_USER |
| -------- | ------------ |
| somedb   | anotheruser  |

6. instanceA: Run step 3 on this instance, results does not match step 5's results as expected, but is still step 3's results.
7. instanceA: Admin > Database > it will show the details that was updated on instanceB in step 5.

Expected behavior
Connection details to be flushed on all instances. Not sure if it should be graceful or hard, but I'm leaning towards hard.
By graceful, I mean only replacing idle connections, so it doesn't interrupt an existing ongoing query, whereas hard drops all connections.
It took a lot of work (years) to gracefully do reloads on Nginx and Apache, so better to do it hard, since the database details are probably not changed often and if they are, it's probably for a good reason.

Information about your Metabase Installation:
Only did two tests, since it took a long time to reproduce - one on 0.36.11 and one on 0.38.0

Severity
I'm unsure what consequence it has for sync/scan too, but it's definitely close to a P1.

Additional information
It's not the same problem as #13270, since that has a workaround with sticky load balancer.

⬇️ Please click the 👍 reaction instead of leaving a +1 or update? comment

[paoliniluis]

Adding one use case to this one (maybe we have to create another issue if needed):

• add a database with an ssh tunnel
• then disable the ssh tunnel
  result: the SSH tunnel will stay enabled