RDS proxy

[codyebberson]

Experimenting with RDS Proxy for better database upgrade recovery time.

Learn more: https://aws.amazon.com/rds/proxy/

Deploying to staging.

---

Update 1

Using RDS Proxy improved the database upgrade / server downtime impact.

When not using RDS proxy, the database upgrade led to 19 seconds of downtime.

When using RDS proxy, the database upgrade led to 4 seconds of downtime.

I believe those 4 seconds are actually due to this open issue in pg: brianc/node-postgres#2112

---

Update 2

Downtime reduced to 2 seconds using this workaround for brianc/node-postgres#2112

  process.on('uncaughtException', (err) => {
    globalLogger.error('Uncaught exception thrown', err);

    if (err.message && typeof err.message === 'string' && err.message.includes('Connection terminated unexpectedly')) {
      // The pg-pool library throws this error when the database connection is lost.
      // This can happen when the database server is restarted.
      // We do *not* want to exit the process in this case.
      return;
    }

    process.exit(1);
  });

Those 2 seconds due to connection error.

Server side log:

{
    "level": "ERROR",
    "timestamp": "2024-01-28T18:36:27.517Z",
    "msg": "Uncaught exception thrown",
    "error": "Error: Connection terminated unexpectedly",
    "stack": [
        "Error: Connection terminated unexpectedly",
        "    at Connection.<anonymous> (/usr/src/medplum/node_modules/pg/lib/client.js:132:73)",
        "    at Object.onceWrapper (node:events:632:28)",
        "    at Connection.emit (node:events:518:28)",
        "    at Socket.<anonymous> (/usr/src/medplum/node_modules/pg/lib/connection.js:63:12)",
        "    at Socket.emit (node:events:530:35)",
        "    at TCP.<anonymous> (node:net:337:12)",
        "    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)"
    ]
}

Client side log:

p [Error]: Client has encountered a connection error and is not queryable
    at Or.<anonymous> (C:\Users\cody\dev\medplum\packages\core\dist\cjs\index.cjs:42:7650)
    at Generator.next (<anonymous>)
    at fulfilled (C:\Users\cody\dev\medplum\packages\core\dist\cjs\index.cjs:5:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  outcome: {
    resourceType: 'OperationOutcome',
    issue: [ [Object] ],
    extension: [ [Object] ]
  },
  cause: undefined
}

That could potentially be fixed or handled by doing some kind of healthcheck on borrow from pool.

---

Update 3

Consider trying postgres-pool as a replacement for pg-pool:

• https://github.com/postgres-pool/postgres-pool
• https://www.npmjs.com/package/postgres-pool

It is less popular, but appears to have a more robust implementation.

---

Update 4

Abandoning postgres-pool - it is not completely API compatible with pg-pool. Given it's relative low popularity, I'm not inclined to do a major refactor just to try it.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

3 Ignored Deployments

Name	Status	Preview	Comments	Updated (UTC)
medplum-app	⬜️ Ignored (Inspect)	Visit Preview		Jan 28, 2024 6:46pm
medplum-storybook	⬜️ Ignored (Inspect)	Visit Preview		Jan 28, 2024 6:46pm
medplum-www	⬜️ Ignored (Inspect)	Visit Preview		Jan 28, 2024 6:46pm

[github-actions]

	Messages
📖	@medplum/core: 153.9 kB
📖	@medplum/react: 338.4 kB

Generated by 🚫 dangerJS against 0609cfa

[sonarcloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
81.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud