Pool cluster connections do not recover after database comes back online

[mafischer]

the mariadb connector is used to connect to a galera cluster via the pool cluster api. the pool cluster is configured as follows:

const cluster = mariadb.createPoolCluster({
    canRetry: true,
    removeNodeErrorCount: 1,
    restoreNodeTimeout: 5000,
    defaultSelector: 'ORDER',
    metaAsArray: true
});

each node in the pool cluster is configured as follows:

const config = {
    "connectionLimit": 5,
    "queueLimit": 20,
    "acquireTimeout": 10000,
    "connectTimeout": 2000,
    "port": 3307,
    "dateStrings": true,
    "timezone": "Etc/UTC",
    "multipleStatements": true,
    "user": "example",
    "password": "example",
    "database": "example",
    "host": "host1.example.com"
};
cluster.add("host1", config);

After some brief database disruption event (i.e. galera cluster is offline for 5 minutes), the pool cluster is not able to recover connections until after an application restart, despite the galera cluster coming back online and returning to normal function.

In the pool cluster configuration, I have attempted many different values for removeNodeErrorCount including null, I do not see a change in behavior.

Error: No node have been added to cluster or nodes have been removed due to too much connection error at _getConnection (/home/app/node_modules/mariadb/lib/pool-cluster.js:122:11) at PoolCluster.getConnection (/home/app/node_modules/mariadb/lib/pool-cluster.js:94:12) at FilteredPoolCluster.getConnection (/home/app/node_modules/mariadb/lib/filtered-pool-cluster.js:21:20)

Error: No Connection available for '.*'. Last connection error was: pool is ending, connection request aborted at PoolCluster._getConnection (/home/app/node_modules/mariadb/lib/pool-cluster.js:135:29) at /home/app/node_modules/mariadb/lib/pool-cluster.js:346:18 at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:97:5)

[mafischer]

What is the intended purpose of removeNodeErrorCount value? Is that the amount of errors before permanently removing the node, or is the amount of error before temporarily removing the node until it is able to be connected to again?

[rusher]

The option removeNodeErrorCount is to remove permanently the node. see poolcluster-options

Default value of 5 was initially for compatibility with mysql/mysql2 connectors. (see PoolCluster options)

I would think that the default value of Infinity would be a lot better (nodes with an error are just blacklisted without being remove at all)

[lukeescude]

@mafischer have you tested setting it to Infinity?

We're encountering the same issue, but I completely missed the "infinity" option in the documentation. We're going to test this out.

[rusher]

As sayed in previous messages, the current option removeNodeErrorCount default to 5 attempts value has been set for compatibility with mysql/mysql2 drivers.

When a node doesn't respond, it is already blacklisted for some time, a better solution for new 3.x version is to have Infinity as default, and document that difference with other driver. Driver must have sane default behavior and removing nodes without explicit instruction isn't.