Database layer for Idempotency Keys persistence.

[marcoonroad]

Let's imagine a clusterized network (either physical or virtual/dockerized). Let's also suppose that our clients are ERP API integrations (no browser here). How can we ensure that duplicated requests don't spend money more than once? The solution is to persist such Idempotency Key + Request/Response information on some shared store for the running micro-service instances.

Good candidates for the backend storage are:

• 🐘 Postgres, due the widespread usage.
• 🔴 Redis, due the TTL of keys support.

I'm just opening this issue to start a nice discussion here, and, if you would approve, I could as well open a Pull-Request to implement that. 😄

Thanks in advance! 👍

[chris-jamieson]

Hey @marcoonroad apologies I didn't see this post earlier.

Absolutely agree that using some persistent store for idempotency keys would make sense.

There are several good cache managers for Node which incorporate support for different stores (a favourite of mine is Node Cache Manager.

Perhaps a good approach to this would be to allow different stores to be supported (e.g. in-memory by default, redis / postgres / mongo etc. if specified by config).

Would be happy to discuss further and PRs warmly welcomed.

[marcoonroad]

No problem, @jamiesoncj, it's quite hard to be up-to-date with GitHub notifications nowadays, they have a poor support for thread notifications anyways.

Redis is indeed one of the best for caching, 'cause it can asynchronously perform an on-disk backup (quite good when the server/machine crashes, different of Memcached, used to optimize data retrieval) and supports Time-to-Live on keys.

What about the initial implementation offer in-memory and Redis backends?
Thanks for the reply. o/

[chris-jamieson]

Great, if you want to produce a PR for Redis option that would be great.

Initially this can be quite simple - the main function should accept an options argument, which can contain cache-related keys, and also might allow us to set some other options e.g. specify the request header for the module to check (example below).

Can I suggest that the signature might be something like this:

var idempotency = require('@optimuspay/express-idempotency');

// Example for in-memory cache
var options = {
  requestHeader: 'Idem-Key', // default: 'Idempotency-Key' - this is not yet implemented
  cache: {
    store: 'memory',
    options: { // lruOptions if store is memory (see https://github.com/isaacs/node-lru-cache)
      max: 500,
      length: function (n, key) { return n * 2 + key.length },
      maxAge: 10000,
    }
  }
};

// Example for Redis cache
var options = {
  requestHeader: 'Idem-Key', // default: 'Idempotency-Key' - this is not yet implemented
  cache: {
    store: 'redis',
    options: { // options for redis.createClient if store is Redis (see https://github.com/NodeRedis/node_redis#options-object-properties)
      host: '127.0.0.1',
      port: 6379,
      path: null,
    },
    // redisClient: client, // instead of options property above, an initialised Redis client from `redis.createClient();` could be passed in here
  }
};

app.use(idempotency(options));

Look forward to your suggestions - the above is just an initial idea.