Can't connect to ElastiCache redis cluster via NAT instance

[jeremytm]

We're using a single node cluster to test our workflow, and cannot connect via a NAT instance.

Having done some debugging, we get an error immediately after the node is connected:

Disconnect ec2-XX-X-XX-XX.compute-1.amazonaws.com:6379 because the node does not hold any slot

Looking at the code this happens because the actual cluster node has an IP address from inside our VPC, while the NAT instance which we use to establish connectivity has a different public hostname. Because ioredis relies on the ip address of each node to create keys (or slots in the context of this error message), it thinks the node is missing and immediately disconnects it.

Is this something you guys would ever fix? I'm thinking some sort of map specified in connection options which maps the NAT hostname and port to the real VPC IP and ports could be a solution?

For anyone asking why we want to run local code with a remote redis implementation:

Usually our code runs remotely in AWS lambda. However our fastest dev workflow is where we test locally for convenience. Usually a local redis server is enough, but we regularly need to connect to a remote redis instance to debug with live data – and uploading our full repo to lambda on every iteration is way too slow.

Up until now we've used public redis instances in redislabs. However, we're moving to a more scalable solution in ElastiCache with redis cluster (which means running inside a VPC) and would like to be able to achieve the same fast workflow.

[luin]

Hi, haven't dug into this issue, but I remember there's an issue on the Redis repo about this issue: redis/redis#2527. The option announce-ip seems to have been added since Redis 4.x.

[jeremytm]

Interesting, however there are a couple of problems:

1. The cluster-announce-ip and cluster-announce-port options need to be specified in redis.conf, which we don't have access to per-node with elasticache.
2. We only need to remap the announced ip and port when connecting remotely via NAT. General connectivity within VPC should run as normal, so it makes more sense for implementation to be contextual, in the client.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[jeremytm]

For now we've made a fork of ioredis in our private npm, and have modified code to support what we need.

[luin]

@jeremytm Sorry for the late response. I've already migrated the code to TypeScript and will submit a pull request for the NAT support (by adding a nat map option) in the next week. I'll let you know then.

[luin]

@jeremytm Would you like to share the modification of the code that works for your case or just make a pull request?

[jeremytm]

It's pretty simple, and potentially not the safest option, but it works for us. It just passes the nodes in a callback provided via connection options. This callback is free to modify the nodes used by ioredis however it pleases.

#738

The callback we use is provided here:

(nodes)=>{
  
  for( let node of nodes ){
    let mapInfo = natMappings[node.host];
    if(mapInfo){
      
      let toInfo = mapInfo[node.port.toString()];
      if(toInfo){
        
        let [toHost,toPort] = toInfo.split(':');
        node.host = toHost;
        node.port = toPort;
        
      } else {
        throw new Error(`Port ${node.port} not found in map for ${node.host}`);
      }
      
    } else {
      throw new Error(`Host ${node.host} not found in map`);
    }
  }
};

Nat mappings would be something like:

{
  'rugby-cache-0001-001.xxxxxx.0001.use1.cache.amazonaws.com': {
    '6379': 'ec2-xx-x-xx-xx.compute-1.amazonaws.com:6379'
  },
  'rugby-cache-0001-002.xxxxxx.0001.use1.cache.amazonaws.com': {
    '6380': 'ec2-xx-x-xx-xx.compute-1.amazonaws.com:6380'
  }
}

[luin]

@jeremytm Neat solution! A question for it though that since Redis Cluster returns IPs instead of hostnames, do you have to specify both IPs & hostnames? E.g. let's say rugby-cache-0001-001.xxxxxx.0001.use1.cache.amazonaws.com points to 172.189.1.2, the nat mappings should be:

{
  'rugby-cache-0001-001.xxxxxx.0001.use1.cache.amazonaws.com': {
    '6379': 'ec2-xx-x-xx-xx.compute-1.amazonaws.com:6379'
  },
  '172.189.1.2': {
    '6379': 'ec2-xx-x-xx-xx.compute-1.amazonaws.com:6379' // same as the above
  },
}

[jeremytm]

Yeah, my bad, That's actually how ours looks. I just messed something up during copying and pasting while obfuscating the addresses.

[ioredis-robot]

🎉 This issue has been resolved in version 4.3.0 🎉

The release is available on:

• npm package (@next dist-tag)
• GitHub release

Your semantic-release bot 📦🚀