Client requests always going to same rack

[marcoivich]

Hi,
We created a 9-node cluster in one datacenter with 3 racks, each having 3 nodes following the example at https://github.com/Netflix/dynomite/wiki/Getting+Started#c-6-node-cluster-3-racks-with-2-nodes-on-each. The racks are in AWS in zones us-east-1b, us-east-1c, and us-east-1d.
We created a client in Java to connect to the above using the following maven dependency:

com.netflix.dyno
dyno-jedis
1.0.4

We also created a test Java app to test failover that uses the client and writes and reads from Redis. However, we noticed that all writes and reads always go to one rack (us-east-1c) and if we shut down the 3 servers in that rack the client errors out in every write and read. This does not happen when we shut down the other 2 racks (us-east-1b, us-east-1d). Is this the intended behavior? Our expectation was that traffic gets routed to whatever racks are available regardless of what rack fails.

Finally, we found that we can set up the environment variable EC2_AVAILABILITY_ZONE to a specific zone (e.g., us-east-1b) while running our test, and that in this case our failover works (e.g., we can successfully shut down any of the racks and requests get routed to the other racks that are up). The only problem with this solution is that we will need to set up the EC2_AVAILABILITY_ZONE variable in every app servers connecting to Dyno. Is there a better way to make this work?

Thank you

[opuneet]

Hey @marcoivich
yes, there is a binding to EC2_AVAILABILITY_ZONE and yes Dyno does use local rack affinity, for best latency results. You can override your own local rack using these method in ConnectionPoolConfigurationImpl.

    @Override
    public String getLocalDC() {
        return localDC;
    }

    public ConnectionPoolConfigurationImpl setLocalDC(String dc) {
        this.localDC = dc;
        return this;
    }

So 2 ways to do it.

1. Cast to ConnectionPoolConfigurationImpl and then call the setLocalDC method.
2. Extend and Override ConnectionPoolConfigurationImpl with your own class and then pass that to the DynoClient buildier object.

Hope this is helpful. I'm closing out the issue. Please let me know if you have any more questions. And please feel free to reopen.