Example code not working

[gonzajg]

Hi,

I've trying to use the library but couldn't make it work. I get the following exception:

2014-11-04 14:01:22 WARN TokenMapSupplierImpl:107 - Could not get json response for token topology [Connection to http://localhost:8080 refused]
2014-11-04 14:01:22 INFO HostConnectionPoolImpl:162 - Shutting down connection pool for host:Host [name=localhost, port=8102, dc: localrack, status: Up]
2014-11-04 14:01:22 WARN HostConnectionPoolImpl:407 - Failed to close connection for host: Host [name=localhost, port=8102, dc: localrack, status: Up] Unknown reply:
Exception in thread "main" com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [name=UNKNOWN, port=0, dc: null, status: Down], latency=0(0), attempts=0]Token not found for key hash: 3303027599
at com.netflix.dyno.connectionpool.impl.hash.BinarySearchTokenMapper.getToken(BinarySearchTokenMapper.java:73)
at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getPoolForOperation(TokenAwareSelection.java:85)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:125)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:114)
at com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl.executeWithFailover(ConnectionPoolImpl.java:299)
at com.netflix.dyno.jedis.DynoJedisClient.d_set(DynoJedisClient.java:946)
at com.netflix.dyno.jedis.DynoJedisClient.set(DynoJedisClient.java:941)
at testElastic.DynoTest.main(DynoTest.java:37)

Server Log:

src/dynomite -c conf/redis.yml
[Tue Nov 4 11:36:20 2014] dynomite.c:192 dynomite-0.1.19 built for Linux 3.13.0-39-generic x86_64 started on pid 18082
[Tue Nov 4 11:36:20 2014] dynomite.c:197 run, rabbit run / dig that hole, forget the sun / and when at last the work is done / don't sit down / it's time to dig another one
[Tue Nov 4 11:36:20 2014] dyn_stats.c:992 m 3 listening on '0.0.0.0:22222'
[Tue Nov 4 11:36:20 2014] dyn_proxy.c:211 p 7 listening on '127.0.0.1:8102' in redis pool 0 'dyn_o_mite' with 1 servers
[Tue Nov 4 11:36:20 2014] dyn_dnode_server.c:195 dyn: p 8 listening on '127.0.0.1:8101' in redis pool 0 'dyn_o_mite' with 1 servers
[Tue Nov 4 11:36:20 2014] dyn_dnode_server.c:286 dyn: accept on sd 10
[Tue Nov 4 11:36:20 2014] dyn_dnode_server.c:326 dyn: accepted c 10 on p 8 from '127.0.0.1:38887'
[Tue Nov 4 11:36:24 2014] dyn_proxy.c:341 accepted c 11 on p 7 from '127.0.0.1:48407'
[Tue Nov 4 11:36:24 2014] dyn_redis.c:1050 parsed unsupported command 'QUIT'
[Tue Nov 4 11:36:24 2014] dyn_core.c:284 close c 11 '127.0.0.1:48407' on event 00FF eof 0 done 0 rb 14 sb 0: Invalid argument
[Tue Nov 4 14:01:22 2014] dyn_proxy.c:341 accepted c 11 on p 7 from '127.0.0.1:53395'
[Tue Nov 4 14:01:22 2014] dyn_redis.c:1050 parsed unsupported command 'QUIT'
[Tue Nov 4 14:01:22 2014] dyn_core.c:284 close c 11 '127.0.0.1:53395' on event 00FF eof 0 done 0 rb 14 sb 0: Invalid argument

Server conf:

dyn_o_mite:
listen: 127.0.0.1:8102
env: network
datacenter: localdc
rack: localrack
dyn_listen: 127.0.0.1:8101
tokens: '3303027599'
servers:
- 127.0.0.1:6379:1
redis: true

Java Code:

import java.util.ArrayList;
import java.util.Collection;
import java.util.List;

import com.netflix.dyno.connectionpool.Host;
import com.netflix.dyno.connectionpool.Host.Status;
import com.netflix.dyno.connectionpool.HostSupplier;
import com.netflix.dyno.jedis.DynoJedisClient;

public class DynoTest {

public static void main(String[] args) {
    final HostSupplier customHostSupplier = new HostSupplier() {

        final List<Host> hosts = new ArrayList<Host>();

        @Override
        public Collection<Host> getHosts() {

            hosts.add(new Host("localhost", 8102, Status.Up).setRack("localrack"));

            return hosts;
        }
    };


    DynoJedisClient dynoClient = new DynoJedisClient.Builder()
            .withApplicationName("MY APP")
            .withDynomiteClusterName("local")
            .withHostSupplier(customHostSupplier).build();

    try{
        dynoClient.set("foo", "puneetTest");
        System.out.println("Value: " + dynoClient.get("foo"));

    } finally{
        dynoClient.stopClient();
    }
}

}

[opuneet]

@gonzajg I see why this is happening. My example was insufficient for the oss community.
Here is what's happening ...

When Dyno starts up, it needs the token map (topology) from the Dynomite servers.
Currently we have an internal mechanism inside Netflix where we can get this information from and we haven't open sourced that as of yet.

This is where you are getting it from
http://localhost:8080
and the class is TokenMapSupplierImpl.

But you can unblock yourself from this. You need to provide a simple TokenMapSupplier impl that just gives the tokens along with the host names to Dyno, and you will get past this error.

I'll start adding an example on how to do this. Thanks for your patience.

[opuneet]

[
{"token":"3051939411","hostname":"foo11" , "zone":"us-east-1c"},
{"token":"188627880"," hostname":"foo12" , "zone":"us-east-1c"},
{"token":"3051939411","hostname":"foo21" , "zone":"us-east-1d"},
{"token":"188627880"," hostname":"foo22" , "zone":"us-east-1d"},
{"token":"3051939411","hostname":"foo31" , "zone":"us-east-1e"},
{"token":"188627880"," hostname":"foo32" , "zone":"us-east-1e"}
]

Here is the example of simple json that TokenMapSupplierImpl uses for 6 nodes in 3 racks, where we have 2 nodes per rack, hence RF = 3.

You just need to add a dummy TokenMapSupplier implementation to the config object for Dyno that uses information about your dynomite servers along with their tokens.

I'm working on adding a class that will do this if you give it the right json. But in the meantime you can construct this yourself.
Hope this is helpful.

[gonzajg]

Thank you opuneet! I'll try this ASAP and get back

[gonzajg]

I got it to work using my own TokenMapSupplier.

BTW there are some limitations I found like Pipeline not supporting more than one key. Will keep investigating! Thanks for your help!

[opuneet]

Yes, I've documented that about Pipelining. Pipeline works well when all keys go to the same server.
Since Dynomite is distributed Redis, this functionality is hard to support.

There will be improvements to pipelining in the near future.

[opuneet]

@gonzajg thanks for your patience. I just uploaded a default class that will make it easier for you to implement your own.

Here is the wiki guide
https://github.com/Netflix/dyno/wiki/Getting-started-with-Redis-client#adding-in-your-own-tokenmapsupplier

[goru1984]

Hello Active Dyno Contributers:

@opuneet and @gonzajg and @ipapapa @quidryan ,

I am following same pattern for calling Dynomite and Redis running on an EC2 instance. I can use Redis CLI and can see Get and Set is working fine in EC2 which means Dynomite and Redis configuration is fine.

In Java Layer, To build the DynoJedisClient, I am providing Host Supplier and Token Map supplier as suggested above. I am verifying this through fetching ConnectionPool object from already built Dyno Client. All supplied values are coming properly. But i execute it i get the same error as @gonzajg got above.
Post debugging, i found that once it tries to create all configuration like Connection Pool, Selection Strategy..etc, everything comes fines except ConcurrentHashMap<Host, HostToken>. This map comes with null value hence there is no host token available error.

Need Immediate help:

Code Reference:

public class TestTokenMapSupplier extends AbstractTokenMapSupplier {

final String json = "[{\"token\":\"12345678\",\"hostname\":\"MyHost Name\",\"zone\":\"us-east-1c\"}]\"";


@Override
public String getTopologyJsonPayload(String hostname) {
    return json;
}

@Override
public String getTopologyJsonPayload(Set<Host> hosts){
    return json;
}

****************************************************************************************************


public class CustomHostSupplier implements HostSupplier {

final List<Host> hosts = new ArrayList<Host>();

@Override
public Collection<Host> getHosts() {

    hosts.add(new Host("myHost", 8102,
            Status.Up).setRack("rack"));
    return hosts;
}

---

Client Code:

// ConnectionPoolConfigurationImpl class to supply Token
dynoClient = new DynoJedisClient.Builder()
.withApplicationName("APP")
.withDynomiteClusterName("CLUSTER")
.withHostSupplier(customHostSupplierClass)
.withCPConfig(new ConnectionPoolConfigurationImpl("APP").withTokenSupplier(Object of TestTokenMapSupplier))
.build();
}

Error:
data::com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [name=UNKNOWN, port=0, dc: null, status: Down], latency=0(0), attempts=0]Token not found for key hash: 339455788
at com.netflix.dyno.connectionpool.impl.hash.BinarySearchTokenMapper.getToken(BinarySearchTokenMapper.java:68)
at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getTokenForKey(TokenAwareSelection.java:110)
at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getPoolForOperation(TokenAwareSelection.java:73)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:111)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:100)
at com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl.executeWithFailover(ConnectionPoolImpl.java:291)
at com.netflix.dyno.jedis.DynoJedisClient.d_get(DynoJedisClient.java:302)
at com.netflix.dyno.jedis.DynoJedisClient.get(DynoJedisClient.java:296)

[ipapapa]

Reopening the issue.

[jcacciatore]

@goru1984 I resurrected a simple working example of using a custom token supplier against redis running on the localhost. See CustomTokenSupplierExample.

The proper way to fix this, however, is not to rely on a sidecar for this information. Dynomite itself should publish this and we are actively working on that. Once that is the case I'll modify the Dyno client appropriately and you will not need to supply your own implementation.

Alternatively until then, you can easily change the configuration of the client to use a round-robin load balancing strategy if that is an option, which would also alleviate providing a custom token supplier implementation.

[goru1984]

@jcacciatore $ @ipapapa

Thanks for your quick turn around. I tried above example in my local, it works fine. but when i try it in an EC2 instance, it fails with below error. Please note, I am able to access Redis through Dynomite in EC2 instance using Redis-CLI so can safely say that configuration should be fine on EC2 side. Please help me with the resolution.

Also, do you have any ETA on releasing the Dynomite support as said in your previous comment?

Error

[main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - registered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [name=UNKNOWN, port=0, dc: null, status: Down], latency=0(0), attempts=0]Could not find host connection pool for key: ABC, hash: 899048765
at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getPoolForOperation(TokenAwareSelection.java:81)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:111)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:100)
at com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl.executeWithFailover(ConnectionPoolImpl.java:291)
at com.netflix.dyno.jedis.DynoJedisClient.d_set(DynoJedisClient.java:1226)
at com.netflix.dyno.jedis.DynoJedisClient.set(DynoJedisClient.java:1216)

[jcacciatore]

I'll get an ETA regarding the Dynomite support.

Can you post your full log with debug logging enabled for the com.netflix.dyno package ?

[ipapapa]

Closing this because of inactivity. We can always reopen it if needed.