Implement clustering, loop over machines until one works

[josephglanville]

Hi,

This is a first attempt at implementing some form of cluster support.
It's very naive at this point but I wanted to get this out there so we could collaborate on a better approach.

[josephglanville]

2 things that are obvious but not yet solved here are behaviour on redirects and a means of not breaking the API whilst introducing cluster support.

[ranjib]

@josephglanville i would love to add cluster awareness, but i am inclined to design it more like the go-etcd client, where a retry logic is exercised upon request failures. Users can pass their own/custom logic, in absence of which the client will fallback to a default retry logic. https://github.com/coreos/go-etcd/blob/master/etcd/requests.go#L291
i'd love to collaborate on getting this done, what do you think?

[josephglanville]

@ranjib Yes this looks like a good approach.

We could easily have a default block that is execute in the case of a connection error and allow the user to pass in their own method with something like Etcd.client(on_request_fail: &retry_logic)

[josephglanville]

I think we can maintain compatibility it with the {host: string, port: int parameters for now but I think the native interface should move to servers: [URI, ... ]. Using full URI is desirable as that matches output of Client#machines, which we should use to discover cluster members on connect.

[Soulou]

I'm also interested by this feature, is there something new about it?

[josephglanville]

I haven't had time to approach this yet, I am using a naive implementation atm.

[tpett]

This isn't quite the same thing, but I have spiked some code that will follow redirects. It fixed some issues I was having in a clustered environment: master...tpett:retryable

It's not currently tested because I'm not sure how to reproduce it in the test environment otherwise I would create a pull request for it.

[ranjib]

@tpett I was thinking of using the retriable rubygem for this.

[tpett]

In the case of a Net::HTTPRetriableError I believe you are generally dealing with a redirect. In that case there isn't really a need for a back off. You just need to update the location based on the response header.

Now in the case of some transient network / server error Retriable would seem like a great option. What I was seeing was a server I had configured was no longer the leader and it was redirecting my update requests to the leader and the library wasn't handling the redirects for me.

If you aren't opposed to the dependency, I would probably use something like Faraday. It is a wrapper over net/http (or pretty much any other HTTP library) that provides both retry and redirect following through its middleware system.

• Retrying
• Follow Redirect

[ranjib]

@tpett faraday is fine. in fact may be it will better, as faraday will allow lot more pluggable middlewares. i think i can use some of faraday's niceties on jruby as well :-)

[tpett]

@ranjib yeah the middleware system is my favorite part of Faraday. And it looks like JRuby is officially supported in the readme!

[luispollo]

Hi all, has there been any progress on this?

@ranjib, I was also wondering if you had considered adding support for DNS-based cluster discovery? I'd like to contribute either way if I can.

[ranjib]

@luispollo I am heavily engaged in golang based client for etcd/consul now, hence this library is lacking some love. I am happy to add @tpett of any of you as collaborator and setup necessary rubygem permission so that these issues can get addressed ,
let me know how you all think

[luispollo]

Thanks @ranjib. I'm trying to determine what's the most efficient way to make progress. My goal is to have a production-grade etcd client that supports cluster discovery and automatic fail-over. Just out of curiosity, have you ever considered joining forces with iconara/etcd-rb? There hasn't been any activity there for a long time, but that client seemed to address some of the limitations we're talking about here. On the other hand, it lacked V2 API support, which your client has. Thoughts?

[ranjib]

@luispollo that is a very good idea. I'd love to hear his opinion, and contribute any way I can.
meanwhile, it will be awesome to jot down the feature we are looking for. I can spend some time to consolidate those and see if we can plan a 1.0 release.