New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pluggable storage backends (was Support for Consul K/V storage) #1957

Closed
Galmido opened this Issue Oct 22, 2014 · 119 comments

Comments

Projects
None yet
@Galmido

Galmido commented Oct 22, 2014

Consul is a self checking service discovery and dynamic DNS server which runs on all machines in our bare-metal datacenter. We use Consul to provide fast and reliable DNS resolving service, mainly for Kubernetes Pods addresses. Consul also provides a simple REST K/V storage system which is very similar to Etcd and we like to use it instead whole Etcd cluster.

Currently we use Consul and Etcd together, where Consul K/V is used by all our services, and Etcd is only used by Kubernetes. This forces us to maintain two independent datacenter-wide services which provides almost exactly the same functionality without any additional benefits.

Both services have very similar REST API and it shouldn't be difficult to provide support for Consul.

@stp-ip

This comment has been minimized.

Member

stp-ip commented Oct 22, 2014

I would probably favour this as I like the ideas of consul and the idea of interchangability.

@rboyer

This comment has been minimized.

rboyer commented Oct 22, 2014

I would also be in favor of this, but given how k8s uses etcd today this is really only viable after real TTL support is added to Consul: hashicorp/consul#172

@Galmido

This comment has been minimized.

Galmido commented Oct 22, 2014

Agree with that. Can we leave this thread open until Consul gets support for TTL?

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Oct 23, 2014

@lavalamp Someone would need to do a feature-by-feature analysis to see whether anything else we need now or the near future is missing from Consul, such as atomic compare and swap, write preconditions, a sequencer for resourceVersion, etc. We should at least document requirements somewhere, in case this comes up again.

@Usnarski Are you planning to do the port? We don't have the bandwidth at the moment.

I'm not sure what we'd do about testing, either, since it would expand the test matrix in a new dimension.

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Oct 23, 2014

We've toyed with taking the underlying concept of the consul store (raft + lmdb transactional tables and secondary indices) and using that to back kube, but haven't looked at the higher level kv store.

We don't really benefit from the etcd hierarchal key space in a meaningful way, and future operations like paging and secondary indices aren't easy within the current API. The raft log and efficient watch are definitely the characteristics that other stores beyond consul and etcd can't essily offer

@tilgovi

This comment has been minimized.

tilgovi commented Dec 21, 2014

hashicorp/consul#172 is closed as of 9 days ago.

@kcq

This comment has been minimized.

kcq commented Jan 3, 2015

+1.01 :)

@thomassun

This comment has been minimized.

thomassun commented Jan 14, 2015

@Galmido I'm very interesting in how you combine Kubernetes with Consul to offer pods address because I have just almost the same requirement. But I'm new to Consul, I wonder how Consul know the lifecycle of a pod and the address(say IP and hostPort) of it. Thanks.

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Jan 14, 2015

FWIW, we've also had a request for Zookeeper.

Things we want out of a k-v store (not all of which are currently delivered by etcd), off the top of my head:

  • Optimistic concurrency
  • Access to past versions of objects
  • Efficient reliable watch
  • Efficient recursive or other multi-object watch
  • Multi-object transactions
  • The ability to somehow build multiple indices
  • Pagination support
  • Access control
  • Low per-object memory footprint
  • Object TTLs (for events, logically deleted objects)
@rkettelerij

This comment has been minimized.

rkettelerij commented Feb 24, 2015

+1 for consul

@MrMMorris

This comment has been minimized.

MrMMorris commented Feb 27, 2015

big +1 for consul

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Feb 27, 2015

This will not be a priority for us in the foreseeable future (see our roadmap). However, we'd be happy to work with someone from the community on this, in perhaps a couple months after we simplify how Kubernetes interacts with the storage layer.

@lavalamp

This comment has been minimized.

Member

lavalamp commented Feb 27, 2015

@saad-ali is going to do some etcd benchmarking, which will give us information to determine the urgency of this issue.

@thockin

This comment has been minimized.

Member

thockin commented Feb 27, 2015

In terms of overall utility - the value of doing a second storage
implementation is that it will force us to keep our API honest. I would
love to put a bounty on someone doing a second impl, even if we don't have
time for it on the core team.

On Fri, Feb 27, 2015 at 2:57 PM, Daniel Smith notifications@github.com
wrote:

@saad-ali https://github.com/saad-ali is going to do some etcd
benchmarking, which will give us information to determine the urgency of
this issue.

Reply to this email directly or view it on GitHub
#1957 (comment)
.

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Feb 27, 2015

More specifically what I think we need to finish first:

  • Post status back #2726
  • Eliminate BoundPods #2483
  • Clean up registry etc. #1451 and make all objects use the generic etcd registry implementation
@jletroui

This comment has been minimized.

jletroui commented Mar 1, 2015

+1 for consul

1 similar comment
@mainframe

This comment has been minimized.

mainframe commented Mar 7, 2015

+1 for consul

@offlinehacker

This comment has been minimized.

offlinehacker commented Mar 17, 2015

+1 consul could be also used as replacement for skydns

@bkleef

This comment has been minimized.

bkleef commented Mar 24, 2015

+1

@sumanthy

This comment has been minimized.

sumanthy commented Mar 28, 2015

+1 consul

@rombob

This comment has been minimized.

rombob commented Mar 30, 2015

+1

@geeknam

This comment has been minimized.

geeknam commented Apr 8, 2015

+1 consul

k8s-merge-robot added a commit that referenced this issue Apr 26, 2016

Merge pull request #24061 from timothysc/kvstore
Automatic merge from submit-queue

Renaming generic registry store from etcd->kvstore

This is part of issue #17546, in preparation for multiple backend storage facilities, renaming the registry structs to reflect.  

xref: #1957
@trong

This comment has been minimized.

trong commented May 10, 2016

I do not know about Zookeeper, but Consul is really must have

@thockin

This comment has been minimized.

Member

thockin commented May 10, 2016

PLEASE do not +1 this thread - use the thumbs-up icon on the first message instead. Every +1 generates email.

@steakknife

This comment has been minimized.

steakknife commented Jul 25, 2016

ZK is worth considering for ginormous enterprise / JVM / Hadoop shops.

@DanyC97

This comment has been minimized.

DanyC97 commented Aug 8, 2016

@thockin who is the owner of this PR ? would be possible to increase the priority from P3 ? is it reasonable to expect it in say 1.4 or maybe 1.5 time frame ?

Much thanks !

@thockin

This comment has been minimized.

Member

thockin commented Aug 9, 2016

This is an issue, not a PR. It is P3 because it is not critical to any launch plans, and because none of the core maintainers (AFAIK) are working on it. Boosting the priority is not going to speed it up - we need people to implement it.

@bgrant0607 bgrant0607 removed the help-wanted label Aug 30, 2016

@blalor

This comment has been minimized.

blalor commented Oct 7, 2016

Implemented by #31622?

@davidopp davidopp referenced this issue Jan 24, 2017

Closed

Pluggable storage backends #172

0 of 23 tasks complete
@docmerlin

This comment has been minimized.

docmerlin commented Jan 31, 2017

+1 consul

@davidopp

This comment has been minimized.

Member

davidopp commented Feb 19, 2017

CoreOS benchmarked etcd, Consul, and ZK. (I was not involved, just saw an announcement and thought folks here might be interested.)

https://coreos.com/blog/performance-of-etcd.html

@jedsmith

This comment has been minimized.

jedsmith commented Feb 20, 2017

@davidopp That benchmark somewhat hints that etcd is evolving into Kubernetes-specific software, which I suppose is unsurprising.

@davidopp

This comment has been minimized.

Member

davidopp commented Feb 20, 2017

Seriously I wasn't trying to troll. :) I think it would be great if Kubernetes could support multiple key-value stores.

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Feb 20, 2017

I actually don't think etcd is evolving into Kubernetes specific software - we just have continually refined it at fairly large scale, and the etcd team has been phenomenal at tuning etcd to eliminate scalability gaps.

I do think the benchmark numbers mirror our own experiences in production, which it is an excellent high performance and reliable K/V store for O(100k-5m) small keys that also provides the right APIs to run Kube's deliberately simple resource structures.

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Jun 14, 2017

I commented here:
#31622 (comment)

@bgrant0607 bgrant0607 changed the title from Support for Consul K/V storage to Pluggable storage backends (was Support for Consul K/V storage) Jun 14, 2017

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Sep 7, 2017

Updated viewpoint:
#31622 (comment)

See also #46351.

We are not ready to do this in the near future, so I'm actually going to close this for now.

@bgrant0607 bgrant0607 closed this Sep 7, 2017

@binq

This comment has been minimized.

binq commented Sep 7, 2017

wow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment