Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pluggable storage backends (was Support for Consul K/V storage) #1957

Closed
Galmido opened this issue Oct 22, 2014 · 123 comments
Closed

Pluggable storage backends (was Support for Consul K/V storage) #1957

Galmido opened this issue Oct 22, 2014 · 123 comments

Comments

@Galmido
Copy link

@Galmido Galmido commented Oct 22, 2014

Consul is a self checking service discovery and dynamic DNS server which runs on all machines in our bare-metal datacenter. We use Consul to provide fast and reliable DNS resolving service, mainly for Kubernetes Pods addresses. Consul also provides a simple REST K/V storage system which is very similar to Etcd and we like to use it instead whole Etcd cluster.

Currently we use Consul and Etcd together, where Consul K/V is used by all our services, and Etcd is only used by Kubernetes. This forces us to maintain two independent datacenter-wide services which provides almost exactly the same functionality without any additional benefits.

Both services have very similar REST API and it shouldn't be difficult to provide support for Consul.

@stp-ip
Copy link
Member

@stp-ip stp-ip commented Oct 22, 2014

I would probably favour this as I like the ideas of consul and the idea of interchangability.

Loading

@rboyer
Copy link

@rboyer rboyer commented Oct 22, 2014

I would also be in favor of this, but given how k8s uses etcd today this is really only viable after real TTL support is added to Consul: hashicorp/consul#172

Loading

@Galmido
Copy link
Author

@Galmido Galmido commented Oct 22, 2014

Agree with that. Can we leave this thread open until Consul gets support for TTL?

Loading

@bgrant0607
Copy link
Member

@bgrant0607 bgrant0607 commented Oct 23, 2014

@lavalamp Someone would need to do a feature-by-feature analysis to see whether anything else we need now or the near future is missing from Consul, such as atomic compare and swap, write preconditions, a sequencer for resourceVersion, etc. We should at least document requirements somewhere, in case this comes up again.

@Usnarski Are you planning to do the port? We don't have the bandwidth at the moment.

I'm not sure what we'd do about testing, either, since it would expand the test matrix in a new dimension.

Loading

@smarterclayton
Copy link
Contributor

@smarterclayton smarterclayton commented Oct 23, 2014

We've toyed with taking the underlying concept of the consul store (raft + lmdb transactional tables and secondary indices) and using that to back kube, but haven't looked at the higher level kv store.

We don't really benefit from the etcd hierarchal key space in a meaningful way, and future operations like paging and secondary indices aren't easy within the current API. The raft log and efficient watch are definitely the characteristics that other stores beyond consul and etcd can't essily offer

Loading

@tilgovi
Copy link

@tilgovi tilgovi commented Dec 21, 2014

hashicorp/consul#172 is closed as of 9 days ago.

Loading

@kcq
Copy link

@kcq kcq commented Jan 3, 2015

+1.01 :)

Loading

@thomassun
Copy link

@thomassun thomassun commented Jan 14, 2015

@Galmido I'm very interesting in how you combine Kubernetes with Consul to offer pods address because I have just almost the same requirement. But I'm new to Consul, I wonder how Consul know the lifecycle of a pod and the address(say IP and hostPort) of it. Thanks.

Loading

@bgrant0607
Copy link
Member

@bgrant0607 bgrant0607 commented Jan 14, 2015

FWIW, we've also had a request for Zookeeper.

Things we want out of a k-v store (not all of which are currently delivered by etcd), off the top of my head:

  • Optimistic concurrency
  • Access to past versions of objects
  • Efficient reliable watch
  • Efficient recursive or other multi-object watch
  • Multi-object transactions
  • The ability to somehow build multiple indices
  • Pagination support
  • Access control
  • Low per-object memory footprint
  • Object TTLs (for events, logically deleted objects)

Loading

@rkettelerij
Copy link

@rkettelerij rkettelerij commented Feb 24, 2015

+1 for consul

Loading

@MrMMorris
Copy link

@MrMMorris MrMMorris commented Feb 27, 2015

big +1 for consul

Loading

@bgrant0607
Copy link
Member

@bgrant0607 bgrant0607 commented Feb 27, 2015

This will not be a priority for us in the foreseeable future (see our roadmap). However, we'd be happy to work with someone from the community on this, in perhaps a couple months after we simplify how Kubernetes interacts with the storage layer.

Loading

@lavalamp
Copy link
Member

@lavalamp lavalamp commented Feb 27, 2015

@saad-ali is going to do some etcd benchmarking, which will give us information to determine the urgency of this issue.

Loading

@thockin
Copy link
Member

@thockin thockin commented Feb 27, 2015

In terms of overall utility - the value of doing a second storage
implementation is that it will force us to keep our API honest. I would
love to put a bounty on someone doing a second impl, even if we don't have
time for it on the core team.

On Fri, Feb 27, 2015 at 2:57 PM, Daniel Smith notifications@github.com
wrote:

@saad-ali https://github.com/saad-ali is going to do some etcd
benchmarking, which will give us information to determine the urgency of
this issue.

Reply to this email directly or view it on GitHub
#1957 (comment)
.

Loading

@bgrant0607
Copy link
Member

@bgrant0607 bgrant0607 commented Feb 27, 2015

More specifically what I think we need to finish first:

  • Post status back #2726
  • Eliminate BoundPods #2483
  • Clean up registry etc. #1451 and make all objects use the generic etcd registry implementation

Loading

@jletroui
Copy link

@jletroui jletroui commented Mar 1, 2015

+1 for consul

Loading

1 similar comment
@mainframe
Copy link

@mainframe mainframe commented Mar 7, 2015

+1 for consul

Loading

@offlinehacker
Copy link

@offlinehacker offlinehacker commented Mar 17, 2015

+1 consul could be also used as replacement for skydns

Loading

@bkleef
Copy link

@bkleef bkleef commented Mar 24, 2015

+1

Loading

@sumanthy
Copy link

@sumanthy sumanthy commented Mar 28, 2015

+1 consul

Loading

@rombob
Copy link

@rombob rombob commented Mar 30, 2015

+1

Loading

@geeknam
Copy link

@geeknam geeknam commented Apr 8, 2015

+1 consul

Loading

@steakknife
Copy link

@steakknife steakknife commented Jul 25, 2016

ZK is worth considering for ginormous enterprise / JVM / Hadoop shops.

Loading

@DanyC97
Copy link

@DanyC97 DanyC97 commented Aug 8, 2016

@thockin who is the owner of this PR ? would be possible to increase the priority from P3 ? is it reasonable to expect it in say 1.4 or maybe 1.5 time frame ?

Much thanks !

Loading

@thockin
Copy link
Member

@thockin thockin commented Aug 9, 2016

This is an issue, not a PR. It is P3 because it is not critical to any launch plans, and because none of the core maintainers (AFAIK) are working on it. Boosting the priority is not going to speed it up - we need people to implement it.

Loading

@blalor
Copy link

@blalor blalor commented Oct 7, 2016

Implemented by #31622?

Loading

@docmerlin
Copy link

@docmerlin docmerlin commented Jan 31, 2017

+1 consul

Loading

@davidopp
Copy link
Member

@davidopp davidopp commented Feb 19, 2017

CoreOS benchmarked etcd, Consul, and ZK. (I was not involved, just saw an announcement and thought folks here might be interested.)

https://coreos.com/blog/performance-of-etcd.html

Loading

@jedsmith
Copy link

@jedsmith jedsmith commented Feb 20, 2017

@davidopp That benchmark somewhat hints that etcd is evolving into Kubernetes-specific software, which I suppose is unsurprising.

Loading

@davidopp
Copy link
Member

@davidopp davidopp commented Feb 20, 2017

Seriously I wasn't trying to troll. :) I think it would be great if Kubernetes could support multiple key-value stores.

Loading

@smarterclayton
Copy link
Contributor

@smarterclayton smarterclayton commented Feb 20, 2017

I actually don't think etcd is evolving into Kubernetes specific software - we just have continually refined it at fairly large scale, and the etcd team has been phenomenal at tuning etcd to eliminate scalability gaps.

I do think the benchmark numbers mirror our own experiences in production, which it is an excellent high performance and reliable K/V store for O(100k-5m) small keys that also provides the right APIs to run Kube's deliberately simple resource structures.

Loading

@bgrant0607
Copy link
Member

@bgrant0607 bgrant0607 commented Jun 14, 2017

I commented here:
#31622 (comment)

Loading

@bgrant0607 bgrant0607 changed the title Support for Consul K/V storage Pluggable storage backends (was Support for Consul K/V storage) Jun 14, 2017
@bgrant0607
Copy link
Member

@bgrant0607 bgrant0607 commented Sep 7, 2017

Updated viewpoint:
#31622 (comment)

See also #46351.

We are not ready to do this in the near future, so I'm actually going to close this for now.

Loading

@bgrant0607 bgrant0607 closed this Sep 7, 2017
@binq
Copy link

@binq binq commented Sep 7, 2017

wow

Loading

@cornfeedhobo
Copy link

@cornfeedhobo cornfeedhobo commented Dec 5, 2020

I've been checking in on this issue, and it's related issues and PRs, for 6 years.

I guess I'll see you all in 2024 at my next check in! 😂

xkcd - time cost

Loading

@o6uoq
Copy link

@o6uoq o6uoq commented Dec 6, 2020

@cornfeedhobo welcome to open source!

Loading

@yuriy-yarosh
Copy link

@yuriy-yarosh yuriy-yarosh commented Jun 20, 2021

It takes forever... I think it would be easier to develop a etcd-consul bridge instead of getting through all the CNCF bureaucracy.

Loading

@lavalamp
Copy link
Member

@lavalamp lavalamp commented Jun 21, 2021

The last few comments here seem to have misunderstood: it's not the case that we can't decide what to do, it is the case that we have decided not to expand to other storage backends, and every time it comes up we've determined that that's still the right thing for the project, and I don't really see that changing.

Fortunately you don't have to wait for us to change our minds, e.g. check out: https://github.com/k3s-io/kine

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet