New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support setting multiple keys in one atomic commit #860

Closed
tristanz opened this Issue Jun 18, 2014 · 18 comments

Comments

Projects
None yet
@tristanz

tristanz commented Jun 18, 2014

Zookeeper supports updating multiple nodes in one atomic commit. This can be very useful.

http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#multi(java.lang.Iterable)

@klizhentas

This comment has been minimized.

Show comment
Hide comment
@klizhentas

klizhentas Jul 13, 2014

+1 This would be very helpful for us as well.

klizhentas commented Jul 13, 2014

+1 This would be very helpful for us as well.

@mauricioabreu

This comment has been minimized.

Show comment
Hide comment
@mauricioabreu

mauricioabreu Aug 8, 2014

Contributor

+1

Contributor

mauricioabreu commented Aug 8, 2014

+1

@victorpoluceno

This comment has been minimized.

Show comment
Hide comment

victorpoluceno commented Aug 8, 2014

+1

@sukrit007

This comment has been minimized.

Show comment
Hide comment
@sukrit007

sukrit007 commented Aug 13, 2014

+1

@yichengq yichengq added the feature label Aug 28, 2014

@ytjohn

This comment has been minimized.

Show comment
Hide comment
@ytjohn

ytjohn commented Oct 17, 2014

+1

@xiang90 xiang90 added this to the v0.6.0maybe milestone Oct 25, 2014

@ehames

This comment has been minimized.

Show comment
Hide comment
@ehames

ehames commented Oct 30, 2014

+1

@sachja

This comment has been minimized.

Show comment
Hide comment
@sachja

sachja Nov 21, 2014

+1. This would be very useful for our use case.

sachja commented Nov 21, 2014

+1. This would be very useful for our use case.

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Dec 11, 2014

Without multi-object transactions, it's harder to write services using etcd as a key-value store. There are many cases where multiple objects need to be kept consistent, and two-phase commit would significantly complicate recovery. Examples:

  • Multiple keys/indices corresponding to the same data (e.g., name and UID)
  • Operations that bi-directionally pair objects (e.g., pods and nodes or services and IP addresses in Kubernetes)
  • API calls that generate multiple objects

This is why Omega didn't use Chubby for its key-value store.

Are there concrete plans to support multi-object transactions?

bgrant0607 commented Dec 11, 2014

Without multi-object transactions, it's harder to write services using etcd as a key-value store. There are many cases where multiple objects need to be kept consistent, and two-phase commit would significantly complicate recovery. Examples:

  • Multiple keys/indices corresponding to the same data (e.g., name and UID)
  • Operations that bi-directionally pair objects (e.g., pods and nodes or services and IP addresses in Kubernetes)
  • API calls that generate multiple objects

This is why Omega didn't use Chubby for its key-value store.

Are there concrete plans to support multi-object transactions?

@xiang90

This comment has been minimized.

Show comment
Hide comment
@xiang90

xiang90 Dec 11, 2014

Contributor

@bgrant0607 We plan to support distributed transaction by MVCC. We plan to support batch requests too (we do not abort the batch if there is a failure in the middle).

Contributor

xiang90 commented Dec 11, 2014

@bgrant0607 We plan to support distributed transaction by MVCC. We plan to support batch requests too (we do not abort the batch if there is a failure in the middle).

@xiang90

This comment has been minimized.

Show comment
Hide comment
@xiang90

xiang90 Dec 11, 2014

Contributor

@bgrant0607 Update: after talked with @barakmich, we decide to support transaction (all or nothing) in the new API.

Contributor

xiang90 commented Dec 11, 2014

@bgrant0607 Update: after talked with @barakmich, we decide to support transaction (all or nothing) in the new API.

@xiang90 xiang90 modified the milestones: v0.6.0, v0.6.0maybe Dec 15, 2014

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Dec 17, 2014

Thanks. We'd be interested in taking a look when you have a draft ready.

/cc @lavalamp

bgrant0607 commented Dec 17, 2014

Thanks. We'd be interested in taking a look when you have a draft ready.

/cc @lavalamp

@yichengq

This comment has been minimized.

Show comment
Hide comment
@yichengq

yichengq Jan 14, 2015

Contributor

@bgrant0607
The design of new kv api is in progress and we have draft thoughts and prototype around. We are actively working on it now, and plan to publish it about two weeks later.
Here are some rough ideas about new api and how it can solve k8s requirements(we get it from k8s issue):

  • Support multiple versioned key. Each key maintains its local version, which increments by 1 whenever it changes. This helps to solve optimistic concurrency, and efficient reliable watch.
  • Support client-driven compaction. History is kept in the cluster until it is compacted. It ensures that client can access to past versions of objects.
  • Support multi-object streaming watch. After connecting to the server, client can watch on multiple objects, and will be notified about all changes of them until the connection is closed.
  • Support transaction.
  • Only keep hot keys in memory, and save others into disk. This largely increases the capacity of the store.
  • Support more efficient TTL mechanism. It is supposed to hold million-level ttl keys for one cluster.

For acl, it will be added in the proxy.
Can you please explain what is "Pagination support"? or point me to a reference?

Contributor

yichengq commented Jan 14, 2015

@bgrant0607
The design of new kv api is in progress and we have draft thoughts and prototype around. We are actively working on it now, and plan to publish it about two weeks later.
Here are some rough ideas about new api and how it can solve k8s requirements(we get it from k8s issue):

  • Support multiple versioned key. Each key maintains its local version, which increments by 1 whenever it changes. This helps to solve optimistic concurrency, and efficient reliable watch.
  • Support client-driven compaction. History is kept in the cluster until it is compacted. It ensures that client can access to past versions of objects.
  • Support multi-object streaming watch. After connecting to the server, client can watch on multiple objects, and will be notified about all changes of them until the connection is closed.
  • Support transaction.
  • Only keep hot keys in memory, and save others into disk. This largely increases the capacity of the store.
  • Support more efficient TTL mechanism. It is supposed to hold million-level ttl keys for one cluster.

For acl, it will be added in the proxy.
Can you please explain what is "Pagination support"? or point me to a reference?

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Jan 14, 2015

Great!

FYI, our high-level feature wishlist (which maybe you saw) is here:
kubernetes/kubernetes#1957 (comment)

"Pagination" is some way to break up large responses into multiple responses. Chunked HTTP would be a simple example, but typically multiple requests are necessary in REST APIs. For instance:
http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api#pagination
https://blog.apigee.com/detail/restful_api_design_can_your_api_give_developers_just_the_information
https://cloud.google.com/appengine/docs/python/datastore/queries#Python_Query_cursors

We use recursive GETs to retrieve lists of objects in a subtree. If there were, say, a hundred thousand objects in a subtree, that shouldn't be returned in one enormous request or list. Internally, we use streaming RPCs for such cases, but pagination is more common in REST APIs. Search for "rest api pagination" and you'll find loads of examples: github, twitter, newrelic, ...

/cc @lavalamp @smarterclayton

bgrant0607 commented Jan 14, 2015

Great!

FYI, our high-level feature wishlist (which maybe you saw) is here:
kubernetes/kubernetes#1957 (comment)

"Pagination" is some way to break up large responses into multiple responses. Chunked HTTP would be a simple example, but typically multiple requests are necessary in REST APIs. For instance:
http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api#pagination
https://blog.apigee.com/detail/restful_api_design_can_your_api_give_developers_just_the_information
https://cloud.google.com/appengine/docs/python/datastore/queries#Python_Query_cursors

We use recursive GETs to retrieve lists of objects in a subtree. If there were, say, a hundred thousand objects in a subtree, that shouldn't be returned in one enormous request or list. Internally, we use streaming RPCs for such cases, but pagination is more common in REST APIs. Search for "rest api pagination" and you'll find loads of examples: github, twitter, newrelic, ...

/cc @lavalamp @smarterclayton

@xiang90

This comment has been minimized.

Show comment
Hide comment
@xiang90

xiang90 Jan 14, 2015

Contributor

@bgrant0607 Thanks!
Actually we have the plan to use the range header in http to achieve the goal. internally etcd keep all versions of keys, so it is simple to do.

Contributor

xiang90 commented Jan 14, 2015

@bgrant0607 Thanks!
Actually we have the plan to use the range header in http to achieve the goal. internally etcd keep all versions of keys, so it is simple to do.

@sckott

This comment has been minimized.

Show comment
Hide comment
@sckott

sckott Apr 8, 2015

Contributor

+1

Contributor

sckott commented Apr 8, 2015

+1

@xiang90

This comment has been minimized.

Show comment
Hide comment
@hot13399

This comment has been minimized.

Show comment
Hide comment
@hot13399

hot13399 Mar 11, 2016

Hi,

does etcd support range now? I want to do pagination through it.

Thanks,
Jerry

hot13399 commented Mar 11, 2016

Hi,

does etcd support range now? I want to do pagination through it.

Thanks,
Jerry

@tejinderss

This comment has been minimized.

Show comment
Hide comment
@tejinderss

tejinderss Jun 27, 2018

Hey how can i achieve this? Should i use transaction put multiple keys without using any precondition ? And does range provide setting the keys ? Also can I use range for fetching the multiple keys under same namespace. I have a use case where i need to store multiple keys under same name corresponding to micro service. How can i do that ?

tejinderss commented Jun 27, 2018

Hey how can i achieve this? Should i use transaction put multiple keys without using any precondition ? And does range provide setting the keys ? Also can I use range for fetching the multiple keys under same namespace. I have a use case where i need to store multiple keys under same name corresponding to micro service. How can i do that ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment