New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow changing multiple key values atomically #886

Closed
mikeys opened this Issue Apr 25, 2015 · 8 comments

Comments

Projects
None yet
8 participants
@mikeys

mikeys commented Apr 25, 2015

We would like to use Consul as our central configuration management tool and as such, we would've expect to change some of the configuration keys atomically, at once.
This will make sure we don't experience bad flows occurring because one flag was set correctly to the expected value but the other one wasn't changed on time.
Is there any pattern which currently permits doing that?

@ryanuber

This comment has been minimized.

Show comment
Hide comment
@ryanuber

ryanuber Apr 29, 2015

Member

At this time Consul doesn't support atomically writing more than one key. You could acquire separate locks on a few different keys and only release once all were acquired/written, but this is by no means atomic, and watches would not be coalesced. It's an interesting idea though, so I'll mark this as thinking for now.

Member

ryanuber commented Apr 29, 2015

At this time Consul doesn't support atomically writing more than one key. You could acquire separate locks on a few different keys and only release once all were acquired/written, but this is by no means atomic, and watches would not be coalesced. It's an interesting idea though, so I'll mark this as thinking for now.

@ryanuber ryanuber added the thinking label Apr 29, 2015

@mikeys

This comment has been minimized.

Show comment
Hide comment
@mikeys

mikeys Apr 29, 2015

@ryanuber Thanks, this feature will be much appreciated.

For anyone's interested: atomicity can also be achieved in the watcher handler level with an implementation close to the following:

Consider that there are N keys we would like to update atomically, at once:
Create each key value in the following format:
<value1>:1:N:<uuid>
<value2>:2:N:<uuid>
...
...
<valueN>:3:N:<uuid>

Where <valueN> is the actual value of the Nth key in the bulk and <uuid> is a unique identifier we assign to the bulk to differentiate it from other bulks.

The watcher handler keeps a map of uuids to a list of KVs.
When a KV update occurs, the handler peeks at it's value and seeks for the format mentioned above.
In case there's a match: it will add the KV to list of KVs which is mapped by the 'uuid'.
It will then check if this list reached the size of N (the handler knows what N is since its part of the above format as well).
If the list reached the bulk size then we can safely execute some action (in our use case it was a curl command which sends the the entire KV list in one bulk to a web server endpoint).

mikeys commented Apr 29, 2015

@ryanuber Thanks, this feature will be much appreciated.

For anyone's interested: atomicity can also be achieved in the watcher handler level with an implementation close to the following:

Consider that there are N keys we would like to update atomically, at once:
Create each key value in the following format:
<value1>:1:N:<uuid>
<value2>:2:N:<uuid>
...
...
<valueN>:3:N:<uuid>

Where <valueN> is the actual value of the Nth key in the bulk and <uuid> is a unique identifier we assign to the bulk to differentiate it from other bulks.

The watcher handler keeps a map of uuids to a list of KVs.
When a KV update occurs, the handler peeks at it's value and seeks for the format mentioned above.
In case there's a match: it will add the KV to list of KVs which is mapped by the 'uuid'.
It will then check if this list reached the size of N (the handler knows what N is since its part of the above format as well).
If the list reached the bulk size then we can safely execute some action (in our use case it was a curl command which sends the the entire KV list in one bulk to a web server endpoint).

@u6f6o

This comment has been minimized.

Show comment
Hide comment
@u6f6o

u6f6o Jun 30, 2015

We are evaluating consul as a central configuration management system as well. For us it would be also important to update several keys atomically at once. So +1 for this feature.

u6f6o commented Jun 30, 2015

We are evaluating consul as a central configuration management system as well. For us it would be also important to update several keys atomically at once. So +1 for this feature.

@RyanGordon

This comment has been minimized.

Show comment
Hide comment
@RyanGordon

RyanGordon Oct 19, 2015

We would love to see this implemented. I don't think it would necessarily need to be atomic. The basic idea is that it would be nice to have a way to update multiple key's that would affect the same file - in one go - and have the file be updated by consul only once.

RyanGordon commented Oct 19, 2015

We would love to see this implemented. I don't think it would necessarily need to be atomic. The basic idea is that it would be nice to have a way to update multiple key's that would affect the same file - in one go - and have the file be updated by consul only once.

@zachmarshall

This comment has been minimized.

Show comment
Hide comment
@zachmarshall

zachmarshall Dec 4, 2015

We would like to see this feature as well. For us, the updates need not technically be atomic, but we'd like to see watches/events triggered or delivered only once for the batch of updates.

zachmarshall commented Dec 4, 2015

We would like to see this feature as well. For us, the updates need not technically be atomic, but we'd like to see watches/events triggered or delivered only once for the batch of updates.

@shahar-davidson

This comment has been minimized.

Show comment
Hide comment
@shahar-davidson

shahar-davidson Dec 17, 2015

+1 for this.
We need this atomic multi-key update ability as well.

shahar-davidson commented Dec 17, 2015

+1 for this.
We need this atomic multi-key update ability as well.

@123BLiN

This comment has been minimized.

Show comment
Hide comment
@123BLiN

123BLiN May 7, 2016

+1, we need something like bulk keys changes transaction possibility as well.
Lets also add some versioning if it is possible ;-) and we are done)
we now use git2consul ref key to watch on changes if someone needs something like workaround here.

123BLiN commented May 7, 2016

+1, we need something like bulk keys changes transaction possibility as well.
Lets also add some versioning if it is possible ;-) and we are done)
we now use git2consul ref key to watch on changes if someone needs something like workaround here.

@slackpad

This comment has been minimized.

Show comment
Hide comment
@slackpad

slackpad Jun 1, 2016

Contributor

This was implemented in #2028!

Contributor

slackpad commented Jun 1, 2016

This was implemented in #2028!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment