New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sequences #4

Open
jberkus opened this Issue Dec 4, 2014 · 6 comments

Comments

Projects
None yet
4 participants
@jberkus
Contributor

jberkus commented Dec 4, 2014

pg_shard needs to support sequences and SERIAL/BIGSERIAL. The simplest way to do these for now is to have the sequence live on the master. Later we'll need to do something more complicated, but 9.5 will have machinery for that, I think.

@jasonmp85 jasonmp85 added the feature label Dec 4, 2014

@jasonmp85 jasonmp85 changed the title from Need sequence support to Support sequences Dec 4, 2014

@jasonmp85

This comment has been minimized.

Show comment
Hide comment
@jasonmp85

jasonmp85 Dec 4, 2014

Member

Having the sequence live on the master is essentially a must right now, as INSERT statements are commutable: conditions could arise wherein two racing INSERT commands result in different sequence values across replicas. Perhaps that's an argument for looking into logical replication, but the feature wasn't mature enough for us to investigate earlier in pg_shard's lifecycle.

Right now we eagerly evaluate any "constant expressions" during planning, e.g. we transform char_length('Jason') into the integer 5. Certain other functions could reasonably be supported by eagerly evaluating them on the master during planning, even things like rand or in your case, nextval. Since sequence values are never rolled back, our loose transactional semantics don't even get in the way here.

For the immediate future, we could expand the types of expressions we eagerly evaluate, or even offer a special eager function to allow users to force eager evaluation of functions on the master.

Member

jasonmp85 commented Dec 4, 2014

Having the sequence live on the master is essentially a must right now, as INSERT statements are commutable: conditions could arise wherein two racing INSERT commands result in different sequence values across replicas. Perhaps that's an argument for looking into logical replication, but the feature wasn't mature enough for us to investigate earlier in pg_shard's lifecycle.

Right now we eagerly evaluate any "constant expressions" during planning, e.g. we transform char_length('Jason') into the integer 5. Certain other functions could reasonably be supported by eagerly evaluating them on the master during planning, even things like rand or in your case, nextval. Since sequence values are never rolled back, our loose transactional semantics don't even get in the way here.

For the immediate future, we could expand the types of expressions we eagerly evaluate, or even offer a special eager function to allow users to force eager evaluation of functions on the master.

@jasonmp85

This comment has been minimized.

Show comment
Hide comment
@jasonmp85

jasonmp85 Dec 5, 2014

Member

Nothing stands out in my mind about special support for this kind of thing in 9.5… do you have a quick link to a pgsql-hackers thread on the topic? I'm a pretty heavy lurker in that list but don't remember anything about this.

Member

jasonmp85 commented Dec 5, 2014

Nothing stands out in my mind about special support for this kind of thing in 9.5… do you have a quick link to a pgsql-hackers thread on the topic? I'm a pretty heavy lurker in that list but don't remember anything about this.

@jberkus

This comment has been minimized.

Show comment
Hide comment
@jberkus

jberkus Dec 5, 2014

Contributor

Jason: there's some stuff buried in the Bi-directional replication code which allows replicas to pull "chunks" of sequences, in effect allowing for distributed sequences. However, this doesn't at all solve the consistency issue between shards.

Maybe we'd be better off implementing a UUID which is compatible with pg_shard instead, something which will be consistent yet unique across shards. Not quite sure how that would work, though.

Contributor

jberkus commented Dec 5, 2014

Jason: there's some stuff buried in the Bi-directional replication code which allows replicas to pull "chunks" of sequences, in effect allowing for distributed sequences. However, this doesn't at all solve the consistency issue between shards.

Maybe we'd be better off implementing a UUID which is compatible with pg_shard instead, something which will be consistent yet unique across shards. Not quite sure how that would work, though.

@jberkus

This comment has been minimized.

Show comment
Hide comment
@jberkus

jberkus Dec 5, 2014

Contributor

Maybe we should borrow code from here? https://github.com/OptionsHouse/shard_manager

Shaun has a fairly advanced distributed ID-generation widget. We may need to work out licensing issues though.

Contributor

jberkus commented Dec 5, 2014

Maybe we should borrow code from here? https://github.com/OptionsHouse/shard_manager

Shaun has a fairly advanced distributed ID-generation widget. We may need to work out licensing issues though.

@digi604

This comment has been minimized.

Show comment
Hide comment
@digi604

digi604 Jul 23, 2015

implementing this would be highly appriciated.

digi604 commented Jul 23, 2015

implementing this would be highly appriciated.

@rodo

This comment has been minimized.

Show comment
Hide comment
@rodo

rodo Aug 9, 2015

Support sequences is definitely a nice to have.
Thanks for pg_shard it's a huge improvement for postgresql.

rodo commented Aug 9, 2015

Support sequences is definitely a nice to have.
Thanks for pg_shard it's a huge improvement for postgresql.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment