Add postgres-specific id generator #26

4word · 2016-05-03T02:12:33Z

Requiring an external-to-the-db id generator is not an option for some use cases. This new field and related migration utilities allow for shard-safe (up to 8191 shards) inserts with guaranteed unique IDs (up to 1024 inserts per millisecond on an individual shard). The ID will return like a standard Django auto field ID. In the future, a given ID could be deconstructed to get the time it was inserted and the shard id (useful for the router to potentially know which shard to grab something from without needing help from the using() statement).

… django updates to the base method rather than returning on their own

… argument that stores the field as an attribute and sets the default manager for the model to ShardManager

…del to get the correct shard based on the arguments passed to filter,create,get,etc. The default behavior is unchanged so existing models do not need any changes

…does not know what shard it should be on but there in a sharded by field defined

…ty on each primary shard. Added and updated unit tests to check for valid shard ids.

…with a too large for sqlite error

…-safe IDs. Added utils for migrations. Updated model_config decorator to enforce an ID strategy if the postgres shard id field does not exist on the model

4word · 2016-05-03T02:14:26Z

Note that much of this code was ported over from a Django 1.4 and python 2.7 deployment and I am not exactly expecting unit tests to pass the first time (although I updated as much as I could think of and find). More commits will come to fix any failures.

…ests.

…generator field is valid

4word · 2016-05-03T05:27:18Z

This is ready for more scrutinizing review. Docs need to be updated but functionality is available and tested for both database and django level usage.

… wrong values on accident

… the base model does is set the manager. Unfortunately, this cannot be done in the decorator for some reason (I cannot figure out why, it simply doesnt work correctly)

…re instantiating the object with them, just in case the parent class messes with the kwargs later

… using())

JBKahn · 2016-05-03T23:45:10Z

django_sharding_library/fields.py

+        if not db_alias:
+            raise EnvironmentError("A pre-migration receiver did not receive a database alias. "
+                                   "Perhaps your app is not registered correctly?")
+        if settings.DATABASES[db_alias]['ENGINE'] == Backends.POSTGRES:


since this only supports postgres, somewhere (maybe here?) we should raise an exception if you try to use it and you're not using postgres.

Hmm. The problem is that the library allows arbitrary backends within the same shard group. Should I just loop through the entire shard group and make sure all the backends are postgres?

Yup, you can do it either for all sharded databases or make it a shard
group level setting?

On Tue, May 3, 2016, 8:59 PM Titus Peterson notifications@github.com
wrote:

In django_sharding_library/fields.py
#26 (comment):

if not hasattr(settings, 'SHARD_EPOCH'):

raise ValueError("PostgresShardGeneratedIDField requires a SHARD_EPOCH to be defined in your settings file.")

if connection.vendor == PostgresDatabaseWrapper.vendor:

return "bigint DEFAULT next_sharded_id()"

else:

return super(PostgresShardGeneratedIDField, self).db_type(connection)

@staticmethod

def migration_receiver(_args, *_kwargs):

sequence_name = "global_id_sequence"

db_alias = kwargs.get('using')

if not db_alias:

raise EnvironmentError("A pre-migration receiver did not receive a database alias. "

"Perhaps your app is not registered correctly?")

if settings.DATABASES[db_alias]['ENGINE'] == Backends.POSTGRES:

Hmm. The problem is that the library allows arbitrary backends within the
same shard group. Should I just loop through the entire shard group and
make sure all the backends are postgres?

—
You are receiving this because you commented.

Reply to this email directly or view it on GitHub
https://github.com/JBKahn/django-sharding/pull/26/files/e534b5ffa93bd79fab7bcb8f90fb315a344f3661#r61980198

Updated to raise an exception

…ferences in code to check "in" rather than equality

…gres field without postgres being the engine. Updated test model and fields to reflect the change

- New Field for Postgres ID, along with migrations - Updated decorators.py to account for new postgres field - Updated settings helpers and constants with new utils, and fixed referencs to the same

- Check that field works as intended - Ensure stored procedure and sequence are created - Added migration to support field tests

4word · 2016-06-06T17:17:25Z

I rebased my commits to squash down the sheer volume of them that there were. I updated the doc to mention the shard limit.

This is ready to merge in, I am planning on doing the management commands in a new PR. I would rather wait for you to merge this in until after you release the manager and router changes on pypi, if you are planning on doing that soon, since its useful for most people. If not, then just merge it whenever and I will plan on working on the management commands later this week.

coveralls · 2016-06-06T17:32:04Z

Coverage decreased (-0.7%) to 93.174% when pulling 10ed281 on 4word:postgres-auto-shard-id into e92cff9 on JBKahn:master.

bronsen · 2016-06-23T08:31:32Z

docs/components/IDGeneration.md

 In the above example, it takes an optional database. However you will find that you can choose to provide additional arguments later on when you make use of the generator. The only real requirements is that it be a class with a `get_next_id` function.

 The two included in the package are:



Shouldn't it say

The three included in the package are

Also, I wonder if one could clarify what is enumerated here, since all these are mentioned above: strategies, additional arguments, or requirements. (<-- this is me being very pedantic)

You are right, it should say three. I will update.

What do you mean clarify what is enumerated? There are no additional requirements other than what is listed, which basically includes Postgres and defining a shard epoch time. All of the exact requirements are in the Using section for this feature, which is in the ShardingAModel.md file. If thats not what you mean, let me know!

…s: the Django default errors out if the FK goes out of integer range, as the FK field in Django is (by default), a standard integer rather than a bigint

coveralls · 2016-07-14T20:42:37Z

Coverage decreased (-1.9%) to 92.027% when pulling c76760d on 4word:postgres-auto-shard-id into e92cff9 on JBKahn:master.

coveralls · 2016-07-15T21:25:01Z

Coverage decreased (-2.03%) to 91.887% when pulling 43b465f on 4word:postgres-auto-shard-id into e92cff9 on JBKahn:master.

JBKahn · 2016-08-17T15:47:26Z

I'm going to try looking at this tonight. I'll probably take a stab at writing the management commands I'd like to include and then we can see about releasing this ASAP since I've been really lazy about this.

JBKahn · 2016-08-17T15:48:31Z

@4word if I write some commands, and this is reviewed, are any pieces missing or is this good to go (both from the point of view of the pull request and for a new release) from your point of view?

4word · 2016-08-17T22:02:14Z

All the pieces are there, should be good to be merged in.

coveralls · 2016-08-17T22:20:07Z

Coverage decreased (-2.03%) to 91.887% when pulling e5b5101 on 4word:postgres-auto-shard-id into e92cff9 on JBKahn:master.

coveralls · 2016-08-17T22:20:07Z

Coverage decreased (-2.03%) to 91.887% when pulling e5b5101 on 4word:postgres-auto-shard-id into e92cff9 on JBKahn:master.

JBKahn · 2016-08-18T02:07:11Z

django_sharding_library/fields.py

+        # If the database needs similar types for key fields however, the only
+        # thing we can do is making AutoField an IntegerField.
+        rel_field = self.target_field
+        if rel_field.get_internal_type() is "BigIntegerField":


Did you mean to put is and not == for the string comparison?

JBKahn · 2016-08-18T02:10:52Z

@4word those are my last two questions, then I should be good to merge this in and write the commands.

JBKahn · 2016-08-20T01:21:50Z

I'm going to merge and make the change and let me know if I should revert it. That way I can continue working on it before you get back to me.

Titus Peterson added 15 commits May 1, 2016 10:09

Added manager and queryset for automatic shard selection in the router

4e7f7f8

Updated custom ShardQueryset to call super() functions to help handle…

5c8d7c1

… django updates to the base method rather than returning on their own

Updated model_config decorator to take an optional "sharded_by_field"…

e3d6849

… argument that stores the field as an attribute and sets the default manager for the model to ShardManager

Updated the router to use the new optional sharded by field on the mo…

93fda3d

…del to get the correct shard based on the arguments passed to filter,create,get,etc. The default behavior is unchanged so existing models do not need any changes

Fixed a bug with staticmethod not working properly in the manager

39a450a

Fixed a bug with staticmethod not working properly in the manager

2f0d7f2

Updated logic for grabbing the shard in the router when the instance …

10b201c

…does not know what shard it should be on but there in a sharded by field defined

Updated the settings helper database_configs to put a shard id proper…

b4955eb

…ty on each primary shard. Added and updated unit tests to check for valid shard ids.

Fixed a test bug when using SQLite where the BigAutoField would fail …

bf13c3e

…with a too large for sqlite error

Fixed another BigAutoField bug with SQLite being too large to store

36722b6

Added common local dev files to gitignore

f272356

Added custom postgres id field that will automatically generate shard…

c6aca05

…-safe IDs. Added utils for migrations. Updated model_config decorator to enforce an ID strategy if the postgres shard id field does not exist on the model

Added unit tests for custom postgres shard id field

87688a9

Fixed a syntax error with bit manipulation

aafe214

Fixed another syntax error with bit manipulation

60ca204

Titus Peterson added 4 commits May 2, 2016 21:09

Added logging to troubleshoot CI

37e674f

More logging for CI

ff8e003

Fixed formatting issue with psycopg2. Removed logging. Updated unit t…

b9c7742

…ests.

Added another test to verify the id coming back from the postgres id …

e534b5f

…generator field is valid

Titus Peterson added 5 commits May 3, 2016 00:59

Fixed multiple syntax errors in the router where it would look up the…

48a5086

… wrong values on accident

Created a base model for sharding to go along with the decorator. All…

3790972

… the base model does is set the manager. Unfortunately, this cannot be done in the decorator for some reason (I cannot figure out why, it simply doesnt work correctly)

Updated the model instace create function to copy out the kwargs befo…

ce7a895

…re instantiating the object with them, just in case the parent class messes with the kwargs later

Updated tests for filter() and create() shard auto detection (no more…

aab706c

… using())

Fixed a bug in a test where the user created does not get pk=1

3ecaad9

JBKahn reviewed May 3, 2016
View reviewed changes

Titus Peterson added 2 commits May 3, 2016 19:00

Updated constants with all of the backend for each vendor. Updated re…

b01bb6c

…ferences in code to check "in" rather than equality

Updated decorator to raise an error if a person tries to use the post…

0158889

…gres field without postgres being the engine. Updated test model and fields to reflect the change

Titus Peterson added 6 commits June 6, 2016 08:21

Merge branch 'master' into shopventory-branch

a3f27f1

Added Postgres specific ID field functionality:

0d16903

- New Field for Postgres ID, along with migrations - Updated decorators.py to account for new postgres field - Updated settings helpers and constants with new utils, and fixed referencs to the same

Added tests for Postgres-specific ID field

f4e7e08

- Check that field works as intended - Ensure stored procedure and sequence are created - Added migration to support field tests

Updated docs with postgres-specific ID information

9d3f513

Added more common development files for git to ignore

85ce1fe

Updated runtests.py with required settings for postgres ID field testing

10ed281

4word force-pushed the postgres-auto-shard-id branch from 838eb4e to 10ed281 Compare June 6, 2016 17:12

bronsen reviewed Jun 23, 2016
View reviewed changes

Titus Peterson added 4 commits July 14, 2016 11:11

Added a foreign key field that can handle Postgres generated ID field…

ef1b6a5

…s: the Django default errors out if the FK goes out of integer range, as the FK field in Django is (by default), a standard integer rather than a bigint

Merge branch 'postgres-auto-shard-id' into shopventory-branch

a402efa

Added one-to-one field for Django to work correctly with

c76760d

Merge branch 'postgres-auto-shard-id' into shopventory-branch

49bc198

Titus Peterson added 2 commits July 15, 2016 14:22

Added rel_db_type function to postgres shard ID field

43b465f

Merge branch 'postgres-auto-shard-id' into shopventory-branch

de1fc41

Fixed formatting in migration command for 1.10

e5b5101

JBKahn reviewed Aug 18, 2016
View reviewed changes

JBKahn merged commit feb6693 into JBKahn:master Aug 20, 2016

4word deleted the postgres-auto-shard-id branch September 12, 2016 20:13

		In the above example, it takes an optional database. However you will find that you can choose to provide additional arguments later on when you make use of the generator. The only real requirements is that it be a class with a `get_next_id` function.

		The two included in the package are:

Add postgres-specific id generator #26

Add postgres-specific id generator #26

Uh oh!

Conversation

4word commented May 3, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

4word commented May 3, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

4word commented May 3, 2016

Uh oh!

JBKahn May 3, 2016

Choose a reason for hiding this comment

Uh oh!

4word May 4, 2016

Choose a reason for hiding this comment

Uh oh!

JBKahn May 4, 2016

Choose a reason for hiding this comment

Uh oh!

4word May 4, 2016

Choose a reason for hiding this comment

Uh oh!

4word commented Jun 6, 2016

Uh oh!

coveralls commented Jun 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bronsen Jun 23, 2016

Choose a reason for hiding this comment

Uh oh!

4word Jun 25, 2016

Choose a reason for hiding this comment

Uh oh!

coveralls commented Jul 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Jul 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JBKahn commented Aug 17, 2016

Uh oh!

JBKahn commented Aug 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

4word commented Aug 17, 2016

Uh oh!

coveralls commented Aug 17, 2016

Uh oh!

coveralls commented Aug 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JBKahn Aug 18, 2016

Choose a reason for hiding this comment

Uh oh!

JBKahn commented Aug 18, 2016

Uh oh!

JBKahn commented Aug 20, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

4word commented May 3, 2016 •

edited

Loading

4word commented May 3, 2016 •

edited

Loading

coveralls commented Jun 6, 2016 •

edited

Loading

coveralls commented Jul 14, 2016 •

edited

Loading

coveralls commented Jul 15, 2016 •

edited

Loading

JBKahn commented Aug 17, 2016 •

edited

Loading

coveralls commented Aug 17, 2016 •

edited

Loading