Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ For information about how to setup sharding in your application, [read the docum

### Developer Experience

I wrote this library after working on this problem for [Wave](https://www.waveapps.com) and not being able to find a library that suited our needs. What we were looking for was something that was powerful, extensible and customizable. This library was created for just that purpose and includes at least one implimentation of each part of the pipeline with room to replace any individual components.
I wrote this library after working on this problem for [Wave](https://www.waveapps.com) and not being able to find a library that suited our needs. What we were looking for was something that was powerful, extensible and customizable. This library was created for just that purpose and includes at least one implementation of each part of the pipeline with room to replace any individual components.

### Influences

Expand Down
2 changes: 1 addition & 1 deletion django_sharding/apps.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def ready(self):
continue

if len(shard_fields) > 1:
raise Exception('The model {} has multuple fields for shard storage: {}'.format(model, shard_fields))
raise Exception('The model {} has multiple fields for shard storage: {}'.format(model, shard_fields))
shard_field = shard_fields[0]
shard_group = getattr(shard_field, 'django_sharding__shard_group', None)

Expand Down
4 changes: 2 additions & 2 deletions docs/components/IDGeneration.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# ID Generation

In order to shard your database, one of the first decisions to makee is how you assign identifiers to the sharded objects. While it is not required, it is highly recommended that you choose a unique identifier. The main reason here being that you may want to either move data across shards later or that you may choose to analyze data across various shards for analytics and you will have to differentiate those objects before moving them to another server.
In order to shard your database, one of the first decisions to make is how you assign identifiers to the sharded objects. While it is not required, it is highly recommended that you choose a unique identifier. The main reason here being that you may want to either move data across shards later or that you may choose to analyze data across various shards for analytics and you will have to differentiate those objects before moving them to another server.

This repository is initially shipping with three strategies but you may impliment your own. The base requirement for defining your own strategy at the moment is that you define a class like this:
This repository is initially shipping with three strategies but you may implement your own. The base requirement for defining your own strategy at the moment is that you define a class like this:

```python
class BaseIDGenerationStrategy(object):
Expand Down
4 changes: 2 additions & 2 deletions docs/components/OtherComponents.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ class ShardedIDFieldMixin(object):

##### Table Sharded ID Field

As an example using the above mixin, one of the included fields uses a secondary table to generate unique IDs, as discussed in the ID generation section of this guide. This takes the class of the table as an argument and impliments to strategy shipped with this package:
As an example using the above mixin, one of the included fields uses a secondary table to generate unique IDs, as discussed in the ID generation section of this guide. This takes the class of the table as an argument and implements to strategy shipped with this package:

```python

Expand Down Expand Up @@ -172,7 +172,7 @@ class ShardStorageCharField(ShardLocalStorageFieldMixin, CharField):

##### Storing The Shard On Another Model Shard Foreign Key Storage Field

As an example, say we have a webapp that serves all TD banks and we wish to store all the branches within a district under the same shard. We could store the shard on the district but perhaps we don't usually touch the district or need to check it and would rather store the shard on the branch itself but still shard by district. One solution is to use a unqiue `shard_key` field on another table to store the shards and create a foreign key to that table. That implimentation has been included in this library.
As an example, say we have a webapp that serves all TD banks and we wish to store all the branches within a district under the same shard. We could store the shard on the district but perhaps we don't usually touch the district or need to check it and would rather store the shard on the branch itself but still shard by district. One solution is to use a unqiue `shard_key` field on another table to store the shards and create a foreign key to that table. That implementation has been included in this library.

As before, we simply store additional data in `__init__` and then retreive stored args and kwargs in `deconstruct`. In this case, we will make use of the `ForeignKey` field's args as well as the `shard_group` from our previous mixin.

Expand Down
2 changes: 1 addition & 1 deletion docs/components/ReadStrategies.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,4 +110,4 @@ class ExampleRatioRoutingStrategy(BaseRoutingStrategy):

##### Note About Using Read Strategies

If you're using one of the above, or a custom read strategy, there are some considerations that are important when choosing them. The system does not currenly have a built-in system to handle replication lag time. For example, if a user updates item A in the primary database then reading from a replication database before that data has propogated will result in the user getting stale data. This is typically handled by reading only from the primary drive during this period, however the system does not currenly include these tools and will need to be written for the project. For more information, check out the section where we discuss replication lag time.
If you're using one of the above, or a custom read strategy, there are some considerations that are important when choosing them. The system does not currently have a built-in system to handle replication lag time. For example, if a user updates item A in the primary database then reading from a replication database before that data has propogated will result in the user getting stale data. This is typically handled by reading only from the primary drive during this period, however the system does not currently include these tools and will need to be written for the project. For more information, check out the section where we discuss replication lag time.
4 changes: 2 additions & 2 deletions docs/components/Router.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ First we check if it's non-sharded but stored on a single database other than `d

When running your migrations, the app needs to be able to determine which databases require the migration in order to same developers from having to do this work manually.

As such, we restrict migrations to only those which provide the model they are migrating as well as migrations to primary databases. In the event that a model is on a sepcific database or sharded then we also restrict the migration to those sets of databases. By using the module loading system in Django, we can determine the shard status of a model instance in order to make an informed decision.
As such, we restrict migrations to only those which provide the model they are migrating as well as migrations to primary databases. In the event that a model is on a specific database or sharded then we also restrict the migration to those sets of databases. By using the module loading system in Django, we can determine the shard status of a model instance in order to make an informed decision.

```python
def allow_migrate(self, db, app_label, model_name=None, **hints):
Expand Down Expand Up @@ -125,4 +125,4 @@ As such, we restrict migrations to only those which provide the model they are m
if shard_group:
return settings.DATABASES[db]['SHARD_GROUP'] == shard_group
return db == 'default'
```
```
2 changes: 1 addition & 1 deletion docs/installation/Settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ DJANGO_FRAGMENTS_SHARD_SETTINGS = {
}
```

Additionally, if you add a shard field on a model to store the shard for that object, the package will automate the process of retreiving and saving the shard on model save. You can skip automatically saving the shard to the User model in this example by adding this setting:
Additionally, if you add a shard field on a model to store the shard for that object, the package will automate the process of retrieving and saving the shard on model save. You can skip automatically saving the shard to the User model in this example by adding this setting:

```python
DJANGO_FRAGMENTS_SHARD_SETTINGS = {
Expand Down
4 changes: 2 additions & 2 deletions tests/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ class ShardedModelIDs(TableStrategyModel):
pass


# An implimentation of the extension of a the Django user to add
# An implementation of the extension of a the Django user to add
# the mixin provided in order to save the shard on the user.
@shard_storage_config()
class User(AbstractUser, ShardedByMixin):
pass


# An implimentation of the extension of a the Django user to add
# An implementation of the extension of a the Django user to add
# the mixin provided in order to save the shard on the user.
@shard_storage_config(shard_group='postgres')
class PostgresShardUser(AbstractUser, ShardedByMixin):
Expand Down
4 changes: 2 additions & 2 deletions tests/test_sharding_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,14 @@ def test_get_shards(self):

self.assertEqual(sorted(result), expected_result)

def test_pick_shard_method_defined_but_unimplimented(self):
def test_pick_shard_method_defined_but_unimplemented(self):
from django.contrib.auth import get_user_model
User = get_user_model()
sut = BaseBucketingStrategy(shard_group='default')
with self.assertRaises(NotImplementedError):
sut.pick_shard(User)

def test_get_shard_method_defined_but_unimplimented(self):
def test_get_shard_method_defined_but_unimplemented(self):
from django.contrib.auth import get_user_model
User = get_user_model()
sut = BaseBucketingStrategy(shard_group='default')
Expand Down