Skip to content

Commit

Permalink
Merge pull request #1 from JBKahn/coveralls
Browse files Browse the repository at this point in the history
Coveralls
  • Loading branch information
JBKahn committed Oct 11, 2015
2 parents 5fa6c93 + 9551b7c commit 5464dc9
Show file tree
Hide file tree
Showing 11 changed files with 359 additions and 285 deletions.
72 changes: 5 additions & 67 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ It helps you to scale your applications by sharding your data across multiple da
[![Circle CI](https://circleci.com/gh/JBKahn/django-sharding.svg?style=svg)](https://circleci.com/gh/JBKahn/django-sharding)
[![PyPI version](https://badge.fury.io/py/django-sharding.svg)](https://badge.fury.io/py/django-sharding)
[![PyPi downloads](https://img.shields.io/pypi/dm/django-sharding.svg)](https://crate.io/packages/django-sharding/)
[![Coverage Status](https://coveralls.io/repos/JBKahn/django-sharding/badge.svg?branch=coveralls&service=github)](https://coveralls.io/github/JBKahn/django-sharding?branch=coveralls)
[![Coverage Status](https://coveralls.io/repos/JBKahn/django-sharding/badge.svg?branch=master&service=github)](https://coveralls.io/github/JBKahn/django-sharding?branch=coveralls)

### What is Sharding?

Expand All @@ -23,73 +23,11 @@ The package was influenced by my experiences at Wave as well as the help and cod

### Installation

To install the package, use pypi:
Check out the [installation section](http://josephkahn.io/django-sharding/docs/installation/Settings.html) of the docs for basic package setup.

```
pip install django-sharding
```

and Add the package to your installed apps:

```python
INSTALLED_APPS=[
...,
"django_sharding",
],
```

### Using The Default Configuration

Refer to the configuration section (link) of the ReadMe for additional information.
### Basis Setup & Usage

Add the following to your settings file:

```python
# Most applications will not need aditional routers but if you need your own then
# remember that order does matter. Read up on them here (link).
DATABASE_ROUTERS=['django_sharding_library.router.ShardedRouter'],

```

Add your databases to you settings file in the following format based on, and using, dj-database (link).
This structure supports unsharded sets of databses as well as replicates. This setting uses a single shard group,
more advanced structures are possible and checkout the other section of the docs for more information (link):

```python
DATABASES = database_configs(databases_dict={
'unsharded_databases': [
{
'name': 'default',
'environment_variable': 'DATABASE_URL',
'default_database_url': 'postgres://user:pw@localhost/sharding'
}
],
'sharded_databases': [
{
'name': 'app_shard_001',
'environment_variable': 'SHARD_001_DATABASE_URL',
'default_database_url': 'postgres://user:pw@localhost/sharding_001',
'replicas': [
{
'name': 'app_shard_001_replica_001',
'environment_variable': 'REPLICA_001_DATABASE_URL',
'default_database_url': 'postgres://user:pw@localhost/shard_replica_001'
},
{
'name': 'app_shard_001_replica_002',
'environment_variable': 'REPLICA_002_DATABASE_URL',
'default_database_url': 'postgres://user:pw@localhost/shard_replica_002'
},
]
},
{
'name': 'app_shard_002',
'environment_variable': 'SHARD_002_DATABASE_URL',
'default_database_url': 'mysql://user:pw@localhost/sharding_002'
},
]
})
```
#### Sharding by User

Select a model to shard by and open up the models.py file. Here we'll use the user model:

Expand All @@ -109,7 +47,7 @@ Add that custom User to your settings file using the string class path:
AUTH_USER_MODEL = '<app_with_user_model>.User'
```

### Create Your First Sharded Model
#### Create Your First Sharded Model

Define your new model, eg:

Expand Down
3 changes: 3 additions & 0 deletions circle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ test:
- createdb -U ubuntu -w -p 5432 shard_002
- createdb -U ubuntu -w -p 5432 sharding_replica_001
- createdb -U ubuntu -w -p 5432 sharding_replica_002
post:
- python ./runcoverage.py
- coveralls

dependencies:
override:
Expand Down
15 changes: 8 additions & 7 deletions django_sharding_library/sharding_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@ def pick_shard(self, model_sharded_by):
Returns the shard for the model which has not previously been bucketed
into a shard.
"""
raise NotImplemented
raise NotImplementedError

def get_shard(self, model_sharded_by):
"""
Returns the shard for a model which has already been assigned a shard.
"""
raise NotImplemented
raise NotImplementedError


class BaseShardedModelBucketingStrategy(BaseBucketingStrategy):
Expand All @@ -38,7 +38,8 @@ class BaseShardedModelBucketingStrategy(BaseBucketingStrategy):
the model.
"""
def get_shard(self, model_sharded_by):
return model_sharded_by.shard
sharded_field = getattr(model_sharded_by, 'django_sharding__shard_field')
return getattr(model_sharded_by, sharded_field)


class RoundRobinBucketingStrategy(BaseShardedModelBucketingStrategy):
Expand Down Expand Up @@ -68,7 +69,7 @@ class RandomBucketingStrategy(BaseShardedModelBucketingStrategy):
the model.
"""
def __init__(self, shard_group, databases):
super(RoundRobinBucketingStrategy, self).__init__(shard_group)
super(RandomBucketingStrategy, self).__init__(shard_group)
self.shards = self.get_shards(databases)

def pick_shard(self, model_sharded_by):
Expand All @@ -83,14 +84,14 @@ class ModBucketingStrategy(BaseBucketingStrategy):
change.
"""
def __init__(self, shard_group, databases):
super(RoundRobinBucketingStrategy, self).__init__(shard_group)
super(ModBucketingStrategy, self).__init__(shard_group)
self.shards = self.get_shards(databases)

def pick_shard(self, model_sharded_by):
self.shards[hash(str(model_sharded_by.pk)) % len(self.shards)]
return self.shards[hash(str(model_sharded_by.pk)) % len(self.shards)]

def get_shard(self, model_sharded_by):
self.shards[hash(str(model_sharded_by.pk)) % len(self.shards)]
return self.pick_shard(model_sharded_by)


class SavedModBucketingStrategy(BaseShardedModelBucketingStrategy, ModBucketingStrategy):
Expand Down
192 changes: 0 additions & 192 deletions docs/Troubleshooting.md

This file was deleted.

Loading

0 comments on commit 5464dc9

Please sign in to comment.