Project Bluebottle

The repository for Project Bluebottle, the crowdsourcing framework initiated by GoodUp.

Contributors

For those who want to try out to the BlueBottle project, here's to get started:

1. Fork and/or clone the repository.
2. Navigate to your local repository directory.
3. Create a bluebottle/settings/local.py base on bluebottle/settings/secrets.py.example

Now you have the option to install the application with, or without Docker. If you wish to not use Docker, continue below. Otherwise, see :ref:`docker`.

1. Create a virtual environment within the repository directory, for example:

   $ virtualenv env
   $ source env/bin/activate
   

2. Make sure you have a recent Python distro (3.7+ recommended).

3. Make sure (a recent) virtualenv is installed.

4. Install the project:

   $ pip install -e .[test]
   

5. Migrate the database schemas:

   $ python manage.py migrate_schemas --shared --settings=bluebottle.settings.testing
   

6. (optional) Create a new tenant (if you haven't imported a database dump):

   $ python manage.py create_tenant
   

7. (optional) Re-index any imported data using Elasticsearch

   • For all tenants:

     $ python manage.py tenant_command search_index --rebuild -f
     

   • If for whatever reason this doesn't work, try running the migration step for a single tenant only, and indexing that tenant's data. E.g.:

     $ python manage.py migrate_schemas -s onepercent --settings=bluebottle.settings.testing
     $ python manage.py tenant_command --schema onepercent search_index --rebuild -f
     

8. Start the server:

   $ python manage.py runserver
   

9. You might still need to

   • Install libraries to get pip install working, like libxmlsec1, postgresql, postgis, elasticsearch 6.x

     • Installing these is not necessary when using Docker

   • Alter your hosts file (e.g. in /etc/hosts on Linux/OSX) to contain tenants you've created like:

     127.0.0.1 tenant35.localhost
     

Docker

It is possible to run thy Python server, PostgreSQL and Elasticsearch in a Docker environment using docker-compose. To get started, make sure to download a Docker client, like Docker Desktop.

Installation:

Make sure to download Docker (e.g. Docker Desktop) first.

In your local.py file, set the DATABASES variable to the following:

DATABASES = {
    'default': {
        'ENGINE': 'bluebottle.clients.postgresql_backend',
        'HOST': 'postgres',
        'PORT': '5432',
        'NAME': 'reef',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'DISABLE_SERVER_SIDE_CURSORS': True # this prevents issues with connection pooling
    },
}

ELASTICSEARCH_DSL = {
    'default': {
        'hosts': 'elasticsearch:9200'
    },
}

• To start the containers (if this is your first time, all dependencies will be installed automatically):

  $ docker-compose -u -d
  

• To import a database dump file (please note that the last two commands run without the -t flag):

  $ docker exec -it -u postgres postgres dropdb reef
  $ docker exec -it -u postgres postgres createdb reef
  $ bzcat reef-prod-current.sql.bz2 | docker exec -i -u postgres postgres psql reef
  $ echo "UPDATE clients_client SET domain_url=CONCAT(client_name, '.localhost');" | docker exec -i -u postgres postgres psql reef
  

• If you are running into errors related to max_map_count, then run the following command in your terminal: sysctl -w vm.max_map_count=262144

Running the containers:

To run the containers:

$ docker-compose -u -d

To run one specific container:

$ docker-compose -u -d [CONTAINER_NAME]

Or on other systems (some OSX):

$ docker compose up -d elasticsearch

To shut them down:

$ docker-compose down

The environment also comes with pgAdmin included so you can inspect the local database. Navigate to http://localhost:5050 and login with these credentials:

• Email: admin@admin.com
• Password: admin

After that, you can add a new server using the details below to inspect the PostgreSQL database:

• Host: host.docker.internal
• Username: postgres
• Password: postgres

To run commands in the Python container:

$ docker exec -it bluebottle python manage.py [YOUR_COMMAND]

- For example::

    $ docker exec -it bluebottle python manage.py migrate_schemas -s onepercent --settings=bluebottle.settings.local

If you need to rebuild the container, for example when you want to apply changes after pulling the latest version of a branch, run:

$ docker compose up --build

Testing

The BlueBottle test suite can be run completely using:

1. Install the dependencies

   $ pip install -e .[test,dev]

2. Create test db and restore testdata

   $ createdb test_reef $ psql test_reef < testdata.sql

3. Run the tests

   $ python manage.py test -k

Translating

To push new strings to crowding:

$ ./locale/push

After checking that master is fully merged in the current branch, this will extract new messages and push them to crowdin. Make sure to commit all found strings.

To pull messages simply do

$ crowdin pull

And commit the changes after that command