- This project may be removed from public view at any time.
- There is no warranty, express or implied.
- Use at your own risk.
Share, load, edit and display data from various mobile data collection platforms. The data itself is stored in MongoDB but metadata and dashboarding is done through Django and a relational database. The Silo app provides most of the functionality. It stores data source and destination locations, manages the data import process, and manages permissions and other meta-processes.
After login, the user can choose to import data from several different platforms, including Google Sheets, Ona, and CommCare. Data from multiple sources can be combined into a single table (aka Silo).
git clone https://github.com/toladata/TolaTables.git
cd TolaTables
First copy the sample config files:
cp tola/settings/local.example.py tola/settings/local.py
cp silo/client_secrets_example.json silo/client_secrets.json
cp fixtures/sites.json fixtures/sites_local.json
cp fixtures/tolasites.json fixtures/tolasites_local.json
cp fixtures/tags.json fixtures/tags_local.json
cp fixtures/read_types_.json fixtures/read_types_local.json
Once copied, modify all new files to suit your own particular installation and organization. The local.py file contains Django settings, client_secrets.py contains Google Auth related settings, and the fixture files are a starter set of metadata for your organization. You will need to load the sites and tolasites fixtures in order to run Tola, and loading read_types is required for importing data.
If you have password protected your MongoDB database, you will need to update the settings in the local.py file to match your setup. If you have not password protected the MongoDB instance, you can comment out the MONGODB section in local.py and uncomment the same section in base.py.
Once you are done configuring the fixtures you can load them:
./manage.py loaddata sites_local
./manage.py loaddata tolasites_local
./manage.py loaddata tags_local
./manage.py loaddata read_types_local
Virtualenv allows us to customize an encapsulated version of python to use with your app.
First install Virtualenv to you system python installation, then initiate a python virtual environment and load the required python modules.
pip install virtualenv
cd ..
virtualenv --no-site-packages venv
source venv/bin/activate
You should now see '(venv)' added to the left side of your prompt. If you don't, you have not successfully activated the Virtualenv.
Now install the python modules into the Virtualenv:
pip install -r TolaTables/requirements.txt
You may need to install selenium as well. On a Mac, the easiest way is to run
brew install selenium
Some MacOS systems have trouble seeing the MySql installation. If you are using MySql, you may need to run this command.
export PATH=$PATH:/usr/local/mysql/bin
python manage.py migrate
This App uses Celery and RabbitMQ as a queueing system for certain data imports and stores data in a MongoDB database. All of these applications require their own servers to be running concurrently with your main application server. These instructions enable you to run interactive servers from the command-line on your local computer; you will eventually have four servers running at once. For server environments (and for your local development environment if you choose), you will likely daemonize these services.
Follow the RabbitMQ installation guide. For macOS, if you have homebrew installed, you should be able to brew install rabbitmq. Once RabbitMQ is installed you can start the server with rabbitmq-server and stop it with rabbitmqctl stop.
The Celery library should have been installed with the rest of the python packages you installed earlier. You can start celery worker using celery -A tola worker -l info. For more information check out its documentation.
MongoDB uses /data/db as the default directory for its database files and if you run a bare mongod command, your data will go into that directory. If you wish to use a different directory, you can run mongod --dbpath <path/to/your/dir>, as described in the MongoDB docs. Your data directory should be outside of the TolaTables repository.
If your using more then one settings file change manage.py to point to local or dev file. Then run
python manage.py runserver
Because Tola Tables relies on several different types of services (with multiple options for each type), upgrading Tola Tables on the server can be a multi-step process. Upgrade procedures for a common deployment configuration can be found below.
Nginx webserver, Celery task queue, RabbitMQ message broker
If you use your browser to navigate to localhost:8000, you should now find a Tola login screen.
To filter data from api/silos/#{pk}/data endpoint add a mongodb query to the modifier at the end of the url Ex. api/silo/2/data?query={"nm":"Henry"} More advanced query language can be found at https://docs.mongodb.com/manual/ To sort data data add onto the url sort=<column_name> for ascending or sort=-<column_name> for descending
0.9.2 changes the way data is stored in MongoDB to increase efficiency and reduce storage space. To accommodate these changes it is necessary to run the collect_silo_columns command otherwise no data will show up in TolaTables. 0.9.2 adds indexes to the MongoDB to make reading and writing faster. To enforce this change run the add_indexes_for_silos command.
Do not run unit tests on a production database. Django is not set up to make a test MongoDB so data is added and removed from the MongoDB in settings. Any data with silo_id 1 will be damaged or deleted.