TotalEmail Core

This is the start of something great!

NOTE: The code is this repository is missing some of the setting and configuration files required to get it working. It is not designed for anyone else to get it working on their own systems and this use-case is not supported. The code is simply open-sourced for transparency and out of necessity.

Guidelines

Git

Most of your work should be done on a branch with your name. In other words, Bob would work on a branch named bob. On occasions, code can be merged into the dev branch. Once code on the dev branch is well tested, it can be merged into master branch.

Setup

Docker (required)

1. Install docker and docker-compose.

• To start the server: docker-compose up
• To run tests: docker-compose run web pytest
• To make migrations and migrate: docker-compose run web python3 manage.py makemigrations and docker-compose run web python3 manage.py migrate respectively
• To list running docker-compose services: docker-compose ps
• To shutdown a running server (ctrl+c will stop the server): docker-compose down

Using Alerta

To log errors to Alerta, you will also need to create environmental variables as follows (with valid data of course):

export ALERTA_KEY=1234567890987654321
export ALERTA_BASE_URL=https://192.168.0.0

Postgres (required)

If you are running the vagrant box for the first time, you will need to install postgres. To do this, run the ./_meta/postgres_install.sh script from inside the vagrant box.

Next, run sudo -u postgres psql, which logs you into the postgres user account so that you can create a database.

Now, enter the following commands:

CREATE DATABASE totalemail;
CREATE USER junkmailer WITH PASSWORD '1234567';

ALTER ROLE junkmailer SET client_encoding TO 'utf8';
ALTER ROLE junkmailer SET default_transaction_isolation TO 'read committed';
ALTER ROLE junkmailer SET timezone TO 'UTC';

GRANT ALL PRIVILEGES ON DATABASE totalemail TO junkmailer;
\q

This will setup a database that Django can talk to.

ToDo Review (recommended)

To make it easier to find todos, I recommend using todo review. If you already have this installed, you should also update your exclusions so that todo review doesn't look through the test_emails directory (assuming you have that created). To add an exclusion to todo-review, following the process here: https://github.com/jonathandelgado/SublimeTodoReview#excluding-files-and-folders. I just changed my default, todo-review settings, but you can change this to a custom format if you like.

Mass Email Creation (optional)

To create a lot of emails in the system while it is running (which is helpful to test ui, ux, searching, etc..), follow the instructions below:

1. Copy the contents of ./test_resources/mass_email_creator.py
2. In the command-line, run: docker-compose run web python3 manage.py shell
3. Paste the contents of ./test_resources/mass_email_creator.py into the shell and you're good to go!

Notes:

• You can change the directory from which the emails are created in ./test_resources/mass_email_creator.py.

Heroku Deployment (optional)

To be able to deploy the app to Heroku, install the Heroku CLI and run:

git remote -v
heroku git:remote -a <NAME OF HEROKU APP>
git remote -v # you will now see that there is a git remote for heroku

You should only push updates to heroku from the master branch of the TE core repository. To push an update to heroku, use:

git push heroku master

You also need to add the Alerta key in heroku so that it is accessible in the app. To do this, update the commands below with the correct information and run:

heroku config:set ALERTA_KEY=1234567890987654321
heroku config:set ALERTA_BASE_URL=https://192.168.0.0

Helpful Scripts

Recreate the database

Delete and re-create the database:

DROP DATABASE totalemail;
CREATE DATABASE totalemail;
GRANT ALL PRIVILEGES ON DATABASE totalemail TO junkmailer;
\q

Deploy to Heroku for the first time

When pushing to Heroku for the first time (if you have been testing it locally for a while), there are a couple of changes you will have to make. All of the steps below assume you are on the master branch. First, open totalemail/settings.py and find the section that says DATABASES = {. Comment out the section that is used for local testing (which is identified with a comment) and uncomment the section for heroku. Save and commit the changes. Now, you can run git push heroku master to push the updates to heroku.

Next, you need to make sure the db configuration on heroku is up to date. To do this, run:

heroku run python manage.py migrate

Assuming this finishes successfully, you are good to go!

Viewing database tables

To view the structure of your database run:

psql

and then:

\dt

Recipes

Adding a New Analysis Engine

To add a new analysis engine:

• create a new file in the analysis_server/engines directory with the appropriate functions
• call the function and process the results from analysis_server/analysis.py
• Add an entry for the new source with the appropriate function to the SOURCE_WEIGHTINGS dictionary in utility/utility.py

Common Errors

• Missing staticfiles manifest entry for 'styles.css': run make static

Misc.

Color Codes

The color codes for the stop lights are:

• #6FEF87 - green
• #F1D86C - yellow
• #E64A48 - red

Licensing

Copyright (C) 2017-2021, TotalEmail authors

This project is currently licensed under the AGPL-3.0 license. If you have any questions, feel free to contact us at info@totalemail.io.