Skip to content
Python Flask application with a strong inclination for social network and blogging features
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci
etc
examples
pili
tests
.gitignore
.pyup.yml
Dockerfile
LICENSE
Makefile
README.rst
config.py
docker-compose.ci.yml
docker-compose.yml
docker-entrypoint.sh
requirements.txt
setup.cfg
setup.py

README.rst

https://circleci.com/gh/pilosus/pili/tree/master.svg?style=svg

Pili App

Pili is a Python Flask application with a strong inclination for social network and blogging features.

About Pili

Stack

Application is based on Flask, a light-weight Python web framework that gets use of Werkzeug toolkit and Jinja2 template engine. Although Flask works fine with both Python 2 and 3, Pili App's written with Python 3 in focus.

Database-agnostic, application uses SQLAlchemy ORM, which enables user to choose between DBMS ranging from a simple SQLite to an enterprise solution of user's choice.

Asynchronous tasks (such as email sending) tackled with Celery distributed task queue, which is to be used with a message broker software of user's choice such as Redis or RabbitMQ.

Features

  1. Users

    • Registration
    • Authentication
    • Resetting/changing password
    • Creating/changing profile
  2. Roles

    • Each user assigned one of the 7 predefined roles
    • Each role has a set of permissions
  3. Posts

    • Tagging
    • Categorization (sectioning, which is useful for websites)
    • File upload
  4. Comments

    • Write/delete comments
    • Screen/unscreen comments (not seen by the non-moderator users, all new comments can be set screened by default)
    • Disable/enable comments (non-moderators that disable comment exists, but its content disabled by the moderator)
    • Replies to the comments (users see replies to their comments)
  5. Following

    • Follow/Unfollow other users to customize a feed
  6. Likes

    • Authenticated users with FOLLOW permission can like/unlike posts or comments
  7. Notifications

    • Get notifications from platform administrators
    • See replies to your comments
  8. REST API

Credits

Pili App could not be possible without Miguel Grinberg's Flasky App developed as an example project for his excellent Flask Web Development book published by O'Reilly Media in 2014.

Application comes with 3rd party libraries preinstalled:

  1. Bootstrap 3 Datetimepicker
  2. Typeahead.js
  3. Bootstrap Tagsinput

These libraries are found under:

app/static/js
app/static/css

The libraries belong to their owners and should not be considered as a part of the application.

Deployment with Docker

Local development

  1. Install docker>=18.06 and docker-compose>=1.23.0

  2. Set environment variable PILI_CONFIG=development (you can place it to .env file in the root directory of the project)

  3. Create file /etc/config/development.env and save enviroment variables needed for the app, e.g.:

    FLASK_CONFIG=development
    FLASK_ENV=development
    FLASK_INIT=1  # initialize DB with python manage.py initialize
    FLASK_DEPLOY=1  # prepopulate DB with python manage.py deploy
    SECRET_KEY=your_key
    SSL_DISABLE=1  # you don't need this in localhost
    DATABASE_URL=postgresql://pili:pili@db/pili  # use DB as docker-compose service
    CELERY_INSTEAD_THREADING=True  # use celery cervice
    CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672/  # use RabbitMQ as celery's broker
    CELERY_RESULT_BACKEND=redis://redis:6379/10  # celery result backend
    FLOWER_PORT=5678  # monitoring tool for celery
    FLOWER_BROKER_API=http://guest:guest@rabbitmq:15672/api/
    MAIL_SERVER=your_smtp
    MAIL_PORT=587
    MAIL_USE_TLS=True
    MAIL_USERNAME=you@your@smtp
    MAIL_PASSWORD=your_password
    
  4. Run services with docker-compose up

  5. Open service with browse http://localhost:8080

  6. Open celery monitoring with browse http://localhost:5678

Use make for the routine operations like:

  1. Start/stop docker services with make up and make down respectively
  2. Run linters with make lint
  3. Run mypy static analysis tool with make mypy
  4. Format code with black formatter

Production

The project uses Circle CI for CI/CD. As its final step CI/CD pushes docker image to a private docker registry. The image can be used then in docker run, docker-compose or in a Kubernetes cluster.

Deployment

This section considered deprecated, see DockerDeployment for the suggested deployment model.

Environment setup

Application's deployment follows the same steps as any other large Flask application.

Setting up environment basically means:

  1. Installing dependencies (Python packages)
  2. Editing application's configurations files
  3. Exporting shell environment variables

List of dependencies is made up of several parts:

  1. Common dependencies
  2. Dependencies specific for the environment (built upon common dependencies):
    • Development
    • Production (Unix server)
    • Heroku

Dependencies lists are found under:

requirements/

virtualenv can be used for creating a virtual environment in the app's working directory in order to install aforementioned dependencies:

$ virtualenv --python=python3 venv

Then virtual environment can be activated/deactivated:

$ source venv/bin/activate
(venv) $ deactivate

Dependencies can be installed then using pip:

(venv) $ pip install -r requirements/unix[prod|dev|...].txt

App's config file

Application gets use of environment variables. The whole list of such variables can be found in config.py.

These environment variables are set using shell-specific commands, such as export in bash or setenv in csh:

(venv) $ export VARIABLE=value

IMPORTANT! Application also relies on .hosting.env file that is to be created by the user in the app's working directory. File format is the following:

ENVVARIABLE=value of the environment variable

.hosting.env is mandatory for manage.py file. It can also be used in production when writing systemd service files (with EnvironmentFile directive).

IMPORTANT! Although manage.py sets environment variables found in .hosting.env users cannot rely on it when working with Celery workers. In this case environment variables are to be set in Celery's own configuration (production) or with the shell's export command (development).

Database deployment

Application uses Flask-Migrate for database migrations with Alembic. Database deployment is made up of the following steps:

  1. Create all databases used by the application, create migration repository:

    (venv) $ python manage.py initialize
    
  2. Generate an initial migration, apply it to the database, then insert roles and add application's administrator:

    (venv) $ python manage.py deploy
    

Run application

Now that the application is configured, DB created and migration repo is created, the last two steps are needed in order to get the application running:

  1. Start Celery workers with:

    (venv) $ celery worker -A celery_worker.celery --loglevel=info
    
  2. Start development server:

    (venv) $ python manage.py runserver
    
  3. Go to http://127.0.0.0:5000 and enjoy!

When application models changed

Every time the database models (app/models.py) change do the following:

(venv) $ python manage.py db migrate [--message MESSAGE]
(venv) $ emacs $( ls -1th migrations/versions/*.py | head -1 ) # check and edit migration
(venv) $ python manage.py db upgrade

Deployment in production

This section considered deprecated, see DockerDeployment for the suggested deployment model.

Reverse-proxy and Application server

Flask's built-in server is not suitable for production. There are quite a few deployment options for production environment, both self-hosted and PaaS.

Being WSGI application, Flask requires WSGI application server (such as uWSGI or Gunicorn), which usually works in conjunction with a reserve-proxy server such as Nginx that serves static files and manages requests. That takes the load off the application server and guarantees better performance:

Client request <-> Reverse-Proxy <-> Application Server (127.0.0.1:port OR socket)
    ^                   |
    └--- static files --┘

Configuration examples

There are configuration examples under:

examples/

These examples include:

  1. Celery systemd service file:
    • pili-celery.conf
    • pili-celery.service
  2. Nginx configuration:
    • pili-nginx.conf
  3. uWSGI systemd service file, uWSGI ini-config file:
    • pili-uwsgi.conf
    • pili-uwsgi.ini
    • pili-uwsgi.service
  4. Git hooks for deployment from a repository:

Permissions

Aforementioned systemd service file examples get use of two directories:

/var/log/pili
/var/run/pili

The best way to create these directories is using the following systemd directives:

PermissionsStartOnly=true # run ExecStartPre with root permissions
ExecStartPre=-/usr/bin/mkdir -p /var/log/pili
ExecStartPre=-/usr/bin/mkdir -p /var/run/pili

Using systemd service files

When tailored to your needs, provided systemd service files can be used this way:

  1. Go to systemd's directory for custom unit files:

    $ cd /etc/systemd/system
    
  2. Create a symlink to a unit file:

    $ ln -s /var/www/pili/your.service your.service
    
  3. Reload systemd daemon:

    $ sudo systemctl daemon-reload
    
  4. Start your service with:

    $ sudo systemctl start your.service
    
  5. Make sure it's running:

    $ sudo systemctl status your.service
    
  6. If service has failed, take a look at systemd's logs:

    $ sudo journalctl -xe
    

Usage

Script options

In addition to providing an apllication entry point manage.py provides several other options to be used with (venv) $ python manage.py option command:

test Run unit-tests test --coverage Run unit-tests with the coverage statistics (report is generated under tmp/coverage directory) profiler Start the application under the code profiler (25 slowest function included by default) profiler --length=N Include N slowest function in profiler report profiler --profile-dir=DIR Save profiler report in the file under DIR initialize Create all databases, initialize migration scripts before deploying deploy Run deployment tasks (to be run after initialize tasks are done) db Perform database migrations shell Run a Python shell inside Flask application context runserver Run the Flask development server i.e. app.run()

Running shell in application context

For testing purposes it's recommended to run Python REPL inside application context with the Flask-Script built-in shell command:

(venv) $ python manage.py shell

Examples:

Look up a body of the comment with id 10:

>>> Comment.query.filter(Comment.id==10).first().body

Get a list of users with the role 'Writer':

>>> [u for u in Role.query.filter(Role.name == 'Writer').first().users]

Get a list of comments to the post with id 111:

>>> [c for c in Post.query.filter(Post.id == 111).first().comments]

Get a list of replies to the comment contining a word 'flask':

>>> [r for r in Comment.query.filter(Comment.body.like("%flask%")).first().replies]

Get a parent comment of the reply with id 29 (parent attribute exists due to backref='parent' in models):

>>> Comment.query.filter(Comment.id == 29).first().parent

Get all replies written by the user 'Pilosus' in descending order (sort by the time of publication):

>>> user = User.query.filter(User.username == 'Pilosus').first()
>>> Comment.query.join(Reply, Comment.author_id == User.id).\
... filter(Comment.parent_id.isnot(None), User.id == user.id).\
... order_by(Comment.timestamp.desc()).all()
>>>
>>> # the same but more concise
>>>
>>> Comment.query.filter(Comment.parent_id.isnot(None), Comment.author == user).\
... order_by(Comment.timestamp.desc()).\
... all()

Get all replies to the comment with id 23:

>>> Comment.query.get(23).replies

Get a thread of all replies to the certain comment:

|- Comment 1
|- Comment 2
|    |- Comment 4
|    |    |- Comment 6
|    |
|    |- Comment 5
|
|- Comment 3

>>> # Use Depth-First Search algorithm for graphs,
>>> #              implemented as a static method
>>>
>>> Comment.dfs(Comment.query.get(2), print)
>>> <Comment 4>
>>> <Comment 6>
>>> <Comment 5>

Get all post likes by the user with id 1, exclude comment likes:

>>> Like.query.filter(Like.user_id==1, Like.comment_id == None).all()
>>> Like.query.filter((Like.user_id==1) & (Like.comment_id == None)).all()

Get information about 'users' table:

>>> User.__table__.columns
>>> User.__table__.foreign_keys
>>> User.__table__.constraints
>>> User.__table__.indexes
You can’t perform that action at this time.