Mininet Repo Docs

Introduction

Mininet Repo is a website for Mininet users and module developers to use and share modules. The source code can be checked out here. Make sure to checkout the gsoc branch.

Mininet Repo is developed heavily based on djangopypi2. The main reason of choosing djangopypi2 is because the packaging system we decided to use is based on plain old Python packaging system, so a PyPI clone will be most suitable as a sharing website.

djangopypi2 is specifically chosen out of so many other clones because the code looks reasonably well done and because it uses Django, a very popular web development framework for Python, so documentation is abundant.

Disclaimer

Many things are already implemented by djangopypi2 when I decided use it as a base. It is, however, by no means complete. I have to make quite a lot of changes to make it to the current state. Changes that deserve to be in original djangopypi2 are always pushed there for its benefit as well. Changes I made are listed here. This document attempts to document Mininet Repo as a whole, not just the part I implemented.

Features

Core Features

• Provide website interface to download, search, finding details, manage permission of packages
• Provide interface for command-line tools to upload/download/query packages

Other Features

• User registration / login with GitHub account

Implementation

Core Features

djangopypi2, by virtue of being a Django project is made of several Django apps. These are:

• pypi_frontend: The app which implements the views for distutils, XML-RPC and normal HTTP request. Note that the views dealing with packages are defined by pypi_packages instead.
• pypi_metadata: The app which defines the models of the packages metadata. It also provides the fixtures for the classifiers.
• pypi_packages: The app which defines the model of Package and Release. It also implements the views for packages and releases.
• pypi_ui: The app which stores the base templates which is overridden by every other apps.
• pypi_users: The app which deals which showing user list and user details. The model of user is the default Django User model, which is why this apps is quite simple.

Setup

On setup, djangopypi2 creates the database tables, load the initial fixtures, and copies the static files to user settings folder (~/.djangopypi2/ on default setting). On first run, djangopypi2 initializes the user settings file settings.json with the default values if it cannot be found in the user settings folder.

Package & Release

Each package can have multiple released versions. A release is usually associated with a file located inside the user settings folder under the media/dists/ folder. Each package can have multiple owners and maintainers.

Both package maintainers and owners may edit metadata or upload additional files associated with a release. However, only package owners may delete both package and releases. Only package owners may add or remove owners and maintainers.

Other Features

User Registration / Login with GitHub Account

I simply use the excellent django-allauth library to implement this. All I have to do is simply configure it and write the templates for the various user management forms. django-allauth uses the Django User as the model for the users, which is a great thing.

Internally, when a new user chooses to login with GitHub account, a new user account with no password will be created in the database associated with the GitHub account. This will force the user to login with GitHub account only on subsequent visit until the user set a password. The GitHub token associated with the session will also be stored in the database. From then on, when a user attempts to login with his GitHub account, django-allauth will see that the user has already associated that GitHub account with a certain User account and allow that certain user to login.

There is also a page for existing non-GitHub account users to associate their GitHub account when they want to as well. This same page can be used to disassociate their GitHub account as well.

User Interface and Client-side JavaScript

Bootstrap framework is used to make the website looks reasonably good with minimum effort. jQuery is used simply because many common JavaScript tasks are easier with it.

Unit Tests

For unit testing, Django testing feature is used. mock is also used for mocking.

Note that pypi_ui doesn't need to be tested as it only stores templates. pypi_metadata only defines models and forms via Django API and does not contain any logic, so writing tests for it is not the best use of time.

Tests for pypi_frontend, pypi_packages, and pypi_users are written. For pypi_frontend, the distutils views are not tested because I'm not sure how to test it.

The test cases can be run by using these commands:

• python manage_pypi_site.py test pypi_frontend
• python manage_pypi_site.py test pypi_packages
• python manage_pypi_site.py test pypi_users

This book is a very good reference on how to test Django application.

Deployment

I followed this tutorial when initially trying to deploy the website.

To make it easier to repeat the deployment, a setup script has been written in Python to automatically clone and setup the server on Ubuntu 12.04. These are the software used as a server:

• Gunicorn runs djangopypi2 locally.
• Supervisor starts Gunicorn process and restart it on failure.
• nginx as a reverse proxy which directs the traffics to the Gunicorn process.

Although I tried to make the setup script does as much as possible, some things either cannot be automated or should be done manually. So these are the steps on how to use the setup script:

• Install a database engine on the server. I recommend PostgreSQL.
• Register a new GitHub application here. Set the callback URL to <homepage_url>/accounts/github/login/callback/.
• Clone the repository, make sure the current branch is gsoc.
• Put the HTTPS certificate and key in djangopypi2/setup/keys/ folder.
• Set the settings for GitHub authentication in djangopypi2/setup/conf/github_oauth.py.
• Change the default values in djangopypi2/setup/config.py. These values are very likely needed to be changed:
• djpp_data_site_*: The name and domain of the website
• djpp_email_*: Settings for email server
• djpp_db_*: Database settings
• gunicorn_user and gunicorn_group: User & group executing the gunicorn process
• supervisor_user: User executing the supervisor process
• nginx_server_name: Server name for nginx
• nginx_cert & nginx_cert_key: Filename for HTTPS certificate and key

Once ready, simply execute the config.py file by using the command python config.py. This is what the script will do:

• Install some required package via apt-get.
• Create a new virtual environment for djangopypi2 in ~/venv/ (if default folder not changed in config.py).
• Make a new clone of djangopypi2 under the virtualenv folder, the gsoc branch is checked out as well.
• Install additional Python package, either using pip or setuptools.
• Copy each of the configuration scripts to its appropriate folder. These are:
• nginx configuration
• Gunicorn configuration
• Supervisor configuration
• Initial database fixtures for djangopypi2
• User settings file for djangopypi2
• Initialize djangopypi2.
• Run nginx, supervisor, gunicorn, djangopypi2.

Migration

In the case of migration, the data files can be easily copied. The uploaded packages, with default settings, are stored in ~/.djangopypi2/media/dists/, so simply copy the content over when migration is needed.

The database content has to be backed-up separately. If using SQLite, simply copy the SQLite database file over and it is done. If using PostgreSQL, this article may be useful.