oClubs

Introduction

This website is hosted in SHSID internal network at http://connect.shs.cn/.

The IP address of this website is 192.168.8.201.

File Hierarchy

• /: Root directory, at /srv/oclubs/repo
  • oclubs/: Main oclubs code (almost everything other than configuration is here)
    • access/: Low level accessing of various cache/configuration/database/storage backends
      • database.py: Main relational database
      • delay.py: helper functions
      • elasticsearch.py: Search engine
      • email.py: Email handling, to Postfix and SendGrid, write-only
      • fs.py: Filesystem, for watching files
      • redis.py: Redis key-value storage
      • secrets.py: Secret settings (passcodes and tokens) access via ini, read-only
      • siteconfig.py: Web-adjustable site configuration access via ini.
    • blueprints/: High level flask code handling various suburls
      • actblueprint.py: http://connect.shs.cn/activity
      • clubblueprint.py: http://connect.shs.cn/club
      • userblueprint.py: http://connect.shs.cn/user
    • email_templates/: Email templates written in Mustache
    • objs/: Python class definitions of high level classes, mostly self-explanatory
      • base.py: Definition on how the classes work
    • static/: Static, usually Gulp-processed files to be served directly by nginx
    • static-dev/: development files for Gulp to output to static/
      • css/: PostCSS code to compile to CSS
      • css-build/: CSS before minification, output by Gulp
      • images/: Images before minification
      • js/: JavaScript before minification
    • templates/: HTML templates, written in Jinja2
      • activity/ & club/ & user/: Corresponds to their blueprints
      • static/: Corresponds to main app.py
      • base.jinja2: Base layout for every page
      • macros.jinja2: Jinja2 macros
    • app.py: Main flask code
    • enums.py: Database Enums
    • exceptions.py: oClubs errors
    • redissession.py: Server side session via redis
    • shared.py: Shared code for Flask
    • worker.py: Celery worker code
  • provision/: Test VM configuration, may also guide server configuration
    • manifests/: Puppet code
    • the rest: static configurations for various services
  • browserlist: Configuration on what browsers we attempt to support
  • Gulpfile.js: Gulp CSS/JS build & minification instructions
  • oclubs-tables.sql Definition of Relational database tables
  • package.json: Node dependencies for Gulp building
  • requirements.in: Raw (for human editing) python dependencies for running the oclubs server
  • requirements.txt: Versioned (for actual machine configuration) python dependencies, generated by pip-compile
  • shell.py: Program invoked by pyshell, to spawn an python interactive shell with many already imported
  • Vagrantfile: Test VM configuration

Architecture

System

Centos 6 on 64-bit system

Services

• uWSGI & Flask: Serves dynamic webpages

• Celery: Backend worker

• Elasticsearch: Current search engine, runs on Java

• MariaDB 10: MySQL replacement, current main relational database storage

• Nginx: Current fast HTTP reverse-proxy, serves custom static files directly, and proxies & caches to some other CDN services for non-custom static files, and proxies to uwsgi for dynamic content

• Postfix: Current main SMTP server for email-delivery

• Redis: Current fast key-value storage with native expiry support, for eg. cache & temporary data storage

• SendGrid (External): Current email relay for difficult-to-deliver situations (such as to Gmail)

• iptables: Firewall

Contributing

Many softwares are required for development, testing, and deployment of this site.

Development

A good IDE

Of course. You may use whatever you like. If you're having trouble making choices:

• For an old-school coder, which I don't expect for students, Vim & Emacs might suit you well.
• For newcomers, I recommend Sublime Text 3 or Atom.

Git command line interface

Believe me, command line is easier to use and more feature-rich than graphical interfaces. And really, read the friendly manual (RTFM).

Best Practice: In case of a conflict, use rebase instead of merge.

Gulp

For build & minification. How to setup:

1. Install NodeJS & Npm
2. Go to command line (yes I'm serious)
3. cd (change current working directory) to oclubs root directory
4. npm install -g gulp-cli, on some installs, you many need to sudo
5. npm install

For builds, just run (in a command line interface with oclubs root directory as current working directory): gulp, and Gulp will watch all target files and do what it's meant to do.

(TL;DR: just keep gulp running if you intend to change the static CSS/JS/Images)

Testing

A supported web browser with developer tools

Having more than one browser is recommended.

Common browsers include:

1. Chrome
2. Firefox
3. Safari
4. Edge
5. Internet Explorer (older versions do not support bootstrap)

Vagrant

Vagrant + VirtualBox is the current way to virtualize the server for local testing. How to setup:

1. Install Vagrant & VirtualBox
2. Grab the latest CentOS 6 box file from https://atlas.hashicorp.com/centos/boxes/6

• You may wish to inject vagrant-box-download-helper.js by everyx to get the download links.
• Use a fast downloader such as XunLei or axel.
• Download it to anywhere, but preferably not within oclubs root directory.

3. Go to command line,
4. Run vagrant plugin install vagrant-vbguest. This plugin is required to install VirtualBox Guest Additions for shared folders, which puppet configuration still uses.
5. cd to wherever that .box file is downloaded, then run vagrant box add centos/6 <filename.box> with <filename.box> as the downloaded filename.
6. cd to oclubs root directory, then vagrant up. Puppet and vagrant will then configure the VM. In case of failed configuration, run vagrant provision
7. An oclubs clone should be viewable in browser at http://127.0.0.1:8080/

Files are shared between VM and host with VirtualBox Shared Folders. Everything should work except for sendfile-related accesses, which is patched in our nginx.conf; please refer to https://www.vagrantup.com/docs/synced-folders/virtualbox.html.

To access the VM's shell, run vagrant ssh.

• You may need to do sudo service uwsgi reload within the shell for each time the python scripts are changed.
• Commands within the shell are basically similar to that of the actual server, so don't sudo rm -rf --no-preserve-root / unless you know what you're doing. (Yeah it deletes the whole filesystem.)

Deployment

SSH client

We use SSH for remote login to server. Linux knowledge is highly recommended. Remember to check if things break!

For Windows: Use PuTTY For Mac / Linux: Use OpenSSH

This section is TODO...