New Dev Onboarding

Tech Stack

This repo primarily uses the following tech stack:

• Language
  • Python3 (3.6 and above, preferably 3.7 and above) as the general programming language for development and scripting
• Web framework
  • Django 3.0.6+ as the web API framework
• CI/CD pipeline
  • Jenkins
• Deployment
  • Docker for the containerization
  • Kubernetes for container orchestration and deployment
• Database
  • PostgreSQL for general database
  • TimeScale for time-series database (TSDB)

Development in Python

TL;DR

• Use Python3.7 for development and production
• Use virtual environment for local development (e.g. virtualenv)
• Use pip3 for dependency management
• Follow PEP8 for coding styles

Version

• In principle, use 3.7
• Experiments on newer versions for recent features are welcome
• 3.6.9 is also acceptable since it is the default on Ubuntu 18.04 and the NoisePage testing scripts are also run with such version
• No Python 2 any more

Virtual Environment

• Why to use virtual environment in Python?
• How to set up a virtual environment with virtualenv
  • Ensure you have Python3.7 on your mac or windows
  • Ensure you have pip3 on your mac or windows
  • Install virtualenv: pip3 install virtualenv
  • Create a virtual environment: virtualenv env -p $(which python3.7)
  • Enter the virtual environment: source env/bin/activate
  • Install the dependencies (if any): pip3 install -r requirements.txt
  • Exit the virtual environment: deactive

Lint and Coding Styles

• Follow PEP8
  • Luckily, there are some tools can help you to conform your code to PEP8
  • Yapf (I have already added this in the requirements.txt, which you should have it upon installing the dependencies).
• Use Pylint to conduct static analysis for your code (I have already added this in the requirements.txt, which you should have it upon installing the dependencies).

Environments

We use the following environments for this repo:

• local: Your local env for active development and testing.
  • Use Docker and minikube for the local environment
• test: A compact but complete environment with all the service dependencies for integration tests
  • Hosted at incrudibles-testing.db.pdl.cmu.edu
• staging: A production-ish environment for beat feature demonstration, user acceptance testing and load testing
  • Hosted at incrudibles-staging.db.pdl.cmu.edu
• production: The live environment of the testing infra, for real uses
  • *Server not yet ready

Git

Branches and Environments

• feature branch: use JIRA board story number and name as the branch name. For example, for the story GEN-82 about the Django web api service, the feature branch name should be like GEN-82-boilerplate-django
  • local env
• test branch: test
  • testing env
• staging branch: staging
  • staging env
• master branch: master
  • production env

Checkout Rule

In principle, for the sake of the stability, a new feature branch should be checkout from the master branch only. However, in the same sprint where different stories may depend one another, a feature branch can be checked out from staging branch as well. Though such behaviour is discouraged and unsafe.

Merging Rule

For the non-workflow features (e.g. features that are not for the GitHub, jenkins, ansible, or any part of the CI/CD pipeline), please follow the following merging rules

• Upon the completion of the development and basic unit testing of a story
  • You should submit a pull request from the feature branch to the test branch and assign a reviewer for code review.
  • After the code review is approved, the merge action will trigger Jenkins build for static analysis and unit tests
  • After the Jenkins build is passed, the pull request is completed
• After thorough integration tests
  • You should submit a pull request from the test branch to the staging branch
  • Similar PR process as above
• After validation from Andy
  • The release manager of the week (round-robin) should submit a pull request from the staging branch to the master branch
  • Similar PR process as above
  • Update the release version by git tags For the workflow related features (e.g. features that are for the GitHub, jenkins, ansible, or any part of the CI/CD pipeline that requires testing upon merging, you can should, in theory, test on your own feature branch with your test jenkins pipelines for the workflow before making a PR.

Tagging and Versions

We use the following format for the versions:

v<MAJOR>.<MINOR>.<PATCH>

Where

• Major version represents a major change or upgrade of the testing infra (e.g. change of the API)
• Minor version represents a sprint release
• Patch represents hotfixes