Skip to content

samKenpachi011/bwhistory-app-api

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Botswana History App API

Objectives of this project

  • To create api endpoint for the history of Botswana
  • To document browsable api's using swagger

Technologies

  • Python
  • Django ( admin site, ORM, url mappings)
  • Django Rest
  • PostgreSQL
  • Docker
  • Swagger

Project Structure

  • app -> holds all application code
  • app/core -> code shared between mulitple apps
  • app/user -> user related code
  • app/history -> history related code
  • requirements.txt -> for the application/production
  • requirements.dev.text -> for local development

API Management

  • API removal and deprecation process

Core App structure

  • app/core/tests/
  • app/core/models
  • app/core/admin
  • app/core/apps
  • app/core/migrations/

EthnicGroup App

  • Used for creating, updating and viewing ethnic groups.

EthnicGroup App structure

  • app/ethnicgroup/tests/
  • app/ethnicgroup/urls
  • app/ethnicgroup/serializers
  • app/ethnicgroup/apps
  • app/ethnicgroup/views

Culture App

  • Used for creating, updating and viewing cultures.

Culture App structure

  • app/culture/tests/
  • app/culture/urls
  • app/culture/serializers
  • app/culture/apps
  • app/culture/views

Event App

  • Used for creating, updating and viewing events.

Event App structure

  • app/event/tests/
  • app/event/urls
  • app/event/serializers
  • app/event/apps
  • app/event/views

Chief App

  • Used for creating, updating and viewing chiefs.

Chiefs App structure

  • app/chief/tests/
  • app/chief/urls
  • app/chief/serializers
  • app/chief/apps
  • app/chief/views

Publisher App

  • Used for creating, updating and viewing published documents.

Publisher App structure

  • app/publisher/tests/
  • app/publisher/urls
  • app/publisher/serializers
  • app/publisher/apps
  • app/publisher/views

Sites App

  • Used for creating, updating and viewing sites.

Sites App structure

  • app/sites/tests/
  • app/sites/urls
  • app/sites/serializers
  • app/sites/apps
  • app/sites/views

Artifacts App

  • Used for creating, updating and viewing artifacts.

Sites App structure

  • app/artifacts/tests/
  • app/artifacts/urls
  • app/artifacts/serializers
  • app/artifacts/apps
  • app/artifacts/views
Syntax Description
docker-compose run --rm app sh -c "python manage.py startapp user" Create a user app

EthnicGroup App image api

Syntax Description
Pillow Contains all the basic image processing functionality
zlib , zlib-dev used for data compression

User App - Used for creating auth tokens, user registration, updating and viewing profiles - API Status Codes Guide

Syntax Description
docker-compose run --rm app sh -c "python manage.py startapp user" Create a user app

Authentication

  • We are using a custom user model to allow for changes moving forward and also to use an email for the username.

  • We implementing the token authentication for its simplicity and security

  • Token Authetication Guide

    • Custom User Model Classes
    • set AUTH_USER_MODEL in settings.py for django to use the custom user model.
    Syntax Description
    AbstractBaseUser Provides the core implementation of a user model.
    BaseUserManager Base class for managing users and has helper methods.
    PermissionsMixin Support for the django permission system.
    UserManager Allows creation of custom logic for creating objects.
    create_user Custom method called when creating a user.
    create_superuser Custom method used to create superusers.
    • User Api
    • There are configurations that allow for the management of users
    Syntax / Files Description
    Serializers Serializers are responsible for converting complex data
    (e.g., querysets and model instances) to native Python datatypes that can then be rendered into
    JSON, XML, or other content types.
    Views Determine how requests will be handled and which policy attributes to use.
    urls.py We include all the endpoints that have view classess
  • Creating a super user docker-compose run --rm app sh -c "python manage.py createsuperuser"


Migrations -Django handels database structure and changes

  • Migrations are handled by django -> python manage.py makemigrations

  • Applying migrations python manage.py migrate

  • Using docker compose docker-compose run --rm app sh -c "python manage.py wait_for_db && python manage.py migrate"

    • Migration issues
    Description Possible Solution
    0001_initial is applied before is dependency clear the devdb volume
    django.db.utils.ProgrammingError: relation "core_user" does not exist clear migrations and make migrations again for the core app

Project Management

  1. Test Driven Development

  • A development practice to write test for functionalities before implementation
  1. Pre-Installs/Configuration

    • Vs-Code / any IDE
    • GIT
    • Set up Docker and Docker-Compose
    • Setup linting
    • Configure Github actions

  1. Documentation

Developers or any user of our APIs needs to know how to use them.

  • Documentation is both Manual and Automatic(endpoints) -> tools

  • What we have documented

    • Available endpoints
    • Application docker configurations
    • Authentication process
    • Endpoint payloads and responses
  1. Testing

    • We use the Django test suite and set tests per Django app

    Steps

    • Import test classes either SimpleTestCase(no database) or TestCase(database) Django Testing Tools

    • Import objects to test

    • Define test class

    • Add test method that will fail

    • Setup inputs

    • Execute code and check failing output

    Useful Links

    Running tests

    Syntax Description
    docker-compose run --rm app sh -c "python manage.py test" Using docker-compose to run application tests
    import pdb; pdb.set_trace() Option 1 to debug
    breakpoint() Option 2 to bebug
  2. Linking the Githup repo to DockerHub


  1. Adding a Docker file and dockerignore

Why Docker

  • Helps to capture all dependencies as code which leads to consistent development and production environments
  • Docker Best Practices

A .dockerignore is a configuration file that describes files and directories that you want to exclude when building a Docker image.

Docker file Description

Syntax Description
FROM python:3.9-alpine3.13 Defining the base image
ENV PYTHONUNBUFFERED 1 To print outputs from the running application to the console
COPY ./source /destination Copy local files to the directory specified in the container
WORKDIR /dir Set the working directory as the default directory to run all commands from
EXPOSE Expose the specified port on the container to our machine
RUN We are using a format to allow us to run multiple commands e.g installing requirements, adding a user to the container
ENV PATH=":$PATH" Adding our executables path to the system path
USER Setting the user inside the container

Docker Compose file Description

Syntax Description
docker-compose up This runs services inside the docker compose file
services Define services that will run containers or any job.
build: context: . Builds the docker file inside the current directory.
args Build arguments, which are environment variables accessible only during the build process.
ports: Specify port mappings such that we connect port 8000 on our local machine to port 8000 inside the container.
volumes: - ./app:/app Map the local app directory to sync with the running container files.
command: >
sh -c "python manage.py runserver 0.0.0.0:8000" Command to run the service which can be overridden from the terminal.
depends_on Creates a network link between services
environment Used to define environment variables
volumes: Named volumes are used here to map container files from one service locally

Why environment variable

  • Easily passed to docker to be used in both local development and production
  1. Linting

    Using Flake 8

    Syntax Description
    docker-compose run --rm app sh -c "flake8" Running flake8

  1. Project Structure

Syntax Description
docker-compose run --rm app sh -c "django-admin startproject app ." Start project in the current directory
docker-compose run --rm app sh -c "django-admin startapp core" Start app

Management Commands

file Description
wait_for_db_command.py We are fixing a database service race issue so that we wait for all the db subfuctions to finish

  1. GitHub Actions

Github Actions Guide Action Features

Why github actions

  • We use github actions to run jobs for (deployment, unit-tests, code linting)
  • Add steps for running test and linting -> .github/workflows/checks.yml
  • Authenticate with docker hub and add secretsto the github project
  • Update or add the DOCKERHUB_TOKEN and DOCKERHUB_USER secrets on github

  1. Postgres DB access

syntax Description
docker-compose up -d run all services in the background
docker ps get the container id running the db
docker exec -it <container_id> //bin/bash access the container using bash
psql -U -d <DB_name> access the psql shell by specifying the username and database name
\c <db_name> connect to the db
\dt list tables

Using PgAdmin

  • define your connection -> create a new server
  • add the database name
  • add the port that postgres container is running on

Steps

  • Create the neccessary docker and docker-compose files.
  • Add requirements files and create the app directory.
  • run docker build . to build the docker image.
  • run docker-compose build to use docker compose to build and tag the image or after updating the requirements.txt.
  • Add flake 8 to requirements.dev.txt and create the .flake8 to ignore application files inside app/ from linting.
  • Add args to the docker-compose file and update the docker file to install dev requirements when in development.
  • Add a database service with environment variables set to the docker-compose file
  • Configure postgresql for django by installing adaptors and update the settings files
  • To fix database race issues create a core app and add a management command to check if the database is ready
  • Add a custom authentication user model and include the admin configs (list, add)
  • Add a user app for creating users, tokens and managing profiles
  • Add a ethnic group app for creating,updating and viewing ethinic groups.
  • Add a tag feature for tagging ethnic groups
  • Add a image api for ethnic groups
  • Add a filtering feature for ethinic groups
  • Add a culture app for creating, updating and viewing different cultures
  • Add a tag feature for tagging cultures
  • Add a image api for culture and create a custom action to upload the image
  • Add a event app for creating, updating and viewing events
  • Add a chief app for creating, updating and viewing chiefs information
  • Add a publisher app for creating, updating and viewing published documents
  • Add a sites app for creating, updating and viewing sites information
  • Add a artifacts app for creating, updating and viewing artifact information

Contributions

  • After cloning the repo change to the dev branch and create pull requests from there. -- git branch -M dev
  • Check the available issues -> pick an issue to manage.
  • Write tests for the feature or bug fix.
  • Submit a pull request.