This project is an end-to-end application that consist of (1) static no-code front-end; (2) stateless back-end that serves request to and from an instance of ArchiveBox; and (3) CI pipeline to test and deploy a custom Docker image.
- archiveso
- Overview
- Things to learn and research
- Place, Affordance, Connection
- Workflow
- Usage
- Shaping
- Building
- Limitation
- Methodology
- Project Structure
- Troubleshooting
- Reference
In no particular order, my plan is to use the following resources to learn and research.
| Title | Author | Publisher Date [Short Code] |
|---|---|---|
| E-Book: Hands-On Docker for Microservices with Python | Jaime Buelta | Packt 2019 [Buel2019] |
| E-Book: Shape Up | Ryan Singer | Basecamp 2021 [Sing2021] |
| Lowdefy Documentation | Lowdefy | 2022 [Lowd2022] |
| Python API Usage - ArchiveBox Documentation | ArchiveBox | 2022 [Arch2022] |
-
Places users can navigate
- Lowdefy app e.g.
https://archiveso.netlify.app - Python Flask server (with cloudflared as a service) e.g.
https://archiveso.markit.work - Docker Hub e.g.
https://hub.docker.com/repository/docker/dennislwm/archiveso
- Lowdefy app e.g.
-
Affordance users can act
- Docker pull
docker pull dennislwm/archiveso:latest
- Docker pull
This project uses several methods and products to optimize your workflow.
- Use a version control system (GitHub) to track your changes and collaborate with others.
- Use a static analyzer (prospector) to help write your clean code.
- Use a build tool (Makefile) to automate your development tasks.
- Use a package manager (Pipenv) to manage your dependencies.
- Use a testing framework (pytest) to automate your testing.
- Use a containerization platform (Docker) to run your application in any environment.
- Use a continuous integration pipeline (CircleCI) to automate your static analysis and image deployment.
- Use an artifactory (Docker Hub) to store and pull your image.
- Use a low-code framework (Lowdefy) to build your dashboard.
- Use a continuous deployment infrastructure (Netlify) to automate your front-end hosting.
- Use a secured tunnel (cloudflared) to manage your back-end locally.
Before running the Lowdefy app on your local workstation, you need the following:
- Configure a MongoDB cluster.
- Configure an OpenID Connect provider.
- Configure the Lowdefy environment.
Click here to get started.
- Open a new terminal and run a Docker container with the Python Flask server on your local workstation. This command pulls the image from Docker Hub if it doesn't exist.
docker run --rm -d -p 8080:8080 --name objArchiveso dennislwm/archiveso:latest- Run the
cloudflaredtunnel to expose your Docker container on port8080to a custom domain, e.g.https://archiveso.mydomain.com.
cloudflared tunnel --config ~/.cloudflared/config-archiveso.yml runThe tunnel remains opened as long as your don't terminate (CTRL-C) the connection.
- Open a new terminal and build the Lowdefy app.
cd front
npx lowdefy@latest build- Run the Lowdefy app on your local workstation.
npx lowdefy@latest dev
As shaping and building have independent cycles, we will define shaping as any work that does not involve implementation of code. This work may include evaluation, feasibility, comparison, research, etc.
We set a time constraint of 9 workdays, for shaping, and an additional 9 workdays for building. Hence, the total time for this project is approximately 20 workdays with a cool-down of 2 workdays.
- Create a single REST endpoint with Python (2)
- Create test cases for each REST endpoint with Python (1)
- Create a Dockerfile and test it locally (1)
- Require status checks to pass before merging PR (1)
- Build, test, tag and upload our web app image using CI (1)
- Create a front-end for our web service (2)
- Expose our web app using Cloudflare Tunnel (1)
This project started on 26-Jan-2022 and has been completed on 15-Feb-2022 (13 days), which was ahead of the expected completion date on 24-Feb-2022. The shaping cycle was completed on 8-Feb-2022 (8 days), while the building cycle took 5 days.
This steps are repeatable, i.e. expose and test an endpoint and build a page to allow user interaction.
- Build a Lowdefy page to allow user interaction (1)
- Build and test an API endpoint to execute a command and return its response (2)
- The CI pipeline does not perform the last mile delivery, i.e deploy the Docker container Python Flask server app after it has been tagged and pushed to Docker Hub. The Continuous Delivery ["CD"] pipeline is not implemented.
- Currently, I'm hosting the Docker container on my local workstation, i.e. http://localhost:8080 and using Cloud Tunnel to access it via https://archiveso.mydomain.com.
- Shaping and building have independent cycles.
- Shaping requires a level of abstraction that is neither too concrete (wireframe) nor too abstract (words).
- Time constraint as an appetite for both shaping and building.
- Integrate one slice from UI to backend during shaping.
- Make a bet for what to build and honor it.
- Choose either a base, bull or bear cycle length with cool-down.
- Break a project into independent scopes, each with a smaller period, on a hill chart.
- Each independent scope should include details such as places a user can navigate (S3 bucket), affordances a user can act (/api/upload), and connections a user are taken to (api/upload -> S3 bucket).
- Integrate all scopes to form a minimal intentional product [MIP].
root
|- README.md <-- This README markdown file
|- .gitignore <-- Git ignore file
+- doc/ <-- Holds documentation files
+- img/
|- shape.drawio
+- app/
|- Pipfile <-- Requirements file for app
|- Pipfile.lock
|- Makefile <-- Makefile for app, image and dependency
|- main.py <-- Python file for app
|- Dockerfile <-- Dockerfile for app
|- docker-compose.yml <-- YAML file for CI static-analysis
+- src/
+- archiveso/
|- __init__.py
|- clsArchiveso.py
+- test/
+- .circleci/ <-- Holds CircleCI files
|- config.yml <-- Config file for CI pipeline implementation
+- front/ <-- Holds Lowdefy front-end static files
|- lowdefy.yaml <-- YAML file for Lowdefy root schema
+- auth0/
+- components/ <-- Holds YAML files for Lowdefy reusable components
+- pages/ <-- Holds YAML files for Lowdefy pages
- Argo Tunnel error
Your cloudflared tunnel is not running as a service.
cloudflared tunnel --config ~/.cloudflared/config-archiveso.yml runThe following resources were used as a single-use reference.