0.0.4 - Backend is here, again

Full Changelog: 0.0.3...0.0.4

Backend reimagined

A complete refactoring of the backend, completely overhauled with almost nothing preserved. The aim is to give us the ability to rapidly add in backup scripts for each task so that we can rapidly add in new feed sources

Changes

• Created utility files for all functions. We have a utility file for all task specific utilities, as well as general purpose, shared utilities like logging, database management & setups for each large language model source
• Centralized configuration into a configuration python and yaml files. This is where things like order of scripts, environment management and things like that are handled
• Defined interfaces for each job
• Created a redundancy manager and broke everything into four tasks, 'fetch_urls' 'scraper' 'summarizer' 'tagging'. This calls the relevant scripts, order defined in our config.yaml, for each task
• Created a main.py, when we want to call all tasks or a subset of any, we call it via main.py using arguments, if needed, to call specific tasks. All tasks are initiated in main.py it handles calling and reporting the status of them
• Removed the old cron job approach, and integrated Celery as our task manager
  • It runs the fetch_urls task every 2 minutes
  • If 0 urls are returned, nothing happens. If 1 or more are returned, it adds all remaining tasks to our queue, to be run back to back. It checks and runs whatever is in the queue every minute
  • Every 30 minutes, it runs all tasks anyway
• Reduced our scripts, which were the bulk of our code, to as thin as possible. So we can easily and quickly add as many scripts to a task as we want
• Finally added requirements.txt
• Added a thorough documentation run in readme.md covering the entire codebase
• Created a staging branch, which pushes to our server into a staging folder