0.0.5 - Adding ability to scale

Improved Task Managent

Lots of little changes around backend to give us the ability to add a significant number of sources, mostly around task management and adding lots of fall back options for all tasks

Changes

• Added a list of acceptable tags and now reject any tag added that isn't on that list to ensure we don't end up with LLM hallucinations causing things like 'TV' 'Television' etc
• Completely changed task management from a simple cron to using Celery which allows us to run much smaller tasks more often, handle failures appropiately and only call all of our scripts when needed
• Continued to break out everything into utility files and makes our scripts as small as possible
• Changed 'feed' to 'topic' to make it a bit clearer
• Added ability to show tags and scores on frontend for debugging
• Improved our LLM workflows, adding descriptions and other prompt adjustments
• Added Gemini Flash as a key part of our workflow and in general added lots and lots of LLM redundancy to all tasks
• Created first pass of BDD which will be a big part of the next push into testing

Full Changelog: 0.0.4...0.0.5

0.0.4 - Backend is here, again

Full Changelog: 0.0.3...0.0.4

Backend reimagined

A complete refactoring of the backend, completely overhauled with almost nothing preserved. The aim is to give us the ability to rapidly add in backup scripts for each task so that we can rapidly add in new feed sources

Changes

• Created utility files for all functions. We have a utility file for all task specific utilities, as well as general purpose, shared utilities like logging, database management & setups for each large language model source
• Centralized configuration into a configuration python and yaml files. This is where things like order of scripts, environment management and things like that are handled
• Defined interfaces for each job
• Created a redundancy manager and broke everything into four tasks, 'fetch_urls' 'scraper' 'summarizer' 'tagging'. This calls the relevant scripts, order defined in our config.yaml, for each task
• Created a main.py, when we want to call all tasks or a subset of any, we call it via main.py using arguments, if needed, to call specific tasks. All tasks are initiated in main.py it handles calling and reporting the status of them
• Removed the old cron job approach, and integrated Celery as our task manager
  • It runs the fetch_urls task every 2 minutes
  • If 0 urls are returned, nothing happens. If 1 or more are returned, it adds all remaining tasks to our queue, to be run back to back. It checks and runs whatever is in the queue every minute
  • Every 30 minutes, it runs all tasks anyway
• Reduced our scripts, which were the bulk of our code, to as thin as possible. So we can easily and quickly add as many scripts to a task as we want
• Finally added requirements.txt
• Added a thorough documentation run in readme.md covering the entire codebase
• Created a staging branch, which pushes to our server into a staging folder

0.0.3 - frontend is a little prettier

Pretty-ing up the frontend

• Used TailwindUI to create a compotent looking website, still lots of improvements, but it almost looks like a real website
• Added an auto generating /topic/[tag] route, showing articles with that tag
• Build components for header (navbar), CardHeader (the header above ArticleList), ArticleList, & Modal. Plus added a layout component just to have a max width we can put components into
• All pages have a navbar, a list of articles with a header, and infinite scroll. Navbar has a login link which shows a 'coming soon' modal. Tag pages have a 'subscribe' CTA with a 'coming soon' modal for accounts
• Removed pagination (And deleted component) and used infinite scroll - need to add a loading animation for initial fetch and all infinite scroll fetches, though
• Set up Storyboard for easy visualizing and messing around with components
• Displaying tags on ArticleList component, which links to their feed page. Temporarily showing ranking scores for future debugging and curiosity

0.0.2 - Frontend is here, again

On the backend...

• Changed LLM output and DB to support new frontend approach
  • Changed format of bullet point output to JSON
  • Created a Supabase table for tagging
  • Set up LLM to output tags and scores, in JSON, which push to that table. The tagging and scoring is absolutely awful, but that's expected, I just needed it to be in a consistent format - will make it good later
  • Added 'productionready' bool for frontend to know when a row is ready to be used, set by completion of tagging
• Adjusted the logic of the Claude fallback summarizing llm script to fire not just if summarized is false, but also if any of the intro, bullet points or concluding paragraphs are empty to counteract the bug where llama just repeats tokens and doesn't give complete output

On the frontend...

• Completely removed Wordpress install, and setup next.js app instead
• Set up connection with Supabase, set up realtime fetching from there upon page render
• Built an article component (zero styling) to render our articles
  • Shows 20 articles initially with pagination for more. Renders titles on a feed, titles are href to original article. Expand opens accordion which shows intro paragraph, bullet points formatted as such, and concluding paragraph. Users can close accordion
• Set up pages.tsx to render that component and nothing else so we can see
• Zero styling, but functionally it does what it is intended to do
• Added posthoq component (analytics) and added that to header of all pages
• Hosted site on Vercel, pointing jamiecostello.net to master branch, and staging.jamiecostello.net to staging branch

Frontend is here!

Frontend is here

Summary

Almost no movement in Github, but a huge milestone as we have a (functional, but awful in almost every way) frontend environment

Original scope for this release

• Set up a frontend environment of some kind
• Have a 'feed' with all stories in it. For now, could just sort them in chronological order. A next step could be to ask the LLM to grade the interest of the story out of 10, both for it's feed and globally (eg this is a 8 for AI enthusiasts, a 2 for the general public) and sort that way

Result

• Wordpress is up and running on my server. It's able to pull from Supabase and create a custom post type for each article with the data points all in there as separate ACF fields
• We can then create pages that pull from those posts in whatever way we want, and render the content in real time
• Right now it's a home feed, pulling all of them, and rendering in chronological order, which will dynamically update as articles come in. You see a list of headlines, and can click on them and see the summaries
• It looks like shit due to a default template with no editing, and I need to adjust the llm output so I can properly render the bullet points, but it works!
• http://jamiecostello.net/home-feed/ is the url (Domain is temporary until I name this)

0.0.01

MVP for databses done, enough for us to move onto starting to build the bones of a frontend for our alpha 0.0.1

0.0.0

• Used The Verge as our playground
• Use a Python script calling feedparser. We can feed an RSS feed into it, and have it extract whatever fields we want. The Verge only has 10 articles in its field, for other ones, we might need some logic that includes the data, and either here or in step two filters to only a certain timeframe (last 24 hours?). It extracts these URLs into a txt file
• Use a js script using a headless browser called puppeteer. Takes the URLs from feedparser, and then dumps the contents of the article into individual txt files
  • This also worked using curl, which is another option. Curl won't grab anything rendered in javascript (so short codes etc), but that's likely a non issue for this kind of job. Curl also returns the full html, so uses more tokens, and that potentially impacts the results and that's why I didn't use it
• Then call Claudes Haiku model, for cost efficiency, for each article on there and ask it for the summary in the format we want.
• Created a Python script then to just run each script sequentially, so it's a one click process
• For now, this creates txt files every step of the way. Powering a website likely requires another approach, but will get to that later
• For now, the proof of concept works and actually automated even more than I expected

Full Changelog: https://github.com/jfcostello/article-summarizer/commits/0.0.0