AI RSS Feeds

This project generates RSS 2.0 feeds for AI news/blog sites that do not publish feeds directly.

You can import all of these feeds into your RSS reader with this feeds.opml file.

Or you can import selected feeds by copying the URL of the XML files in the below table.

Feeds

Name	File
Ai2 News (Allen Institute for AI)	feeds/allenai-news.xml
AISI Blog (AI Security Institute)	feeds/aisi-blog.xml
Anthropic News	feeds/anthropic-news.xml
Anthropic Research	feeds/anthropic-research.xml
* The Batch	feeds/the-batch.xml
Claude Blog	feeds/claude-blog.xml
Cohere Blog	feeds/cohere-blog.xml
Mila News (Quebec AI Institute)	feeds/mila-news.xml
Mistral News	feeds/mistral-news.xml
TLDR AI	feeds/tldr-ai.xml
Turing Blog (Alan Turing Institute)	feeds/turing-blog.xml
Turing News (Alan Turing Institute)	feeds/turing-news.xml

* These feeds are currently broken and not updating.

Developer Guide

Generate Feeds

Use the generator script:

uv run python generate_feeds.py

Generated feed files are written to ./feeds.

By default all feeds are generated, but you can specify which to generate:

uv run python generate_feeds.py aisi-blog allenai-news

Options:

• --no-cache: disable Scrapy HTTP cache for that run
• --skip-unchanged: skip writing a feed file if its only change would be lastBuildDate

The scheduled GitHub Actions workflow uses --skip-unchanged, so it does not create a commit when feeds are otherwise unchanged.

Validation and Failure Behavior

Feed generation is fail-fast.

• If the source URL does not return HTTP 200, that spider raises an error.
• If item_container_selector matches nothing, that spider raises an error.
• If too few items are extracted, that spider raises an error.

By default, each spider enforces:

• min_item_count = 1
• min_item_ratio_vs_previous = 0.6 when an existing feeds/<name>.xml file is present

This prevents silently writing empty or unexpectedly tiny feeds when page markup changes. uv run python generate_feeds.py exits with a non-zero status if any spider errors.

HTTP Cache

Scrapy HTTP cache is enabled by default in src/settings.py. Use uv run python generate_feeds.py --no-cache to disable cache for a single run.

To refresh cached source pages, delete:

rm -rf .scrapy/httpcache

Add A New Feed

1. Add a new [feeds.<feed-key>] table in feeds.toml.
2. Set required fields for HTML feeds:
   • feed_title
   • source_url
   • item_container_selector
   • item_title_selector
   • item_link_selector
3. For Next.js feeds, set:
   • format = "nextjs"
   • item_container_selector as a jq query that returns item objects (for example .page.sections[] | select(._type == "publicationList") | .posts[])
   • item_title_selector, item_link_selector, and optional item_date_selector / item_description_selector as jq queries scoped to each item
4. Set optional fields as needed:
   • item_date_selector, item_date_regex, item_description_selector, feed_description, language
   • item_guid_is_permalink, min_item_count, min_item_ratio_vs_previous
   • save a local source snapshot in snapshots/ and develop selectors against that copy
   • comments above the feed table to keep source/structure notes alongside selectors
5. Add the new feed entry to the table above, keeping it sorted by name.
6. Run uv run python generate_feeds.py and verify output in feeds/.