An analytics platform that monitors the Electronic Code of Federal Regulations (eCFR), providing real-time insights into regulatory evolution across federal agencies.
https://ecfr-analyzer-chi.vercel.app/word-counts
http://localhost:3000/word-counts
- Agency Word Counts: A visualization of word counts across federal agencies, featuring:
- Interactive bar chart with horizontal/vertical toggle
- Raw data display option
- Calculates aggregate word counts for agencies and their subordinate organizations
- Comprehensive agency hierarchy representation by word count
- Scrapes agency word counts from the eCFR API versioner endpoint and saves them to the database
components/ecfr/WordCountDisplay.tsx
https://ecfr-analyzer-chi.vercel.app/agency-titles
http://localhost:3000/agency-titles
- Agency CFR Titles & Text: An advanced search interface that:
- Parses complex CFR references (title/subtitle/chapter/subchapter/part/subpart/section) from source XML and matches them to agency regulations
- Displays mapped truncated subset text of all related CFR references
- Leverages scraped data to display agency word counts along with the full text of all related CFR titles
https://ecfr-analyzer-chi.vercel.app/title-compare
http://localhost:3000/title-compare
- Historical Title Changes: A comparison tool that:
- Allows selection of specific titles and date ranges
- Fetches and parses XML data directly from the eCFR API
- Provides visual diff analysis of regulatory changes of title between selected dates
- Displays the full text of the CFR title between the selected dates
- Database: PostgreSQL in Supabase
- Backend: Django REST API
- Frontend: Next.js/React with TypeScript
- Deployment: Railway (backend) and Vercel (frontend)
- Dev:
- bun for Next.js/React (https://bun.sh/guides/ecosystem/nextjs)
- biomejs for frontend linting
- uv for Python/Django (https://blog.pecar.me/uv-with-django)
- ruff for backend linting
- Visualization: Highcharts
- Features efficient data fetching and caching strategies
https://ecfr-analyzer-chi.vercel.app/
Django REST API backend deployed to https://ecfr-analyzer-production.up.railway.app/api/ through Railway
https://github.com/railwayapp-templates/django/tree/main
Next.js/React with TypeScript frontend deployed to https://ecfr-analyzer-chi.vercel.app/ through Vercel
https://nextjs.org/docs/app/api-reference/cli/create-next-app
PostgreSQL database through Supabase
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
$ brew install bun uv biome ruff
$ brew install --cask postgres-unofficial
$ export PATH=/Applications/Postgres.app/Contents/Versions/@latest/bin/:$PATH
$ git clone git@github.com:JohnDDuncanIII/ecfr-analyzer.git
$ bun install
$ cd ecfr-django
$ uv sync
<new tab>
$ cd ecfr-analyzer
$ bun --bun run dev
<new tab>
$ cd ecfr-analyzer/ecfr-django
$ uv run manage.py runserverOpen http://localhost:3000.
<new tab>
$ cd ecfr-analyzer/ecfr-django
$ uv run manage.py dbshell
<new tab>
$ cd ecfr-analyzer/ecfr-django
$ uv run manage.py shell_plus --ipython --print-sql
$ uv run manage.py scrape_agencies
$ uv run manage.py scrape_cfr_text --title <title_number> --include-headers (optional)
$ uv run manage.py update_agencies_wordcountsThere are overlapping Title/Chapter cfr_references for the following Agencies:
- Title 40, Chapter VII is shared by:
- Department of Defense
- Department of Energy
- Title 47, Chapter II is shared by:
- National Security Council
- Office of Science and Technology Policy
- Title 47, Chapter IV is shared by:
- National Telecommunications and Information Administration
- National Highway Traffic Safety Administration
- Title 50, Chapter IV is shared by:
- National Oceanic and Atmospheric Administration
- United States Fish and Wildlife Service
{
"name": "National Telecommunications and Information Administration",
...
"cfr_references": [
// this no longer exists the ecfr as of 2017-09-20 and was ignored in http://localhost:3000/agency-titles
// https://www.ecfr.gov/current/title-15/chapter-XXIII
{
"title": 15,
"chapter": "XXIII"
},
]
}{
"name": "Interstate Commerce Commission",
...
"cfr_references": [
// this no longer exists the ecfr as of 2024-12-26 and was ignored in http://localhost:3000/agency-titles
// https://www.ecfr.gov/current/title-5/chapter-XL
{
"title": 5,
"chapter": "XL"
}
]
}In [1]: lst = list(Agency.objects.order_by("short_name").distinct("short_name").values_list("short_name", flat=True))
{
"name": "Gulf Coast Ecosystem Restoration Council",
// this short_name is potentially wrong — should be GCERC
"short_name": "Gulf Restoration Council",
}{
"name": "Military Compensation and Retirement Modernization Commission",
// this short_name is seemingly wrong — should be "MCRMC"
"short_name": "Military Compensation and Retirement Modernization Commission",
}This API endpoint is strange — you would expect it to return truncated XML for queries with a date, title, and subtitle/chapter/subchapter, but even if you include any one of the three aforementionedsubset parameters, the endpoint returns the entire title's XML.
While this is explicitly stated in the API docs, it's not clear to me why the developers decided not to truncate the XML responses if a the subtitle/chapter/subchapter is provided.
I do this work manually in CFR Text Scraper.
For example, it's reasonable to assume https://www.ecfr.gov/api/versioner/v1/full/2025-02-06/title-1.xml?chapter=II or https://www.ecfr.gov/api/versioner/v1/full/2025-02-06/title-49.xml?subtitle=A would return the XML for just the chapter, subchapter, or subtitle, but it actually returns the entire title's full XML.
Requests can be for entire titles or part level and below. Downloadable XML document is returned for title requests. Processed XML is returned if part, subpart, section, or appendix is requested.
Code of Federal Regulations (ECFR) (Used to scrape 2024 data and to display 2010-2023 historical data through /api/versioner/v1/full/{date}/title-{title}.xml endpoint in Title Compare feature)
https://www.ecfr.gov/api/admin/v1/agencies.json
Federal Register
Code of Federal Regulations (CFR) (Unused, bulk 2010-2023 data)