Feat/astro fr

[chris-c-thomas]

This pull request introduces significant improvements and fixes across the LexBuild monorepo, especially for the Federal Register (FR) pipeline. The major highlights include implementing a bulk XML downloader for the Federal Register, adding concurrent downloads for faster throughput, and addressing several conversion and formatting issues. Additionally, documentation and developer notes have been updated to reflect these changes and to provide best practices for batch processing and Meilisearch operations.

Federal Register Downloader and Conversion Enhancements

• Implemented a bulk XML downloader for the Federal Register, enabling efficient downloading of large document sets.
• Added support for concurrent downloads in the FR downloader, replacing sequential fetching and significantly improving speed; CLI now exposes a --concurrency flag.
• Fixed FR conversion logic and addressed formatting issues with both raw and rendered FR text output. [1] [2]
• Improved downloader progress status reporting and reduced the request delay from 100ms to 25ms for 4x faster throughput.
• Fixed a parser error affecting multiple packages.

Documentation and Developer Experience

• Updated documentation with new scripts, usage notes, and best practices:
  • Added guidance for using gray-matter with { cache: false } in batch scripts to prevent memory leaks. [1] [2]
  • Clarified Meilisearch full reindexing and incremental indexing scripts, and documented the need for database deletion on version upgrades. [1] [2] [3]
  • Updated README with more precise descriptions of data sources, corpus statistics, and the monorepo structure. [1] [2] [3] [4] [5] [6]
• Minor updates to CLAUDE.md files for better accuracy and clarity.

[sonarqubecloud]

Quality Gate failed

Failed conditions
3 Security Hotspots
10.6% Duplication on New Code (required ≤ 3%)
D Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE