fix(auto_source): pin request budget for token-based auto-source feeds - #1062
Merged
Conversation
Explicitly set max_requests=4 in token_generator_input to match the html2rss gem auto_source default. Previously the gem policy default of 1 prevented sitemap sub-fetches from firing. No breaking change: requests beyond 1 previously returned empty feeds.
gildesmarais
commented
Aug 10, 2026
Co-authored-by: Gil Desmarais <gildesmarais@users.noreply.github.com>
gildesmarais
changed the title
gildesmarais
enabled auto-merge (squash)
August 10, 2026 15:02
gildesmarais
added a commit
to html2rss/html2rss
that referenced
this pull request
Aug 10, 2026
…uests` to 4 (#411) ## Summary Adds sitemap index (`sitemapindex`) discovery and fan-out support to auto-source (e.g. `github.blog`, Yoast SEO-generated sitemaps, YC blog). ### Background & Context 1. **Sitemap Index Support**: Many modern sites serve a `sitemap_index.xml` containing `<sitemap><loc>` elements pointing to sub-sitemaps (e.g., `post-sitemap.xml`) rather than direct `<url><loc>` elements. `Discovery::Sitemap` previously parsed flat `<urlset>` documents, returning `[]` for index documents. 2. **Request Budget for Auto Mode**: `auto_source` defaulted to `max_requests: 1` (inherited from `Policy::DEFAULTS`), which prevented follow-up sub-sitemap HTTP fetches from executing out of the box. --- ## New Capabilities & Changes - **`Discovery::Sitemap`** ([lib/html2rss/auto_source/discovery/sitemap.rb](file:///Users/gil/versioned/html2rss/html2rss/lib/html2rss/auto_source/discovery/sitemap.rb)): - Returns a `Result` value object (`entries:` and `sub_sitemap_urls:`). - Automatically detects `<sitemapindex>` documents and extracts child sitemap URLs in document order (naturally prioritizing post/news sitemaps). - **`Scraper::Sitemap`** ([lib/html2rss/auto_source/scraper/sitemap.rb](file:///Users/gil/versioned/html2rss/html2rss/lib/html2rss/auto_source/scraper/sitemap.rb)): - Implements fan-out through `sub_sitemap_urls` up to `MAX_SUB_SITEMAPS = 3`. - Catches `RequestBudgetExceeded` gracefully to log an operational warning and yield all accumulated entries without crashing. - **`Html2rss.auto_source` & `auto_json_feed`** ([lib/html2rss.rb](file:///Users/gil/versioned/html2rss/html2rss/lib/html2rss.rb)): - Defaults `max_requests: 4` for auto-sourcing (1 initial page + 1 root sitemap + up to 2 sub-sitemaps), while keeping the conservative `max_requests: 1` policy default for explicit feed configs. --- ## Verification - **RSpec**: 1,156/1,156 unit tests passing (0 failures). Added test coverage for sitemapindex parsing, scraper fan-out, `MAX_SUB_SITEMAPS` capping, and budget exhaustion. - **RuboCop**: 255 files inspected, 0 offenses detected. - **YARD Lint**: 100% documentation coverage. - **Quality Gate**: `make ready` passed with exit code 0. --- ## Related - Paired with `html2rss-web` PR: html2rss/html2rss-web#1062
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
html2rss-web calls Html2rss.feed (not Html2rss.auto_source), so the gem method-level max_requests default of 4 does not apply here. Without this change, token-backed auto-source feeds inherit Policy::DEFAULTS[:max_requests]=1, blocking sitemap sub-fetches. Pins max_requests=4 explicitly in token_generator_input. No breaking change.