Skip to content
This repository was archived by the owner on Jun 22, 2026. It is now read-only.

fix: harden upstream feed fetching#8

Merged
phatpham9 merged 1 commit into
mainfrom
fix/hackernews-timeout
Jun 22, 2026
Merged

fix: harden upstream feed fetching#8
phatpham9 merged 1 commit into
mainfrom
fix/hackernews-timeout

Conversation

@phatpham9

Copy link
Copy Markdown
Member

Summary

  • replace the Hacker News fetch path from hnrss.org/frontpage to Algolia's direct front-page API
  • apply the configured outbound FEEDREADER_USER_AGENT to upstream requests
  • add a bounded shared retry path for upstream GET fetches used by all feed sources
  • add parser and transport/retry tests for the new behavior

Root cause

The Hacker News source depended on https://hnrss.org/frontpage, an extra third-party hop that intermittently stalled before response headers arrived, surfacing context deadline exceeded (Client.Timeout exceeded while awaiting headers).

The repo also documented FEEDREADER_USER_AGENT, but the HTTP client never actually applied it.

Details

  • internal/sources/hackernews.go
    • switched HN ingestion from RSS-on-hnrss.org to direct Algolia front-page JSON
    • preserved story identity, comments URL, score, comments count, and published time mapping
  • internal/service/service.go
    • wrapped the shared client transport to inject the configured User-Agent unless a request already sets one
  • internal/sources/util.go
    • added bounded retry logic for upstream GETs
    • retries transport errors and retryable status codes (408, 425, 429, 500, 502, 503, 504)
  • internal/sources/{github,huggingface,alphaxiv,hackernews}.go
    • moved source fetches onto the shared retry path
  • tests
    • added HN parser coverage
    • added User-Agent transport coverage
    • added retry helper coverage

Verification

docker run --rm -v "$PWD":/src -w /src golang:1.24 /bin/sh -lc '/usr/local/go/bin/gofmt -w internal/service/service.go internal/service/http_test.go internal/sources/util.go internal/sources/hackernews.go internal/sources/hackernews_test.go internal/sources/http_test.go internal/sources/github.go internal/sources/huggingface.go internal/sources/alphaxiv.go && /usr/local/go/bin/go test ./...'

@phatpham9 phatpham9 merged commit 658d135 into main Jun 22, 2026
1 check passed
@phatpham9 phatpham9 deleted the fix/hackernews-timeout branch June 22, 2026 11:03
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant