Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github-scraper to download all issues/PRs and keep them in sync #4

Open
milahu opened this issue Feb 7, 2023 · 1 comment
Open

github-scraper to download all issues/PRs and keep them in sync #4

milahu opened this issue Feb 7, 2023 · 1 comment

Comments

@milahu
Copy link
Owner

milahu commented Feb 7, 2023 •
edited
Loading

aka "live mirror" of github issues and PRs

better than the online version, because we can use custom search

should be a daemon = run continuously, watch github events, fetch incremental updates

should use the official github API (dont scrape html)

different than #1 (webhooks, push events) because use polling (pull events)
for clients with no access to the nixos org (to configure webhooks)

https://docs.github.com/en/rest/activity/events

question: how fine-grained are pull events?

keywords

  • github sync issues
  • github issues downloader
  • github scraper
  • github watcher
  • github archiver
  • github offline mirror
  • github proxy
  • github bot
  • github client

https://docs.github.com/en/rest/guides/scripting-with-the-rest-api-and-javascript

incremental web scraping

https://stackoverflow.com/questions/2576785/web-crawler-update-strategy

keywords

  • event driven web scraping
  • web scraping live data
  • real time web scraping
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@milahu