Implement Reddit scraper #159
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements Reddit scraping capabilities using the Apify API, adding a new Reddit job type to the worker's capabilities.
Key changes:
- Added Reddit scraper job type with support for URL scraping, post search, user search, and community search
- Refactored Apify client interface to enable reuse across different scrapers
- Updated type definitions and capabilities to support Reddit functionality
Reviewed Changes
Copilot reviewed 21 out of 23 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/client/apify_client.go | Added Apify interface and refactored types to use uint instead of int for pagination |
| internal/jobserver/worker.go | Added error logging improvements for job execution |
| internal/jobserver/jobserver_test.go | Added WorkerID field to test job structures |
| internal/jobserver/jobserver.go | Registered new Reddit scraper job type |
| internal/jobs/webscraper.go | Changed log levels from Info to Debug/Warn for reduced verbosity |
| internal/jobs/twitterapify/client.go | Updated to use new Apify interface and uint types |
| internal/jobs/twitter.go | Cleaned up unused function and updated import statements |
| internal/jobs/redditapify/client_test.go | Comprehensive tests for Reddit Apify client functionality |
| internal/jobs/redditapify/client.go | New Reddit Apify client implementation |
| internal/jobs/reddit_test.go | Unit tests for Reddit scraper job execution |
| internal/jobs/reddit.go | New Reddit scraper implementation |
| internal/capabilities/detector.go | Added Reddit capabilities detection when Apify key is available |
| internal/api/routes.go | Improved error message clarity |
| go.mod | Added temporary replace directive for tee-types dependency |
| api/types/reddit/reddit_test.go | Tests for Reddit response type marshalling/unmarshalling |
| api/types/reddit/reddit_suite_test.go | Test suite setup for Reddit types |
| api/types/reddit/reddit.go | Reddit data type definitions and JSON marshalling logic |
| api/types/job.go | Added Reddit configuration support and Job string method |
| api/types/encrypted.go | Enhanced error messages with proper error wrapping |
| README.md | Added documentation for Reddit job types and capabilities |
| Makefile | Added test target for Reddit functionality |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Collaborator
|
Depends on gopher-lab/tee-types#15 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Add jobs for scraping Reddit
Testing
Tested with
curlon the devbox, using an Apify API key