Add types for Reddit scraper

[mcamou]

No description provided.

[Copilot]

Pull Request Overview

This PR adds comprehensive type definitions and support for Reddit scraping functionality. It introduces Reddit-specific job types, capabilities, validation logic, and custom JSON marshaling/unmarshaling for polymorphic Reddit item types.

• Adds Reddit job type and capabilities to the job system with validation
• Implements polymorphic Reddit item types (User, Post, Comment, Community) with custom JSON handling
• Creates comprehensive test coverage for Reddit argument validation and type marshaling

Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
types/reddit.go	Core Reddit types with polymorphic JSON marshaling for different item types
types/jobs.go	Extends job system with Reddit job type and capabilities
args/reddit.go	Reddit job arguments with validation and default value handling
pkg/util/set.go	Enhances Set methods to return self for method chaining
types/reddit_test.go	Tests for Reddit item JSON marshaling/unmarshaling
args/reddit_test.go	Tests for Reddit argument validation and defaults
args/unmarshaller.go	Adds Reddit job argument unmarshaling support
args/unmarshaller_test.go	Tests for job argument unmarshaling including Reddit
types/types_suite_test.go	Test suite setup for types package
args/args_suite_test.go	Test suite setup for args package

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The method assignment to q.Method only modifies the local copy since Go passes structs by value. This means the validation passes but the original struct is not updated. Use a pointer to modify the original: r.URLs[i].Method = strings.ToUpper(r.URLs[i].Method)

Suggested change

	errs = append(errs, fmt.Errorf("invalid Reddit URL %s", q.URL))
	for i := range r.URLs {
	r.URLs[i].Method = strings.ToUpper(r.URLs[i].Method)
	if r.URLs[i].Method == "" {
	r.URLs[i].Method = "GET"
	}
	if !allowedHttpMethods.Contains(r.URLs[i].Method) {
	errs = append(errs, fmt.Errorf("%s is not a valid HTTP method", r.URLs[i].Method))
	}
	u, err := url.Parse(r.URLs[i].URL)
	if err != nil {
	errs = append(errs, fmt.Errorf("%s is not a valid URL", r.URLs[i].URL))
	} else {
	if !strings.HasSuffix(u.Host, redditDomainSuffix) {
	errs = append(errs, fmt.Errorf("invalid Reddit URL %s", r.URLs[i].URL))

[Copilot]

Similar to the previous issue, this assignment only affects the local copy. Use r.URLs[i].Method = "GET" to modify the original struct.

Suggested change

	errs = append(errs, fmt.Errorf("invalid Reddit URL %s", q.URL))
	for i := range r.URLs {
	r.URLs[i].Method = strings.ToUpper(r.URLs[i].Method)
	if r.URLs[i].Method == "" {
	r.URLs[i].Method = "GET"
	}
	if !allowedHttpMethods.Contains(r.URLs[i].Method) {
	errs = append(errs, fmt.Errorf("%s is not a valid HTTP method", r.URLs[i].Method))
	}
	u, err := url.Parse(r.URLs[i].URL)
	if err != nil {
	errs = append(errs, fmt.Errorf("%s is not a valid URL", r.URLs[i].URL))
	} else {
	if !strings.HasSuffix(u.Host, redditDomainSuffix) {
	errs = append(errs, fmt.Errorf("invalid Reddit URL %s", r.URLs[i].URL))

[rapidfix]

this slightly confuses me, why not unmarshalling on r directly?

[mcamou]

According to #7 (comment), this pattern is a go idiom to prevent infinite recursion w/ custom UnmarshalJSON methods... go will call your custom unmarshal method when using json.Unmarshal, which then uses json.Unmarshal...

I'll add an appropriate comment.

[rapidfix]

Oh I see. OK cool 👍

[rapidfix]

should be actually safe to call errors.Join instead of appending and checking at the end. errors.Join checks if the error is nil and discards it:

Join returns an error that wraps the given errors. Any nil error values are discarded. Join returns nil if every value in errs is nil. The error formats as the concatenation of the strings obtained by calling the Error method of each element of errs, with a newline between each string.

https://pkg.go.dev/errors#Join

[rapidfix]

Just small nits, but overall should be safe to go