v0.7.0

Changelog

• 01d3b41 Merge pull request #12 from shubh1855/feature/redirect-handling
• 8150a3c docs(fix): proper gotcha commands where added
• f0cb7c7 feat(fetch): add configurable redirect handling
• 311d38d feat(fetch): add configurable request rate limiting
• 75933b1 feat(fetch): added configurable redirect limit
• be899e4 feat(fetch): support configurable HTTP redirects
• 2869f66 fix(lint): fixed a few missed errors for fmt.Fprint
• 0223e62 test(fetch): add redirect handling tests

v0.6.0

Changelog

• c1319e2 Merge pull request #10 from shubh1855/feature/crawl-depth
• 1279d85 Merge pull request #11 from shubh1855/feature/request-delay
• 86c886e ci: added golangci-lint action
• 5aa1d44 doc: fixed grammar mistakes
• 0784f54 doc: updated readme for new version release
• 240866c feat(fetch): added configurable request delay
• 2b457d8 feat: crawling depth feature. Using --depth flag
• b8b2f00 fix(CI): added golangci-lint version 2.12
• 08266af tests: add crawl depth tests

v0.5.0

Changelog

• 9269373 Added a function to get the page data given an URL and return a struct.
• d00cc5d Added basic fetching of Sites.
• e271ccb Added code check to see the number of arguments provided to the program
• 0ee506e Added json based report generation
• 9c63a7b Adding configurable user agent. (#3)
• 3fb20e4 Create LICENSE
• bef53ad Feature/user agent (#4)
• 145c9fa Merge branch 'main' into feature/user-agent
• 4bc3837 Merge pull request #1 from shubh1855/feature/error-handling
• d59818d Merge pull request #5 from shubh1855/feature/structured-logging
• af6b800 Merge pull request #6 from shubh1855/feature/cli-flags
• 3df204f Merge pull request #7 from shubh1855/feature/version-info
• 8c66d66 Merge pull request #8 from shubh1855/feature/github-actions
• bc9c649 chore: added goquery as a dependency
• 0f3e9e1 ci: add automated release workflow
• 1c48791 doc: Added a brief readme with usage and installation steps
• 5b882ec doc: added updated readme for the 0.5.0 version
• aae6cdf doc: changed readme with better details and future plans
• cf3000b doc: updated roadmap
• 99819a4 feat(cli): add version info
• 120f222 feat(cli): allow configuring the user agent
• 1fea40e feat(cli): migrate CLI to use pflag instead of fmt print statements
• aa392d6 feat(crawler): respect robots.txt crawl rules
• 060b7d1 feat(logging): added slog logger
• 8abecbb feat(logging): added structured logging for fetching requests
• 3a2997c feat(parser): added parser to parse the URLs and images from the Html file.
• 4490cb3 feat(robots): parse basic robots.txt rules
• 1dc43a4 feat: Added URL normalizer. > Added test cases for URL normaliser. > Updated go.sum
• 91e09c6 feat: Added proper web crawling. The Crawler will crawl each outgoing link with only the URL as the same baseURL The code logic is currently single threaded
• f26fc5e feat: classify internal and external links (#2)
• bd50df1 feat: feat(crawler): add configurable concurrency and page limits
• 9b764b3 fix(report bug): The keys created earlier added empty reports. Fixed it so that initialized with zero empty ones. New report for each URL is ammended
• 488ff75 fix: isAllowed for robots.txt file parsing. - Disallow wasn't handled - Exact match wasn't handled
• 6c15505 initial commit: Add initial files. > Add a placeholder main.go file. > Add a normalize url code. > Add test cases for it.
• 14a88e6 merge PR to add release workflow using goreleaser
• 4e88d27 refactor(crawler): add thread-safe concurrent crawling - Introduce shared config struct - Convert crawlPage into a config method - Add mutex-protected page tracking - Add WaitGroup synchronization - Add semaphore-based concurrency limiting - Store PageData instead of visit counts
• 64112f6 refactor(fetch): move user agent into crawler configuration
• 1902091 refactor(logging): replace fmt logging with slog