🔗 Follow URLs

[roniemartinez]

Resolves #62 (easier alternative, no code change for decorated functions)

TODO

• Extract URLs
• Fix relative paths
• [ ] Prevent following external URLs
• Prevent duplicates Prevent requesting current URL
• Specify allowed domains (For now, allowed domains are extracted from URL input.)

NOTES

This is just a simple Spider/Crawler for now. Will update in the future.

[codecov-commenter]

Codecov Report

Merging #90 (d07a92b) into master (d067127) will decrease coverage by 2.51%.
The diff coverage is 86.23%.

@@            Coverage Diff             @@
##           master      #90      +/-   ##
==========================================
- Coverage   99.24%   96.72%   -2.52%     
==========================================
  Files          13       13              
  Lines         929     1070     +141     
==========================================
+ Hits          922     1035     +113     
- Misses          7       35      +28     

Impacted Files	Coverage Δ
dude/__init__.py	100.00% <ø> (ø)
dude/scraper.py	100.00% <ø> (ø)
dude/playwright_scraper.py	82.85% <46.15%> (-12.95%)	⬇️
dude/base.py	97.75% <86.20%> (-2.25%)	⬇️
dude/optional/parsel_scraper.py	98.21% <93.75%> (-1.79%)	⬇️
dude/optional/lxml_scraper.py	98.24% <93.93%> (-1.76%)	⬇️
dude/optional/beautifulsoup_scraper.py	99.09% <96.66%> (-0.91%)	⬇️
dude/optional/pyppeteer_scraper.py	98.19% <100.00%> (+0.28%)	⬆️
dude/optional/selenium_scraper.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d067127...d07a92b. Read the comment docs.