A simple script that scrapes a website, extracting texts in a CSV file with the format below, and saving images.
| Page | Tag | Text | Link | Image |
|---|---|---|---|---|
| page path | element tag (h{1,6}, a, p, etc) | text content | link url (if any) | image address (if any) |
First, install dependencies (python3):
pip install -r requirements
Then create a file containing urls of the websites you want to scrape, one line for each website, for example (I'll call this file test_websites):
https://theread.me
https://theguardian.com
Now you are ready to execute the script:
python index.py test_websites
# ^ path to your file
After the script is done with it's job, you can find the results in results/<website_hostname> folder.
To see available options, try python index.py -h.