This program takes online book data on books.toscrape.com and stores it in a local SQLite database. Developed with Linux Mint and Python 3.8. I've written a full write up on my website at https://trumandaniels.com/2022/04/15/building-a-webscraper-with-beautifulsoup/
- bookscraper.py ~ where all the magic happens: this file contains all the scraping logic and function documentation
- test.py ~ a small unit test suite
- books.db ~ the SQLite database where everything is stored
First (Optional) Step: create and activate python virtual environemnt or anaconda environment first (skipping this step could cause package dependency/compatibility issues if your machine is used for many projects)
Changing the working directory to wherever you extract the downloaded folder to
truman@laptop ~ $ cd /path/to/extracted/bookscraper-main
either (after installing virtualenv) choosing a specific python version
truman@laptop ~/bookscraper-main ~ $ virtualenv --python=/usr/bin/python3.9 /path/to/new/environment/VIRTUALENVNAME
or using whatever your PATH is set to (probably fine)
truman@laptop ~/bookscraper-main ~ $ python3 -m venv /path/to/new/environment/VIRTUALENVNAME
truman@laptop ~/bookscraper-main ~ $ source /path/to/new/environment/VIRTUALENVNAME/bin/activate
Then install neccesary packages to run the scripts
(VIRTUALENVNAME) truman@laptop ~/bookscraper-main ~ $ pip install -r requirements.txt
(VIRTUALENVNAME) truman@laptop ~/bookscraper-main ~ $ pytests tests.py
collected 2 items
tests.py .. [100%]
============================== 2 passed in 0.19s ==============================
(VIRTUALENVNAME) truman@laptop ~/bookscraper-main ~ $ python bookscraper.py
Row appended: 1
Row appended: 2
Row appended: 3
...
Row appended: 1000
Done scraping
Sources:
- https://www.digitalocean.com/community/tutorials/how-to-work-with-web-data-using-requests-and-beautiful-soup-with-python-3
- https://www.programiz.com/python-programming/datetime/current-time
- https://stackoverflow.com/questions/36768068/get-meta-tag-content-property-with-beautifulsoup-and-python
- https://www.sqlitetutorial.net/sqlite-python/insert/
- https://stackoverflow.com/questions/4098008/create-table-in-sqlite-only-if-it-doesnt-exist-already
- https://dev.to/albertulysses/unit-testing-your-web-scraper-1aha