git clone https://github.com/Tawheed-tariq/crypto-scrapper
cd crypto-scrapper/crypto_data
install all requirements
pip3 install -r requirements.txt
To run the spider use
scrapy crawl crypSpi
Follow this step-by-step section to set up Playwright in Scrapy.
First create a project folder and a virtual environment inside it
mkdir <project_name>
cd <project_name>
virtualenv venv
Activate and enter the environment (command for linux)
source ./venv/bin/activate
Now, install scrapy and scrapy-playwright
pip3 install scrapy scrapy-playwright
create a scrapy project
scrapy startproject <project_name>
also install browser in playwright
playwright install
or
playwright install <browser_name>
Open setting.py
and add the following lines to configure ScrapyPlaywrightDownloadHandler
as the default http/https
handler. In other words, Scrapy will now perform HTTP or HTTPS requests through Playwright.
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
You also need the following line to enable the asyncio-based Twisted reactor
, yet the most recent versions of Playwright already have it in setting.py
.
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
By default, Playwright operates in headless mode. If you'd want to see the actions performed by your scraping script in the browser, add this value:
PLAYWRIGHT_LAUNCH_OPTIONS = {
"headless": False,
"timeout" : 60000,
}
if you have ideas for improvements or adding features , feel free to fork , send pull requests or create issues.