Scrapy middleware to handle javascript pages using seleniumbase driver.

Based on scrapy-selenium library, scrapy-seleniumbase was made for those who want to make use of Seleniumbase Driver convenient features.

Installation

pip install git+https://github.com/Quartz-Core/scrapy-seleniumbase

Configuration

Provide keyword arguments for Driver in dict. For example:

SELENIUMBASE_DRIVER_KWARGS = {
"browser": "chrome",
"uc": True,
"headless": True,
"block_images": True,
}

Add the SeleniumBaseMiddleware to the downloader middlewares:

DOWNLOADER_MIDDLEWARES = {
    'scrapy_seleniumbase.SeleniumBaseMiddleware': 800
}

Usage

Use the scrapy_seleniumbase.SeleniumBaseRequest instead of the scrapy built-in Request like below:

from scrapy_seleniumbase import SeleniumBaseRequest

yield SeleniumBaseRequest(url=url, callback=self.parse_result)

The request will be handled by seleniumbase, and the request will have an additional meta key, named driver containing the seleniumbase driver with the request processed.

def parse_result(self, response):
    print(response.request.meta['driver'].title)

For more information about the available driver methods and attributes, refer to the selenium python documentation (all vanilla selenium driver methods are available) and seleniumbase documentation (look for "driver" specific methods, located at the end of the page).

The selector response attribute work as usual (but contains the html processed by the selenium driver).

def parse_result(self, response):
    print(response.selector.xpath('//title/@text'))

Additional arguments

The scrapy_selenium.SeleniumBaseRequest accept 5 additional arguments:

`wait_time` / `wait_until`

When used, webdriver will perform an Explicit wait before returning the response to the spider.

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    wait_time=10,
    wait_until=EC.element_to_be_clickable((By.ID, 'someid'))
)

`screenshot`

When used, webdriver will take a screenshot of the page and the binary data of the .png captured will be added to the response meta:

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    screenshot=True
)

def parse_result(self, response):
    with open('image.png', 'wb') as image_file:
        image_file.write(response.meta['screenshot'])

`script`

When used, webdriver will execute custom JavaScript code.

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    script='window.scrollTo(0, document.body.scrollHeight);',
)

`driver_methods`

When used, seleniumbase webdriver will execute methods, provided as strings in a list, before returning page's html.

def start_requests(self):
    for url in self.start_urls:
        yield SeleniumRequest(
                        url=url,
                        driver_methods=['''.find_element("xpath","some_xpath").click()'''])
)

License

Quartz-Core/scrapy-seleniumbase

ErrorLooks like something went wrong!

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Error
Looks like something went wrong!

Packages