missing 1 required positional argument: 'url'

[1quintana]

Hello I followed the installation guide, but I am having the next error. I am using python 3.7.0

    yield SeleniumRequest(url, self.parse)
  File "/usr/local/lib/python3.7/site-packages/scrapy_selenium/http.py", line 29, in __init__
    super().__init__(*args, **kwargs)
TypeError: __init__() missing 1 required positional argument: 'url'

I added to the settings the next configuration

from shutil import which
SELENIUM_DRIVER_NAME='chrome'
SELENIUM_DRIVER_EXECUTABLE_PATH=which('geckodriver')
SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox

here is my code

def start_requests(self):
      url = "http://egov.sos.state.or.us/br/pkg_web_name_srch_inq.do_name_srch?p_name=aa&p_regist_nbr=&p_srch=PHASE3P&p_print=FALSE&p_entity_status=ACTINA"
      yield SeleniumRequest(url, self.parse)

def parse(self, response):
   rows = response.selector.xpath("//table/tbody/tr")

One more tip, if I use the next way to implement selenium is working with no problem.

chromedriver = os.getcwd()+'/leadsScraper/chromedriver'
driver = webdriver.Chrome(chromedriver)
delay = 10

[clemfromspace]

Hi @1quintana ,

It looks like a difference between SeleniumRequest and the Scrapy Request classes.
This way, it should work (passing explicitly the url and the callback parameters):

def start_requests(self):
      url = "http://egov.sos.state.or.us/br/pkg_web_name_srch_inq.do_name_srch?p_name=aa&p_regist_nbr=&p_srch=PHASE3P&p_print=FALSE&p_entity_status=ACTINA"
      yield SeleniumRequest(url=url, callback=self.parse) # Here pass url=url instead of just url

I will open an issue to fix this thing, thank for your feedback.

[1quintana]

Hello @clemfromspace

Thank you so much for the fast answer. It did work good with your solution.

[wzyboy]

Hi @clemfromspace ,

I have encountered this as well. Passing url as kw arg instead of positional arg does suppress the error, but it seems that the request is no longer processed by the middleware at all: trying to access response.meta['driver'] raises KeyError now.

The browser still opens, but no tabs would open, no webpages would be navigated to, while the console output continues and the crawling goes on without the processing of the middleware.

[wzyboy]

Here is an example:

#!/usr/bin/env python

import scrapy
from scrapy_selenium import SeleniumRequest


class QuotesSpider(scrapy.Spider):

    name = 'quotes'

    custom_settings = {
        'FEED_EXPORT_ENCODING': 'utf-8',
        'SELENIUM_DRIVER_NAME': 'chrome',
        'SELENIUM_DRIVER_EXECUTABLE_PATH': '/usr/local/bin/chromedriver',
        'SELENIUM_DRIVER_ARGUMENTS': ['--user-data-dir=DATA'],
        'SPIDER_MIDDLEWARES': {
            'scrapy_selenium.SeleniumMiddleware': 800
        }
    }

    def start_requests(self):

        #url = 'http://quotes.toscrape.com/'
        url = 'http://quotes.toscrape.com/js/'
        yield SeleniumRequest(url=url, callback=self.parse)

    def parse(self, response):

        #driver = response.meta['driver'].title
        quotes = response.selector.xpath('//div[@class="quote"]/span[@class="text"]/text()').extract()
        for quote in quotes:
            yield {'quote': quote}

[wrightwriter]

Still have this problem