Issue: Selenium Webdriver parameter executable_path deprecated

[aaravindan101]

System Info

langchain version 0.0.273
selenium version 4.11.2

In version 4.11.2, Selenium fully deprecated and removed executable_path parameter from webdriver in favor of using Service class object to pass in path/executable_path parameter. This results in a crash when using SeleniumURLLoader with executable_path upon upgrading selenium to 4.11.2.

Who can help?

@eyurtsev

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

Upgrade selenium to version 4.11.2. Load SeleniumURLLoader with executable_path parameter.

Expected behavior

Expected Result: SeleniumURLLoader is instantiated
Actual Result: ERROR: WebDriver.init() got an unexpected keyword argument 'executable_path'

[dosubot]

🤖

Thank you for bringing this issue to our attention. It seems that the 'executable_path' parameter has indeed been deprecated in Selenium version 4.11.2, and this is causing the SeleniumURLLoader in LangChain version 0.0.273 to crash.

To resolve this issue, you can modify the _get_driver method in the SeleniumURLLoader class to use the Service class from Selenium. The Service class is used to start the browser driver service and it takes the path to the driver executable as an argument.

Here is how you can modify the _get_driver method:

def _get_driver(self) -> Union["Chrome", "Firefox"]:
    """Create and return a WebDriver instance based on the specified browser.
Raises:
        ValueError: If an invalid browser is specified.

    Returns:
        Union[Chrome, Firefox]: A WebDriver instance for the specified browser.
    """
    if self.browser.lower() == "chrome":
        from selenium.webdriver import Chrome
        from selenium.webdriver.chrome.options import Options as ChromeOptions
        from selenium.webdriver.chrome.service import Service as ChromeService

        chrome_options = ChromeOptions()

        for arg in self.arguments:
            chrome_options.add_argument(arg)

        if self.headless:
            chrome_options.add_argument("--headless")
            chrome_options.add_argument("--no-sandbox")
        if self.binary_location is not None:
            chrome_options.binary_location = self.binary_location
        if self.executable_path is None:
            return Chrome(options=chrome_options)
        return Chrome(service=ChromeService(self.executable_path), options=chrome_options)
    elif self.browser.lower() == "firefox":
        from selenium.webdriver import Firefox
        from selenium.webdriver.firefox.options import Options as FirefoxOptions
        from selenium.webdriver.firefox.service import Service as FirefoxService

        firefox_options = FirefoxOptions()

        for arg in self.arguments:
            firefox_options.add_argument(arg)

        if self.headless:
            firefox_options.add_argument("--headless")
        if self.binary_location is not None:
            firefox_options.binary_location = self.binary_location
        if self.executable_path is None:
            return Firefox(options=firefox_options)
        return Firefox(service=FirefoxService(self.executable_path), options=firefox_options)
    else:
        raise ValueError("Invalid browser specified. Use 'chrome' or 'firefox'.")

In this modification, we import the Service class from selenium.webdriver.chrome.service and selenium.webdriver.firefox.service for Chrome and Firefox respectively. Then, instead of passing executable_path directly to the Chrome or Firefox constructor, we create a Service instance with executable_path and pass it to the service parameter of the Chrome or Firefox constructor.

Please note that this issue is still present in the current version of LangChain (v0.0.274). We will consider this modification for the next release of LangChain. Thank you for your patience and understanding.

Sources

• libs/langchain/langchain/document_loaders/url_selenium.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[aaravindan101]

Wow, great investigation from Dosu-bot!

Tried to create a pull request with my suggestion already but unable to create pull request, my recommended changes are:

    def _get_driver(self) -> Union["Chrome", "Firefox"]:
        """Create and return a WebDriver instance based on the specified browser.

        Raises:
            ValueError: If an invalid browser is specified.

        Returns:
            Union[Chrome, Firefox]: A WebDriver instance for the specified browser.
        """
        if self.browser.lower() == "chrome":
            from selenium.webdriver import Chrome
            from selenium.webdriver.chrome.options import Options as ChromeOptions
            from selenium.webdriver.chrome.service import Service

            chrome_options = ChromeOptions()
            chrome_service = Service()

            for arg in self.arguments:
                chrome_options.add_argument(arg)

            if self.headless:
                chrome_options.add_argument("--headless")
                chrome_options.add_argument("--no-sandbox")
            if self.binary_location is not None:
                chrome_options.binary_location = self.binary_location
            if self.executable_path is None:
                chrome_service.path = self.executable_path
            return Chrome(options=chrome_options, service=chrome_service)
        elif self.browser.lower() == "firefox":
            from selenium.webdriver import Firefox
            from selenium.webdriver.firefox.options import Options as FirefoxOptions
            from selenium.webdriver.firefox.service import Service

            firefox_options = FirefoxOptions()
            firefox_service = Service()

            for arg in self.arguments:
                firefox_options.add_argument(arg)

            if self.headless:
                firefox_options.add_argument("--headless")
            if self.binary_location is not None:
                firefox_options.binary_location = self.binary_location
            if self.executable_path is None:
                firefox_service.path = self.executable_path
            return Firefox(options=firefox_options, service=firefox_service)
        else:
            raise ValueError("Invalid browser specified. Use 'chrome' or 'firefox'.")

[dosubot]

🤖

Thank you for reaching out and for your contribution to LangChain! Your proposed solution to the issue with the SeleniumURLLoader and the deprecation of the 'executable_path' parameter in Selenium seems like a step in the right direction.

To create a pull request with your changes, you can follow these general steps:

1. Fork the LangChain repository on GitHub.
2. Clone your forked repository to your local machine.
3. Create a new branch for your changes.
4. Make the necessary modifications to the _get_driver method in the SeleniumURLLoader class.
5. Commit your changes with a descriptive commit message.
6. Push your branch to your forked repository on GitHub.
7. Open a pull request from your branch to the main LangChain repository.

Please note that the LangChain maintainers will review your pull request and decide if it should be merged. They may provide feedback or request further changes before merging.

If you encounter any specific issues or need further assistance during the process, feel free to ask. Happy contributing!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot]

Hi, @aaravindan101! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue you reported was regarding the executable_path parameter being deprecated and removed in Selenium version 4.11.2. This was causing a crash when using SeleniumURLLoader. However, I provided a solution by modifying the _get_driver method in the SeleniumURLLoader class to use the Service class from Selenium.

I see that you attempted to create a pull request with your suggested changes, but encountered difficulties. Don't worry, I've provided guidance on how to create a pull request to address the issue.

Now, we would like to know if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution and we appreciate your understanding! Let us know if you have any further questions or need any assistance.