browse_website is broken on Apple M1 & M1 Max due to chromedriver mismatch

[Pwuts]

⚠️ Search for existing issues first ⚠️

• I have searched the existing issues, and there is no existing issue for my problem

Which Operating System are you using?

MacOS

GPT-3 or GPT-4?

GPT-4

Steps to reproduce 🕹

Let the GPT execute a browse_website command

Current behavior 😯

Browsing fails because Selenium can't find an appropriate chromedriver for Apple M1 arch. It tries to download chromedriver_mac64_m1.zip, while only chromedriver_mac64.zip exists (see repo).

EDIT: apparently a similar issue occurs on Apple M1 Max which should use mac_arm64 arch.

Your Logs 📒

====== WebDriver manager ======
Current google-chrome version is 112.0.5615
Get LATEST chromedriver version for 112.0.5615 google-chrome
There is no [mac64_m1] chromedriver for browser 112.0.5615 in cache
Trying to download new driver from https://chromedriver.storage.googleapis.com/112.0.5615.49/chromedriver_mac64_m1.zip
SYSTEM:  Command browse_website returned: Error: There is no such driver by url https://chromedriver.storage.googleapis.com/112.0.5615.49/chromedriver_mac64_m1.zip
 THOUGHTS:  It seems like we need to use a web driver to browse the website. I think we should use Selenium to automate the browsing process and extract the information we need.
REASONING:  Selenium is a powerful tool for automating web browsing and can be used to extract information from websites. By using Selenium, we can automate the browsing process and extract the information we need without having to manually navigate the website.

[camolechowski]

I have been hitting a very similar issue, albeit a different error code and @Pwuts suggested I add my logs to this Issue thread:

I am running AutoGPT within a Docker container in continuous mode and everything works great until it tries to access a website and I receive the following error:

SPEAK:  I will use the 'browse_website' command to visit the third website on the list and take notes on important segments that will help me form my thesis on the optimal strategy.
Attempting to fix JSON by finding outermost brackets 
Apparently json was fixed. 
NEXT ACTION:  COMMAND = browse_website ARGUMENTS = {'url': 'https://www.investopedia.com/terms/q/quantitative-trading.asp', 'question': 'optimal strategies for quantitative trading using Pinescript'}
SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

This results in a near-iterative loop as it works through the array of websites it initially finds in the Google API search. I am on a MacBook Pro with an M1 Max chip and I have configured my Docker container to use Pinecone, Google, Hugging Face, and my OpenAI GPT-4 API Key but the issue seems to be solely related to the browse_website function. I have tested on master, stable, and some other branches that claimed to solve similar issues to no avail.

It sounds like another user has ran into the same issue and we are pretty certain it is due to the Mac M1 architecture but we are unclear on how to work around. Any help would be much appreciated!

[camolechowski]

Let me know if anymore information would be helpful!

[Minger]

https://github.com/Significant-Gravitas/Auto-GPT/blob/e2accab87e3b5d339cb5695eb55908ff52b235c3/Dockerfile#L14

[vanities]

I was the guy having issues with @camolechowski.. It seems like it's that particular error is happening on master and stable on my M1 mac while using docker-compose. I've tried removing images and rebuilding multiple times as suggested by other users to no avail. The issue appears to be surrounding https://github.com/Significant-Gravitas/Auto-GPT/blob/0bf4987e1ad71e853fae7fb5f38983428fcb1626/autogpt/commands/web_selenium.py#L96

╰─ docker-compose exec auto-gpt python -it
Python 3.10.11 (main, Apr 12 2023, 11:49:01) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from selenium import webdriver
>>> from webdriver_manager.chrome import ChromeDriverManager
>>>
>>> driver = webdriver.Chrome(ChromeDriverManager().install())
[WDM] - Downloading: 100%|███████| 6.75M/6.75M [00:00<00:00, 24.8MB/s]
<stdin>:1: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/appuser/.local/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriver.py", line 80, in __init__
    super().__init__(
  File "/home/appuser/.local/lib/python3.10/site-packages/selenium/webdriver/chromium/webdriver.py", line 101, in __init__
    self.service.start()
  File "/home/appuser/.local/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 106, in start
    self.assert_process_still_running()
  File "/home/appuser/.local/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 119, in assert_process_still_running
    raise WebDriverException(f"Service {self.path} unexpectedly exited. Status code was: {return_code}")
selenium.common.exceptions.WebDriverException: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

https://github.com/Significant-Gravitas/Auto-GPT/blob/e2accab87e3b5d339cb5695eb55908ff52b235c3/Dockerfile#L14

I just made removed and made sure I had no images built, and rebuilt using [arch=arm64] and it's still giving me

SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

[vanities]

Getting closer!

appuser@9945be5e2d08:~/.wdm/drivers/chromedriver/linux64/112.0.5615.49$ ls -al .
total 20692
drwxr-xr-x 2 appuser appuser     4096 Apr 20 17:36 .
drwxr-xr-x 3 appuser appuser     4096 Apr 20 17:36 ..
-rw-r--r-- 1 appuser appuser   326400 Apr 20 17:36 LICENSE.chromedriver
-rwxr-xr-x 1 appuser appuser 13768328 Apr 20 17:36 chromedriver
-rw-r--r-- 1 appuser appuser  7079001 Apr 20 17:36 driver.zip
appuser@9945be5e2d08:~/.wdm/drivers/chromedriver/linux64/112.0.5615.49$ ./chromedriver
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
appuser@9945be5e2d08:~/.wdm/drivers/chromedriver/linux64/112.0.5615.49$

I do have... /lib/ld-linux-aarch64.so.1

[bobinson]

I am having same issues and logs:

git commit hash : 14d3eca

-=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= 
SYSTEM:  Command browse_website returned: Error: Message: unknown error: Chrome failed to start: crashed. (chrome not reachable) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x0040006e6fe3 <unknown> #1 0x004000425d36 <unknown> #2 0x00400044eb20 <unknown> #3 0x00400044aa9b <unknown> #4 0x00400048caf7 <unknown> #5 0x00400048c11f <unknown> #6 0x004000483693 <unknown> #7 0x00400045603a <unknown> #8 0x00400045717e <unknown> #9 0x0040006a8dbd <unknown> #10 0x0040006acc6c <unknown> #11 0x0040006b64b0 <unknown> #12 0x0040006add63 <unknown> #13 0x004000680c35 <unknown> #14 0x0040006d1138 <unknown> #15 0x0040006d12c7 <unknown> #16 0x0040006df093 <unknown> #17 0x00400258aea7 start_thread

[vanities]

I am having same issues and logs:

git commit hash : 14d3eca

-=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= 
SYSTEM:  Command browse_website returned: Error: Message: unknown error: Chrome failed to start: crashed. (chrome not reachable) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x0040006e6fe3 <unknown> #1 0x004000425d36 <unknown> #2 0x00400044eb20 <unknown> #3 0x00400044aa9b <unknown> #4 0x00400048caf7 <unknown> #5 0x00400048c11f <unknown> #6 0x004000483693 <unknown> #7 0x00400045603a <unknown> #8 0x00400045717e <unknown> #9 0x0040006a8dbd <unknown> #10 0x0040006acc6c <unknown> #11 0x0040006b64b0 <unknown> #12 0x0040006add63 <unknown> #13 0x004000680c35 <unknown> #14 0x0040006d1138 <unknown> #15 0x0040006d12c7 <unknown> #16 0x0040006df093 <unknown> #17 0x00400258aea7 start_thread

I get that error If i change the build arch to emulate amd64

FROM --platform=linux/amd64 python:3.10-slim

[Pwuts]

Please try adding these packages to the end of line 16 in the Dockerfile: libgconf-2-4 libfontconfig1

https://github.com/Significant-Gravitas/Auto-GPT/blob/14d3ecaae75a80f03cd118b5767e1c6affb4e3cc/Dockerfile#L16

Then rebuild and try again. Does that help?

[camolechowski]

@vanities @Pwuts: I found a pretty ridiculous workaround that solves the issue:
Dockerfile

# Use an official Python base image from the Docker Hub
FROM python:3.10-slim

# Install git
RUN apt-get -y update
RUN apt-get -y install git chromium-driver

# Install Xvfb and other dependencies for headless browser testing
RUN apt-get update \
    && apt-get install -y wget gnupg2 libgtk-3-0 libdbus-glib-1-2 dbus-x11 xvfb ca-certificates

# Install Firefox / Chromium
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list \
    && apt-get update \
    && apt-get install -y chromium firefox-esr

# Install unzip
RUN apt-get update && \
    apt-get install -y unzip

# Install Correct Chromedriver
RUN wget -q -O /tmp/chromedriver.zip https://chromedriver.storage.googleapis.com/112.0.5615.49/chromedriver_mac_arm64.zip \
    && unzip /tmp/chromedriver.zip -d /usr/local/bin \
    && rm /tmp/chromedriver.zip \
    && chmod +x /usr/local/bin/chromedriver

# Set environment variables
ENV PIP_NO_CACHE_DIR=yes \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

# Create a non-root user and set permissions
RUN useradd --create-home appuser
WORKDIR /home/appuser
RUN chown appuser:appuser /home/appuser
USER appuser

# Copy the requirements.txt file and install the requirements
COPY --chown=appuser:appuser requirements.txt .
RUN sed -i '/Items below this point will not be included in the Docker Image/,$d' requirements.txt && \
	pip install --no-cache-dir --user -r requirements.txt

# Copy the application files
COPY --chown=appuser:appuser autogpt/ ./autogpt

# Set the entrypoint
ENTRYPOINT ["python", "-m", "autogpt"]

web_selenium.py

def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
    """Scrape text from a website using selenium

    Args:
        url (str): The url of the website to scrape

    Returns:
        Tuple[WebDriver, str]: The webdriver and the text scraped from the website
    """
    logging.getLogger("selenium").setLevel(logging.CRITICAL)

    options_available = {
        "chrome": ChromeOptions,
        "safari": SafariOptions,
        "firefox": FirefoxOptions,
    }

    options = options_available[CFG.selenium_web_browser]()
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
    )

    if CFG.selenium_web_browser == "firefox":
        driver = webdriver.Firefox(
            executable_path=GeckoDriverManager().install(), options=options
        )
    elif CFG.selenium_web_browser == "safari":
        # Requires a bit more setup on the users end
        # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
        driver = webdriver.Safari(options=options)
    else:
        if platform == "linux" or platform == "linux2":
            options.add_argument("--disable-dev-shm-usage")
            options.add_argument("--remote-debugging-port=9222")

        options.add_argument("--no-sandbox")
        if CFG.selenium_headless:
            options.add_argument("--headless")
            options.add_argument("--disable-gpu")

        driver = webdriver.Chrome(
            executable_path="/usr/local/bin/chromedriver", options=options
        )
    driver.get(url)

    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "body"))
    )

    # Get the HTML content directly from the browser's DOM
    page_source = driver.execute_script("return document.body.outerHTML;")
    soup = BeautifulSoup(page_source, "html.parser")

    for script in soup(["script", "style"]):
        script.extract()

    text = soup.get_text()
    lines = (line.strip() for line in text.splitlines())
    chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
    text = "\n".join(chunk for chunk in chunks if chunk)
    return driver, text

Just manually added a step to install unzip and the correct driver in the Dockerfile and reference the directory directly. I know that doesn't help you much @Pwuts but @vanities I figured you were just trying to get it working, so I figured I'd share.

UPDATE: That's embarrassing. I might have spoken too soon. Just a moment.

[axldarealest]

hey I'm a noob feeling concerned about that here on a M1 max chip I get

Command browse_website returned: Error: Message: unknown error: cannot find Chrome binary Stacktrace: 0 chromedriver 0x0000000105409670 chromedriver + 4298352 1 chromedriver 0x0000000105401bbc chromedriver + 4266940 2 chromedriver 0x0000000105034758 chromedriver + 280408 3 chromedriver 0x0000000105059e40 chromedriver + 433728 4 chromedriver 0x0000000105058308 chromedriver + 426760 5 chromedriver 0x0000000105098994 chromedriver + 690580 6 chromedriver 0x0000000105098114 chromedriver + 688404 7 chromedriver 0x00000001050622d0 chromedriver + 467664 8 chromedriver 0x0000000105063354 chromedriver + 471892 9 chromedriver 0x00000001053c96c4 chromedriver + 4036292 10 chromedriver 0x00000001053cdc64 chromedriver + 4054116 11 chromedriver 0x00000001053d42d8 chromedriver + 4080344 12 chromedriver 0x00000001053ce970 chromedriver + 4057456 13 chromedriver 0x00000001053a58dc chromedriver + 3889372 14 chromedriver 0x00000001053ed25c chromedriver + 4182620 15 chromedriver 0x00000001053ed3b4 chromedriver + 4182964 16 chromedriver 0x00000001053fc0f4 chromedriver + 4243700 17 libsystem_pthread.dylib 0x00000001a99de06c _pthread_start + 148 18 libsystem_pthread.dylib 0x00000001a99d8e2c thread_start + 8

So i'm subscribing to this issue

[camolechowski]

@Pwuts: I tried your suggestion and unfortunately it did not work. Same error code:

SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

[Pwuts]

SeleniumHQ/docker-selenium#1076

[tdimino]

I'm encountering the same error, in the same context, albeit on a Windows machine. Set my browser to Firefox, if that has any bearing on this.

SYSTEM: Command browse_website returned: Error: Message: Process unexpectedly closed with status 1

[vanities]

The only thing I can do that makes it work is changing the Dockerfile

FROM --platform=linux/amd64 python:3.10-slim

the build time 5x's though

[bobinson]

@Pwuts

Please try adding these packages to the end of line 16 in the Dockerfile: libgconf-2-4 libfontconfig1

https://github.com/Significant-Gravitas/Auto-GPT/blob/14d3ecaae75a80f03cd118b5767e1c6affb4e3cc/Dockerfile#L16

Then rebuild and try again. Does that help?

Tried within devcontainer but failed as follows:

NEXT ACTION:  COMMAND = browse_website ARGUMENTS = {'url': 'https://freebird.in', 'question': 'What services does Freebird offer?'}
[WDM] - Downloading: 100%|██████████████████████████████████| 6.75M/6.75M [00:01<00:00, 4.55MB/s]
SYSTEM:  Command browse_website returned: Error: Message: unknown error: Chrome failed to start: crashed. (chrome not reachable) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x0040006e6fe3 <unknown> #1 0x004000425d36 <unknown> #2 0x00400044eb20 <unknown> #3 0x00400044aa9b <unknown> #4 0x00400048caf7 <unknown> #5 0x00400048c11f <unknown> #6 0x004000483693 <unknown> #7 0x00400045603a <unknown> #8 0x00400045717e <unknown> #9 0x0040006a8dbd <unknown> #10 0x0040006acc6c <unknown> #11 0x0040006b64b0 <unknown> #12 0x0040006add63 <unknown> #13 0x004000680c35 <unknown> #14 0x0040006d1138 <unknown> #15 0x0040006d12c7 <unknown> #16 0x0040006df093 <unknown> #17 0x00400258aea7 start_thread

[vanities]

Adding those libs isn't going to create a /lib64 folder inside the container.

[jcalderonzumba]

Joining late to the party.

I had the same issue and this is what I did that solved the issue for me in my M1

• Dockerfile is as it comes from master
• I modified web_selenium.py and changed:

driver = webdriver.Chrome( executable_path="/usr/bin/chromedriver", options=options )

it seems that the original line that installs the chrome driver is installing a wrong version while the version installed i the docker file is the proper one because:

• Chromium installed is v122
• Chromedriver installed by Dockerfile is v122
• Chromedriver installed in the python code is the WRONG one

Hope this helps others.

[tjhils]

The only thing I can do that makes it work is changing the Dockerfile

FROM --platform=linux/amd64 python:3.10-slim

the build time 5x's though

This didn't resolve the issue for me. It produces a new error instead. Is this the only change you made?

[vanities]

This didn't resolve the issue for me. It produces a new error instead. Is this the only change you made?

You're right, the error is captured here #2600 (comment)

[camolechowski]

@jcalderonzumba

Joining late to the party.

I had the same issue and this is what I did that solved the issue for me in my M1

• Dockerfile is as it comes from master

• I modified web_selenium.py and changed:

` driver = webdriver.Chrome(

        executable_path="/usr/bin/chromedriver", options=options

    )`

it seems that the original line that installs the chrome driver is installing a wrong version while the version installed i the docker file is the proper one because:

• Chromium installed is v122

• Chromedriver installed by Dockerfile is v122

• Chromedriver installed in the python code is the WRONG one

Hope this helps others.

#2600 (comment)

Did you not receive the same error I did that claims it cannot find the driver located at /usr/local/bin/chromedriver?

[bobinson]

@Pwuts

Extended test results:

Environment : macbook pro with M2 : on master with git commit a8fe3085fd708afd9bd870f44122f61e78455f1d

Test result on Dockerfile with the modification suggested in

=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-=
[WDM] - Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 9.09MB/s]
SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

Test result in .devcontainer/Dockerfile in vscode

[WDM] - Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 7.73MB/s]
SYSTEM:  Command browse_website returned: Error: Message: unknown error: Chrome failed to start: crashed. (chrome not reachable) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x0040006e6fe3 <unknown> #1 0x004000425d36 <unknown> #2 0x00400044eb20 <unknown> #3 0x00400044aa9b <unknown> #4 0x00400048caf7 <unknown> #5 0x00400048c11f <unknown> #6 0x004000483693 <unknown> #7 0x00400045603a <unknown> #8 0x00400045717e <unknown> #9 0x0040006a8dbd <unknown> #10 0x0040006acc6c <unknown> #11 0x0040006b64b0 <unknown> #12 0x0040006add63 <unknown> #13 0x004000680c35 <unknown> #14 0x0040006d1138 <unknown> #15 0x0040006d12c7 <unknown> #16 0x0040006df093 <unknown> #17 0x00400258cea7 start_thread

[Desttro]

Just tried the latest stable, and here are results from my M1:

[WDM] - Downloading: 100%|███████████████████████████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 15.7MB/s]
SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

Status code was: 255

when enabled Dockers Rosetta for x86/amd64 emulation on Apple Silicon:

[WDM] - Downloading: 100%|███████████████████████████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 12.8MB/s]
SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: -5

Status code was: -5

[jcalderonzumba]

@jcalderonzumba

Joining late to the party.
I had the same issue and this is what I did that solved the issue for me in my M1

• Dockerfile is as it comes from master
• I modified web_selenium.py and changed:

` driver = webdriver.Chrome(

        executable_path="/usr/bin/chromedriver", options=options

    )`

it seems that the original line that installs the chrome driver is installing a wrong version while the version installed i the docker file is the proper one because:

• Chromium installed is v122
• Chromedriver installed by Dockerfile is v122
• Chromedriver installed in the python code is the WRONG one

Hope this helps others.

#2600 (comment)

Did you not receive the same error I did that claims it cannot find the driver located at /usr/local/bin/chromedriver?

You are using /usr/local/bin/chromedriver, but you need to use /usr/bin/chromedriver because that is the location of the chromedriver installed in the Dockerfile definition.

[Desttro]

@jcalderonzumba

Joining late to the party.
I had the same issue and this is what I did that solved the issue for me in my M1

• Dockerfile is as it comes from master
• I modified web_selenium.py and changed:

` driver = webdriver.Chrome(

        executable_path="/usr/bin/chromedriver", options=options

    )`

it seems that the original line that installs the chrome driver is installing a wrong version while the version installed i the docker file is the proper one because:

• Chromium installed is v122
• Chromedriver installed by Dockerfile is v122
• Chromedriver installed in the python code is the WRONG one

Hope this helps others.

#2600 (comment)
Did you not receive the same error I did that claims it cannot find the driver located at /usr/local/bin/chromedriver?

You are using /usr/local/bin/chromedriver, but you need to use /usr/bin/chromedriver because that is the location of the chromedriver installed in the Dockerfile definition.

Just tried it, and it works! Even when Rosetta emulation is disabled in Docker.

Have a stable branch, just changed one line in web_selenium.py
So the problem is with webdriver-manager setup, link to repo.

This could be a solution, detect the platform, then use this setup (from README.md of webdriver_manager repo):

If the Opera browser is installed in a location other than C:/Program Files or C:/Program Files (x86) on windows
and /usr/bin/opera for all unix variants and mac, then use the below code,

options = webdriver.ChromeOptions()
options.binary_location = "path/to/opera.exe"
driver = webdriver.Remote(webdriver_service.service_url, options=options)

[tinalasisi]

          executable_path="/usr/bin/chromedriver", options=options
   )

Editing web_selenium.py fixed my issue with the browse_website command without doing anything else! (Mac M1 Max; using Dockerfile to run in a container)

This is the complete function I'm using in case it's easier to copy and paste:

def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
    """Scrape text from a website using selenium

    Args:
        url (str): The url of the website to scrape

    Returns:
        Tuple[WebDriver, str]: The webdriver and the text scraped from the website
    """
    logging.getLogger("selenium").setLevel(logging.CRITICAL)

    options_available = {
        "chrome": ChromeOptions,
        "safari": SafariOptions,
        "firefox": FirefoxOptions,
    }

    options = options_available[CFG.selenium_web_browser]()
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
    )

    if CFG.selenium_web_browser == "firefox":
        driver = webdriver.Firefox(
            executable_path=GeckoDriverManager().install(), options=options
        )
    elif CFG.selenium_web_browser == "safari":
        # Requires a bit more setup on the users end
        # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
        driver = webdriver.Safari(options=options)
    else:
        if platform == "linux" or platform == "linux2":
            options.add_argument("--disable-dev-shm-usage")
            options.add_argument("--remote-debugging-port=9222")

        options.add_argument("--no-sandbox")
        if CFG.selenium_headless:
            options.add_argument("--headless")
            options.add_argument("--disable-gpu")

        driver = webdriver.Chrome(
            executable_path="/usr/bin/chromedriver", options=options
    )
    driver.get(url)

    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "body"))
    )

    # Get the HTML content directly from the browser's DOM
    page_source = driver.execute_script("return document.body.outerHTML;")
    soup = BeautifulSoup(page_source, "html.parser")

    for script in soup(["script", "style"]):
        script.extract()

    text = soup.get_text()
    lines = (line.strip() for line in text.splitlines())
    chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
    text = "\n".join(chunk for chunk in chunks if chunk)
    return driver, text

Thank you so much for the fix!

[AdamiecRadek]

@jcalderonzumba Thank you so much. This one works for me as well! 🎉

[Pwuts]

@jcalderonzumba thanks a lot! We'll try matrix testing this and pushing a fix asap.

[knutli]

Chipping in to say that @jcalderonzumba 's fix worked for me too.

browse_website didn't work previously, it just errored out with different error messages like SYSTEM: Command browse_website returned: Error: Message: unknown error: cannot find Chrome binary Stacktrace.

I am running AutoGPT locally on macOS M1.

[Pwuts]

This will be fixed for Docker users and users who have already installed chromium-driver with #1843

[bobinson]

Docker + Chromedriver on M2 works with #1843

[iba-ragi]

#2600 (comment)
Thank you! I worked.

[gbhall]

def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
"""Scrape text from a website using selenium

Args:
    url (str): The url of the website to scrape

Returns:
    Tuple[WebDriver, str]: The webdriver and the text scraped from the website
"""
logging.getLogger("selenium").setLevel(logging.CRITICAL)

options_available = {
    "chrome": ChromeOptions,
    "safari": SafariOptions,
    "firefox": FirefoxOptions,
}

options = options_available[CFG.selenium_web_browser]()
options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
)

if CFG.selenium_web_browser == "firefox":
    driver = webdriver.Firefox(
        executable_path=GeckoDriverManager().install(), options=options
    )
elif CFG.selenium_web_browser == "safari":
    # Requires a bit more setup on the users end
    # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
    driver = webdriver.Safari(options=options)
else:
    if platform == "linux" or platform == "linux2":
        options.add_argument("--disable-dev-shm-usage")
        options.add_argument("--remote-debugging-port=9222")

    options.add_argument("--no-sandbox")
    if CFG.selenium_headless:
        options.add_argument("--headless")
        options.add_argument("--disable-gpu")

    driver = webdriver.Chrome(
        executable_path="/usr/bin/chromedriver", options=options
)
driver.get(url)

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.TAG_NAME, "body"))
)

# Get the HTML content directly from the browser's DOM
page_source = driver.execute_script("return document.body.outerHTML;")
soup = BeautifulSoup(page_source, "html.parser")

for script in soup(["script", "style"]):
    script.extract()

text = soup.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
text = "\n".join(chunk for chunk in chunks if chunk)
return driver, text

I tried this on my Macbook Pro M1 Pro but get this

Command browse_website returned: Error: Message: invalid argument (Session info: headless chrome=112.0.5615.138) Stacktrace: #0 0xaaaae639439c <unknown> #1 0xaaaae60fbcc4 <unknown> #2 0xaaaae60e9988 <unknown> #3 0xaaaae60e7f7c <unknown> #4 0xaaaae60e80dc <unknown> #5 0xaaaae60fe840 <unknown> #6 0xaaaae616dea8 <unknown> #7 0xaaaae616d920 <unknown> #8 0xaaaae6126adc <unknown> #9 0xaaaae6127d68 <unknown> #10 0xaaaae636200c <unknown> #11 0xaaaae6364878 <unknown> #12 0xaaaae636439c <unknown> #13 0xaaaae636af90 <unknown> #14 0xaaaae63651ec <unknown> #15 0xaaaae6343de8 <unknown> #16 0xaaaae637f458 <unknown> #17 0xaaaae637f620 <unknown> #18 0xaaaae638cd84 <unknown> #19 0xffffbc144648 start_thread #20 0xffffbb8ebc1c <unknown>

[stitchchau]

          executable_path="/usr/bin/chromedriver", options=options
   )

Editing web_selenium.py fixed my issue with the browse_website command without doing anything else! (Mac M1 Max; using Dockerfile to run in a container)

This is the complete function I'm using in case it's easier to copy and paste:

def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
    """Scrape text from a website using selenium

    Args:
        url (str): The url of the website to scrape

    Returns:
        Tuple[WebDriver, str]: The webdriver and the text scraped from the website
    """
    logging.getLogger("selenium").setLevel(logging.CRITICAL)

    options_available = {
        "chrome": ChromeOptions,
        "safari": SafariOptions,
        "firefox": FirefoxOptions,
    }

    options = options_available[CFG.selenium_web_browser]()
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
    )

    if CFG.selenium_web_browser == "firefox":
        driver = webdriver.Firefox(
            executable_path=GeckoDriverManager().install(), options=options
        )
    elif CFG.selenium_web_browser == "safari":
        # Requires a bit more setup on the users end
        # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
        driver = webdriver.Safari(options=options)
    else:
        if platform == "linux" or platform == "linux2":
            options.add_argument("--disable-dev-shm-usage")
            options.add_argument("--remote-debugging-port=9222")

        options.add_argument("--no-sandbox")
        if CFG.selenium_headless:
            options.add_argument("--headless")
            options.add_argument("--disable-gpu")

        driver = webdriver.Chrome(
            executable_path="/usr/bin/chromedriver", options=options
    )
    driver.get(url)

    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "body"))
    )

    # Get the HTML content directly from the browser's DOM
    page_source = driver.execute_script("return document.body.outerHTML;")
    soup = BeautifulSoup(page_source, "html.parser")

    for script in soup(["script", "style"]):
        script.extract()

    text = soup.get_text()
    lines = (line.strip() for line in text.splitlines())
    chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
    text = "\n".join(chunk for chunk in chunks if chunk)
    return driver, text

Thank you so much for the fix!

It worked with me, Thanks a lot!

[cdeliens]

          executable_path="/usr/bin/chromedriver", options=options
   )

Editing web_selenium.py fixed my issue with the browse_website command without doing anything else! (Mac M1 Max; using Dockerfile to run in a container)

This is the complete function I'm using in case it's easier to copy and paste:

def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
    """Scrape text from a website using selenium

    Args:
        url (str): The url of the website to scrape

    Returns:
        Tuple[WebDriver, str]: The webdriver and the text scraped from the website
    """
    logging.getLogger("selenium").setLevel(logging.CRITICAL)

    options_available = {
        "chrome": ChromeOptions,
        "safari": SafariOptions,
        "firefox": FirefoxOptions,
    }

    options = options_available[CFG.selenium_web_browser]()
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
    )

    if CFG.selenium_web_browser == "firefox":
        driver = webdriver.Firefox(
            executable_path=GeckoDriverManager().install(), options=options
        )
    elif CFG.selenium_web_browser == "safari":
        # Requires a bit more setup on the users end
        # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
        driver = webdriver.Safari(options=options)
    else:
        if platform == "linux" or platform == "linux2":
            options.add_argument("--disable-dev-shm-usage")
            options.add_argument("--remote-debugging-port=9222")

        options.add_argument("--no-sandbox")
        if CFG.selenium_headless:
            options.add_argument("--headless")
            options.add_argument("--disable-gpu")

        driver = webdriver.Chrome(
            executable_path="/usr/bin/chromedriver", options=options
    )
    driver.get(url)

    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "body"))
    )

    # Get the HTML content directly from the browser's DOM
    page_source = driver.execute_script("return document.body.outerHTML;")
    soup = BeautifulSoup(page_source, "html.parser")

    for script in soup(["script", "style"]):
        script.extract()

    text = soup.get_text()
    lines = (line.strip() for line in text.splitlines())
    chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
    text = "\n".join(chunk for chunk in chunks if chunk)
    return driver, text

Thank you so much for the fix!

Worked for me, I'm on M1, Thanks a lot!

[CharlieCappn]

I did the above edit of the web_selenium.py on my M1 macbook, and have had no success. (using visual studio code)
This is the result of my attempt on auto-gpt

Command browse_website returned: Error: Message: unknown error: Chrome failed to start: exited abnormally. (chrome not reachable) (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x0040006e6fe3 #1 0x004000425d36 #2 0x00400044eb20 #3 0x00400044aa9b #4 0x00400048caf7 #5 0x00400048c11f #6 0x004000483693 #7 0x00400045603a #8 0x00400045717e #9 0x0040006a8dbd #10 0x0040006acc6c #11 0x0040006b64b0 #12 0x0040006add63 #13 0x004000680c35 #14 0x0040006d1138 #15 0x0040006d12c7 #16 0x0040006df093 #17 0x004002585ea7 start_thread

Additionally, this is my entire file, if it helps figure out why its not working.

"""Selenium web scraping module."""
from future import annotations

import logging
from pathlib import Path
from sys import platform

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium.webdriver.remote.webdriver import WebDriver
from selenium.webdriver.safari.options import Options as SafariOptions
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager

import autogpt.processing.text as summary
from autogpt.config import Config
from autogpt.processing.html import extract_hyperlinks, format_hyperlinks

FILE_DIR = Path(file).parent.parent
CFG = Config()

def browse_website(url: str, question: str) -> tuple[str, WebDriver]:
"""Browse a website and return the answer and links to the user

Args:
    url (str): The url of the website to browse
    question (str): The question asked by the user

Returns:
    Tuple[str, WebDriver]: The answer and links to the user and the webdriver
"""
driver, text = scrape_text_with_selenium(url)
add_header(driver)
summary_text = summary.summarize_text(url, text, question, driver)
links = scrape_links_with_selenium(driver, url)

# Limit links to 5
if len(links) > 5:
    links = links[:5]
close_browser(driver)
return f"Answer gathered from website: {summary_text} \n \n Links: {links}", driver

def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
"""Scrape text from a website using selenium

Args:
    url (str): The url of the website to scrape

Returns:
    Tuple[WebDriver, str]: The webdriver and the text scraped from the website
"""
logging.getLogger("selenium").setLevel(logging.CRITICAL)

options_available = {
    "chrome": ChromeOptions,
    "safari": SafariOptions,
    "firefox": FirefoxOptions,
}

options = options_available[CFG.selenium_web_browser]()
options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
)

if CFG.selenium_web_browser == "firefox":
    driver = webdriver.Firefox(
        executable_path=GeckoDriverManager().install(), options=options
    )
elif CFG.selenium_web_browser == "safari":
    # Requires a bit more setup on the users end
    # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
    driver = webdriver.Safari(options=options)
else:
    if platform == "linux" or platform == "linux2":
        options.add_argument("--disable-dev-shm-usage")
        options.add_argument("--remote-debugging-port=9222")

    options.add_argument("--no-sandbox")
    if CFG.selenium_headless:
        options.add_argument("--headless")
        options.add_argument("--disable-gpu")

    driver = webdriver.Chrome(
        executable_path="/usr/bin/chromedriver", options=options
)
driver.get(url)

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.TAG_NAME, "body"))
)

# Get the HTML content directly from the browser's DOM
page_source = driver.execute_script("return document.body.outerHTML;")
soup = BeautifulSoup(page_source, "html.parser")

for script in soup(["script", "style"]):
    script.extract()

text = soup.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
text = "\n".join(chunk for chunk in chunks if chunk)
return driver, text

def scrape_links_with_selenium(driver: WebDriver, url: str) -> list[str]:
"""Scrape links from a website using selenium

Args:
    driver (WebDriver): The webdriver to use to scrape the links

Returns:
    List[str]: The links scraped from the website
"""
page_source = driver.page_source
soup = BeautifulSoup(page_source, "html.parser")

for script in soup(["script", "style"]):
    script.extract()

hyperlinks = extract_hyperlinks(soup, url)

return format_hyperlinks(hyperlinks)

def close_browser(driver: WebDriver) -> None:
"""Close the browser

Args:
    driver (WebDriver): The webdriver to close

Returns:
    None
"""
driver.quit()

def add_header(driver: WebDriver) -> None:
"""Add a header to the website

Args:
    driver (WebDriver): The webdriver to use to add the header

Returns:
    None
"""
driver.execute_script(open(f"{FILE_DIR}/js/overlay.js", "r").read())

[danielvoigt]

Even if you try all suggested solutions to get it working on the latest release v0.2.2 on Docker & Mac M1, you will still get:

Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

There are two main issues:

1. you need to reference the chromedriver path in the docker container directly in the scrape_text_with_selenium function, i.e.

driver = webdriver.Chrome( executable_path="/usr/bin/chromedriver", options=options )

you can verify this is the docker container path for chromedriver by doing the following:

docker exec -it <container-id> /bin/bash 

and then once in the container

which chromedriver

2. this is where most people are going wrong, if you are on mac m1 you need to use:

FROM --platform=linux/arm64 python:3.10-slim

not linux/amd64 as others have suggested. mac m1 is arm not amd

Doing both of the changes above fixed it

[0xG3ckers]

@danielvoigt I tried what you recommended and it's still not working

[CharlieCappn]

i changed the first part as you recommended, but im not sure where to go in order to type in "FROM --platform=linux/arm64 python:3.10-slim"

[danielvoigt]

@CharlieCappn you go to the Dockerfile in the root of the directory. You change the first line in the Dockerfile to the following (adding in the --platform=linux/arm64):

FROM --platform=linux/arm64 python:3.10-slim

@0xG3ckers I don't see how that's possible. Are you running with Docker? What error are you getting? I would do the following steps.

1. follow my steps above and enter the container via docker exec after you start up the container. Verify that chromedriver is installed (i.e. chromedriver --version) and also verify the path of the executable which chromedriver.
2. take that path and update the executable_path for chromedriver in the scrape_text_with_selenium function:

driver = webdriver.Chrome( executable_path="/usr/bin/chromedriver", options=options )

It should work if you are pulling the correct image for M1 and referencing the correct path as the executable.

Also, I think it would make sense to provide these directions in the README for M1 users. Apple M1 is highly used by engineers and it shouldn't be expected that everyone that uses AutoGPT knows how to debug Docker / platform dependency errors.

[0xG3ckers]

@danielvoigt damn I really don't know what I'm doing wrong. I tried fresh installs before following your instructions too. I ran the docker exec command yesterday and confirmed it's the same executable_path thats in scrape_text_with_selenium . Haven't tried chromedriver --version just tried running everything again along with that but now I get docker: not found . I'm supposed to be running this through the docker terminal right?

Just checked through the file system in docker and found the chromedriver in the right location but the file was illegible. Maybe thats where the problem is?

edit: finally got the chromedriver --version to confirm its installed. still not working tho
Error: Message: unknown error: Chrome failed to start: exited abnormally. (chrome not reachable) (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)