browse_website fails without --headless flag

[contributorai]

⚠️ Search for existing issues first ⚠️

• I have searched the existing issues, and there is no existing issue for my problem

GPT-3 or GPT-4

• I am using Auto-GPT with GPT-3 (GPT-3.5)

Steps to reproduce 🕹

does NOT work on stable branch
does NOT work on current master

install AutoGPT on gcp VM: SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
observe the issue both when running without and with docker

commit-hash (latest version tried): 9589334
commit-hash (stable version tried): 60b2b61

Current behavior 😯

on unix prompt (without docker):

 python -m autogpt
Welcome back!  Would you like me to return to being Entrepreneur-GPT?
Continue with the last settings?
Name:  Entrepreneur-GPT
Role:  browse to github.com
Goals: ['browse to github.com', 'summarize what you see on github.com']
Continue (y/n): y
Using memory of type:  LocalCache
Using Browser:  chrome
 THOUGHTS:  I will start by browsing to github.com to see what is currently on the website.
REASONING:  Browsing to the website will give me an idea of what is currently trending on the platform and what projects are currently being worked on.
PLAN: 
-  Browse to github.com
-  Summarize what is currently on the website
CRITICISM:  I need to ensure that I am not distracted by any irrelevant information on the website and focus on the main content.
NEXT ACTION:  COMMAND = browse_website ARGUMENTS = {'url': 'https://github.com', 'question': 'What is currently on the website?'}
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for ...
Input:y
-=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= 
SYSTEM:  Command browse_website returned: Error: Message: unknown error: Chrome failed to start: exited abnormally. (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x55b98ffb5fe3 <unknown> #1 0x55b98fcf4d36 <unknown> #2 0x55b98fd1db20 <unknown> #3 0x55b98fd19a9b <unknown> #4 0x55b98fd5baf7 <unknown> #5 0x55b98fd5b11f <unknown> #6 0x55b98fd52693 <unknown> #7 0x55b98fd2503a <unknown> #8 0x55b98fd2617e <unknown> #9 0x55b98ff77dbd <unknown> #10 0x55b98ff7bc6c <unknown> #11 0x55b98ff854b0 <unknown> #12 0x55b98ff7cd63 <unknown> #13 0x55b98ff4fc35 <unknown> #14 0x55b98ffa0138 <unknown> #15 0x55b98ffa02c7 <unknown> #16 0x55b98ffae093 <unknown> #17 0x7f26f448bea7 start_thread
 THOUGHTS:  It seems like there was an error with the previous command. I will try to browse to github.com again.

=======================================
Docker behavior:

$ docker run -it --env-file=./.env -v $PWD/auto_gpt_workspace:/app/auto_gpt_workspace autogpt
Warning: The file 'AutoGpt.json' does not exist. Local memory would not be saved to a file.
Welcome to Auto-GPT!  Enter the name of your AI and its role below. Entering nothing will load defaults.
Name your AI:  For example, 'Entrepreneur-GPT'
AI Name: browse_test
browse_test here!  I am at your service.
Describe your AI's role:  For example, 'an AI designed to autonomously develop and run businesses with the sole goal of increasing your net worth.'
browse_test is: browse to github.com and fetch a page
Enter up to 5 goals for your AI:  For example: Increase net worth, Grow Twitter Account, Develop and manage multiple businesses autonomously'
Enter nothing to load defaults, enter nothing when finished.
Goal 1: browse to github.com and fetch a page
Goal 2: summarize the page
Goal 3: 
Using memory of type:  LocalCache
Using Browser:  chrome
 THOUGHTS:  I will browse to github.com and fetch a page to summarize it.
REASONING:  The goal is to summarize the page, so I need to fetch it first. I will use the browse_website command to browse to github.com and fetch the page.
PLAN: 
-  Use the browse_website command to browse to github.com and fetch the page
-  Use the evaluate_code command to summarize the page
CRITICISM:  I need to ensure that I am able to accurately summarize the page, and that I am not missing any important information.
NEXT ACTION:  COMMAND = browse_website ARGUMENTS = {'url': 'https://github.com', 'question': 'fetch page'}
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for ...
Input:y
-=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= 
[WDM] - Downloading: 100%|█████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 318MB/s]
SYSTEM:  Command browse_website returned: Error: Message: unknown error: Chrome failed to start: exited abnormally. (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x559ec2396fe3 <unknown> #1 0x559ec20d5d36 <unknown> #2 0x559ec20feb20 <unknown> #3 0x559ec20faa9b <unknown> #4 0x559ec213caf7 <unknown> #5 0x559ec213c11f <unknown> #6 0x559ec2133693 <unknown> #7 0x559ec210603a <unknown> #8 0x559ec210717e <unknown> #9 0x559ec2358dbd <unknown> #10 0x559ec235cc6c <unknown> #11 0x559ec23664b0 <unknown> #12 0x559ec235dd63 <unknown> #13 0x559ec2330c35 <unknown> #14 0x559ec2381138 <unknown> #15 0x559ec23812c7 <unknown> #16 0x559ec238f093 <unknown> #17 0x7fa4ba72bea7 start_thread
 THOUGHTS:  I will try to browse to github.com again using the browse_website command.
REASONING:  The browse_website command is the best option to fetch the page from github.com. I will try to use it again to see if it works this time.
PLAN: 
-  Use the browse_website command to browse to github.com and fetch the page
-  Use the evaluate_code command to summarize the page
CRITICISM:  If the browse_website command fails again, I will need to investigate the issue further.

Expected behavior 🤔

some pages from github.com are fetched...

Your prompt 📝

ai_goals:
- browse to github.com
- summarize what you see on github.com
ai_name: Entrepreneur-GPT
ai_role: browse to github.com

[montanaflynn]

Same issue, using debian and docker compose

[calevans]

same issue using the docker container
=C=

[k-boikov]

Fixed with #1473
Tried it myself and adding options.add_argument('--headless') works.

[telometto]

@k-boikov
Where does it go inside web.py?

[k-boikov]

@k-boikov Where does it go inside web.py?

under if platform == "linux" or platform == "linux2": in autogpt/commands/web_selenium.py

[Pwuts]

Closing as duplicate of #1641