-
Notifications
You must be signed in to change notification settings - Fork 44.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
browse_website fails with non-headless browser #1641
Comments
The error you are encountering is related to Chrome not starting correctly when using Selenium WebDriver. You can try adding a few more arguments to the options object to resolve this issue. Update the scrape_text_with_selenium(url) function in your script as follows: File: autogpt/web.py:
|
I added it, but it didn't seem to fix the problem unfortunately. Maybe it has something to do with the pathing? I don't know what the file system usually looks like in a docker container? ` options.binary_location = "/usr/local/bin/chromedriver" |
I've got this same issue. Tried the suggested arguments to add to options as well as |
Make sure chrome is actually installed in your container, the current Dockerfile doesn't install it for example. |
Running in a Win10 vm, not a docker container. I'm getting some traction with
Still getting some issues: ` DevTools listening on ws://127.0.0.1:52210/devtools/browser/92ead2e2-e12e-4d14-93fd-1ea445cf6fd5 |
@aeiberra comments worked along with :
|
This worked for me! Thanks so much! |
looks like web.py got removed from Stable in a recent merge. This broken again and the error persists even with the commands from @darrynv
|
I have this error in latest pull
|
Modifying |
For me, the Chat-GPT fixed the problem in web_selenium.py file:
I am using headless server for this. |
Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011:
Plus running the commands: Solved for me, running on Kali Linux Subsystem for windows. PS: last commands produced the following errors:
But browse_website COMMAND works now. Thanks for the continuous support! |
I tried this but the suggestion did not work, running on macOS. |
this all worked for me. I am also on macOS |
Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011: Change this part of code: else:
if platform == "linux" or platform == "linux2":
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--no-sandbox") # indent this line
options.add_argument("--headless") # Add this line
driver = webdriver.Chrome(
executable_path=ChromeDriverManager().install(), options=options
)
driver.get(url)
|
I'll try again and see if it works, thanks |
Grafting together various responses and threads, my web_selenium.py now looks like this and it finally runs without any errors. ` options = options_availableCFG.selenium_web_browser
` |
I think this is a specific problem for M1 or other ARM processor users. I assume you also have a problem with macOS. |
I add upper code to autogpt/commands/web_selenium.py, but still have problem.
def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
"""Scrape text from a website using selenium
Args:
url (str): The url of the website to scrape
Returns:
Tuple[WebDriver, str]: The webdriver and the text scraped from the website
"""
logging.getLogger("selenium").setLevel(logging.CRITICAL)
options_available = {
"chrome": ChromeOptions,
"safari": SafariOptions,
"firefox": FirefoxOptions,
}
options = options_available[CFG.selenium_web_browser]()
options.binary_location = "/usr/bin/chromedriver"
options.add_argument(
"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
)
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--headless")
if CFG.selenium_web_browser == "firefox":
driver = webdriver.Firefox(
executable_path=GeckoDriverManager().install(), options=options
)
elif CFG.selenium_web_browser == "safari":
# Requires a bit more setup on the users end
# See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
driver = webdriver.Safari(options=options)
else:
if platform == "linux" or platform == "linux2":
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument('--headless')
options.add_argument("--no-sandbox")
if CFG.selenium_headless:
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome(
executable_path="/usr/bin/chromedriver", options=options
)
#driver = webdriver.Chrome(
# executable_path=ChromeDriverManager().install(), options=options
#)
driver.get(url)
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
# Get the HTML content directly from the browser's DOM
page_source = driver.execute_script("return document.body.outerHTML;")
soup = BeautifulSoup(page_source, "html.parser")
for script in soup(["script", "style"]):
script.extract()
text = soup.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
text = "\n".join(chunk for chunk in chunks if chunk)
return driver, text |
@busyfree please try with the latest version, and if it's still broken, provide more info about how you are running Auto-GPT, e.g. what OS/environment. |
I'm still having this problem on an M1 mac on master with docker-compose. |
This exact issue by any chance? |
This is the exact error I'm getting:
Which seems to be a little different from yours.. more:
|
I have update to lastest commit(
|
Thanks for posting; please move further discussion about M1-specific issues to #2600 |
I tried to add the modifications to posted by @darkcount2011 @dods30 in a Mac 2018 (not M1) using docker run and Im still receiving the error. I tried to execute Auto-GPT with the goal to fix it is own browse_website function, but didn't work 😂 |
@Robsonsjre no use applying those mods as a fix is already in |
#2600 (comment) This solution worked for me |
using Fukol's 'if' from Significant-Gravitas#1641 (comment)
Thank you this did the trick for me as well |
Duplicates
Steps to reproduce 🕹
3, Have docker desktop up and running.
Current behavior 😯
Once when it tries to browse a website with the command : browse_website and Arguments : {'url': , question:
it returns an error in the System message:
SYSTEM: Command browse_website returned: Error: Message: unknown error: cannot find Chrome binary Stacktrace: #0 0x55ac2df50fe3 <unknown> #1 0x55ac2dc8fd36 <unknown> #2 0x55ac2dcb6f4a <unknown> #3 0x55ac2dcb4a9b <unknown> #4 0x55ac2dcf6af7 <unknown> #5 0x55ac2dcf611f <unknown> #6 0x55ac2dced693 <unknown> #7 0x55ac2dcc003a <unknown> #8 0x55ac2dcc117e <unknown> #9 0x55ac2df12dbd <unknown> #10 0x55ac2df16c6c <unknown> #11 0x55ac2df204b0 <unknown> #12 0x55ac2df17d63 <unknown> #13 0x55ac2deeac35 <unknown> #14 0x55ac2df3b138 <unknown> #15 0x55ac2df3b2c7 <unknown> #16 0x55ac2df49093 <unknown> #17 0x7f0432fd2ea7 start_thread
Expected behavior 🤔
It should be able to run the browse_website command normally and browse the website with the question.
Your prompt 📝
# Paste your prompt here
The text was updated successfully, but these errors were encountered: