Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improvement in web commands functionalities #7068

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
+22 −2
Open

Improvement in web commands functionalities #7068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
78c9d8e
use of different backend for ddg
amirdaaee Apr 3, 2024
7dbe78f
selenium proxy support
amirdaaee Apr 3, 2024
3fcee82
added example in .env.template
amirdaaee Apr 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 9 additions & 0 deletions autogpts/autogpt/.env.template
View file
Open in desktop
Expand Up @@ -134,6 +134,12 @@ OPENAI_API_KEY=your-openai-api-key
## HUGGINGFACE_API_TOKEN - HuggingFace API token (Default: None)
# HUGGINGFACE_API_TOKEN=

### DuckDuckGo

## DUCKDUCKGO_BACKEND - backend to be used for DDG sdk. possible values are api, html and lite. (Default: api)
# DUCKDUCKGO_BACKEND=


### Stable Diffusion (IMAGE_PROVIDER=sdwebui)

## SD_WEBUI_AUTH - Stable Diffusion Web UI username:password pair (Default: None)
Expand Down Expand Up @@ -172,6 +178,9 @@ OPENAI_API_KEY=your-openai-api-key
## USE_WEB_BROWSER - Sets the web-browser driver to use with selenium (default: chrome)
# USE_WEB_BROWSER=chrome

## SELENIUM_PROXY - Sets the http proxy to be used by selenium (default: None)
# SELENIUM_PROXY=

## BROWSE_CHUNK_MAX_LENGTH - When browsing website, define the length of chunks to summarize (Default: 3000)
# BROWSE_CHUNK_MAX_LENGTH=3000

Expand Down
5 changes: 4 additions & 1 deletion autogpts/autogpt/autogpt/commands/web_search.py
View file
Open in desktop
Expand Up @@ -47,8 +47,11 @@ def web_search(query: str, agent: Agent, num_results: int = 8) -> str:
while attempts < DUCKDUCKGO_MAX_ATTEMPTS:
if not query:
return json.dumps(search_results)
ddc_backend = agent.legacy_config.duckduckgo_backend

search_results = DDGS().text(query, max_results=num_results)
search_results = DDGS().text(
query, max_results=num_results, backend=ddc_backend
)

if search_results:
break
Expand Down
4 changes: 3 additions & 1 deletion autogpts/autogpt/autogpt/commands/web_selenium.py
View file
Open in desktop
Expand Up @@ -261,7 +261,9 @@
if config.selenium_headless:
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")

proxy = config.selenium_proxy
if proxy:
options.add_argument("--proxy-server=%s" % proxy)

Check warning on line 266 in autogpts/autogpt/autogpt/commands/web_selenium.py

View check run for this annotation

Codecov / codecov/patch

autogpts/autogpt/autogpt/commands/web_selenium.py#L266 

Added line #L266 was not covered by tests
_sideload_chrome_extensions(options, config.app_data_dir / "assets" / "crx")

if (chromium_driver_path := Path("/usr/bin/chromedriver")).exists():
Expand Down
6 changes: 6 additions & 0 deletions autogpts/autogpt/autogpt/config/config.py
View file
Open in desktop
Expand Up @@ -188,6 +188,7 @@ class Config(SystemSettings, arbitrary_types_allowed=True):
selenium_headless: bool = UserConfigurable(
default=True, from_env=lambda: os.getenv("HEADLESS_BROWSER", "True") == "True"
)
selenium_proxy: str = UserConfigurable("", from_env="SELENIUM_PROXY")
user_agent: str = UserConfigurable(
default="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36", # noqa: E501
from_env="USER_AGENT",
Expand Down Expand Up @@ -237,6 +238,11 @@ class Config(SystemSettings, arbitrary_types_allowed=True):
from_env=lambda: os.getenv("GOOGLE_CUSTOM_SEARCH_ENGINE_ID"),
)

# duckduckgo
duckduckgo_backend: Optional[str] = UserConfigurable(
default="api", from_env="DUCKDUCKGO_BACKEND"
)

# Huggingface
huggingface_api_token: Optional[str] = UserConfigurable(
from_env="HUGGINGFACE_API_TOKEN"
Expand Down