feat(agent): Add URL whitelisting/blacklisting to config and URL validation #6848
Conversation
✅ Deploy Preview for auto-gpt-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
remove falcon from the PR |
| ### White/Black Listing URLS | ||
| ################################################################################ | ||
| ##WEB_POLICY - Specifies whether the url list is black list(0) or a whilte list(1) | ||
| #WEB_POLICY=0 |
There was a problem hiding this comment.
I think we need a policy for users who want to continue using AutoGPT as normal. This looks like it forces them to choose between whitelisting and blacklisting, which some users might not want either. There needs to be a default option that simply disables this feature.
| ##WEB_POLICY - Specifies whether the url list is black list(0) or a whilte list(1) | ||
| #WEB_POLICY=0 | ||
| ##URL_LIST - Comma separated URLs to be black or whitelisted, sets according to WEB_POLICY value | ||
| #URL_LIST=https://www.google.com,http://www.google.com |
There was a problem hiding this comment.
The only thing I don't like about putting it here, is that these lists might get pretty long. That's why in my version I was attempting to parse files.
| ##WEB_POLICY - Specifies whether the url list is black list(0) or a whilte list(1) | ||
| #WEB_POLICY=0 |
There was a problem hiding this comment.
- Formatting diverges from the rest of the file
- Rather make this an enum with values
blacklistandwhitelist
| # White/Black Listing URLs | ||
| web_policy: Optional[str] = os.getenv("WEB_POLICY", 0) | ||
| url_list: Optional[list] = os.getenv("URL_LIST",[]).split(",") |
There was a problem hiding this comment.
This is not how the config of the application works, as you can see from how all the other config attributes are defined.
|
|
||
| P = ParamSpec("P") | ||
| T = TypeVar("T") | ||
|
|
||
| config = Config |
There was a problem hiding this comment.
@Pwuts can you please help with why this wont work?
There was a problem hiding this comment.
from autogpt.config import ConfigBuilder
config = ConfigBuilder.build_config_from_env()
web_policy = config.web_policy
url_list = config.url_list
will this work? @Pwuts
| @@ -2,10 +2,11 @@ | |||
| import re | |||
| from typing import Any, Callable, ParamSpec, TypeVar | |||
| from urllib.parse import urljoin, urlparse | |||
| from autogpt.config import Config | |||
There was a problem hiding this comment.
this is gonna throw a linting error
There was a problem hiding this comment.
Added flake8 to VS code, sorry there was some issue with flake earlier.
| raise ValueError("URL Not Whitelisted") | ||
| elif url in url_list: | ||
| raise ValueError("URL Blacklisted.") |
There was a problem hiding this comment.
I have fixed the linting this update the PR with the above config changes @Pwuts
Pwuts
changed the title
github-actions
bot
added
the
conflicts
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
Background
This PR refers to the solution for enhancement suggested here: #5289
This basically takes account white/black listing of URLs during context setting in agents.
Changes 🏗️
I have added 2 env variables for web policy and URL list. With a combination of both, a URL is white or blacklisted by the read_webpage function.
PR Quality Scorecard ✨
+2 pts+5 pts+5 pts+5 pts-4 pts+4 pts+5 pts-5 ptsagbenchmarkto verify that these changes do not regress performance?+10 pts