feat: honor HTTP_PROXY/HTTPS_PROXY env vars when -proxy is unset#2493
Conversation
Falls back to http.ProxyFromEnvironment for the HTTP transport and reads the same vars manually for the headless launcher (chromedp does not read env). Closes #2492
WalkthroughThis PR adds support for automatic proxy detection from environment variables ( ChangesEnvironment Variable Proxy Support
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@runner/headless.go`:
- Around line 87-94: The current logic collapses proxies into a single proxy
value and ignores NO_PROXY; update the proxy resolution to read
HTTP_PROXY/https_proxy and HTTPS_PROXY/https_proxy separately along with
NO_PROXY/no_proxy, and when any are present use the existing Set() path (not
.Proxy()) to add Chromium flags: construct
--proxy-server="http=HTTP_PROXY;https=HTTPS_PROXY" using only the schemes that
exist (omit missing schemes) and add --proxy-bypass-list="NO_PROXY_VALUE" when
NO_PROXY is set; ensure you still respect an already-set proxy variable (proxy)
if provided but prefer explicit per-scheme env vars when present and remove the
.Proxy() call in favor of setting these flags via Set().
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 5192cae7-ec23-4cb2-bf4c-fde6ac107830
📒 Files selected for processing (3)
common/httpx/httpx.gorunner/headless.gorunner/options.go
| if proxy == "" { | ||
| for _, k := range []string{"HTTPS_PROXY", "https_proxy", "HTTP_PROXY", "http_proxy"} { | ||
| if v := os.Getenv(k); v != "" { | ||
| proxy = v | ||
| break | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
`For go-rod v0.116.2 / Chromium launch flags, what is the correct way to:
- apply different HTTP_PROXY and HTTPS_PROXY values, and
- honor NO_PROXY/no_proxy via proxy bypass settings
when launching Chrome?`
💡 Result:
To launch Chromium with go-rod v0.116.2 using different HTTP_PROXY and HTTPS_PROXY values, and honoring NO_PROXY/no_proxy bypasses: Use the launcher.Set method with Chromium's --proxy-server flag for scheme-specific proxies, and --proxy-bypass-list for bypass rules matching NO_PROXY format. Example code: import ( \"github.com/go-rod/rod/lib/launcher\" \"github.com/go-rod/rod/lib/launcher/flags\" \"github.com/go-rod/rod\" ) httpProxy := \"http://http-proxy-host:8080\" httpsProxy := \"http://https-proxy-host:8443\" noProxy := \"localhost,127.0.0.1,.example.com\" // matches NO_PROXY env var format, comma-separated l := launcher.New(). Set(\"--proxy-server\", \"http=\"+httpProxy+\";https=\"+httpsProxy). Set(\"--proxy-bypass-list\", noProxy). Headless(true) // or false controlURL := l.MustLaunch() defer l.Cleanup() browser := rod.New().ControlURL(controlURL).MustConnect() defer browser.MustClose() // Test page := browser.MustPage(\"https://httpbin.org/ip\") println(page.MustElement(\"body\").MustText()) Key points: - --proxy-server="http=PROXY;https=PROXY" sets different proxies for HTTP vs HTTPS schemes [1][2][3][4]. - --proxy-bypass-list="host1,host2,domain.com" bypasses proxy for matching hosts/domains (semicolon or comma separated), directly honors NO_PROXY format [5][1][6][7]. - Use launcher.Proxy(PROXY) or launcher.Set(flags.ProxyServer, PROXY) for single proxy [3][8][6]. - Environment variables like HTTP_PROXY are not directly used by Chromium command line; flags override [9][10]. - For auth, embed in proxy URL like "http://user:pass@host:port\" [8][11], or use browser.MustHandleAuth [3]. - Confirmed in go-rod examples and issues for v0.116.x [6][3][12].
Citations:
- 1: https://chromium.googlesource.com/chromium/src/+/HEAD/net/docs/proxy.md
- 2: https://superuser.com/questions/478608/setting-http-proxy-for-chromium-in-shell
- 3: https://github.com/go-rod/rod/blob/main/examples_test.go
- 4: https://chromium.googlesource.com/chromium/src/+/main/net/docs/proxy.md
- 5: https://superuser.com/questions/1418848/how-to-avoid-google-chrome-proxy-bypass-for-localhost
- 6: How do I connect with authenticated proxy and load extension with launcher manager go-rod/rod#1105
- 7: https://github.com/go-rod/rod/blob/master/lib/launcher/rod-manager/main.go
- 8: How to use http proxy, it is best to give a sample code go-rod/rod#857
- 9: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/linux/proxy_config.md
- 10: https://dev.to/luckypipewrench/politeness-vs-enforcement-why-set-httpsproxy-isnt-a-security-control-1hka
- 11: https://superuser.com/questions/1790294/how-to-use-a-proxy-server-flag-in-chromium-browser
- 12: Regarding the browser issue, I would like to ask the author for help. Thank you very much for your open source help. go-rod/rod#1170
🏁 Script executed:
# View the actual code at lines 87-94 in runner/headless.go
head -95 runner/headless.go | tail -20Repository: projectdiscovery/httpx
Length of output: 641
🏁 Script executed:
# Search for NO_PROXY handling in the file
rg -n "NO_PROXY|no_proxy" runner/headless.goRepository: projectdiscovery/httpx
Length of output: 48
🏁 Script executed:
# Search for any proxy-related environment variable handling in the repository
rg -n "HTTP_PROXY|HTTPS_PROXY|proxy" runner/headless.go -A 2 -B 2Repository: projectdiscovery/httpx
Length of output: 542
🏁 Script executed:
# Search for chromeLauncher.Proxy usage and context in the file
rg -n "Proxy" runner/headless.go -B 2 -A 2Repository: projectdiscovery/httpx
Length of output: 153
🏁 Script executed:
# Check if there's any TODO or FIXME related to proxy handling
rg -n "TODO|FIXME|proxy|PROXY" runner/headless.goRepository: projectdiscovery/httpx
Length of output: 446
🏁 Script executed:
# Check the go.mod or imports to understand which go-rod version is being used
fd "go.mod" | head -5 | xargs catRepository: projectdiscovery/httpx
Length of output: 8886
🏁 Script executed:
# Search for .Set usage with chromeLauncher in the file
rg -n "chromeLauncher\." runner/headless.go -A 1Repository: projectdiscovery/httpx
Length of output: 364
🏁 Script executed:
# Check for launcher package imports and usage patterns
rg -n "launcher\." runner/headless.go | head -20Repository: projectdiscovery/httpx
Length of output: 170
Headless env-proxy resolution drops NO_PROXY and collapses HTTP/HTTPS into one proxy.
Lines 87–94 pick a single env proxy and never consider NO_PROXY/no_proxy. This can route hosts through the proxy that should bypass it, and mishandles setups where HTTP_PROXY and HTTPS_PROXY differ.
Chromium requires --proxy-server="http=PROXY;https=PROXY" (scheme-specific) and --proxy-bypass-list="..." to honor NO_PROXY. The .Set() mechanism already exists in the code (line 100); use it to construct and pass these flags from the environment variables instead of collapsing them into a single .Proxy() call.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@runner/headless.go` around lines 87 - 94, The current logic collapses proxies
into a single proxy value and ignores NO_PROXY; update the proxy resolution to
read HTTP_PROXY/https_proxy and HTTPS_PROXY/https_proxy separately along with
NO_PROXY/no_proxy, and when any are present use the existing Set() path (not
.Proxy()) to add Chromium flags: construct
--proxy-server="http=HTTP_PROXY;https=HTTPS_PROXY" using only the schemes that
exist (omit missing schemes) and add --proxy-bypass-list="NO_PROXY_VALUE" when
NO_PROXY is set; ensure you still respect an already-set proxy variable (proxy)
if provided but prefer explicit per-scheme env vars when present and remove the
.Proxy() call in favor of setting these flags via Set().
2 participants
Summary
http.ProxyFromEnvironmentfor the HTTP transport when-proxyis empty, picking upHTTP_PROXY/HTTPS_PROXY/NO_PROXY(and lowercase variants).socks5://URLs in the same vars.Closes #2492
Summary by CodeRabbit
New Features
Documentation