Respect HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables

[gajus]

Currently Got allows use of proxies by configuring agent (https://github.com/sindresorhus/got#proxies).

This approach suffers from the same issues I have described in the issue about logging (#559 (comment)).

tl;dr;

When you have an application which depends on modules that use got, there is no way to enable got proxy without digging through node_modules/ and patching the code.

I have no knowledge of where the HTTP_PROXY convention originates, but it is prevalent practice for the exactly the reason described above – ability to inspect all application traffic without configuring individual HTTP client instances.

"request" package already implements HTTP_PROXY configuration (https://github.com/request/request#controlling-proxy-behaviour-using-environment-variables):

Controlling proxy behaviour using environment variables

The following environment variables are respected by request:

• HTTP_PROXY / http_proxy
• HTTPS_PROXY / https_proxy
• NO_PROXY / no_proxy

When HTTP_PROXY / http_proxy are set, they will be used to proxy non-SSL requests that do not have an explicit proxy configuration option present. Similarly, HTTPS_PROXY / https_proxy will be respected for SSL requests that do not have an explicit proxy configuration option. It is valid to define a proxy in one of the environment variables, but then override it for a specific request, using the proxy configuration option. Furthermore, the proxy configuration option can be > explicitly set to false / null to opt out of proxying altogether for that request.

request is also aware of the NO_PROXY/no_proxy environment variables. These variables provide a granular way to opt out of proxying, on a per-host basis. It should contain a comma separated list of hosts to opt out of proxying. It is also possible to opt of proxying when a particular destination port is used. Finally, the variable may be set to * to opt out of the implicit proxy configuration of the other environment variables.

Here's some examples of valid no_proxy values:

• google.com - don't proxy HTTP/HTTPS requests to Google.
• google.com:443 - don't proxy HTTPS requests to Google, but do proxy HTTP requests to Google.
• google.com:443, yahoo.com:80 - don't proxy HTTPS requests to Google, and don't proxy HTTP requests to Yahoo!
• * - ignore https_proxy/http_proxy environment variables altogether.

I suggest that got implement equivalent logic.

agent configuration should remain and it should override HTTP_PROXY setting (it then becomes the responsibility of the HTTP agent to respect the environment variables).

[sindresorhus]

I think we should just recommend https://github.com/np-maintain/global-tunnel for that. Proxy support is really hard to get right and not something I'm interested in spending time on and I don't want to bloat Got with dependencies to support it.

[gajus]

Just saw that this is a duplicate of:

• http_proxy env support? #200
• Proxy support #79

[sindresorhus]

@gajus What do you think about recommending https://github.com/np-maintain/global-tunnel ? With that module it would work for any request library in the dependency-tree, not just Got.

[gajus]

@gajus What do you think about recommending https://github.com/np-maintain/global-tunnel ? With that module it would work for any request library in the dependency-tree, not just Got.

It is a good solution, but unless every package author that uses Got also adds global-tunnel, then it isn't useful for the purpose that I have described (enabling HTTP traffic inspection across the entire application). And if every author needs to add global-tunnel as a dependency, then it might as well be a direct dependency of Got.

Alternatively... Got could check for global.GOT_AGENT variable and default to it if it is configured. This way I could quickly override default agent for all dependencies that use Got.

[sindresorhus]

global-tunnel is not for package authors. You use it in your app to enable proxy support for everything that uses http.request internally, which is most request libs.

---

Got could check for global.GOT_AGENT variable and default to it if it is configured. This way I could quickly override default agent for all dependencies that use Got.

This sounds fragile. What if a Got wrapper expects some special functionality from an agent and stops working when it gets a different agent?

[gajus]

global-tunnel is not for package authors. You use it in your app to enable proxy support for everything that uses http.request internally, which is most request libs.

Sorry, assumed the contents of the package based on the name of the package (tunnel being the other one).

https://github.com/np-maintain/global-tunnel looks decent.

This sounds fragile. What if a Got wrapper expects some special functionality from an agent and stops working when it gets a different agent?

Anything when used in a global space is fragile. Absolutely valid concern.

You could of course workaround it by making global.GOT_AGENT a factory function that is called with the version of Got package and returns Agent compatible to that version of Got (similar to how Roarr handles logger registration). I don't recommend this approach though.

[rarkins]

FYI, the global-tunnel package does not support NO_PROXY so that's a dealbreaker in many corporate environments that need the ability to bypass the proxy for certain hosts.