proposal: net/http: support PAC proxy configuration via libproxy

[janbrummer]

What version of Go are you using (go version)?

go version go1.20.5

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

GOARCH="amd64" GOOS="linux"

What did you do?

Trying to access a page behinde a pac proxy.

What did you expect to see?

Page can be fetched.

What did you see instead?

Abort.

Please add libproxy support to take care of this.

[bcmills]

This issue does not contain enough detail to be actionable.

Please post or link to a Go program that demonstrates the problem and the commands run to reproduce it.

[janbrummer]

package main

import (
    "fmt"
    "net/http"
    "os"
)

func main() {
    client := &http.Client{}
    requestURL := fmt.Sprintf("https://www.tabos.org")
    res, err := client.Get(requestURL)
    if err != nil {
        fmt.Printf("error making http request: %s\n", err)
        os.Exit(1)
    }

    fmt.Printf("client: got response!\n")
    fmt.Printf("client: status code: %d\n", res.StatusCode)
}

Output:

go run example.go 
error making http request: Get "https://www.tabos.org": read tcp 10.115.1.93:47912->35.185.44.232:443: read: connection reset by peer
exit status 1

[janbrummer]

We do have a pac proxy in place which returns different proxy server ip's depending on the given host url. So a standard HTTP_PROXY setting does not help. It needs to download a js file which must be parsed and then transferred to the client. That's what libproxy is doing: https://github.com/libproxy/libproxy

[bcmills]

See previously #21943.

[arp242]

PAC support requires a full JavaScript interpreter; this is why e.g. curl doesn't support it either. libproxy relies on Duktape for this. I'm fairly confident in stating this sort of functionality is never going to be included in the standard library.

[janbrummer]

We are in close exchange with curl and libproxy will be integrated there as well. From a company perspective (for me it's Volkswagen) this is a quite important feature set as we are fighting several issues with docker and other go apps which do not support it. Our developers are moving from office to home office and need different configurations depending on the host, so this seems to be the only viable solution which benefits all apps written in go. I've already prepared a package for go and could need help to integrate it.

[arp242]

Of course it's useful in some cases, but it's an enormous amount of complexity, with very non-trivial security implications, and not something that's really a good fit for the standard library.

For your case an independent tool which sets HTTP_PROXY might be a good solution?

% HTTP_PROXY=$(get-proxy) my-go-app

[janbrummer]

That's not feasible as different hosts uses different proxy servers. Therefore setting an individual HTTP_PROXY is not sufficient.

[janbrummer]

You just need to make use of libproxy, nothing else is needed on your side.

[janbrummer]

FTR: Someone already wrote a package for go: https://github.com/mvo5/libproxy-golang which might be a good starting point

[ianlancetaylor]

CC @neild @bradfitz

[neild]

We're not going to add a full JavaScript interpreter to the standard library to support PAC. Third-party packages can support this via the Transport.Proxy hook, but it's just not feasible as a stdlib feature.

[janbrummer]

FTR: I'm not asking you to implement a JS interpreter but to make use of libproxy. Furthermore beside supporting pac proxies libproxy is also capable to retrieve the system proxy configuration (Windows, OSX, Linux (GTK/KDE/sysconfig/...)). So once configured in your favourite system / DE all golang application are working out of the box with the selected proxy. Let it be for privacy reasons, bandwidth control or data protection reasons, it just works and the user does not need to find the correct configuration and configuration option in other go lang application.

[arp242]

I'm not asking you to implement a JS interpreter but to make use of libproxy.

And libproxy uses a JS interpreter; that is the problem, and to make things "work out of the box" you need to allow any Go binary that accesses the internet to run arbitrary JS code. It seems so obvious to me this is an enormous extra surface area for security issues, ddos attacks, surprising behaviour, and other problems that I don't know how to explain it.

And libproxy/duktape is in C, which means that all builds will need to use cgo which makes all builds significantly harder and will impose a huge burden on everyone building Go applications. And to top it all off libproxy is LGPL and all of that has very non-trivial implications as well.

[janbrummer]

First license: I'm the maintainer of libproxy and we can discuss a license change (actually also a request from a different project) to something else. This is no problem.

In the past libproxy made use of mozjs and webkitgtk to run js code. During the rewrite we decided to go with duktape as it has a very small footprint and is really portable for embedded devices. Also i would like to add that it is a compile time option so libproxy does not need duktape (of course pac parsing no longer works then) but the other features are working.

I can do nothing about the C code as then other projects would be requiring to rewrite it in their preferred language. And this might or will most likely effect embedded devices.

[ianlancetaylor]

I'm sorry, as a matter of policy the Go standard library is not going to depend on a C library. I'm sure it's a good library but the Go standard library is Go code. This is the kind of thing that needs to be done in a third-party library outside of the Go standard library.

If there is something that the standard library can do to make it easier for people to plug in a third-party library that uses libproxy, we can discuss that. But the Go standard library itself is not going to depend on libproxy.

[janbrummer]

Then there might be a possible alternative. Offload your proxy detection to an external helper. Check whether helper is available and then execute it with the url in question. As result you will receive the proxy. In case this one is not available just follow your current code. This would be possible in Go and does not necessarily add an internal js interpreter.

[neild]

I'm sorry, but we're not going to execute a program on every HTTP request (or any HTTP request) either.

A third-party library outside the standard library is the right way to do this.