net/http: no way to set Fetch options when making outgoing http calls with js/wasm

[eginez]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version devel +c814ac44c0 Thu Jul 19 21:30:27 2018 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="wasm"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="js"
GOPATH="/Users/eginez/repos/goland"
GOPROXY=""
GORACE=""
GOROOT="/Users/eginez/repos/go"
GOTMPDIR=""
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/pm/tlg5p5nx4vj3s05h4rfsv3t40000gn/T/go-build136489492=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

The native http client, maps to the fetch() api. With the current client it is impossible to set additional options as supported by the fetch call, see: https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch.

What did you expect to see?

A way to specify options for the fetch call

[agnivade]

Yes, this is known. Currently, there is no good way to expose fetch parameters from user level to the http.RoundTripper which is what we are using to make the fetch call. It is not very straightforward, as we have to make some decisions on how best to integrate browser options into a Go API.

See a very small discussion here - https://go-review.googlesource.com/c/go/+/114515/8/src/net/http/roundtrip_js.go#34

The main tracking issue for a better wasm http client is #25695. I would suggest that you comment on that issue with your suggestions on how best we can take this forward.

[bradfitz]

@agnivade, things aren't that doomed. We can use http.Request.WithContext or HTTP headers to something to set options for Fetch.

/cc @neelance @dmitshur

[bradfitz]

(That is, we don't need to wait for a new HTTP client package design to make progress on this)

[agnivade]

Ah, using WithContext sounds like a good idea. I was overcomplicating things.

Also @johanbrandhorst

[johanbrandhorst]

That's a cool idea, but how do we make it obvious to the user? Is using the context as a transport for arbitrary downstream method invocations something well established? I'd be happy to implement this if there's a proposal.

[bradfitz]

Using contexts for options is actually generally considered a gross practice. But we can do so if needed.

But before designing anything, what are the specific option(s) we're talking about here? Maybe they already map to something we have in Go.

[agnivade]

To me, mode and credentials are the 2 most important options that we should expose.

• mode can take cors, no-cors, same-origin.
• credentials can take omit, same-origin, include.

These are settings which arise only in a browser context. So as I see, there is no clear way to map them in Go.

We also have redirect which somehow maps with client.CheckRedirect.

See here for remaining options - https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch#Parameters

[bradfitz]

Another option: we could use magic header values, like https://golang.org/pkg/net/http/#TrailerPrefix (it contains a colon which is illegal in general, so we special case its semantics).

[johanbrandhorst]

I have the beginnings of a CL ready for this. Will put it up later today.

[gopherbot]

Change https://golang.org/cl/127718 mentions this issue: net/http: support configuring fetch options

[eginez]

@johanbrandhorst thanks for the quick turnaround on this issue. One quick comment I have on your PR. As a stop gap and until we can rehaul parts of the client, I see no problems with this approach. On the other hand I am really not inclined with the solution as presented if it this is going to be the long term answer. The idea that @bradfitz had about using context seems less magical to me. Thoughts?

[johanbrandhorst]

Please put comments on the CL: https://go-review.googlesource.com/c/go/+/127718

[johanbrandhorst]

With #26784 merged we now have mode and credentials covered, so this issue is now about adding support for configuring redirect. Any more thoughts on how we might accomplish that?

[bradfitz]

I don't see an obvious mapping from anything already in net/http to the Fetch redirect knob. So probably we should just do the same thing as the other two.

[johanbrandhorst]

I intend to submit a PR to add a way to configure the redirect behavior using the same mechanism as the others.

[johanbrandhorst]

@agnivade What other options do you think would be reasonable to expose via the header mechanism? I hesitate to close this issue until we cover all the existing options at least. Using your link we have:

Covered by existing API

method
headers
body
mode
credentials
redirect (soon)
signal (covered by context cancellation)

That leaves

cache: The cache mode you want to use for the request.

Not sure if there's some existing API we can map this to

referrer: A USVString specifying no-referrer, client, or a URL. The default is client.

Could presumably be configured via a header like the others.

referrerPolicy: Specifies the value of the referer HTTP header. May be one of no-referrer, no-referrer-when-downgrade, origin, origin-when-cross-origin, unsafe-url.

Could be configured via a header. This seems to affect the Referer header sent, which may already be configurable via existing API (unless that header is ignored when making the request).

integrity: Contains the subresource integrity value of the request (e.g., sha256-BpfBw7ivV8q2jLiT13fxDYAe2tJllusRSZ273h2nFSE=).

This is surely impossible for us to emulate? Or if we can figure out how this is calculated perhaps we could always send it?

keepalive: The keepalive option can be used to allow the request to outlive the page. Fetch with the keepalive flag is a replacement for the Navigator.sendBeacon() API. 

This too might map to some existing API?

Inputs appreciated.

[gopherbot]

Change https://golang.org/cl/164666 mentions this issue: net/http: support configuring redirect fetch option

[agnivade]

For keepalive, we have this https://golang.org/pkg/net/http/#Transport.DisableKeepAlives. Is it possible to map to that ?

Other than that, I would just want to add cache. I have not used the other fields in practice. Will leave it to others to decide.

[johanbrandhorst]

I don't think they mean the same concept of keepalive. The Fetch keepalive sounds like an instruction to keep the request alive beyond the lifetime of the client (like a goroutine kind of?). I think HTTP keepalives are used to keep the connection alive. I might be wrong.

[JackKCWong]

May I know what's the latest magic headers to set Fetch options? The wiki suggests js.fetch but it seems broken on go1.18 linux/amd64

$(go env GOROOT)/misc/wasm/go_js_wasm_exec wasm

net/http: invalid header field name "js.fetch:mode"

[neelance]

@JackKCWong Please check that fetch is available in your JS environment. Note that Node.js does not provide fetch by default.