net/http: “protocol not available” and “fetch failed” when running with Denoland and GOARCH=wasm go run but not with browser

[3goats]

What version of Go are you using (go version)?

$ go version
go version go1.14.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/user1/Library/Caches/go-build"
GOENV="/Users/user1/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/user1/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/l7/08p741zd43xbsnqvxz68wdch0000gn/T/go-build062348627=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm hitting a problem when using the GOOS=js GOARCH=wasm go run -exec="$(go env GOROOT)/misc/wasm/go_js_wasm_exec" . to test my WASM build via NodeJS which uses wasm_exec.js

This is returning a auth error: Post "https://server/authorize/": dial tcp: Protocol not available exit status 1 error.

However, if test the same .wasm using wasm_exec.html it works perfectly, obviously in a browser. It also works if I compile to a normal go binary.

I also used the following to test it using the Deno (https://deno.land/v1) toolchain:

import * as _ from "./wasm_exec.js";
const go = new window.Go();
const f = await Deno.open("test.wasm")
const buf = await Deno.readAll(f);
const inst = await WebAssembly.instantiate(buf, go.importObject);
go.run(inst.instance);

This exposed a bit more information that relates to fetch() failing. i.e. net/http: fetch() failed: <object>

[odeke-em]

Thank you for this report @carlskii.

Kindly looping in some experts @agnivade @neelance

[neelance]

Is the fetch function available in Deno?

[3goats]

Yes I believe so. Wondering if Node handles CORS differently to Chrome.

Here's the deno error message:

auth error: Post "https://cbtpp.se.venafi.com/vedsdk/authorize/": dial tcp: Protocol not available
exit code: 1

Essentially the same error I get using go_js_wasm_exec directly.

$GOOS=js GOARCH=wasm go run -exec="$(go env GOROOT)/misc/wasm/go_js_wasm_exec" .
auth error: Post "https://cbtpp.se.venafi.com/vedsdk/authorize/": dial tcp: Protocol not available
exit status 1

[agnivade]

NodeJS doesn't have fetch, so it won't work there. I am pretty confident that Deno doesn't have fetch too. You will get that line when the fakeNetwork is used. 220552f

Please confirm that fetch is available as a global function in the Deno runtime.

[3goats]

Thanks - I will check that and get back to you.

[3goats]

OK - So deno does include fetch but the network needs to be allowed. i.e.

deno run --allow-read --allow-net deno-test.js this works perfectly.

So I tried patching the global object in node to include fetch. e.g.

require('./wasm_exec.js');
const fetch = require('node-fetch');

if (!globalThis.fetch) {
    globalThis.fetch = fetch;
}

const buf = fs.readFileSync('vcert.wasm');
const go = new Go();
async function run() {
    const inst = await WebAssembly.instantiate(buf, go.importObject);
    go.run(inst.instance);
}
run()

This seemed to get things further, however its now causeing a panic. e.g.

panic: syscall/js: call of Value.Invoke on undefined

goroutine 1 [running]:
syscall/js.Value.New(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x81aaa8)
	/usr/local/go/src/syscall/js/js.go:436 +0xa
net/http.(*Transport).RoundTrip(0x870a00, 0x8c2100, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/roundtrip_js.go:80 +0x25
net/http.send(0x8c2000, 0x10c120, 0x870a00, 0xbfaeb8e24de34c40, 0x6fc4b5301, 0x4156c0, 0x0, 0x80c090, 0x8253b0, 0x1)
	/usr/local/go/src/net/http/client.go:252 +0x5c
net/http.(*Client).send(0x825380, 0x8c2000, 0xbfaeb8e24de34c40, 0x6fc4b5301, 0x4156c0, 0x80c090, 0x0, 0x1, 0xa84bd)
	/usr/local/go/src/net/http/client.go:176 +0x13
net/http.(*Client).do(0x825380, 0x8c2000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:699 +0x38
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:567
github.com/Venafi/vcert/pkg/venafi/tpp.(*Connector).request(0x8284e0, 0xa7618, 0x4, 0xab35a, 0x11, 0x68e20, 0x80b000, 0x0, 0x0, 0x0, ...)
	/Users/cbourne/development/godev/src/github.com/Venafi/vcert/pkg/venafi/tpp/tpp.go:367 +0x35
github.com/Venafi/vcert/pkg/venafi/tpp.processAuthData(0x8284e0, 0xab35a, 0x11, 0x68e20, 0x80b000, 0x23, 0x23, 0x1c, 0xb1fd9)
	/Users/cbourne/development/godev/src/github.com/Venafi/vcert/pkg/venafi/tpp/connector.go:210 +0x2
github.com/Venafi/vcert/pkg/venafi/tpp.(*Connector).Authenticate(0x8284e0, 0x888200, 0x0, 0x0)
	/Users/cbourne/development/godev/src/github.com/Venafi/vcert/pkg/venafi/tpp/connector.go:119 +0x28
github.com/Venafi/vcert.(*Config).NewClient(0x82a640, 0x1010101007e0005, 0x10, 0x21cb0007, 0xa)
	/Users/cbourne/development/godev/src/github.com/Venafi/vcert/client.go:62 +0x13
github.com/Venafi/vcert.NewClient(...)
	/Users/cbourne/development/godev/src/github.com/Venafi/vcert/client.go:71
main.main()
	/Users/cbourne/development/godev/src/github.com/carlskii/vcert-wasm/main.go:51 +0x4
exit code: 2

So woking fine in the browser and using deno, but not NodeJS

[neelance]

The line of this crash refers to the global Headers constructor:

go/src/net/http/roundtrip_js.go

Line 80 in cd8f802

headers := js.Global().Get("Headers").New()

Seems like you need to set this, too.

[3goats]

OK so I tried adding the following to my NodeJS code.

global.Headers = {'content-type': 'application/json'};

This gave me the same error using NodeJS.

Interestingly, if I set this header in my Deno test script which does work without it, Deno then throws the same error.

[neelance]

https://www.npmjs.com/package/node-fetch#class-headers

[3goats]

So I added the headers to the globals globalThis.Headers = fetch.Headers

Then created a new Headers object. But still getting the same error.

Code is like this now:

require('./wasm_exec.js');
const fetch = require('node-fetch');


globalThis.fetch = fetch;
globalThis.Headers = fetch.Headers;

new Headers({'content-type': 'application/json'})

const buf = fs.readFileSync('/Users/user/development/godev/src/github.com/carlskii/vcert-wasm/vcert.wasm');
const go = new Go();

async function run() {
    const inst = await WebAssembly.instantiate(buf, go.importObject);
    go.run(inst.instance);
}
run()

[johanbrandhorst]

If you're still getting the same error, it's still failing to instantiate the headers. Not sure what other help we can provide, you need to emulate the environment that exists in the browser.