Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syscall/js: allow synchronous callbacks #26045

Closed
myitcv opened this issue Jun 25, 2018 · 23 comments

Comments

@myitcv
Copy link
Member

@myitcv myitcv commented Jun 25, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version devel +6fdbed0543 Mon Jun 25 02:22:05 2018 +0000 linux/amd64

6fdbed0

Does this issue reproduce with the latest release?

No; this is only present in tip.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPROXY=""
GORACE=""
GOROOT="/home/myitcv/dev/go"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/dev/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build521317867=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Picking up a conversation with @neelance on Slack, which picks up on the following exchange: https://go-review.googlesource.com/c/go/+/114197/8/src/syscall/js/callback.go#57.

Migrating this bit of code from GopherJS to WASM:

compDef.Set(reactCompShouldComponentUpdate, js.MakeFunc(func(this *js.Object, arguments []*js.Object) interface{} {

	// ... 

	return !curProps.EqualsIntf(nextProps)
}))

requires that I provide a function that (synchronously) returns a boolean value. Given the current async nature of callbacks this is not possible.

Furthermore, I think the use of async callbacks for handling of events is a potential problem/source of bugs. Let's assume I have a calIback for an onChange event. And let's assume I've used the syscall/js.NewEventCallback function to ensure PreventDefault. By the time our WASM "callback" has fired, the actual JS event callback will have already returned. Meaning the state of the thing for which we got the event callback might well have changed.

Therefore, to my mind this requires that we provide a way to build synchronous callbacks (functions), adding the caveats/warnings etc that @neelance describes in https://go-review.googlesource.com/c/go/+/114197/8/src/syscall/js/callback.go#57 that people should not block in such a function (or use a go routine to carry out blocking calls).

@neelance hopefully this accurately captures our conversation. I've added the "release-blocker" tag... but that can obviously be removed if we conclude that it's not critical, but based on our discussion I think, for now, it is.

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Jun 25, 2018

@myitcv Thanks for creating this issue.

Unfortunately I don't have a clean solution for this yet. It is kind of a fundamental conflict between Go's and JS' execution models.

The solution that GopherJS uses works, but I'm not really happy about. I'd rather find a cleaner way if possible. This will become even more complicated as soon as threads are available for WebAssembly.

Maybe other people have ideas. Let's discuss. /cc @bradfitz @aclements

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Jun 25, 2018

The main issue here is that there are JS APIs that expect a callback to synchronously return a value, in contrast to accepting a promise. I haven't yet seen such a case in the DOM API, but some 3rd party JS libraries (e.g. React, mentioned above) seem to do it.

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Jun 25, 2018

@myitcv

This comment has been minimized.

Copy link
Member Author

@myitcv myitcv commented Jun 25, 2018

Related: https://www.youtube.com/watch?v=cCOL7MC4Pl0

Not sure what the TL;DR of that video is, but I'd guess it's related to the point about event callbacks?

JS APIs that expect a callback to synchronously return a value, in contrast to accepting a promise. I haven't yet seen such a case in the DOM API

But I suspect that's largely because the event callbacks in the DOM don't have a return value. Instead, their "contract" is that the event passed as an argument be handled synchronously, and the "contact" of the element for which the event fired is also that the state will be X for the duration of the callback, but afterwards "all bets are off".

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Jun 25, 2018

I agree that the syscall/js package should probably stay closer to JS than to Go. Higher level packages can then hide these concepts from the user.

Still, we would need some way to express in Go that some code is special in the regard that it runs on the main JS event loop, whereas other Go code is not bound to the JS event loop. So what happens if you block while being bound to the event loop? Do we want to crash, like GopherJS does? If you want to access shared data and you want to use a mutex, can that mutex make it crash if it blocks? Or do we only want to crash if the goroutine on the event loop is blocked AND there is no other goroutine to run?

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Jun 25, 2018

I've investigated this some more, with two new insights:

  1. If a goroutine is bound to the event loop and it blocks, then it should be okay to simply block the event loop by not returning from WebAssembly until the goroutine got unblocked (by another thread). This is the most straightforward solution and easy to reason about.
  2. There may be callbacks inside of callbacks. If JS calls Go, which calls JS, which calls Go again synchronously, then the current way of how callbacks are implemented does not fit at all any more.
@aclements

This comment has been minimized.

Copy link
Member

@aclements aclements commented Jun 25, 2018

One thing I would worry about with an approach that involves continuing to schedule on the Go side until the callback returns is that it could cause deadlocks across the JS -> Go boundary, even without nested callbacks. For example, if the Go callback blocks because it's waiting on a channel message from another callback, but that other callback can't run until the JS event loop runs again, then we're stuck. I'm not sure, but this would seem like a desirable and common pattern. OTOH, any potential solution to this may be fundamentally incompatible with JavaScript's single-threaded event loop, and when programming in mixed JS/wasm perhaps you just have to be conscious of the single-threaded event loop and consider that in your concurrency model.

I wonder if this is something that's worth pushing back up to the WebAssembly standards committee. It's possible this could dovetail nicely with the work on threads. I'm not sure what form that's taking, but, for example, if the JavaScript event loop were thought of as just another thread communicating calls and returns with the wasm thread(s), then the Go callback returning could be treated as just a message to resume the blocked JavaScript call. (Implementation-wise, it would be nice if any thread could resume the blocked JavaScript call, but we could build that behavior even if it wasn't directly supported by wasm.)

@myitcv

This comment has been minimized.

Copy link
Member Author

@myitcv myitcv commented Jun 25, 2018

OTOH, any potential solution to this may be fundamentally incompatible with JavaScript's single-threaded event loop, and when programming in mixed JS/wasm perhaps you just have to be conscious of the single-threaded event loop and consider that in your concurrency model.

That's my gut feel on all of this too.

My distant memory of C# and .Net days is that there's a single GUI thread. I think of JS's event-loop thread in much the same way. Else race conditions on things like DOM access become horrendous.

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Jun 25, 2018

For example, if the Go callback blocks because it's waiting on a channel message from another callback, but that other callback can't run until the JS event loop runs again, then we're stuck.

Yes, this is accurate and I currently think there is no way to avoid it. I hope that higher level libraries will be able to hide this from users.

I wonder if this is something that's worth pushing back up to the WebAssembly standards committee.

From looking at the proposal for WebAssembly threads, it looks as if each thread will run in its own JavaScript execution context, similar to Web Workers. This means that only the main thread (on the event loop) is able to access the DOM. WebAssembly code on the main thread will always have to be called from JS code and has to always return the execution to JS so the event loop can continue. In short: The main thread IS the event loop.

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Jun 27, 2018

I'd say we should remove the release-blocker tag on this issue. Yes, it is something we should definitely address in the future and it limits what can be done with the first release, but the solution will be involved so it is unlikely to land for 1.11.

@myitcv

This comment has been minimized.

Copy link
Member Author

@myitcv myitcv commented Jun 27, 2018

@neelance sounds totally sensible to me.

@bradfitz @aclements - you've been looking at this work more than me, any thoughts/objections?

@myitcv myitcv removed the release-blocker label Jul 1, 2018
@myitcv

This comment has been minimized.

Copy link
Member Author

@myitcv myitcv commented Jul 1, 2018

I've removed the release-blocker label for now.

@bradfitz @aclements please shout if that's an issue from your perspective.

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Jul 5, 2018

@myitcv We should also change the milestone of this issue.

@theclapp

This comment has been minimized.

Copy link
Contributor

@theclapp theclapp commented Sep 1, 2018

For example, if the Go callback blocks because it's waiting on a channel message from another callback, but that other callback can't run until the JS event loop runs again, then we're stuck.

Yes, this is accurate and I currently think there is no way to avoid it. I hope that higher level libraries will be able to hide this from users.

Isn't this also true in pure Javascript too? They just get around it with promises or something -- if they do it at all. Frankly making the innards of one callback synchronously dependent on the innards of a different callback seems like a bad idea. What would that even mean? You can't finish pressing this button until you also press that button? No, I think you need to just start a separate goroutine for both of them. (Disclaimer: my frontend experience is limited.)

That said, I could imagine that it'd be pretty cool if the runtime could run the handler as far as it could go in the main thread until it blocks, and then run the rest of the handler in a different goroutine. So, like, a blocking operation in a callback would automagically go runtime.WithCurrentContinuation() or something.

@theclapp

This comment has been minimized.

Copy link
Contributor

@theclapp theclapp commented Sep 3, 2018

Once this is fixed, I think NewCallback should be renamed. Perhaps NewCallable or NewJSCallable or NewCallableFromJS? Since NewCallback would be used to expose any Go function of any kind ("callback" or not) to Javascript.

@BenLubar

This comment has been minimized.

Copy link

@BenLubar BenLubar commented Sep 4, 2018

How about just NewFunc? You're giving it a func on the Go side and getting something that can be used like a function on the JS side.

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Oct 11, 2018

I'm currently working on this. The WIP can be found at https://github.com/neelance/go/tree/wasm-sync-callbacks. Anyone interested in giving this a try can give feedback to me directly on the Gophers slack.

neelance added a commit to neelance/go that referenced this issue Oct 14, 2018
With this change, callbacks returned by syscall/js.NewCallback
get executed synchronously. This is necessary for the APIs of
many JavaScript libraries.

Fixes golang#26045
Fixes golang#27441

Change-Id: I591b9e85ab851cef0c746c18eba95fb02ea9e85b
neelance added a commit to neelance/go that referenced this issue Oct 14, 2018
With this change, callbacks returned by syscall/js.NewCallback
get executed synchronously. This is necessary for the APIs of
many JavaScript libraries.

Fixes golang#26045
Fixes golang#27441

Change-Id: I591b9e85ab851cef0c746c18eba95fb02ea9e85b
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 14, 2018

Change https://golang.org/cl/142004 mentions this issue: all: add support for synchronous callbacks to js/wasm

neelance added a commit to neelance/go that referenced this issue Oct 14, 2018
With this change, callbacks returned by syscall/js.NewCallback
get executed synchronously. This is necessary for the APIs of
many JavaScript libraries.

Synchronous calls to callbacks that have been triggered by calls from
Go to JavaScript get executed on the same goroutine. Asynchronous
callbacks get executed on an extra goroutine.

Fixes golang#26045
Fixes golang#27441

Change-Id: I591b9e85ab851cef0c746c18eba95fb02ea9e85b
@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Oct 14, 2018

Hey everyone. The CL is now polished and passing all tests. I would appreciate if you could test it with your own use cases and post the results here or on the CL. Thanks!

PS: The CL's code can also be found here: https://github.com/neelance/go/tree/wasm-sync-callbacks

@johanbrandhorst

This comment has been minimized.

Copy link
Member

@johanbrandhorst johanbrandhorst commented Oct 14, 2018

Tested branch with https://github.com/johanbrandhorst/grpcweb-wasm-example and https://github.com/johanbrandhorst/wasm-experiments, both which make extensive use of the callbacks in the WASM Roundtripper. No problems encountered.

HuckRidgeSW added a commit to HuckRidgeSW/hvue that referenced this issue Oct 15, 2018
See golang/go#26045 (comment) and
https://github.com/neelance/go/tree/wasm-sync-callbacks.

All examples work (as good as they did before, at least) with native
syscall/js.  I'd ripped out some code that required sync callbacks,
which I haven't put back in yet.  Also GopherJS not tested at all.
@theclapp

This comment has been minimized.

Copy link
Contributor

@theclapp theclapp commented Oct 18, 2018

I've updated my Vue.js wrapper hvue (see automated mentions above) to work with your wasm-sync-callbacks branch. Synchronous callbacks which return values and can access this all seem to work correctly.

I didn't fiddle around with blocking calls in event handlers and whether or not (or how) this blocks the main JS event loop.

Any thoughts on renaming NewCallback to NewFunc or something? (See earlier comments.) Or do you feel like any JS calling any Go is appropriately called a "callback"?

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Oct 22, 2018

@theclapp I agree that the term "callback" is a bit arbitrary. I just tried to change it to NewFunc, but it was hard to get rid of the term "callback" while still having documentation that is easy to read, since the term "function" is even more broad. So for now I'd prefer to not include this change in the current CL. We can have a subsequent issue and CL to investigate the name change some more.

@neelance

This comment has been minimized.

Copy link
Member

@neelance neelance commented Nov 10, 2018

@theclapp Now is the time to investigate a name change of Callback. Please take a look at #28711.

@golang golang locked and limited conversation to collaborators Nov 10, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
7 participants
You can’t perform that action at this time.