Should workers support py-* events?

[antocuni]

This is a sub-issue of #1762.
I expect some discussion here, so better to open a separate one.

Currently, we have 4 tests which works in the main thread but not in the worker:

pyscript/pyscript.core/tests/integration/test_02_display.py

Lines 174 to 189 in c6aaacd

	@skip_worker("NEXT: py-click doesn't work")
	def test_no_explicit_target(self):
	self.pyscript_run(
	"""
	<script type="py">
	from pyscript import display
	def display_hello(error):
	display('hello world')
	</script>
	<button id="my-button" py-click="display_hello">Click me</button>
	"""
	)
	self.page.locator("button").click()
	text = self.page.locator("script-py").text_content()
	assert "hello world" in text

pyscript/pyscript.core/tests/integration/test_02_display.py

Lines 208 to 222 in c6aaacd

	@skip_worker("NEXT: display target does not work properly")
	def test_explicit_target_on_button_tag(self):
	self.pyscript_run(
	"""
	<script type="py">
	from pyscript import display
	def display_hello(error):
	display('hello', target='my-button')
	</script>
	<button id="my-button" py-click="display_hello">Click me</button>
	"""
	)
	self.page.locator("text=Click me").click()
	text = self.page.locator("id=my-button").inner_text()
	assert "hello" in text

pyscript/pyscript.core/tests/integration/test_01_basic.py

Lines 113 to 140 in c6aaacd

	@skip_worker("NEXT: py-click doesn't work inside workers")
	def test_python_exception_in_event_handler(self):
	self.pyscript_run(
	"""
	<button py-click="onclick">Click me</button>
	<script type="py">
	def onclick(event):
	raise Exception("this is an error inside handler")
	</script>
	"""
	)
	self.page.locator("button").click()
	self.wait_for_console(
	"Exception: this is an error inside handler", match_substring=True
	)
	self.check_py_errors("Exception: this is an error inside handler")
	## error in console
	tb_lines = self.console.error.lines[-1].splitlines()
	assert tb_lines[0] == "PythonError: Traceback (most recent call last):"
	## error in DOM
	tb_lines = self.page.locator(".py-error").inner_text().splitlines()
	assert tb_lines[0] == "Traceback (most recent call last):"
	assert tb_lines[-1] == "Exception: this is an error inside handler"

pyscript/pyscript.core/tests/integration/test_01_basic.py

Lines 357 to 372 in c6aaacd

	@skip_worker("NEXT: py-click doesn't work inside workers")
	def test_py_attribute_without_id(self):
	self.pyscript_run(
	"""
	<button py-click="myfunc">Click me</button>
	<script type="py">
	def myfunc(event):
	print("hello world!")
	</script>
	"""
	)
	btn = self.page.wait_for_selector("button")
	btn.click()
	self.wait_for_console("hello world!")
	assert self.console.log.lines[-1] == "hello world!"
	assert self.console.error.lines == []

They all share the same fundamental reason:

<button py-click="myfunc">Click me</button>
<script type="py" worker>
    def myfunc(event):
        print("hello world!")
</script>

Should this work or not?
Let me try to play devil's advocate for both camps.

Team "it should not work"

One reason for saying "not" is that it's unclear what to do in case of multiple workers, or maybe main-thread+workers, e.g.:

<button py-click="myfunc">Click me</button>
<script type="py" src="aaa.py"></script>
<script type="py" src="bbb.py" worker></script>
<script type="py" src="ccc.py" worker></script>

In a situation like the above, we have three different interpreters: one in the main thread and one for each of the two separate workers, and it's not obvious/clear/well defined where to search and find myfunc, so better to avoid it entirely.

Team "it should work"

In case there is only one interpreter running in a worker, it is obvious where to look.
Moreover, we make it easier to switch from "main thread" to "worker", and it makes the two cases behaving more similarly.

Also, I expect that in the majority of cases people will have a single interpreter running anyway (either main thread or worker), so this is a solution which "just works" for the common case.

In case of multiple interpreters, we should think of a more explicit syntax and/or require them to use @when.

Personally, I am in the "it should work camp" :)

/cc @WebReflection @ntoll @tedpatrick @JeffersGlass

[WebReflection]

I'm team it should not work for the following reason: we export a @when decorator that supposed to be superior to py-xxx events and it should be clear that workers operate in a different world, they are not 1:1 related to main, no matter how much magic we introduce (magic that in this specific proposal screams for troubles).

However, for the it should work team I'd like to list all issues that comes to mind just now, without much further thinking of even more issues to consider:

In case there is only one interpreter running in a worker, it is obvious where to look.

not at all ... those events are nowhere leaked in the main thread, those are worker defined methods and I've no idea how to expose those functions automatically to the outer world out of user code. With main thread code those leak globally but the pyodide in a worker cannot be passed along a postMessage and it stays sandboxed in the worker ... we could orchestrate something like "leak all global methods to the main world for listeners after a single worked is executed and map it by the interpeter or custom type name" which might work but it requires a lot of checks and validations in the main thread around how many tags, what happens if a program add a tag at runtime, and so on ... this is the first slippery slope to me, but in an ideal static only HTML page this might be solved.

we make it easier to switch from "main thread" to "worker", and it makes the two cases behaving more similarly.

from a user perspective, this is the only argument that makes me think that maybe we should at least try to investigate this possibility

I expect that in the majority of cases people will have a single interpreter running anyway (either main thread or worker)

yes and no ... I expect mpy on the main thread any time and py in workers all other times when more complex stuff is needed ... luckily enough neither interfere with each other (mpy-click vs py-click) but it's worth mentioning this use case which I think will flourish in the near future.

In case of multiple interpreters, we should think of a more explicit syntax

that's unnecessary ... if you have multiple custom types (mpy and py) nothing conflicts with each other ... so that if there is only one <script type="py"> or one <py-script> on the page, nobody cares if <script type="mpy"> or <mpy-script> tags are somewhere else ... the contract here is 1 source of truth only and that's the per custom type where each custom type has only one interpreter still by contract so ... we're good there, this is not the issue we'll face to implement this.

and/or require them to use @when.

I think @when should be suggested in general because it keeps logic and expectations within the Python world, separating concerns with the layout, but I also believe py-click and friends are too damn and deadly easy to sell so I understand where you come from but then again, there are technical difficulties to fix.

My mild push back is mostly to avoid bloating the current achievement of shipping a tiny library with stuff that's maybe both early and too new to decide we should spend time fixing issues we never had to date, but on the other hand, if you folks are happy with me taking a chance to implement this idea, I'd be game!

[WebReflection]

P.S. this issue feels like it should've been a discussion instead ... just saying ... it's even presented with a question mark 😅

[ntoll]

@antocuni this is a really important discussion to have, thank you for bringing it up.

I'm in the "it should NOT work" camp. I think there are simply too many edge cases or work-arounds and a feeling of "magic" (one abstraction too far IYSWIM) needed to make it work when, as @WebReflection suggests, there are already ways to achieve the same effect (use @when rather than py-*) and it would complicate the implementation.

Also, we can be very explicit in the documentation about HOW to handle such things in workers.

[WebReflection]

I'd like to add an extra limitation of the "too magic" world we'd like to offer:

• there is no way to have 1:1 event listeners in main thread and worker thread, mostly due event lifecycle

To expand, the main thread cannot block the thread, it's not allowed by atomics, it's even disabled and deprecated by workarounds via synchronous XHR in the near future.

"... and where is the issue?" ... well, to start with, we cannot event.preventDefault() or event.stopPropagation() because by the time the worker receives the event from main this is kinda "dead" already. It will carry its own details, which is fine, but the borwser will throw an error or silently fail expectations because the event is forwarded in another world where it cannot prevent any default or stop any propagation ... so this is the biggest limitation out there that makes code impossible to guarantee as 1:1 from both worlds.

coincident solves this issue by providing an extra field to the third argument of addEventlistener but that works procedurally, not imperatively, and by no mean those operations can be decided at runtime (for input cases, as example) as these must be known at event listener attach time, not a tick later.

We could find some hack/workaround to make then code able to understand those operations in both main and worker thread, but it must be clear the code itself cannot be 100% ported across these two worlds, and specially for events this limitation must be understood.

This "gotcha" is bigger than any "how do I expose listeners to main" friction because it's a limitation backed in by the very same standard Web specifications we use as platform, so if any other argument doesn't convince we shouldn't move forward with this idea, this technical limitation will be a nightmare to implement and explain to our users.

Again, I am the least person that would ever say "it's impossible", we live on the edge out of modern features and little hacks here and there, but the event system in the main thread is unfortunately what it is and I've asked already WHATWG to provide an event.waitUntil(...) utility while solving this in coincident and there were crickets: whatwg/html#9540

P.S. even if we took over the entirity of the event lifecycle, we cannot re-dispatch trusted events in the wild, assuming our convoluted dance allowed that, 'cause isTrusted field can't rally be changed, or better, the browser won't ever consider user-land events in some particular case (mostly input and form or validation related) so ... there's that too.

[antocuni]

this issue feels like it should've been a discussion instead

yes I considered it but I think it's appropriate for it to be an issue, because there is an actionable item: depending on the outcome, the currently skipped tests should be either marked as @only_main or fixed.

there is no way to have 1:1 event listeners in main thread and worker thread, mostly due event lifecycle
[cut]

I didn't fully understand the in-depth explanation, mostly because I lack a lot of background but I also admit I didn't try too hard :). That said, I am thinking about a case like the following:

<button id="btn">Click me</button>

<script type="py" worker>
from pyscript import when

@when("click", selector="#btn")
def onclick(event):
    ...
</script>

This is supposed to work, isn't it?
Basically my proposal is that <button py-click="onclick"> should have the very same effect as using @when. Super simple to explain and to understand.

However, I seem to understand that for technical reasons it is not possible to guarantee that they behave in the very same way?
IF that's the case, then I think we should really reconsider whether we want to have py-click at all: having two things which are similar-but-slightly-different sounds like a good recipe to cause confusion.

[ntoll]

I think we should really reconsider whether we want to have py-click at all: having two things which are similar-but-slightly-different sounds like a good recipe to cause confusion.

@antocuni I wondered about this too... but py-click makes it very easy for "first steps" examples (i.e. it's explicit in the HTML that such-and-such a button will be handled by a Python function called whatever).

[antocuni]

@antocuni I wondered about this too... but py-click makes it very easy for "first steps" examples (i.e. it's explicit in the HTML that such-and-such a button will be handled by a Python function called whatever).

yes, I agree that it's super nice -- that's why I want it for workers as well :)

But only if the semantics of py-click and @when are equivalent: if not, we need to discuss. Let's first try to understand if there are differences and what they are.

[WebReflection]

This is supposed to work, isn't it?

not quite ...

@when("click", selector="#btn")
def onclick(event):
    # this works in main AND fails in workers
    event.preventDefault()

but @when is a declarative approach and it can use the coincident patch to make thing working from a worker:

@when("click", selector="#btn", invoke=["preventDefault"])
def onclick(event):
    # this works in both main AND workers

the details is that @when can then decide to add a listener that event.preventDefault() out of the box in main and, when we're running in a worker, just pass along that invoke to addEventListener as third argument.

This requires documentation around but there is no fix for the first example I've shown because it will throw or not do what users wanted to do out of Web standard constraints so there's nothing we can do about it.

I think we should really reconsider whether we want to have py-click at all

absoltely not because worker is a great use case for pyodide but a completely unnecessary use case for micropython or whatever PL we'll like to offer in the future.

In all main thread scenarios the xxx-click event is an absolute wonder and ease to explain and implement for anything that is not in worker ... it seems to me it's still hard to not think pyodide only with current offer ... we should try harder to think more broadly, imho.

P.S. also backward compatibility matters here ... and I think we agreed what works already and it's not broken, should still work. Adding explicitly a worker attribute out of the box and expect everything is fine is not what we should promote out there ... most of the time it is fine, but caveats must be considered, and only docs can save us, or our users, out there.

[antocuni]

absoltely not because worker is a great use case for pyodide but a completely unnecessary use case for micropython or whatever PL we'll like to offer in the future.

I disagree. MicroPython has the very same problems as pyodide w.r.t. blocking calls.

My not-so-secret idea is that a future version of pyscript should switch to worker-by-default, that's why I'd like the two cases to behave as similar as they can.
(but let's not discuss "worker-by-default" here, I know it's a can of worms)

[WebReflection]

MicroPython has the very same problems as pyodide w.r.t. blocking calls.

and so does JS ... that's why we can have asynchronous listeners ... MicroPython doesn't need worker because it doesn't really block in a relevant way the main thread on bootstrap. Performance on clicks are always good enough, being either pyodide or micropython, but pyodide can do much more thanks to micropip.

Asynchronous events still cannot preventDefault or stopPropagation at distance, these operations must be part of the still synchronous part and we cannot offer that due Web standard constraints.

[antocuni]

MicroPython doesn't need worker because it doesn't really block in a relevant way the main thread on bootstrap.

well, it depends what you do. If you have an infinite loop, it surely blocks :)

But it's not only about the performance: we want input() in micropython as well.

I'm not saying that "micropython in the main thread" is not a valid use case, it totally is. But there are tons of reasons for wanting "micropython in the worker" as well

[WebReflection]

If you have an infinite loop, it surely blocks :)

same in JS ... if you have an infinite loop there's no worker that can save your broken site or code, imho.

[antocuni]

same in JS ... if you have an infinite loop there's no worker that can save your broken site or code, imho.

yes, because if the infinite loop is in a worker, we can have a button to kill/stop it, and possibly start a debugger.
We are targeting the famous 99%, and I'm pretty sure they will write infinite loops eventually :)

[ntoll]

I'm 100% sure the 99% will while True:.