how does async work / how is async going to work in starlight

[andrieshiemstra]

Based on what you said in #44 i had a look at function.rs and generator.rs and i don't quite understand how that is going to work async

JavaScript engines typically rely on running in an EventLoop and provide a way to expose jobs which should be added to that loop.

QuickJS has an internal vec of pending jobs accessed by calling JS_IsJobPending() and then JS_ExecutePendingJob() (the downside of this is that you have call those after everything you do with the quickjs runtime).

Spidermonkey has a more elegant solution, you can add a hook (fn/closure) for when an async job has to run (SetJobQueue())

For quickjs I implemented an generic EventLoop here: https://github.com/HiRoFa/utils/blob/master/src/eventloop.rs

For Promises in starlight i was planning to add a simple hook (vm.registerJobAddHookThingy(Fn(Job)) which in turn adds the job to the EventLoop so it wil run async (but in the same thread) but i'm wondering if that's the way you want to go or have an other solution for async jobs...?

[playXE]

Based on what you said in #44 i had a look at function.rs and generator.rs and i don't quite understand how that is going to work async

I was talking on how async function state should be saved/restored and not the actual impl.

I don't actually know what is the best way to implement promises and async fns for JS engine actually. All my experience with coroutines is all in greenie which implements green threads and nothing more.

[andrieshiemstra]

i'll see what I can come up with and get back to you with a suggestion later ok?

[playXE]

That would be awesome! I still haven't finished exams so I do not have much time to work on such complex things like async. :(

[andrieshiemstra]

That would be awesome! I still haven't finished exams so I do not have much time to work on such complex things like async. :(

Well get back to studying then! i don't want to be the one that distracted you from your exams! ;)

Good luck with your exams!

[playXE]

Good luck with your exams!

Thank you!

Btw if you have questions about internals of Starlight I guess GH Discussions is better place. :)

[andrieshiemstra]

ok, so i made some first steps for async support

you can see my suggestion here https://github.com/HiRoFa/starlight/commit/a6e32c222fb3706a5fcdfccae4f3d45a51301938

i'm going for the spidermonkey strategy of beeing able to register jobs as async, is added this as an option in the RuntimeParams struct

I also added a test as an example which implements a setImmediate function (the sole purpose of which is do something async)

First of all of course i'd like to know what you think of the direction i'm going, and also what is the simplest way to turn a String into a JsValue :)?

Later i'm going to try an implement a Promise class using your define_jsclass! macro

[andrieshiemstra]

@playXE
ok so i managed to get to a point where i can construct a Promise in js and run the resolve method and such (Promise still needs a lot of work like .then/.catch and creating resulting sub promises) i just have a question about the API:

Can i regard a JsValue as a Handle to a GcPointer which means that as long as i don't drop all the JsValues the contained object will not be garbage collected? or do i need to do something extra to prevent garbage collection of an Object like my JsPromise?

[andrieshiemstra]

Can i regard a JsValue as a Handle to a GcPointer which means that as long as i don't drop all the JsValues the contained object will not be garbage collected? or do i need to do something extra to prevent garbage collection of an Object like my JsPromise?

looks like a No, your shadowstack seems closely modelled to spidermonkey, Can i just move a Rooted<JsValue> in a closure to prevent contained objects from beeing garbage collected? or do i need something like spidermonkey's PersistentRooted struct?

[andrieshiemstra]

@playXE
ok, i got the basics working and 1 out of 10 times my test case works.. the other 9 times i get a SIGABORT

see my code here
https://github.com/HiRoFa/starlight

i guess it has to do with GC but i'm unsure how to keep a JsValue from getting garbage collected...

[playXE]

@andrieshiemstra sorry for not answering your questions for too long, I have been busy for these few days. To be sure it is GC problem debugging is needed or you can enable conservative scanning for GC references in GcParams or using sl --conservative-marking It will check stack and registers for potential GC pointers so even If you didn't root some value GC will scan it.

Can i regard a JsValue as a Handle to a GcPointer which means that as long as i don't drop all the JsValues the contained object will not be garbage collected? or do i need to do something extra to prevent garbage collection of an Object like my JsPromise?

Simply use letroot!(val = stack, JsValue::<smth>) and GC will scan it. In fact you can use letroot! on any type that implements Trace trait.

First of all of course i'd like to know what you think of the direction i'm going, and also what is the simplest way to turn a String into a JsValue :)?

Your design looks nice, I think SerenityOS's LibJS does a similar thing.

and also what is the simplest way to turn a String into a JsValue :)?

Just invoke JsValue::to_string but it requires runtime instance because internally this function might execute JS code. EDIT:bruh I did read your question in reverse lol. Just invoke JsString::new,it accepts AsRef as input and you'll get GCed JS string and then you can simply invoke JsValue::new on iy

[playXE]

looks like a No, your shadowstack seems closely modelled to spidermonkey, Can i just move a Rooted in a closure to prevent contained objects from beeing garbage collected? or do i need something like spidermonkey's PersistentRooted struct?

Rooted is strictly bonded to shadow stack instance lifetime so you can't move it to the closure. I haven't worked on persistent roots but if you really need them I guess I could try to implement 'em.

[andrieshiemstra]

@andrieshiemstra sorry for not answering your questions for too long

No problem at all, i know you're busy. Are your exams going ok?

Your design looks nice, I think SerenityOS's LibJS does a similar thing.

Thanks, i'll continue this way. i'll also have a look at SerenityOS's LibJS for educational purposes :)

bruh I did read your question in reverse lol

Haha , yeah i managed to figure that one out :)

Rooted is strictly bonded to shadow stack instance lifetime so you can't move it to the closure. I haven't worked on persistent roots but if you really need them I guess I could try to implement 'em.

Well in order to keep objects rooted which are moved into a closure i think we kinda need something like that, basicly we just need something like spidermonkeys addRoot and removeRoot methods which just add a Trace to some collection and make sure the GC traces those.

A PersistentRooted struct would just call those methods on init and drop so we have a nice wapper.

My first idea would be to add that collection and methods to Shadowstack and trace them from the impl Trace for ShadowStack but i don't know if thats where you want it architecturaly..

Hmm, maybe it would be nicer to just have a HashMap or something in Runtime and trace that in the closure passed to this.gc.add_constraint() at line 595 in vm.rs?

[playXE]

Yeah idea with hashmap in runtime sounds fine to me

[playXE]

No problem at all, i know you're busy. Are your exams going ok?

Exams are OK, thanks :)

[andrieshiemstra]

Question, still getting SIGABORT with my promise code

In https://github.com/HiRoFa/starlight/blob/dev/crates/starlight/src/vm/promise.rs#L372

I impld Trace for my Promise class, but is that enough? or do i need to implement something somewhere to actually trace my JsPromise struct?

my testcase is still randomly passing or failing (if i disable GC the results are the same so i'm also wondering if realy is a GC issue..)

[playXE]

You need special treatment for ObjectTag::Promise in object.rs: https://github.com/HiRoFa/starlight/blob/dev/crates/starlight/src/vm/object.rs#L184in serialization/deserialization routines. In particular, you have to add ObjectTag::Promise => variants where it might need to exist. BTW There's a different way to define JS class: https://github.com/Starlight-JS/starlight/blob/dev/crates/starlight/src/jsrt/jsstd/file.rs#L314. This way is much easier to handle and does not require you to modify half of starlight code, I guess I'll transition to using this way soon because ObjectTag is not the easiest one to follow...

[playXE]

There's also a weird object_size_with_tag function so I guess you haven't added Promise object tag to this function so GC allocates less memory than object + promise data requires and this causes sigabrt. I think it would be much better if you used the second way like in file.rs. I'm going to deprecate object tags really soon.

[andrieshiemstra]

i altered the code to resemble your File example

now it fails just as often but always with a stackoverflow, vm.rs/Runtime has a with_heap() method.. at the end it calls this.gc.collect_if_necessary(); 9 out of 10 times this will cause:

thread 'vm::promise::tests::test_promise' has overflowed its stack
fatal runtime error: stack overflow

i'll continue debugging tomorrow but insights are appreciated :)

[andrieshiemstra]

hmm, if i remove all Promise init code and such i still get the stack overflow.. i'm going to rebranch and test first, then reapply my changes