Support R&T referencing objects in WeakMap/Set/Ref and FinalizationRegistry

[nicolo-ribaudo]

Updating the spec for WeakRef was hard and I'm not sure that I did it correctly. My goal was to allow the first console.log in this example to log undefined, true, while the second one must always log #[ Box({}) ], true.

{
  let o = {};
  let t = #[ Box(o) ];
  let ref = new WeakRef(t);
  await null;
  console.log(ref.deref(), t[0] === Box(o));
}

{
  let o = {};
  let t = #[ Box(o) ];
  let ref = new WeakRef(t);
  await null;
  console.log(ref.deref(), t === #[ Box(o) ]);
}

Fixes #233

[mhofman]

Based purely on the example given, I don't think that's the right semantics.
In both cases o is the identity conferring value, and since it stays "reachable" (aka live), then any value it confers its identity to is also reachable.

Here is a more appropriate example:

let oRef, tRef;
{
  let o = {};
  oRef = new WeakRef(o);
  let t = #[ Box(o) ];
  tRef = new WeakRef(t);
}
// In most implementations you'll need to do `await new Promise(resolve => setTimeout(resolve, 0));`
await null;
gc();
// Should be false if gc actually ran
if (oRef.deref()) {
  assert(tRef.deref() === #[ Box(oRef.deref()) ]);
} else {
  assert(!tRef.deref());
}

[mhofman]

I did a quick pass, but it's hard to spot all the modifications. Is this PR rendered anywhere to visually review the changes?

The definition of "unforgeable value" looks fine by me. We may want to strengthen the Box part when/if we forbid non-objects in boxes.

We may want to explicitly note that Symbol is not considered an unforgeable value, even though Symbol identities are in effect unforgeable, and that nothing here prescribes implementations from optimizing Symbol values when their identity is not observable.

The mechanics of the other changes look fine as this is simply renaming object -> unforgeable value. There are however a few instances that replace object by value instead of unforgeable value.

[mhofman]

We may want to replace "values" by "collection values", especially when nearby the definition of a collection key as an "unforgeable value"

[mhofman]

Based purely on the example given, I don't think that's the right semantics.

My reading of the spec changes match my expectation of how it'd work, but doesn't match the example given.

Edit: I hadn't thought this through fully, see below.

[ljharb]

I'm a bit confused; the diff is a lot to read through and the examples aren't super clear either.

Does a Box strongly or weakly reference the value it contains? Whatever the answer, it shouldn't be affected by whether the Box has been weakly referenced or not.

[mhofman]

Does a Box strongly or weakly reference the value it contains? Whatever the answer, it shouldn't be affected by whether the Box has been weakly referenced or not.

To put it in spec terms, having a live box value, and if the unbox operation in a given realm (assuming realm checks) is itself live, would allow the observation of the object value contained in the box. In simplified terms, the box is a strong reference.

If the box of an object is the target of a weakref, and there are no other ways to observe the identity of that object than through the weakref of the box, then there is similarly no way to observe the identity of the box through a weakref oblivious execution, and the object and the box can both be collected.

[acutmore]

Based purely on the example given, I don't think that's the right semantics.
In both cases o is the identity conferring value, and since it stays "reachable" (aka live), then any value it confers its identity to is also reachable.

That was my initial reaction too, but it does go against the optimisation allowed by https://tc39.es/ecma262/#sec-weakref-execution note-1-paragraph-2. A sufficiently smart compiler could see that t's equality is never directly observed, so it could be optimised out iff the WeakRef is allowed to drop the reference.

[mhofman]

I think that's where the "set of objects/unforgeable values" comes in. The identity of o is observed in both cases so it cannot be collected. And since t can be forged from the identity of o, its identity can be observed as long as o is itself is observed.

We should clarify that box/record/tuple are not intrinsically unforgeable, but they are a composite unforgeable.

Btw, that reminds me that a tuple/records which contains 2 or more boxes of different objects is no longer live as soon as any one of the composing object is not live. So the current "unforgeable" definition is not quite right.

cc @syg, we could use your liveness definition skills.

[acutmore]

And since t can be forged from the identity of o, its identity can be observed as long as o is itself is observed.

100% agree, t could be re-made from o. I think the question is, if a compiler chooses to see that it is not forged (maybe it can see that o does not escape). Should it be allowed to collect t (or maybe never allocate it in the first place)?

Or do we think that (cost-to-implement-optimisation / occurrence-of-optimisable-code) > value-added, i.e. allowing for this optimisation in the spec is not worth it as no-one would ever utilise it ?

[mhofman]

allowing for this optimisation in the spec is not worth it as no-one would ever utilise it ?

I doubt any implementation would ever be able to ascertain a record/tuple is never reconstructed while the unforgeable object it contains is still live.

[mhofman]

I think the problem comes from mixing the identity observation and the creation of said identity.

Objects and unregistered symbols have an unforgeable identity. The lifetime of an object's identity can be observed through weakrefs and finalization registry. The lifetime of a symbol identity cannot be observed. Values can be associated to the lifetime of an object's identity through weak collections.

A box has an identity derived solely from the object's identity it contains. A record/tuple has an identity only if it contains at least one object or symbol, in which case its identity is forged as a composite of these contained identities. The composite identity of a box/record/tuple can only be compared if all of the objects and symbols it's deriving its identity from can have their identity observed. If one of the composing object identities is no longer observable, the composite identity of the box/record/tuple is no longer observable. The composite identity, of a box/record/tuple can always be forged from all the objects and symbols it contains.

A box/record/tuple which contains at least one object has a composite identity whose lifetime can be observed. Its lifetime is equal to the shortest of the lifetimes of the composing objects. Since it'd be possible to observe the liveness of the composing objects through the box/record/tuple, an implementation must collect and prevent liveness observation of a composite box/record/tuple as soon as a composing object is no longer live (this is the same requirement as "set of objects" that need to be atomically collected).

[nicolo-ribaudo]

@mhofman The specification allows the .deref() call in this code to return undefined:

{
  let o = {};
  let arr = [o];
  let ref = new WeakRef(arr);

  await null;

  console.log(ref.deref());
  console.log(arr[0]);
}

how is it different from this example, given that tup has the same lifetime as arr (because it contains o which has the same lifetime as arr)?

{
  let o = {};
  let tup = #[ Box(o) ];
  let ref = new WeakRef(tup);

  await null;

  console.log(ref.deref());
  console.log(tup[0]);
}

[mhofman]

how is it different from this example, given that tup has the same lifetime as arr

It's different because while the scopes are the same, the identity have different lifetimes.

The identity of tup is tied to the identity of o, aka it can be forged solely from o. arr has it's own independent identity.
Since we're observing o, that means the identity of tup can be rebuilt, and is still observable.

Edit:
@acutmore made a similar point earlier but as I mentioned in #270 (comment), the engine would be allowed to clear the weakref if and only if it can be certain a tuple with the same structure and with the same boxed o is never rebuilt in a future execution. I doubt that's ever possible.

To put it in equivalent terms to the symbol as weakmap keys, it'd be allowing a registered symbol to be used as a weakref target, and allowing that symbol to be collected while the weakref is itself live, if the engine can be sure the program will never make a future call to Symbol.for() to retrieve that specific registered symbol. The difference is that for registered symbol the identity construction is always forgeable (from string), where for records/tuples the identity construction is tied to the objects contained.

[acutmore]

As already mentioned. I think the key input we need here is from a few different engine implementors. While the chances of engines using this optimisation may seem low, I'd like to imagine a world where JavaScript is still a popular language 50 years from now - and who knows what optimisation tricks they will be pulling by then 😎 and we wouldn't want to preclude something too hastily.

[mhofman]

Should we put this on the agenda for the Record & Tuple Monthly Call on Tuesday? I plan on joining. @syg, would you be able to attend?

[acutmore]

related conversations for context: tc39/proposal-weakrefs#179

[mhofman]

related conversations for context: tc39/proposal-weakrefs#179

That's the "set of objects" problem, right? And why I said if any object in a composite record or tuple is collected, the record or tuple must itself be collected atomically.

[nicolo-ribaudo]

@mhofman I rewrote this PR to so that my understanding matches yours (it's possible that the old text had the same semantics, but I saw it as different from what you were saying). This rewording only defines liveness for objects, and describes things in terms of "unforgeable values that contain a live object".

You can see a preview at https://jsfiddle.net/9sc48xyn/show.

[mhofman]

I mostly went over the liveness definition for now. This is a great direction, thanks for putting it together. However there is still the problem that if an observable value contains an object candidate for collection, then that observable value must be in the same collection set as obj. I believe the suggestions I offered do that.

Edit: thanks so much for the preview!!!

[mhofman]

whose referent is an unforgeable value that contains an element of S

In this case the referent would itself have to be in S, so an unforgeable value that contains should not have been added.

Suggested change

	<p>For some set of objects _S_, a <dfn>hypothetical WeakRef-oblivious</dfn> execution with respect to _S_ is an execution whereby the abstract operation WeakRefDeref of a WeakRef whose referent is <ins>an unforgeable value that <emu-xref href="#sec-unforgeable-values">contains</emu-xref></ins> an element of _S_ always returns *undefined*.</p>
	<p>For some set of <ins>unforgeable values</ins><del>objects</del> _S_, a <dfn>hypothetical WeakRef-oblivious</dfn> execution with respect to _S_ is an execution whereby the abstract operation WeakRefDeref of a WeakRef whose referent is an element of _S_ always returns *undefined*.</p>

[mhofman]

Same here, I think we can simplify by replacing all objects occurrences by unforgeable value, without touching the concept of "contains"

Suggested change

WeakRef-obliviousness, together with liveness, capture two notions. One, that a WeakRef itself does not keep an <del>object</del><ins>unforgeable value</ins> alive. Two, that cycles in liveness does not imply that an <del>object</del><ins>unforgeable value</ins> is live. To be concrete, if determining _obj_'s liveness depends on determining the liveness of another WeakRef referent, <del>_obj2_</del><ins>_val2_</ins>, <del>_obj2's liveness</del><ins>the liveness of the objects _obj2_ such that _val2_ <emu-xref href="#sec-unforgeable-values">contains</emu-xref> _obj2_</ins> cannot assume _obj_'s liveness, which would be circular reasoning.

WeakRef-obliviousness, together with liveness, capture two notions. One, that a WeakRef itself does not keep an <del>object</del><ins>unforgeable value</ins> alive. Two, that cycles in liveness does not imply that an <del>object</del><ins>unforgeable value</ins> is live. To be concrete, if determining <ins>_val_</ins><del>_obj_</del>'s liveness depends on determining the liveness of another WeakRef referent, <del>_obj2_</del><ins>_val2_</ins>, <del>_obj2_</del><ins>_val2_</ins>'s liveness cannot assume <ins>_val_</ins><del>_obj_</del>'s liveness, which would be circular reasoning.

[mhofman]

Suggested change

WeakRef-obliviousness is defined on sets of objects instead of individual objects to account for cycles. If it were defined on individual objects, then an object in a cycle will be considered live even though its Object value is only observed via WeakRefs of other <del>objects</del><ins>unforgeable values</ins> in the cycle.

WeakRef-obliviousness is defined on sets of <ins>unforgeable values</ins><del>objects</del> instead of individual <ins>unforgeable values</ins><del>objects</del> to account for cycles. If it were defined on individual <ins>unforgeable values</ins><del>objects</del>, then an <ins>unforgeable value</ins><del>object</del>in a cycle will be considered live even though its <del>Object</del>value is only observed via WeakRefs of other <del>objects</del><ins>unforgeable values</ins> in the cycle.

[mhofman]

Suggested change

	Colloquially, we say that an individual object is live if every set of objects containing it is live.
	Colloquially, we say that an individual <ins>value</ins><del>object</del> is live if every set of <ins>unforgeable values</ins><del>objects</del> containing it is live.

[mhofman]

Suggested change

	<p>At any point during evaluation, a set of objects _S_ is considered <dfn>live</dfn> if either of the following conditions is met:</p>
	<p>At any point during evaluation, a set of <ins>unforgeable values</ins><del>objects</del> _S_ is considered <dfn>live</dfn> if either of the following conditions is met:</p>

[mhofman]

Suggested change

	There exists a valid future hypothetical WeakRef-oblivious execution with respect to _S_ that observes the Object value of any object in _S_.
	There exists a valid future hypothetical WeakRef-oblivious execution with respect to _S_ that observes <del>the Object value of </del>any <ins>of the unforgeable values></ins><del>object</del> in _S_.

[mhofman]

Suggested change

The second condition above intends to capture the intuition that an object is live if its identity is observable via non-WeakRef means. An object's identity may be observed by observing a strict equality comparison between <del>objects</del><ins>unforgeable values</ins> or observing <del>the object</del><ins>an unforgeable value that <emu-xref href="#sec-unforgeable-values">contains</emu-xref> the object</ins> being used as key in a Map.</del>

The second condition above intends to capture the intuition that an <ins>unforgeable value</ins><del>object</del> is live if its identity is observable via non-WeakRef means. An <ins>unforgeable value</ins><del>object</del>'s identity may be observed by observing a strict equality comparison between <del>objects</del><ins>unforgeable values</ins> or observing <del>the object</del><ins>the unforgeable value</ins> being used as key in a Map.

[mhofman]

Suggested change

<p>Presence of an <del>object</del><ins>unforgeable value</ins> in a field, an internal slot, or a property does not imply that the <del>object</del><ins>unforgeable value</ins> is live. For example if the <del>object</del><ins>unforgeable value</ins> in question is <ins>an object that is</ins> never passed back to the program, then it cannot be observed.</p>

<p>Presence of an <del>object</del><ins>unforgeable value</ins> in a field, an internal slot, or a property does not imply that the <del>object</del><ins>unforgeable value</ins> is live. For example if the <del>object</del><ins>unforgeable value</ins> in question is never passed back to the program, then it cannot be observed.<ins>Similarly, if an object _obj_ is not observed by the program, any unforgeable value that <emu-xref href="#sec-unforgeable-values">contains</emu-xref> the object _obj_ cannot be observed.</ins></p>

I think this addition is the crux of it to say that a record/tuple/box identity cannot be observed if the program no longer has the ability to observe any of the objects it contains (the obvious observation path being the record/tuple/box itself if that is still observable on its own).

[mhofman]

There is more object -> unforgeable value work to do in this section.

[mhofman]

I am wondering if we could not merge this Unforgeable values definition and the contains object definition below. That would make it painfully obvious why symbols cannot be considered in this definition of unforgeable.

[nicolo-ribaudo]

I think there is a bit of misunderstanding. I'm only defining "liveness" for objects and not for R&T, so that in this example:

let o1 = { x: 1 };
let o2 = { x: 2 };
let rec = #{ o1: Box(o1), o2: Box(o2) };
let ref1 = new WeakRef(o1);
let ref2 = new WeakRef(rec);

o1 = null;
rec = null;

• The set { o1 } is not live, because a WeakRef-oblivious execution cannot call ref1.deref() (because o1 contains o1) and cannot call ref2.deref() (because rec contains o1).
• When garbage-collecting the set { o1 }, engines must empty ref1 and ref2 at the same time because both o1 contains o1 and rec contains o1.

I'm only defining liveness of objects (as it already was) and not of R&T containing objects because the "identity" of a R&T can be interpreted in two ways:

• The union (or intersection, idk) of all the identities of the objects within the record.
• The union (or intersection) of all the identities of the objects within the record, union (or intersection) all their location within the record.

In other words, I think in your mental model of how all of this should work, in this example you are observing the identity of the tuple:

{
  let o1 = {}, o2 = {};
  let tup = #[Box(o1), Box(o2)];
  let ref = new WeakRef(tup);
  await null;
  console.log(ref.deref(), tup[0].unbox() === o1, tup[1].unbox() === o2);
}

and thus ref cannot be emptied. However, in my mental model here you are not observing the identity of the tuple, and thus that ref.deref() call could return undefined.

(Note: I don't think that one approach is better than the other, but from what I understood those are the two "visions" we had of the original spec text I wrote in this PR).

By defining liveness only for objects, and by using the _V_ contains _o_ proposition (that I should probably rewrite to an AO) wherever interacting with the value stored in a WeakRef, I think I unequivocally specify the behavior that you were expecting.

[mhofman]

I think I don't understand the motivation for keeping unforgeable values other than object out of the collectible set. These are values with an unforgeable identity just as much as object, and the most important is that records/tuples (and box if unbox is available to the program for the box's realm) do have a strong reference to the objects they contain that the program can deference (by property walk). Because of that there is no way for an object to be collectible if all the records/tuples values that contain it are not also collectible, and the collectible set is the way to express that. Any WeakMap, WeakSet or WeakRef/FR targets must be cleared atomically in the Execution steps of all the collectible values.

The mental model I have is that a record/tuple/box have a strong reference to the objects they contain, allowing the program to observe the object's identity through them. An object contained in records/tuples/boxes may have an internal weak reference to these unforgeable values to optimize finding related identities.

the "identity" of a R&T can be interpreted in two ways:

I disagree. The identity of a R/T is the combination of its shape and the values contained in that shape, and can always be forged by building a R/T of the same shape with the same values. All values that are not boxes confer a R/T its structure (maybe we could find another word for that?). If a R/T doesn't have any boxes, it's structure is its identity, which can always be forged from its values (if you discount the unique symbol values for now). If a R/T does have a box, it's identity is the combination of its structure, of the object in contains, and of the position these objects have in the structure.

Said differently, two R/T do not have the same identity if they have the same structure and contains the same objects, but have them in different positions. And similarly if they are in the same position but the structure is different (e.g. a record with a different value for another field).

When you construct a record/tuple/box with some contained objects, the engine checks if any record/tuple/box of that structure exists and if they have the same objects in the same positions. If one exists, it returns the existing identity, if not it forges a new identity for the record/tuple/box. But to the program it always appears as it deterministically forging an identity from existing values.

In other words, I think in your mental model of how all of this should work, in this example you are observing the identity of the tuple, and thus ref cannot be emptied. However, in my mental model here you are not observing the identity of the tuple, and thus that ref.deref() call could return undefined.

I think this is getting into the definition of what observing an identity exactly is, and what optimization are allowable, which apparently I still don't fully understand.

Assuming the craziest of optimizations I don't believe tup[0].unbox() === o1 is observing the tuple's identity any more than it's observing o1 identity.

However if the program later uses both o1 and o2 to rebuild tup, the identity of tup is definitely observed again.

{
  let o1 = {}, o2 = {};
  let tup = #[Box(o1), Box(o2)];
  let wm = new WeakMap();
  wm.set(tup, true);
  await null;
  console.log(ref.deref());
  let tup2 = #[Box(o1), Box(o2)];
  assert(wm.has(tup2));
}

[mhofman]

I realize now that unforgeable isn't quite the right word for R/T/B containing objects. I just don't know what word would work to describe them and regular objects. Basically "object bearing value".

[mhofman]

We'll still need a similar PR with symbols as WeakMap keys, right?

[acutmore]

Supporting R/T that contain symbols as weak-targets could be a follow on proposal (post-mvp style) as it can be implemented in user-land.
Not sure what the chaimpion's opinion on this is.

[mhofman]

I'm not sure I follow. Are you saying that if symbol as WeakMap keys come first, that records/tuples containing symbols might not themselves be usable as WeakMap keys? I would find that particularly limiting and don't see how it could be remediated in user land.

[acutmore]

Rough gist of the concept if done in userland: https://gist.github.com/acutmore/e7e70306dbc02e2e5fa6e340b09eaa15

Updated gist: https://gist.github.com/acutmore/2aebf65223783bb452c37e4c3f1ac2a6