Symbol.private vs WeakMap semantics

[Igmat]

Originally I posted following messages in #149 answering to @erights concern about Membrane pattern implementation with Symbol.private approach. I've decided to create new issue here for several reasons:

1. Previous thread become too big to follow it
2. While this topic is related to original discussion it's actually important enough to be separated
3. 21 days passed from the moment I posted it, 16 from @erights promise to review it - and NO response so far (even though he was active in What would it mean to drop branding? rdking/proposal-class-members#10 (comment))
4. According to this: Developer experience of the syntax and semantics of this proposal #175 (comment), [Private] yet another alternative to #-based syntax #149 (comment) and September meeting it seems to be a major concern against Symbol.private
   • Unfortunately, there are no notes from latest meeting where it was discussed
   • there is also concern about lack of ergonomic syntax, but I already proposed few solutions (e.g. [Private] yet another alternative to #-based syntax #149, [Private] Metaprogramming vs Security #134 and [Private] yet another alternative to #-based syntax #149 (comment)), and I'm not the only one who did it (e.g. What would it mean to drop branding? rdking/proposal-class-members#10), so IMO it could be solved if Symbol.private will reach stage 1, so we'll be able to work on it or create follow-on proposals

@ljharb, @erights, @littledan, @zenparsing could you, please answer to this:

1. Is Membrane pattern a major concern?
2. Did I address Membrane+Symbol.private issues with following solution?
3. Is lack of ergonomic syntax is also a major concern?
4. Should I summarize syntax solutions for Symbol.private (including my yet unpublished ideas)?

Copied from #149 (comment)

---

@erights, as I said before there is a solution that makes your test reachable using Symbol.private, here's a sample:

function isPrimitive(obj) {
    return obj === undefined
        || obj === null
        || typeof obj === 'boolean'
        || typeof obj === 'number'
        || typeof obj === 'string'
        || typeof obj === 'symbol'; // for simplicity let's treat symbols as primitives
}

function createWrapFn(originalsToProxies, proxiesToOriginals, unwrapFn) {
    // `privateHandlers` are special objects dedicated to keep invariants built
    // on top of exposing private symbols via public API
    // we also need one-to-one relation between `privateHandler` and `original`
    const privateHandlersOriginals = new WeakMap();
    // we're keep track of created handlers, so we'll be able to adjust them with
    // newly exposed private symbols
    const allHandlers = new Set();

    // just simple helper that creates getter/setter pair for specific
    // private symbol and object that gets through membrane
    function handlePrivate(handler, privateSymbol) {
        const original = privateHandlersOriginals.get(handler);
        Object.defineProperty(handler, privateSymbol, {
            get() {
                return wrap(original[privateSymbol]);
            },
            set(v) {
                original[privateSymbol] = unwrapFn(v);
            }
        })
    }

    function wrap(original) {
        // we don't need to wrap any primitive values
        if (isPrimitive(original)) return original;
        // we also don't need to wrap already wrapped values
        if (originalsToProxies.has(original)) return originalsToProxies.get(original);
        const privateHandler = {};
        privateHandlersOriginals.set(privateHandler, original);
        allHandlers.add(privateHandler);

        //       note that we don't use `original` here as proxy target
        //                      ↓↓↓↓↓↓↓↓↓↓↓↓↓↓
        const proxy = new Proxy(privateHandler, {
            apply(target, thisArg, argArray) {
                thisArg = unwrapFn(thisArg);
                for (let i = 0; i < argArray; i++) {
                    if (!isPrimitive(argArray[i])) {
                        argArray[i] = unwrapFn(argArray[i]);
                    }
                }

                //          but we use `original` here instead of `target`
                //                           ↓↓↓↓↓↓↓↓
                const retval = Reflect.apply(original, thisArg, argArray);

                // in case when private symbols is exposed via some part of public API
                // we have to add such symbol to all possible targets where it could appear
                if (typeof retval === 'symbol' && retval.private) {
                    allHandlers.forEach(handler => handlePrivate(handler, retval));
                }

                return wrap(retval);
            },
            get(target, p, receiver) {
                receiver = unwrapFn(receiver);
                //       but we use `original` here instead of `target`
                //                         ↓↓↓↓↓↓↓↓
                const retval = Reflect.get(original, p, receiver);

                // in case when private symbols is exposed via some part of public API
                // we have to add such symbol to all possible targets where it could appear
                if (typeof retval === 'symbol' && retval.private) {
                    allHandlers.forEach(handler => handlePrivate(handler, retval));
                }

                return wrap(retval);
            },
            // following methods also should be implemented,
            // but it they are skipped for simplicity
            getPrototypeOf(target) { },
            setPrototypeOf(target, v) { },
            isExtensible(target) { },
            preventExtensions(target) { },
            getOwnPropertyDescriptor(target, p) { },
            has(target, p) { },
            set(target, p, value, receiver) { },
            deleteProperty(target, p) { },
            defineProperty(target, p, attributes) { },
            enumerate(target) { },
            ownKeys(target) { },
            construct(target, argArray, newTarget) { },
        });

        originalsToProxies.set(original, proxy);
        proxiesToOriginals.set(proxy, original);

        return proxy;
    }

    return wrap;
}

function membrane(obj) {
    const originalProxies = new WeakMap();
    const originalTargets = new WeakMap();
    const outerProxies = new WeakMap();

    const wrap = createWrapFn(originalProxies, originalTargets, unwrap);
    const wrapOuter = createWrapFn(outerProxies, originalProxies, wrap)

    function unwrap(proxy) {
        return originalTargets.has(proxy)
            ? originalTargets.get(proxy)
            : wrapOuter(proxy);
    }

    return wrap(obj);
}

const privateSymbol = Symbol.private();
const Left = {
    base: {
        [privateSymbol]: ''
    },
    value: '',
    field: privateSymbol,
};
const Right = membrane(Left);

const { base: bT, field: fT, value: vT } = Left;
const { base: bP, field: fP, value: vP } = Right;

// # set on left side of membrane
// ## set using left side field name
bT[fT] = vT;
assert(bP[fP] === vP);

bT[fT] = vP;
assert(bP[fP] === vT);

// ## set using right side field name
bT[fP] = vT;
assert(bP[fT] === vP);

bT[fP] = vP;
assert(bP[fT] === vT);

// # set on right side of membrane
// ## set using left side field name
bP[fT] = vT;
assert(bT[fP] === vP);

bP[fT] = vP;
assert(bT[fP] === vT);

// ## set using right side field name
bP[fP] = vT;
assert(bT[fT] === vP);

bP[fP] = vP;
assert(bT[fT] === vT);

Copied from #149 (comment)

---

I skipped fP->fT and fT->fP transformations in previous example for simplicity. But I'll mention such transformation here, since they are really easy handled when private symbols crosses boundary using public API of membraned object

set on left side of membrane

set using left side field name

bT[fT] = vT;

goes as usual, since happens on one (left) side of membrane

assert(bP[fP] === vP);

• since fP is private symbol, get called on proxy target
• which is special privateHandler object
• call privateHandler[fP] getter:
  • unwrap fP to fT
  • get bT[fT] which equals vT
  • wrap vT to vP
• assertion is TRUE

---

bT[fT] = vP;

• unwrap vP to vT
• set vT to bt[fT] as usual

assert(bP[fP] === vT);

• proxy skipped to target (privateHandler)
• call privateHandler[fP] getter:
  • unwrap fP to fT
  • get bT[fT] which equals vT
  • wrap vT to vP
• wrap vT to vP
• assertion is TRUE

set using right side field name

bT[fP] = vT;

• unwrap fP to fT
• set vT to bt[fT] as usual

assert(bP[fT] === vP);

• wrap fT to fP
• proxy skipped to target (privateHandler)
• call privateHandler[fP] getter:
  • unwrap fP to fT
  • get bT[fT] which equals vT
  • wrap vT to vP
• assertion is TRUE

---

bT[fP] = vP;

• unwrap fP to fT
• unwrap vP to vT
• set vT to bt[fT] as usual

assert(bP[fT] === vT);

• wrap fT to fP
• proxy skipped to target (privateHandler)
• call privateHandler[fP] getter:
  • unwrap fP to fT
  • get bT[fT] which equals vT
  • wrap vT to vP
• wrap vT to vP
• assertion is TRUE

set on right side of membrane

set using left side field name

bP[fT] = vT;

• wrap fT to fP
• wrap vT to vP
• set vP to bP[fP]
• proxy skipped to target (privateHandler)
• call privateHandler[fP] setter:
  • unwrap fP to fT
  • unwrap vP to vT
  • set vT to bt[fT] as usual

assert(bT[fP] === vP);

• unwrap fP to fT
• get bT[fT] which equals vT
• unwrap vP to vT
• assertion is TRUE

---

bP[fT] = vP;

• wrap fT to fP
• set vP to bP[fP]
• proxy skipped to target (privateHandler)
• call privateHandler[fP] setter:
  • unwrap fP to fT
  • unwrap vP to vT
  • set vT to bt[fT] as usual

assert(bT[fP] === vT);

• unwrap fP to fT
• get bT[fT] which equals vT
• assertion is TRUE

set using right side field name

bP[fP] = vT;

• wrap vT to vP
• set vP to bP[fP]
• proxy skipped to target (privateHandler)
• call privateHandler[fP] setter:
  • unwrap fP to fT
  • unwrap vP to vT
  • set vT to bt[fT] as usual

assert(bT[fT] === vP);

• get bT[fT] which equals vT
• unwrap vP to vT
• assertion is TRUE

---

bP[fP] = vP;

• set vP to bP[fP]
• proxy skipped to target (privateHandler)
• call privateHandler[fP] setter:
  • unwrap fP to fT
  • unwrap vP to vT
  • set vT to bt[fT] as usual

assert(bT[fT] === vT);

• get bT[fT] which equals vT
• assertion is TRUE

[jridgewell]

Is Membrane pattern a major concern?

Yes. Proposals will not gain consensus if they break membranes.

Did I address Membrane+Symbol.private issues with following solution?

Not all of them, see below.

Is lack of ergonomic syntax is also a major concern?

"Ergonomic syntax" meaning something like obj.#priv instead of obj[priv]? Then yes, this was a concern at the Sept meeting. It'll be hard to get everyone to agree without a specific syntax.

Should I summarize syntax solutions for Symbol.private (including my yet unpublished ideas)?

I think most of the committee likes .#. Anything that uses private for declaration and not # will be nearly impossible.

---

Membrane stuff:

• bT[fT] = vP;
  • You're first assumption is that vP is immediately unwrapped to vT. This is impossible, there's not proxy interaction to do the unwrapping.
• bT[fP] = vT;
  • Again, you're unwrapping as the first step which is impossible.

There's a few more like this, but you're getting close.

A big issue is all your proxy handlers will fail "has" checks, since you're using defineProperty, even if the original value doesn't actually have that private symbol. What's needed isn't installing these hooks onto the shadow target, but a way for proxies to learn of private symbols and then begin trapping them.

[Igmat]

@jridgewell thank you for response. I guess it's a good start for constructive discussion.

1. • bT[fT] = vP;

     • You're first assumption is that vP is immediately unwrapped to vT. This is impossible, there's not proxy interaction to do the unwrapping.

   • bT[fP] = vT;

     • Again, you're unwrapping as the first step which is impossible.

   You probably missed the main point - it's actually double membrane (one for code we are wrapping, second for everything outer). There is always a Proxy that does wrap/unwrap operations.

2. A big issue is all your proxy handlers will fail "has" checks

   Thanks for pointing this out - you inspired me to use Object.getOwnPropertyDescriptor which may also lead to avoiding unneseccary memory and time consumptions.

3. It'll be hard to get everyone to agree without a specific syntax.

   Should I treat this answer as yes to original question?

I also just get an idea how to create runnable tests that emulates Symbol.private for Membrane implementation. I'll publish small repo tomorrow with it, so we could have more material for investigation.

[erights]

@Igmat thanks for creating a self-contained issue. I will respond here. I would like to indicate by when, but I've already exceeded previous expectations --- my apologies.

[Igmat]

@jridgewell, @erights I just realized that I accidentally broke layout of my answer in #183 (comment). I updated the comment, so my points become visible - you probably may review it since those answers could be helpful for understanding.

[jridgewell]

it's actually double membrane (one for code we are wrapping, second for everything outer). There is always a Proxy that does wrap/unwrap operations.

Ahh, I didn't see that. I'm not sure how Mark feels about it, but that's not the membrane design as I understand it. On the left side and right sides, values shouldn't be wrapped. Only a left-sided object pulled through to the right side should be wrapped (and the other direction too).

you inspired me to use Object.getOwnPropertyDescriptor which may also lead to avoiding unneseccary memory and time consumptions.

I'm not sure I follow. The code I'm thinking of is fT in bP, which would go directly to the shadow target. Since there's a getter/setter there, it'd return true, even if the original object does not have a value on fT.

Should I treat this answer as yes to original question?

Yah, I have a feeling it's like a 90% requirement. Not strictly necessary, but the committee likes it enough that it must be a very, very good reason not to include it.

[Igmat]

Sorry for the delay (Christmas and New Year holidays takes too much preparation =/), but continuing the topic.

As I promised, here a little test project for Membrane pattern with Symbol.private:
https://github.com/Igmat/membrane-test-for-symbol-private

Important notes:

1. Since there are no Symbol.private in any environment, I treated all Symbols as private
2. This file implements Proxy replacement emulating how Proxy will interact with Symbol.private
   • Unfortunately, it seems to be limited, because it may not implement my vision of tunneling as I described in Private fields/properties & the membrane pattern... #158 (comment) in So why do we need tunneling? section
   • Most probably, it works like @rdking described in his Private fields/properties & the membrane pattern... #158 (comment)
   • From another hand, both previous statements could be wrong, since I didn't have enough time to test these assumptions
   • But important thing is that it actually has NO EFFECT on discussed problem of Membrane and Symbol.private issue
3. I made few minor fixes comparing to original post (proper proxy for function and correct usage of argArray)
4. No additional setup, just run npm test even without installing packages (I've used only node's native API)
5. If you find ANY mistakes or new important edge cases that isn't covered in tests, feel free to comment them here, or via issues/PR's in my repo

Next steps:

1. I, probably, can create more real-world like test-cases if needed. What do you think?
2. I haven't checked my idea about work-around for @jridgewell issue about has check. Is it really important? I realize that it could break some invariants, but at first glance it looks very minor problem.
3. Do you need better infrastructure around it (test-runner, compiler, readme, etc.)?
4. I'll create separate issue for Symbol.private syntax later, since it's a deal-breaker

@erights, @jridgewell, @jridgewell, @littledan does it help? Do you have any comments about it?

[rdking]

@Igmat Take a look here. I took a shot at a rough implementation of Symbol.private in ES6. See if you can't implement your testing with that.

[Igmat]

@erights, @jridgewell, @littledan I updated my Membrane implementation, so now it passes all tests.

Big thanks to @jridgewell with his help explaining some points I missed and providing better test suites.

[littledan]

I am sorry about the delay in these reviews. I want to just point out that a single slide might include constraints that are considered important, but might not be exhaustive, so an implementation passing certain tests might not be sufficient.

I think we had some pretty specific reasons for each aspect of the WeakMap semantics in this proposal. I wrote some notes about it in https://gist.github.com/littledan/f4f6b698813f813ee64a6fb271173fc8 . This might turn into a TC39 presentation if private symbols semantics are re-raised to the committee.

[rdking]

@littledan Would it be fair to say that the document you've got at that link is at least somewhat representative of TC39's desires for a private data proposal in general?

[littledan]

I'm wondering, would folks here be interested in moving the discussion to @zenparsing's repository on private symbols? I don't plan to champion private symbols, and so working out the details in this repository might be misplaced effort.

[bakkot]

If latest one still works after refactoring from _method() to #method(), then brand-checking for methods invocation is effectively dropped.

Yes, that's the case. They're checked only on access, not invocation.

I don't understand why do you agree to drop brand-check for invocation if it leads to problems for engine implementations

It wasn't dropped because of problems for engine implementations, but rather because it was a refactoring hazard, as you identified. I consider the check on access to be much more important than a check on invocation would be.

Also, don't you think that brand-checking in one cases and not brand-checking in another will lead to hard-to-reason-about situations?

It's always possible, but I think the current state will be less likely to lead to such situations than most possible alternatives.

[Igmat]

I'm wondering, would folks here be interested in moving the discussion to @zenparsing's repository on private symbols?

@littledan, sorry, but since private-symbols were rejected few times and haven't been moved even to stage 1, it looks like you ask us to stop bothering you with unimportant stuff. I hope I'm wrong.

[littledan]

I've been clear in this issue thread about my reasons for pursuing this proposal rather than private symbols and my expectation that private symbols will not get consensus in TC39. We've also been in contact by email about how to improve community outreach in TC39, and I look forward to working with you on the meetup in Minsk. I'm not the right person to cheer you on in your continued discussion of private symbols, but feel free to work with @jridgewell and @zenparsing on this effort.

[zenparsing]

@bakkot You picked on the semantics of "method" but didn't address the elephant in my post 😄

Let me state it more explicitly: the hash-stuff part of this proposal doesn't carry it's (significant) weight. I think it should be separated out from public fields.

[ljharb]

The fields proposal has already been merged, split on new lines, and re-merged, with consensus. Can you share the arguments you think would make it more likely for such a change to attain consensus?

[littledan]

OK, we're getting pretty far off-topic of Symbol.private vs WeakMap semantics. What do you all think we should do with this issue? I don't know how productive it is to have an general purpose chat room to continue repeating ourselves in.

[bakkot]

@zenparsing Sorry, let me respond explicitly, then, though I imagine you know my position.

I think your hidden-state library is great, but (as a user) I don't think it's anything like a substitute for language-level support for privacy, and the addition of some syntax sugar would not be enough to bridge that gap. And I'm confused by the implication that it would solve any of the semantic concerns that people have with this proposal, since it has almost identical semantics including interactions with proxies and so on. Or am I confused about what you're proposing?

And, separately, I think it's a very bad idea to develop public fields in a different proposal from private state - after having gone through that process, I think it ought to be clear that many of the design decisions are at least sometimes closely linked. If we're revisiting private state, I think we would also have to retract public fields.

[bakkot]

@littledan I don't think it does much harm to let the discussion continue here, despite being somewhat off-topic for the original purpose of the thread (and despite the fact that identical discussions have been happening for a decade). It may or may not be especially productive, but I don't think we need to shut it down unless the OP wants to keep the issue on-topic or it gets particularly acrimonious.

[zenparsing]

@bakkot

Perhaps this will clarify: I don't think developers should be encouraged to use hard private state in their workaday abstractions. For those that need it (polyfills, platform code, high profile libraries...maybe) "hidden-state" or something like it is good enough.

For most code, soft private via symbols, with some syntactic support, is a better solution (b/c it doesn't suffer from all of those "hard case" issues). We can develop that separately from the (public) fields proposal.

[bakkot]

@zenparsing Hm, ok. We've talked about that a lot, and I understand why people hold that position. But since we've talked about it a lot, I imagine you know that I disagree (rather strongly); I think all developers ought to be explicit about what their public API is, and it's pretty clear that anything shy of actual privacy will be considered by a fairly large fraction of developers to be a de-facto public API. (Not just in the obvious adversarial situations - this comes up a fair bit between two sufficiently separated teams within individual companies developing purely internal code, leaving us with this sort of thing.) I know a lot of the committee shares this position.

I don't know how we can reconcile this. We have to pick one - either the language encourages hard privacy as a first-class feature, or it doesn't. What can we do from here?

[ljharb]

I disagree very much; i think almost all code needs hard private - there are a long list of issues caused by soft private that already occur in the ecosystem, and have for decades - other than Proxy, i haven’t seen any problems in the wild from using hard private (which includes all variables closed over by functions).

[bigopon]

i think almost all code needs hard private

This is pretty untrue, or at least only true for your current job. It's not true for all of what I have written, and my current job. I'm raising my concern one more time: my moderately paid job, on a Friday afternoon with a production bug introduced by my coworkers who are not TC39 or any kind of JS experts, with the help of all the issues caused by enforcement from this proposal, will make me very miserable. Maybe I should get better job, but that's a different story.

[bakkot]

@bigopon, the claim is that bugs of that kind will be less likely if the language supports hard privacy as a first-class feature - that is, if it enforces that classes are used only according to the public API the author intended, rather than the public API they as non-experts accidentally introduced, not realizing that was what they were doing. This is very much the same as the design of closures.

[zenparsing]

The downsides of hard private being the default (with the associated syntax) are well documented. I believe that:

• The sum of those downsides outweighs the benefit
• That symbols are the most appropriate solution for the vast majority of application developers
• My vision results in a language that is easier to learn, because it's simpler

[bakkot]

Well, yes. As you know I and a number of other committee members disagree with all three of those points. We've talked about them all (excepting the third, insofar as your specific vision is fairly recent) at very, very great length. Again, what might we do from here?

[rdking]

@zenparsing Notice you said "vast majority of application developers"? Doesn't that mean that there are those of us who would benefit from hard private support? The "vast majority" of developers don't write libraries. However, the "vast majority" of developers are library users. If that small minority is the library developers, then won't the "vast majority" still benefit from a feature needed by the small minority?

[littledan]

I'm closing this issue, as we have been having an unstructured conversation about all aspects of several class proposals, on points which have been made repeatedly elsewhere, rather than discussing the topic of this issue. I encourage folks who are interested in private symbols to develop them in the private symbols repository and @jridgewell 's fork. We'll use this repository to develop the WeakMap-based class fields proposal.

[zenparsing]

For @bakkot

[bakkot]

@zenparsing As long as the language continues to exist, either it will or will not encourage hard privacy as a first-class feature. There's no "not to play" option other than burning the language to the ground.

And burning the language to the ground seems not to be an option at the moment, so...

[erights]

@bakkot Not playing does not necessarily mean not encouraging hard private. It can also be the position that the language already has adequate mechanism to support hard private, and that the status quo is a better way to support hard private than any of the mechanisms we propose to help support it. For example, all the examples at

https://github.com/erights/PNLSOWNSF/blob/master/examples/

are examples of hard private. The position that

https://github.com/erights/PNLSOWNSF/blob/master/examples/hiddenExample.js
https://github.com/erights/PNLSOWNSF/blob/master/examples/stateExample.js
https://github.com/erights/PNLSOWNSF/blob/master/examples/xyExample.js

dominate the others is "not playing", but is not an anti-hard private position. I am not yet saying what my ranking is of the examples in that directory. But I do consider this not-playing position to be a reasonable perspective on how to support hard private.

[bakkot]

@erights, the particular question is not what is supported, but what is encouraged. There does not appear to be much disagreement on the question of whether hard private state for classes is currently encouraged (it is not); the debate is whether it ought to be.

[rdking]

@bakkot @erights @zenparsing Rotate the question a bit and ask "Is hard private being discouraged by the current difficulty of staging and maintaining it?" I think the answer is a resounding "yes!"

[Igmat]

I've been clear in this issue thread about my reasons for pursuing this proposal rather than private symbols and my expectation that private symbols will not get consensus in TC39. We've also been in contact by email about how to improve community outreach in TC39, and I look forward to working with you on the meetup in Minsk. I'm not the right person to cheer you on in your continued discussion of private symbols, but feel free to work with @jridgewell and @zenparsing on this effort.

@littledan, ok, I understand your position. As I said already in emails, the outcome of Symbol.private vs WeakMap wouldn't affect our other collaborations (e.g. Minsk meetup and extended numeric literals).
Sorry if I made you feel uncomfortable or distracted from your works - I didn't want this.
So, from this point I stop add noise here and try to choose more appropriate places for discussions.

[hax]

but what is encouraged

@bakkot Yes, current proposal encourage hard private, but it also encourage many footguns (for example, encourage break Proxy by default).