feat($parse): secure expressions by hiding "private" properties #4509

Merged
merged 1 commit into from Oct 31, 2013

Conversation

Projects
None yet
Contributor

chirayuk commented Oct 18, 2013

BREAKING CHANGE:

This commit introduces the notion of "private" properties (properties
whose names begin and/or end with an underscore) on the scope chain.
These properties will not be available to Angular expressions (i.e. {{
}} interpolation in templates and strings passed to $parse) They are
freely available to JavaScript code (as before).

Motivation

Angular expressions execute in a limited context.  They do not have
direct access to the global scope, Window, Document or the Function
constructor.  However, they have direct access to names/properties on
the scope chain.  It has been a long standing best practice to keep
sensitive APIs outside of the scope chain (in a closure or your
controller.)  That's easier said that done for two reasons: (1)
JavaScript does not have a notion of private properties so if you need
someone on the scope chain for JavaScript use, you also expose it to
Angular expressions, and (2) the new "controller as" syntax that's now
in increased usage exposes the entire controller on the scope chain
greatly increaing the exposed surface.  Though Angular expressions are
written and controlled by the developer, they (1) typically deal with
user input and (2) don't get the kind of test coverage that JavaScript
code would.  This commit provides a way, via a naming convention, to
allow publishing/restricting properties from controllers/scopes to
Angular expressions enabling one to only expose those properties that
are actually needed by the expressions.

Oh golly!  Another PR from a favorite contributor.  I'm giddy with joy!

IgorMinar was assigned Oct 18, 2013

Contributor

chirayuk commented Oct 18, 2013

@IgorMinar: I have another idea for this that I want to run by you.  I'm considering allowing access to "private" fields if they resolve to a non-function object.  One could still use it to perform mischief depending on the app (e.g. if some controller published a function looked something like function foo() { this.dispatchMap[this._state](…); } and the attacker was able to update _state to some unsafe string.  And this is just with strings).  So my thought was to only allow read/getter access to such fields and only when using the [] index operator syntax if the resulting value was not a function.  Thoughts?  If we do that, I think we should do it as a follow up commit – possible even after the 1.2 release.

Member

petebacondarwin commented Oct 28, 2013

I am not convinced by this change.

In the first place, the only real pattern that this allows is using scope to share private properties between controllers. Other aspects of private properties can be secured by placing them in the closure of the controller and exposing secure getters and setters on the scope.

Using scope to share private data between controllers seems like a bad idea to me. This would be better done using services that are responsible for holding this shared state, which is injected into the closure of the controller when it is instantiated.

Also, security issues like this are hard for most programmers to comprehend and I am not convinced that, even with this change in place, programmers would use it correctly to secure their code.

Finally, it makes me feel even more strongly that the "controller as" syntax is not a great pattern. Controllers were originally an excellent place to put helper functions, aspects of which can be published, securely, to the scope. By allowing the template designer to attach the entire controller object to the scope, you basically open up this security issue.

@vojtajina vojtajina commented on the diff Oct 30, 2013

src/ng/parse.js
@@ -726,7 +739,7 @@ Parser.prototype = {
return v;
}, {
assign: function(self, value, locals) {
- var key = indexFn(self, locals);
+ var key = ensureSafeMemberName(indexFn(self, locals), parser.text);
@vojtajina

vojtajina Oct 30, 2013

Contributor

We can't I set "constructor", when I can get it?

@vojtajina vojtajina commented on the diff Oct 30, 2013

src/ng/parse.js
@@ -31,10 +35,16 @@ var promiseWarning;
// In general, it is not possible to access a Window object from an angular expression unless a window or some DOM
// object that has a reference to window is published onto a Scope.
-function ensureSafeMemberName(name, fullExpression) {
- if (name === "constructor") {
+function ensureSafeMemberName(name, fullExpression, allowConstructor) {
+ if (typeof name !== 'string' && toString.apply(name) !== "[object String]") {
@vojtajina

vojtajina Oct 30, 2013

Contributor

Just asking, how can name be a non string ?

@vojtajina vojtajina commented on the diff Oct 30, 2013

src/ng/parse.js
@@ -31,10 +35,16 @@ var promiseWarning;
// In general, it is not possible to access a Window object from an angular expression unless a window or some DOM
// object that has a reference to window is published onto a Scope.
-function ensureSafeMemberName(name, fullExpression) {
- if (name === "constructor") {
+function ensureSafeMemberName(name, fullExpression, allowConstructor) {
+ if (typeof name !== 'string' && toString.apply(name) !== "[object String]") {
+ return name;
+ }
+ if (name === "constructor" && !allowConstructor) {
@vojtajina

vojtajina Oct 30, 2013

Contributor

style: use single quotes (line 39 too)

Contributor

vojtajina commented Oct 30, 2013

Discussed with @chirayuk in person. LGTM, just do the little style fixes.

@vojtajina vojtajina commented on an outdated diff Oct 30, 2013

src/ng/parse.js
throw $parseMinErr('isecfld',
'Referencing "constructor" field in Angular expressions is disallowed! Expression: {0}', fullExpression);
+ } else if (name.charAt(0) === '_' || name.charAt(name.length-1) === '_') {
@vojtajina

vojtajina Oct 30, 2013

Contributor

style: remove the else

Contributor

vojtajina commented Oct 30, 2013

@petebacondarwin This change does not make sharing code between controllers through scope inheritance (which we do not recommend) any simpler.

I agree with you, that sharing stuff between controllers through scope inheritance is not a good thing, but I don't think this change promotes this style in any way. Please explain more, if you think it does...

You can achieve "private" state for controllers even with "controller as" syntax:

var MyCtrl = function() {
  var privateMethod = function() {};
  this.publicMethod = function() {};
};

I think this change is mostly for people using the "prototype" style (eg. Google internal projects).

@chirayuk chirayuk feat($parse): secure expressions by hiding "private" properties
BREAKING CHANGE:
This commit introduces the notion of "private" properties (properties
whose names begin and/or end with an underscore) on the scope chain.
These properties will not be available to Angular expressions (i.e. {{
}} interpolation in templates and strings passed to `$parse`)  They are
freely available to JavaScript code (as before).

Motivation
----------
Angular expressions execute in a limited context.  They do not have
direct access to the global scope, Window, Document or the Function
constructor.  However, they have direct access to names/properties on
the scope chain.  It has been a long standing best practice to keep
sensitive APIs outside of the scope chain (in a closure or your
controller.)  That's easier said that done for two reasons: (1)
JavaScript does not have a notion of private properties so if you need
someone on the scope chain for JavaScript use, you also expose it to
Angular expressions, and (2) the new "controller as" syntax that's now
in increased usage exposes the entire controller on the scope chain
greatly increaing the exposed surface.  Though Angular expressions are
written and controlled by the developer, they (1) typically deal with
user input and (2) don't get the kind of test coverage that JavaScript
code would.  This commit provides a way, via a naming convention, to
allow publishing/restricting properties from controllers/scopes to
Angular expressions enabling one to only expose those properties that
are actually needed by the expressions.
3d6a89e

@chirayuk chirayuk merged commit 3d6a89e into angular:master Oct 31, 2013

1 check passed

default The Travis CI build passed
Details

You just broke the apps of everyone who is building with CouchDB. The convention there is that document ID's and revisions are prefixed with underscores.

Contributor

btford commented Nov 8, 2013

@krotscheck this only "breaks" them in the sense that you cannot directly bind to these properties.

You can easily:

  1. alias them
  2. write a method to return the id given a document

Developers should have to explicitly choose to expose these kinds of properties in their app. This is consistent with the intention of the change

MongoDB also has primary keys stored on an _id field, iirc.

@btford That may be the case, however javascript already provides the ability to make variables private via scope isolation. Why enforce this convention? Furthermore, why fix something that isn't broken?

angular.module('mymodule').controller('myController', function() {
      var privateVariable = 'this.variable.is.private';

      return {
            getPrivateVariable: function () { return privateVariable; }
      }
});

Note, I'm using "Scope isolation" in the context of "Scope inside of the function/lambda", not "AngularJS scope"

Contributor

cburgdorf commented Nov 8, 2013

I try to imagine a use case where this actually safes us from anything? I can't see the real benefit.

@krotscheck Agreed. I don't understand why you can just use javascript scopes to hide state like this. If this feature is needed, it shouldn't be done by detecting an underscore. It could be done by declaring that variable is private, perhaps a helper method.

Contributor

boneskull commented Nov 8, 2013

For a breaking change this is of very small benefit and probably even to a smaller number of users. I like this idea, but as a compromise: let us toggle this behavior.

@btford Bindings are not the only ones impacted. More investigation suggests that the introduced bug exists in ngResource, where resource variable interpolation triggers ensureSafeMemberName().

return $resource(myBaseUri + '/resource/:_id', {_id: '@_id'});

This benefits a prototype-style controller using the "controller as" syntax:

function WidgetCtrl() {
}

WidgetCtrl.prototype._doSomethingPrivate = function () {
};

WidgetCtrl.prototype.doSomethingPublic = function () {
  // ...
};

angular.module('myApp').controller('WidgetCtrl', WidgetCtrl);
<div ng-controller="WidgetCtrl as widget">
  <button ng-click="widget.doSomethingPublic()">do something</button>
</div>

where _-prefixed members on the controller prototype are hidden from the scope.

Contributor

btford commented Nov 8, 2013

of very small benefit and probably even to a smaller number of users

In a constructive conversation, I'd avoid making these types of assumptions. A feature that is of "small benefit" to you might be important to others. This makes security audits much easier for large applications using controllers with methods defined on their prototype.

I understand that this is annoying issue, and the implications @krotscheck pointed out in ngResource seem undesirable.

let us toggle this behavior

A flag that disables this feature seems reasonable. @chirayuk, what do you think? Would anyone like to prepare a PR?

Contributor

cburgdorf commented Nov 8, 2013

@btford just out of curiosity, why exactly does it make security audits easier? What exactly makes it more secure that such method can't be accessed from within expressions? Since we all know that true privacy is not possible when using a prototype based programming style in JavaScript we have long lived by the rule to just accept things as private when they were communicated as such.

Contributor

cburgdorf commented Nov 8, 2013

@btford or by "secure" do you mean "securing" that developers don't abuse the code base? I'm just curious if we mean the same thing with "security" in this context.

Mparaiso commented Nov 8, 2013

I quote :

Hi thanks for the opportunity to discuss that "feature".

JavaScript does not have a notion of private properties

Yet you want to do that ? I understand the point, but convention over configuration is a bad,thing.Want that ? let developpers configure filtered methods through an Angular API. _ for "private" members is not a good practice in any language. it's basically HUNGARIAN NOTATION and not understanding the nature of javascript;

Some mark their private with private prefix, some other use @@ or other symbols so members can only be called when they are quoted ( myobject['@@mymember'] ) , Angular owns the $$ namespace . It will make things confusing if you go that route.

Regards.

Contributor

cburgdorf commented Nov 8, 2013

@Mparaiso good point with $$ being used internally by Angular to mark private stuff.

Contributor

boneskull commented Nov 8, 2013

@btford Fair enough. As @vojtajina commented, this is important to devs writing internal apps for Google.

I'll look at writing a PR later tonight if nobody has beaten me to it. I'll hazard a guess the code would be similar to a simple toggle like parseProvider.unwrapPromises() provides.

tortlieb commented Nov 9, 2013

I'd rather see it removed than being a flag, but if it's a flag, please make the default that underscores are allowed, as this was the 1.2.0-rc.3 behavior.

On another note, this should have been added to one of the release candidates instead of being shoved into the actual release, so the full impact of the code could be assessed. Is it too late to call this rc4?

Contributor

kzar commented Nov 9, 2013

@tortlieb agreed, I know it's not really my place to say but I feel bringing in breaking changes like this between rc3 and the actual release is crazy. Personally I'd like 1.2.0 to be a really solid version that works as expected and that is documented well.

Contributor

btford commented Nov 9, 2013

The _id issue is obviously a nuisance for databases that have this convention. We're investigating options, but suggestions (aside from outright reverting the change :P ) are welcome.

I'm going to try to address some of the questions below.

why exactly does it make security audits easier?

The short answer is "because Angular now guarantees that foo._private is not available to expression bindings."

It's hard to think of a simple, clear, obvious example off the top of my head, because most exploits are none of these things.

function Ctrl () {}
Ctrl.prototype._deleteUser = function () {};
Ctrl.prototype.promptToDeleteUser = function () {
  // bring up dialog box, do some sanity checks, then _delete
};
<div ng-controller="Ctrl as ctrl">
  <select ng-model="action">
    <option value="promptToDeleteUser"></option>
  </select>
  <button ng-click="ctrl[action]()">
</div>

Consider in AngularJS pre-1.2.0: you'd think that because there is no <option value="_deleteUser">, the ngClick couldn't call it. But an attacker, through some elaborate method, might inject a script that changes <option value="promptToDeleteUser"> to <option value="_deleteUser">.

If you use the prefix, it makes it more obvious to verify that your app is safe: you know that ngClick cannot call _deleteUser, so you know that only promptToDeleteUser needs to perform these sanity checks. This means an auditor can focus their efforts into testing just this one method rather than several. This seems small, but in a large app with hundreds of controllers and thousands of methods, the ability to really focus your security efforts on the places that matter is huge.

Why not just defensively program and do these checks in all methods? There are cases where for performance implications, it would be slow to always do a full set of checks/sanitization. This offers developers a pragmatic choice.

Why not just use closures? This means you get a bound method for each instance of a controller. With prototypes, you only have one. This potentially saves memory on large apps, or for apps with more constraints (like ones targeting mobile). It's also easier to use prototypes with tools like Google Closure Compiler. We think developers should have the choice to use either in a way that is safe.

You can never be completely certain with security, but offering developers this guarantee makes it easier to cull some of the possible attack vectors. This, in turn means that security reviewers can focus on the un-prefixed parts of the app that are exposed to users/attackers.

I don't think that security alone is the only benefit from this change. I've switched to this style for a few apps and I find that it helps communicate intentions better. If you intend for some API to be "private" Angular now binds you to use this. This is consistent with the principle of least surprise; if you refactor some method names that are "private," you know your template does not rely on these APIs.

this was the 1.2.0-rc.3 behavior.

Our policy is that RCs can have breaking changes. Yes we could keep doing RCs until everything "stabilizes," but we found that doing that in the past meant slower releases. Remember that no one is forcing you to update to 1.2.0. You're free to keep using 1.2.0-rc.3 and update at your own pace.

If you want to know about breaking changes earlier, you can subscribe to AngularJS's Github notifications. All major changes like this are discussed publicly via PR, just like this one.

Mparaiso commented Nov 9, 2013

This is consistent with the principle of least surprise

if you use that convention at first place. Yes. The real surprise is a JS framework enforcing _ as private anywhere.
_ doesnt not make something private in javascript. So one could use that argument against the same point. It is surprising.

It isn't a JS framework enforcing _ as private in a Javascript expression though. Angular expressions aren't Javascript.

Contributor

btford commented Nov 9, 2013

_ doesnt not make something private in javascript. So one could use that argument against the same point.

As @tomjakubowski said, {{ binding }} isn't JavaScript. It's an "Angular expression." Your expectation should be that Angular expressions are a "safe" subset for use in templates. However, it seems that other contexts (ngResource) are affected as well.

At this point, arguing about whether or not this is a good idea is just adding noise to this thread. Suggestions for accommodating both use cases are more productive.

kofalt commented Nov 9, 2013

Everything about this change has a "rushed in to meet a merge window" feeling about it.
I would argue that this is not how quality software is made.

While sharing state outside the reach of expressions is an idea that holds merit, I'm not at all convinced the above behavior is the best approach, or even a reasonable one. Implicit in the merge request and @btford's comments is a need to handle state that is not & should not be available in expressions.

This begs the question why it's on the scope at all - the issue as stated makes it sound like there should be another, entirely separate way to handle this sort of interaction. Angular apps already tend to end up with heavily-loaded scopes; moving anything possible out of the "hash cycle" seems desirable.

Introducing this rough-cut attempt after the final release candidate is a strange choice, and it would appear the (albeit limited) community response has been universally confused and disappointed. This problem would be better served by an extended discussion of the actual problem at hand. The lack of a PoC does not help; working examples are necessary for reasonable discussion on this complicated topic.

It would be difficult to argue that calling charAt() all day is the best we can do here.

I would advocate moving this from 1.2 to 1.3, or to compromise releasing 1.2.1 to disable this behavior (by default). There simply wasn't enough forethought around this change.

Contributor

btford commented Nov 9, 2013

@kofalt this type of comment is rude and unhelpful. This PR was openly discussed for ~3 weeks. We cannot foresee how this interacts with every software or usage under the sun. We are happy to get feedback, but keep it civil and constructive.

Can someone please give an example of how this change breaks for you? How are you using _id ? That would be most helpful for us to move the discussion forward.

Can we start with getting ngResource fixed? I can't find a place in our code where we use _id in view expressions.

Mparaiso commented Nov 9, 2013

@tomjakubowski

I believe the point was the "least surprise" . Do you really think triggering that behavior with _ is the least surprise ? Do you think introducing a semantic change that doesnt mean anything in javascript is the least surprise? expressions are not javascript , yet how do they figure out which member to call ? by their name. Now i cant call members the way I want though it is valid javascript.

Class based controllers were introduced so people coding with prototypal inheritance could go on coding that way. Now Angular tells me I need to code the Angular way and introduces semantics that dont exist in js and in my code, where _ could be used for something else than a private flag because of a dev team convention.

@btford

INTENT AND SOLUTION

My point is your point is flawed. It is a unilateral decision that doesnt have any justification other than being convenient for some people using that convention at first place. I'm ok with the intent, I just think people should be honest about the why of the solution : there is no why. one decided _ is private, period.

kofalt commented Nov 9, 2013

@btford I'm confused how our exchange could be interpreted as rude; that was not my intention. My comment represents an honest assessment of the situation, and I very specifically tried to take a neutral tone. Let's not resort to name-calling.

To repeat and clarify: it would seem rather evident that there was not nearly enough visibility on this change, that the implementation's approach is quite possibly flawed, and a new discussion is needed, centered on the problem.

Moving this change to 1.3 or releasing 1.2.1 to turn this behavior off seems like a mature response to this issue.

kofalt commented Nov 9, 2013

Further, IMO asking for clarification about usage of _id is focusing on the symptom and not the disease: a blacklist of prefixes (and postfixes) is a strange way to handle the problem as stated.

In my opinion this solution is architecturally flawed, not superficially flawed.
I wish I could have made this point sooner, but I only found out about it post-release :)

Contributor

btford commented Nov 9, 2013

@kofalt

rough-cut attempt
rushed
that this is not how quality software is made

These are rude, and your overall dismissive tone is not welcome. You're free to disagree, but you should do so respectfully. Disrespectful comments are harmful to community discourse. They dissuade developers from becoming involved.

This change was discussed in public on this thread for weeks before being merged. If you'd like to receive a notification whenever there is a pending code change (via PR), you can opt into email notifications on Github so that you never miss anything.

We will happily try to accommodate a change that works well for everyone. To do that, we need to understand how you are using underscore prefixed properties that is incompatible with this change. "Roll back the change" is not a reasonable solution. We added this feature to support another valid way to use Angular.

Contributor

caitp commented Nov 9, 2013

I didn't really want to jump into this, but is there any good reason that it is necessary to decide a property is "private" based simply on its name?

What about providing a means to simply register properties as private, and simply ensure that the property is not found in "private registry" in ensureSafeMemberName()? It seems like it wouldn't be too difficult to use prototypal inheritance to easily provide "globally private properties" as well as ones specific to a particular scope (and its children) --- This way, those who need to take advantage of the feature can do so in a way which affects their entire app, without imposing restrictions on property name patterns.

Scope.prototype.privatize = function(properties) {
  if (isString(properties)) {
    properties = string.split(/\s+/);
  }
  if (isArray(properties)) {
    forEach(properties, function(name) {
      this.$$privateProperties[name] = true;
    });
  }

// ...

function ensureSafeMemberName(name, fullExpression, allowConstructor, context) {
  if (typeof name !== 'string' && toString.apply(name) !== "[object String]") {
    return name;
  }
  if (name === "constructor" && !allowConstructor) {
    throw $parseMinErr('isecfld',
        'Referencing "constructor" field in Angular expressions is disallowed! Expression: {0}',
        fullExpression);
  }
  if (isObject(context.$$privateProperties) &&
     context.$$privateProperties[name] === true) {
    throw $parseMinErr('isecprv',
        'Referencing private fields in Angular expressions is disallowed! Expression: {0}',
        fullExpression);
  }
  return name;
}

});
Contributor

btford commented Nov 9, 2013

@caitp That's a reasonable suggestion, but I worry that would make it too cumbersome to use. During an audit, you'd have to constantly check back and forth between the registry and the controller. I think the locality of the private flag is important.

Besides @krotscheck's case with ngResource are there any other places that this is causing trouble for real apps using 1.2.0? If not, I'm going to make a new issue to discuss that bug.

kofalt commented Nov 9, 2013

@btford I'm afraid I don't see how I could possibly convey a dissenting opinion with language that is milder than the examples you quoted. Bringing up various Github features isn't really relevant; it's too late. We're already here :)

Let's move the conversation back on track. To be frank, I think this blacklisting-by-name approach is fundamentally flawed. Above I've laid out a few reasons why; I don't think stuff like this belongs on the scope, and if there's a need for certain properties to behave differently than their brethren it definitely feels like something else not the scope should hold these... privileged citizens. The stated desire makes sense; I just think there's better ways to approach the problem.

Which is why I think discussions about underscores are rather missing the point. Shoving copious functions on the scope was the biggest mistake I made starting out with Angular, and it was also the mistake that took the longest time to correct. This is what makes me feel like a more robust approach to 'private' properties is needed.

I'm curious what you think about this proposal, because I feel something like the above is going to be a better long-term solution than blacklisting.

Contributor

btford commented Nov 9, 2013

@kofalt you crossed the line when you started making assumptions about the quality of the code and development methodology. The entire sentiment of "this feature is rushed" is totally unproductive.

The "private method" feature becomes more useful with "controller as" and prototype-based controllers. The issue is not with adding additional things to the scope but to the controller, which is made available on the scope. There are plenty of real cases where this is a useful practice. See my comment above.

Do you have any real code that is negatively impacted by this change? If so, a short snippet or Plunkr would be helpful. I'm more concerned with helping users who are negatively impacted by this change than holding an endless debate.

Contributor

caitp commented Nov 9, 2013

I agree that my suggested solution may be cumbersome, however I would say this:

  • Things exposed via scope should typically be exposed via scope because they're SUPPOSED to be accessible via expressions / templates (with the possible exception of things like the Scope-graph data structures and $$phase) (I admit that this doesn't help the controller as syntax, but meh, I think people have been warned about the "security implications" of controller as syntax in the past so there's nothing new there)
  • If it is desirable for whatever reason to have things in scope which are not accessible via expressions, a single API call for setting private property names / patterns is probably not that much work to provide it
  • While inheriting private property rules may not necessarily be a great idea, at the very least it could be doable to have a set of "global" rules which affect all scopes, as well as a set of per-scope rules
  • I think, if we need to have these sorts of audits, a good place to perform them would be during testing or some kind of static analysis, although I haven't really thought of a good way to do this yet.

kofalt commented Nov 9, 2013

Basically, a library with surprising (and arguably arbitrary) semantics is worse off than a library without them. @Mparaiso's comment was very on point when he talked about confusing namespace problems. To answer your direct question, I'd argue that all code is negatively impacted by this change. The fact that apps which use underscores will break in 1.2 is entirely coincidental.

Despite multiple attempts to propose a hopefully-superior alternative, I haven't heard a response or acknowledgement that there's more than one way to accomplish this, and that the best way might not be the one which was chosen. This is unfortunate, because I really do think there's a missed opportunity here.

I hope everyone's able to reach a good outcome here, and nobody wants to get bogged down pointing fingers. Though I do hope you consider at minimum defaulting this behavior to off in a 1.2.1 release, which was earlier proposed.

Contributor

btford commented Nov 9, 2013

Things exposed via scope should typically be exposed via scope because they're SUPPOSED to be accessible via expressions / templates

I think this point misses the import case of prototype-based controllers + controller as.

We could go back and forth on this all day, but I'd much rather look at this through the lens of example code for real use cases.

Contributor

boneskull commented Nov 9, 2013

See #4849

eknkc commented Nov 9, 2013

While I can see the possible benefit of the functionality, this implementation does not make a lot of sense. Just rendering _ prefixed items unaccessible is really overkill for such functionality.

I think @caitp 's suggestion is handles this better than the current implementation.

I have another suggestion; we could have "safety check interceptor" somewhere. You could supply the function that checks if a name is private or not.

This would mean that I can have $ as a private member prefix and someone else can have private_ for that.

leisms commented Nov 9, 2013

I use CouchDB and this was a huge breaking change for my apps due to the inability to reference '_id's or '_rev's in expressions.

I see the points being made for the edge cases where private fields would be nice such as in controller as functionality, but we should use something besides leading underscores, which in general use don't reflect private vars.

Contributor

boneskull commented Nov 9, 2013

@leisms On the contrary I see _ reflecting private variables as quite widespread.

Contributor

boneskull commented Nov 9, 2013

@leisms Sounds like you might have some example code that breaks as @btford is asking to see...?

leisms commented Nov 9, 2013

@boneskull how so? In MongoDB and CouchDB they represent reserved words rather than private vars.

@btford this is a simple use case that's breaking
<span>{{foo._id}}</span>

Contributor

boneskull commented Nov 9, 2013

@leisms Yes, but in JavaScript since there are really no "private" variables, people tend to prefix members with _ as a way to say, "whoever is reading this code: don't use this".

leisms commented Nov 9, 2013

@boneskull well, again I see that as a reserved word pattern. If you override an object with underscored vars, who's to say which variable is private and shouldn't be overridden, vs. variables that just shouldn't be used by people in general? The point is that conventions from JS will bleed into Angular with expressions being the way they are now.

After thinking a bit more about it I think the best way to fix this issue and make everyone happy is to go the registry path mentioned by @caitp and apply private markers only after fields have been added to the $scope. It seems like the only way to ensure that the local object doesn't get polluted while maintaining existing conventions (and prevent this new feature from breaking code).

Contributor

btford commented Nov 9, 2013

@leisms Thanks for the response. I have some follow-ups to try to understand the issue better, if you don't mind.

To be clear, this is to say that in an actual app of yours, you dump out the _id and display it to the user? Or is it for a link or something? If not, why does the user need to see the id?

Have you tried/considered using either HTTP response interceptors to alias _id to something else, or a getter to retrieve it? If so, have you found these solutions to be somehow inadequate or painful?

I want to understand because I worry this thread has turned into bikeshedding.

leisms commented Nov 9, 2013

@btford no prob. I have a few administration apps that do require rendering _id and _rev fields as well as using them for linking. Going with the interceptor route is a pain, since I'd need to refactor back into underscored keys when saving data.

Btw, did it make sense how I described the convention of underscored elements as reserved words?

Contributor

btford commented Nov 9, 2013

@leisms Thanks. Do you ever need to two-way bind to these properties? Would being able to just read these properties but not write to them be acceptable, for instance?

I understand your point, but that's an entirely different (and more bikeshed-y) issue.

leisms commented Nov 9, 2013

@btford fair enough. For those specific use cases, two-way binding isn't needed on those specific variables. I am exposing the root object (foo) to $scope though.

anshul commented Nov 9, 2013

One solution might be to replace a single underscore with double underscore. That is to mark properties beginning or ending with two underscores as private. This feels more idiomatic and would perhaps break much less pre existing code?

leisms commented Nov 9, 2013

@anshul I was thinking about that too, or something else that's obscure. Double underscores probably are a bad choice as they could lead to name-collision with reserved browser variables (see link), but maybe something new might not. It might also pollute the local object.

Contributor

chirayuk commented Nov 9, 2013

On $resource + MongoDB breakage

This is currently broken as it's using $parse which is
overkill.  For folks broken by this, I'm preparing a fix for
$resource (ref PR #4850)

It's my understanding that this is the real issue that's causing the pain. 

On _name in {{ … }} expressions

You would change _name to name or use a getter
function.  Could you give me a real world example where
making that change to your code is non-trivial?  While you
must update your code for this (like for all other breaking
changes), I'd like to know of cases where it's really a big
deal.  (A made up / fake example would be: "My database
returns deeply nested JSON and all the names begin with
underscores.  I don't want to / can't change the JSON
response and it's not cool to have an $http/$resource interceptor
rename them.")  This far, I've only see simple cases that
can be are easily changed to work.

Angular expressions are not JavaScript

Angular expressions and JavaScript expressions are quite
different.  a.b.c = 1 in Angular will create {a: {b: {c: 1}}} when a isn't defined on the scope but in JavaScript,
you'll get an ReferenceError that a isn't
defined.  foo | bar in Angular will either filter foo
with the bar filter or give you a Unknown provider: barFilterProvider <- barFilter error while in JavaScript,
it's coerced into an integer and bitwise OR'd with
bar.  There's plenty more (JavaScript has the new
operator, Angular can optionally (deprecated) unwrap promises
, etc) so if you want a JavaScript expression – just use a
JavaScript function and call it from the Angular expression.

A huge Thank You to @btford for all his responses here
while I was away.  I'm off again now but I'll be back
tomorrow.

leisms commented Nov 9, 2013

@chirayuk thanks for bringing us up to speed. Regarding underscore usage, you and @btford are correct that users can simply work around the issue by writing more code.

That said, it seems absolutely everyone else in this thread is opposed to this change. Why are you guys maintaining such an opinionated and contrarian view?

Contributor

caitp commented Nov 9, 2013

@leisms it's very hard to answer that question without making this look like a nasty argument :p

From my perspective, I am not going to have a hard time with the change, but I think that there are a lot of developers who probably will be confused by this, not understand the reason behind it (I admit, I think the reasoning is a bit silly, the $parser is probably not the place to be policing the developer). I think for the benefit of people who will be confused or discouraged by the change, or for people who do not care to use the imposed convention, we can probably find some middle ground.

However, as I said, I'm not terribly concerned (which is why I didn't bother to enter the discussion until my inbox started catching fire from the notifications), I just think that on average, it's not going to be taken as an "oh thank you for this glorious enhancement" kind of change.

From my perspective, it broke my app, and that's my primary concern.

Having said that, I do this kind of convention enforcement all the time: I just do it via linting, not within the framework itself. If I wanted to use an opinionated framework, I'd be developing in ember.

@leisms I think only the people who are opposed to this change are posting in this thread. There's little incentive for supporters to post here, and none for apathetics (which I imagine is the silent majority of Angular users). From my POV, this change doesn't affect my application and seems like a logical next step in the direction Angular's supported best practices seem to be heading — exposing a controller object's (and the properties on its prototype) to the template as its scope via the "Ctrl as" syntax.

Contributor

btford commented Nov 9, 2013

That said, it seems absolutely everyone else in this thread is opposed to this change.

I wouldn't confuse the handful of those who are loudly opposing this change with unanimity. I think the main reason for the vocal opposition is that the benefits are clearly more subtle than the drawbacks. In all cases, we try to make the right choice. This may not necessarily the popular one.

It's difficult to see the benefit of conventions upfront. For some users, this change means they have to change their code to update to 1.2.0, which can understandably be frustrating.

To summarize:

Pros: Introduces conventions that make it easier to reason about your app's security.
Cons: Conflicts with binding to some databases' keys, potentially clashes with some developer's naming schemes.

I the value of making security easier hugely outweighs the slight inconvenience that this introduces. Maybe security isn't that much of an issue in the applications you build. That's fine, but Angular should also support large-scale, security-conscious apps as well. That being said, we're always open to suggestions, provided they take into account the entire spectrum of reasonable use cases.

@caitp's suggestion is alright in the sense that it makes it technically possible to do the same thing, but not as good in the sense that it doesn't make reasoning about the security any easier (since it doesn't maintain the locality).

@chirayuk and I talked about having numeric and string underscore-prefixed properties available as read-only, and non-invokable. We're hesitant to do this because it's potentially confusing that _id works but _id = foo and _id() don't. This solution makes @leisms use case a bit easier.

@krotscheck your case is a legitimate bug in ngResource that will be fixed (see #4850). In theory, you shouldn't run into this issue unless you are doing {{_id}}, which outside of @leisms's case where you have some sort of app that administers a database, shouldn't be too common.

If I wanted to use an opinionated framework...

This is a false dichotomy. Opinionated-ness isn't black and white. Angular has some areas where it is opinionated, and some where it is more lax. Security is an area where conventions are more valuable than flexibility.

leisms commented Nov 9, 2013

Thanks for the candor. Like @krotscheck, I'm annoyed because the breaking change is coincidental and due to an arbitrary character selection. I'd like to be able to talk about alternatives (choose another letter, use another method) and have the maintainers join in rather than being shut down by them as having a 'made up/fake problem'.

leisms commented Nov 9, 2013

@btford thanks for the outline, it helped me understand the issue better. I don't think we're flighting to remove private fields –security conventions are important. Rather, it's the unintended side-effects that happened to pop up due to using an underscore in the implementation.

So, the simplest fix may just be to chose another identifier besides underscore.

Contributor

btford commented Nov 9, 2013

@leisms I hope you don't think I've been patronizing. I was being totally candid in my questions, and I appreciate your thoughtful replies. I did not mean to imply that your problem was not real nor important, but it's also hard to talk about these things without seeing examples of how developers are using these types of names in their app.

We can discuss using another name as well, but doing that runs the risk of stomping all over someone else's favorite namespace. My intention was to defer the discussion of "what should the convention be" until after the dust settles around "what are the real world use cases that we need to support to make everyone happy."

leisms commented Nov 9, 2013

@btford you weren't being patronizing at all. I don't think we were all on the same page. For me the convention was the issue, not the feature, so hopefully we can move on to the best way to fix that soon.

👎 I'm very much against this change. _id is used in and through out our mongo based apps. We receive, display (in URLs) and post back records with _ids. Further more we have extensive test coverage that generates client side data fixtures along with stubs and mocks that do the same. We would have to change all of that and introduce pretty brittle middleware on the client and server side just to rename _id to id and back.

I think if this were to stay it should be configurable or at least have an option to turn this off. Picking underscores and locking that down seems very arbitrary to me. I think this should be solved through best practices recommendations and not like this.

Finally, to calm everyone down a bit... you can always keep a fork that pulls this out. We'll probably have to do this.

Contributor

cburgdorf commented Nov 9, 2013

I left a comment in this other PR but I'm also cross posting it here. I think we should do something like this:

myModule.config(function($parseProvider) {
  //set it to an empty string to completely ignore it.
  $parseProvider.privateFieldPrefix('_');
});

This means for security audits one can be sure that you only need to look up the setting upfront and it can only be made in the config phase. And it also means that you can completely remove the feature or just change the prefix to anything other than _ e.g. $$.

Contributor

revolunet commented Nov 9, 2013

@cburgdorf : Nice proposal that should answer most case raised here. Undrscore should ne the configurable default.

Contributor

btford commented Nov 9, 2013

$ and/or $$ is not a good choice for the prefix. $$ means something special to Angular's digest cycle and is considered "angular-reserved." $ is reserved for Angular APIs.

_ is far from arbitrary; is already a convention in many style guides and even other languages. It's also the convention used by Closure Tools.

Having the prefix be configurable/optional largely mitigates its usefulness as a best practice. A best practice is useful because it allows us to share components. If you install a component that uses one prefix, and someone else installs a component that uses another, you break the security guarantees of one (depending on the order of the configuration functions). This is why I don't think that prefix configurability is a good fix.

Changing the prefix runs the risk of introducing yet another breaking change that messes with someone else's preferred prefix. Unless someone is able to construct a corpus of all JavaScript prefix conventions, this is going to be tough.

This case is easily mitigated by changing the templates that reply on {{_id}} to {{id}} and using $http response interceptors:

$provide.factory('myHttpInterceptor', function() {
  return {
    'request': function(config) {
      config.data._id = config.data.id;
      return config;
    },
    'response': function(response) {
      config.data.id = config.data._id;
      return config;
    }
});
$httpProvider.responseInterceptors.push('myHttpInterceptor');

Or using a getter with {{doc._id}} to {{getId(doc)}}:

$scope.getId = function (doc) {
  return $scope.doc._id;
};

Unless someone has an example otherwise, I think these changes are relatively painless. This means you wouldn't have to make any changes to your server-side code, but just to your AngularJS templates. A simple "find and replace" would even work in many cases.

I think this is also a good choice for helping new users become more security conscious. It causes them to think upfront about whether they really want to be exposing these indices to end users. If they do, they must be explicit about it.

For those who are really, really opposed to this change, you can always provide your own $parse to Angular to revert to the old behavior.

eknkc commented Nov 9, 2013

Why is this being dragged like this? Using _ is at best a convention. Please do not mandate your conventions on me.

@btford you are getting hostile, saying things like "someone else's favorite prefix". It's not my favorite. I happen to use it to build things right now and I happen to depend on it functioning, in my job, that I make money for living. Do not underestimate how such an arbitrary change causes problems for other people.

This will also effect new people, getting into angular. If they are already using Mongo or Couch for persistence or working on already established APIs. What will the docs say? "Traverse your entire request/response graphs using http interceptors to rename entries starting with _. And carry a mental burden of having thnigs named diffferently on your server and client side. Cause you know, you are an idiot having them on public."

Angular is such a configuration heaven, this one should be just as easily configurable and if I want to shoot my feet, it's my feet.

I'm working on an administration app built with angular and i am using a MongoDB driven backend. I read all the discussion going on here after this "feature" made it into the release.

Updating to version 1.2 causes a lot of trouble since now it is no longer possible to:

  • use _id in ng-src and ng-href
  • accessing my own object cache using things like ng-href={{cache[pic._id].fullSizeUrl}}
  • let the admin choose a STI-attribute that is called _type since it is not readable/writable

See this plunker as an example.

@btford The "solution" to implement a http interceptor is not really a solution because it would add an indifference between the server's data and the angular model which is not present in the backend or database. Also it would add to the potential points of failure - i'm thinking about arrays of objects, embedded objects and so on. It is not trivial to certainly fix all occurrences.

I think nailing it down to (a part of) the name of a property or function is not an optimal way to enforce pseudo-private behavior. In my app's case _ was a bad choice - same as $$ or __ or _$ or ... would have been for someone else.

I don't see why my use case of angular is less valid. The

entire spectrum of reasonable use cases

should include this one as well and currently, it is broken. My feeling is that the only need to introduce this behavior is the controller as syntax, as @petebacondarwin pointed out even before this got merged. IMO "securing" the exposed controller should be achieved by another way than blacklisting by a chosen "convention". A better approach would be to make this configurable by providing some sort of callback like suggested before, allow me to disable this behavior (#4849) or at least make the prefix customizable.

mlandalv commented Nov 9, 2013

1,5 years or so ago, we migrated lots of KnockoutJS code to Angular. Along with that we kept the _destroy convention that is used for observable arrays in knockout. (Basically _destroy = true is used to mark a row for deletion on the server.)

We have places where we check this _destroy value to either tell the user that something is going to be removed, to give them the ability to undo it, to mark a table row as red if the item will be destroyed etc. Nothing of this works in 1.2.0.

This plunker illustrates the concept (can't use _destroy in ng-class).

First I thought this "feature" would only prevent access to "private" variables directly on $scope (that is where the prototype members end up, right?), but not on nested properties, but that's not the case.

I agree with @spiderpug that implementing a http interceptor isn't really much of a solution.

I vote for making this configurable, and disabled by default.

Contributor

wesleycho commented Nov 9, 2013

As far as I can tell, this addition looks like it can be hugely inconveniencing.

For example, in one app I do app.run(function ($rootScope) { $rootScope._ = _; }); so that I have access to Lodash in my templates. I can alias it to something else with no problem, so for me it is only a minor inconvenience.

The other example people have pointed out is with MongoDB - IMO, I think you should configure the backend to transform the data before sending it to the client-side app, but I can see this eating massive dev time depending on the code base, especially since using Angular with Node.js & MongoDB seems to have a cult following.

And there is the issue that has been mentioned by conventions some devs have already adopted regarding _ prefixing of keys.

Lastly, the issue of urls with _ being interpolated incorrectly is also an issue.

I propose that this should have been a toggle in the first place, with the announcement that using _ as a prefix to a key on $scope with normal behavior will be deprecated in favor of the new one to give enough advance notice to transition.

I'm personally indifferent towards this change, as I've avoided prefixing with _ in general, but it's not that difficult to see how this is a huge breaking change. I think the issue of url interpolation is the most serious one, as it's the main truly unfixable problem. The solution for that would be to add the ability to whitelist a string when parsing & the string potentially contains an underscore.

kofalt commented Nov 9, 2013

It would be nice to get a maintainer's opinion on #4849, it's a straightforward compromise.
Merging that PR (or something like it) would IMO be easier than arguing about underscores all day :)

leisms commented Nov 9, 2013

Agreed w/ @kofalt, that PR is an easy compromise until we figure out a better solution. Follows existing toggle conventions.

lucsky commented Nov 9, 2013

I'm with @eknkc, please do not enforce these conventions on me and break all my code in the process.

As far as I am concerned, I am using a few REST APIs which allow to embed related data objects to others when making queries, these related data being added as childs of an _embedded property. Note the _ in _embedded ? None of those objects can be bound anymore. And those aren't databases _id metadata, those are real data fields which I can't access anymore because of this.

I'm sticking to 1.2.0-rc3 and surely hope this will at least be configurable.

Member

petebacondarwin commented Nov 9, 2013

The core team are going to look into the concerns around this issue. Unfortunately a large number of them are away at conferences for the next few days.

In the meantime, if you have concrete examples that are affected by this change, feel free to post them so that the impact on them can be considered.

@zdwolfe zdwolfe added a commit to librecms/librecms that referenced this pull request Nov 10, 2013

@zdwolfe zdwolfe Fix for Angular release of version 1.2.0 breaking change for _id data…
… members. See angular/angular.js#4509
3cc589c

I agree with all angry guys here:
forcing security with a code convention, enabling a breaking change on existing apps, it's not a big idea for me.
If revert is a no no way, than make it configurable.

from parse.js current/before commit (line 29/26):

// This sandboxing technique is not perfect and doesn't aim to be. The goal is to prevent exploits
// against the expression language, but not to prevent exploits that were enabled by exposing
// sensitive JavaScript or browser apis on Scope. Exposing such objects on a Scope is never a good
// practice and therefore we are not even trying to protect against interaction with an object
// explicitly exposed in this way.

beyond this; I'm also dependent on performance of large mongo and couch datasets. Would love to see configurable solution in terms of being able to choose prefix/postfix to mark privates.

oh and brian asked about two-way bindings and regarding the code i've written, i cannot think of a sensible use-case where underscore-prefixed values had to be editable directly.

should underscore remain angularjs' private, keeping read-only would totally suffice imho

@ghost

ghost commented Nov 10, 2013

We also use Mongo extensively, and have our ids that come across in arrays and embedded objects, that we must display to our users as they typically email them and convey them back and forth even over the phone. They are used in our URL's as well. The response / request interceptor only works easily on the top level, if you have an _id embedded four levels deep it will become extremely cumbersome.

I also don't like that I would be forced to set a configuration on such a common setup on every new project. I like the fact it's more secure, I really do, but aliasing _id to id for so many of our collections, that also happen to be embedded and replicated to an ElasticSearch set of indexes, and replicated again to a data warehouse is really cumbersome. Agree with @btford that I have no need for two-way binding to them.

If it really must stay though, please provide a configuration mechanism to turn it off.

Contributor

lrlopez commented Nov 10, 2013

As Brian suggested in #4509 (comment), I've proposed PR #4859 as a posible solution. You can disable the new behavior or change the matching pattern to any other that suits your project so you can keep the security features.

Would this fix your problems?

@ghost

ghost commented Nov 10, 2013

@lrlopez Yes, it would, the only minor inconvenience it would cause is that we're currently using the ngBootstrap style project layout, which means we create a ton of modules; 1 for each major section of functionality. This would mean we would need to configure it on each one, but to be honest, I'm willing to live with that.

goya commented Nov 11, 2013

let me get this straight: a breaking change affecting every mongodb and couch angular application using direct model bindings was added between 1.2.0-rc3 and 1.2.0 and nobody raised a red flag on this? this should NOT have been pulled in. regardless of the usefulness of this "feature" a breaking change should not be added during the release candidate phase.

Contributor

wesleycho commented Nov 11, 2013

I'm personally not a huge fan of this breaking change, or at least the pick of _ instead of maybe __.

However, I think the Angular team does have the right to push breaking changes in RC. Development progresses with breaking changes often.

ditto @goya and @dariusriggins . Not something that should appear between the last RC and the release.

As to where to go from here, why not allow the developer to either turn off the feature or specify a custom function or regex to test against?

EDIT: just saw @lrlopez is ahead of me! :)

lucsky commented Nov 11, 2013

@btford wrote:

Our policy is that RCs can have breaking changes.

And that's perfectly fine.The problem is that 1.2.0 is not an RC, it's a final release.

Yes we could keep doing RCs until everything "stabilizes," but we found that doing that in the past meant slower releases.

I'm not sure if creating a shitstorm of angry users is better than adding a couple weeks to a release which was already (approximately) a year late.

Remember that no one is forcing you to update to 1.2.0. You're free to keep using 1.2.0-rc.3 and update at your own pace.

Absolutely right. But this doesn't excuse the incredibly sloppy way this "feature" has been introduced. And I assure you that I am really trying hard to keep being civil here...

If you want to know about breaking changes earlier, you can subscribe to AngularJS's Github notifications. All major changes like this are discussed publicly via PR, just like this one.

I find this suggestion very funny, coming from the guy who brags about having "the world's largest collection of unread Github alerts" in his Twitter profile... :>

mrdungx commented Nov 11, 2013

Agree with @goya. Retarded changes like this are what really put people off using AngularJS as a long term solution.

eknkc commented Nov 11, 2013

Is it possible that this was done just to have feature parity with https://github.com/angular/angular.dart given that Dart uses _ prefix to denote private fields at language level? It was released recently too (http://news.dartlang.org/2013/11/angular-announces-angulardart.html)

I'm not looking for an evil plan or something (someone implied that on HN), just genuinely curious if the dart port will have any influence on the main project and cause such changes. What do core devs think about it?

kloy commented on 3d6a89e Nov 11, 2013

While I understand the intent of this change, I completely disagree with the release strategy. AngularJS had over a year of "unstable" point releases and 3 release candidates to try out breaking changes. Why would you release a breaking change of this type in the first major release of 1.2.x?

This is functionality we use at our company across multiple apps and we keep an eye on all unstable releases to ensure forward compatibility. We do not have time to monitor every commit and rely on the AngularJS team to follow the best practices that have been set forth and utilize the processes the team already has in place such as unstable branches and release candidates.

What's the recommended practice for conforming to this change? We typically define RESTful javascript models and use private naming conventions for filtering out properties that are only used to store and display ui-related state but shouldn't be sent to the server when saving the model. In other words, we use _ primarily to store ui-related information that would be used in an Angular expression, so being unable to access these properties in Angular is a very big inconvenience and we'll need to rethink our practices to upgrade to 1.2. Surely the solution can't be to define accessor functions on scope for each private property that we need to access in an Angular expression?

kloy replied Nov 12, 2013

@jr314159 We took the approach of prefixing UI specific properties with ui_ when dealing with a similar situation where we would decorate collections of data from our REST services. We considered using $ since that is another popular convention in the JS world of denoting limited privilege but decided not to since $ and $$ are popularly used in the AngularJS library.

Wouldn't it be pretty easy to make this new functionality configurable through $parseProvider? That way the check could be enabled by default, but be disabled in a .config block to ease version upgrades. Or am I not seeing the whole picture here?

This is totally unexpected. Broke my MongoDB app as well.

This seems very unreasonable. MongoDB and CouchDB use _id for, well, id. Forcing everyone to write a wrapper to hide 'id' to pretend it to be "not private" is not a very efficient way to enforce this capricious restriction. If someone wants such a restriction, it should be a configuration option with prefix being specified (i.e. "privateFieldPrefix" so people can set it to '$' or '' or something else if they want to enforce the restriction).

I very much agree with @kloy . Why would you wait until AFTER all of the release candidates to make such a change? This is very disruptive and unexpected from such an otherwise well run project. Can you at least not make this behavior configurable like you did for the unwrapping of promises?

8-uh replied Nov 14, 2013

While this doesn't "fix" this change, this is the solution I've come up with to get around it. Yes, it's obvious, but it gets the work done.

http://stackoverflow.com/questions/19983635/angular-1-2-0-error-referencing-private-fields-in-angular-expressions-is-disa

Someone explain to me what this accomplishes that

// some js code that establishes a scope
var somePrivateVariableHiddenFromTheTemplate = "supersecret";

wouldn't accomplish, without breaking things?

Contributor

jbdeboer replied Nov 15, 2013

This commit has been reverted in AngularJS 1.2.1. http://blog.angularjs.org/2013/11/angularjs-121-underscore-empathy.html

It is good hear this change was reverted. While I understand the need to mark something as private in certain cases, using such a common naming pattern to indicate a private member is not a good idea. It broke my app. :(

Siyfion commented Nov 11, 2013

Just have to say that this change was really badly dealt with, to the point that my entire trust in the AngularJS core team and release process has been shaken. We've been through no less than three release candidates to get ready for the 1.2.x final and not one of them contained this major breaking change.

To thrust this change upon your users as you make the release final has caused a significant proportion of your user-base a great deal of issues. To the point that many will now prefer running 1.2.0-rc3/2 over the final (better the enemy you know, right?).

My specific issue? I allow users to effectively "hand-make" their schema for their products (which is stored in MongoDB) so firstly I loose all the ngResource code that uses '_id', next up, if any user were to define a product field with an underscore (say an internal flag, eg. '_batch_no') that too, will no longer be accessible.

This needs to be fixed and made optional ASAP. I hope the team will learn from this...

Sorry, I commented on the commit before I found this PR discussion. Per @petebacondarwin's request, here is a little clearer illustration of our use case that breaks with this change. Pretend we've defined a Book service that is our angular representation of a server-side restful resource. When we call book.save() on an instance of a Book, it serializes all non-private properties and sends these to the server, filtering out all properties beginning with underscore, which we've reserved for ui state-related properties that have nothing to do with our server-side model. Here is an example controller (pardon the coffeescript):

.controller 'BooksCtrl', ['$scope', 'Book', ($scope, Book) ->
  Book.query().then (books) ->
    $scope.books = books
  $scope.editBook = (book) ->
    book._editing = true
  $scope.saveBook = (book) ->
    book._saving = true
    book.save().then -> 
      book._saving = book._editing = false

and the markup:

<div ng-repeat="book in books" ng-switch="book._editing">
  <span ng-show="book._saving">[saving]</span>
  <div ng-switch-when="true">
    <input ng-model="book.title">
    <input ng-model="book.author">
    <button ng-click="saveBook(book)">Save</button><button ng-click="book._editing = false">Cancel</button>
  </div>
  <div ng-switch-when="false">
    {{ book.title }} by {{ book.author }} [<a ng-click="editBook(book)">edit</a>]
  </div>
</div>

This is a pretty typical pattern for us, and I'm not sure what's the best way to rewrite this if we can't access underscore-prefixed properties in angular anymore. fwiw I'd like to see #4849 merged. I thought this was a nice approach for dealing with assigning promises to scope and has made it easier for us to upgrade to rc3 and deal with deprecation notices at our own pace.

Member

petebacondarwin commented Nov 12, 2013

@jr314159 - thanks for this concrete example of your problems. Without entering into a discussion of whether blocking _ prefixes is the right thing to do:

  • one simple fix for you would be to use a different prefix, such as $. $resource does this already - it applies "helper" methods to the model object such as $save().
  • alternatively, and the approach that I prefer to take in this situation, is to keep the UI related properties in the controller. Rather than a single "BooksCtrl" for the entire array, you have a "BookCtrl" for each book, which is responsible for this UI logic and state. This has the additional benefit of separating the view logic from the model and means you don't have to remember to "clean" your model before sending it to the server. Something like this:
<div ng-controller="BooksCtrl">
<div ng-repeat="book in books" ng-controller="BookCtrl">
  <span ng-show="isSaving()">[saving]</span>
  <div ng-if="isEditing()">
    <input ng-model="book.title">
    <input ng-model="book.author">
    <button ng-click="saveBook(book)">Save</button>
    <button ng-click="stopEditing()">Cancel</button>
  </div>
  <div ng-if="!isEditing()">
    {{ book.title }} by {{ book.author }} [<a ng-click="editBook(book)">edit</a>]
  </div>
</div>
</div>
.controller 'BooksCtrl', ['$scope', 'Book', ($scope, Book) ->
  Book.query().then (books) ->
    $scope.books = books

.controller 'BookCtrl', ['$scope', ($scope)->
  @editing = false
  @saving = false
  $scope.editBook = (book) ->
    @editing = true
  $scope.saveBook = (book) ->
    @saving = true
    book.save().then -> 
      @saving = @editing = false
  $scope.isSaving = () ->
    @saving
  $scope.isEditing = () ->
    @editing

@petebacondarwin, thanks, that's a nice example. It gets a little tougher though when the line between ui related properties and model properties blurs. For example:

<div ng-controller="BooksCtrl">
  <div ng-repeat="book in books" ng-class="{changed: book._changed}">
    <input ng-model="book.title" ng-change="book._changed = true">
    <input ng-model="book.author" ng-change="book._changed = true">
  </div>
  <button ng-click="save()">Save Changes</button>
</div>
.controller 'BooksCtrl', ['$scope', 'Book', ($scope, Book) ->
  Book.query().then (books) ->
    $scope.books = books
  $scope.save = ->
    booksToSave = _.filter $scope.books, (book) -> book._changed
    Book.save(booksToSave)

Here, our parent controller is responsible for bulk saving all books that have been changed, and we use the _changed flag for UI display as well. If we were storing this flag on the child controllers, it's not clear how we'd expose this property to the parent controller without more complicated wiring.

So sure we could use $ instead but _ is a more standard convention, and we've become accustomed to avoiding the dollar as something reserved for Angular services.

Siyfion commented Nov 12, 2013

@petebacondarwin I think the point that @jr314159 highlights perfectly is that while it this is a methodology that would be nice to implement if you were starting a project from scratch, it causes huge issues for existing codebases, potentially requiring them to be significantly rewritten. And when it comes to MongoDB/CouchDB it just causes headaches.

Contributor

kzar commented Nov 12, 2013

So to chime in again, we tried upgrading to 1.2.0 at work on our
application and ended up rolling back to rc3 until this is sorted.

On Tue, Nov 12, 2013 at 3:58 PM, Simon Mansfield
notifications@github.comwrote:

@petebacondarwin https://github.com/petebacondarwin I think the point
that @jr314159 https://github.com/jr314159 highlights perfectly is that
while it this is a methodology that would be nice to implement if you were
starting a project from scratch, it causes huge issues for existing
codebases, potentially requiring them to be significantly rewritten. And
when it comes to MongoDB/CouchDB it just causes headaches.


Reply to this email directly or view it on GitHubhttps://github.com/angular/angular.js/pull/4509#issuecomment-28304906
.

It's a shame this was implemented so fast. It seems to me that those suffering the consequences are those not having the problem.

The suggestion that we should prefix the variables with something else is quite sad. People could stop putting private stuff on $scope too...

Revert. Please.

Can't agree more with you, @mlandalv. If it's private, why it's putting it into the $scope?

Contributor

iammerrick commented Nov 12, 2013

Please revert this...

Contributor

iammerrick commented Nov 12, 2013

In some cases, this could break based on data... E.G.

var columns = ['Name', 'Age', '_last_updated'];
<h1>{{columns[2]}}</h1>

Now say, that columns is a result of the users data, pulling in their data from google docs for example. It has nothing to do with our code and remapping for this convention is just silly.

Valorum commented Nov 12, 2013

I understand the desire to introduce better security, which is a worthy goal. But why should Angular have an opinion on my back-end's naming convention for properties on objects? What good can possibly come from that?

The "solution" of us writing more code (such as the mapping from _id to id in ngResource based apps for example) leaves me a bit bewildered. That extra code we need to write does nothing to add functionality; Its only purpose is to work around a (IMO artificial) limitation put into place by the framework. The net result is that we need to spend time to write more code which has no direct benefit in any way. In fact, it just increases the code's footprint, leading to more code that can break. Perhaps Google can afford to do such exercises, but for many small teams, such time is wasted time that could be used to focus on product features instead.

It also seems to go against what frameworks and such are trying to do nowadays. One of the reasons to use a MEAN-like stack for example (which is something the Angular team is not reluctant to promote it seems) is the convenience of not having to map your data structures at each level; It's JSON all the way down, and you can use the same objects and properties throughout. This change goes against that philosophy and means we need to add mapping of properties, just as it seems there's a movement to move away from such (IMO silly) requirements.

Finally it strikes me as odd in general to introduce things that require more work. I feel that the general direction things are moving into is one in which libraries and frameworks and tools are trying to make it so that you can do less work to build applications. In that light, I would expect any enhancements such as this one to be introduced along with changes elsewhere to buffer the impact of the change. In this case, perhaps an associated change in ngResource to handle the transparent mapping of _id to id could have been introduced, thereby reducing the pain of this change.

We're a small team, and I've been moving us towards using AngularJS to build our web apps. It's hard enough to get everybody up to speed and proficient as it is (it requires a shift in mindset), but to then also have to deal with these seemingly arbitrary quirks makes that job that much harder.

It seems that the scope of the impact of this change is larger than intended. I think it would be beneficial if we discuss a way to make this change transparent for the cases that are not intended to be impacted by this change. I understand the need to protect certain things, but it seems a bit overzealous to apply this change to everything.

MEAN stack and schema/model sharing between Angular and Mongoose is a great reason to merge PR #4859

ryn101 commented Nov 12, 2013

One simple feature which breaks with this new change is the ng-repeat 'track by' functionality when using MongoDb.

<li ng-repeat="obj in objects track by obj._id"></li>

This is the most common use case for me and unfortunately will not work with the final version without resorting to unnecessary work around's/applying new naming conventions.

Just need to comment out lines 9097 thru 9101 (5 lines) of angular.js 1.2.0.

  if (name.charAt(0) === '_' || name.charAt(name.length-1) === '_') {
    throw $parseMinErr('isecprv',
        'Referencing private fields in Angular expressions is disallowed! Expression: {0}',
        fullExpression);
  }

Or this snippet on the minified version:

if("_"===b.charAt(0)||"_"===b.charAt(b.length-1))throw pa("isecprv",a)

Someone gave the tip here: http://docs.angularjs.org/error/$parse:isecprv
Regards.

goya commented Nov 13, 2013

i shall be forking and making a bower package that does this for you.

Contributor

iammerrick commented Nov 13, 2013

It just needs to be taken out. I don't understand the rationale for this.

Contributor

iammerrick commented Nov 13, 2013

Can we start a petition on whitehouse.gov?

Contributor

ocombe commented Nov 13, 2013

Why the hell would you put something like that from a RC to a release ?!
You just broke everyones apps who use mongoDB, couchDB, elasticSearch, etc...
As you stated it's a convention, but we don't have to follow yours if we don't want.
Make this optionnal or remove it asap.
Thanks.

Member

petebacondarwin commented Nov 13, 2013

See #4926

Contributor

kzar commented Nov 13, 2013

@petebacondarwin So what version will this be disabled by default from?

@kzar Just check the milestone of the PR: 1.2.1

Contributor

vojtajina commented Nov 14, 2013

Guys, sorry for the troubles this change caused. This "feature" was mostly for people using Closure compiler and Google JS style, but I didn't realize that many people were relying on accessing _* properties in templates. For now we are reverting that change (4ab16aa).

We gonna release 1.2.1 (which will contain this "fix") within next days.

The main outcome of this issue is: we should not put any breaking changes into RC.
Lesson learned.

Thanks @vojtajina! 👍

I think the problem is not so much with templates where the change is not unreasonable (I'm not sure I still think it's worth it but that's kind of beside the point). The big deal is that use of _id in $resource declarations is extremely common for anyone interacting with APIs using Mongo or Couch, especially "direct" APIs like MongoHQ.

I have some use of _id in templates but nothing that couldn't be refactored into a function in a controller (and probably be better off for it). Changing every $resource declaration is more problematic and confusing to anyone coming along later to understand.

The main outcome of this issue is: we should not put any breaking changes into RC.
Lesson learned.

I don't think a breaking change in an RC is necessarily a problem. A breaking change in the final release that wasn't in any RC is.

Contributor

cburgdorf commented Nov 14, 2013

@robfletcher actually it's best to completely avoid putting breaking features even between RCs. A release candidate actually means "Hey, folks, we think this should be the final version if we don't find any new unknown bugs. Please drop it into your apps and if we don't find any new bugs, it's going to be the final version"

Contributor

guilbep commented Nov 14, 2013

@robfletcher got a point 👍 / I'm moving back to rc-3 so I can play with $interval and use _id along // long live mongodb.

@cburgdorf I agree but I think the screaming would have been a notch lower in volume if this had been done in an RC4 rather than final.

Contributor

cburgdorf commented Nov 14, 2013

@robfletcher absolutely :)

Contributor

iammerrick commented Nov 14, 2013

Thank you! This made dynamic binding impossible (if a user entered something with an _ which was then used to look up something in a map later.)

Contributor

boneskull commented Nov 14, 2013

@vojtajina This sounds like the right thing to do considering the "shitstorm" of angry users. Thank you for making this decision!

Thank you! I will wait for 1.2.1. There was just too much overhead to adjust for such a change. I had to change the entire Angular codebase, and use a function to return the private object. I couldn't do that in Angular expressions, so the next thing I would have had to change was all the APIs and traverse through every object just to remove the underscore in "_id".

goya commented Nov 14, 2013

excellent news! awesome.

Thank you very much!

Valorum commented Nov 15, 2013

Thanks for the fix. I see the 1.2.1 release, but bower does not yet have it? Am I too impatient?

@Valorum it should work if the tag is there

Valorum commented Nov 15, 2013

Let me start by apologizing for not understanding the inner workings completely. When I ask bower about angular, this is what I get (after clearing Bower's cache):

$ bower info angular
bower not-cached    git://github.com/angular/bower-angular.git#*
bower resolve       git://github.com/angular/bower-angular.git#*
bower download      https://github.com/angular/bower-angular/archive/v1.2.0.tar.gz
bower extract       angular#* archive.tar.gz
bower resolved      git://github.com/angular/bower-angular.git#1.2.0

{
  name: 'angular',
  version: '1.2.0',
  main: './angular.js',
  dependencies: {},
  homepage: 'https://github.com/angular/bower-angular'
}

Available versions:
  - 1.2.0
  - 1.2.0-rc.3
  - 1.2.0-rc.2
  - 1.2.0-rc.1
  - 1.0.8
  - 1.0.7
  - 1.0.6
  - 1.0.5
  - 1.0.4
  - 1.0.3

You can request info for a specific version with 'bower info angular#<version>'

https://github.com/angular/bower-angular/releases does not list 1.2.1. I do not know how this normally finds its way into Bower. I assume somebody needs to do something for this to happen?

1.2.1 hasn't been deployed yet, AFAIK.

dagumak commented Nov 15, 2013

I am not very clear if 1.2.1 is out yet. I see the version reflected in the documentation, but I don't see it available for download. The inconsistency is very confusing.

Member

petebacondarwin commented Nov 15, 2013

It's getting there. Patience is a virtue :-) Wait for the blog posting.

On 15 November 2013 19:23, Douglas Mak notifications@github.com wrote:

I am not very clear if 1.2.1 is out yet. I see the version reflected in
the documentation, but I don't see it available for download. The
inconsistency is very confusing.


Reply to this email directly or view it on GitHubhttps://github.com/angular/angular.js/pull/4509#issuecomment-28596155
.

lucsky commented Nov 15, 2013

@vojtajina Thank you Vojta, much appreciated.

Contributor

mgcrea commented Nov 17, 2013

Thanks for listening to the community. I'm using MongoDB for several apps and this was a pain.

Theses multiple RC's had quite shaken my confidence as I've wasted a ton of time fighting broken changes between theses releases (ngAnimate, $parse). In my opinion, that should never ever happen during RC stage, especially if you use an unstable branch: RC means ready for production to me, like it's time to try to port my apps to it. If it's not ready, it should be called an alpha/beta.

I'm looking forward to finally work with this new branch ;-). Kudos for the hard work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment