250 precursor #293

e2tha-e · 2016-03-16T10:38:59Z

Addresses #250

Summary of changes:
Some edits to paramToJson in parameter_hunter.js required when parsing an enterprise Pattern Lab instance and a unit test for it.

e2tha-e · 2016-03-16T10:39:55Z

@bmuenzenmeyer Some more precursor work for issue #250 . Please review and merge.

e2tha-e · 2016-03-16T14:06:19Z

@bmuenzenmeyer
cc @james-nash
#292 seems like it's doing parallel, and possibly superior work in paramToJson. Please review, but hold off on merging, while we come to a consensus on what should be merged.

james-nash · 2016-03-16T15:42:38Z

Makes sense. If there's anything I can do to help (e.g. if any of my code is unclear) just shout :-)

bmuenzenmeyer · 2016-03-16T16:49:10Z

I haven't made time to review either yet - but agree this area of code is undergoing a lot of change and likely one of these two PRs needs to win out or consume the other. I'll appreciate any degree of review you both can do in that regard in the meantime.

james-nash · 2016-03-17T14:17:57Z

Sure. I have a bit of time this afternoon, so I'll take a closer look at the changes in this pull request.

e2tha-e · 2016-03-17T14:37:00Z

@james-nash Could you please hold off review of commit 695471c ? I'll be submitting something more up-to-date hopefully later today. Thanks.

james-nash · 2016-03-17T15:11:09Z

Sure thing. Just shout whenever it's ready.

…ternhunter)

e2tha-e · 2016-03-18T12:12:03Z

@james-nash @bmuenzenmeyer Ready for review. Notes on the param parsing are in the docblock to paramToJson.

FYI, I added the JSON5 package and replaced the JSON object with it, for better error messaging on malformed JSON. Running npm install and grunt nodeunit will give a great example of its advantages.

james-nash · 2016-03-18T16:48:39Z

core/lib/list_item_hunter.js

@@ -13,6 +13,7 @@
 var list_item_hunter = function () {

  var extend = require('util')._extend,
+    JSON = require('json5'),


I wonder if it might be more appropriate to name this JSON5, so that people don't mistake it for the built-in JSON object.

@james-nash Had to debate over this myself, and I'm still torn. @bmuenzenmeyer What are your thoughts?

I'm not too fussed myself. However, if you were to change it to JSON5 then I'd suggest doing that throughout all files where you use that library.

Forgive my naivety on JSON5 (it looks like a nice solution to a problem we've all faced), but it doesn't sit right with me. My biggest worry is downstream consumption of .json5 files (I'd also argue that things like trailing commas are not good form. Yes comments would be nice). If people get accustomed to writing these files, or perhaps worse, writing them as "invalid" pure .json files, they might shoot themselves or others in the foot. Is this fear overblown?

Another consideration in my rather scattershot comment is that it looks like they don't follow semver (yet), which could create problems.

... more thinking ...

Put simply, can you explain how adding JSON5 makes #250 (which PR references by name) and the parameter_hunter.paramToJson() method better?

I am an outsider on the specifics of this issue, and have no doubt that a lot of thought has been put into getting us this far, but I need your help in understanding if this is straying too far from the goal of this issue/fix. c8acb5a mentions better error handling, but in what way is it better than the existing try catch logic? http://codepen.io/bmuenzenmeyer/pen/66eb18de343ab1954abc53537fde09f0 has an example that is descriptive enough, coupled with a bit more logging on PL's part, would seem to do the trick in my mind.

Thoughts?

cc @geoffp

It's important to note that the pattern parameter syntax is only vaguely defined here http://patternlab.io/docs/pattern-parameters.html#pattern-parameter-syntax , but since the examples have keys that are not double-quoted, it's definitely not pure JSON. It's JSON-ish :-)

I'd be in favour of keeping our parsing of the pattern parameter syntax quite lax, since it's conceivable that users might copy-paste chunks of JSON data (so then keys might be double-quoted) and, as per issue #145 it's desirable to allow HTML code to passed in as a value so allowing single- as well as double-quoted string values makes sense to me (saves you having to escape every quote around HTML attributes and stuff like that).

The changes in this PR which make use of JSON5 enable all of that, so I think they're worth keeping.

Since, I've been focussed primarily on the parsing of pattern parameters, which is in parameter_hunter.paramToJson(), I'm not as familiar with the other code though. Perhaps @e2tha-e can comment on that.

FWIW, where proper JSON data is expected, I'd agree that it might be better to be stricter (or at least display warnings in the build output) so as not to encourage sloppy JSON code that may not be interoperable with other tools or PatternLab versions.

james-nash · 2016-03-18T17:13:52Z

Ok, I've finished reviewing. It looks great!
There's few minor things I commented inline (and I see that you've already responded to them), but otherwise no issues.

I see that you brought across my additional test cases, so this solution successfully fixes the same problems I was attempting to fix in my PR #292. As far as I'm concerned, #292 can be rejected and this PR accepted instead.

e2tha-e · 2016-03-18T17:18:10Z

@james-nash Thanks a million for the code review! I just pushed that typo-fix, and so am waiting on further review and merge. So hopefully issue #291 can be put to bed soon!

e2tha-e · 2016-03-18T22:34:06Z

@bmuenzenmeyer @james-nash I'll be offline on vacation for a few days, so don't mind the radio silence too much. Looking forward to be back at it when I'm back 😄

james-nash · 2016-03-21T11:10:49Z

@e2tha-e: No worries. Enjoy your vacation!

james-nash · 2016-03-21T11:55:27Z

core/lib/parameter_hunter.js

-      paramString = paramString.replace(/(^\s*[\{|\,]\s*)'([^']+)'(\s*\:)/, '$1"$2"$3');
+      //check if searching for a key
+      if (paramString[0] === '{' || paramString[0] === ',') {
+        paramString = paramString.substring([1], paramString.length).trim();


I found another instance of using an array instead of a number for substring()'s first argument. Must have missed that last time around - sorry!

e2tha-e · 2016-03-29T11:30:29Z

@bmuenzenmeyer I fixed that typo pointed out by @james-nash . I've also renamed the instances of JSON5 to be clear that it is not the native JSON object. I'm happy to replace JSON5 entirely, but I don't think it detracts from Pattern Lab in any way. I don't think it by default encourages lax JSON in the actual JSON files. Unless you peak under the hood, how would you know? JSON5 does, however, provide much better error handling than native JSON. I initially experimented with JSON5 to see if it would it could help with converting params to JSON. I found that it wasn't necessary, but boy were those error-handling message helpful. As such, I left it in. Any thoughts or suggestions are welcome.

e2tha-e · 2016-04-06T11:33:10Z

@bmuenzenmeyer Here's an example of the difference between native JSON and JSON5 exception handling. It's from running parameter_hunter_tests.js:

Native JSON:

[SyntaxError: Unexpected token ,]

JSON5:

{ [SyntaxError: Unexpected ',' at line 1 column 17 of the JSON5 data. Still to read: ",\"\":missing-key,\"\":,"]
  message: 'Unexpected \',\' at line 1 column 17 of the JSON5 data. Still to read: ",\\"\\":missing-key,\\"\\":,"',
  at: 16,
  lineNumber: 1,
  columnNumber: 17 }

In issue #250 , @vebersol and I mentioned we worked on enterprise Pattern Lab projects with hundreds of patterns. I know personally, that the "Unexpected token" syntax error was the bane of my existence. There's no way to lint parameters, and how was anyone expected to be able to quickly track down where that error came from?

Again, strictly syntaxed JSON is a requirement for .json files because they are read by fs.readJSONSync(). JSON5 has nothing to do with those files. I also wouldn't mind revisiting JSON5 being a drop-in replacement for JSON in the few instances where it's used, in the same manner that fs-extra is a drop-in replacement for fs.

…-precursor

bmuenzenmeyer · 2016-04-07T04:13:51Z

I'm coming around to this.

I made 00-homepage.json malformed and it silently failed, failing to apply any data to the page from the template. It's because this.

This test is probably outside the scope of the current issue/fix since it relies on fs - but that's the sort of error you are striving to make better. Let me give this some more thought

bmuenzenmeyer · 2016-04-08T03:36:05Z

I'm realizing more and more that users are taking PL Node into territory and scale I haven't yet experienced personally or professionally. And that there is merit to making the library more robust, even at the expense of some sense of "purity" I define in my head. Your contribution addresses a need, is thoughtfully proposed, thorough in its implementation, and covered with unit tests. What more could one ask for?

I'll need to be careful to forward port this into dev-2.0-core

Thanks!

e2tha-e added 3 commits March 15, 2016 20:24

more accurate paramToJson in parameter_hunter

0343df1

updating unit tests for parameter_hunter paramToJson

92e9a00

minor edit

695471c

e2tha-e changed the title ~~250 precursor~~ 250 precursor: DO NOT MERGE Mar 16, 2016

e2tha-e and others added 9 commits March 17, 2016 18:32

another param string parser

8b3551f

bugs fixed, unit tests working

77a2089

variable name change

e3bb31e

Added (failing for now) test to highlight the bug.

df0e804

Added another test for another bug (parenthesis in value confuses pat…

695a936

…ternhunter)

Added additional parameter hunter tests.

2a297f4

replacing JSON with JSON5 for better error messaging

c8acb5a

working paramToJson

9348441

stricter unit tests

cf080af

e2tha-e changed the title ~~250 precursor: DO NOT MERGE~~ 250 precursor Mar 18, 2016

e2tha-e added 8 commits March 18, 2016 08:16

comment updates

68a6c35

simplfying paramToJson further

37ad825

simplfying paramToJson further

a7e52cd

comment update

f3ee0f1

tightened parameter key search

db17e81

comment update

83f6a4f

accommodating decimals and exponentials as valid numeric values

ed13128

comment update

7bcea0a

james-nash reviewed Mar 18, 2016
View reviewed changes

fixing array instead of number typo

2317cae

james-nash mentioned this pull request Mar 18, 2016

Support HTML entities within pattern parameters #145

Closed

e2tha-e added 2 commits March 18, 2016 15:13

more comments

3583251

spacing to pass eslint

704346a

removing unnecessary escape

a0d33d3

bmuenzenmeyer mentioned this pull request Mar 19, 2016

#291: Fixes & improvements to parameter hunter #292

Closed

james-nash reviewed Mar 21, 2016
View reviewed changes

e2tha-e added 2 commits March 29, 2016 07:14

fixing substring param typo

4690f79

renaming json5 instances

eedaecc

comment update

7fa7f6b

bmuenzenmeyer self-assigned this Apr 6, 2016

Merge branch 'dev' of github.com:pattern-lab/patternlab-node into 250…

3f7d984

…-precursor

bmuenzenmeyer merged commit 739e507 into pattern-lab:dev Apr 8, 2016

bmuenzenmeyer mentioned this pull request Apr 8, 2016

Pattern parameters with non-string values cause the param values to disappear #291

Closed

e2tha-e deleted the 250-precursor branch April 9, 2016 22:15

james-nash mentioned this pull request Apr 11, 2016

145 test #307

Merged

bmuenzenmeyer mentioned this pull request Mar 7, 2017

JavaScript heap out of memory #637

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

250 precursor #293

250 precursor #293

e2tha-e commented Mar 16, 2016

e2tha-e commented Mar 16, 2016

e2tha-e commented Mar 16, 2016

james-nash commented Mar 16, 2016

bmuenzenmeyer commented Mar 16, 2016

james-nash commented Mar 17, 2016

e2tha-e commented Mar 17, 2016

james-nash commented Mar 17, 2016

e2tha-e commented Mar 18, 2016

james-nash Mar 18, 2016

e2tha-e Mar 18, 2016

james-nash Mar 18, 2016

bmuenzenmeyer Mar 23, 2016

james-nash Mar 23, 2016

james-nash commented Mar 18, 2016

e2tha-e commented Mar 18, 2016

e2tha-e commented Mar 18, 2016

james-nash commented Mar 21, 2016

james-nash Mar 21, 2016

e2tha-e commented Mar 29, 2016

e2tha-e commented Apr 6, 2016

bmuenzenmeyer commented Apr 7, 2016

bmuenzenmeyer commented Apr 8, 2016

250 precursor #293

250 precursor #293

Conversation

e2tha-e commented Mar 16, 2016

e2tha-e commented Mar 16, 2016

e2tha-e commented Mar 16, 2016

james-nash commented Mar 16, 2016

bmuenzenmeyer commented Mar 16, 2016

james-nash commented Mar 17, 2016

e2tha-e commented Mar 17, 2016

james-nash commented Mar 17, 2016

e2tha-e commented Mar 18, 2016

james-nash Mar 18, 2016

Choose a reason for hiding this comment

e2tha-e Mar 18, 2016

Choose a reason for hiding this comment

james-nash Mar 18, 2016

Choose a reason for hiding this comment

bmuenzenmeyer Mar 23, 2016

Choose a reason for hiding this comment

james-nash Mar 23, 2016

Choose a reason for hiding this comment

james-nash commented Mar 18, 2016

e2tha-e commented Mar 18, 2016

e2tha-e commented Mar 18, 2016

james-nash commented Mar 21, 2016

james-nash Mar 21, 2016

Choose a reason for hiding this comment

e2tha-e commented Mar 29, 2016

e2tha-e commented Apr 6, 2016

bmuenzenmeyer commented Apr 7, 2016

bmuenzenmeyer commented Apr 8, 2016