-
Notifications
You must be signed in to change notification settings - Fork 2k
/
lexer.html
650 lines (591 loc) · 140 KB
/
lexer.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
<!DOCTYPE html> <html> <head> <title>lexer.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <div id="jump_to"> Jump To … <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="browser.html"> browser.coffee </a> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="index.html"> index.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> </div> </div> </div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> lexer.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-1">¶</a> </div> <p>The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt
matches against the beginning of the source code. When a match is found,
a token is produced, we consume the match, and start again. Tokens are in the
form:</p>
<pre><code>[tag, value, locationData]
</code></pre>
<p>where locationData is {first<em>line, first</em>column, last<em>line, last</em>column}, which is a
format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>. These
are read by jison in the <code>parser.lexer</code> function defined in coffee-script.coffee.</p> </td> <td class="code"> <div class="highlight"><pre><span class="p">{</span><span class="nx">Rewriter</span><span class="p">,</span> <span class="nx">INVERSES</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'./rewriter'</span></pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-2">¶</a> </div> <p>Import the helpers we need.</p> </td> <td class="code"> <div class="highlight"><pre><span class="p">{</span><span class="nx">count</span><span class="p">,</span> <span class="nx">starts</span><span class="p">,</span> <span class="nx">compact</span><span class="p">,</span> <span class="nx">last</span><span class="p">,</span> <span class="nx">locationDataToString</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'./helpers'</span></pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-3">¶</a> </div> <h2>The Lexer Class</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-4">¶</a> </div> <p>The Lexer class reads a stream of CoffeeScript and divvies it up into tagged
tokens. Some potential ambiguity in the grammar has been avoided by
pushing some extra smarts into the Lexer.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">exports.Lexer = </span><span class="k">class</span> <span class="nx">Lexer</span></pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-5">¶</a> </div> <p><strong>tokenize</strong> is the Lexer's main method. Scan by attempting to match tokens
one at a time, using a regular expression anchored at the start of the
remaining code, or a custom recursive token-matching method
(for interpolations). When the next token has been recorded, we move forward
within the code past the token, and begin again.</p>
<p>Each tokenizing method is responsible for returning the number of characters
it has consumed.</p>
<p>Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a>
unless explicitly asked not to.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tokenize: </span><span class="nf">(code, opts = {}) -></span>
<span class="vi">@literate = </span><span class="nx">opts</span><span class="p">.</span><span class="nx">literate</span> <span class="c1"># Are we lexing literate CoffeeScript?</span>
<span class="nv">code = </span><span class="nx">@clean</span> <span class="nx">code</span> <span class="c1"># The stripped, cleaned original source code.</span>
<span class="vi">@indent = </span><span class="mi">0</span> <span class="c1"># The current indentation level.</span>
<span class="vi">@indebt = </span><span class="mi">0</span> <span class="c1"># The over-indentation at the current level.</span>
<span class="vi">@outdebt = </span><span class="mi">0</span> <span class="c1"># The under-outdentation at the current level.</span>
<span class="vi">@indents = </span><span class="p">[]</span> <span class="c1"># The stack of all current indentation levels.</span>
<span class="vi">@ends = </span><span class="p">[]</span> <span class="c1"># The stack for pairing up tokens.</span>
<span class="vi">@tokens = </span><span class="p">[]</span> <span class="c1"># Stream of parsed tokens in the form `['TYPE', value, line]`.</span>
<span class="vi">@chunkLine =</span>
<span class="nx">opts</span><span class="p">.</span><span class="nx">line</span> <span class="o">or</span> <span class="mi">0</span> <span class="c1"># The start line for the current @chunk.</span>
<span class="vi">@chunkColumn =</span>
<span class="nx">opts</span><span class="p">.</span><span class="nx">column</span> <span class="o">or</span> <span class="mi">0</span> <span class="c1"># The start column of the current @chunk.</span></pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-6">¶</a> </div> <p>At every position, run through this list of attempted matches,
short-circuiting if any of them succeed. Their order determines precedence:
<code>@literalToken</code> is the fallback catch-all.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">i = </span><span class="mi">0</span>
<span class="k">while</span> <span class="vi">@chunk = </span><span class="nx">code</span><span class="p">[</span><span class="nx">i</span><span class="p">..]</span>
<span class="nv">consumed = </span><span class="o">\</span>
<span class="nx">@identifierToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@commentToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@whitespaceToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@lineToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@heredocToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@stringToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@numberToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@regexToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@jsToken</span><span class="p">()</span> <span class="o">or</span>
<span class="nx">@literalToken</span><span class="p">()</span></pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-7">¶</a> </div> <p>Update position</p> </td> <td class="code"> <div class="highlight"><pre> <span class="p">[</span><span class="nx">@chunkLine</span><span class="p">,</span> <span class="nx">@chunkColumn</span><span class="p">]</span> <span class="o">=</span> <span class="nx">@getLineAndColumnFromChunk</span> <span class="nx">consumed</span>
<span class="nx">i</span> <span class="o">+=</span> <span class="nx">consumed</span>
<span class="nx">@closeIndentation</span><span class="p">()</span>
<span class="nx">@error</span> <span class="s">"missing </span><span class="si">#{</span><span class="nx">tag</span><span class="si">}</span><span class="s">"</span> <span class="k">if</span> <span class="nv">tag = </span><span class="nx">@ends</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="k">return</span> <span class="nx">@tokens</span> <span class="k">if</span> <span class="nx">opts</span><span class="p">.</span><span class="nx">rewrite</span> <span class="o">is</span> <span class="kc">off</span>
<span class="p">(</span><span class="k">new</span> <span class="nx">Rewriter</span><span class="p">).</span><span class="nx">rewrite</span> <span class="nx">@tokens</span></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-8">¶</a> </div> <p>Preprocess the code to remove leading and trailing whitespace, carriage
returns, etc. If we're lexing literate CoffeeScript, strip external Markdown
by removing all lines that aren't indented by at least four spaces or a tab.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">clean: </span><span class="nf">(code) -></span>
<span class="nv">code = </span><span class="nx">code</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="k">if</span> <span class="nx">code</span><span class="p">.</span><span class="nx">charCodeAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">is</span> <span class="nx">BOM</span>
<span class="nv">code = </span><span class="s">"\n</span><span class="si">#{</span><span class="nx">code</span><span class="si">}</span><span class="s">"</span> <span class="k">if</span> <span class="nx">WHITESPACE</span><span class="p">.</span><span class="nx">test</span> <span class="nx">code</span>
<span class="nv">code = </span><span class="nx">code</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/\r/g</span><span class="p">,</span> <span class="s">''</span><span class="p">).</span><span class="nx">replace</span> <span class="nx">TRAILING_SPACES</span><span class="p">,</span> <span class="s">''</span>
<span class="k">if</span> <span class="nx">@literate</span>
<span class="nv">lines = </span><span class="k">for</span> <span class="nx">line</span> <span class="k">in</span> <span class="nx">code</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="s">'\n'</span><span class="p">)</span>
<span class="k">if</span> <span class="nv">match = </span><span class="nx">LITERATE</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">line</span>
<span class="nx">line</span><span class="p">[</span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">length</span><span class="p">..]</span>
<span class="k">else</span>
<span class="s">'# '</span> <span class="o">+</span> <span class="nx">line</span>
<span class="nv">code = </span><span class="nx">lines</span><span class="p">.</span><span class="nx">join</span> <span class="s">'\n'</span>
<span class="nx">code</span></pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-9">¶</a> </div> <h2>Tokenizers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-10">¶</a> </div> <p>Matches identifying literals: variables, keywords, method names, etc.
Check to ensure that JavaScript reserved words aren't being used as
identifiers. Because CoffeeScript reserves a handful of keywords that are
allowed in JavaScript, we're careful not to tag them as keywords when
referenced as property names here, so you can still do <code>jQuery.is()</code> even
though <code>is</code> means <code>===</code> otherwise.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">identifierToken: </span><span class="nf">-></span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">IDENTIFIER</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="p">[</span><span class="nx">input</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">colon</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span></pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-11">¶</a> </div> <p>Preserve length of id for location data</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">idLength = </span><span class="nx">id</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">poppedToken = </span><span class="kc">undefined</span>
<span class="k">if</span> <span class="nx">id</span> <span class="o">is</span> <span class="s">'own'</span> <span class="o">and</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s">'FOR'</span>
<span class="nx">@token</span> <span class="s">'OWN'</span><span class="p">,</span> <span class="nx">id</span>
<span class="k">return</span> <span class="nx">id</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">forcedIdentifier = </span><span class="nx">colon</span> <span class="o">or</span>
<span class="p">(</span><span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">)</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="p">[</span><span class="s">'.'</span><span class="p">,</span> <span class="s">'?.'</span><span class="p">,</span> <span class="s">'::'</span><span class="p">]</span> <span class="o">or</span>
<span class="o">not</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'@'</span><span class="p">)</span>
<span class="nv">tag = </span><span class="s">'IDENTIFIER'</span>
<span class="k">if</span> <span class="o">not</span> <span class="nx">forcedIdentifier</span> <span class="o">and</span> <span class="p">(</span><span class="nx">id</span> <span class="k">in</span> <span class="nx">JS_KEYWORDS</span> <span class="o">or</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">COFFEE_KEYWORDS</span><span class="p">)</span>
<span class="nv">tag = </span><span class="nx">id</span><span class="p">.</span><span class="nx">toUpperCase</span><span class="p">()</span>
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'WHEN'</span> <span class="o">and</span> <span class="nx">@tag</span><span class="p">()</span> <span class="k">in</span> <span class="nx">LINE_BREAK</span>
<span class="nv">tag = </span><span class="s">'LEADING_WHEN'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'FOR'</span>
<span class="vi">@seenFor = </span><span class="kc">yes</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'UNLESS'</span>
<span class="nv">tag = </span><span class="s">'IF'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">in</span> <span class="nx">UNARY</span>
<span class="nv">tag = </span><span class="s">'UNARY'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">in</span> <span class="nx">RELATION</span>
<span class="k">if</span> <span class="nx">tag</span> <span class="o">isnt</span> <span class="s">'INSTANCEOF'</span> <span class="o">and</span> <span class="nx">@seenFor</span>
<span class="nv">tag = </span><span class="s">'FOR'</span> <span class="o">+</span> <span class="nx">tag</span>
<span class="vi">@seenFor = </span><span class="kc">no</span>
<span class="k">else</span>
<span class="nv">tag = </span><span class="s">'RELATION'</span>
<span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s">'!'</span>
<span class="nv">poppedToken = </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nv">id = </span><span class="s">'!'</span> <span class="o">+</span> <span class="nx">id</span>
<span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">JS_FORBIDDEN</span>
<span class="k">if</span> <span class="nx">forcedIdentifier</span>
<span class="nv">tag = </span><span class="s">'IDENTIFIER'</span>
<span class="nv">id = </span><span class="k">new</span> <span class="nb">String</span> <span class="nx">id</span>
<span class="nv">id.reserved = </span><span class="kc">yes</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">RESERVED</span>
<span class="nx">@error</span> <span class="s">"reserved word \"</span><span class="si">#{</span><span class="nx">id</span><span class="si">}</span><span class="s">\""</span>
<span class="k">unless</span> <span class="nx">forcedIdentifier</span>
<span class="nv">id = </span><span class="nx">COFFEE_ALIAS_MAP</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">COFFEE_ALIASES</span>
<span class="nv">tag = </span><span class="k">switch</span> <span class="nx">id</span>
<span class="k">when</span> <span class="s">'!'</span> <span class="k">then</span> <span class="s">'UNARY'</span>
<span class="k">when</span> <span class="s">'=='</span><span class="p">,</span> <span class="s">'!='</span> <span class="k">then</span> <span class="s">'COMPARE'</span>
<span class="k">when</span> <span class="s">'&&'</span><span class="p">,</span> <span class="s">'||'</span> <span class="k">then</span> <span class="s">'LOGIC'</span>
<span class="k">when</span> <span class="s">'true'</span><span class="p">,</span> <span class="s">'false'</span> <span class="k">then</span> <span class="s">'BOOL'</span>
<span class="k">when</span> <span class="s">'break'</span><span class="p">,</span> <span class="s">'continue'</span> <span class="k">then</span> <span class="s">'STATEMENT'</span>
<span class="k">else</span> <span class="nx">tag</span>
<span class="nv">tagToken = </span><span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">idLength</span>
<span class="k">if</span> <span class="nx">poppedToken</span>
<span class="p">[</span><span class="nx">tagToken</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">first_line</span><span class="p">,</span> <span class="nx">tagToken</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">first_column</span><span class="p">]</span> <span class="o">=</span>
<span class="p">[</span><span class="nx">poppedToken</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">first_line</span><span class="p">,</span> <span class="nx">poppedToken</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">first_column</span><span class="p">]</span>
<span class="k">if</span> <span class="nx">colon</span>
<span class="nv">colonOffset = </span><span class="nx">input</span><span class="p">.</span><span class="nx">lastIndexOf</span> <span class="s">':'</span>
<span class="nx">@token</span> <span class="s">':'</span><span class="p">,</span> <span class="s">':'</span><span class="p">,</span> <span class="nx">colonOffset</span><span class="p">,</span> <span class="nx">colon</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">input</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-12">¶</a> </div> <p>Matches numbers, including decimals, hex, and exponential notation.
Be careful not to interfere with ranges-in-progress.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">numberToken: </span><span class="nf">-></span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">NUMBER</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="nv">number = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="sr">/^0[BOX]/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">number</span>
<span class="nx">@error</span> <span class="s">"radix prefix '</span><span class="si">#{</span><span class="nx">number</span><span class="si">}</span><span class="s">' must be lowercase"</span>
<span class="k">else</span> <span class="k">if</span> <span class="sr">/E/</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">number</span><span class="p">)</span> <span class="o">and</span> <span class="o">not</span> <span class="sr">/^0x/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">number</span>
<span class="nx">@error</span> <span class="s">"exponential notation '</span><span class="si">#{</span><span class="nx">number</span><span class="si">}</span><span class="s">' must be indicated with a lowercase 'e'"</span>
<span class="k">else</span> <span class="k">if</span> <span class="sr">/^0\d*[89]/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">number</span>
<span class="nx">@error</span> <span class="s">"decimal literal '</span><span class="si">#{</span><span class="nx">number</span><span class="si">}</span><span class="s">' must not be prefixed with '0'"</span>
<span class="k">else</span> <span class="k">if</span> <span class="sr">/^0\d+/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">number</span>
<span class="nx">@error</span> <span class="s">"octal literal '</span><span class="si">#{</span><span class="nx">number</span><span class="si">}</span><span class="s">' must be prefixed with '0o'"</span>
<span class="nv">lexedLength = </span><span class="nx">number</span><span class="p">.</span><span class="nx">length</span>
<span class="k">if</span> <span class="nv">octalLiteral = </span><span class="sr">/^0o([0-7]+)/</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">number</span>
<span class="nv">number = </span><span class="s">'0x'</span> <span class="o">+</span> <span class="p">(</span><span class="nb">parseInt</span> <span class="nx">octalLiteral</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="mi">8</span><span class="p">).</span><span class="nx">toString</span> <span class="mi">16</span>
<span class="k">if</span> <span class="nv">binaryLiteral = </span><span class="sr">/^0b([01]+)/</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">number</span>
<span class="nv">number = </span><span class="s">'0x'</span> <span class="o">+</span> <span class="p">(</span><span class="nb">parseInt</span> <span class="nx">binaryLiteral</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="mi">2</span><span class="p">).</span><span class="nx">toString</span> <span class="mi">16</span>
<span class="nx">@token</span> <span class="s">'NUMBER'</span><span class="p">,</span> <span class="nx">number</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">lexedLength</span>
<span class="nx">lexedLength</span></pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-13">¶</a> </div> <p>Matches strings, including multi-line strings. Ensures that quotation marks
are balanced within the string's contents, and within nested interpolations.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">stringToken: </span><span class="nf">-></span>
<span class="k">switch</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
<span class="k">when</span> <span class="s">"'"</span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">SIMPLESTR</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="nv">string = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="nx">string</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">MULTILINER</span><span class="p">,</span> <span class="s">'\\\n'</span><span class="p">),</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">string</span><span class="p">.</span><span class="nx">length</span>
<span class="k">when</span> <span class="s">'"'</span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">string = </span><span class="nx">@balancedString</span> <span class="nx">@chunk</span><span class="p">,</span> <span class="s">'"'</span>
<span class="k">if</span> <span class="mi">0</span> <span class="o"><</span> <span class="nx">string</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s">'#{'</span><span class="p">,</span> <span class="mi">1</span>
<span class="nx">@interpolateString</span> <span class="nx">string</span><span class="p">[</span><span class="mi">1</span><span class="p">...</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="nv">strOffset: </span><span class="mi">1</span><span class="p">,</span> <span class="nv">lexedLength: </span><span class="nx">string</span><span class="p">.</span><span class="nx">length</span>
<span class="k">else</span>
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="nx">@escapeLines</span> <span class="nx">string</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">string</span><span class="p">.</span><span class="nx">length</span>
<span class="k">else</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="k">if</span> <span class="nv">octalEsc = </span><span class="sr">/^(?:\\.|[^\\])*\\(?:0[0-7]|[1-7])/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">string</span>
<span class="nx">@error</span> <span class="s">"octal escape sequences </span><span class="si">#{</span><span class="nx">string</span><span class="si">}</span><span class="s"> are not allowed"</span>
<span class="nx">string</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-14">¶</a> </div> <p>Matches heredocs, adjusting indentation to the correct level, as heredocs
preserve whitespace, but ignore indentation to the left.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">heredocToken: </span><span class="nf">-></span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">HEREDOC</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="nv">heredoc = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="nv">quote = </span><span class="nx">heredoc</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
<span class="nv">doc = </span><span class="nx">@sanitizeHeredoc</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="nv">quote: </span><span class="nx">quote</span><span class="p">,</span> <span class="nv">indent: </span><span class="kc">null</span>
<span class="k">if</span> <span class="nx">quote</span> <span class="o">is</span> <span class="s">'"'</span> <span class="o">and</span> <span class="mi">0</span> <span class="o"><=</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s">'#{'</span>
<span class="nx">@interpolateString</span> <span class="nx">doc</span><span class="p">,</span> <span class="nv">heredoc: </span><span class="kc">yes</span><span class="p">,</span> <span class="nv">strOffset: </span><span class="mi">3</span><span class="p">,</span> <span class="nv">lexedLength: </span><span class="nx">heredoc</span><span class="p">.</span><span class="nx">length</span>
<span class="k">else</span>
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="nx">@makeString</span><span class="p">(</span><span class="nx">doc</span><span class="p">,</span> <span class="nx">quote</span><span class="p">,</span> <span class="kc">yes</span><span class="p">),</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">heredoc</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">heredoc</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-15">¶</a> </div> <p>Matches and consumes comments.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">commentToken: </span><span class="nf">-></span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">COMMENT</span>
<span class="p">[</span><span class="nx">comment</span><span class="p">,</span> <span class="nx">here</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
<span class="k">if</span> <span class="nx">here</span>
<span class="nx">@token</span> <span class="s">'HERECOMMENT'</span><span class="p">,</span>
<span class="p">(</span><span class="nx">@sanitizeHeredoc</span> <span class="nx">here</span><span class="p">,</span>
<span class="nv">herecomment: </span><span class="kc">true</span><span class="p">,</span> <span class="nv">indent: </span><span class="nb">Array</span><span class="p">(</span><span class="nx">@indent</span> <span class="o">+</span> <span class="mi">1</span><span class="p">).</span><span class="nx">join</span><span class="p">(</span><span class="s">' '</span><span class="p">)),</span>
<span class="mi">0</span><span class="p">,</span> <span class="nx">comment</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">comment</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-16">¶</a> </div> <p>Matches JavaScript interpolated directly into the source via backticks.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">jsToken: </span><span class="nf">-></span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">is</span> <span class="s">'`'</span> <span class="o">and</span> <span class="nv">match = </span><span class="nx">JSTOKEN</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="nx">@token</span> <span class="s">'JS'</span><span class="p">,</span> <span class="p">(</span><span class="nv">script = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">])[</span><span class="mi">1</span><span class="p">...</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">script</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">script</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-17">¶</a> </div> <p>Matches regular expression literals. Lexing regular expressions is difficult
to distinguish from division, so we borrow some basic heuristics from
JavaScript and Ruby.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">regexToken: </span><span class="nf">-></span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">if</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">isnt</span> <span class="s">'/'</span>
<span class="k">if</span> <span class="nv">match = </span><span class="nx">HEREGEX</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="nv">length = </span><span class="nx">@heregexToken</span> <span class="nx">match</span>
<span class="k">return</span> <span class="nx">length</span>
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">if</span> <span class="nx">prev</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="p">(</span><span class="k">if</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span> <span class="k">then</span> <span class="nx">NOT_REGEX</span> <span class="k">else</span> <span class="nx">NOT_SPACED_REGEX</span><span class="p">))</span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">REGEX</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="p">[</span><span class="nx">match</span><span class="p">,</span> <span class="nx">regex</span><span class="p">,</span> <span class="nx">flags</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
<span class="k">if</span> <span class="nx">regex</span><span class="p">[..</span><span class="mi">1</span><span class="p">]</span> <span class="o">is</span> <span class="s">'/*'</span> <span class="k">then</span> <span class="nx">@error</span> <span class="s">'regular expressions cannot begin with `*`'</span>
<span class="k">if</span> <span class="nx">regex</span> <span class="o">is</span> <span class="s">'//'</span> <span class="k">then</span> <span class="nv">regex = </span><span class="s">'/(?:)/'</span>
<span class="nx">@token</span> <span class="s">'REGEX'</span><span class="p">,</span> <span class="s">"</span><span class="si">#{</span><span class="nx">regex</span><span class="si">}#{</span><span class="nx">flags</span><span class="si">}</span><span class="s">"</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">match</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">match</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-18">¶</a> </div> <p>Matches multiline extended regular expressions.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">heregexToken: </span><span class="nf">(match) -></span>
<span class="p">[</span><span class="nx">heregex</span><span class="p">,</span> <span class="nx">body</span><span class="p">,</span> <span class="nx">flags</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
<span class="k">if</span> <span class="mi">0</span> <span class="o">></span> <span class="nx">body</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s">'#{'</span>
<span class="nv">re = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">HEREGEX_OMIT</span><span class="p">,</span> <span class="s">''</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/\//g</span><span class="p">,</span> <span class="s">'\\/'</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">re</span><span class="p">.</span><span class="nx">match</span> <span class="sr">/^\*/</span> <span class="k">then</span> <span class="nx">@error</span> <span class="s">'regular expressions cannot begin with `*`'</span>
<span class="nx">@token</span> <span class="s">'REGEX'</span><span class="p">,</span> <span class="s">"/</span><span class="si">#{</span> <span class="nx">re</span> <span class="o">or</span> <span class="s">'(?:)'</span> <span class="si">}</span><span class="s">/</span><span class="si">#{</span><span class="nx">flags</span><span class="si">}</span><span class="s">"</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">heregex</span><span class="p">.</span><span class="nx">length</span>
<span class="k">return</span> <span class="nx">heregex</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">@token</span> <span class="s">'IDENTIFIER'</span><span class="p">,</span> <span class="s">'RegExp'</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span>
<span class="nx">@token</span> <span class="s">'CALL_START'</span><span class="p">,</span> <span class="s">'('</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span>
<span class="nv">tokens = </span><span class="p">[]</span>
<span class="k">for</span> <span class="nx">token</span> <span class="k">in</span> <span class="nx">@interpolateString</span><span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="nv">regex: </span><span class="kc">yes</span><span class="p">)</span>
<span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">]</span> <span class="o">=</span> <span class="nx">token</span>
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'TOKENS'</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">value</span><span class="p">...</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'NEOSTRING'</span>
<span class="k">continue</span> <span class="k">unless</span> <span class="nv">value = </span><span class="nx">value</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">HEREGEX_OMIT</span><span class="p">,</span> <span class="s">''</span></pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-19">¶</a> </div> <p>Convert NEOSTRING into STRING</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">value = </span><span class="nx">value</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/\\/g</span><span class="p">,</span> <span class="s">'\\\\'</span>
<span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'STRING'</span>
<span class="nx">token</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="nx">@makeString</span><span class="p">(</span><span class="nx">value</span><span class="p">,</span> <span class="s">'"'</span><span class="p">,</span> <span class="kc">yes</span><span class="p">)</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">token</span>
<span class="k">else</span>
<span class="nx">@error</span> <span class="s">"Unexpected </span><span class="si">#{</span><span class="nx">tag</span><span class="si">}</span><span class="s">"</span>
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
<span class="nv">plusToken = </span><span class="p">[</span><span class="s">'+'</span><span class="p">,</span> <span class="s">'+'</span><span class="p">]</span>
<span class="nx">plusToken</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="c1"># Copy location data</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">plusToken</span></pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-20">¶</a> </div> <p>Remove the extra "+"</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="k">unless</span> <span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">?</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'STRING'</span>
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="s">'""'</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span>
<span class="nx">@token</span> <span class="s">'+'</span><span class="p">,</span> <span class="s">'+'</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">tokens</span><span class="p">...</span>
<span class="k">if</span> <span class="nx">flags</span></pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-21">¶</a> </div> <p>Find the flags in the heregex</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">flagsOffset = </span><span class="nx">heregex</span><span class="p">.</span><span class="nx">lastIndexOf</span> <span class="nx">flags</span>
<span class="nx">@token</span> <span class="s">','</span><span class="p">,</span> <span class="s">','</span><span class="p">,</span> <span class="nx">flagsOffset</span><span class="p">,</span> <span class="mi">0</span>
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="s">'"'</span> <span class="o">+</span> <span class="nx">flags</span> <span class="o">+</span> <span class="s">'"'</span><span class="p">,</span> <span class="nx">flagsOffset</span><span class="p">,</span> <span class="nx">flags</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">@token</span> <span class="s">')'</span><span class="p">,</span> <span class="s">')'</span><span class="p">,</span> <span class="nx">heregex</span><span class="p">.</span><span class="nx">length</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span>
<span class="nx">heregex</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-22"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-22">¶</a> </div> <p>Matches newlines, indents, and outdents, and determines which is which.
If we can detect that the current line is continued onto the the next line,
then the newline is suppressed:</p>
<pre><code>elements
.each( ... )
.map( ... )
</code></pre>
<p>Keeps track of the level of indentation, because a single outdent token
can close multiple indents, so we need to know how far in we happen to be.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">lineToken: </span><span class="nf">-></span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">MULTI_DENT</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="nv">indent = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="vi">@seenFor = </span><span class="kc">no</span>
<span class="nv">size = </span><span class="nx">indent</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span> <span class="o">-</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">lastIndexOf</span> <span class="s">'\n'</span>
<span class="nv">noNewlines = </span><span class="nx">@unfinished</span><span class="p">()</span>
<span class="k">if</span> <span class="nx">size</span> <span class="o">-</span> <span class="nx">@indebt</span> <span class="o">is</span> <span class="nx">@indent</span>
<span class="k">if</span> <span class="nx">noNewlines</span> <span class="k">then</span> <span class="nx">@suppressNewlines</span><span class="p">()</span> <span class="k">else</span> <span class="nx">@newlineToken</span> <span class="mi">0</span>
<span class="k">return</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
<span class="k">if</span> <span class="nx">size</span> <span class="o">></span> <span class="nx">@indent</span>
<span class="k">if</span> <span class="nx">noNewlines</span>
<span class="vi">@indebt = </span><span class="nx">size</span> <span class="o">-</span> <span class="nx">@indent</span>
<span class="nx">@suppressNewlines</span><span class="p">()</span>
<span class="k">return</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">diff = </span><span class="nx">size</span> <span class="o">-</span> <span class="nx">@indent</span> <span class="o">+</span> <span class="nx">@outdebt</span>
<span class="nx">@token</span> <span class="s">'INDENT'</span><span class="p">,</span> <span class="nx">diff</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">@indents</span><span class="p">.</span><span class="nx">push</span> <span class="nx">diff</span>
<span class="nx">@ends</span><span class="p">.</span><span class="nx">push</span> <span class="s">'OUTDENT'</span>
<span class="vi">@outdebt = @indebt = </span><span class="mi">0</span>
<span class="k">else</span>
<span class="vi">@indebt = </span><span class="mi">0</span>
<span class="nx">@outdentToken</span> <span class="nx">@indent</span> <span class="o">-</span> <span class="nx">size</span><span class="p">,</span> <span class="nx">noNewlines</span><span class="p">,</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
<span class="vi">@indent = </span><span class="nx">size</span>
<span class="nx">indent</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-23"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-23">¶</a> </div> <p>Record an outdent token or multiple tokens, if we happen to be moving back
inwards past several recorded indents.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">outdentToken: </span><span class="nf">(moveOut, noNewlines, outdentLength) -></span>
<span class="k">while</span> <span class="nx">moveOut</span> <span class="o">></span> <span class="mi">0</span>
<span class="nv">len = </span><span class="nx">@indents</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
<span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o">is</span> <span class="kc">undefined</span>
<span class="nv">moveOut = </span><span class="mi">0</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o">is</span> <span class="nx">@outdebt</span>
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">@outdebt</span>
<span class="vi">@outdebt = </span><span class="mi">0</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o"><</span> <span class="nx">@outdebt</span>
<span class="nx">@outdebt</span> <span class="o">-=</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span>
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span>
<span class="k">else</span>
<span class="nv">dent = </span><span class="nx">@indents</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="o">+</span> <span class="nx">@outdebt</span>
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">dent</span>
<span class="vi">@outdebt = </span><span class="mi">0</span>
<span class="nx">@pair</span> <span class="s">'OUTDENT'</span>
<span class="nx">@token</span> <span class="s">'OUTDENT'</span><span class="p">,</span> <span class="nx">dent</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">outdentLength</span>
<span class="nx">@outdebt</span> <span class="o">-=</span> <span class="nx">moveOut</span> <span class="k">if</span> <span class="nx">dent</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">while</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s">';'</span>
<span class="nx">@token</span> <span class="s">'TERMINATOR'</span><span class="p">,</span> <span class="s">'\n'</span><span class="p">,</span> <span class="nx">outdentLength</span><span class="p">,</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s">'TERMINATOR'</span> <span class="o">or</span> <span class="nx">noNewlines</span>
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-24"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-24">¶</a> </div> <p>Matches and consumes non-meaningful whitespace. Tag the previous token
as being "spaced", because there are some cases where it makes a difference.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">whitespaceToken: </span><span class="nf">-></span>
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="p">(</span><span class="nv">match = </span><span class="nx">WHITESPACE</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span><span class="p">)</span> <span class="o">or</span>
<span class="p">(</span><span class="nv">nline = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">is</span> <span class="s">'\n'</span><span class="p">)</span>
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
<span class="nx">prev</span><span class="p">[</span><span class="k">if</span> <span class="nx">match</span> <span class="k">then</span> <span class="s">'spaced'</span> <span class="k">else</span> <span class="s">'newLine'</span><span class="p">]</span> <span class="o">=</span> <span class="kc">true</span> <span class="k">if</span> <span class="nx">prev</span>
<span class="k">if</span> <span class="nx">match</span> <span class="k">then</span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">length</span> <span class="k">else</span> <span class="mi">0</span></pre></div> </td> </tr> <tr id="section-25"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-25">¶</a> </div> <p>Generate a newline token. Consecutive newlines get merged together.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">newlineToken: </span><span class="nf">(offset) -></span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">while</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s">';'</span>
<span class="nx">@token</span> <span class="s">'TERMINATOR'</span><span class="p">,</span> <span class="s">'\n'</span><span class="p">,</span> <span class="nx">offset</span><span class="p">,</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s">'TERMINATOR'</span>
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-26">¶</a> </div> <p>Use a <code>\</code> at a line-ending to suppress the newline.
The slash is removed here once its job is done.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">suppressNewlines: </span><span class="nf">-></span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s">'\\'</span>
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-27"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-27">¶</a> </div> <p>We treat all other single characters as a token. E.g.: <code>( ) , . !</code>
Multi-character operators are also literal tokens, so that Jison can assign
the proper order of operations. There are some symbols that we tag specially
here. <code>;</code> and newlines are both treated as a <code>TERMINATOR</code>, we distinguish
parentheses that indicate a method call from regular parentheses, and so on.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">literalToken: </span><span class="nf">-></span>
<span class="k">if</span> <span class="nv">match = </span><span class="nx">OPERATOR</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
<span class="p">[</span><span class="nx">value</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
<span class="nx">@tagParameters</span><span class="p">()</span> <span class="k">if</span> <span class="nx">CODE</span><span class="p">.</span><span class="nx">test</span> <span class="nx">value</span>
<span class="k">else</span>
<span class="nv">value = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
<span class="nv">tag = </span><span class="nx">value</span>
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">'='</span> <span class="o">and</span> <span class="nx">prev</span>
<span class="k">if</span> <span class="o">not</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">reserved</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">in</span> <span class="nx">JS_FORBIDDEN</span>
<span class="nx">@error</span> <span class="s">"reserved word \"</span><span class="si">#{</span><span class="nx">@value</span><span class="p">()</span><span class="si">}</span><span class="s">\" can't be assigned"</span>
<span class="k">if</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">in</span> <span class="p">[</span><span class="s">'||'</span><span class="p">,</span> <span class="s">'&&'</span><span class="p">]</span>
<span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'COMPOUND_ASSIGN'</span>
<span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="s">'='</span>
<span class="k">return</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span>
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">';'</span>
<span class="vi">@seenFor = </span><span class="kc">no</span>
<span class="nv">tag = </span><span class="s">'TERMINATOR'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">MATH</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'MATH'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">COMPARE</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'COMPARE'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">COMPOUND_ASSIGN</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'COMPOUND_ASSIGN'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">UNARY</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'UNARY'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">SHIFT</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'SHIFT'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">LOGIC</span> <span class="o">or</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">'?'</span> <span class="o">and</span> <span class="nx">prev</span><span class="o">?</span><span class="p">.</span><span class="nx">spaced</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'LOGIC'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">prev</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span>
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">'('</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="nx">CALLABLE</span>
<span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'FUNC_EXIST'</span> <span class="k">if</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'?'</span>
<span class="nv">tag = </span><span class="s">'CALL_START'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">'['</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="nx">INDEXABLE</span>
<span class="nv">tag = </span><span class="s">'INDEX_START'</span>
<span class="k">switch</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">when</span> <span class="s">'?'</span> <span class="k">then</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'INDEX_SOAK'</span>
<span class="k">switch</span> <span class="nx">value</span>
<span class="k">when</span> <span class="s">'('</span><span class="p">,</span> <span class="s">'{'</span><span class="p">,</span> <span class="s">'['</span> <span class="k">then</span> <span class="nx">@ends</span><span class="p">.</span><span class="nx">push</span> <span class="nx">INVERSES</span><span class="p">[</span><span class="nx">value</span><span class="p">]</span>
<span class="k">when</span> <span class="s">')'</span><span class="p">,</span> <span class="s">'}'</span><span class="p">,</span> <span class="s">']'</span> <span class="k">then</span> <span class="nx">@pair</span> <span class="nx">value</span>
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
<span class="nx">value</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-28"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-28">¶</a> </div> <h2>Token Manipulators</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-29"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-29">¶</a> </div> <p>Sanitize a heredoc or herecomment by
erasing all external indentation on the left-hand side.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">sanitizeHeredoc: </span><span class="nf">(doc, options) -></span>
<span class="p">{</span><span class="nx">indent</span><span class="p">,</span> <span class="nx">herecomment</span><span class="p">}</span> <span class="o">=</span> <span class="nx">options</span>
<span class="k">if</span> <span class="nx">herecomment</span>
<span class="k">if</span> <span class="nx">HEREDOC_ILLEGAL</span><span class="p">.</span><span class="nx">test</span> <span class="nx">doc</span>
<span class="nx">@error</span> <span class="s">"block comment cannot contain \"*/\", starting"</span>
<span class="k">return</span> <span class="nx">doc</span> <span class="k">if</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="s">'\n'</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span>
<span class="k">else</span>
<span class="k">while</span> <span class="nv">match = </span><span class="nx">HEREDOC_INDENT</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">doc</span>
<span class="nv">attempt = </span><span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="nv">indent = </span><span class="nx">attempt</span> <span class="k">if</span> <span class="nx">indent</span> <span class="o">is</span> <span class="kc">null</span> <span class="o">or</span> <span class="mi">0</span> <span class="o"><</span> <span class="nx">attempt</span><span class="p">.</span><span class="nx">length</span> <span class="o"><</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">doc = </span><span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/// \n </span><span class="si">#{</span><span class="nx">indent</span><span class="si">}</span><span class="sr"> ///g</span><span class="p">,</span> <span class="s">'\n'</span> <span class="k">if</span> <span class="nx">indent</span>
<span class="nv">doc = </span><span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/^\n/</span><span class="p">,</span> <span class="s">''</span> <span class="k">unless</span> <span class="nx">herecomment</span>
<span class="nx">doc</span></pre></div> </td> </tr> <tr id="section-30"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-30">¶</a> </div> <p>A source of ambiguity in our grammar used to be parameter lists in function
definitions versus argument lists in function calls. Walk backwards, tagging
parameters specially in order to make things easier for the parser.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tagParameters: </span><span class="nf">-></span>
<span class="k">return</span> <span class="k">this</span> <span class="k">if</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">isnt</span> <span class="s">')'</span>
<span class="nv">stack = </span><span class="p">[]</span>
<span class="p">{</span><span class="nx">tokens</span><span class="p">}</span> <span class="o">=</span> <span class="k">this</span>
<span class="nv">i = </span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">tokens</span><span class="p">[</span><span class="o">--</span><span class="nx">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'PARAM_END'</span>
<span class="k">while</span> <span class="nv">tok = </span><span class="nx">tokens</span><span class="p">[</span><span class="o">--</span><span class="nx">i</span><span class="p">]</span>
<span class="k">switch</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">when</span> <span class="s">')'</span>
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nx">tok</span>
<span class="k">when</span> <span class="s">'('</span><span class="p">,</span> <span class="s">'CALL_START'</span>
<span class="k">if</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="k">then</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'('</span>
<span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'PARAM_START'</span>
<span class="k">return</span> <span class="k">this</span>
<span class="k">else</span> <span class="k">return</span> <span class="k">this</span>
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-31"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-31">¶</a> </div> <p>Close up all remaining open blocks at the end of the file.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">closeIndentation: </span><span class="nf">-></span>
<span class="nx">@outdentToken</span> <span class="nx">@indent</span></pre></div> </td> </tr> <tr id="section-32"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-32">¶</a> </div> <p>Matches a balanced group such as a single or double-quoted string. Pass in
a series of delimiters, all of which must be nested correctly within the
contents of the string. This method allows us to have strings within
interpolations within strings, ad infinitum.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">balancedString: </span><span class="nf">(str, end) -></span>
<span class="nv">continueCount = </span><span class="mi">0</span>
<span class="nv">stack = </span><span class="p">[</span><span class="nx">end</span><span class="p">]</span>
<span class="k">for</span> <span class="nx">i</span> <span class="k">in</span> <span class="p">[</span><span class="mi">1</span><span class="p">...</span><span class="nx">str</span><span class="p">.</span><span class="nx">length</span><span class="p">]</span>
<span class="k">if</span> <span class="nx">continueCount</span>
<span class="o">--</span><span class="nx">continueCount</span>
<span class="k">continue</span>
<span class="k">switch</span> <span class="nv">letter = </span><span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span> <span class="nx">i</span>
<span class="k">when</span> <span class="s">'\\'</span>
<span class="o">++</span><span class="nx">continueCount</span>
<span class="k">continue</span>
<span class="k">when</span> <span class="nx">end</span>
<span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="k">unless</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">length</span>
<span class="k">return</span> <span class="nx">str</span><span class="p">[</span><span class="mi">0</span><span class="p">..</span><span class="nx">i</span><span class="p">]</span>
<span class="nv">end = </span><span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="k">in</span> <span class="p">[</span><span class="s">'"'</span><span class="p">,</span> <span class="s">"'"</span><span class="p">]</span>
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="nx">letter</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'/'</span> <span class="o">and</span> <span class="nv">match = </span><span class="p">(</span><span class="nx">HEREGEX</span><span class="p">.</span><span class="nx">exec</span><span class="p">(</span><span class="nx">str</span><span class="p">[</span><span class="nx">i</span><span class="p">..])</span> <span class="o">or</span> <span class="nx">REGEX</span><span class="p">.</span><span class="nx">exec</span><span class="p">(</span><span class="nx">str</span><span class="p">[</span><span class="nx">i</span><span class="p">..]))</span>
<span class="nx">continueCount</span> <span class="o">+=</span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'{'</span>
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="s">'}'</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s">'"'</span> <span class="o">and</span> <span class="nx">prev</span> <span class="o">is</span> <span class="s">'#'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'{'</span>
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="s">'}'</span>
<span class="nv">prev = </span><span class="nx">letter</span>
<span class="nx">@error</span> <span class="s">"missing </span><span class="si">#{</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="si">}</span><span class="s">, starting"</span></pre></div> </td> </tr> <tr id="section-33"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-33">¶</a> </div> <p>Expand variables and expressions inside double-quoted strings using
Ruby-like notation for substitution of arbitrary expressions.</p>
<pre><code>"Hello #{name.capitalize()}."
</code></pre>
<p>If it encounters an interpolation, this method will recursively create a
new Lexer, tokenize the interpolated contents, and merge them into the
token stream.</p>
<ul>
<li><code>str</code> is the start of the string contents (IE with the " or """ stripped
off.)</li>
<li><code>options.offsetInChunk</code> is the start of the interpolated string in the
current chunk, including the " or """, etc... If not provided, this is
assumed to be 0. <code>options.lexedLength</code> is the length of the
interpolated string, including both the start and end quotes. Both of these
values are ignored if <code>options.regex</code> is true.</li>
<li><code>options.strOffset</code> is the offset of str, relative to the start of the
current chunk.</li>
</ul> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">interpolateString: </span><span class="nf">(str, options = {}) -></span>
<span class="p">{</span><span class="nx">heredoc</span><span class="p">,</span> <span class="nx">regex</span><span class="p">,</span> <span class="nx">offsetInChunk</span><span class="p">,</span> <span class="nx">strOffset</span><span class="p">,</span> <span class="nx">lexedLength</span><span class="p">}</span> <span class="o">=</span> <span class="nx">options</span>
<span class="nv">offsetInChunk = </span><span class="nx">offsetInChunk</span> <span class="o">||</span> <span class="mi">0</span>
<span class="nv">strOffset = </span><span class="nx">strOffset</span> <span class="o">||</span> <span class="mi">0</span>
<span class="nv">lexedLength = </span><span class="nx">lexedLength</span> <span class="o">||</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-34"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-34">¶</a> </div> <p>Clip leading \n from heredoc</p> </td> <td class="code"> <div class="highlight"><pre> <span class="k">if</span> <span class="nx">heredoc</span> <span class="o">and</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span> <span class="o">></span> <span class="mi">0</span> <span class="o">and</span> <span class="nx">str</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s">'\n'</span>
<span class="nv">str = </span><span class="nx">str</span><span class="p">[</span><span class="mi">1</span><span class="p">...]</span>
<span class="nx">strOffset</span><span class="o">++</span></pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-35">¶</a> </div> <p>Parse the string.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tokens = </span><span class="p">[]</span>
<span class="nv">pi = </span><span class="mi">0</span>
<span class="nv">i = </span><span class="o">-</span><span class="mi">1</span>
<span class="k">while</span> <span class="nv">letter = </span><span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span> <span class="nx">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'\\'</span>
<span class="nx">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">continue</span>
<span class="k">unless</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'#'</span> <span class="o">and</span> <span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="nx">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">is</span> <span class="s">'{'</span> <span class="o">and</span>
<span class="p">(</span><span class="nv">expr = </span><span class="nx">@balancedString</span> <span class="nx">str</span><span class="p">[</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">..],</span> <span class="s">'}'</span><span class="p">)</span>
<span class="k">continue</span></pre></div> </td> </tr> <tr id="section-36"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-36">¶</a> </div> <p>NEOSTRING is a fake token. This will be converted to a string below.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">@makeToken</span><span class="p">(</span><span class="s">'NEOSTRING'</span><span class="p">,</span> <span class="nx">str</span><span class="p">[</span><span class="nx">pi</span><span class="p">...</span><span class="nx">i</span><span class="p">],</span> <span class="nx">strOffset</span> <span class="o">+</span> <span class="nx">pi</span><span class="p">)</span> <span class="k">if</span> <span class="nx">pi</span> <span class="o"><</span> <span class="nx">i</span>
<span class="nv">inner = </span><span class="nx">expr</span><span class="p">[</span><span class="mi">1</span><span class="p">...</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="nx">inner</span><span class="p">.</span><span class="nx">length</span>
<span class="p">[</span><span class="nx">line</span><span class="p">,</span> <span class="nx">column</span><span class="p">]</span> <span class="o">=</span> <span class="nx">@getLineAndColumnFromChunk</span><span class="p">(</span><span class="nx">strOffset</span> <span class="o">+</span> <span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="nv">nested = </span><span class="k">new</span> <span class="nx">Lexer</span><span class="p">().</span><span class="nx">tokenize</span> <span class="nx">inner</span><span class="p">,</span> <span class="nv">line: </span><span class="nx">line</span><span class="p">,</span> <span class="nv">column: </span><span class="nx">column</span><span class="p">,</span> <span class="nv">rewrite: </span><span class="kc">off</span>
<span class="nv">popped = </span><span class="nx">nested</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nv">popped = </span><span class="nx">nested</span><span class="p">.</span><span class="nx">shift</span><span class="p">()</span> <span class="k">if</span> <span class="nx">nested</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">?</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'TERMINATOR'</span>
<span class="k">if</span> <span class="nv">len = </span><span class="nx">nested</span><span class="p">.</span><span class="nx">length</span>
<span class="k">if</span> <span class="nx">len</span> <span class="o">></span> <span class="mi">1</span>
<span class="nx">nested</span><span class="p">.</span><span class="nx">unshift</span> <span class="nx">@makeToken</span> <span class="s">'('</span><span class="p">,</span> <span class="s">'('</span><span class="p">,</span> <span class="nx">strOffset</span> <span class="o">+</span> <span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span>
<span class="nx">nested</span><span class="p">.</span><span class="nx">push</span> <span class="nx">@makeToken</span> <span class="s">')'</span><span class="p">,</span> <span class="s">')'</span><span class="p">,</span> <span class="nx">strOffset</span> <span class="o">+</span> <span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">+</span> <span class="nx">inner</span><span class="p">.</span><span class="nx">length</span><span class="p">,</span> <span class="mi">0</span></pre></div> </td> </tr> <tr id="section-37"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-37">¶</a> </div> <p>Push a fake 'TOKENS' token, which will get turned into real tokens below.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">'TOKENS'</span><span class="p">,</span> <span class="nx">nested</span><span class="p">]</span>
<span class="nx">i</span> <span class="o">+=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">pi = </span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">@makeToken</span><span class="p">(</span><span class="s">'NEOSTRING'</span><span class="p">,</span> <span class="nx">str</span><span class="p">[</span><span class="nx">pi</span><span class="p">..],</span> <span class="nx">strOffset</span> <span class="o">+</span> <span class="nx">pi</span><span class="p">)</span> <span class="k">if</span> <span class="nx">i</span> <span class="o">></span> <span class="nx">pi</span> <span class="o"><</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-38"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-38">¶</a> </div> <p>If regex, then return now and let the regex code deal with all these fake tokens</p> </td> <td class="code"> <div class="highlight"><pre> <span class="k">return</span> <span class="nx">tokens</span> <span class="k">if</span> <span class="nx">regex</span></pre></div> </td> </tr> <tr id="section-39"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-39">¶</a> </div> <p>If we didn't find any tokens, then just return an empty string.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="k">return</span> <span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="s">'""'</span><span class="p">,</span> <span class="nx">offsetInChunk</span><span class="p">,</span> <span class="nx">lexedLength</span> <span class="k">unless</span> <span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-40"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-40">¶</a> </div> <p>If the first token is not a string, add a fake empty string to the beginning.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">tokens</span><span class="p">.</span><span class="nx">unshift</span> <span class="nx">@makeToken</span><span class="p">(</span><span class="s">'NEOSTRING'</span><span class="p">,</span> <span class="s">''</span><span class="p">,</span> <span class="nx">offsetInChunk</span><span class="p">)</span> <span class="k">unless</span> <span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'NEOSTRING'</span>
<span class="nx">@token</span> <span class="s">'('</span><span class="p">,</span> <span class="s">'('</span><span class="p">,</span> <span class="nx">offsetInChunk</span><span class="p">,</span> <span class="mi">0</span> <span class="k">if</span> <span class="nv">interpolated = </span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">></span> <span class="mi">1</span></pre></div> </td> </tr> <tr id="section-41"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-41">¶</a> </div> <p>Push all the tokens</p> </td> <td class="code"> <div class="highlight"><pre> <span class="k">for</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">i</span> <span class="k">in</span> <span class="nx">tokens</span>
<span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">]</span> <span class="o">=</span> <span class="nx">token</span>
<span class="k">if</span> <span class="nx">i</span></pre></div> </td> </tr> <tr id="section-42"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-42">¶</a> </div> <p>Create a 0-length "+" token.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">plusToken = </span><span class="nx">@token</span> <span class="s">'+'</span><span class="p">,</span> <span class="s">'+'</span> <span class="k">if</span> <span class="nx">i</span>
<span class="nv">locationToken = </span><span class="k">if</span> <span class="nx">tag</span> <span class="o">==</span> <span class="s">'TOKENS'</span> <span class="k">then</span> <span class="nx">value</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">else</span> <span class="nx">token</span>
<span class="nx">plusToken</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span>
<span class="nv">first_line: </span><span class="nx">locationToken</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">first_line</span>
<span class="nv">first_column: </span><span class="nx">locationToken</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">first_column</span>
<span class="nv">last_line: </span><span class="nx">locationToken</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">first_line</span>
<span class="nv">last_column: </span><span class="nx">locationToken</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">first_column</span>
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'TOKENS'</span></pre></div> </td> </tr> <tr id="section-43"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-43">¶</a> </div> <p>Push all the tokens in the fake 'TOKENS' token. These already have
sane location data.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">value</span><span class="p">...</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'NEOSTRING'</span></pre></div> </td> </tr> <tr id="section-44"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-44">¶</a> </div> <p>Convert NEOSTRING into STRING</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'STRING'</span>
<span class="nx">token</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="nx">@makeString</span> <span class="nx">value</span><span class="p">,</span> <span class="s">'"'</span><span class="p">,</span> <span class="nx">heredoc</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">token</span>
<span class="k">else</span>
<span class="nx">@error</span> <span class="s">"Unexpected </span><span class="si">#{</span><span class="nx">tag</span><span class="si">}</span><span class="s">"</span>
<span class="nx">@token</span> <span class="s">')'</span><span class="p">,</span> <span class="s">')'</span><span class="p">,</span> <span class="nx">offsetInChunk</span> <span class="o">+</span> <span class="nx">lexedLength</span><span class="p">,</span> <span class="mi">0</span> <span class="k">if</span> <span class="nx">interpolated</span>
<span class="nx">tokens</span></pre></div> </td> </tr> <tr id="section-45"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-45">¶</a> </div> <p>Pairs up a closing token, ensuring that all listed pairs of tokens are
correctly balanced throughout the course of the token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">pair: </span><span class="nf">(tag) -></span>
<span class="k">unless</span> <span class="nx">tag</span> <span class="o">is</span> <span class="nv">wanted = </span><span class="nx">last</span> <span class="nx">@ends</span>
<span class="nx">@error</span> <span class="s">"unmatched </span><span class="si">#{</span><span class="nx">tag</span><span class="si">}</span><span class="s">"</span> <span class="k">unless</span> <span class="s">'OUTDENT'</span> <span class="o">is</span> <span class="nx">wanted</span></pre></div> </td> </tr> <tr id="section-46"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-46">¶</a> </div> <p>Auto-close INDENT to support syntax like this:</p>
<pre><code>el.click((event) ->
el.hide())
</code></pre> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">@indent</span> <span class="o">-=</span> <span class="nv">size = </span><span class="nx">last</span> <span class="nx">@indents</span>
<span class="nx">@outdentToken</span> <span class="nx">size</span><span class="p">,</span> <span class="kc">true</span>
<span class="k">return</span> <span class="nx">@pair</span> <span class="nx">tag</span>
<span class="nx">@ends</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span></pre></div> </td> </tr> <tr id="section-47"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-47">¶</a> </div> <h2>Helpers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-48"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-48">¶</a> </div> <p>Returns the line and column number from an offset into the current chunk.</p>
<p><code>offset</code> is a number of characters into @chunk.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">getLineAndColumnFromChunk: </span><span class="nf">(offset) -></span>
<span class="k">if</span> <span class="nx">offset</span> <span class="o">is</span> <span class="mi">0</span>
<span class="k">return</span> <span class="p">[</span><span class="nx">@chunkLine</span><span class="p">,</span> <span class="nx">@chunkColumn</span><span class="p">]</span>
<span class="k">if</span> <span class="nx">offset</span> <span class="o">>=</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">string = </span><span class="nx">@chunk</span>
<span class="k">else</span>
<span class="nv">string = </span><span class="nx">@chunk</span><span class="p">[..</span><span class="nx">offset</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="nv">lineCount = </span><span class="nx">count</span> <span class="nx">string</span><span class="p">,</span> <span class="s">'\n'</span>
<span class="nv">column = </span><span class="nx">@chunkColumn</span>
<span class="k">if</span> <span class="nx">lineCount</span> <span class="o">></span> <span class="mi">0</span>
<span class="nv">lines = </span><span class="nx">string</span><span class="p">.</span><span class="nx">split</span> <span class="s">'\n'</span>
<span class="nv">column = </span><span class="p">(</span><span class="nx">last</span> <span class="nx">lines</span><span class="p">).</span><span class="nx">length</span>
<span class="k">else</span>
<span class="nx">column</span> <span class="o">+=</span> <span class="nx">string</span><span class="p">.</span><span class="nx">length</span>
<span class="k">return</span> <span class="p">[</span><span class="nx">@chunkLine</span> <span class="o">+</span> <span class="nx">lineCount</span><span class="p">,</span> <span class="nx">column</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-49"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-49">¶</a> </div> <p>Same as "token", exception this just returns the token without adding it
to the results.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">makeToken: </span><span class="nf">(tag, value, offsetInChunk, length) -></span>
<span class="nv">offsetInChunk = </span><span class="nx">offsetInChunk</span> <span class="o">||</span> <span class="mi">0</span>
<span class="k">if</span> <span class="nx">length</span> <span class="o">is</span> <span class="kc">undefined</span> <span class="k">then</span> <span class="nv">length = </span><span class="nx">value</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">locationData = </span><span class="p">{}</span>
<span class="p">[</span><span class="nx">locationData</span><span class="p">.</span><span class="nx">first_line</span><span class="p">,</span> <span class="nx">locationData</span><span class="p">.</span><span class="nx">first_column</span><span class="p">]</span> <span class="o">=</span>
<span class="nx">@getLineAndColumnFromChunk</span> <span class="nx">offsetInChunk</span></pre></div> </td> </tr> <tr id="section-50"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-50">¶</a> </div> <p>Use length - 1 for the final offset - we're supplying the last<em>line and the last</em>column,
so if last<em>column == first</em>column, then we're looking at a character of length 1.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">lastCharacter = </span><span class="k">if</span> <span class="nx">length</span> <span class="o">></span> <span class="mi">0</span> <span class="k">then</span> <span class="p">(</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="k">else</span> <span class="mi">0</span>
<span class="p">[</span><span class="nx">locationData</span><span class="p">.</span><span class="nx">last_line</span><span class="p">,</span> <span class="nx">locationData</span><span class="p">.</span><span class="nx">last_column</span><span class="p">]</span> <span class="o">=</span>
<span class="nx">@getLineAndColumnFromChunk</span> <span class="nx">offsetInChunk</span> <span class="o">+</span> <span class="p">(</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="nv">token = </span><span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">locationData</span><span class="p">]</span>
<span class="k">return</span> <span class="nx">token</span></pre></div> </td> </tr> <tr id="section-51"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-51">¶</a> </div> <p>Add a token to the results.
<code>offset</code> is the offset into the current @chunk where the token starts.
<code>length</code> is the length of the token in the @chunk, after the offset. If
not specified, the length of <code>value</code> will be used.</p>
<p>Returns the new token.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">token: </span><span class="nf">(tag, value, offsetInChunk, length) -></span>
<span class="nv">token = </span><span class="nx">@makeToken</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">offsetInChunk</span><span class="p">,</span> <span class="nx">length</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">token</span>
<span class="k">return</span> <span class="nx">token</span></pre></div> </td> </tr> <tr id="section-52"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-52">¶</a> </div> <p>Peek at a tag in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag: </span><span class="nf">(index, tag) -></span>
<span class="p">(</span><span class="nv">tok = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">and</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="nx">tag</span> <span class="k">else</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-53"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-53">¶</a> </div> <p>Peek at a value in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">value: </span><span class="nf">(index, val) -></span>
<span class="p">(</span><span class="nv">tok = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">and</span> <span class="k">if</span> <span class="nx">val</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="nx">val</span> <span class="k">else</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-54"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-54">¶</a> </div> <p>Are we in the midst of an unfinished expression?</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">unfinished: </span><span class="nf">-></span>
<span class="nx">LINE_CONTINUER</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">@chunk</span><span class="p">)</span> <span class="o">or</span>
<span class="nx">@tag</span><span class="p">()</span> <span class="k">in</span> <span class="p">[</span><span class="s">'\\'</span><span class="p">,</span> <span class="s">'.'</span><span class="p">,</span> <span class="s">'?.'</span><span class="p">,</span> <span class="s">'UNARY'</span><span class="p">,</span> <span class="s">'MATH'</span><span class="p">,</span> <span class="s">'+'</span><span class="p">,</span> <span class="s">'-'</span><span class="p">,</span> <span class="s">'SHIFT'</span><span class="p">,</span> <span class="s">'RELATION'</span>
<span class="s">'COMPARE'</span><span class="p">,</span> <span class="s">'LOGIC'</span><span class="p">,</span> <span class="s">'THROW'</span><span class="p">,</span> <span class="s">'EXTENDS'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-55"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-55">¶</a> </div> <p>Converts newlines for string literals.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">escapeLines: </span><span class="nf">(str, heredoc) -></span>
<span class="nx">str</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">MULTILINER</span><span class="p">,</span> <span class="k">if</span> <span class="nx">heredoc</span> <span class="k">then</span> <span class="s">'\\n'</span> <span class="k">else</span> <span class="s">''</span></pre></div> </td> </tr> <tr id="section-56"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-56">¶</a> </div> <p>Constructs a string token by escaping quotes and newlines.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">makeString: </span><span class="nf">(body, quote, heredoc) -></span>
<span class="k">return</span> <span class="nx">quote</span> <span class="o">+</span> <span class="nx">quote</span> <span class="k">unless</span> <span class="nx">body</span>
<span class="nv">body = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/\\([\s\S])/g</span><span class="p">,</span> <span class="nf">(match, contents) -></span>
<span class="k">if</span> <span class="nx">contents</span> <span class="k">in</span> <span class="p">[</span><span class="s">'\n'</span><span class="p">,</span> <span class="nx">quote</span><span class="p">]</span> <span class="k">then</span> <span class="nx">contents</span> <span class="k">else</span> <span class="nx">match</span>
<span class="nv">body = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/// </span><span class="si">#{</span><span class="nx">quote</span><span class="si">}</span><span class="sr"> ///g</span><span class="p">,</span> <span class="s">'\\$&'</span>
<span class="nx">quote</span> <span class="o">+</span> <span class="nx">@escapeLines</span><span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="nx">heredoc</span><span class="p">)</span> <span class="o">+</span> <span class="nx">quote</span></pre></div> </td> </tr> <tr id="section-57"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-57">¶</a> </div> <p>Throws a syntax error on the current <code>@line</code>.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">error: </span><span class="nf">(message) -></span></pre></div> </td> </tr> <tr id="section-58"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-58">¶</a> </div> <p>TODO: Are there some cases we could improve the error line number by
passing the offset in the chunk where the error happened?</p> </td> <td class="code"> <div class="highlight"><pre> <span class="k">throw</span> <span class="nx">SyntaxError</span> <span class="s">"</span><span class="si">#{</span><span class="nx">message</span><span class="si">}</span><span class="s"> on line </span><span class="si">#{</span> <span class="nx">@chunkLine</span> <span class="o">+</span> <span class="mi">1</span> <span class="si">}</span><span class="s">"</span></pre></div> </td> </tr> <tr id="section-59"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-59">¶</a> </div> <h2>Constants</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-60"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-60">¶</a> </div> <p>Keywords that CoffeeScript shares in common with JavaScript.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_KEYWORDS = </span><span class="p">[</span>
<span class="s">'true'</span><span class="p">,</span> <span class="s">'false'</span><span class="p">,</span> <span class="s">'null'</span><span class="p">,</span> <span class="s">'this'</span>
<span class="s">'new'</span><span class="p">,</span> <span class="s">'delete'</span><span class="p">,</span> <span class="s">'typeof'</span><span class="p">,</span> <span class="s">'in'</span><span class="p">,</span> <span class="s">'instanceof'</span>
<span class="s">'return'</span><span class="p">,</span> <span class="s">'throw'</span><span class="p">,</span> <span class="s">'break'</span><span class="p">,</span> <span class="s">'continue'</span><span class="p">,</span> <span class="s">'debugger'</span>
<span class="s">'if'</span><span class="p">,</span> <span class="s">'else'</span><span class="p">,</span> <span class="s">'switch'</span><span class="p">,</span> <span class="s">'for'</span><span class="p">,</span> <span class="s">'while'</span><span class="p">,</span> <span class="s">'do'</span><span class="p">,</span> <span class="s">'try'</span><span class="p">,</span> <span class="s">'catch'</span><span class="p">,</span> <span class="s">'finally'</span>
<span class="s">'class'</span><span class="p">,</span> <span class="s">'extends'</span><span class="p">,</span> <span class="s">'super'</span>
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-61"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-61">¶</a> </div> <p>CoffeeScript-only keywords.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COFFEE_KEYWORDS = </span><span class="p">[</span><span class="s">'undefined'</span><span class="p">,</span> <span class="s">'then'</span><span class="p">,</span> <span class="s">'unless'</span><span class="p">,</span> <span class="s">'until'</span><span class="p">,</span> <span class="s">'loop'</span><span class="p">,</span> <span class="s">'of'</span><span class="p">,</span> <span class="s">'by'</span><span class="p">,</span> <span class="s">'when'</span><span class="p">]</span>
<span class="nv">COFFEE_ALIAS_MAP =</span>
<span class="o">and</span> <span class="o">:</span> <span class="s">'&&'</span>
<span class="o">or</span> <span class="o">:</span> <span class="s">'||'</span>
<span class="o">is</span> <span class="o">:</span> <span class="s">'=='</span>
<span class="o">isnt</span> <span class="o">:</span> <span class="s">'!='</span>
<span class="o">not</span> <span class="o">:</span> <span class="s">'!'</span>
<span class="kc">yes</span> <span class="o">:</span> <span class="s">'true'</span>
<span class="kc">no</span> <span class="o">:</span> <span class="s">'false'</span>
<span class="kc">on</span> <span class="o">:</span> <span class="s">'true'</span>
<span class="kc">off</span> <span class="o">:</span> <span class="s">'false'</span>
<span class="nv">COFFEE_ALIASES = </span><span class="p">(</span><span class="nx">key</span> <span class="k">for</span> <span class="nx">key</span> <span class="k">of</span> <span class="nx">COFFEE_ALIAS_MAP</span><span class="p">)</span>
<span class="nv">COFFEE_KEYWORDS = </span><span class="nx">COFFEE_KEYWORDS</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">COFFEE_ALIASES</span></pre></div> </td> </tr> <tr id="section-62"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-62">¶</a> </div> <p>The list of keywords that are reserved by JavaScript, but not used, or are
used by CoffeeScript internally. We throw an error when these are encountered,
to avoid having a JavaScript error at runtime.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">RESERVED = </span><span class="p">[</span>
<span class="s">'case'</span><span class="p">,</span> <span class="s">'default'</span><span class="p">,</span> <span class="s">'function'</span><span class="p">,</span> <span class="s">'var'</span><span class="p">,</span> <span class="s">'void'</span><span class="p">,</span> <span class="s">'with'</span><span class="p">,</span> <span class="s">'const'</span><span class="p">,</span> <span class="s">'let'</span><span class="p">,</span> <span class="s">'enum'</span>
<span class="s">'export'</span><span class="p">,</span> <span class="s">'import'</span><span class="p">,</span> <span class="s">'native'</span><span class="p">,</span> <span class="s">'__hasProp'</span><span class="p">,</span> <span class="s">'__extends'</span><span class="p">,</span> <span class="s">'__slice'</span><span class="p">,</span> <span class="s">'__bind'</span>
<span class="s">'__indexOf'</span><span class="p">,</span> <span class="s">'implements'</span><span class="p">,</span> <span class="s">'interface'</span><span class="p">,</span> <span class="s">'package'</span><span class="p">,</span> <span class="s">'private'</span><span class="p">,</span> <span class="s">'protected'</span>
<span class="s">'public'</span><span class="p">,</span> <span class="s">'static'</span><span class="p">,</span> <span class="s">'yield'</span>
<span class="p">]</span>
<span class="nv">STRICT_PROSCRIBED = </span><span class="p">[</span><span class="s">'arguments'</span><span class="p">,</span> <span class="s">'eval'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-63"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-63">¶</a> </div> <p>The superset of both JavaScript keywords and reserved words, none of which may
be used as identifiers or properties.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_FORBIDDEN = </span><span class="nx">JS_KEYWORDS</span><span class="p">.</span><span class="nx">concat</span><span class="p">(</span><span class="nx">RESERVED</span><span class="p">).</span><span class="nx">concat</span><span class="p">(</span><span class="nx">STRICT_PROSCRIBED</span><span class="p">)</span>
<span class="nv">exports.RESERVED = </span><span class="nx">RESERVED</span><span class="p">.</span><span class="nx">concat</span><span class="p">(</span><span class="nx">JS_KEYWORDS</span><span class="p">).</span><span class="nx">concat</span><span class="p">(</span><span class="nx">COFFEE_KEYWORDS</span><span class="p">).</span><span class="nx">concat</span><span class="p">(</span><span class="nx">STRICT_PROSCRIBED</span><span class="p">)</span>
<span class="nv">exports.STRICT_PROSCRIBED = </span><span class="nx">STRICT_PROSCRIBED</span></pre></div> </td> </tr> <tr id="section-64"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-64">¶</a> </div> <p>The character code of the nasty Microsoft madness otherwise known as the BOM.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">BOM = </span><span class="mi">65279</span></pre></div> </td> </tr> <tr id="section-65"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-65">¶</a> </div> <p>Token matching regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">IDENTIFIER = </span><span class="sr">/// ^</span>
<span class="sr"> ( [$A-Za-z_\x7f-\uffff][$\w\x7f-\uffff]* )</span>
<span class="sr"> ( [^\n\S]* : (?!:) )? # Is this a property name?</span>
<span class="sr">///</span>
<span class="nv">NUMBER = </span><span class="sr">///</span>
<span class="sr"> ^ 0b[01]+ | # binary</span>
<span class="sr"> ^ 0o[0-7]+ | # octal</span>
<span class="sr"> ^ 0x[\da-f]+ | # hex</span>
<span class="sr"> ^ \d*\.?\d+ (?:e[+-]?\d+)? # decimal</span>
<span class="sr">///i</span>
<span class="nv">HEREDOC = </span><span class="sr">/// ^ ("""|''') ([\s\S]*?) (?:\n[^\n\S]*)? \1 ///</span>
<span class="nv">OPERATOR = </span><span class="sr">/// ^ (</span>
<span class="sr"> ?: [-=]> # function</span>
<span class="sr"> | [-+*/%<>&|^!?=]= # compound assign / compare</span>
<span class="sr"> | >>>=? # zero-fill right shift</span>
<span class="sr"> | ([-+:])\1 # doubles</span>
<span class="sr"> | ([&|<>])\2=? # logic / shift</span>
<span class="sr"> | \?\. # soak access</span>
<span class="sr"> | \.{2,3} # range or splat</span>
<span class="sr">) ///</span>
<span class="nv">WHITESPACE = </span><span class="sr">/^[^\n\S]+/</span>
<span class="nv">COMMENT = </span><span class="sr">/^###([^#][\s\S]*?)(?:###[^\n\S]*|(?:###)$)|^(?:\s*#(?!##[^#]).*)+/</span>
<span class="nv">LITERATE = </span><span class="sr">/^([ ]{4}|\t)/</span>
<span class="nv">CODE = </span><span class="sr">/^[-=]>/</span>
<span class="nv">MULTI_DENT = </span><span class="sr">/^(?:\n[^\n\S]*)+/</span>
<span class="nv">SIMPLESTR = </span><span class="sr">/^'[^\\']*(?:\\.[^\\']*)*'/</span>
<span class="nv">JSTOKEN = </span><span class="sr">/^`[^\\`]*(?:\\.[^\\`]*)*`/</span></pre></div> </td> </tr> <tr id="section-66"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-66">¶</a> </div> <p>Regex-matching-regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">REGEX = </span><span class="sr">/// ^</span>
<span class="sr"> (/ (?! [\s=] ) # disallow leading whitespace or equals signs</span>
<span class="sr"> [^ [ / \n \\ ]* # every other thing</span>
<span class="sr"> (?:</span>
<span class="sr"> (?: \\[\s\S] # anything escaped</span>
<span class="sr"> | \[ # character class</span>
<span class="sr"> [^ \] \n \\ ]*</span>
<span class="sr"> (?: \\[\s\S] [^ \] \n \\ ]* )*</span>
<span class="sr"> ]</span>
<span class="sr"> ) [^ [ / \n \\ ]*</span>
<span class="sr"> )*</span>
<span class="sr"> /) ([imgy]{0,4}) (?!\w)</span>
<span class="sr">///</span>
<span class="nv">HEREGEX = </span><span class="sr">/// ^ /{3} ([\s\S]+?) /{3} ([imgy]{0,4}) (?!\w) ///</span>
<span class="nv">HEREGEX_OMIT = </span><span class="sr">/\s+(?:#.*)?/g</span></pre></div> </td> </tr> <tr id="section-67"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-67">¶</a> </div> <p>Token cleaning regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">MULTILINER = </span><span class="sr">/\n/g</span>
<span class="nv">HEREDOC_INDENT = </span><span class="sr">/\n+([^\n\S]*)/g</span>
<span class="nv">HEREDOC_ILLEGAL = </span><span class="sr">/\*\//</span>
<span class="nv">LINE_CONTINUER = </span><span class="sr">/// ^ \s* (?: , | \??\.(?![.\d]) | :: ) ///</span>
<span class="nv">TRAILING_SPACES = </span><span class="sr">/\s+$/</span></pre></div> </td> </tr> <tr id="section-68"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-68">¶</a> </div> <p>Compound assignment tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COMPOUND_ASSIGN = </span><span class="p">[</span>
<span class="s">'-='</span><span class="p">,</span> <span class="s">'+='</span><span class="p">,</span> <span class="s">'/='</span><span class="p">,</span> <span class="s">'*='</span><span class="p">,</span> <span class="s">'%='</span><span class="p">,</span> <span class="s">'||='</span><span class="p">,</span> <span class="s">'&&='</span><span class="p">,</span> <span class="s">'?='</span><span class="p">,</span> <span class="s">'<<='</span><span class="p">,</span> <span class="s">'>>='</span><span class="p">,</span> <span class="s">'>>>='</span><span class="p">,</span> <span class="s">'&='</span><span class="p">,</span> <span class="s">'^='</span><span class="p">,</span> <span class="s">'|='</span>
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-69"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-69">¶</a> </div> <p>Unary tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">UNARY = </span><span class="p">[</span><span class="s">'!'</span><span class="p">,</span> <span class="s">'~'</span><span class="p">,</span> <span class="s">'NEW'</span><span class="p">,</span> <span class="s">'TYPEOF'</span><span class="p">,</span> <span class="s">'DELETE'</span><span class="p">,</span> <span class="s">'DO'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-70"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-70">¶</a> </div> <p>Logical tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">LOGIC = </span><span class="p">[</span><span class="s">'&&'</span><span class="p">,</span> <span class="s">'||'</span><span class="p">,</span> <span class="s">'&'</span><span class="p">,</span> <span class="s">'|'</span><span class="p">,</span> <span class="s">'^'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-71"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-71">¶</a> </div> <p>Bit-shifting tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">SHIFT = </span><span class="p">[</span><span class="s">'<<'</span><span class="p">,</span> <span class="s">'>>'</span><span class="p">,</span> <span class="s">'>>>'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-72"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-72">¶</a> </div> <p>Comparison tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COMPARE = </span><span class="p">[</span><span class="s">'=='</span><span class="p">,</span> <span class="s">'!='</span><span class="p">,</span> <span class="s">'<'</span><span class="p">,</span> <span class="s">'>'</span><span class="p">,</span> <span class="s">'<='</span><span class="p">,</span> <span class="s">'>='</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-73"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-73">¶</a> </div> <p>Mathematical tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">MATH = </span><span class="p">[</span><span class="s">'*'</span><span class="p">,</span> <span class="s">'/'</span><span class="p">,</span> <span class="s">'%'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-74"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-74">¶</a> </div> <p>Relational tokens that are negatable with <code>not</code> prefix.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">RELATION = </span><span class="p">[</span><span class="s">'IN'</span><span class="p">,</span> <span class="s">'OF'</span><span class="p">,</span> <span class="s">'INSTANCEOF'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-75"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-75">¶</a> </div> <p>Boolean tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">BOOL = </span><span class="p">[</span><span class="s">'TRUE'</span><span class="p">,</span> <span class="s">'FALSE'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-76"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-76">¶</a> </div> <p>Tokens which a regular expression will never immediately follow, but which
a division operator might.</p>
<p>See: http://www.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions</p>
<p>Our list is shorter, due to sans-parentheses method calls.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">NOT_REGEX = </span><span class="p">[</span><span class="s">'NUMBER'</span><span class="p">,</span> <span class="s">'REGEX'</span><span class="p">,</span> <span class="s">'BOOL'</span><span class="p">,</span> <span class="s">'NULL'</span><span class="p">,</span> <span class="s">'UNDEFINED'</span><span class="p">,</span> <span class="s">'++'</span><span class="p">,</span> <span class="s">'--'</span><span class="p">,</span> <span class="s">']'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-77"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-77">¶</a> </div> <p>If the previous token is not spaced, there are more preceding tokens that
force a division parse:</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">NOT_SPACED_REGEX = </span><span class="nx">NOT_REGEX</span><span class="p">.</span><span class="nx">concat</span> <span class="s">')'</span><span class="p">,</span> <span class="s">'}'</span><span class="p">,</span> <span class="s">'THIS'</span><span class="p">,</span> <span class="s">'IDENTIFIER'</span><span class="p">,</span> <span class="s">'STRING'</span></pre></div> </td> </tr> <tr id="section-78"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-78">¶</a> </div> <p>Tokens which could legitimately be invoked or indexed. An opening
parentheses or bracket following these tokens will be recorded as the start
of a function invocation or indexing operation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">CALLABLE = </span><span class="p">[</span><span class="s">'IDENTIFIER'</span><span class="p">,</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="s">'REGEX'</span><span class="p">,</span> <span class="s">')'</span><span class="p">,</span> <span class="s">']'</span><span class="p">,</span> <span class="s">'}'</span><span class="p">,</span> <span class="s">'?'</span><span class="p">,</span> <span class="s">'::'</span><span class="p">,</span> <span class="s">'@'</span><span class="p">,</span> <span class="s">'THIS'</span><span class="p">,</span> <span class="s">'SUPER'</span><span class="p">]</span>
<span class="nv">INDEXABLE = </span><span class="nx">CALLABLE</span><span class="p">.</span><span class="nx">concat</span> <span class="s">'NUMBER'</span><span class="p">,</span> <span class="s">'BOOL'</span><span class="p">,</span> <span class="s">'NULL'</span><span class="p">,</span> <span class="s">'UNDEFINED'</span></pre></div> </td> </tr> <tr id="section-79"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-79">¶</a> </div> <p>Tokens that, when immediately preceding a <code>WHEN</code>, indicate that the <code>WHEN</code>
occurs at the start of a line. We disambiguate these from trailing whens to
avoid an ambiguity in the grammar.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">LINE_BREAK = </span><span class="p">[</span><span class="s">'INDENT'</span><span class="p">,</span> <span class="s">'OUTDENT'</span><span class="p">,</span> <span class="s">'TERMINATOR'</span><span class="p">]</span>
</pre></div> </td> </tr> </tbody> </table> </div> </body> </html>