Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Support for license comments /*! license */ #332

Open
wants to merge 6 commits into from

9 participants

Kenneth Kufluk Robert Gust‑Bardon Jacob Ben Alman Chris Campbell Christoph Mal Graty Mathias Bynens Jörn Zaefferer
Kenneth Kufluk

Adding support for license statements marked like this:
/*!
license
*/
These will be new elements in the AST, and will always be generated into the minified code.
This fixes #85, #306, some of #275, maybe some others.

I also pass the comments and line numbers out of the parser, by adding extra properties to the AST arrays.
This is useful to those using the parser without the minifier.

Kenneth Kufluk Support for licenses statements marked like this:
/*!
  license
*/
These will be new elements in the AST, and will always be generated into the minified code.

I also pass the comments and line numbers out of the parser, by adding extra properties to the AST arrays.
This is useful to those using the parser without the minifier.
28efb89
Robert Gust‑Bardon

Due to an extra next() (line 579 of ./lib/parse-js.js):

  • $ echo '/**/' | uglifyjs results in an error,
  • $ echo '/*a*/' | uglifyjs results in another error,
  • $ echo '/*ab*/' | uglifyjs results in /*b*/;.
Kenneth Kufluk

Hmm. Yes, good point. Wondered whether that next() would affect anything.

Jacob
fat commented

This might break the bin implementation which already extracts comments and then readds them.

https://github.com/mishoo/UglifyJS/blob/master/bin/uglifyjs#L265

Kenneth Kufluk

I think it's compatible with show_copyight. Doesn't cause me errors with a few simple test cases.

(Sorry for weird last commit message. Forgot !s didn't work in commit msgs.)

lib/parse-js.js
@@ -258,6 +258,12 @@ function JS_Parse_Error(message, line, col, pos) {
258 258 this.col = col + 1;
259 259 this.pos = pos + 1;
260 260 this.stack = new Error().stack;
  261 + try {
4
Jörn Zaefferer
jzaefferer added a note

What does this have to do with the pull request?

Nothing. It just reports errors with context, which makes it far far easier to debug this sort of thing.

Jörn Zaefferer
jzaefferer added a note

Shouldn't you then remove line 260?

I've removed this, for a cleaner pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Ben Alman

A huge +1 for this functionality being added in!

Chris Campbell

Another +1... I'm using @kennethkufluk's fork in the meantime (thanks!).

Christoph
xrstf commented

Another +1 -- this makes UglifyJS more suitable for people wanting to use the "official" (1) way to embed their license.

(1) jQuery does it, so it must be "the right way". ;-)

Mal Graty
mal commented

Unfortunately adding the license to the AST means that it breaks contextual minifications that look around using prev() and peek(), this is especially apparent when using --no-licenses mode.

in.js

/*! License 1 */
/*! Multiline
    License 2
*/
var a=1;
/*! License 3 */
var b=1;

var a=1,b=1;

mishoo/master

$ uglifyjs -nc in.js
var a=1,b=1;

kennethkufluk/master

$ bin/uglifyjs -nc -nl in.js
var a=1;var b=1;

The new license node prevents the code processing the var b from seeing that the previous node was var a and combining the two.

Ben Alman

@mal that makes sense, and is the behavior I'd expect. The most common use-case for preserving inline license comments assumes they are at the top of individual files concatenated together into a single script, in which case each sub-script will most likely be inside an IIFE anyways.

Kenneth Kufluk

@mal Hmm, good catch.

Mal Graty
mal commented

@cowboy I agree with the common use-case, however there's still some loss with IIFEs; consider:

/*! License 1 */
(function(){var a=1;})();
/*! License 2 */
(function(){var b=1;})();

mishoo/master

(function(){var a=1})(),function(){var a=1}();

kennethkufluk/master

(function(){var a=1})();(function(){var a=1})();
Kenneth Kufluk

@mal I still think the license should be in the AST, to prevent loss & confusion during mangling.

I guess I could add exceptions to the next, prev and peek methods? Do you think that would work?

Mal Graty
mal commented

@kennethkufluk In theory ignoring non-code AST nodes unless processing the node itself should resolve that particular problem, but there's other perils associated with having non-code items as first class nodes in the AST:

Function Hoisting

/*! License goes walk about with no IIFE */
function a () {}
var b;
function a(){}/*! License goes walk about with no IIFE */var b;

Sequences

var a=1,
/*! Very suspiciously placed license comment kills the parser */
b=1;
DEBUG: Error
    at new JS_Parse_Error (/home/mal/code/js/UglifyJS/lib/parse-js.js:260:22)
    ...
Kenneth Kufluk

@mal Yes, I see the problem.

I always knew that badly placed licenses would cause havoc. You can put a comment pretty much anywhere. Maybe we need to exclude badly-placed licenses. But, hey, licenses are not hard to place.

Maybe the solution is to simply not AST them if --no-licenses is enabled, solving the issue for those that don't care.
And otherwise leave as-is - the licenses forming barriers between licensed blocks of code.

Ben Alman

FWIW, keeping /*! ... */ comments is great, but it would be nice if there was also an option to keep all /* ... */ comments.

Mal Graty
mal commented

In an ideal world I think we'd just keep the comments_before property while the AST gets manipulated, and then either output as many or as few comments as we like from gen_code. The problem is that it's easier said than done; all node properties get dropped almost every time the tree is walked. I think it's the scale of the refactor required that has made it more of a 2.0 feature in past discussions.

For now though @kennethkufluk's suggestion of ignoring weird license placements sounds like the best short term solution. Potential rules being something like: they must be in the toplevel immediately preceding an IIFE or function?

Mal Graty
mal commented

Alternatively; another, much simpler, solution that I imagine solves most use-cases (command line only) is to add documentation for, and make use of, the --make argument. It allows a JSON file to specify a set of files to be uglified, and uglifies them independently before concatenating the output. This means that the first comment in each file is preserved in place.

Ben Alman

Adding complex rules that dictate where to-be-preserved comments need to be located might be overly complex. What if you just documented that comment nodes between statements that would otherwise be combined will prevent them from being combined?

Mathias Bynens

RT @cowboy

Adding complex rules that dictate where to-be-preserved comments need to be located might be overly complex. What if you just documented that comment nodes between statements that would otherwise be combined will prevent them from being combined?

Mal Graty
mal commented

@cowboy Agreed, documentation > complex rules, but at a minimum weirdly placed comments should not be allowed to crash the parser.

Ben Alman

@mal fair enough!

hui leoner referenced this pull request in spmjs/spm
Closed

注释问题 #334

Terin Stock terinjokes referenced this pull request in lodash/lodash
Closed

Add "@license" to copyright header block #138

Kyle Robinson Young shama referenced this pull request in gruntjs/grunt
Closed

Preserve /*! comments from being stripped #596

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 6 unique commits by 2 authors.

Mar 02, 2012
Kenneth Kufluk Support for licenses statements marked like this:
/*!
  license
*/
These will be new elements in the AST, and will always be generated into the minified code.

I also pass the comments and line numbers out of the parser, by adding extra properties to the AST arrays.
This is useful to those using the parser without the minifier.
28efb89
Mar 05, 2012
Kenneth Kufluk kennethkufluk Remove extra next() from multiline comments. eba6613
Kenneth Kufluk kennethkufluk Remove duplicate ./apachectl stop Remove comments. ea7f505
Mar 18, 2012
Kenneth Kufluk kennethkufluk Tidy error messages in parser. a2ba573
Kenneth Kufluk kennethkufluk Add option -nl to remove license statements from generated code. 3321026
Kenneth Kufluk kennethkufluk Add tests for license statement. 7551bf1
This page is out of date. Refresh to see the latest.
5 README.html
@@ -449,6 +449,11 @@ <h3 id="sec-1-4"><span class="section-number-3">1.4</span> Usage </h3>
449 449 etc.). If you pass this it will discard it.
450 450
451 451 </li>
  452 +<li><code>-nl</code> or <code>--no-licenses</code> &mdash; by default, <code>uglifyjs</code> will keep the text
  453 + marked /*! */ in the generated code (assumed to be license information
  454 + etc.). If you pass this it will discard it.
  455 +
  456 +</li>
452 457 <li><code>-o filename</code> or <code>--output filename</code> &mdash; put the result in <code>filename</code>. If
453 458 this isn't given, the result goes to standard output (or see next one).
454 459
4 README.org
Source Rendered
@@ -224,6 +224,10 @@ Supported options:
224 224 comment tokens in the generated code (assumed to be copyright information
225 225 etc.). If you pass this it will discard it.
226 226
  227 +- =-nl= or =--no-licenses= --- by default, =uglifyjs= will keep the text
  228 + marked /*! */ in the generated code (assumed to be license information
  229 + etc.). If you pass this it will discard it.
  230 +
227 231 - =-o filename= or =--output filename= --- put the result in =filename=. If
228 232 this isn't given, the result goes to standard output (or see next one).
229 233
9 bin/uglifyjs
@@ -3,7 +3,7 @@
3 3
4 4 global.sys = require(/^v0\.[012]/.test(process.version) ? "sys" : "util");
5 5 var fs = require("fs");
6   -var uglify = require("uglify-js"), // symlink ~/.node_libraries/uglify-js.js to ../uglify-js.js
  6 +var uglify = require("../uglify-js"), // symlink ~/.node_libraries/uglify-js.js to ../uglify-js.js
7 7 consolidator = uglify.consolidator,
8 8 jsp = uglify.parser,
9 9 pro = uglify.uglify;
@@ -32,7 +32,8 @@ var options = {
32 32 indent_start: 0,
33 33 quote_keys: false,
34 34 space_colon: false,
35   - inline_script: false
  35 + inline_script: false,
  36 + licenses: true
36 37 },
37 38 make: false,
38 39 output: true // stdout
@@ -86,6 +87,10 @@ out: while (args.length > 0) {
86 87 case "-nc":
87 88 options.show_copyright = false;
88 89 break;
  90 + case "--no-licenses":
  91 + case "-nl":
  92 + options.codegen_options.licenses = false;
  93 + break;
89 94 case "-o":
90 95 case "--output":
91 96 options.output = args.shift();
37 lib/parse-js.js
@@ -466,7 +466,6 @@ function tokenizer($TEXT) {
466 466 };
467 467
468 468 function read_multiline_comment() {
469   - next();
470 469 return with_eof_error("Unterminated multiline comment", function(){
471 470 var i = find("*/", true),
472 471 text = S.text.substring(S.pos, i);
@@ -546,6 +545,18 @@ function tokenizer($TEXT) {
546 545 return token("operator", grow(prefix || next()));
547 546 };
548 547
  548 + function read_license() {
  549 + return with_eof_error("Unterminated license statement", function(){
  550 + var i = find("*/", true),
  551 + text = S.text.substring(S.pos, i);
  552 + for (var j=0; j<text.length+2; j++) {
  553 + next();
  554 + }
  555 + S.newline_before = text.indexOf("\n") >= 0;
  556 + return token("license", text);
  557 + });
  558 + };
  559 +
549 560 function handle_slash() {
550 561 next();
551 562 var regex_allowed = S.regex_allowed;
@@ -555,8 +566,15 @@ function tokenizer($TEXT) {
555 566 S.regex_allowed = regex_allowed;
556 567 return next_token();
557 568 case "*":
558   - S.comments_before.push(read_multiline_comment());
  569 + next();
559 570 S.regex_allowed = regex_allowed;
  571 + if (peek()=='!') {
  572 + // A license statment starts /*!
  573 + next();
  574 + return read_license();
  575 + } else {
  576 + S.comments_before.push(read_multiline_comment());
  577 + }
560 578 return next_token();
561 579 }
562 580 return S.regex_allowed ? read_regexp("") : read_operator("/");
@@ -682,7 +700,6 @@ function NodeWithToken(str, start, end) {
682 700 NodeWithToken.prototype.toString = function() { return this.name; };
683 701
684 702 function parse($TEXT, exigent_mode, embed_tokens) {
685   -
686 703 var S = {
687 704 input : typeof $TEXT == "string" ? tokenizer($TEXT, true) : $TEXT,
688 705 token : null,
@@ -755,7 +772,12 @@ function parse($TEXT, exigent_mode, embed_tokens) {
755 772 };
756 773
757 774 function as() {
758   - return slice(arguments);
  775 + var arr = slice(arguments);
  776 + if (S.token.comments_before && S.token.comments_before.length) {
  777 + arr.comments_before = S.token.comments_before;
  778 + }
  779 + if (S.token.line) arr.line = S.token.line;
  780 + return arr;
759 781 };
760 782
761 783 function parenthesised() {
@@ -792,6 +814,9 @@ function parse($TEXT, exigent_mode, embed_tokens) {
792 814 case "atom":
793 815 return simple_statement();
794 816
  817 + case "license":
  818 + return as("license", prog1(S.token.value, next));
  819 +
795 820 case "name":
796 821 return is_token(peek(), "punc", ":")
797 822 ? labeled_statement(prog1(S.token.value, next, next))
@@ -886,6 +911,10 @@ function parse($TEXT, exigent_mode, embed_tokens) {
886 911 return as("label", label, stat);
887 912 };
888 913
  914 + function license() {
  915 + return as("license", prog1(expression));
  916 + };
  917 +
889 918 function simple_statement() {
890 919 return as("stat", prog1(expression, semicolon));
891 920 };
12 lib/process.js
@@ -202,6 +202,9 @@ function ast_walker() {
202 202 },
203 203 "atom": function(name) {
204 204 return [ this[0], name ];
  205 + },
  206 + "license": function(name) {
  207 + return [ this[0], name ];
205 208 }
206 209 };
207 210
@@ -1418,7 +1421,8 @@ function gen_code(ast, options) {
1418 1421 space_colon : false,
1419 1422 beautify : false,
1420 1423 ascii_only : false,
1421   - inline_script: false
  1424 + inline_script: false,
  1425 + licenses: true
1422 1426 });
1423 1427 var beautify = !!options.beautify;
1424 1428 var indentation = 0,
@@ -1547,6 +1551,12 @@ function gen_code(ast, options) {
1547 1551 "string": encode_string,
1548 1552 "num": make_num,
1549 1553 "name": make_name,
  1554 + "license": function(text){
  1555 + if (!options.licenses) {
  1556 + return '';
  1557 + }
  1558 + return "/*!" + text + '*/';
  1559 + },
1550 1560 "debugger": function(){ return "debugger" },
1551 1561 "toplevel": function(statements) {
1552 1562 return make_block_statements(statements)
1  test/testparser.js
@@ -23,6 +23,7 @@ function ParserTestSuite(callback){
23 23 ["var abc = 5;", "Regular variable statement with assignment"],
24 24 ["/* */;", "Multiline comment"],
25 25 ['/** **/;', 'Double star multiline comment'],
  26 + ["/*! */;", "License statement"],
26 27 ["var f = function(){;};", "Function expression in var assignment"],
27 28 ['hi; // moo\n;', 'single line comment'],
28 29 ['var varwithfunction;', 'Dont match keywords as substrings'], // difference between `var withsomevar` and `"str"` (local search and lits)
3  test/unit/compress/expected/license.js
... ... @@ -0,0 +1,3 @@
  1 +/*! License 1 *//*! Multiline
  2 + License 2
  3 +*/var a=1;/*! License 3 */var b=1
7 test/unit/compress/test/license.js
... ... @@ -0,0 +1,7 @@
  1 +/*! License 1 */
  2 +/*! Multiline
  3 + License 2
  4 +*/
  5 +var a=1;
  6 +/*! License 3 */
  7 +var b=1;

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.