Permalink
Browse files

added a function to split the code into multiple lines. requires this

parse-js commit:

marijnh/parse-js@5483bf5
  • Loading branch information...
1 parent 5e64455 commit d5194ca509868dc07c7f20142981c602f1c0a724 @mishoo committed Jan 25, 2011
Showing with 62 additions and 4 deletions.
  1. +18 −2 README.html
  2. +7 −0 README.org
  3. +2 −1 cl-uglify-js.asd
  4. +2 −1 src/package.lisp
  5. +33 −0 src/split.lisp
View
@@ -7,7 +7,7 @@
<title>cl-uglify-js -- JavaScript compressor/beautifier for Common Lisp</title>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<meta name="generator" content="Org-mode"/>
-<meta name="generated" content="2010-11-13 16:18:47 CET"/>
+<meta name="generated" content="2011-01-24 00:09:50 EET"/>
<meta name="author" content="Mihai Bazon"/>
<meta name="description" content="JavaScript compressor/beautifier for Common Lisp"/>
<meta name="keywords" content="javascript, js, parser, compiler, compressor, mangle, minify, minifier, common lisp"/>
@@ -122,6 +122,7 @@ <h2>Table of Contents</h2>
<li><a href="#sec-2_1">2.1 ast-squeeze (ast &amp;key (sequences T) (dead-code T)) </a></li>
<li><a href="#sec-2_2">2.2 ast-mangle (ast &amp;key toplevel) </a></li>
<li><a href="#sec-2_3">2.3 ast-gen-code (ast &amp;key (beautify T) (indent-level 4) (indent-start 0) quote-keys) </a></li>
+<li><a href="#sec-2_4">2.4 split-code (code &amp;optional (maxlen (* 32 1024))) </a></li>
</ul>
</li>
<li><a href="#sec-3">3 Compress one file </a></li>
@@ -274,6 +275,21 @@ <h4 id="sec-2_3"><span class="section-number-4">2.3</span> ast-gen-code (ast &am
</ul>
+</div>
+
+</div>
+
+<div id="outline-container-2_4" class="outline-4">
+<h4 id="sec-2_4"><span class="section-number-4">2.4</span> split-code (code &amp;optional (maxlen (* 32 1024))) </h4>
+<div class="outline-text-4" id="text-2_4">
+
+
+<p>
+Given `code' (a string) it will split it by adding a newline every `maxlen'
+(or so) characters. I found both Firefox and Chrome croak with weird errors
+when the whole code was on a single 680K line, so even if it adds a few more
+bytes this step should be done for safety. The default `maxlen' is 32K.
+</p>
</div>
</div>
@@ -343,7 +359,7 @@ <h3 id="sec-4"><span class="section-number-3">4</span> License </h3>
<div id="postamble">
<p class="author"> Author: Mihai Bazon
</p>
-<p class="date"> Date: 2010-11-13 16:18:47 CET</p>
+<p class="date"> Date: 2011-01-24 00:09:50 EET</p>
<p class="creator">HTML generated by org-mode 7.01trans in emacs 23</p>
</div>
</div>
View
@@ -93,6 +93,13 @@ The other arguments only make sense if =beautify= is *true*:
(i.e. reserved words such as "while"). Pass this *true* to quote all keys
regardless.
+*** split-code (code &optional (maxlen (* 32 1024)))
+
+Given `code' (a string) it will split it by adding a newline every `maxlen'
+(or so) characters. I found both Firefox and Chrome croak with weird errors
+when the whole code was on a single 680K line, so even if it adds a few more
+bytes this step should be done for safety. The default `maxlen' is 32K.
+
** Compress one file
To compress one file, you would do something like this:
View
@@ -9,4 +9,5 @@
(:file "walker")
(:file "mangle")
(:file "squeeze")
- (:file "codegen")))))
+ (:file "codegen")
+ (:file "split")))))
View
@@ -4,5 +4,6 @@
#:ast-mangle
#:ast-squeeze
#:ast-case
- #:ast-walk)
+ #:ast-walk
+ #:split-code)
(:nicknames #:uglify-js))
View
@@ -0,0 +1,33 @@
+(in-package #:uglify-js)
+
+(defun split-code (code &optional (maxlen (* 32 1024)))
+ "Inserts newlines into the given `code' (must be a string) as soon
+as lines are bigger than `maxlen' (default 32K). Note that the lines
+in the output will generally be bigger than `maxlen'. This function
+splits code only *before* a :KEYWORD, :ATOM, :NAME or :PUNC, and it
+never splits *after* a :KEYWORD (in order not to break the JS
+semantics, which say that no newline is allowed between return or
+throw and their arguments)."
+ (with-input-from-string (stream code)
+ (let ((tokenizer (lex-js stream :include-comments t))
+ (this-token nil)
+ (prev-token nil)
+ (splits (list 0)))
+ (labels ((next-token ()
+ (setf prev-token this-token
+ this-token (funcall tokenizer)))
+ (current-length ()
+ (- (token-pos this-token) (car splits)))
+ (split-here ()
+ (push (token-pos this-token) splits)))
+ (iter (for i = (next-token))
+ (until (eq :eof (token-type i)))
+ (when (and prev-token
+ (eq :keyword (token-type prev-token)))
+ (next-iteration))
+ (when (> (current-length) maxlen)
+ (when (member (token-type this-token) '(:keyword :atom :name :punc))
+ (split-here))))
+ (format nil "~{~A~^~%~}"
+ (iter (for (from to) on (nreverse splits))
+ (collect (subseq code from to))))))))

0 comments on commit d5194ca

Please sign in to comment.