Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop #307

Merged
merged 113 commits into from Aug 26, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
aa64e78
Redirect go.mod tdewolff/parse
tdewolff Feb 25, 2020
a026c67
Update JS minifier
tdewolff Mar 2, 2020
0adde17
Update JS minifier
tdewolff Mar 4, 2020
5dbe71a
Update JS minifier
tdewolff Mar 4, 2020
16e1299
Update JS minifier
tdewolff Mar 11, 2020
6ec6ab1
Add benchmark tests for new JS parser
tdewolff Mar 12, 2020
c0e16d4
Update JS minifier
tdewolff Mar 12, 2020
bcb1bfd
Update JS minifier
tdewolff Mar 13, 2020
0fff2c4
Remove '_, err = w.Write(); err != nil' checks, improves performance …
tdewolff Mar 13, 2020
ce17301
Use predefined byte-slices, increases performance by ~1%
tdewolff Mar 13, 2020
dfe7d6b
Use new parse.Input data structure instead of buffer.Lexer
tdewolff Mar 13, 2020
ea71dc9
Update JS minifier
tdewolff Mar 17, 2020
d52eb5d
Add first variable naming implementation
tdewolff Mar 18, 2020
e357418
Update JS parser to add name mangling
tdewolff Mar 22, 2020
64d306d
JS: update renamer and semicolon inserter
tdewolff Jun 1, 2020
16c129a
JS: more semicolon and renaming tests
tdewolff Jun 2, 2020
d9d3d39
JS: update tests
tdewolff Jun 2, 2020
4914561
Merge master
tdewolff Jun 2, 2020
455cb3f
JS: minify true,false,undefined
tdewolff Jun 3, 2020
7db1aab
JS: fix variable renaming sequence
tdewolff Jun 3, 2020
c270434
JS: minify if statement to expression
tdewolff Jun 3, 2020
fe50d84
JS: minify if to expression; remove group parenthesis
tdewolff Jun 3, 2020
30b5c37
JS: fix group parameter removal
tdewolff Jun 3, 2020
122ef33
Update comments
tdewolff Jun 3, 2020
c436d2e
JS: add optional chaining tests
tdewolff Jun 3, 2020
258acb1
Update readme
tdewolff Jun 3, 2020
1f6d34a
JS: merge expr statements; minify arrow functions
tdewolff Jun 5, 2020
e359286
Update go.mod
tdewolff Jun 5, 2020
aed709e
JS: function/class remain as expression; merge if-return/if-throw fol…
tdewolff Jun 6, 2020
f0cdd23
JS: minify strings
tdewolff Jun 6, 2020
38190b4
JS: merge var declarations with for/while
tdewolff Jun 6, 2020
f67acdf
JS: keep class/function as expression even after merged with other ex…
tdewolff Jun 6, 2020
f73b92d
JS: use _,$ as variable names; add tests
tdewolff Jun 7, 2020
20bc47c
JS: fix operator precedence, many new tests
tdewolff Jun 18, 2020
714070f
JS: use execution order of && and || to remove groups
tdewolff Jun 18, 2020
2b5064b
JS: update parser, keep single statement of let/const/class in block …
tdewolff Jun 20, 2020
5e42e86
JS: improve var renaming; improve cond exprs
tdewolff Jun 23, 2020
48aee65
cmd: add cpu and mem profile output temporarily
tdewolff Jun 23, 2020
bdfe53a
JS: add tests
tdewolff Jun 28, 2020
37c6bd4
JS: add and fix tests, add octal/binary to number
tdewolff Jun 30, 2020
06575f6
JS: changes to var renaming
tdewolff Jul 2, 2020
2a400d3
JS: update variable renaming
tdewolff Jul 4, 2020
ab2259a
Update benchmarks
tdewolff Jul 4, 2020
ab1ece3
JS: update from parse
tdewolff Jul 4, 2020
53a4e2d
JS: update var renaming and fix expr stmts that are also stmts
tdewolff Jul 13, 2020
9e80a50
JS: Var.Data => Var.Name
tdewolff Jul 13, 2020
b4ea71a
JS: use predeclared byte slices for performance
tdewolff Jul 13, 2020
2e22212
JS: improvements
tdewolff Jul 14, 2020
b94e957
JS: improve PropertyName
tdewolff Jul 14, 2020
3511dda
JS: improvements
tdewolff Jul 15, 2020
75eeaea
JS: improvements
tdewolff Jul 15, 2020
cd9a72b
JS: improvements
tdewolff Jul 15, 2020
f416641
JS: improvements
tdewolff Jul 15, 2020
b63ebb2
JS: improvements
tdewolff Jul 16, 2020
c791c99
JS: keep license comment, start variable declaration hoisting
tdewolff Jul 17, 2020
88389e5
JS: improvements
tdewolff Jul 19, 2020
89a1c12
JS: improvements
tdewolff Jul 19, 2020
61689bf
JS: remove Count
tdewolff Jul 19, 2020
5e5169c
JS: regression
tdewolff Jul 20, 2020
8d88c9d
JS: regression
tdewolff Jul 20, 2020
565ad9d
JS: remove dead code removal
tdewolff Jul 23, 2020
b25a7d4
JS: update
tdewolff Jul 29, 2020
7b5c5e0
JS: remove duplicate Scope for func/method/arrow
tdewolff Jul 29, 2020
f362a14
Explicit GC for benchmarks
tdewolff Jul 30, 2020
3af20d5
JS: use pointers for some AST objects
tdewolff Jul 30, 2020
eb4c1bd
JS: improve parens around new; improve optchain; add spaceBefore for …
tdewolff Jul 30, 2020
d450478
JS: improve test coverage
tdewolff Jul 31, 2020
5a33cf3
JS: fix test
tdewolff Jul 31, 2020
093c2a7
JS: improve var decls
tdewolff Aug 1, 2020
70f476f
JS: fix fuzz bugs
tdewolff Aug 1, 2020
2ef03d2
JS: bugfixes
tdewolff Aug 3, 2020
3f2954b
JS: bugfixes and using Refs in Scopes
tdewolff Aug 4, 2020
6d8ea39
JS: merge definitions with var decl
tdewolff Aug 4, 2020
387c1f8
JS: update tests
tdewolff Aug 8, 2020
ca751f3
JS: add Link to Var
tdewolff Aug 14, 2020
3ad4bb6
JS: optimize Var links
tdewolff Aug 14, 2020
d9676e2
JS: remove AST.Vars, 9% faster
tdewolff Aug 14, 2020
19bd04a
JS: optimize AsIdentifierName/AsDecimalLiteral
tdewolff Aug 14, 2020
c4ffc35
JS: merge var decl with for/while; fix bug for undeclaring var to par…
tdewolff Aug 14, 2020
1af29d0
cmd: remove cpu/mem profile
tdewolff Aug 14, 2020
25700d9
JS: fix lexical scope for for statements
tdewolff Aug 14, 2020
6437a93
JS: fix renaming in for stmt
tdewolff Aug 15, 2020
75a7fd9
JS: refactor
tdewolff Aug 15, 2020
af08bee
JS: use shorter globals array
tdewolff Aug 15, 2020
b1444e7
JS: fix tests
tdewolff Aug 17, 2020
c52ef41
JS: don't rename variables with using with or eval
tdewolff Aug 17, 2020
8ff57be
JS: fix TODO for truthy/falsy of numbers
tdewolff Aug 17, 2020
41dc508
JS: merge concatenating strings
tdewolff Aug 18, 2020
766c997
JS: consider \!expr where expr is truthy/falsy as false/true
tdewolff Aug 18, 2020
612744a
Merge master
tdewolff Aug 18, 2020
8e02634
Fix merge with master
tdewolff Aug 18, 2020
96aff0e
JS: don't track eval()
tdewolff Aug 18, 2020
8340d95
JS: don't merge expressions with for-init/while as they may contain i…
tdewolff Aug 21, 2020
4ac5821
JS: fix string concatenation
tdewolff Aug 21, 2020
9b7b508
JS: don't merge expressions with for-init/while as they may contain i…
tdewolff Aug 21, 2020
c2dbd2e
JS: fix precendence of call expression arguments
tdewolff Aug 21, 2020
7a0a96d
JS: add extra dot after integer in dot expression
tdewolff Aug 21, 2020
8db32e4
JS: fix add extra dot after integer in dot expression
tdewolff Aug 21, 2020
334a4fd
JS: fix string minification
tdewolff Aug 21, 2020
0b1f580
JS: fix string minification
tdewolff Aug 21, 2020
d3464ec
JS: fix flattening of if statement with flow ending
tdewolff Aug 21, 2020
55ea310
JS: fix counting of variable uses when hoisting var decl (not 100% ac…
tdewolff Aug 22, 2020
2bf9997
JS: refactor optimizeStmtList
tdewolff Aug 25, 2020
644cb45
JS: refactor optimizeStmtList
tdewolff Aug 25, 2020
6344a65
JS: fix nested if/else mixing
tdewolff Aug 25, 2020
4d7ace4
JS: fix nested if/else mixing
tdewolff Aug 25, 2020
760650f
JS: improve coverage
tdewolff Aug 25, 2020
6fa4676
JS: fix usage of in keyword in expressions
tdewolff Aug 25, 2020
b6bfc02
cmd: small code refactor
tdewolff Aug 25, 2020
54cf4fc
Update readme
tdewolff Aug 25, 2020
521f53e
Update to tdewolff/parse v2.5.0
tdewolff Aug 26, 2020
f6d2476
Add initial design for sourcemaps
tdewolff Aug 26, 2020
72c2a9e
Remove sourcemap.go
tdewolff Aug 26, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .travis.yml
Expand Up @@ -3,6 +3,7 @@ go:
- 1.13.x
env:
- GO111MODULE=on
- CGO_ENABLED=0
before_install:
- go get github.com/mattn/goveralls
script:
Expand Down
29 changes: 13 additions & 16 deletions README.md
Expand Up @@ -50,19 +50,10 @@ The core functionality associates mimetypes with minification functions, allowin
- [Templates](#templates)
- [License](#license)

### Status

* CSS: **fully implemented**
* HTML: **fully implemented**
* JS: improved JSmin implementation
* JSON: **fully implemented**
* SVG: partially implemented; in development
* XML: **fully implemented**

### Roadmap

- [ ] Use ASM/SSE to further speed-up core parts of the parsers/minifiers
- [ ] Improve JS minifiers by shortening variables and proper semicolon omission
- [x] Improve JS minifiers by shortening variables and proper semicolon omission
- [ ] Speed-up SVG minifier, it is very slow
- [x] Proper parser error reporting and line number + column information
- [ ] Generation of source maps (uncertain, might slow down parsers too much if it cannot run separately nicely)
Expand Down Expand Up @@ -227,19 +218,25 @@ Options:

## JS

The JS minifier is pretty basic. It removes comments, whitespace and line breaks whenever it can. It employs all the rules that [JSMin](http://www.crockford.com/javascript/jsmin.html) does too, but has additional improvements. For example the prefix-postfix bug is fixed.
The JS minifier typically shaves off about 35% -- 65% of filesize depening on the file, which is a compression close to many other minifiers. Common speeds of PHP and JS implementations are about 100-300kB/s (see [Uglify2](http://lisperator.net/uglifyjs/), [Adventures in PHP web asset minimization](https://www.happyassassin.net/2014/12/29/adventures-in-php-web-asset-minimization/)). This implementation is orders of magnitude faster at around ~25MB/s.

Common speeds of PHP and JS implementations are about 100-300kB/s (see [Uglify2](http://lisperator.net/uglifyjs/), [Adventures in PHP web asset minimization](https://www.happyassassin.net/2014/12/29/adventures-in-php-web-asset-minimization/)). This implementation or orders of magnitude faster, around ~80MB/s.
The following features are implemented:

TODO:
- shorten local variables / function parameters names
- precise semicolon and newline omission
- remove superfluous whitespace
- remove superfluous semicolons
- shorten `true`, `false`, and `undefined` to `!0`, `!1` and `void 0`
- rename variables and functions to shorter names (not in global scope)
- collapse if/else statements to expressions
- merge sequential expression statements to one, including into `return` and `throw`
- remove superfluous grouping in expressions
- shorten or remove string escapes
- convert object key or index expression from string to identifier or decimal

## JSON

Minification typically shaves off about 15% of filesize for common indented JSON such as generated by [JSON Generator](http://www.json-generator.com/).

The JSON minifier only removes whitespace, which is the only thing that can be left out.
The JSON minifier only removes whitespace, which is the only thing that can be left out, and minifies numbers (`1000` => `1e3`).

Options:

Expand Down
107 changes: 107 additions & 0 deletions benchmarks/benchmark_test.go
@@ -0,0 +1,107 @@
package benchmarks

import (
"io/ioutil"
"runtime"
"testing"

"github.com/tdewolff/minify/v2/minify"
"github.com/tdewolff/parse/v2"
"github.com/tdewolff/parse/v2/buffer"
)

func benchmark(b *testing.B, mediatype string, sample string) {
m := minify.Default
buf, err := ioutil.ReadFile(sample)
if err != nil {
panic(err)
}
b.Run(sample, func(b *testing.B) {
b.SetBytes(int64(len(buf)))
for i := 0; i < b.N; i++ {
b.StopTimer()
runtime.GC()
r := buffer.NewReader(parse.Copy(buf))
w := buffer.NewWriter(make([]byte, 0, len(buf)))
b.StartTimer()

if err := m.Minify(mediatype, w, r); err != nil {
b.Fatal(err)
}
}
})
}

func BenchmarkCSS(b *testing.B) {
var samples = []string{
"sample_bootstrap.css",
"sample_gumby.css",
"sample_fontawesome.css",
"sample_normalize.css",
}
for _, sample := range samples {
benchmark(b, "text/css", sample)
}
}

func BenchmarkHTML(b *testing.B) {
var samples = []string{
"sample_amazon.html",
"sample_bbc.html",
"sample_blogpost.html",
"sample_es6.html",
"sample_stackoverflow.html",
"sample_wikipedia.html",
}
for _, sample := range samples {
benchmark(b, "text/html", sample)
}
}

func BenchmarkJS(b *testing.B) {
var samples = []string{
"sample_ace.js",
"sample_dot.js",
"sample_jquery.js",
"sample_jqueryui.js",
"sample_moment.js",
}
for _, sample := range samples {
benchmark(b, "application/javascript", sample)
}
}

func BenchmarkJSON(b *testing.B) {
var samples = []string{
"sample_large.json",
"sample_testsuite.json",
"sample_twitter.json",
}
for _, sample := range samples {
benchmark(b, "application/json", sample)
}
}

func BenchmarkSVG(b *testing.B) {
var samples = []string{
"sample_arctic.svg",
"sample_gopher.svg",
"sample_usa.svg",
"sample_car.svg",
"sample_tiger.svg",
}
for _, sample := range samples {
benchmark(b, "image/svg+xml", sample)
}
}

func BenchmarkXML(b *testing.B) {
var samples = []string{
"sample_books.xml",
"sample_catalog.xml",
"sample_omg.xml",
}
for _, sample := range samples {
benchmark(b, "application/xml", sample)
}
}
34 changes: 0 additions & 34 deletions benchmarks/css_test.go

This file was deleted.

36 changes: 0 additions & 36 deletions benchmarks/html_test.go

This file was deleted.

23 changes: 0 additions & 23 deletions benchmarks/jquery_test.go

This file was deleted.

35 changes: 0 additions & 35 deletions benchmarks/js_test.go

This file was deleted.

33 changes: 0 additions & 33 deletions benchmarks/json_test.go

This file was deleted.

18 changes: 0 additions & 18 deletions benchmarks/main_test.go

This file was deleted.

35 changes: 0 additions & 35 deletions benchmarks/svg_test.go

This file was deleted.