Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tokenizer: update token structure #1326

Merged
merged 4 commits into from Mar 5, 2020
Merged

Tokenizer: update token structure #1326

merged 4 commits into from Mar 5, 2020

Conversation

graberzz
Copy link
Contributor

@graberzz graberzz commented Feb 17, 2020

graberzz added 3 commits Feb 17, 2020
Tokenizer returns [type, content, offset] tokens instead of [type, content, line, column]
Update tokenizer tests according to new token structure
line/column now caclulated using `line-column` package
@graberzz graberzz requested a review from ai Feb 17, 2020
@ai
Copy link
Member

@ai ai commented Feb 17, 2020

What is benchmark differences before and after this PR?

@graberzz
Copy link
Contributor Author

@graberzz graberzz commented Feb 17, 2020

Benchmark's npm test logs from my machine:

Before
[21:08:42] Starting 'default'...
[21:08:42] Starting 'preprocessors'...
[21:08:42] Starting 'bootstrap'...
[21:08:42] Finished 'bootstrap' after 412 μs
[21:08:42] Starting '<anonymous>'...
[21:08:43] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/preprocessors.js]...
[21:08:49]    LibSass x 9.14 ops/sec ±1.23% (48 runs sampled)
[21:08:55]    LibSass sync x 10.31 ops/sec ±0.65% (30 runs sampled)
[21:09:02]    Dart Sass x 5.59 ops/sec ±17.57% (35 runs sampled)
[21:09:08]    Dart Sass sync x 9.88 ops/sec ±5.80% (30 runs sampled)
[21:09:14]    Rework x 20.92 ops/sec ±3.45% (59 runs sampled)
[21:09:20]    PostCSS x 17.31 ops/sec ±8.82% (58 runs sampled)
[21:09:25]    PostCSS dev x 19.82 ops/sec ±3.63% (51 runs sampled)
[21:09:32]    Stylecow x 3.08 ops/sec ±7.17% (20 runs sampled)
[21:09:39]    Less x 2.63 ops/sec ±6.54% (18 runs sampled)
[21:09:45]    Stylis x 33.64 ops/sec ±12.65% (62 runs sampled)
[21:09:45] Fastest test is Stylis at 1.61x faster than Rework

Stylis:         30 ms  (1.9 times faster)
Rework:         48 ms  (1.2 times faster)
PostCSS dev:    50 ms  (1.1 times faster)
PostCSS:        58 ms
LibSass sync:   97 ms  (1.7 times slower)
Dart Sass sync: 101 ms (1.8 times slower)
LibSass:        109 ms (1.9 times slower)
Dart Sass:      179 ms (3.1 times slower)
Stylecow:       325 ms (5.6 times slower)
Less:           381 ms (6.6 times slower)

[21:09:45] Finished '<anonymous>' after 1.05 min
[21:09:45] Finished 'preprocessors' after 1.05 min
[21:09:45] Starting 'parsers'...
[21:09:45] Starting 'bootstrap'...
[21:09:45] Finished 'bootstrap' after 581 μs
[21:09:45] Starting '<anonymous>'...
[21:09:45] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/parsers.js]...
[21:09:51]    Rework x 29.46 ops/sec ±2.09% (54 runs sampled)
[21:09:57]    PostCSS dev x 33.18 ops/sec ±7.85% (71 runs sampled)
[21:10:03]    PostCSS x 34.66 ops/sec ±2.49% (63 runs sampled)
[21:10:09]    PostCSS Full x 11.41 ops/sec ±9.61% (49 runs sampled)
[21:10:14]    CSSOM x 43.17 ops/sec ±9.76% (61 runs sampled)
[21:10:20]    Mensch x 32.66 ops/sec ±8.87% (61 runs sampled)
[21:10:26]    Gonzales x 11.57 ops/sec ±3.70% (33 runs sampled)
[21:10:32]    Gonzales PE x 10.98 ops/sec ±4.82% (32 runs sampled)
[21:10:37]    CSSTree x 180 ops/sec ±0.94% (86 runs sampled)
[21:10:44]    ParserLib x 6.21 ops/sec ±3.20% (20 runs sampled)
[21:10:50]    Stylecow x 20.26 ops/sec ±4.16% (39 runs sampled)
[21:10:55]    Stylis x 65.24 ops/sec ±4.51% (71 runs sampled)
[21:10:55] Fastest test is CSSTree at 2.8x faster than Stylis

CSSTree:      6 ms   (5.2 times faster)
Stylis:       15 ms  (1.9 times faster)
CSSOM:        23 ms  (1.2 times faster)
PostCSS:      29 ms
PostCSS dev:  30 ms  (1.0 times slower)
Mensch:       31 ms  (1.1 times slower)
Rework:       34 ms  (1.2 times slower)
Stylecow:     49 ms  (1.7 times slower)
Gonzales:     86 ms  (3.0 times slower)
PostCSS Full: 88 ms  (3.0 times slower)
Gonzales PE:  91 ms  (3.2 times slower)
ParserLib:    161 ms (5.6 times slower)

[21:10:55] Finished '<anonymous>' after 1.17 min
[21:10:55] Finished 'parsers' after 1.17 min
[21:10:55] Starting 'prefixers'...
[21:10:55] Starting 'bootstrap'...
[21:10:55] Finished 'bootstrap' after 436 μs
[21:10:55] Starting '<anonymous>'...
[21:10:56] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/prefixers.js]...
[21:11:02]    Autoprefixer x 19.22 ops/sec ±8.06% (54 runs sampled)
[21:11:09]    Stylecow x 2.89 ops/sec ±5.23% (19 runs sampled)
[21:11:15]    Stylis x 44.16 ops/sec ±10.03% (76 runs sampled)
[21:11:15] Fastest test is Stylis at 2.3x faster than Autoprefixer

Stylis:       23 ms  (2.3 times faster)
Autoprefixer: 52 ms
Stylecow:     346 ms (6.6 times slower)

[21:11:15] Finished '<anonymous>' after 20 s
[21:11:15] Finished 'prefixers' after 20 s
[21:11:15] Starting 'tokenizers'...
[21:11:15] Starting 'bootstrap'...
[21:11:15] Finished 'bootstrap' after 170 μs
[21:11:15] Starting '<anonymous>'...
[21:11:15] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/tokenizers.js]...
[21:11:21]    PostCSS x 104 ops/sec ±3.30% (70 runs sampled)
[21:11:26]    PostCSS dev x 106 ops/sec ±2.94% (67 runs sampled)
[21:11:26] Fastest tests are PostCSS dev,PostCSS

PostCSS dev: 9 ms  (1.0 times faster)
PostCSS:     10 ms

[21:11:26] Finished '<anonymous>' after 12 s
[21:11:26] Finished 'tokenizers' after 12 s
[21:11:26] Finished 'default' after 2.73 min
After
[21:04:59] Starting 'default'...
[21:04:59] Starting 'preprocessors'...
[21:04:59] Starting 'bootstrap'...
[21:04:59] Finished 'bootstrap' after 408 μs
[21:04:59] Starting '<anonymous>'...
[21:05:00] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/preprocessors.js]...
[21:05:06]    LibSass x 9.22 ops/sec ±0.82% (48 runs sampled)
[21:05:12]    LibSass sync x 10.49 ops/sec ±0.50% (31 runs sampled)
[21:05:19]    Dart Sass x 5.79 ops/sec ±17.06% (36 runs sampled)
[21:05:25]    Dart Sass sync x 10.52 ops/sec ±3.46% (31 runs sampled)
[21:05:31]    Rework x 21.31 ops/sec ±3.30% (57 runs sampled)
[21:05:36]    PostCSS x 17.06 ops/sec ±11.02% (52 runs sampled)
[21:05:42]    PostCSS dev x 18.53 ops/sec ±2.53% (64 runs sampled)
[21:05:49]    Stylecow x 3.07 ops/sec ±7.42% (20 runs sampled)
[21:05:56]    Less x 2.56 ops/sec ±6.43% (18 runs sampled)
[21:06:02]    Stylis x 33.50 ops/sec ±12.27% (64 runs sampled)
[21:06:02] Fastest test is Stylis at 1.57x faster than Rework

Stylis:         30 ms  (2.0 times faster)
Rework:         47 ms  (1.2 times faster)
PostCSS dev:    54 ms  (1.1 times faster)
PostCSS:        59 ms
Dart Sass sync: 95 ms  (1.6 times slower)
LibSass sync:   95 ms  (1.6 times slower)
LibSass:        108 ms (1.8 times slower)
Dart Sass:      173 ms (2.9 times slower)
Stylecow:       326 ms (5.6 times slower)
Less:           390 ms (6.7 times slower)

[21:06:02] Finished '<anonymous>' after 1.03 min
[21:06:02] Finished 'preprocessors' after 1.03 min
[21:06:02] Starting 'parsers'...
[21:06:02] Starting 'bootstrap'...
[21:06:02] Finished 'bootstrap' after 565 μs
[21:06:02] Starting '<anonymous>'...
[21:06:02] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/parsers.js]...
[21:06:08]    Rework x 31.22 ops/sec ±2.12% (57 runs sampled)
[21:06:14]    PostCSS dev x 31.33 ops/sec ±7.69% (64 runs sampled)
[21:06:20]    PostCSS x 32.29 ops/sec ±1.72% (77 runs sampled)
[21:06:26]    PostCSS Full x 11.15 ops/sec ±9.67% (49 runs sampled)
[21:06:31]    CSSOM x 45.47 ops/sec ±0.52% (61 runs sampled)
[21:06:37]    Mensch x 29.93 ops/sec ±10.35% (57 runs sampled)
[21:06:43]    Gonzales x 11.41 ops/sec ±4.19% (33 runs sampled)
[21:06:49]    Gonzales PE x 10.53 ops/sec ±8.67% (33 runs sampled)
[21:06:55]    CSSTree x 197 ops/sec ±1.02% (85 runs sampled)
[21:07:01]    ParserLib x 6.11 ops/sec ±3.64% (20 runs sampled)
[21:07:07]    Stylecow x 21.32 ops/sec ±4.99% (41 runs sampled)
[21:07:13]    Stylis x 65.20 ops/sec ±4.33% (71 runs sampled)
[21:07:13] Fastest test is CSSTree at 3.0x faster than Stylis

CSSTree:      5 ms   (6.1 times faster)
Stylis:       15 ms  (2.0 times faster)
CSSOM:        22 ms  (1.4 times faster)
PostCSS:      31 ms
PostCSS dev:  32 ms  (1.0 times slower)
Rework:       32 ms  (1.0 times slower)
Mensch:       33 ms  (1.1 times slower)
Stylecow:     47 ms  (1.5 times slower)
Gonzales:     88 ms  (2.8 times slower)
PostCSS Full: 90 ms  (2.9 times slower)
Gonzales PE:  95 ms  (3.1 times slower)
ParserLib:    164 ms (5.3 times slower)

[21:07:13] Finished '<anonymous>' after 1.17 min
[21:07:13] Finished 'parsers' after 1.17 min
[21:07:13] Starting 'prefixers'...
[21:07:13] Starting 'bootstrap'...
[21:07:13] Finished 'bootstrap' after 384 μs
[21:07:13] Starting '<anonymous>'...
[21:07:14] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/prefixers.js]...
[21:07:20]    Autoprefixer x 18.00 ops/sec ±7.42% (61 runs sampled)
[21:07:27]    Stylecow x 2.84 ops/sec ±4.86% (19 runs sampled)
[21:07:33]    Stylis x 44.26 ops/sec ±10.70% (70 runs sampled)
[21:07:33] Fastest test is Stylis at 2.5x faster than Autoprefixer

Stylis:       23 ms  (2.5 times faster)
Autoprefixer: 56 ms
Stylecow:     353 ms (6.3 times slower)

[21:07:33] Finished '<anonymous>' after 20 s
[21:07:33] Finished 'prefixers' after 20 s
[21:07:33] Starting 'tokenizers'...
[21:07:33] Starting 'bootstrap'...
[21:07:33] Finished 'bootstrap' after 142 μs
[21:07:33] Starting '<anonymous>'...
[21:07:33] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/tokenizers.js]...
[21:07:38]    PostCSS x 130 ops/sec ±4.21% (64 runs sampled)
[21:07:44]    PostCSS dev x 120 ops/sec ±4.18% (66 runs sampled)
[21:07:44] Fastest test is PostCSS at 1.09x faster than PostCSS dev

PostCSS:     8 ms
PostCSS dev: 8 ms (1.1 times slower)

[21:07:44] Finished '<anonymous>' after 12 s
[21:07:44] Finished 'tokenizers' after 12 s
[21:07:44] Finished 'default' after 2.73 min

@ai
Copy link
Member

@ai ai commented Feb 18, 2020

Can you trim offset from tests by regexp? We will add it letter.

@graberzz
Copy link
Contributor Author

@graberzz graberzz commented Feb 18, 2020

You mean create a PR with updated tests here https://github.com/postcss/postcss-parser-tests ?

@ai
Copy link
Member

@ai ai commented Feb 18, 2020

I will release it when I will release PoatCSS 8.0.

(I marked you as an author of Cult of Martians task.)

@ai ai merged commit 7d1c9df into postcss:ose Mar 5, 2020
1 check was pending
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants