Tokenizer: update token structure

[graberzz]

https://cultofmartians.com/tasks/postcss-offset.html#task

#1325

As I can see those changes also require test fixtures and documentation update.

[ai]

What is benchmark differences before and after this PR?

[graberzz]

Benchmark's npm test logs from my machine:

Before

[21:08:42] Starting 'default'...
[21:08:42] Starting 'preprocessors'...
[21:08:42] Starting 'bootstrap'...
[21:08:42] Finished 'bootstrap' after 412 μs
[21:08:42] Starting '<anonymous>'...
[21:08:43] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/preprocessors.js]...
[21:08:49]    LibSass x 9.14 ops/sec ±1.23% (48 runs sampled)
[21:08:55]    LibSass sync x 10.31 ops/sec ±0.65% (30 runs sampled)
[21:09:02]    Dart Sass x 5.59 ops/sec ±17.57% (35 runs sampled)
[21:09:08]    Dart Sass sync x 9.88 ops/sec ±5.80% (30 runs sampled)
[21:09:14]    Rework x 20.92 ops/sec ±3.45% (59 runs sampled)
[21:09:20]    PostCSS x 17.31 ops/sec ±8.82% (58 runs sampled)
[21:09:25]    PostCSS dev x 19.82 ops/sec ±3.63% (51 runs sampled)
[21:09:32]    Stylecow x 3.08 ops/sec ±7.17% (20 runs sampled)
[21:09:39]    Less x 2.63 ops/sec ±6.54% (18 runs sampled)
[21:09:45]    Stylis x 33.64 ops/sec ±12.65% (62 runs sampled)
[21:09:45] Fastest test is Stylis at 1.61x faster than Rework

Stylis:         30 ms  (1.9 times faster)
Rework:         48 ms  (1.2 times faster)
PostCSS dev:    50 ms  (1.1 times faster)
PostCSS:        58 ms
LibSass sync:   97 ms  (1.7 times slower)
Dart Sass sync: 101 ms (1.8 times slower)
LibSass:        109 ms (1.9 times slower)
Dart Sass:      179 ms (3.1 times slower)
Stylecow:       325 ms (5.6 times slower)
Less:           381 ms (6.6 times slower)

[21:09:45] Finished '<anonymous>' after 1.05 min
[21:09:45] Finished 'preprocessors' after 1.05 min
[21:09:45] Starting 'parsers'...
[21:09:45] Starting 'bootstrap'...
[21:09:45] Finished 'bootstrap' after 581 μs
[21:09:45] Starting '<anonymous>'...
[21:09:45] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/parsers.js]...
[21:09:51]    Rework x 29.46 ops/sec ±2.09% (54 runs sampled)
[21:09:57]    PostCSS dev x 33.18 ops/sec ±7.85% (71 runs sampled)
[21:10:03]    PostCSS x 34.66 ops/sec ±2.49% (63 runs sampled)
[21:10:09]    PostCSS Full x 11.41 ops/sec ±9.61% (49 runs sampled)
[21:10:14]    CSSOM x 43.17 ops/sec ±9.76% (61 runs sampled)
[21:10:20]    Mensch x 32.66 ops/sec ±8.87% (61 runs sampled)
[21:10:26]    Gonzales x 11.57 ops/sec ±3.70% (33 runs sampled)
[21:10:32]    Gonzales PE x 10.98 ops/sec ±4.82% (32 runs sampled)
[21:10:37]    CSSTree x 180 ops/sec ±0.94% (86 runs sampled)
[21:10:44]    ParserLib x 6.21 ops/sec ±3.20% (20 runs sampled)
[21:10:50]    Stylecow x 20.26 ops/sec ±4.16% (39 runs sampled)
[21:10:55]    Stylis x 65.24 ops/sec ±4.51% (71 runs sampled)
[21:10:55] Fastest test is CSSTree at 2.8x faster than Stylis

CSSTree:      6 ms   (5.2 times faster)
Stylis:       15 ms  (1.9 times faster)
CSSOM:        23 ms  (1.2 times faster)
PostCSS:      29 ms
PostCSS dev:  30 ms  (1.0 times slower)
Mensch:       31 ms  (1.1 times slower)
Rework:       34 ms  (1.2 times slower)
Stylecow:     49 ms  (1.7 times slower)
Gonzales:     86 ms  (3.0 times slower)
PostCSS Full: 88 ms  (3.0 times slower)
Gonzales PE:  91 ms  (3.2 times slower)
ParserLib:    161 ms (5.6 times slower)

[21:10:55] Finished '<anonymous>' after 1.17 min
[21:10:55] Finished 'parsers' after 1.17 min
[21:10:55] Starting 'prefixers'...
[21:10:55] Starting 'bootstrap'...
[21:10:55] Finished 'bootstrap' after 436 μs
[21:10:55] Starting '<anonymous>'...
[21:10:56] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/prefixers.js]...
[21:11:02]    Autoprefixer x 19.22 ops/sec ±8.06% (54 runs sampled)
[21:11:09]    Stylecow x 2.89 ops/sec ±5.23% (19 runs sampled)
[21:11:15]    Stylis x 44.16 ops/sec ±10.03% (76 runs sampled)
[21:11:15] Fastest test is Stylis at 2.3x faster than Autoprefixer

Stylis:       23 ms  (2.3 times faster)
Autoprefixer: 52 ms
Stylecow:     346 ms (6.6 times slower)

[21:11:15] Finished '<anonymous>' after 20 s
[21:11:15] Finished 'prefixers' after 20 s
[21:11:15] Starting 'tokenizers'...
[21:11:15] Starting 'bootstrap'...
[21:11:15] Finished 'bootstrap' after 170 μs
[21:11:15] Starting '<anonymous>'...
[21:11:15] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/tokenizers.js]...
[21:11:21]    PostCSS x 104 ops/sec ±3.30% (70 runs sampled)
[21:11:26]    PostCSS dev x 106 ops/sec ±2.94% (67 runs sampled)
[21:11:26] Fastest tests are PostCSS dev,PostCSS

PostCSS dev: 9 ms  (1.0 times faster)
PostCSS:     10 ms

[21:11:26] Finished '<anonymous>' after 12 s
[21:11:26] Finished 'tokenizers' after 12 s
[21:11:26] Finished 'default' after 2.73 min

After

[21:04:59] Starting 'default'...
[21:04:59] Starting 'preprocessors'...
[21:04:59] Starting 'bootstrap'...
[21:04:59] Finished 'bootstrap' after 408 μs
[21:04:59] Starting '<anonymous>'...
[21:05:00] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/preprocessors.js]...
[21:05:06]    LibSass x 9.22 ops/sec ±0.82% (48 runs sampled)
[21:05:12]    LibSass sync x 10.49 ops/sec ±0.50% (31 runs sampled)
[21:05:19]    Dart Sass x 5.79 ops/sec ±17.06% (36 runs sampled)
[21:05:25]    Dart Sass sync x 10.52 ops/sec ±3.46% (31 runs sampled)
[21:05:31]    Rework x 21.31 ops/sec ±3.30% (57 runs sampled)
[21:05:36]    PostCSS x 17.06 ops/sec ±11.02% (52 runs sampled)
[21:05:42]    PostCSS dev x 18.53 ops/sec ±2.53% (64 runs sampled)
[21:05:49]    Stylecow x 3.07 ops/sec ±7.42% (20 runs sampled)
[21:05:56]    Less x 2.56 ops/sec ±6.43% (18 runs sampled)
[21:06:02]    Stylis x 33.50 ops/sec ±12.27% (64 runs sampled)
[21:06:02] Fastest test is Stylis at 1.57x faster than Rework

Stylis:         30 ms  (2.0 times faster)
Rework:         47 ms  (1.2 times faster)
PostCSS dev:    54 ms  (1.1 times faster)
PostCSS:        59 ms
Dart Sass sync: 95 ms  (1.6 times slower)
LibSass sync:   95 ms  (1.6 times slower)
LibSass:        108 ms (1.8 times slower)
Dart Sass:      173 ms (2.9 times slower)
Stylecow:       326 ms (5.6 times slower)
Less:           390 ms (6.7 times slower)

[21:06:02] Finished '<anonymous>' after 1.03 min
[21:06:02] Finished 'preprocessors' after 1.03 min
[21:06:02] Starting 'parsers'...
[21:06:02] Starting 'bootstrap'...
[21:06:02] Finished 'bootstrap' after 565 μs
[21:06:02] Starting '<anonymous>'...
[21:06:02] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/parsers.js]...
[21:06:08]    Rework x 31.22 ops/sec ±2.12% (57 runs sampled)
[21:06:14]    PostCSS dev x 31.33 ops/sec ±7.69% (64 runs sampled)
[21:06:20]    PostCSS x 32.29 ops/sec ±1.72% (77 runs sampled)
[21:06:26]    PostCSS Full x 11.15 ops/sec ±9.67% (49 runs sampled)
[21:06:31]    CSSOM x 45.47 ops/sec ±0.52% (61 runs sampled)
[21:06:37]    Mensch x 29.93 ops/sec ±10.35% (57 runs sampled)
[21:06:43]    Gonzales x 11.41 ops/sec ±4.19% (33 runs sampled)
[21:06:49]    Gonzales PE x 10.53 ops/sec ±8.67% (33 runs sampled)
[21:06:55]    CSSTree x 197 ops/sec ±1.02% (85 runs sampled)
[21:07:01]    ParserLib x 6.11 ops/sec ±3.64% (20 runs sampled)
[21:07:07]    Stylecow x 21.32 ops/sec ±4.99% (41 runs sampled)
[21:07:13]    Stylis x 65.20 ops/sec ±4.33% (71 runs sampled)
[21:07:13] Fastest test is CSSTree at 3.0x faster than Stylis

CSSTree:      5 ms   (6.1 times faster)
Stylis:       15 ms  (2.0 times faster)
CSSOM:        22 ms  (1.4 times faster)
PostCSS:      31 ms
PostCSS dev:  32 ms  (1.0 times slower)
Rework:       32 ms  (1.0 times slower)
Mensch:       33 ms  (1.1 times slower)
Stylecow:     47 ms  (1.5 times slower)
Gonzales:     88 ms  (2.8 times slower)
PostCSS Full: 90 ms  (2.9 times slower)
Gonzales PE:  95 ms  (3.1 times slower)
ParserLib:    164 ms (5.3 times slower)

[21:07:13] Finished '<anonymous>' after 1.17 min
[21:07:13] Finished 'parsers' after 1.17 min
[21:07:13] Starting 'prefixers'...
[21:07:13] Starting 'bootstrap'...
[21:07:13] Finished 'bootstrap' after 384 μs
[21:07:13] Starting '<anonymous>'...
[21:07:14] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/prefixers.js]...
[21:07:20]    Autoprefixer x 18.00 ops/sec ±7.42% (61 runs sampled)
[21:07:27]    Stylecow x 2.84 ops/sec ±4.86% (19 runs sampled)
[21:07:33]    Stylis x 44.26 ops/sec ±10.70% (70 runs sampled)
[21:07:33] Fastest test is Stylis at 2.5x faster than Autoprefixer

Stylis:       23 ms  (2.5 times faster)
Autoprefixer: 56 ms
Stylecow:     353 ms (6.3 times slower)

[21:07:33] Finished '<anonymous>' after 20 s
[21:07:33] Finished 'prefixers' after 20 s
[21:07:33] Starting 'tokenizers'...
[21:07:33] Starting 'bootstrap'...
[21:07:33] Finished 'bootstrap' after 142 μs
[21:07:33] Starting '<anonymous>'...
[21:07:33] Running suite Bootstrap [/Users/ayaz/Documents/benchmark/tokenizers.js]...
[21:07:38]    PostCSS x 130 ops/sec ±4.21% (64 runs sampled)
[21:07:44]    PostCSS dev x 120 ops/sec ±4.18% (66 runs sampled)
[21:07:44] Fastest test is PostCSS at 1.09x faster than PostCSS dev

PostCSS:     8 ms
PostCSS dev: 8 ms (1.1 times slower)

[21:07:44] Finished '<anonymous>' after 12 s
[21:07:44] Finished 'tokenizers' after 12 s
[21:07:44] Finished 'default' after 2.73 min

[ai]

Can you trim offset from tests by regexp? We will add it letter.

[graberzz]

You mean create a PR with updated tests here https://github.com/postcss/postcss-parser-tests ?

[ai]

I will release it when I will release PoatCSS 8.0.

(I marked you as an author of Cult of Martians task.)