Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite Lexer & Parser #1

Merged
merged 14 commits into from
Feb 14, 2024
Merged

Rewrite Lexer & Parser #1

merged 14 commits into from
Feb 14, 2024

Conversation

keithamus
Copy link
Owner

@keithamus keithamus commented Feb 4, 2024 •
edited

This PR rewrites the lexer to:

  • Half the size of tokens from 32B to 16B.
  • Skip whitespace & comments in the lexer, not the parser.
  • Simplify some tokens (e.g a Percentage token is not needed).
  • Move span data from the lexer to the parser.

It also rewrites the parser to:

  • Move all individual parsing logic to hdx_ast, colocating it with the AST nodes
  • Make use of more macros & Derives to massively DRY up parsing logic

The aim is to:

  • Speed up overall parsing
  • Make it easier to develop
  • Make the whole thing more robust.

@keithamus keithamus changed the title Rewrite lexer Rewrite Lexer & Parser Feb 11, 2024
@keithamus keithamus force-pushed the rewrite-lexer branch 3 times, most recently from b3e2809 to 228d365 Compare February 11, 2024 21:44
@keithamus
Copy link
Owner Author

keithamus commented Feb 11, 2024 •
edited

Lexer benchmark comparison:

Old

lexer/960
                        time:   [53.217 µs 53.278 µs 53.341 µs]
                        thrpt:  [178.63 MiB/s 178.84 MiB/s 179.04 MiB/s]
lexer/animate.4.1.1
                        time:   [772.47 µs 773.50 µs 774.62 µs]
                        thrpt:  [117.48 MiB/s 117.65 MiB/s 117.81 MiB/s]
lexer/blueprint.1.0.1
                        time:   [108.56 µs 108.78 µs 109.05 µs]
                        thrpt:  [152.35 MiB/s 152.74 MiB/s 153.05 MiB/s]
lexer/bootstrap.4.6.2
                        time:   [1.3628 ms 1.3638 ms 1.3649 ms]
                        thrpt:  [140.01 MiB/s 140.12 MiB/s 140.23 MiB/s]
lexer/bootstrap.5.3.0
                        time:   [1.7953 ms 1.7989 ms 1.8035 ms]
                        thrpt:  [148.49 MiB/s 148.87 MiB/s 149.17 MiB/s]
lexer/font-awesome-all.5.15.4
                        time:   [463.86 µs 464.59 µs 465.58 µs]
                        thrpt:  [150.71 MiB/s 151.03 MiB/s 151.27 MiB/s]
lexer/foundation.6.7.5
                        time:   [1.3646 ms 1.3660 ms 1.3675 ms]
                        thrpt:  [138.61 MiB/s 138.76 MiB/s 138.90 MiB/s]
lexer/foundation.6.7.5.min
                        time:   [1.0481 ms 1.0490 ms 1.0499 ms]
                        thrpt:  [122.46 MiB/s 122.57 MiB/s 122.68 MiB/s]
lexer/inuitcss.6.0.0
                        time:   [189.41 µs 189.63 µs 189.86 µs]
                        thrpt:  [266.88 MiB/s 267.21 MiB/s 267.53 MiB/s]
lexer/mini.css.3.0.1
                        time:   [361.28 µs 361.82 µs 362.55 µs]
                        thrpt:  [145.40 MiB/s 145.69 MiB/s 145.91 MiB/s]
lexer/open-props.1.5.10.min
                        time:   [174.38 µs 174.62 µs 174.92 µs]
                        thrpt:  [107.06 MiB/s 107.25 MiB/s 107.40 MiB/s]
lexer/pure.2.0.3
                        time:   [149.10 µs 149.36 µs 149.65 µs]
                        thrpt:  [183.81 MiB/s 184.16 MiB/s 184.48 MiB/s]
lexer/reset.2.0
                        time:   [6.9250 µs 6.9410 µs 6.9608 µs]
                        thrpt:  [149.75 MiB/s 150.18 MiB/s 150.52 MiB/s]
lexer/tailwind.2.2.19.min
                        time:   [23.446 ms 23.477 ms 23.515 ms]
                        thrpt:  [118.99 MiB/s 119.19 MiB/s 119.34 MiB/s]

New

lexer/960 
                       time:   [46.043 µs 46.146 µs 46.276 µs]
                        thrpt:  [205.90 MiB/s 206.48 MiB/s 206.94 MiB/s]
lexer/animate.4.1.1
                        time:   [637.93 µs 639.03 µs 640.15 µs]
                        thrpt:  [142.16 MiB/s 142.41 MiB/s 142.66 MiB/s]
lexer/blueprint.1.0.1
                        time:   [92.470 µs 92.547 µs 92.635 µs]                                                                                                                            
                        thrpt:  [179.36 MiB/s 179.53 MiB/s 179.68 MiB/s]
lexer/bootstrap.4.6.2
                        time:   [1.2340 ms 1.2352 ms 1.2368 ms]
                        thrpt:  [154.51 MiB/s 154.71 MiB/s 154.87 MiB/s]
lexer/bootstrap.5.3.0
                        time:   [1.6490 ms 1.6531 ms 1.6578 ms]
                        thrpt:  [161.54 MiB/s 162.00 MiB/s 162.40 MiB/s]
lexer/font-awesome-all.5.15.4
                        time:   [428.82 µs 429.36 µs 429.98 µs]
                        thrpt:  [163.19 MiB/s 163.43 MiB/s 163.63 MiB/s]
lexer/foundation.6.7.5
                        time:   [1.2253 ms 1.2265 ms 1.2278 ms]
                        thrpt:  [154.37 MiB/s 154.54 MiB/s 154.70 MiB/s]
lexer/foundation.6.7.5.min
                        time:   [982.61 µs 983.17 µs 983.87 µs]
                        thrpt:  [130.68 MiB/s 130.78 MiB/s 130.85 MiB/s]
lexer/inuitcss.6.0.0
                        time:   [175.43 µs 175.64 µs 175.87 µs]
                        thrpt:  [288.12 MiB/s 288.49 MiB/s 288.85 MiB/s]
lexer/mini.css.3.0.1
                        time:   [333.29 µs 333.56 µs 333.85 µs]
                        thrpt:  [157.90 MiB/s 158.04 MiB/s 158.16 MiB/s]
lexer/open-props.1.5.10.min
                        time:   [153.09 µs 153.28 µs 153.56 µs]
                        thrpt:  [121.96 MiB/s 122.17 MiB/s 122.33 MiB/s]
lexer/pure.2.0.3
                        time:   [137.57 µs 137.69 µs 137.81 µs]
                        thrpt:  [199.60 MiB/s 199.78 MiB/s 199.95 MiB/s]
lexer/reset.2.0
                        time:   [6.2455 µs 6.2580 µs 6.2724 µs]
                        thrpt:  [166.18 MiB/s 166.57 MiB/s 166.90 MiB/s]
lexer/tailwind.2.2.19.min
                        time:   [21.954 ms 21.977 ms 22.007 ms]
                        thrpt:  [127.15 MiB/s 127.32 MiB/s 127.45 MiB/s]

This is a decent speed up in the lexer.

@keithamus
Copy link
Owner Author

Parser:

Old

parser/960
                        time:   [238.14 µs 239.17 µs 239.98 µs]
                        thrpt:  [39.704 MiB/s 39.838 MiB/s 40.012 MiB/s]
parser/animate.4.1.1
                        time:   [1.8714 ms 1.8779 ms 1.8832 ms]
                        thrpt:  [48.324 MiB/s 48.462 MiB/s 48.629 MiB/s]
parser/blueprint.1.0.1
                        time:   [376.84 µs 378.47 µs 380.12 µs]
                        thrpt:  [43.710 MiB/s 43.900 MiB/s 44.090 MiB/s]
parser/bootstrap.4.6.2  time:   [3.4237 ms 3.4373 ms 3.4490 ms]
                        thrpt:  [55.408 MiB/s 55.596 MiB/s 55.818 MiB/s]
parser/bootstrap.5.3.0
                        time:   [5.6988 ms 5.7163 ms 5.7347 ms]
                        thrpt:  [46.699 MiB/s 46.849 MiB/s 46.993 MiB/s]
parser/bootstrap.5.3.0.min
                        time:   [5.4250 ms 5.4368 ms 5.4512 ms]
                        thrpt:  [40.747 MiB/s 40.856 MiB/s 40.945 MiB/s]
parser/font-awesome-all.5.15.4
                        time:   [1.2422 ms 1.2471 ms 1.2509 ms]
                        thrpt:  [56.096 MiB/s 56.267 MiB/s 56.489 MiB/s]
parser/foundation.6.7.5
                        time:   [3.2582 ms 3.2672 ms 3.2745 ms]
                        thrpt:  [57.886 MiB/s 58.016 MiB/s 58.176 MiB/s]
parser/foundation.6.7.5.min
                        time:   [2.7389 ms 2.7446 ms 2.7490 ms]
                        thrpt:  [46.771 MiB/s 46.847 MiB/s 46.944 MiB/s]
parser/inuitcss.6.0.0
                        time:   [432.22 µs 433.98 µs 435.35 µs]
                        thrpt:  [116.39 MiB/s 116.76 MiB/s 117.24 MiB/s]

New

parser/960
                        time:   [235.08 µs 236.44 µs 237.53 µs]
                        thrpt:  [40.113 MiB/s 40.299 MiB/s 40.532 MiB/s]
parser/animate.4.1.1
                        time:   [4.1180 µs 4.1238 µs 4.1307 µs]
                        thrpt:  [21.515 GiB/s 21.551 GiB/s 21.581 GiB/s]
parser/blueprint.1.0.1
                        time:   [472.47 µs 475.41 µs 477.69 µs]
                        thrpt:  [34.781 MiB/s 34.949 MiB/s 35.166 MiB/s]
parser/bootstrap.4.6.2
                        time:   [3.3183 ms 3.3243 ms 3.3305 ms]
                        thrpt:  [57.380 MiB/s 57.487 MiB/s 57.591 MiB/s]
parser/bootstrap.5.3.0
                        time:   [4.1224 µs 4.1316 µs 4.1430 µs]
                        thrpt:  [63.125 GiB/s 63.299 GiB/s 63.440 GiB/s]
parser/font-awesome-all.5.15.4
                        time:   [1.2166 ms 1.2312 ms 1.2448 ms]
                        thrpt:  [56.369 MiB/s 56.992 MiB/s 57.677 MiB/s]
parser/foundation.6.7.5.min
                        time:   [4.1225 µs 4.1281 µs 4.1348 µs]
                        thrpt:  [30.367 GiB/s 30.416 GiB/s 30.457 GiB/s]
parser/inuitcss.6.0.0 
                        time:   [4.1154 µs 4.1244 µs 4.1364 µs]
                        thrpt:  [11.963 GiB/s 11.998 GiB/s 12.024 GiB/s]
parser/mini.css.3.0.1
                        time:   [4.1268 µs 4.1334 µs 4.1412 µs]
                        thrpt:  [12.431 GiB/s 12.455 GiB/s 12.474 GiB/s]
parser/open-props.1.5.10.min
                        time:   [341.44 µs 342.10 µs 342.84 µs]
                        thrpt:  [54.623 MiB/s 54.741 MiB/s 54.848 MiB/s]
parser/pure.2.0.3
                        time:   [365.75 µs 366.51 µs 367.11 µs]
                        thrpt:  [74.929 MiB/s 75.050 MiB/s 75.206 MiB/s]
parser/tailwind.2.2.19.min
                        time:   [60.206 ms 60.488 ms 60.805 ms]
                        thrpt:  [46.018 MiB/s 46.259 MiB/s 46.476 MiB/s]

I think these GiB/s readings are failing fast and swallowing the rest of the file, but the speedup from lexing to parsing is far more modest than the lexer gains.

@keithamus keithamus marked this pull request as ready for review February 14, 2024 00:16
@keithamus keithamus merged commit 85ad752 into main Feb 14, 2024
2 of 3 checks passed
@keithamus keithamus deleted the rewrite-lexer branch February 14, 2024 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@keithamus