List of CSS Tokenizers

[romainmenke]

This is a list of CSS Tokenizers.
This issue is not intended for in depth discussion about any individual tokenizer or any aspect of CSS tokenizing.

• csslex
• parse-css
• csstree
• @csstools/css-tokenizer
• @csstools/tokenizer
• @adguard/css-tokenizer

[scripthunter7]

Hi @romainmenke, thank you for this compilation! We also created a tokenizer for CSS & Extended CSS:
https://www.npmjs.com/package/@adguard/css-tokenizer

A detailed benchmark has also been prepared for the tokenizers listed here. Results can be found at the following link:
https://github.com/AdguardTeam/tsurlfilter/blob/master/packages/css-tokenizer/benchmark/RESULTS.md

Note

Our API and token list is also compatible with the CSSTree's tokenizer API, and in the long term, we plan to integrate the tokenizer library into CSSTree via our ECSSTree library. See this issue for more details.

[romainmenke]

Hi @scripthunter7,

Interesting, added to the list!
How does it score on these tests : https://github.com/romainmenke/css-tokenizer-tests

[romainmenke]

test results for @adguard/css-tokenizer https://www.romainmenke.com/css-tokenizer-tests

• perfect score for widely used CSS in the community (bootstrap, tailwind, ...)
• a few issues in more exotic CSS
• it's faster than any other tokenizer currently tested

[Mouvedia]

Rome used to have a tokenizer: rome/tools#707