-
Notifications
You must be signed in to change notification settings - Fork 13
CSSTokenizer #197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
CSSTokenizer #197
Changes from all commits
Commits
Show all changes
68 commits
Select commit
Hold shift + click to select a range
d332205
Kickoff migrating URLs in CSS
adamziel adb07a9
Support Unicode escapes
adamziel 40380e5
Simplify the replacements, format the code
adamziel f6710aa
Improve clarity of the CSSUrlProcessor
adamziel ff59ffd
Test CSS unicode escapes decoder
adamziel 0813667
Ditch regexp
adamziel 3b69730
PHPCS
adamziel 95a1302
Do not allocate memory for every match optimistically
adamziel e98c3ba
Test for data URI
adamziel 3bdbda6
Skip data URIs in the replacement logic
adamziel 8a5e734
Optimize get_parsed_url() for data uris
adamziel 3739a95
Simplify the CSS URL Processor
adamziel 0d5d95f
Move URL parsing from CSS processor to BlockMarkupURLProcessor
adamziel 5feafb5
Use wp.org as a test domain
adamziel c387bd5
Simplify the css processor integration
adamziel 2b2170b
Add a generic CSS Processor
adamziel ee3ed64
Simplify consume_string()
adamziel cd32ab2
Pass most CSS tokenizer test cases
adamziel 4b75739
Less failures
adamziel d3d1b07
1 last failure
adamziel 0245453
Remove the offending fuzzer test
adamziel 8996fd4
Adjust details
adamziel 38f89af
Use codepoints instead of bytes for decoding idents
adamziel 2382057
Use the bundled unicode decoder
adamziel 663db21
Do not concat to repr when consuming numeric values
adamziel c647699
Comments, renaming for clarity
adamziel 8227327
Simplify consume_ident_sequence()
adamziel 2182023
Fix inconsistencies in CSSProcessor
adamziel 20947cb
Simplify is_valid_escape
adamziel 43301e6
Simplify would_next_3_code_points_start_an_ident
adamziel 2c0b357
Simplify is_ident_code_point_at
adamziel 46ce619
Reformat and further simplify
adamziel eeba23f
Reformat and further simplify
adamziel b427f9d
PHPCS
adamziel 0842b67
Remove CSS URL Processor
adamziel 0bbca21
Regenerate tests
adamziel 034bf87
Decide on the ident vs delim test – it seems faulty, I removed it.
adamziel c2e5e08
List specific token values in tests
adamziel 685400c
Simplify the CSS processor and the test suite
adamziel ab1b3c6
multiline CSS sample in test_tokenize_labels_core_tokens
adamziel 2b66102
Clearer tests, document the string normalization tradeoff
adamziel 93daf5a
Rename Processor to Tokenizer
adamziel 04578ab
Remove EOF token
adamziel 7b1f632
Document token_unit
adamziel 0e97f55
Reorder props
adamziel a9a8357
Consolidate decode_escape_at and consume_escape_at
adamziel d565a53
Small cleanup
adamziel 26df610
Lint, simplify, lazy decode tokens
adamziel a6567b5
Consolidate decode_string_or_url and get_normalized_token
adamziel 7791785
Static constructor requiring UTF-8 declaration
adamziel 439a665
Use \u unicode escape syntax
adamziel f8e90c7
Use escape sequences in css-test-cases.php
adamziel 266a0e5
Do not convert CSS numbers to PHP floats
adamziel b2da468
Remove this->token_value_needs_decoding
adamziel 82c32a5
Simplify ident sequence consumption – use indexes, do not optimistica…
adamziel d7ef9b1
PHP 7.2 syntax adjustment
adamziel 5cbf7f2
Brush up the test set
adamziel 2f008a8
code style
adamziel c474de1
PHP 7.2 syntax adjustment
adamziel 394a789
remove the generator script
adamziel 8a72b92
Add attribution
adamziel bc2efc9
Use _wp_scan_utf8
adamziel bf2140a
Adjust tests
adamziel b1e3d77
Test invalid utf-8 sequences
adamziel e06f08d
Test invalid utf-8 sequences
adamziel b1a9283
Brush up tests
adamziel f0c0ec7
phpcs
adamziel 1b7b65b
more usage examples
adamziel File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.