Add support for whitespace control character. by evulse · Pull Request #30 · Shopify/liquid-c

evulse · 2016-06-27T18:40:44Z

This matches up with Shopify/liquid#746 to resolve issues Shopify/liquid#216, Shopify/liquid#215, Shopify/liquid#214, Shopify/liquid#194, Shopify/liquid#171, Shopify/liquid#162

This pull request is designed to be standalone and will pass all tests without its counterpart Shopify/liquid#746 however Shopify/liquid#746 requires this pull request and will be updated with this dependancy once merged.

fw42 · 2016-06-27T18:42:55Z

ping @Shopify/liquid

evulse · 2016-06-27T18:51:56Z

This adds support for {{- and {%- syntax which will lstrip! and -}} and -%} which will rstrip!

evulse · 2016-06-27T23:21:13Z

Now matches up to Shopify/liquid#773

tobi · 2016-06-27T23:23:07Z

I'd love to avoid the cycle through the Ruby VM to call rstrip!/lstrip!.

Ideally we should parse white-space as a special token, then modify the token to be marked for render-skipping if {%- or -%} are encountered.

tobi · 2016-06-27T23:24:11Z

I see now that this would make it harder to match with Shopify/liquid#746 - alright, let's not overthink it. Thanks!

evulse · 2016-06-27T23:29:58Z

Yeah I would have liked to avoid it but the trade off isn't worth it at this stage. To keep the functions inline on both sides makes maintenance easier. Also as I'm pulling back the last token if needed it requires conversion from a ruby object then back again where as this approach keeps it clean and only a few lines

pushrax · 2016-06-28T17:42:59Z

ext/liquid_c/block.c

                VALUE str = rb_enc_str_new(token.str, token.length, utf8_encoding);
+
+                if(token.trim_whitespace)
+                   rb_funcall(str, rb_intern("lstrip!"), 0);


Why not find the offset after any whitespace (see lstrip_offset) and create the string with that offset? Avoids a memmove.

I'm now thinking that it could be a good idea to make read_while respect the codepoint length (see other usages of read_while for what it does).

pushrax · 2016-06-28T17:58:31Z

Makes sense and looks pretty good overall.

dylanahsmith · 2016-06-30T15:06:13Z

ext/liquid_c/block.c

+                const char *start = token.str + 2;
+                long length = token.length - 4;
+
+                if (token.str[2] == '-') {


You can use start[0] or *start instead

evulse · 2016-07-07T12:18:06Z

Alright we are passing again. Detection of the whitespace control characters has now been pushed back in to the tokenizer so there is no need to push and pop any existing values.

tobi · 2016-07-07T19:59:27Z

really nice work Mike.

tobi
CEO Shopify

On Thu, Jul 7, 2016 at 8:18 AM, Mike Angell notifications@github.com
wrote:

Alright we are passing again. Detection of the whitespace control
characters has now been pushed back in to the tokenizer so there is no need
to push and pop any existing values.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#30 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAABW7m4s55YijtSESQGVQYhVRGXmrgdks5qTO6AgaJpZM4I_Yjh
.

pushrax · 2016-07-07T20:17:38Z

ext/liquid_c/block.c

+
+                VALUE str = rb_enc_str_new(token_start, end - token_start, utf8_encoding);
+                if(token.rstrip)
+                    rb_funcall(str, intern_rstrip, 0);


What do you think about adding a reverse_read_while if the lstrip part is using read_while anyway?

Yeah I actually was thinking to do exactly that. I really only stuck to this while refactoring so I could confirm i didn't introduce anything new. Should have some time tonight to pull this out and bring it up to where the lstrip sits.

pushrax · 2016-07-07T20:18:21Z

Definitely the better solution, nice.

…o replace current intern_rstrip

evulse · 2016-07-08T10:32:48Z

Ok new change from using ruby rstrip is done which makes this pretty final. As such I've just run the final performance benchmark

Current

Calculating -------------------------------------
          parse:     80.921  (± 3.7%) i/s -      4.851k in  60.028478s
    parse & run:     32.196  (± 3.1%) i/s -      1.932k in  60.069380s

This feature

Calculating -------------------------------------
          parse:     80.791  (± 3.7%) i/s -      4.844k in  60.031186s
    parse & run:     32.109  (± 3.1%) i/s -      1.926k in  60.046058s

The results show the performance impact of this feature is very minimal.

evulse · 2016-07-12T00:59:48Z

@pushrax @tobi @dylanahsmith Where to from here? We would love to be able to start using this

dylanahsmith · 2016-07-12T17:14:12Z

ext/liquid_c/tokenizer.c

            continue;

        char c = *cursor++;
+        char w = *cursor++;


This can read past the end of the string. E.g. tokenizing "{{" results in a token with "{{\u0000". The while (cursor < last) check only allows us to read two characters before having to do another bounds check to see if we can read another byte.

dylanahsmith · 2016-07-12T17:46:27Z

There were some corner cases that you missed as noted above. But overall this is looking very good now. Nice work

evulse · 2016-07-13T00:26:05Z

@dylanahsmith is this what you were thinking?

dylanahsmith · 2016-07-13T04:35:00Z

ext/liquid_c/tokenizer.c

        char c = *cursor++;
+        if (cursor <= last && *cursor == '-') {
+          cursor++;
+          token->rstrip = 1;


If the following if (c != '%' && c != '{') branch is taken, then this change won't be reversed.

… work incorrectly.

evulse · 2016-07-14T01:19:17Z

I think just shifting this removes this issue. So flowing through this previously this would have allowed {.- where . being any character. However i think shifting this only allows {%- and {{- as the tag. So the only flows are

These 3 characters are at the end of a string and become a raw token
These 3 characters are at the start and become a token or variable and run their own loops
This is the last token and as such there will be no other tag following that will trigger the lstrip

evulse · 2016-07-18T22:47:41Z

Is there anything else needed for this to progress?

dylanahsmith · 2016-07-18T23:03:10Z

It looks like https://github.com/Shopify/liquid-c/pull/30/files#r70563502 still hasn't been addressed. That was my last concern.

…bles set these based on their behaviour to raw tags

evulse · 2016-07-20T06:40:27Z

Ok i've added a reset after this line so it will only be set if it is found.

evulse · 2016-08-01T05:08:01Z

Ok this and the ruby version look like they are ready to go. What do we need to do to get this moving.

Add support for whitespace control character.

66f3bf9

This was referenced Jun 27, 2016

Add support for whitespace control character allowing for ERB like trim Shopify/liquid#746

Closed

Add whitespace control character and associated tests Shopify/liquid#773

Merged

pushrax reviewed Jun 28, 2016
View reviewed changes

cache rb_intern, avoid lstrip and edit last node in place.

596cf4c

dylanahsmith reviewed Jun 30, 2016
View reviewed changes

Add new rstrip and lstrip properties to the tokens during tokenizing

cfc2f58

pushrax reviewed Jul 7, 2016
View reviewed changes

Create new read_while that scans from the end of a string. Use this t…

df751f9

…o replace current intern_rstrip

Clean up TOKEN_RAW initial variable char assignment

40f701c

dylanahsmith reviewed Jul 12, 2016
View reviewed changes

New changes to avoid extending past end of cursor

11acad9

dylanahsmith reviewed Jul 13, 2016
View reviewed changes

Shift whitespace detection block as previously this would allow {- to…

d332aa9

… work incorrectly.

reset rstrip after deciding this is not a raw token as tags and varia…

c904976

…bles set these based on their behaviour to raw tags

fw42 merged commit 1fa04f1 into Shopify:master Aug 11, 2016

fw42 mentioned this pull request Aug 11, 2016

Bump LiquidC for whitespace changes Shopify/liquid#786

Merged

cappadona mentioned this pull request Aug 17, 2016

Merge upstream changes to liquid fork? locomotivecms/steam#76

Closed

shopify-shipit bot had a problem deploying to rubygems-4-0-stable January 11, 2023 15:26 Failure

shopify-shipit bot temporarily deployed to rubygems-4-0-stable January 11, 2023 15:29 Inactive

Conversation

evulse commented Jun 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fw42 commented Jun 27, 2016

Uh oh!

evulse commented Jun 27, 2016

Uh oh!

evulse commented Jun 27, 2016

Uh oh!

tobi commented Jun 27, 2016

Uh oh!

tobi commented Jun 27, 2016

Uh oh!

evulse commented Jun 27, 2016

Uh oh!

pushrax Jun 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pushrax Jun 28, 2016

Choose a reason for hiding this comment

Uh oh!

pushrax commented Jun 28, 2016

Uh oh!

dylanahsmith Jun 30, 2016

Choose a reason for hiding this comment

Uh oh!

evulse commented Jul 7, 2016

Uh oh!

tobi commented Jul 7, 2016

Uh oh!

pushrax Jul 7, 2016

Choose a reason for hiding this comment

Uh oh!

evulse Jul 8, 2016

Choose a reason for hiding this comment

Uh oh!

pushrax commented Jul 7, 2016

Uh oh!

evulse commented Jul 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evulse commented Jul 12, 2016

Uh oh!

dylanahsmith Jul 12, 2016

Choose a reason for hiding this comment

Uh oh!

dylanahsmith commented Jul 12, 2016

Uh oh!

evulse commented Jul 13, 2016

Uh oh!

dylanahsmith Jul 13, 2016

Choose a reason for hiding this comment

Uh oh!

evulse commented Jul 14, 2016

Uh oh!

evulse commented Jul 18, 2016

Uh oh!

dylanahsmith commented Jul 18, 2016

Uh oh!

evulse commented Jul 20, 2016

Uh oh!

evulse commented Aug 1, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

evulse commented Jun 27, 2016 •

edited

Loading

pushrax Jun 28, 2016 •

edited

Loading

evulse commented Jul 8, 2016 •

edited

Loading