Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Liquid scanner #123

Open
wants to merge 27 commits into
from

Conversation

Projects
None yet
2 participants

Asherlc commented Apr 4, 2013

This is a scanner for the Liquid templating language: https://github.com/Shopify/liquid

@Asherlc Asherlc referenced this pull request in rubychan/coderay-scanner-tests Apr 4, 2013

Open

Liquid scanner tests #3

It's not a good idea to use the CSV scanner for scanning. CodeRay should never remove or change whitespace from the input.

@korny korny commented on an outdated diff Apr 7, 2013

lib/coderay/helpers/file_type.rb
@@ -1,5 +1,5 @@
module CodeRay
-
+
@korny

korny Apr 7, 2013

Owner

Can you please remove these whitespace changes? :-)

@korny korny commented on an outdated diff Apr 7, 2013

lib/coderay/scanners/liquid.rb
+ FILTER_WITH_VALUE_KEYWORDS = /date|replace_first|replace|remove_first|remove_first|remove|minus|times|divided_by|modulo|mod|split|join|truncatewords|truncate|prepend|append/
+
+ FILTER_KEYWORDS = /#{FILTER_WITH_VALUE_KEYWORDS}|textilize|capitalize|downcase|upcase|first|last|sort|map|size|escape_once|escape|strip_html|strip_newlines|newline_to_br/
+
+ SELECTOR_KEYWORDS = /in|with|snippet|script|content_item|folder|widget|wrapper|category|asset_folder|asset/
+
+ LIQUID_DIRECTIVE_BLOCK = /
+ {{1,2}%
+ (.*?)
+ %}{1,2}
+ /x
+
+ KEY_VALUE_REGEX = /(\w+)(:)(\w+|".*"|'.*?')/
+
+ def setup
+ @html_scanner = CodeRay.scanner(:html, tokens: @tokens, keep_tokens: true, keep_state: true)
@korny

korny Apr 7, 2013

Owner

For now, CodeRay is still Ruby 1.8-compatible. Can you use the old hash syntax, please?

@korny korny commented on an outdated diff Apr 7, 2013

lib/coderay/scanners/liquid.rb
+ column_index = 0
+ row.each do |value|
+ column_index += 1
+ unless scan_string(encoder, value)
+ encoder.text_token value, :value
+ end
+ unless column_index >= row.length
+ encoder.text_token ',', :delimiter
+ end
+ end
+ end
+ end
+
+ def scan_key_value_pair(encoder, options, match)
+ scan_spaces(encoder)
+ if match = check(/#{KEY_VALUE_REGEX}/)
@korny

korny Apr 7, 2013

Owner

The /…/o modifier makes sure that the expression is only generated once, which is important for performance. You should use it wherever you include constants into RegExps.

@korny korny commented on an outdated diff Apr 7, 2013

lib/coderay/scanners/liquid.rb
+ false
+ end
+ end
+
+ def scan_selector(encoder, options, match)
+ scan_spaces(encoder)
+ if scan_key_value_pair(encoder, options, match)
+ scan_selector(encoder, options, match)
+ else
+ false
+ end
+ end
+
+ def scan_directive(encoder, options, match)
+ encoder.text_token match, :tag
+ state = :liquid
@korny

korny Apr 7, 2013

Owner

It does not make sense to set local variables that you don't use; you should have gotten a Ruby warning about this. If you want to use the value of "state" between methods, you need an instance variable (which is slower, that's why I try to keep it all in the scan_tokens method.)

@korny korny commented on an outdated diff Apr 7, 2013

lib/coderay/token_kinds.rb
@@ -76,7 +76,7 @@ module CodeRay
:eyecatcher => 'eyecatcher',
:ident => false,
- :operator => false,
+ :operator => 'operator',
@korny

korny Apr 7, 2013

Owner

The :operator token kind explicitly has no associated class; I don't want to change that. If you use this for something that's important in liquid, you should use another token kind.

@korny

korny Apr 7, 2013

Owner

And btw, this is what breaks the tests ;-)

Asherlc commented Apr 8, 2013

Ok, I think I've set everything as per your comments. Let me know if there's more to improve/fix. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment