Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add setting to customise delimiters #26

Merged
merged 3 commits into from Jul 27, 2017

Conversation

thessem
Copy link
Contributor

@thessem thessem commented Jul 21, 2017

I would like to be able to customise the delimiters I use in templating (this is intended as an analog of https://golang.org/pkg/text/template/#Template.Delims).

This PR has the following problems

  1. Delimiters is a byte array to limit them to single ASCII characters
  2. The tag delimiters are formed using the objectLeft and objectRight argument, I would like to specify all 4 combinations completely, e.g., "<<", ">>", "<%", "%>".
  3. I think moving compilation of the tokenMatcher regex into the function call will dramatically slow things down.
  4. There are no tests.
    I did it like this so I didn't have to change the existing code much.

I am happy to continue work on this, but I'm lodging a PR to see if you're willing to accept this sort of functionality, if I'm going about implementing in the right way, and to check if I haven't missed some other locations where the delimiters are hardcoded.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.08%) to 87.862% when pulling 9dd9191 on Proximaio:feature/customise_delimiters into de5fffa on osteele:master.

// Scan breaks a string into a sequence of Tokens.
func Scan(data string, loc SourceLoc) (tokens []Token) {
func Scan(data string, loc SourceLoc, delims []byte) (tokens []Token) {
// Configure the token matcher to respect the delimeters passed to it
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“delimeters” → “delimiters”

@osteele
Copy link
Owner

osteele commented Jul 22, 2017

This looks like a reasonable way to do this. I added a benchmark to the project; it shows no performance penalty for this. Also using it on the Jekyll doc sources didn't show a measurable effect.

[I also tried to remove the AppVeyor check from CI. Since master doesn't pass, and has never passed, this is just adding noise.]

$ go test -bench=.  # before
BenchmarkEngine_Parse-8               30          45243520 ns/op
BenchmarkTemplate_Render-8           500           2304345 ns/op
PASS
ok      github.com/osteele/liquid       2.868s

$ go test -bench=.  # after
BenchmarkEngine_Parse-8               30          45490013 ns/op
BenchmarkTemplate_Render-8          1000           2339035 ns/op
PASS
ok      github.com/osteele/liquid       4.053s

@coveralls
Copy link

Coverage Status

Coverage increased (+1.3%) to 89.289% when pulling dd4d967 on Proximaio:feature/customise_delimiters into de5fffa on osteele:master.

@thessem
Copy link
Contributor Author

thessem commented Jul 24, 2017

Please ignore that last commit, I'm editing it to support full strings instead of bytes right now, I'll post benchmarks when I think I'm done

@coveralls
Copy link

Coverage Status

Coverage increased (+1.4%) to 89.308% when pulling b7ef67f on Proximaio:feature/customise_delimiters into de5fffa on osteele:master.

@thessem
Copy link
Contributor Author

thessem commented Jul 24, 2017

I added support for delimiters of any length. I had to mess up the regex expression building a little bit as the (?:\s+((?:[^%]|%[^}])+?)) part needed to be constructed based off whatever the custom delimiter was. I hope that I got the logic correct here as I had a little trouble parsing it.

% go test -bench=. -benchtime=10s -benchmem # before
BenchmarkEngine_Parse-4      	     500	  38626146 ns/op	11447556 B/op	   12103 allocs/op
BenchmarkTemplate_Render-4   	   10000	   2001773 ns/op	 1152073 B/op	   23020 allocs/op
PASS
ok  	github.com/osteele/liquid	43.451s

% go test -bench=. -benchtime=10s -benchmem # after
BenchmarkEngine_Parse-4      	     300	  39019683 ns/op	11463230 B/op	   12250 allocs/op
BenchmarkTemplate_Render-4   	   10000	   2060736 ns/op	 1184109 B/op	   23020 allocs/op
PASS
ok  	github.com/osteele/liquid	36.694s

If you are worried about the complexity of this I think it might be prudent to provide a default case with a hardcoded regex, so if the defaults are being used (as they are in 99.9% of cases) the hardcoded expression is used.

@thessem
Copy link
Contributor Author

thessem commented Jul 26, 2017

I am finished working on this PR unless you have any concerns that you'd like me to address.

@osteele osteele merged commit c17c901 into osteele:master Jul 27, 2017
@osteele
Copy link
Owner

osteele commented Jul 27, 2017

Great, looks good to go!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants