Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate contextualization of inputs via the context #1537

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions History.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

### Fixes
* StandardFilter: Fix missing @context on iterations (#1525) [Thierry Joyal]
* Consolidate contextualization of inputs via the context (#XXX) [Thierry Joyal]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the potential "breaking change" in StandardFilters and VariableLookup where Proc are now properly handled, we might to look at this to be similar to #1527 flagged as deprecation for 6.0.

I do not think anyone expected to have Proc to be exposed directly to templates.

For standard filters specifically, that respond_to?(:to_liquid) might have left non-liquid aware object in by potential design error. I could see use cases leveraging this unexpectedly. They have an easy path forward to migrate, but a deprecation warning might be of interest here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"contextualization" is a new concept, which makes this change log entry very unclear about what is being fixed. That seems like just a symptom of the fact that this PR combines multiple changes, a new feature (added method), behaviour fixes and some refactoring (which doesn't need to be in the changelog).


### Deprecation
* Condition#evaluate to require mandatory context argument in Liquid 6.0.0 (#1527) [Thierry Joyal]
Expand Down
25 changes: 17 additions & 8 deletions lib/liquid/context.rb
Original file line number Diff line number Diff line change
Expand Up @@ -187,16 +187,11 @@ def find_variable(key, raise_on_not_found: true)
# path and find_index() is optimized in MRI to reduce object allocation
index = @scopes.find_index { |s| s.key?(key) }

variable = if index
if index
lookup_and_evaluate(@scopes[index], key, raise_on_not_found: raise_on_not_found)
else
try_variable_find_in_environments(key, raise_on_not_found: raise_on_not_found)
end

variable = variable.to_liquid
variable.context = self if variable.respond_to?(:context=)

variable
end

def lookup_and_evaluate(obj, key, raise_on_not_found: true)
Expand All @@ -207,9 +202,9 @@ def lookup_and_evaluate(obj, key, raise_on_not_found: true)
value = obj[key]

if value.is_a?(Proc) && obj.respond_to?(:[]=)
obj[key] = value.arity == 0 ? value.call : value.call(self)
obj[key] = contextualize(value)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not changing the memoization in this PR as it let to too large of a change with even more unrelated code alterations.

I feel we should look at:

  • Exploring memoization everywhere (eg.: for code path like StandardFilters to benefit from it)
  • Not only memoize if value.is_a?(Proc). There are some test failures appearing when trying to remove the condition, can be iterated further in subsequent work if required.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would have been clearer, but achieve the same behaviour with more code. No strong opinion on my end.

Suggested change
obj[key] = contextualize(value)
value = contextualize(value)
obj[key] = value
value

else
value
contextualize(value)
end
end

Expand All @@ -228,6 +223,20 @@ def tag_disabled?(tag_name)
@disabled_tags.fetch(tag_name, 0) > 0
end

# Convert input objects into liquid aware representations
# Procs will be resolved
# Assigns the context (self) through context=
def contextualize(object)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method name makes it seem like its primary purpose is to add a liquid context to an object. I had proposed to_liquid as the method name in #1205 (without the Proc conversion) because I want to get to the point where liquid conversion and assigning the context are done together. That way context.to_liquid(object) can be a thin wrapper around object.to_liquid(context) after a deprecation cycle to combine these.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get from the name that is sets context, but it's not clear that it converts it to a "liquid aware representations". Maybe that means the method is doing too much?

Could we instead move the "contextualization" to Drop#invoke_drop: add a context arg to invoke_drop, and wrap procs in drops? Or is there are use case outside of drops & procs for this?

if object.is_a?(Proc)
object = object.arity == 0 ? object.call : object.call(self)
end
Comment on lines +230 to +232
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should add support for Procs in more places. It's purpose is to make the evaluation of something relatively expensive lazy so that it isn't done when it isn't needed. Therefore, doing it in places where it isn't memoized introduces a performance trap. We should be deprecating any place where we are already doing that, instead of centralizing it so it is easy to do elsewhere.

lookup_and_evaluate is a better place to handle this concern for now. That is a safe API, because it ensures the Proc result is memoized. Although, it could also be improved by giving a deprecation warning if value.is_a?(Proc) is true but not obj.respond_to?(:[]=).

Longer term, it would be better to reduce support for Proc to just find_variable.


object = object.to_liquid
object.context = self if object.respond_to?(:context=)

object
end

protected

attr_writer :base_scope_depth, :warnings, :errors, :strainer, :filters, :disabled_tags
Expand Down
3 changes: 1 addition & 2 deletions lib/liquid/standardfilters.rb
Original file line number Diff line number Diff line change
Expand Up @@ -582,8 +582,7 @@ def empty?

def each
@input.each do |e|
e = e.respond_to?(:to_liquid) ? e.to_liquid : e
e.context = @context if e.respond_to?(:context=)
e = @context.contextualize(e)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted in the description, this change no longer exclude e.respond_to?(:to_liquid)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found it hard to separate isolated deliverables as there is a lot of inter-dependency at play.

Seems like making the to_liquid call unconditional would be pretty easy to split into its own PR or at least its own commit.

Combining refactoring and behaviour changes in the same PR can end up making the behaviour changes quite subtle. Instead, one PR could be based on another. E.g. the behaviour change PR could be done first, then the contextualize change can refactor it. Alternatively, the refactor PR could just replace the code that doesn't result in a behaviour change, then the behaviour change PR can use a new help method from the refactor PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can have a closer look. Making it unconditional was making a few tests of StandardFilters (specifically around map) no longer pass since there is Proc handling in said method and Proc does not respond to_liquid.

If we wanted to make the call unconditional in isolation, I will have to either:

  1. Move support for Proc here (similar to the current PR, extending support for all use cases)
  2. Drop support for Proc in StandardFilters#map (potentially what we will want to do down the road?)

I think going with 1 is a good tradeoff for now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's assume that a Proc type is the only non-liquid value that needs to be handled here. In that case, we can use a if e.is_a?(Proc) check instead of e.respond_to?(:to_liquid). We can then preserve backwards compatibility with a deprecation warning for a Proc. Doing that here would also give a desirable deprecation warning for the e = e.call if e.is_a?(Proc) line in StandardFilters#map.

This deprecation should be communicated as deprecating support for having a Proc item in an enumerable.

On the other hand, I think we should continue supporting a Proc value in liquid Hash objects and should make sure to memoize the result of evaluating the Proc. This means that the property lookup in StandardFilters#map should memoize the result of calling any Proc value.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are other filters that support looking properties, which should be able to just use context.lookup_and_evaluate(item, property, raise_on_not_found: false). Basically, that should be used as the helper method to consolidate Proc Hash lookup support.

That helper method could be used in the map filter implementation as well, although we still may want to keep the r.is_a?(Proc) ? r.call : r line to unconditionally evaluate the Proc even when memoization isn't supported, for backwards compatibility.

yield(e)
end
end
Expand Down
10 changes: 3 additions & 7 deletions lib/liquid/variable_lookup.rb
Original file line number Diff line number Diff line change
Expand Up @@ -49,15 +49,14 @@ def evaluate(context)
((object.respond_to?(:key?) && object.key?(key)) ||
(object.respond_to?(:fetch) && key.is_a?(Integer)))

# if its a proc we will replace the entry with the proc
res = context.lookup_and_evaluate(object, key)
object = res.to_liquid
object = context.lookup_and_evaluate(object, key)

# Some special cases. If the part wasn't in square brackets and
# no key with the same name was found we interpret following calls
# as commands and call them on the current object
elsif @command_flags & (1 << i) != 0 && object.respond_to?(key)
object = object.send(key).to_liquid
object = object.send(key)
object = context.contextualize(object)

# No key was present with the desired value and it wasn't one of the directly supported
# keywords either. The only thing we got left is to return nil or
Expand All @@ -66,9 +65,6 @@ def evaluate(context)
return nil unless context.strict_variables
raise Liquid::UndefinedVariable, "undefined variable #{key}"
end

# If we are dealing with a drop here we have to
object.context = context if object.respond_to?(:context=)
end

object
Expand Down
38 changes: 35 additions & 3 deletions test/integration/standard_filter_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ def initialize(value:)
def registers
@context.registers
end

def to_s
"TestDrop(value:#{@value})"
end
end

class TestModel
Expand Down Expand Up @@ -467,10 +471,38 @@ def test_sort_calls_to_liquid
end

def test_map_over_proc
drop = TestDrop.new(value: "testfoo")
drop = TestDrop.new(value: "123")
p = proc { drop }
templ = '{{ procs | map: "value" }}'
assert_template_result("testfoo", templ, "procs" => [p])
assert_template_result("123", templ, "procs" => [p])
end

def test_join_over_proc
drop = TestDrop.new(value: "123")
p = proc { drop }
templ = '{{ procs | join }}'
assert_template_result('TestDrop(value:123)', templ, "procs" => [p])
end

def test_sort_over_proc
drop = TestDrop.new(value: "123")
p = proc { drop }
templ = '{{ procs | sort: "value" }}'
assert_template_result("TestDrop(value:123)", templ, "procs" => [p])
end

def test_where_over_proc
drop = TestDrop.new(value: "123")
p = proc { drop }
templ = '{{ procs | where: "value", "123" }}'
assert_template_result("TestDrop(value:123)", templ, "procs" => [p])
end

def test_uniq_over_proc
drop = TestDrop.new(value: "123")
p = proc { drop }
templ = '{{ procs | uniq }}'
assert_template_result("TestDrop(value:123)", templ, "procs" => [p])
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are to cover the new behavior for Proc over filters where historically they would have raised on missing method to_liquid

end

def test_map_over_drops_returning_procs
Expand Down Expand Up @@ -888,7 +920,7 @@ def test_all_filters_never_raise_non_liquid_exception
{ foo: "bar" },
[{ "foo" => "bar" }, { "foo" => 123 }, { "foo" => nil }, { "foo" => true }, { "foo" => ["foo", "bar"] }],
{ 1 => "bar" },
["foo", 123, nil, true, false, Drop, ["foo"], { foo: "bar" }],
["foo", 123, nil, true, false, ["foo"], { foo: "bar" }],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps the intention was to test against a drop instance here, which should be supported

Suggested change
["foo", 123, nil, true, false, ["foo"], { foo: "bar" }],
["foo", 123, nil, true, false, Drop.new, ["foo"], { foo: "bar" }],

]
StandardFilters.public_instance_methods(false).each do |method|
arg_count = @filters.method(method).arity
Expand Down
34 changes: 28 additions & 6 deletions test/integration/tags/standard_tag_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -272,14 +272,36 @@ def test_multiple_named_cycles_with_names_from_context
'{%cycle var1: "one", "two" %} {%cycle var2: "one", "two" %} {%cycle var1: "one", "two" %} {%cycle var2: "one", "two" %} {%cycle var1: "one", "two" %} {%cycle var2: "one", "two" %}', assigns)
end

def test_size_of_array
assigns = { "array" => [1, 2, 3, 4] }
assert_template_result('array has 4 elements', "array has {{ array.size }} elements", assigns)
def test_command_methods_of_array
assigns = { "array" => [11, 22, 33] }

assert_template_result("3", "{{ array.size }}", assigns)

assert_template_result("11", "{{ array.first }}", assigns)

assert_template_result("33", "{{ array.last }}", assigns)
end

def test_command_methods_of_hash
assigns = { "hash" => { a: 11, b: 22, c: 33 } }

assert_template_result("3", "{{ hash.size }}", assigns)

assert_template_result("a11", "{{ hash.first }}", assigns)
assert_template_result("a", "{{ hash.first.first }}", assigns)
assert_template_result("11", "{{ hash.first.last }}", assigns)

assert_template_result("", "{{ hash.last }}", assigns)
end

def test_size_of_hash
assigns = { "hash" => { a: 1, b: 2, c: 3, d: 4 } }
assert_template_result('hash has 4 elements', "hash has {{ hash.size }} elements", assigns)
def test_command_methods_with_proc
skip("Liquid-C does not properly resolve Procs in with command methods") if ENV['LIQUID_C'] == '1'
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Liquid-C does not pass this test. Skipping for now as I think we need to iterate further on Proc handling without blocking iterative work.


assigns = { "array" => [proc { "test" }] }
assert_template_result("test", "{{ array.first }}", assigns)

assigns = { "hash" => { a: proc { "test" } } }
assert_template_result("test", "{{ hash.first.last }}", assigns)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are to cover the new behavior for Proc where historically they would have returned the Proc as string (eg.: "#<Proc:0xXXXXXX /Users/tjoyal/projects/liquid/test/integration/standard_filter_test.rb:485>")

end

def test_illegal_symbols
Expand Down