Improved IN handling

[benwilson512]

IN improvements for Ecto.CH

Hey folks, thanks for the library! We ran into a problem using the in operator over at CargoSense and we're hoping you're open to a change in the SQL those queries produce to make more efficient use of Clickhouse params.

Problem

Currently when you have an ecto query like:

ids = [1,2,3]
from q in queryable, where: q.id in ^ids

This results in clickhouse SQL:

WHERE q.id in ($0:Int64,$1:Int64,$2:Int64)

This becomes an issue when ids or any other array parameter becomes very large, which is common when using Ecto CH to perform bulk operations in Clickhouse.

Solution

Now in this PR the same Ecto query produces this Clickhouse SQL:

WHERE q.id in {$0:Array(Int64)}

We have just one parameter no matter the size of the array.

[benwilson512]

Notably this functionality only happens with ^ values, literals maintain their current functionality, which helps this remain compatible with complex uses of IN such as with subqueries.

[benwilson512]

This line here is what tells Ecto itself that in ^val should result in a single parameter for val

[ruslandoga]

👋 @benwilson512

Thank you!

I'd like to convince myself that it's indeed equivalent to the previous approach (lists with nulls, different types, etc.) before merging so I'm taking some time :)

---

Btw, just in case, https://clickhouse.com/docs/sql-reference/operators/in suggests

Don't list too many values explicitly (i.e. millions). If a data set is large, put it in a temporary table (for example, see the section External data for query processing), then use a subquery.

But I guess this change is not about millions of values but rather maybe hundreds to thousands

---

The CI is failing for unrelated issues, "fixed" in #252.

[ukutaht]

Nice! We also started recently using IN queries with fairly large payloads for the new consolidated views feature. We currently limit it to 14k sites so we have at maximum 14k IDs in the where site_id IN (list_of_ids) query.

We also tested creating dictionaries and using these for lookups so the list of IDs does not need to be part of the query payload. It was more efficient as expected but for now we didn't go for that solution to reduce maintenance burden.

tagging @aerosol. I don't think we have problems with the limit at the moment but this is interesting in case we need to bump the limit in the future.

[aerosol]

Just on a quick glance, it looks like this is equivalent to the workaround we're applying to avoid hitting ClickHouse limits due to too many bindings: https://github.com/plausible/analytics/blob/fa09b73ff1acff2248a664d652f818c999c7dcca/lib/plausible/stats/sql/where_builder.ex#L54

[benwilson512]

Hey folks! Sorry I didn't follow up. This PR should be up to date with master now.

Understand that this isn't the right answer for millions, but there are definitely "medium" number of IN uses where this helps.

[ruslandoga]

👋 @benwilson512

Thank you!

I still wonder if lists with nil still behave the same.

Sorry for using Codex instead of checking myself, but here's what it "says":

test "nil after typed value changes semantics" do
  TestRepo.insert!(%Post{title: "hello"})
  TestRepo.insert!(%Post{title: ""})

  query =
    from(p in Post,
      where: p.title in ^["hello", nil],
      order_by: p.title,
      select: p.title
    )

  # master:
  # error: FunctionClauseError in param_type(nil)

  # PR:
  # sql:    SELECT p0."title" FROM "posts" AS p0 WHERE (p0."title" IN {$0:Array(String)}) ORDER BY p0."title"
  # params: [["hello", nil]]
  # result: ["", "hello"]
  #
  # This is the bad case: nil is encoded inside Array(String) as the ClickHouse default string.
end

---

CI probably failed due to something similar to this test, which is probably fine and is the expected new behaviour.

test "nested empty array has same result but different SQL shape" do
  query =
    from(n in fragment("numbers(1)"),
      select: fragment("array(?)", n.number) in ^[[]]
    )

  # master:
  # sql:    SELECT array(f0."number") IN ({$0:Array(Nothing)}) FROM numbers(1) AS f0
  # params: [[]]
  # result: [0]

  # PR:
  # sql:    SELECT array(f0."number") IN {$0:Array(Array(Nothing))} FROM numbers(1) AS f0
  # params: [[[]]]
  # result: [0]
end