DataFusion's implementation of length() follows the SQL standard and is equivalent to character_length(), returning the number of Unicode characters.
However, the ClickBench queries currently use:
AVG(length(URL))
AVG(length(Referer))
For DataFusion, these expressions count characters rather than bytes. Computing character length requires scanning and decoding UTF-8, whereas octet_length() returns the number of bytes and is significantly cheaper.
The original ClickBench benchmark semantics appear to be based on byte length (e.g. ClickHouse's length() and DuckDB's strlen() return the number of bytes for UTF-8 strings), so using octet_length() in the DataFusion benchmark would better match those semantics while avoiding unnecessary overhead.
Proposed change
In the DataFusion benchmark queries (Q27 and Q28), replace:
AVG(length(URL))
AVG(length(Referer))
with:
AVG(octet_length(URL))
AVG(octet_length(Referer))
Benefits
- Preserves the intended byte-length semantics.
- Avoids unnecessary UTF-8 character counting in DataFusion.
- Makes DataFusion's ClickBench results more comparable with other engines.
For reference, the corresponding DataFusion issue is:
apache/datafusion#23086
DataFusion's implementation of
length()follows the SQL standard and is equivalent tocharacter_length(), returning the number of Unicode characters.However, the ClickBench queries currently use:
For DataFusion, these expressions count characters rather than bytes. Computing character length requires scanning and decoding UTF-8, whereas
octet_length()returns the number of bytes and is significantly cheaper.The original ClickBench benchmark semantics appear to be based on byte length (e.g. ClickHouse's
length()and DuckDB'sstrlen()return the number of bytes for UTF-8 strings), so usingoctet_length()in the DataFusion benchmark would better match those semantics while avoiding unnecessary overhead.Proposed change
In the DataFusion benchmark queries (Q27 and Q28), replace:
with:
Benefits
For reference, the corresponding DataFusion issue is:
apache/datafusion#23086