Skip to content

fix ^ evaluates as bitwise XOR instead of exponentiation#22314

Open
xiedeyantu wants to merge 3 commits into
apache:mainfrom
xiedeyantu:cifang
Open

fix ^ evaluates as bitwise XOR instead of exponentiation#22314
xiedeyantu wants to merge 3 commits into
apache:mainfrom
xiedeyantu:cifang

Conversation

@xiedeyantu
Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

In PostgreSQL, the ^ operator represents exponentiation, but DataFusion was interpreting it as bitwise XOR. This caused PostgreSQL-dialect queries such as SELECT 2 ^ 3; to return 1 instead of 8.

This change aligns DataFusion's PostgreSQL SQL semantics with PostgreSQL for this operator while preserving existing non-PostgreSQL behavior.

What changes are included in this PR?

  • Added PostgreSQL-specific handling for BinaryOperator::PGExp during SQL expression lowering.
  • Lowered PostgreSQL ^ expressions to the built-in power(left, right) scalar function instead of treating them as bitwise XOR.
  • Kept existing generic-dialect ^ behavior unchanged.
  • Kept PostgreSQL bitwise XOR behavior unchanged via #.
  • Added a sqllogictest regression covering SELECT 2 ^ 3; under the PostgreSQL parser dialect.

Are these changes tested?

Yes.

Added a regression test in scalar.slt to verify that PostgreSQL-dialect ^ is evaluated as exponentiation.

Validated with:

cargo test -p datafusion-sqllogictest --test sqllogictests scalar

Are there any user-facing changes?

Yes.

For the PostgreSQL SQL parser dialect, ^ now behaves as exponentiation instead of bitwise XOR, matching PostgreSQL semantics. Generic-dialect behavior is unchanged, and PostgreSQL bitwise XOR remains available through #.

@github-actions github-actions Bot added sql SQL Planner sqllogictest SQL Logic Tests (.slt) labels May 17, 2026
@Dandandan Dandandan added the api change Changes the API exposed to users of the crate label May 18, 2026
Comment thread datafusion/sql/src/expr/mod.rs Outdated
}

let RawBinaryExpr { op, left, right } = binary_expr;
if op == BinaryOperator::PGExp {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be handled somewhere else?

Do we neef to update documentation / tests, also covering bitwise XOR?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see it correctly uses # in the tests, so the previous behavior was probably an oversight.
I think we should move the change to self.parse_sql_binary_op(&op) though?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've moved the method to binary_op.rs because BinaryOperator::PGExp doesn't have an operator chain like "#", so I used this approach to solve the problem. I also tried implementing "^" in a similar way to "#", but it required a lot of modifications and ultimately still relies on the "power" function. Would you accept this modification?

@github-actions
Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-sql v53.1.0 (current)
       Built [  41.720s] (current)
     Parsing datafusion-sql v53.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-sql v53.1.0 (baseline)
       Built [  40.809s] (baseline)
     Parsing datafusion-sql v53.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-sql v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.319s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [  84.197s] datafusion-sql
    Building datafusion-sqllogictest v53.1.0 (current)
error: running cargo-doc on crate 'datafusion-sqllogictest' failed with output:
-----
   Compiling proc-macro2 v1.0.106
   Compiling unicode-ident v1.0.24
   Compiling quote v1.0.45
   Compiling libc v0.2.186
    Checking cfg-if v1.0.4
    Checking bytes v1.11.1
    Checking memchr v2.8.0
   Compiling serde_core v1.0.228
   Compiling syn v2.0.117
   Compiling autocfg v1.5.0
   Compiling jobserver v0.1.34
   Compiling find-msvc-tools v0.1.9
   Compiling shlex v1.3.0
    Checking itoa v1.0.18
   Compiling cc v1.2.62
   Compiling libm v0.2.16
   Compiling num-traits v0.2.19
    Checking allocator-api2 v0.2.21
    Checking equivalent v1.0.2
    Checking once_cell v1.21.4
    Checking foldhash v0.2.0
    Checking hashbrown v0.17.1
    Checking pin-project-lite v0.2.17
   Compiling zmij v1.0.21
    Checking indexmap v2.14.0
    Checking futures-core v0.3.32
   Compiling zerocopy v0.8.48
    Checking futures-sink v0.3.32
    Checking errno v0.3.14
    Checking mio v1.2.0
    Checking signal-hook-registry v1.4.8
    Checking socket2 v0.6.3
   Compiling version_check v0.9.5
   Compiling serde v1.0.228
    Checking num-integer v0.1.46
    Checking slab v0.4.12
   Compiling serde_json v1.0.149
    Checking futures-channel v0.3.32
    Checking smallvec v1.15.1
   Compiling getrandom v0.3.4
    Checking num-bigint v0.4.6
   Compiling synstructure v0.13.2
    Checking http v1.4.0
    Checking base64 v0.22.1
    Checking futures-io v0.3.32
    Checking futures-task v0.3.32
    Checking iana-time-zone v0.1.65
    Checking chrono v0.4.44
    Checking tracing-core v0.1.36
   Compiling zerocopy-derive v0.8.48
   Compiling tokio-macros v2.7.0
   Compiling serde_derive v1.0.228
    Checking tokio v1.52.3
   Compiling futures-macro v0.3.32
    Checking futures-util v0.3.32
   Compiling zerofrom-derive v0.1.7
   Compiling tracing-attributes v0.1.31
    Checking siphasher v1.0.3
    Checking zerofrom v0.1.8
   Compiling yoke-derive v0.8.2
    Checking stable_deref_trait v1.2.1
    Checking num-complex v0.4.6
   Compiling zerovec-derive v0.11.3
    Checking tracing v0.1.44
    Checking getrandom v0.2.17
   Compiling pkg-config v0.3.33
    Checking half v2.7.1
    Checking phf_shared v0.12.1
   Compiling displaydoc v0.2.5
   Compiling ahash v0.8.12
    Checking yoke v0.8.2
    Checking percent-encoding v2.3.2
   Compiling chrono-tz v0.10.4
   Compiling thiserror v2.0.18
    Checking phf v0.12.1
    Checking zerovec v0.11.6
    Checking arrow-buffer v58.3.0
   Compiling thiserror-impl v2.0.18
    Checking arrow-schema v58.3.0
   Compiling ring v0.17.14
    Checking arrow-data v58.3.0
   Compiling semver v1.0.28
   Compiling getrandom v0.4.2
    Checking rand_core v0.10.1
    Checking log v0.4.29
    Checking tinystr v0.8.3
    Checking litemap v0.8.2
    Checking untrusted v0.9.0
    Checking writeable v0.6.3
    Checking bitflags v2.11.1
    Checking icu_locale_core v2.2.0
    Checking potential_utf v0.1.5
    Checking zerotrie v0.2.4
   Compiling zstd-sys v2.0.16+zstd.1.5.7
   Compiling async-trait v0.1.89
    Checking utf8_iter v1.0.4
   Compiling icu_normalizer_data v2.2.0
   Compiling icu_properties_data v2.2.0
    Checking icu_collections v2.2.0
    Checking icu_provider v2.2.0
    Checking aho-corasick v1.1.4
   Compiling zstd-safe v7.2.4
    Checking lexical-util v1.0.7
   Compiling object v0.37.3
    Checking regex-syntax v0.8.10
    Checking arrow-array v58.3.0
    Checking ryu v1.0.23
    Checking regex-automata v0.4.14
    Checking arrow-select v58.3.0
    Checking icu_properties v2.2.0
    Checking icu_normalizer v2.2.0
   Compiling rustix v1.1.4
    Checking regex v1.12.3
    Checking form_urlencoded v1.2.2
    Checking idna_adapter v1.2.2
    Checking either v1.15.0
    Checking typenum v1.20.0
    Checking unicode-width v0.2.2
   Compiling crc32fast v1.5.0
   Compiling parking_lot_core v0.9.12
    Checking idna v1.1.0
    Checking lexical-parse-integer v1.0.6
    Checking lexical-write-integer v1.0.6
   Compiling rustc_version v0.4.1
    Checking futures-executor v0.3.32
   Compiling pin-project-internal v1.1.13
    Checking adler2 v2.0.1
    Checking simd-adler32 v0.3.9
    Checking scopeguard v1.2.0
    Checking lock_api v0.4.14
    Checking miniz_oxide v0.8.9
   Compiling flatbuffers v25.12.19
    Checking futures v0.3.32
    Checking lexical-write-float v1.0.6
    Checking pin-project v1.1.13
    Checking lexical-parse-float v1.0.6
    Checking url v2.5.8
    Checking zlib-rs v0.6.3
    Checking unicode-segmentation v1.13.2
    Checking byteorder v1.5.0
   Compiling ar_archive_writer v0.5.1
    Checking comfy-table v7.2.2
    Checking lexical-core v1.0.6
    Checking itertools v0.14.0
    Checking flate2 v1.1.9
   Compiling psm v0.1.31
    Checking arrow-ord v58.3.0
    Checking atoi v2.0.0
   Compiling stacker v0.1.24
    Checking hex v0.4.3
    Checking alloc-no-stdlib v2.0.4
    Checking twox-hash v2.1.2
   Compiling snap v1.1.1
    Checking lz4_flex v0.13.1
    Checking alloc-stdlib v0.2.2
    Checking arrow-cast v58.3.0
    Checking parking_lot v0.12.5
    Checking csv-core v0.1.13
    Checking simdutf8 v0.1.5
    Checking humantime v2.3.0
   Compiling paste v1.0.15
    Checking same-file v1.0.6
    Checking walkdir v2.5.0
    Checking csv v1.4.0
    Checking brotli-decompressor v5.0.0
   Compiling recursive-proc-macro-impl v0.1.1
    Checking ordered-float v2.10.1
    Checking integer-encoding v3.0.4
    Checking subtle v2.6.1
    Checking arrow-csv v58.3.0
    Checking thrift v0.17.0
    Checking arrow-json v58.3.0
    Checking brotli v8.0.2
    Checking recursive v0.1.1
    Checking object_store v0.13.2
    Checking arrow-string v58.3.0
    Checking arrow-arith v58.3.0
    Checking zstd v0.13.3
    Checking arrow-ipc v58.3.0
    Checking arrow-row v58.3.0
    Checking uuid v1.23.1
    Checking tokio-util v0.7.18
   Compiling sqlparser_derive v0.5.0
   Compiling seq-macro v0.3.6
    Checking arrow v58.3.0
    Checking ppv-lite86 v0.2.21
    Checking sqlparser v0.62.0
    Checking cpufeatures v0.3.0
    Checking hybrid-array v0.4.12
    Checking parquet v58.3.0
    Checking linux-raw-sys v0.12.1
    Checking cmov v0.5.3
    Checking crypto-common v0.2.1
    Checking ctutils v0.4.2
    Checking block-buffer v0.12.0
    Checking rand_core v0.9.5
   Compiling generic-array v0.14.7
    Checking const-oid v0.10.2
    Checking digest v0.11.3
    Checking rand_chacha v0.9.0
    Checking rand v0.9.4
    Checking datafusion-doc v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
   Compiling crossbeam-utils v0.8.21
    Checking foldhash v0.1.5
    Checking hashbrown v0.15.5
    Checking block-buffer v0.10.4
    Checking crypto-common v0.1.7
   Compiling heck v0.5.0
    Checking fixedbitset v0.5.7
    Checking fastrand v2.4.1
    Checking tempfile v3.27.0
    Checking petgraph v0.8.3
    Checking digest v0.10.7
    Checking sha2 v0.11.0
    Checking md-5 v0.11.0
    Checking hashbrown v0.14.5
    Checking dashmap v6.2.1
   Compiling blake3 v1.8.5
   Compiling datafusion-macros v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
   Compiling anyhow v1.0.102
    Checking arrayref v0.3.9
    Checking arrayvec v0.7.6
    Checking constant_time_eq v0.4.2
    Checking blake2 v0.10.6
   Compiling liblzma-sys v0.4.6
    Checking libbz2-rs-sys v0.2.5
    Checking bzip2 v0.6.1
    Checking datafusion-common-runtime v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
    Checking compression-core v0.4.32
   Compiling httparse v1.10.1
   Compiling prost-derive v0.14.3
    Checking http-body v1.0.1
    Checking glob v0.3.3
    Checking tower-service v0.3.3
    Checking try-lock v0.2.5
    Checking atomic-waker v1.1.2
    Checking fnv v1.0.7
    Checking h2 v0.4.14
    Checking want v0.3.1
    Checking tokio-stream v0.1.18
    Checking zeroize v1.8.2
    Checking httpdate v1.0.3
    Checking hyper v1.9.0
    Checking rustls-pki-types v1.14.1
    Checking datafusion-common v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
   Compiling prost v0.14.3
    Checking hyper-util v0.1.20
    Checking http-body-util v0.1.3
    Checking tower-layer v0.3.3
    Checking sync_wrapper v1.0.2
   Compiling rustls v0.23.40
   Compiling prettyplease v0.2.37
   Compiling prost-types v0.14.3
    Checking datafusion-expr-common v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
    Checking liblzma v0.4.6
    Checking compression-codecs v0.4.38
    Checking async-compression v0.4.42
    Checking rustls-webpki v0.103.13
   Compiling serde_derive_internals v0.29.1
    Checking datafusion-physical-expr-common v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)
   Compiling schemars v0.8.22
    Checking mime v0.3.17
    Checking datafusion-functions-aggregate-common v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common)
    Checking datafusion-functions-window-common v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window-common)
   Compiling schemars_derive v0.8.22
   Compiling hashbrown v0.16.1
    Checking axum-core v0.5.6
    Checking datafusion-expr v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/expr)
    Checking tower v0.5.3
   Compiling dyn-clone v1.0.20
   Compiling multimap v0.10.1
    Checking matchit v0.8.4
   Compiling prost-build v0.14.3
    Checking axum v0.8.9
   Compiling regress v0.10.5
    Checking datafusion-physical-expr v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)
    Checking datafusion-execution v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/execution)
   Compiling pbjson-build v0.8.0
    Checking datafusion-functions v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)
    Checking hyper-timeout v0.5.2
    Checking chacha20 v0.10.0
   Compiling strsim v0.11.1
   Compiling portable-atomic v1.13.1
   Compiling ident_case v1.0.1
   Compiling darling_core v0.23.0
   Compiling typify-impl v0.5.0
    Checking rand v0.10.1
    Checking tonic v0.14.6
   Compiling serde_tokenstream v0.2.3
    Checking ureq-proto v0.6.0
   Compiling bigdecimal v0.4.10
    Checking utf8parse v0.2.2
   Compiling bollard-buildkit-proto v0.7.0
    Checking powerfmt v0.2.0
    Checking crc-catalog v2.5.0
    Checking tinyvec_macros v0.1.1
    Checking utf8-zero v0.8.1
   Compiling darling_macro v0.23.0
    Checking ureq v3.3.0
    Checking datafusion-physical-plan v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)
    Checking datafusion-physical-expr-adapter v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-adapter)
    Checking tinyvec v1.11.0
    Checking crc v3.4.0
    Checking deranged v0.5.8
    Checking anstyle-parse v1.0.0
    Checking tonic-prost v0.14.6
    Checking datafusion-functions-aggregate v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)
   Compiling typify-macro v0.5.0
   Compiling strum_macros v0.28.0
   Compiling structmeta-derive v0.3.0
    Checking anstyle v1.0.14
    Checking anstyle-query v1.1.5
    Checking num-conv v0.2.1
    Checking is_terminal_polyfill v1.70.2
    Checking colorchoice v1.0.5
    Checking time-core v0.1.8
   Compiling unsafe-libyaml v0.2.11
    Checking time v0.3.47
    Checking anstream v1.0.0
    Checking arrow-avro v58.3.0
   Compiling serde_yaml v0.9.34+deprecated
   Compiling structmeta v0.3.0
    Checking datafusion-functions-nested v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-nested)
   Compiling typify v0.5.0
    Checking datafusion-session v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/session)
    Checking datafusion-datasource v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource)
    Checking unicode-normalization v0.1.25
   Compiling darling v0.23.0
   Compiling pbjson-types v0.8.0
    Checking tokio-rustls v0.26.4
   Compiling async-stream-impl v0.3.6
   Compiling serde_repr v0.1.20
    Checking num-rational v0.4.2
    Checking num-iter v0.1.45
    Checking unicode-bidi v0.3.18
    Checking clap_lex v1.1.0
    Checking datafusion-pruning v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/pruning)
    Checking datafusion-catalog v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/catalog)
    Checking unicode-properties v0.1.4
    Checking openssl-probe v0.2.1
    Checking rustls-native-certs v0.8.3
    Checking clap_builder v4.6.0
    Checking stringprep v0.1.5
    Checking datafusion-datasource-parquet v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-parquet)
    Checking datafusion-physical-optimizer v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-optimizer)
    Checking num v0.4.3
    Checking datafusion-datasource-csv v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-csv)
    Checking datafusion-datasource-json v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-json)
    Checking datafusion-datasource-avro v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-avro)
    Checking datafusion-datasource-arrow v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-arrow)
    Checking bollard-stubs v1.52.1-rc.29.1.3
    Checking async-stream v0.3.6
    Checking hyper-rustls v0.27.9
    Checking datafusion-sql v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/sql)
   Compiling serde_with_macros v3.20.0
error[E0277]: can't compare `&BinaryOperator` with `BinaryOperator`
  --> /home/runner/work/datafusion/datafusion/datafusion/sql/src/expr/binary_op.rs:84:15
   |
84 |         if op == BinaryOperator::PGExp {
   |               ^^ no implementation for `&BinaryOperator == BinaryOperator`
   |
   = help: the trait `PartialEq<BinaryOperator>` is not implemented for `&BinaryOperator`
help: consider dereferencing here
   |
84 |         if *op == BinaryOperator::PGExp {
   |            +

error[E0308]: mismatched types
   --> /home/runner/work/datafusion/datafusion/datafusion/sql/src/expr/mod.rs:145:32
    |
145 |         self.build_binary_expr(op, left, right)
    |              ----------------- ^^ expected `&BinaryOperator`, found `BinaryOperator`
    |              |
    |              arguments to this method are incorrect
    |
note: method defined here
   --> /home/runner/work/datafusion/datafusion/datafusion/sql/src/expr/binary_op.rs:78:19
    |
 78 |     pub(crate) fn build_binary_expr(
    |                   ^^^^^^^^^^^^^^^^^
 79 |         &self,
 80 |         op: &BinaryOperator,
    |         -------------------
help: consider borrowing here
    |
145 |         self.build_binary_expr(&op, left, right)
    |                                +

   Compiling substrait v0.63.0
   Compiling parse-display-derive v0.9.1
Some errors have detailed explanations: E0277, E0308.
For more information about an error, try `rustc --explain E0277`.
error: could not compile `datafusion-sql` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...

-----

error: failed to build rustdoc for crate datafusion-sqllogictest v53.1.0
note: this is usually due to a compilation error in the crate,
      and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/sqllogictest --features avro,backtrace,bytes,chrono,datafusion-substrait,parquet_encryption,postgres,postgres-types,substrait,testcontainers-modules,tokio-postgres &&
          cargo check &&
          cargo doc

error: aborting due to failure to build rustdoc for crate datafusion-sqllogictest v53.1.0

@github-actions github-actions Bot added the auto detected api change Auto detected API change label May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate auto detected api change Auto detected API change sql SQL Planner sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PostgreSQL compatibility: ^ evaluates as bitwise XOR instead of exponentiation

2 participants