Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 32 additions & 9 deletions datafusion/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,29 @@ config_namespace! {
/// written, it may be necessary to increase this size to avoid errors from
/// the remote end point.
pub objectstore_writer_buffer_size: usize, default = 10 * 1024 * 1024

/// Whether to enable ANSI SQL mode.
///
/// The flag is experimental and relevant only for DataFusion Spark built-in functions
///
/// When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL
/// semantics for expressions, casting, and error handling. This means:
/// - **Strict type coercion rules:** implicit casts between incompatible types are disallowed.
/// - **Standard SQL arithmetic behavior:** operations such as division by zero,
/// numeric overflow, or invalid casts raise runtime errors rather than returning
/// `NULL` or adjusted values.
/// - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling.
///
/// When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive,
/// non-ANSI mode designed for user convenience and backward compatibility. In this mode:
/// - Implicit casts between types are allowed (e.g., string to integer when possible).
/// - Arithmetic operations are more lenient — for example, `abs()` on the minimum
/// representable integer value returns the input value instead of raising overflow.
/// - Division by zero or invalid casts may return `NULL` instead of failing.
///
/// # Default
/// `false` — ANSI SQL mode is disabled by default.
pub enable_ansi_mode: bool, default = false
}
}

Expand Down Expand Up @@ -1124,6 +1147,15 @@ pub struct ConfigOptions {
}

impl ConfigField for ConfigOptions {
fn visit<V: Visit>(&self, v: &mut V, _key_prefix: &str, _description: &'static str) {
self.catalog.visit(v, "datafusion.catalog", "");
self.execution.visit(v, "datafusion.execution", "");
self.optimizer.visit(v, "datafusion.optimizer", "");
self.explain.visit(v, "datafusion.explain", "");
self.sql_parser.visit(v, "datafusion.sql_parser", "");
self.format.visit(v, "datafusion.format", "");
}

fn set(&mut self, key: &str, value: &str) -> Result<()> {
// Extensions are handled in the public `ConfigOptions::set`
let (key, rem) = key.split_once('.').unwrap_or((key, ""));
Expand All @@ -1137,15 +1169,6 @@ impl ConfigField for ConfigOptions {
_ => _config_err!("Config value \"{key}\" not found on ConfigOptions"),
}
}

fn visit<V: Visit>(&self, v: &mut V, _key_prefix: &str, _description: &'static str) {
self.catalog.visit(v, "datafusion.catalog", "");
self.execution.visit(v, "datafusion.execution", "");
self.optimizer.visit(v, "datafusion.optimizer", "");
self.explain.visit(v, "datafusion.explain", "");
self.sql_parser.visit(v, "datafusion.sql_parser", "");
self.format.visit(v, "datafusion.format", "");
}
}

impl ConfigOptions {
Expand Down
2 changes: 2 additions & 0 deletions datafusion/sqllogictest/test_files/information_schema.slt
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,7 @@ datafusion.catalog.newlines_in_values false
datafusion.execution.batch_size 8192
datafusion.execution.coalesce_batches true
datafusion.execution.collect_statistics true
datafusion.execution.enable_ansi_mode false
datafusion.execution.enable_recursive_ctes true
datafusion.execution.enforce_batch_size_in_joins false
datafusion.execution.keep_partition_by_columns false
Expand Down Expand Up @@ -338,6 +339,7 @@ datafusion.catalog.newlines_in_values false Specifies whether newlines in (quote
datafusion.execution.batch_size 8192 Default batch size while creating new batches, it's especially useful for buffer-in-memory batches since creating tiny batches would result in too much metadata memory consumption
datafusion.execution.coalesce_batches true When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting
datafusion.execution.collect_statistics true Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Applies to the default `ListingTableProvider` in DataFusion. Defaults to true.
datafusion.execution.enable_ansi_mode false Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.
datafusion.execution.enable_recursive_ctes true Should DataFusion support recursive CTEs
datafusion.execution.enforce_batch_size_in_joins false Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower.
datafusion.execution.keep_partition_by_columns false Should DataFusion keep the columns used for partition_by in the output RecordBatches
Expand Down
Loading