[BREAKING] Modify implicit casting rules to differentiate between string literals and VARCHAR columns, and disable implicit casting to VARCHAR for many types

[Mytherin]

Fixes #8529
Fixes #9948

This PR modifies the way that the implicit casting rules work by treating string literals different from VARCHAR columns, and disables implicit casting to VARCHAR for many types. This aligns DuckDB's implicit casting rules with that of Postgres, and prevents certain confusing situations from occurring.

Breaking Change

This PR introduces a breaking SQL change. The setting old_implicit_casting can be used to revert to the old behavior (for now - potentially to be deprecated in the future). For example:

SELECT a>=42 FROM (VALUES ('42')) t(a);
-- Binder Error: Cannot compare values of type VARCHAR and type INTEGER - an explicit cast is required
SET old_implicit_casting=true;
SELECT a>=42 FROM (VALUES ('42')) t(a);
┌──────────┐
│ (a = 42) │
│ boolean  │
├──────────┤
│ true     │
└──────────┘

New Rules

This core changes of this PR are as follows:

• String literals are treated as a separate type from VARCHAR - and can be implicitly cast to any other type where-ever they are used
• Non-equality comparisons (> >= etc) with type VARCHAR and different types require explicit casts on either side
• Equality comparisons with type VARCHAR are still allowed, but we perform a comparison of the different type in more scenarios (also for datetime types)
• Columns of non-varchar types will no longer implicitly cast to VARCHAR when picking functions, an explicit cast is now required (e.g. length(42) now fails)
• Certain functions (such as CONCAT, PRINTF) have been modified to allow for ANY parameters and explicitly introduce a varchar cast, allowing e.g. CONCAT('hello', 42) to keep working

String Literals vs VARCHAR

String literals are effectively treated as "untyped" instead of as type "varchar". Similar to prepared statement parameters, the string literal type will be modified to the surrounding types if possible (or the VARCHAR type if not).

This allows users to use string literals as a literal of any other type. For example:

D select date '1992-01-01' > '1993-01-01' AS r;
┌─────────┐
│    r    │
│ boolean │
├─────────┤
│ false   │
└─────────┘
D select 2 * '02' AS r;
┌───────┐
│   r   │
│ int32 │
├───────┤
│     4 │
└───────┘

Varchar Auto-Casting

In the current version of DuckDB, we allow all types to be implicitly cast to VARCHAR and we also allow every type to be compared with VARCHAR. For example, the following is valid in DuckDB, but not in Postgres:

SELECT date '2000-01-01' = '2000-01-01'::VARCHAR;

This implicit casting for comparisons becomes confusing as depending on which side we cast to a different result might be obtained. For example, the values 2000-01-01 00:00:00 and 2000-01-01 12:00:00 are equal as dates due to truncation, but not as varchar columns:

SELECT d::VARCHAR=v AS varchar_comp FROM (VALUES (DATE '2000-01-01 00:00:00', '2000-01-01 12:00:00')) t(d, v);
┌──────────────┐
│ varchar_comp │
│   boolean    │
├──────────────┤
│ false        │
└──────────────┘
SELECT d=v::DATE AS date_comp FROM (VALUES (DATE '2000-01-01 00:00:00', '2000-01-01 12:00:00')) t(d, v);
┌───────────┐
│ date_comp │
│  boolean  │
├───────────┤
│ true      │
└───────────┘

The cast to date truncates the timestamp values (i.e. turns 2000-01-01 12:00:00 into 2000-01-01).

This can become even more confusing with greater/less than comparisons, for example:

SELECT DATE '2023-12-11 15:54:45.119' < '2023-12-11 00:00:00' AS r;
┌─────────┐
│    r    │
│ boolean │
├─────────┤
│ true    │
└─────────┘

The reason for this result is that 2023-12-11 15:54:45.119 gets truncated to 2023-12-11 in the date conversion. Then a string comparison is performed, and '2023-12-11' < '2023-12-11 00:00:00' is true as a string comparison.

Examples

Here are some examples of the new rule in action - all of these examples now match Postgres' behavior with the exception of DuckDB being more lenient on equality comparisons with VARCHAR columns:

Date - Literal Comparison

Old

SELECT DATE '2023-12-11 15:54:45.119' < '2023-12-11 00:00:00' AS r;
┌─────────┐
│    r    │
│ boolean │
├─────────┤
│ true    │
└─────────┘

New

SELECT DATE '2023-12-11 15:54:45.119' < '2023-12-11 00:00:00' AS r;
┌─────────┐
│    r    │
│ boolean │
├─────────┤
│ false   │
└─────────┘

Date - Varchar Comparison

Old

WITH cte AS (SELECT '2023-12-11 00:00:00' AS d)
SELECT DATE '2023-12-11 15:54:45.119' < d AS r FROM cte;
┌─────────┐
│    r    │
│ boolean │
├─────────┤
│ true    │
└─────────┘

New

WITH cte AS (SELECT '2023-12-11 00:00:00' AS d)
SELECT DATE '2023-12-11 15:54:45.119' < d AS r FROM cte;
-- Binder Error: Cannot compare values of type DATE and type VARCHAR - an explicit cast is required
-- LINE 2: SELECT DATE '2023-12-11 15:54:45.119' < d AS r FROM cte;

Integer - Literal IN vs Comparison

Old

SELECT 1 IN ('01') AS in_res;
┌─────────┐
│ in_res  │
│ boolean │
├─────────┤
│ false   │
└─────────┘
SELECT 1='01' AS comp_res;
┌──────────┐
│ comp_res │
│ boolean  │
├──────────┤
│ true     │
└──────────┘

New

SELECT 1 IN ('01') AS in_res;
┌─────────┐
│ in_res  │
│ boolean │
├─────────┤
│ true    │
└─────────┘
SELECT 1='01' AS comp_res;
┌──────────┐
│ comp_res │
│ boolean  │
├──────────┤
│ true     │
└──────────┘

Integer - Varchar IN vs Comparison

Old

WITH cte AS (SELECT '01' AS s)
SELECT 1 IN (s) AS in_res FROM cte;
┌─────────┐
│ in_res  │
│ boolean │
├─────────┤
│ false   │
└─────────┘
WITH cte AS (SELECT '01' AS s)
SELECT 1=s AS comp_res FROM cte;
┌──────────┐
│ comp_res │
│ boolean  │
├──────────┤
│ true     │
└──────────┘

New

WITH cte AS (SELECT '01' AS s)
SELECT 1 IN (s) AS in_res FROM cte;
┌─────────┐
│ in_res  │
│ boolean │
├─────────┤
│ true    │
└─────────┘

WITH cte AS (SELECT '01' AS s)
SELECT 1=s AS comp_res FROM cte;
┌──────────┐
│ comp_res │
│ boolean  │
├──────────┤
│ true     │
└──────────┘

Element Extraction

Old

-- auto-cast to varchar causes this to succeed
SELECT (42)[1] AS r;
┌─────────┐
│    r    │
│ varchar │
├─────────┤
│ 4       │
└─────────┘

New

SELECT (42)[1] AS r;
Error: Binder Error: No function matches the given name and argument types 'array_extract(INTEGER, INTEGER)'. You might need to add explicit type casts.
	Candidate functions:
	array_extract(ANY[], BIGINT) -> ANY
	array_extract(VARCHAR, BIGINT) -> VARCHAR
	array_extract(STRUCT, VARCHAR) -> ANY

[manticore-projects]

In my opinion this is a rather unfortunate choice since Postgres' requirement for explicitly casting everything is its weakest spot. Literally every other RDBMS does this better.

Why does it matter? Imagine an application supporting multiple RDBMS based on SQL:2016 standard. You actually get quite far with that on various RDBMS, unless you start querying on Postgres -- suddenly you need to explicitly cast every parameter and NULL test.

Forcing:

select length(42:VARCHAR)

is just not good. The DB knows that a VARCHAR is expected and it knows the data type provided. It costs nothing to cast it, but forcing me to write out this cast for DuckDB/Postgres feels really annoying and seriously limits chances to replace an existing RDBMS with DuckDB (I myself would rather stick with H2 simply for the reason that it gets this right, despite many other disadvantages).

So please, at least consider guaranteeing the old_implicit_casting option at least. potentially to be deprecated in the future) does not instill any courage here.

Your best chance of support contracts comes from migrating Oracle and MS SQL Server DBs to DuckDB. As lower the barrier, as better your chances. At least to me this is pretty much a showstopper because we would have to rewrite every single query and report -- even when already SQL:2016 compliant.