Implement joins, unions, subqueries and extend tests

[alex-clickhouse]

Summary

• INNER / LEFT / CROSS JOIN, with LEFT JOIN now returning real null (not column defaults) via injected set_join_use_nulls=1
• Correlated subqueries: IN, EXISTS/Any, All, scalar subqueries in projections
• Set operations: Concat, Union, Intersect, Except
• LINQ joins against in-memory collections (Join(localIds, ...), Contains(localList))

SQL generator overrides (ClickHouseQuerySqlGenerator)

Each addresses a documented dialect quirk:

• GenerateSetOperation — emit explicit ALL/DISTINCT (bare UNION is rejected)
• VisitSqlFunction — CAST(COUNT(...) AS Int32/Int64) so set operations type-match
• VisitOrdering — NULLS FIRST/LAST for nullable expressions only (avoids reordering NaN in non-nullable floats)
• VisitNonNullableScalarSubquery — wrap with ifNull(..., 0) (ClickHouse returns NULL for no-match aggregates)
• GenerateValues — rewrite VALUES (...) as SELECT ... UNION ALL SELECT ... (ClickHouse rejects VALUES as a subquery source)

Type mapping changes

The ValuesExpression path requires three coordinated changes:

• ClickHouseInt32TypeMapping (new) — inherits from EF Core's IntTypeMapping to satisfy a hard cast in SqlNullabilityProcessor. Other integers continue to use the shared mapping.
• ClickHouseArrayTypeMapping — wires ElementTypeMapping into CoreTypeMappingParameters, hidden for composite elements (nested arrays, tuples, variants) to avoid the model validator's "primitive collection of a primitive collection" rejection.
• ClickHouseTypeMappingSource — prioritizes FindArrayMapping when EF supplies ElementTypeMapping + a recognized collection CLR type (T[], List<T>, IEnumerable<T>, etc.).

Tests

• New conformance suites: NorthwindJoinQueryClickHouseTest, NorthwindSetOperationsQueryClickHouseTest (skips documented inline)
• Northwind.ClickHouse.sql: 42 columns → Nullable(String), ~712 empty strings → NULL so LEFT JOIN assertions see real NULLs

[codecov]

Codecov Report

❌ Patch coverage is 88.77551% with 22 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ouse/Query/Internal/ClickHouseQuerySqlGenerator.cs	88.23%	4 Missing and 4 partials ⚠️
...se/Storage/Internal/ClickHouseTypeMappingSource.cs	88.23%	2 Missing and 4 partials ⚠️
...age/Internal/Mapping/ClickHouseInt32TypeMapping.cs	72.72%	3 Missing ⚠️
...age/Internal/Mapping/ClickHouseArrayTypeMapping.cs	89.47%	1 Missing and 1 partial ⚠️
...se/Storage/Internal/ClickHouseDataSourceManager.cs	90.90%	0 Missing and 1 partial ⚠️
...Storage/Internal/ClickHouseRelationalConnection.cs	95.00%	0 Missing and 1 partial ⚠️
...age/Internal/Mapping/EnumerableToArrayConverter.cs	90.90%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

This PR extends the ClickHouse EF Core provider to support more relational query patterns (joins, set operations, and correlated subqueries), aligns type mapping to EF Core’s ValuesExpression/nullability processing expectations, and updates/expands test coverage (including Northwind conformance suites) to reflect full Northwind data and ClickHouse-specific behaviors.

Changes:

• Add/override SQL generation for set operations, COUNT() typing, NULL ordering, scalar subquery null behavior, and VALUES-source rewriting.
• Adjust type mappings and mapping resolution for int and array/collection parameters used in inline VALUES expansion.
• Update and expand functional/integration tests, including new Northwind join/set-ops suites and updated expected results.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/EFCore.ClickHouse.Tests/JoinAndSetOperationTests.cs	Adds integration tests for joins, correlated subqueries, and set operations against a seeded ClickHouse schema.
test/EFCore.ClickHouse.FunctionalTests/TestUtilities/ClickHouseTestStore.cs	Forces test contexts to use the connection-string path and appends set_join_use_nulls=1 to ensure correct LEFT JOIN null semantics.
test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindWhereQueryClickHouseTest.cs	Updates assertions to match full Northwind dataset results.
test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindSetOperationsQueryClickHouseTest.cs	Introduces ClickHouse-specific set-operations conformance suite overrides/skips for known server limitations.
test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs	Introduces ClickHouse-specific join conformance suite overrides for unsupported APPLY patterns and known translation limitations.
test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindFunctionsQueryClickHouseTest.cs	Updates expected results for string function tests to match full dataset.
test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindCompiledQueryClickHouseTest.cs	Updates compiled query expectations to match full dataset.
test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindAsTrackingQueryClickHouseTest.cs	Updates tracking test expectations to match full dataset size.
test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindAsNoTrackingQueryClickHouseTest.cs	Updates no-tracking test expectations to match full dataset size.
test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindAggregateOperatorsQueryClickHouseTest.cs	Updates aggregate expectations (count/sum) to match full dataset values.
src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseInt32TypeMapping.cs	Adds an IntTypeMapping-derived mapping to satisfy EF Core hard-cast paths and safe materialization for int.
src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs	Wires element type mapping into CoreTypeMappingParameters and conditionally hides it for composite elements to satisfy EF model validation.
src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs	Prioritizes array mapping when EF provides ElementTypeMapping and broadens collection CLR type handling for inline VALUES scenarios.
src/EFCore.ClickHouse/Storage/Internal/ClickHouseRelationalConnection.cs	Ensures default session settings are applied even when constructing a connection directly from a connection string.
src/EFCore.ClickHouse/Storage/Internal/ClickHouseDataSourceManager.cs	Normalizes connection strings via EnsureDefaultSettings before pooling/caching data sources; introduces shared settings injection helper.
src/EFCore.ClickHouse/Query/Internal/ClickHouseQuerySqlGenerator.cs	Adds ClickHouse-specific SQL generation for set ops, COUNT casting, NULL ordering, scalar subquery behavior, and VALUES rewriting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.