Fix | Enable SqlBulkCopy to target tables in Azure Synapse Analytics dedicated SQL pools

[edwardneal]

Description

This is designed to fix #4149, which blocked SqlBulkCopy from being used to copy data to an Azure Synapse dedicated SQL pool.

The original issue emerged as a result of #3590, which changed the query used to gather column metadata from simply running SELECT * FROM [Source] to executing a SELECT statement with a dynamically-constructed list of columns.

I originally had a single @Column_Names variable and I built a list of columns using a query similar to:

SELECT @Column_Names = COALESCE(@Column_Names + ', ', '') + QUOTENAME([name])
FROM [sys].[all_columns]
WHERE [object_id] = @Object_ID
ORDER BY [column_id] ASC

On an Azure Synapse dedicated SQL pool, this doesn't work. The error below is thrown:

Msg 104473, Level 16, State 1, Line 11
A variable that has been assigned in a SELECT statement cannot be included in a expression or assignment when used in conjunction with a from clause.

To fix this, I've replaced it with a reference to STRING_AGG.

Muddying the water slightly is legacy compatibility, since STRING_AGG was only introduced in SQL 2017 and SQL 2016 remains a supported version. I can't directly compare version numbers: SQL 2016 is version 13.x, while Synapse reports version 10.x. To work around this, I just evaluate SERVERPROPERTY('EngineEdition'), since a value of 6 indicates Azure Synapse Analytics.

We also vary the WHERE clause based upon whether or not the SQL Graph columns exist. To avoid generating a mishmash of different criteria, I've decided to break the query into its SELECT, WHERE and ORDER BY components and join them up at the end of the process.

Issues

Fixes #4149. This will need to be backported to 7.0 though.

Testing

Automated tests continue to pass, against both SQL 2016 and SQL 2025.
Manual testing against an Azure Synapse dedicated SQL pool works.

It's worth noting that we don't currently have a test job for this, that most of the existing SqlBulkCopy tests are exempted from running against this target, and that when I override the exemption I encounter a handful of unrelated issues. It might be worth adding a test environment so that we can get CI coverage.

[Copilot]

Pull request overview

Fixes SqlBulkCopy destination-table metadata discovery for Azure Synapse dedicated SQL pools by avoiding the unsupported “self-referencing variable assignment” concatenation pattern and using a STRING_AGG-based query path.

Changes:

• Update the dynamic column-name query generation in SqlBulkCopy.CreateInitialQuery() to use STRING_AGG when SERVERPROPERTY('EngineEdition') = 6 (Synapse).
• Refactor the dynamic SQL construction into SELECT/FILTER/SORT components to keep filtering logic consistent across branches.
• Update ManualTests expectations for connection statistics impacted by the revised metadata query.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlBulkCopy.cs	Adjusts metadata query generation to support Synapse (EngineEdition=6) via STRING_AGG and refactors query assembly.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/SqlBulkCopyTest/CopyAllFromReader.cs	Updates expected SelectCount/SelectRows stats after the metadata-query change.

[paulmedynski]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[paulmedynski]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[paulmedynski]

@edwardneal - Our PR pipelines don't test against SQL Server 2016 or 2017 to keep our resource footprint reasonable. Those tests only run as part of the CI pipelines, which occur post-merge to main. I can't manually run the CI pipelines against a fork, so we will have to wait until this merges to see how things work against 2016 and 2017. This won't be an issue with the new pipeline structure under development.

[edwardneal]

Thanks. Will the new pipelines also have testing for a Synapse dedicated SQL pool?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.13%. Comparing base (52c7149) to head (5397173).
⚠️ Report is 6 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (52c7149) and HEAD (5397173). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (52c7149)	HEAD (5397173)
CI-SqlClient	1	0

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4176      +/-   ##
==========================================
- Coverage   74.27%   65.13%   -9.15%     
==========================================
  Files         279      274       -5     
  Lines       42980    65815   +22835     
==========================================
+ Hits        31922    42866   +10944     
- Misses      11058    22949   +11891     

Flag	Coverage Δ
CI-SqlClient	?
PR-SqlClient-Project	65.13% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[paulmedynski]

@edwardneal - Not planned imminently. We're also missing integration coverage for Fabric. Definitely on the radar, just not quite yet.