FEAT: BULK COPY #125

jahnvi480 · 2025-07-04T11:25:04Z

ADO Work Item Reference

AB#34946

Summary

This pull request introduces significant enhancements to the mssql_python package, primarily focused on implementing and supporting the Bulk Copy Program (BCP) functionality for SQL Server. The changes include the addition of a new BCPClient class, updates to constants and data types, and new classes to configure BCP operations. These updates aim to provide a Pythonic interface for bulk data operations, improving usability and flexibility.

BCP Functionality Implementation:

Addition of BCPClient class: Introduced a new class in mssql_python/bcp_main.py to handle bulk copy operations using SQL Server's BCP utility. The class provides methods for initializing and executing BCP operations, with robust error handling and support for both file-based and in-memory operations.

Configuration Enhancements:

New BCP configuration classes: Added BindData, ColumnFormat, and BCPOptions classes in mssql_python/bcp_options.py to define data bindings, column formats, and overall BCP operation settings. These classes enable detailed configuration of bulk copy operations, including data types, terminators, and column mappings.

Constants and Accessibility:

Expanded constants for BCP data types: Updated mssql_python/__init__.py to include direct variables for BCP data type constants (e.g., SQLTEXT, SQLVARCHAR, SQLBIGCHAR). This provides easier access to commonly used SQL Server data types.
Import pooling functionality: Added import for PoolingManager in mssql_python/__init__.py to ensure pooling functionality is accessible.

… jahnvi/bulk_copy_options

…ft/mssql-python into jahnvi/bulk_copy_main_apis

…soft/mssql-python into jahnvi/bulk_copy_wrapper_cpp

Copilot

Pull Request Overview

This PR adds full support for SQL Server’s Bulk Copy Program (BCP) in the mssql_python package by introducing new client classes, configuration options, and C++ bindings.

Introduces BCPClient, BCPOptions, ColumnFormat, and BindData for Python-side BCP workflows
Extends ddbc_bindings with BCP API function pointers and implements BCPWrapper in C++
Adds integration and unit tests covering format files, column formatting, queries, and in-memory BCP

Reviewed Changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tests/testBCP/test_002_bcpMain.py	New integration tests exercising BCPClient end-to-end
tests/testBCP/test_001_bulkOptions.py	Unit tests for `BCPOptions`, `ColumnFormat`, and `BindData`
mssql_python/pybind/ddbc_bindings.h	Declares BCP API function pointer types
mssql_python/pybind/ddbc_bindings.cpp	Loads BCP API pointers and exports new `BCPWrapper` class
mssql_python/pybind/connection/connection.h	Exposes DBC handle accessor for BCP initialization
mssql_python/pybind/connection/connection.cpp	Applies `SQL_COPT_SS_BCP` attribute before connect
mssql_python/pybind/bcp/bcp_wrapper.h	Declares `BCPWrapper` interface
mssql_python/pybind/bcp/bcp_wrapper.cpp	Implements BCP logic, column binding, and error handling
mssql_python/pybind/CMakeLists.txt	Includes `bcp_wrapper.cpp` in the build
mssql_python/constants.py	Adds `BCPControlOptions` and `BCPDataTypes` enums
mssql_python/bcp_options.py	Defines Python dataclasses for BCP options and bindings
mssql_python/bcp_main.py	Implements `BCPClient` orchestration logic
mssql_python/init.py	Exposes BCP classes and constants in the package API

Comments suppressed due to low confidence (1)

tests/testBCP/test_002_bcpMain.py:1

[nitpick] Consider adding negative or error‐path tests (e.g., invalid format file, unsupported direction) to cover exceptional flows in BCPClient.

import os

Copilot · 2025-07-04T11:28:12Z

mssql_python/pybind/bcp/bcp_wrapper.cpp

+        + ", direction code: " + std::to_string(dir_code));
+    // Call BCPInitW with the correct parameters
+    SQLRETURN ret = BCPInitW_ptr(_hdbc, pTable, pDataFile, pErrorFile, dir_code);
+    LOG("BCPWrapper: HELLOOOO " + std::to_string(ret));


Remove or replace this informal debug log (HELLOOOO) with a more descriptive message or eliminate it if no longer needed.

Suggested change

LOG("BCPWrapper: HELLOOOO " + std::to_string(ret));

LOG("BCPWrapper: BCPInitW_ptr returned value: " + std::to_string(ret));

Copilot · 2025-07-04T11:28:12Z

mssql_python/pybind/bcp/bcp_wrapper.cpp

+    std::string value_utf8 = py::cast(value).cast<std::string>();
+    // Fix the BCPHINTS issue: use the wide string directly rather than converting to narrow
+    SQLRETURN ret = BCPControlW_ptr(_hdbc, it->second.option_code, (LPVOID)value_utf8.c_str());


You're passing a narrow char* to the wide‐string BCPControlW function. You should convert value to std::wstring and pass its c_str() so the API receives a LPCWSTR.

Suggested change

std::string value_utf8 = py::cast(value).cast<std::string>();

// Fix the BCPHINTS issue: use the wide string directly rather than converting to narrow

SQLRETURN ret = BCPControlW_ptr(_hdbc, it->second.option_code, (LPVOID)value_utf8.c_str());

// Fix the BCPHINTS issue: use the wide string directly rather than converting to narrow

SQLRETURN ret = BCPControlW_ptr(_hdbc, it->second.option_code, (LPVOID)value.c_str());

Copilot · 2025-07-04T11:28:12Z

mssql_python/pybind/bcp/bcp_wrapper.cpp

+    if (SQLGetConnectAttr_ptr) {
+        SQLINTEGER bcp_attr_value = 0;
+        SQLINTEGER string_length = 0;
+        std::cout << "BCPWrapper: Checking BCP attribute just before initialization." << std::endl;


Replace direct std::cout usage with the existing LOG(...) utility to keep all diagnostic output consistent and configurable.

Suggested change

std::cout << "BCPWrapper: Checking BCP attribute just before initialization." << std::endl;

LOG("BCPWrapper: Checking BCP attribute just before initialization.");

Copilot · 2025-07-04T11:28:12Z

mssql_python/pybind/ddbc_bindings.h

+typedef SQLRETURN (SQL_API* BCPColFmtWFunc)(SQLHDBC, INT, INT, INT, DBINT, LPCBYTE, INT, INT);
+typedef SQLRETURN  (SQL_API* BCPExecFunc)(SQLHDBC, DBINT*); 
+typedef SQLRETURN (SQL_API* BCPDoneFunc)(SQLHDBC);
+typedef SQLRETURN (SQL_API* BCPBindFunc)(HDBC, LPCBYTE, INT, DBINT, LPCBYTE, INT, INT, INT);


The first parameter should be SQLHDBC instead of the undefined HDBC type to match the BCP API signature.

Suggested change

typedef SQLRETURN (SQL_API* BCPBindFunc)(HDBC, LPCBYTE, INT, DBINT, LPCBYTE, INT, INT, INT);

typedef SQLRETURN (SQL_API* BCPBindFunc)(SQLHDBC, LPCBYTE, INT, DBINT, LPCBYTE, INT, INT, INT);

Copilot · 2025-07-04T11:28:13Z

mssql_python/pybind/ddbc_bindings.cpp

@@ -716,6 +728,17 @@ DriverHandle LoadDriverOrThrowException() {

    SQLGetDiagRec_ptr = GetFunctionPointer<SQLGetDiagRecFunc>(handle, "SQLGetDiagRecW");

+    // Load BCP functions
+    BCPInitW_ptr = (BCPInitWFunc)GetProcAddress(handle, "bcp_initW");
+    BCPControlW_ptr = (BCPControlWFunc)GetProcAddress(handle, "bcp_control");


You’ve bound BCPControlWFunc but are loading the ANSI entry bcp_control instead of the wide‐char bcp_controlW. Update to GetProcAddress(handle, "bcp_controlW").

Suggested change

BCPControlW_ptr = (BCPControlWFunc)GetProcAddress(handle, "bcp_control");

BCPControlW_ptr = (BCPControlWFunc)GetProcAddress(handle, "bcp_controlW");

Copilot · 2025-07-04T11:28:13Z

mssql_python/bcp_main.py

+            )
+
+        self.wrapper = BCPWrapper(connection._conn)
+        print(f"connection: {connection._conn}")


This debug print should be removed or replaced with a logger call to avoid unintended console output in production.

Suggested change

print(f"connection: {connection._conn}")

logger.debug(f"connection: {connection._conn}")

Copilot · 2025-07-04T11:28:13Z

mssql_python/bcp_main.py

+                        col_fmt_obj.file_col,
+                        col_fmt_obj
+                    )
+                    print("col_fmt_obj: %s", col_fmt_obj)


Remove this ad-hoc print. Use the logging framework (logger.debug(...)) if you need to inspect col_fmt_obj.

Suggested change

print("col_fmt_obj: %s", col_fmt_obj)

logger.debug("col_fmt_obj: %s", col_fmt_obj)

Copilot · 2025-07-04T11:28:14Z

mssql_python/bcp_main.py

+                # self.wrapper.finish()
+                # self.wrapper.close()


Commented-out cleanup calls (finish/close) may lead to leaked BCP resources. Ensure you explicitly finalize and close the wrapper in all paths.

Suggested change

# self.wrapper.finish()

# self.wrapper.close()

self.wrapper.finish()

self.wrapper.close()

Copilot · 2025-07-04T11:28:14Z

mssql_python/pybind/bcp/bcp_wrapper.cpp

+            LOG("BCPWrapper: Terminator bytes provided: " + py::cast(terminator_str_holder).cast<std::string>());
+            pTerminator = reinterpret_cast<const BYTE*>(terminator_str_holder.data());
+            LOG("BCPWrapper: Terminator pointer: " + std::to_string(reinterpret_cast<uintptr_t>(pTerminator)));
+            LOG("BCPWRapper: Terminator pointer type: " + std::string(typeid(pTerminator).name()));


Typo in log label: BCPWRapper should be BCPWrapper.

Suggested change

LOG("BCPWRapper: Terminator pointer type: " + std::string(typeid(pTerminator).name()));

LOG("BCPWrapper: Terminator pointer type: " + std::string(typeid(pTerminator).name()));

Jahnvi Thakkar added 17 commits June 25, 2025 18:15

FEAT: Bulk Copy Options

9738ac6

FEAT: Bulk Copy Options

78627bc

FEAT: Adding Bulk Copy Python APIs

ed529ba

FEAT: Bulk Copy Wrapper

6db1383

Merge branch 'main' of https://github.com/microsoft/mssql-python into…

fe2fa67

… jahnvi/bulk_copy_options

FEAT: Adding binding options

ae1f289

FEAT: Removing pylint comments

e312279

Merge branch 'jahnvi/bulk_copy_options' of https://github.com/microso…

4aadf2f

…ft/mssql-python into jahnvi/bulk_copy_main_apis

FEAT: Adding logic for bcp_bind

f1602f7

FEAT: Adding logic for bcp_bind

00b2734

FEAT: Resolving merge conflicts

95950ea

FEAT: Pushing the required changes for bcp bind

3a5d5db

Resolving conflicts

6266120

Resolving conflicts

53e7f6a

Resolving conflicts

13eb539

Merge branch 'jahnvi/bulk_copy_main_apis' of https://github.com/micro…

2c65bf6

…soft/mssql-python into jahnvi/bulk_copy_wrapper_cpp

Resolving comments

8ab840d

Copilot AI review requested due to automatic review settings July 4, 2025 11:25

github-actions bot added the pr-size: large Substantial code update label Jul 4, 2025

Copilot AI reviewed Jul 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

FEAT: BULK COPY #125

FEAT: BULK COPY #125

Uh oh!

jahnvi480 commented Jul 4, 2025 •

edited by azure-boards bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Copilot AI Jul 4, 2025

Uh oh!

Uh oh!

	LOG("BCPWrapper: HELLOOOO " + std::to_string(ret));
	LOG("BCPWrapper: BCPInitW_ptr returned value: " + std::to_string(ret));

	std::cout << "BCPWrapper: Checking BCP attribute just before initialization." << std::endl;
	LOG("BCPWrapper: Checking BCP attribute just before initialization.");

	typedef SQLRETURN (SQL_API* BCPBindFunc)(HDBC, LPCBYTE, INT, DBINT, LPCBYTE, INT, INT, INT);
	typedef SQLRETURN (SQL_API* BCPBindFunc)(SQLHDBC, LPCBYTE, INT, DBINT, LPCBYTE, INT, INT, INT);

	BCPControlW_ptr = (BCPControlWFunc)GetProcAddress(handle, "bcp_control");
	BCPControlW_ptr = (BCPControlWFunc)GetProcAddress(handle, "bcp_controlW");

	print(f"connection: {connection._conn}")
	logger.debug(f"connection: {connection._conn}")

	print("col_fmt_obj: %s", col_fmt_obj)
	logger.debug("col_fmt_obj: %s", col_fmt_obj)

	LOG("BCPWRapper: Terminator pointer type: " + std::string(typeid(pTerminator).name()));
	LOG("BCPWrapper: Terminator pointer type: " + std::string(typeid(pTerminator).name()));

FEAT: BULK COPY #125

Are you sure you want to change the base?

FEAT: BULK COPY #125

Uh oh!

Conversation

jahnvi480 commented Jul 4, 2025 • edited by azure-boards bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ADO Work Item Reference

Summary

BCP Functionality Implementation:

Configuration Enhancements:

Constants and Accessibility:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jahnvi480 commented Jul 4, 2025 •

edited by azure-boards bot

Loading