Skip to content

cursor.executemany throws RuntimeError when decimals change signs #557

@taehoon-song

Description

@taehoon-song

Describe the bug

I'm loading a polars DataFrame and one of the columns has negative and positive decimals. If there is a sign change, the driver throws a RuntimeError. I've tried to narrow it down, but the behavior is a little inconsistent.

Exception message: RuntimeError: Input string exceeds allowed column size at parameter index 0
Stack trace:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    df_to_sql(table_name="pytest", df=df, credential=cred)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tsong\repo\pam\src\pam\sql\write_db.py", line 37, in df_to_sql
    cursor.executemany(stmt, df.rows())
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\tsong\repo\pam\.venv\Lib\site-packages\mssql_python\cursor.py", line 2338, in executemany
    ret = ddbc_bindings.SQLExecuteMany(
        self.hstmt, operation, columnwise_params, parameters_type, row_count, encoding_settings
    )

At the moment, I work around this by casting all Decimal columns to Strings.

df = df.with_columns(pl.col(pl.Decimal).cast(pl.String).name.keep())

To reproduce

Include a complete code listing (or project/solution) that we can run to reproduce the issue.

Partial code listings, or multiple fragments of code, will slow down our response or cause us to push the issue back to you to provide code to reproduce the issue.

# I have a table named [pytest]
# CREATE TABLE [dbo].[pytest] ([col_1] [decimal](28, 14))

import polars as pl
import mssql_python as mssql
from decimal import Decimal

df = pl.DataFrame(
        {
            "col_1": [
                Decimal("-0.1"),
                Decimal("1.0"),
            ]
        }
    )

stmt = "INSERT INTO [pytest] ([col_1]) VALUES (?)"

connection_url = "Server=<SERVER_ADDRESS>;Database=<DATABASE>;Encrypt=yes;Authentication=ActiveDirectoryDefault;"
conn = mssql.connect(connection_url)
cursor = conn.cursor()
cursor.executemany(stmt, df.rows())
cursor.commit()
conn.close()

Expected behavior

Insert should be successful.

Further technical details

Python version: 3.14.4
SQL Server version: Azure SQL Managed Instance (locked to SQL Server 2022)
Operating system: Windows 11

Additional context
There's definitely something weird going on with how the driver is inserting decimals with executemany. The examples below all work normally:

# These dataframes all work
df = pl.DataFrame(
        {
            "col_1": [
                Decimal("-1.1"),
                Decimal("1.0"),
            ]
        }
    )

df = pl.DataFrame(
        {
            "col_1": [
                Decimal("-0.1"),
                Decimal("-1.0"),
            ]
        }
    )

df = pl.DataFrame(
        {
            "col_1": [
                Decimal("0.1"),
                Decimal("1.0"),
            ]
        }
    )

df = pl.DataFrame(
        {
            "col_1": [
                Decimal("0.1"),
                Decimal("-1.0"),
            ]
        }
    )

Weirdly enough, the order also affects the outcome:

# This one works
df = pl.DataFrame(
        {
            "col_1": [
                Decimal("-0.1"),
                Decimal("0.1"),
            ]
        }
    )

# But this one doesn't
df = pl.DataFrame(
        {
            "col_1": [
                Decimal("0.1"),
                Decimal("-0.1"),
            ]
        }
    )

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage neededFor new issues, not triaged yet.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions