Skip to content

Conversation

@sumitmsft
Copy link
Contributor

Added technical details about the description of the driver.

  • Summary of the driver
  • Key Features
  • Example\Sample

@sumitmsft sumitmsft merged commit 3d4079e into main Feb 11, 2025
1 check was pending
@sumitmsft
Copy link
Contributor Author

sumitmsft commented Feb 11, 2025 via email

bewithgaurav added a commit that referenced this pull request Nov 10, 2025
- Replace placeholder allocation (rows.append(py::none())) with direct PyList_New
- Use PyList_Append + Py_DECREF for cleaner refcount management
- Eliminates double allocation: no more placeholder → real row replacement
- Uses rowWrapper for pybind11 compatibility while row is raw PyObject*
- Removed final rows[initialSize + i] = row assignment (already appended)
- Expected improvement: 15-20ms for 10K rows, reduces memory allocator pressure
- Added PERF_TIMER for rows_append to measure append overhead
bewithgaurav added a commit that referenced this pull request Nov 10, 2025
- Replaced placeholder allocation (py::none()) with direct PyList_New + PyList_Append
- Pre-allocate all row lists before populating them
- Uses PyList_GET_ITEM to retrieve pre-allocated rows (no bounds checking)
- Wrap raw PyObject* in py::list for pybind11 compatibility
- Eliminates redundant row assignment at end of loop (row already in list)
- Renamed PERF_TIMER from allocate_placeholder_rows to rows_append
- Reduces memory allocator calls from O(2×rows) to O(rows)
bewithgaurav added a commit that referenced this pull request Nov 10, 2025
Problem:
--------
Row creation and assignment had multiple layers of overhead:
1. Per-row allocation: py::list(numCols) creates pybind11 wrapper for each row
2. Cell assignment: row[col-1] = value uses pybind11 operator[] with bounds checking
3. Final assignment: rows[i] = row uses pybind11 list assignment with refcount overhead
4. Fragmented allocation: 1,000 separate py::list() calls instead of batch allocation

For 1,000 rows: ~30-50 CPU cycles × 1,000 = 30K-50K wasted cycles

Solution:
---------
Replace pybind11 wrappers with direct Python C API throughout:

1. Row creation: PyList_New(numCols) instead of py::list(numCols)
2. Cell assignment: PyList_SET_ITEM(row, col-1, value) instead of row[col-1] = value
3. Final assignment: PyList_SET_ITEM(rows.ptr(), i, row) instead of rows[i] = row

This completes the transition to direct Python C API started in OPT #2.

Changes:
--------
- Replaced py::list row(numCols) → PyObject* row = PyList_New(numCols)
- Updated all NULL/SQL_NO_TOTAL handlers to use PyList_SET_ITEM
- Updated all zero-length data handlers to use direct Python C API
- Updated string handlers (SQL_CHAR, SQL_WCHAR) to use PyList_SET_ITEM
- Updated complex type handlers (DECIMAL, DATETIME, DATE, TIME, TIMESTAMPOFFSET, GUID, BINARY)
- Updated final row assignment to use PyList_SET_ITEM(rows.ptr(), i, row)

All cell assignments now use direct Python C API:
- Numeric types: Already done in OPT #2 (PyLong_FromLong, PyFloat_FromDouble, etc.)
- Strings: PyUnicode_FromStringAndSize, PyUnicode_FromString
- Binary: PyBytes_FromStringAndSize
- Complex types: .release().ptr() to transfer ownership

Impact:
-------
- ✅ Eliminates pybind11 wrapper overhead for row creation
- ✅ No bounds checking in hot loop (PyList_SET_ITEM is a macro)
- ✅ Clean reference counting (objects created with refcount=1, transferred to list)
- ✅ Consistent with OPT #2 (entire row/cell management via Python C API)
- ✅ Expected 5-10% improvement (smaller than OPT #3, but completes the stack)

All type handlers now bypass pybind11 for maximum performance.
bewithgaurav added a commit that referenced this pull request Nov 10, 2025
…ild fix)

Same issue as OPT #3 - Windows compiler treats warnings as errors (/WX).
The columnSize variable was extracted but unused in SQL_CHAR and SQL_WCHAR
cases after OPTIMIZATION #4.

Changes:
--------
- Removed unused 'SQLULEN columnSize' from SQL_CHAR/VARCHAR/LONGVARCHAR
- Removed unused 'SQLULEN columnSize' from SQL_WCHAR/WVARCHAR/WLONGVARCHAR
- Retained fetchBufferSize and isLob which are actively used

Fixes Windows build error C4189 treated as error C2220.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants