-
Notifications
You must be signed in to change notification settings - Fork 25
Open
Labels
triage doneIssues that are triaged by dev team and are in investigation.Issues that are triaged by dev team and are in investigation.under development
Description
### Is your feature request related to a problem? Please describe.
Multi-statement SQL queries with temporary tables execute successfully but return empty result sets, while the same queries work correctly in SSMS(SQL Server Management Studio) and pyodbc. This occurs because each `cursor.execute()` call in mssql-python creates separate execution contexts, making local temp tables (`#temp`) created in one statement invisible in subsequent statements.
**Example that fails silently:**
```sql
-- Works in SSMS, returns empty results in mssql-python
SELECT col1, col2 INTO #temp FROM table1 WHERE condition
SELECT * FROM #temp JOIN table2 ON #temp.col1 = table2.col1
This prevents migration from legacy SQL Server applications and breaks existing complex reporting queries, forcing developers to completely rewrite working SQL.
Describe the solution you'd like
Add transparent temporary table support to the cursor.execute()
method using pyodbc's proven approach:
- Auto-detection: Automatically identify multi-statement queries containing local temp tables
- SET NOCOUNT ON: Add
SET NOCOUNT ON;
prefix to eliminate intermediate result sets that interfere with multi-statement execution - Batch execution: Execute the entire query as a single batch to preserve session scope for temp table visibility
- Zero API changes: Existing code works without modification
Benefits:
- Transparent - no API changes required
- Reliable - uses proven pyodbc approach
- Compatible - works with existing SQL without modification
- Minimal impact - adds approximately 20 lines of code
Describe alternatives you've considered
We evaluated and tested multiple approaches:
-
CTE Conversion (Tested - Works but Complex)
- Convert
SELECT ... INTO #temp
statements to Common Table Expressions (CTEs) - Replace temp table references with CTE names in subsequent queries
- ✅ Works: Successfully converts simple temp table patterns
- ❌ Complex: Requires extensive SQL parsing and transformation logic
- ❌ Maintenance burden: Custom conversion logic needs constant updates for edge cases
- ❌ Logic changes: CTEs have different scope and behavior than temp tables
- Convert
-
Statement Splitting (Tested - Partial Success)
- Split multi-statement SQL into individual statements
- Execute statements sequentially within the same transaction
- ✅ Works: Maintains temp table visibility between statements
- ❌ Session scope issues: Still suffers from mssql-python's execution context isolation
- ❌ Complex parsing: Difficult to reliably split complex SQL statements
- ❌ Parameter handling: Complications with parameter distribution across statements
-
Global Temp Tables (Works but Intrusive)
- Convert local temp tables (
#temp
) to global temp tables (##temp
) - ✅ Works: Global temp tables are visible across execution contexts
- ❌ Breaking changes: Requires modifying user SQL
- ❌ Cleanup complexity: Requires manual cleanup of global temp tables
- ❌ Concurrency issues: Global temp tables can conflict between parallel operations
- Convert local temp tables (
-
Custom Wrapper Methods (Works but Breaking)
- Add new methods like
execute_multistatement()
orexecute_with_temp_tables()
- ✅ Works: Can handle temp tables with specialized logic
- ❌ Breaking changes: Requires application code changes
- ❌ Adoption barrier: Developers must learn new API methods
- ❌ Inconsistent: Creates two different execution paths
- Add new methods like
-
SET NOCOUNT ON Approach (Chosen Solution)
- Add
SET NOCOUNT ON;
prefix to problematic queries - Based on proven pyodbc implementation
- ✅ Works: Eliminates intermediate result sets that cause issues
- ✅ Simple: Minimal code changes required
- ✅ Proven: Already successfully used by pyodbc for same issue
- ✅ Non-intrusive: No API changes, works with existing code
- ✅ Reliable: Preserves original SQL structure and logic
- Add
Additional context
Research Sources:
- pyodbc Issue #993: Temp table is not available to cursor when arguments are passed into the cursor.execute method mkleehammer/pyodbc#993
- pyodbc Issue FIX: cursor.rowcount #263: select into temporary table does not work with parameters mkleehammer/pyodbc#263
- Stack Overflow: https://stackoverflow.com/questions/49516278/python-running-sql-query-with-temp-tables
Real-World Test Case:
This enhancement was validated with a complex 50+ line production query involving:
- Temp table creation with
SELECT ... INTO #TempEdi
- Multiple JOINs across different databases
- Complex aggregation and filtering logic
- Previously failed silently (returned empty results)
- Now works correctly after applying SET NOCOUNT approach
Impact:
- Enables seamless migration from SSMS-based applications to Python
- Eliminates need for extensive query rewrites
- Brings mssql-python to feature parity with pyodbc for temp table handling
- Maintains backward compatibility while adding powerful new functionality
bewithgaurav
Metadata
Metadata
Assignees
Labels
triage doneIssues that are triaged by dev team and are in investigation.Issues that are triaged by dev team and are in investigation.under development