Skip to content

Support multi-statement execution with temp tables #228

@arvis108

Description

@arvis108
### Is your feature request related to a problem? Please describe.

Multi-statement SQL queries with temporary tables execute successfully but return empty result sets, while the same queries work correctly in SSMS(SQL Server Management Studio) and pyodbc. This occurs because each `cursor.execute()` call in mssql-python creates separate execution contexts, making local temp tables (`#temp`) created in one statement invisible in subsequent statements.

**Example that fails silently:**
```sql
-- Works in SSMS, returns empty results in mssql-python
SELECT col1, col2 INTO #temp FROM table1 WHERE condition
SELECT * FROM #temp JOIN table2 ON #temp.col1 = table2.col1

This prevents migration from legacy SQL Server applications and breaks existing complex reporting queries, forcing developers to completely rewrite working SQL.

Describe the solution you'd like

Add transparent temporary table support to the cursor.execute() method using pyodbc's proven approach:

  1. Auto-detection: Automatically identify multi-statement queries containing local temp tables
  2. SET NOCOUNT ON: Add SET NOCOUNT ON; prefix to eliminate intermediate result sets that interfere with multi-statement execution
  3. Batch execution: Execute the entire query as a single batch to preserve session scope for temp table visibility
  4. Zero API changes: Existing code works without modification

Benefits:

  • Transparent - no API changes required
  • Reliable - uses proven pyodbc approach
  • Compatible - works with existing SQL without modification
  • Minimal impact - adds approximately 20 lines of code

Describe alternatives you've considered

We evaluated and tested multiple approaches:

  1. CTE Conversion (Tested - Works but Complex)

    • Convert SELECT ... INTO #temp statements to Common Table Expressions (CTEs)
    • Replace temp table references with CTE names in subsequent queries
    • Works: Successfully converts simple temp table patterns
    • Complex: Requires extensive SQL parsing and transformation logic
    • Maintenance burden: Custom conversion logic needs constant updates for edge cases
    • Logic changes: CTEs have different scope and behavior than temp tables
  2. Statement Splitting (Tested - Partial Success)

    • Split multi-statement SQL into individual statements
    • Execute statements sequentially within the same transaction
    • Works: Maintains temp table visibility between statements
    • Session scope issues: Still suffers from mssql-python's execution context isolation
    • Complex parsing: Difficult to reliably split complex SQL statements
    • Parameter handling: Complications with parameter distribution across statements
  3. Global Temp Tables (Works but Intrusive)

    • Convert local temp tables (#temp) to global temp tables (##temp)
    • Works: Global temp tables are visible across execution contexts
    • Breaking changes: Requires modifying user SQL
    • Cleanup complexity: Requires manual cleanup of global temp tables
    • Concurrency issues: Global temp tables can conflict between parallel operations
  4. Custom Wrapper Methods (Works but Breaking)

    • Add new methods like execute_multistatement() or execute_with_temp_tables()
    • Works: Can handle temp tables with specialized logic
    • Breaking changes: Requires application code changes
    • Adoption barrier: Developers must learn new API methods
    • Inconsistent: Creates two different execution paths
  5. SET NOCOUNT ON Approach (Chosen Solution)

    • Add SET NOCOUNT ON; prefix to problematic queries
    • Based on proven pyodbc implementation
    • Works: Eliminates intermediate result sets that cause issues
    • Simple: Minimal code changes required
    • Proven: Already successfully used by pyodbc for same issue
    • Non-intrusive: No API changes, works with existing code
    • Reliable: Preserves original SQL structure and logic

Additional context

Research Sources:

Real-World Test Case:
This enhancement was validated with a complex 50+ line production query involving:

  • Temp table creation with SELECT ... INTO #TempEdi
  • Multiple JOINs across different databases
  • Complex aggregation and filtering logic
  • Previously failed silently (returned empty results)
  • Now works correctly after applying SET NOCOUNT approach

Impact:

  • Enables seamless migration from SSMS-based applications to Python
  • Eliminates need for extensive query rewrites
  • Brings mssql-python to feature parity with pyodbc for temp table handling
  • Maintains backward compatibility while adding powerful new functionality

Metadata

Metadata

Assignees

Labels

triage doneIssues that are triaged by dev team and are in investigation.under development

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions