Skip to content

SIGSEGV in DefaultDialect.connect() during concurrent NullPool connection creation with pyodbc do_connect event #13144

@thegoodwinner

Description

@thegoodwinner

Describe the bug

SIGSEGV (segmentation fault) when multiple threads concurrently create connections through a NullPool engine with a do_connect event listener and the pyodbc DBAPI on Linux - and Windows.

The crash occurs when the do_connect event fires concurrently on multiple threads - typically triggered when a cached authentication token expires and several threads simultaneously need new connections. The crash is at default.py line 630 (DefaultDialect.connect()), inside self.loaded_dbapi.connect().

Critical finding: Raw pyodbc.connect() under identical concurrency (same threads, same auth, same connection string, same MARS setting) does NOT crash. The segfault only occurs when connections are created through SQLAlchemy's engine/dialect/event layer. This was verified through extensive isolation testing.

I've been tracking production issues for 3 months now and finally replicated via the script as below.

Optional link from https://docs.sqlalchemy.org which documents the behavior that is expected

https://docs.sqlalchemy.org/en/20/core/events.html#sqlalchemy.events.DialectEvents.do_connect

SQLAlchemy Version in Use

2.0.46 & 2.0.47

DBAPI (i.e. the database driver)

pyodbc 5.3.0

Database Vendor and Major Version

Microsoft SQL Server (Azure SQL Database) -> via ODBC Driver 18 for SQL Server (libmsodbcsql-18.5.so.1.1 & msodbcsql18.dll)

Python Version

3.13.2 & 3.14.2

Operating system

Linux x86_64 (Azure App Service, Debian-based container)

To Reproduce

"""
SIGSEGV repro: concurrent NullPool + do_connect event + pyodbc on Linux.

Requirements:
- Azure SQL Database accessible via Managed Identity
- ODBC Driver 18 for SQL Server installed
- pip install sqlalchemy pyodbc azure-identity

The crash occurs ~60s in when the token cache expires and multiple threads
simultaneously enter the do_connect event path.

IMPORTANT: Raw pyodbc.connect() with identical concurrency does NOT crash.
The issue is specific to SQLAlchemy's connection creation path.
"""
import faulthandler
import os
import struct
import time
import logging
import urllib.parse
from datetime import UTC, datetime, timedelta
from concurrent.futures import ThreadPoolExecutor, as_completed

import pyodbc
from azure.identity import DefaultAzureCredential
from sqlalchemy import create_engine, event, text
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import NullPool

faulthandler.enable(all_threads=True)
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(threadName)s] %(message)s")
log = logging.getLogger(__name__)

# ── Config (set via environment) ──
MSSQL_HOST = os.environ["MSSQL_HOST"]
MSSQL_DATABASE = os.environ["MSSQL_DATABASE"]
SQL_COPT_SS_ACCESS_TOKEN = 1256

# ── Token cache: 60s forced expiry, NO lock (intentional - exposes the race) ──
_credential = DefaultAzureCredential()
_token_cache = None
_refresh_count = 0

def _get_token_bytes():
    global _token_cache, _refresh_count
    now = datetime.now(UTC)
    if _token_cache and _token_cache[1] > now:
        return _token_cache[0]
    _refresh_count += 1
    n = _refresh_count
    log.info("Token refresh #%d...", n)
    token = _credential.get_token("https://database.windows.net/.default")
    expires_utc = datetime.now(UTC) + timedelta(seconds=60)
    raw = token.token.encode("utf-16-le")
    token_bytes = struct.pack("<I", len(raw)) + raw
    _token_cache = (token_bytes, expires_utc)
    log.info("Token refresh #%d done", n)
    return token_bytes

# ── Engine: NullPool + do_connect event ──
odbc_raw = (
    "Driver={ODBC Driver 18 for SQL Server};"
    f"Server=tcp:{MSSQL_HOST},1433;"
    f"Database={MSSQL_DATABASE};"
    "Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;"
    "MARS_Connection=yes"
)
conn_str = "mssql+pyodbc:///?odbc_connect=" + urllib.parse.quote_plus(odbc_raw)
engine = create_engine(conn_str, poolclass=NullPool, fast_executemany=True)

@event.listens_for(engine, "do_connect")
def _provide_token(dialect, conn_rec, cargs, cparams):
    cparams.setdefault("attrs_before", {})[SQL_COPT_SS_ACCESS_TOKEN] = _get_token_bytes()

Session = sessionmaker(bind=engine, autoflush=False, autocommit=False)

# ── Worker: 5 sessions per iteration (mirrors real app pattern) ──
def worker(wid):
    ok = err = 0
    for i in range(500):
        try:
            with Session() as s:
                s.execute(text("SELECT 1")).scalar()
                s.commit()
            for _ in range(3):
                with Session() as s:
                    s.execute(text("SELECT 1")).scalar()
                    s.commit()
            with Session() as s:
                s.execute(text("WAITFOR DELAY '00:00:00.050'; SELECT 1")).scalar()
                s.commit()
            ok += 1
        except Exception as e:
            log.error("W%d iter %d: %s: %s", wid, i, type(e).__name__, e)
            err += 1
        if (i + 1) % 100 == 0:
            log.info("W%d: %d/%d done", wid, i + 1, 500)
    return (ok, err)

# ── Warm up ──
with Session() as s:
    s.execute(text("SELECT 1")).scalar()
    s.commit()
log.info("pyodbc %s + SQLAlchemy, NullPool, do_connect event, MARS=yes", pyodbc.version)
log.info("16 threads x 500 iters, 60s token (no lock)")

start = time.monotonic()
with ThreadPoolExecutor(max_workers=16, thread_name_prefix="t") as pool:
    futs = [pool.submit(worker, i) for i in range(16)]
    tok = terr = 0
    for f in as_completed(futs):
        o, e = f.result()
        tok += o; terr += e
elapsed = time.monotonic() - start
log.info("---")
log.info("Done: %d/%d in %.1fs (%d refreshes)", tok, tok+terr, elapsed, _refresh_count)
if terr: log.warning("Errors: %d", terr)
else: log.info("No crashes.")

Error

Linux:

2026-03-01 04:20:05,539 [t_2] DefaultAzureCredential acquired a token from ManagedIdentityCredential
Fatal Python error: Segmentation fault

Thread 0x00007b1e63fff6c0 (most recent call first):
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/default.py", line 630 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/create.py", line 661 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 896 in __connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 674 in __init__
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 389 in _create_connection
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/impl.py", line 306 in _do_get
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 712 in checkout
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 1272 in _checkout
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 448 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/base.py", line 3317 in raw_connection
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/base.py", line 143 in __init__
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/base.py", line 3293 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/session.py", line 1187 in _connection_for_bind
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/state_changes.py", line 137 in _go
  File "<string>", line 2 in _connection_for_bind
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/session.py", line 2108 in _connection_for_bind
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/session.py", line 2239 in _execute_internal
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/session.py", line 2351 in execute

Thread 0x00007b1e80ff96c0 (most recent call first):
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/default.py", line 630 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/create.py", line 661 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 896 in __connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 674 in __init__
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 389 in _create_connection
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/impl.py", line 306 in _do_get
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 712 in checkout
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 1272 in _checkout
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 448 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/base.py", line 3317 in raw_connection
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/base.py", line 143 in __init__
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/base.py", line 3293 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/session.py", line 1187 in _connection_for_bind
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/state_changes.py", line 137 in _go
  File "<string>", line 2 in _connection_for_bind
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/session.py", line 2108 in _connection_for_bind
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/session.py", line 2239 in _execute_internal
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/orm/session.py", line 2351 in execute

Thread 0x00007b1e817fa6c0 (most recent call first):
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/default.py", line 630 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/engine/create.py", line 661 in connect
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sqlalchemy/pool/base.py", line 896 in __connect
  ...

Segmentation fault (core dumped)

Three threads simultaneously in DefaultDialect.connect() at default.py:630 which calls self.loaded_dbapi.connect(). Crash occurs immediately after a token refresh cycle.

Windows:

2026-03-01 13:41:59,155 [MainThread] DefaultAzureCredential acquired a token from AzureCliCredential
2026-03-01 13:42:01,395 [MainThread] pyodbc 5.3.0 + SQLAlchemy, NullPool, do_connect event, MARS=yes
2026-03-01 13:42:01,396 [MainThread] 16 threads x 500 iters, 60s token (no lock)
2026-03-01 13:43:03,478 [t_8] AzureCliCredential.get_token succeeded
2026-03-01 13:43:03,481 [t_8] DefaultAzureCredential acquired a token from AzureCliCredential
2026-03-01 13:43:03,567 [t_1] AzureCliCredential.get_token succeeded
2026-03-01 13:43:03,568 [t_1] DefaultAzureCredential acquired a token from AzureCliCredential
2026-03-01 13:43:03,568 [t_5] AzureCliCredential.get_token succeeded
2026-03-01 13:43:03,568 [t_5] DefaultAzureCredential acquired a token from AzureCliCredential
2026-03-01 13:43:03,568 [t_7] AzureCliCredential.get_token succeeded
2026-03-01 13:43:03,569 [t_7] DefaultAzureCredential acquired a token from AzureCliCredential
2026-03-01 13:43:03,640 [t_10] AzureCliCredential.get_token succeeded
2026-03-01 13:43:03,641 [t_10] DefaultAzureCredential acquired a token from AzureCliCredential
2026-03-01 13:43:03,710 [t_2] AzureCliCredential.get_token succeeded
2026-03-01 13:43:03,711 [t_2] DefaultAzureCredential acquired a token from AzureCliCredential
Windows fatal exception: access violation

Windows fatal exception: access violation


Windows fatal exception: access violation

Windows fatal exception: access violation

Current thread's C stack trace (most recent call first):
  <cannot get C stack on this system>
Thread 0x00004fa8 [Thread-34 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x00009770 [Thread-33 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py2026-03-01 13:43:04,504 [t_5] W5 iter 26: DBAPIError: (pyodbc.Error) ('HY000', 'The driver did not supply an error!')
(Background on this error at: https://sqlalche.me/e/20/dbapi)
2026-03-01 13:43:04,505 [t_1] W1 iter 26: DBAPIError: (pyodbc.Error) ('HY000', 'The driver did not supply an error!')
(Background on this error at: https://sqlalche.me/e/20/dbapi)
2026-03-01 13:43:04,509 [t_7] W7 iter 26: DBAPIError: (pyodbc.Error) ('HY000', 'The driver did not supply an error!')
(Background on this error at: https://sqlalche.me/e/20/dbapi)
", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x00008680 [Thread-32 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x0000c5c4 [Thread-31 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x000065e4 [Thread-30 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x0000a2f8 [Thread-29 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x0000c7642026-03-01 13:43:04,892 [t_11] AzureCliCredential.get_token succeeded
 [Thread-28 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x00002d502026-03-01 13:43:04,893 [t_9] AzureCliCredential.get_token succeeded
 [Thread-27 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x0000bcd0 [Thread-22 (_readerthread2026-03-01 13:43:04,893 [t_15] AzureCliCredential.get_token succeeded
)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x000024b02026-03-01 13:43:04,893 [t_12] AzureCliCredential.get_token succeeded
 [Thread-21 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1044 in _bootstrap

Thread 0x0000b6582026-03-01 13:43:04,894 [t_4] AzureCliCredential.get_token succeeded
 [Thread-16 (_readerthread)] (most recent call first):
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\subprocess.py", line 1613 in _readerthread
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 1024 in run
  File "C:\Users\goodwinner\AppData\Local\Python\pythoncore-3.14-64\Lib\threading.py", line 2026-03-01 13:43:05,082 [t_11] DefaultAzureCredential acquired a token from AzureCliCredential
Windows fatal exception: access violation


Current thread's C stack trace (most recent call first):
  <cannot get C stack on this system>

Additional context

Isolation testing confirms this is SQLAlchemy-specific:

I ran the same concurrency test (16 threads, 500 iterations, 60s token refresh, NullPool, MARS_Connection=yes) in multiple configurations:

Test SQLAlchemy SIGSEGV?
Raw pyodbc.connect(), no lock, 16 threads No No crash
Raw pyodbc.connect(), barrier-synced 16 simultaneous calls No No crash
Raw pyodbc.connect(), barrier + 10s token refresh No No crash
create_engine(NullPool) + do_connect event + sessionmaker Yes SIGSEGV on first token refresh (~60s)

The only variable that produces the crash is SQLAlchemy's engine/dialect/event layer. Raw pyodbc.connect() with identical connection strings, auth tokens, and thread concurrency is stable.

Cross-platform reproduction:

Platform Python ODBC Driver Auth method Result
Linux x86_64 (Azure App Service) 3.13.2 libmsodbcsql-18.5.so.1.1 ManagedIdentityCredential SIGSEGV on first token refresh
Windows x64 3.14 msodbcsql18.dll AzureCliCredential Access violation on first token refresh

On Windows, the crash manifests as multiple Windows fatal exception: access violation events. Some threads survive but the driver is left in a corrupted state, returning pyodbc.Error: ('HY000', 'The driver did not supply an error!') on subsequent operations.

On Linux, the corruption results in an immediate SIGSEGV (process kill). On Windows, SEH catches the access violation and the process continues, but the driver is left in a corrupted state - subsequent operations return pyodbc.Error: ('HY000', 'The driver did not supply an error!'). This means the same underlying bug may manifest as intermittent HY000 errors on Windows rather than a hard crash, making it harder to diagnose.

The bug is not platform-specific, not Python-version-specific, and not auth-method-specific. The common factor is SQLAlchemy's NullPool + do_connect event + pyodbc + concurrent threads.

Production context: This crash has occurred 5+ times over 3 months in an Azure Functions worker process (Python 3.13, Linux, ODBC Driver 18) running concurrent timer functions. Each timer opens database sessions through a shared engine with a do_connect event listener that injects Azure Managed Identity tokens. Every production faulthandler trace shows 2-3 threads simultaneously in DefaultDialect.connect().

Hypothesis: There may be shared mutable state in the pyodbc dialect's connection creation path (or in how do_connect event arguments are constructed/passed) that is not thread-safe when NullPool forces every session to create a fresh connection.

Metadata

Metadata

Assignees

No one assigned

    Labels

    blockerissue that must be resolved asap as it is preventing things from workingconnection pooleventsSQLAlchemy event hooks, the event system

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions