Skip to content

feat: improve event transport throughput#635

Merged
cofin merged 8 commits into
mainfrom
feat/event-transport-throughput
Jul 10, 2026
Merged

feat: improve event transport throughput#635
cofin merged 8 commits into
mainfrom
feat/event-transport-throughput

Conversation

@cofin

@cofin cofin commented Jul 9, 2026

Copy link
Copy Markdown
Member

Summary

  • dedicate one long-lived listener resource per native backend instance while publishers use short pooled sessions
  • add typed sync/async publish_many() with ordered IDs, explicit grouped transactions, and an ordered fallback
  • bulk-insert durable rows and emit one compact {marker_id, batch_size} wakeup marker per channel atomically
  • recover missing markers through independent event_poll_interval reconciliation and immediately drain every recovered row
  • cancel async listener waits concurrently, bound sync joins, and prevent table polling faster than the configured interval
  • preserve durable batch delivery order and add SQLSpecChannelsBackend.publish_many(data, channels) for consumer integration
  • document canonical transport semantics, adapter support, connection ownership, and recovery behavior

Integration

This PR targeted main directly after #633 and #634 merged. Its event-only commits were rebased onto merge commit ff0b8cb51, so the final branch validates the consolidated C4 contracts and lazy-typing implementation together.

Correctness notes

Durable rows are the source of truth. Native markers are wakeup hints, not batch envelopes. Row insertion and marker publication share one explicit transaction, marker publication failure rolls back the rows, duplicate markers are suppressed, and one reconciliation timeout enters drain mode until the queue is empty. Durable transports remain competing-consumer queues rather than browser fan-out.

JSON payloads and metadata remain native values through batch parameter preparation so adapter-specific encoders receive the same input shape as single-event publication.

Validation

  • listener lifecycle and native backend factory tests
  • 1,000-event batch, ordering, transaction rollback, autocommit restoration, session/statement count, empty/fallback, marker loss/duplication, polling, and metadata tests
  • real asyncpg, psqlpy, and psycopg PostgreSQL integration coverage for single-plus-batch IDs, delivery order, payloads, and metadata
  • SQLite durable order and Litestar channel integrations
  • focused event suite, Ruff, mypy, pyright, docs, and docs linkcheck
  • full GitHub quality, docs, Python 3.10-3.14 unit/integration, source/wheel, MyPyC, and package-integrity checks pass

@codecov-commenter

codecov-commenter commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.95565% with 133 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.04%. Comparing base (ff0b8cb) to head (85cd58c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
sqlspec/adapters/psycopg/events/backend.py 81.40% 32 Missing and 21 partials ⚠️
sqlspec/adapters/psqlpy/events/backend.py 86.66% 10 Missing and 12 partials ⚠️
sqlspec/adapters/asyncpg/events/backend.py 87.27% 10 Missing and 11 partials ⚠️
sqlspec/extensions/events/_channel.py 87.09% 7 Missing and 5 partials ⚠️
sqlspec/extensions/events/_queue.py 89.18% 4 Missing and 4 partials ⚠️
sqlspec/adapters/psycopg/events/_hub.py 92.04% 6 Missing and 1 partial ⚠️
sqlspec/adapters/asyncpg/events/_hub.py 85.29% 5 Missing ⚠️
sqlspec/adapters/psqlpy/events/_hub.py 85.71% 4 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #635      +/-   ##
==========================================
+ Coverage   75.85%   76.04%   +0.19%     
==========================================
  Files         473      473              
  Lines       65285    66107     +822     
  Branches     9731     9841     +110     
==========================================
+ Hits        49522    50273     +751     
- Misses      12504    12542      +38     
- Partials     3259     3292      +33     
Flag Coverage Δ
integration 59.75% <61.98%> (+0.04%) ⬆️
py3.10 74.35% <82.68%> (+0.24%) ⬆️
py3.11 74.35% <82.68%> (+11.97%) ⬆️
py3.12 74.35% <82.68%> (+0.24%) ⬆️
py3.13 74.35% <82.68%> (+0.24%) ⬆️
py3.14 75.23% <85.30%> (+0.19%) ⬆️
unit 63.08% <76.13%> (+0.47%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sqlspec/config.py 92.20% <100.00%> (+0.02%) ⬆️
sqlspec/extensions/events/__init__.py 100.00% <ø> (ø)
sqlspec/extensions/events/_protocols.py 100.00% <ø> (ø)
sqlspec/extensions/litestar/channels.py 76.27% <100.00%> (+0.20%) ⬆️
sqlspec/adapters/asyncpg/events/_hub.py 75.30% <85.29%> (+3.33%) ⬆️
sqlspec/adapters/psqlpy/events/_hub.py 78.53% <85.71%> (+3.13%) ⬆️
sqlspec/adapters/psycopg/events/_hub.py 80.61% <92.04%> (+4.59%) ⬆️
sqlspec/extensions/events/_queue.py 86.66% <89.18%> (+2.25%) ⬆️
sqlspec/extensions/events/_channel.py 78.84% <87.09%> (+4.43%) ⬆️
sqlspec/adapters/asyncpg/events/backend.py 86.44% <87.27%> (+1.76%) ⬆️
... and 2 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cofin cofin force-pushed the feat/event-transport-throughput branch from e9bd451 to c9dbb2a Compare July 10, 2026 00:24
@cofin cofin changed the base branch from main to feat/c4-consolidation July 10, 2026 00:24
@cofin cofin force-pushed the feat/event-transport-throughput branch from 4e9d86b to 151ff07 Compare July 10, 2026 00:28
@cofin cofin changed the title feat: unify event transport backends feat: improve event transport throughput Jul 10, 2026
@cofin cofin force-pushed the feat/event-transport-throughput branch 3 times, most recently from f020dd4 to 05f00a1 Compare July 10, 2026 04:58
cofin added a commit that referenced this pull request Jul 10, 2026
## Summary

- align bounded row-stream cleanup and transaction behavior across
adapters
- consolidate Arrow fallback behavior and adapter contract cleanup
- define canonical event transports: `notify`, `notify_queue`,
`poll_queue`, `aq`, and `txeventq`
- add typed adapter event configuration and accurate durable-delivery
documentation

## Review fixes

- context-manager failures roll back and cleanup `TypeError` is not
retried
- legacy duck-typed stream sources with `close()` still clean up
correctly
- mysql-connector early close consumes unread rows without replacing an
active transaction session
- retired event names fail explicitly with canonical migration guidance
rather than silently selecting polling
- adapter `events_backend` features are honored when extension
configuration does not override them
- native event fixtures remain collected and Python 3.14 type
introspection remains supported

## Stack

This is the base PR for #635, which contains listener lifecycle,
batching, polling/recovery, and final transport documentation.

## Validation

- focused stream, Arrow, event, mysql-connector, and configuration tests
- Ruff, mypy, and pyright
- full GitHub unit/integration matrix on Python 3.10-3.14
- docs, source distribution, pure wheel, MyPyC wheel, and package
integrity checks

All GitHub checks are green.
Base automatically changed from feat/c4-consolidation to main July 10, 2026 05:27
@cofin cofin force-pushed the feat/event-transport-throughput branch from ccfd08e to 85cd58c Compare July 10, 2026 05:30
@cofin cofin marked this pull request as ready for review July 10, 2026 05:49
@cofin cofin merged commit b81f90b into main Jul 10, 2026
23 checks passed
@cofin cofin deleted the feat/event-transport-throughput branch July 10, 2026 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants