Skip to content

Refactor messaging adapters and config#127

Merged
kbukum merged 5 commits intomainfrom
feat/messaging-adapter-architecture
May 5, 2026
Merged

Refactor messaging adapters and config#127
kbukum merged 5 commits intomainfrom
feat/messaging-adapter-architecture

Conversation

@kbukum
Copy link
Copy Markdown
Owner

@kbukum kbukum commented May 4, 2026

Description

Rework pykit-messaging around a clean core registry plus opt-in Kafka, NATS, and RabbitMQ adapters. Core imports stay dependency-light, while adapter modules own their SDK imports, protocol config, validation, redaction, and lifecycle behavior.

Motivation

Applications should be able to use in-memory or core messaging without installing broker SDKs or inheriting Kafka-specific assumptions. This matches the cache/storage optional-adapter pattern and keeps adapter choices explicit at the application boundary.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Package(s) Affected

  • pykit-messaging
  • pykit-resilience

Changes Made

  • Added config-free adapter registration with creation-time config and clean optional SDK boundaries.
  • Added real NATS and RabbitMQ adapters with typed config, validation, redaction, and async lifecycle handling.
  • Kept provider integration shape-specific: producers compose as Sink[Message] and consumers compose as Stream[None, Message].
  • Bounded in-memory broker cache/history and added cleanup support for tests.
  • Aligned retry, DLQ, circuit breaker, and middleware behavior with canonical resilience/messaging patterns.

Testing

  • Added new tests for my changes
  • All existing tests pass (uv run pytest)
  • Linter passes (uv run ruff check packages/)
  • Type-check passes (uv run mypy packages/)
  • Import layering passes (uv run lint-imports)
  • Manual testing performed (describe below if applicable)

Test Evidence

$ uv run ruff format --check packages/pykit-messaging
# passed

$ uv run ruff check packages/pykit-messaging
# passed

$ uv run pytest packages/pykit-messaging/tests
# 280 passed

$ python -c 'import pykit_messaging'
# passed without optional broker SDKs

Breaking Changes

Messaging adapter setup now uses explicit adapter registration and adapter-specific config types. Code that imported Kafka behavior from the core package should import/register the adapter module instead.

Sibling Parity

Parity is implemented in the matching Go and Rust PRs. Go PR: kbukum/gokit#137; Rust PR: kbukum/rskit#137

  • Sibling-parity not required (internal change)
  • Sibling-parity tracked in matching gokit and rskit PRs

Checklist

  • My code follows the coding standards in CONTRIBUTING.md
  • I have run uv lock and committed uv.lock if dependencies changed
  • I have added Google-style docstrings for new public functions/classes
  • I have updated relevant documentation (README.md, package README, etc.)
  • I have added tests that prove my fix/feature works
  • I have considered backward compatibility
  • New dependencies (if any) are justified and minimal
  • CHANGELOG entry added under [Unreleased]

Additional Notes

Review focus: optional import boundaries, adapter config validation, provider-shape bridge behavior, and the shared DLQ/retry behavior reused by Kafka/NATS/RabbitMQ.

…ration handling

- Updated JsonTranslator to use JsonValue type for serialization and deserialization.
- Introduced JsonValue type in types.py for better JSON structure representation.
- Enhanced Event class to use JsonValue for data field and updated serialization logic.
- Modified tests to ensure original headers are preserved and sensitive information is redacted in dead letter queue.
- Refactored retry logic to use calculate_backoff function from resilience package.
- Added comprehensive tests for messaging registry and configuration validation.
- Implemented memory broker tests to ensure bounded history and proper message handling.
- Improved error handling in configuration and registry to reject invalid settings.
- Ensured that optional broker SDKs are not imported unless explicitly registered.
Copy link
Copy Markdown

@github-advanced-security github-advanced-security AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kbukum kbukum requested a review from Copilot May 4, 2026 05:53
@kbukum kbukum requested review from Copilot and removed request for Copilot May 4, 2026 05:58
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated 6 comments.

Comment thread packages/pykit-messaging/src/pykit_messaging/rabbitmq/consumer.py Outdated
Comment thread packages/pykit-messaging/src/pykit_messaging/config.py
Comment thread packages/pykit-messaging/src/pykit_messaging/kafka/config.py Outdated
Comment thread packages/pykit-messaging/src/pykit_messaging/nats/consumer.py Outdated
Comment thread packages/pykit-messaging/src/pykit_messaging/rabbitmq/consumer.py Outdated
Comment thread packages/pykit-messaging/src/pykit_messaging/rabbitmq/consumer.py
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated 3 comments.

Comment thread packages/pykit-messaging/src/pykit_messaging/nats/consumer.py
Comment thread packages/pykit-messaging/src/pykit_messaging/middleware/retry.py Outdated
Comment thread packages/pykit-messaging/README.md Outdated
kbukum and others added 2 commits May 4, 2026 02:09
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@kbukum kbukum merged commit 3b51d21 into main May 5, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants