Add integration tests for rate-limited streaming, safe transaction consumer, and snapshot export workflows

[isdaniel]

• Implemented rate_limited_streaming.rs to test the EventStream API for rate-controlled event processing, including event type categorization and LSN feedback.
• Created safe_transaction_consumer.rs to verify transaction boundaries, buffering changes until commit, and graceful shutdown with cancellation.
• ensuring snapshot lifecycle management and consistency between snapshot reads and streaming events.
• Enhance integration tests by increasing replication slot limits and adding cleanup for lingering slots

for #25

[gemini-code-assist]

Summary of Changes

Hello @isdaniel, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the pg-walstream library by introducing new integration tests for key replication scenarios, including rate-limited event processing, transaction-buffered consumption, and the lifecycle of exported PostgreSQL snapshots. It also clarifies and documents the snapshot export functionality, making it easier for users to achieve consistent initial data reads. These changes aim to improve the reliability and usability of the replication stream library.

Highlights

• New Integration Tests: Added comprehensive integration tests for three critical replication patterns: rate-limited streaming, safe transaction consumption, and PostgreSQL snapshot export workflows.
• Snapshot Export Workflow Enhancement: Improved the ensure_replication_slot method to be public and provided detailed documentation on how to use exported snapshots for consistent initial data reads before starting the replication stream.
• Documentation and Examples Update: Updated the README.md and basic-streaming example to reflect the new, clarified snapshot export workflow, guiding users on proper implementation.
• Internal Test Coverage: Expanded internal test coverage for stream initialization, connection recovery logic, and validation of replication options, ensuring robustness.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• Cargo.toml
  • Added snapshot_export, rate_limited_streaming, and safe_transaction_consumer as new integration test targets.
• README.md
  • Updated the basic streaming example to explicitly show the ensure_replication_slot() call and how to use the exported_snapshot_name() for consistent initial data reads.
  • Clarified the note regarding the validity of exported snapshots.
• examples/basic-streaming/src/main.rs
  • Imported ReplicationSlotOptions.
  • Modified the stream configuration to include temporary slot options with snapshot export.
  • Added calls to stream.ensure_replication_slot() and stream.exported_snapshot_name() to demonstrate the snapshot export workflow.
• integration-tests/rate_limited_streaming.rs
  • Added a new integration test file for verifying rate-limited event processing using the EventStream API, including LSN feedback and event type categorization.
• integration-tests/safe_transaction_consumer.rs
  • Added a new integration test file to validate transaction boundaries, buffering changes until commit, ordered commit processing, LSN feedback after commit, and graceful shutdown with cancellation.
• integration-tests/snapshot_export.rs
  • Added a new integration test file to verify the PostgreSQL snapshot lifecycle when using ensure_replication_slot() with EXPORT_SNAPSHOT, including tests for snapshot readability, invalidation after stream start, idempotency, and consistency.
• src/lib.rs
  • Updated example code comments to reflect the new ensure_replication_slot() and exported_snapshot_name() workflow.
• src/stream.rs
  • Changed ensure_replication_slot visibility from async fn to pub async fn.
  • Added extensive documentation to ensure_replication_slot explaining the initial snapshot workflow and its lifecycle.
  • Updated documentation for exported_snapshot_name to clarify its validity period.
  • Added numerous internal tests covering new() initialization, retry handler configuration, parser protocol version, ensure_replication_slot() idempotency, exported_snapshot_name() behavior, recover_connection() logic, timeout_or_error() behavior, build_replication_options() edge cases, validate_replication_options() edge cases (protocol versions, two-phase, parallel streaming), initialize() path coverage, check_connection_health() behavior, ReplicationStreamConfig builder chain, default slot options, and SharedLsnFeedback in stream context.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/ci.yml

Activity

• No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly enhances the project's test coverage by adding comprehensive integration tests for rate-limited streaming, safe transaction consumption, and the snapshot export workflow. The changes also improve the snapshot export API, making it more explicit and better documented. The overall quality of the changes is high. I have identified a couple of issues in the safe_transaction_consumer.rs test: a misleading doc comment and, more importantly, fragile logic in the test consumer that would not correctly handle interleaved transactions, which is a possibility with the streaming replication mode enabled in the test.

[codecov]

Codecov Report

❌ Patch coverage is 99.28876% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.26%. Comparing base (5453adb) to head (ff024ed).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/stream.rs	99.28%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #27      +/-   ##
==========================================
+ Coverage   89.03%   91.26%   +2.23%     
==========================================
  Files           8        8              
  Lines        7448     8150     +702     
==========================================
+ Hits         6631     7438     +807     
+ Misses        817      712     -105     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.