Skip to content

feat: transparent leader hint reconnect on consume#2

Merged
vieiralucas merged 3 commits intomainfrom
feat/leader-hint-reconnect
Mar 24, 2026
Merged

feat: transparent leader hint reconnect on consume#2
vieiralucas merged 3 commits intomainfrom
feat/leader-hint-reconnect

Conversation

@vieiralucas
Copy link
Copy Markdown
Member

@vieiralucas vieiralucas commented Mar 22, 2026

Summary

  • When consume receives gRPC UNAVAILABLE with x-fila-leader-addr trailing metadata, the client transparently reconnects to the leader address and retries once
  • Applies to both sync Client and async AsyncClient
  • Connection parameters (TLS certs, API key) are stored and reused when creating the redirected channel
  • Max 1 redirect per consume call to prevent infinite loops

Test plan

  • Verify existing integration tests still pass (no behavior change for non-cluster setups)
  • Test with a multi-node cluster where consumer connects to a follower node
  • Verify UNAVAILABLE without leader hint metadata still raises RPCError
  • Verify second UNAVAILABLE after redirect raises RPCError (no infinite loop)

🤖 Generated with Claude Code


Summary by cubic

Automatically reconnects to the hinted leader and retry once when a consume call hits a follower. Works for both sync Client and AsyncClient, reuses TLS/mTLS credentials and API key, and closes the old channel before reconnecting.

  • New Features

    • Detects gRPC UNAVAILABLE with x-fila-leader-addr and retries consume against the leader once.
    • Reuses stored TLS CA and optional client cert/key plus API key when creating the new channel.
    • If no leader hint is present, or the retry fails, the client raises the mapped RPCError.
  • Bug Fixes

    • Closes the previous gRPC channel before reconnecting to avoid leaked connections.
    • Resolves mypy no-any-return issues by adding return types and channel factory helpers (no behavior change).

Written for commit 273746e. Summary will update on new commits.

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="fila/client.py">

<violation number="1" location="fila/client.py:235">
P2: Close the existing gRPC channel before replacing it during redirect to avoid leaking connections.</violation>
</file>

<file name="fila/async_client.py">

<violation number="1" location="fila/async_client.py:245">
P1: Old gRPC channel is leaked on reconnect. `self._channel` is overwritten without closing the previous channel first, leaving its file descriptors and background tasks open until GC (if ever).</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread fila/async_client.py
Comment thread fila/client.py
@vieiralucas vieiralucas force-pushed the feat/leader-hint-reconnect branch from 38bf90c to 1f5753e Compare March 24, 2026 02:02
@vieiralucas vieiralucas merged commit fd28ba0 into main Mar 24, 2026
3 checks passed
@vieiralucas vieiralucas deleted the feat/leader-hint-reconnect branch March 24, 2026 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant