Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(phoenix-channel): re-queue message upon send failure #4294

Merged
merged 2 commits into from
Mar 25, 2024

Conversation

thomaseizinger
Copy link
Member

Previously, we would lose one message to the portal upon failing to send it. We now mitigate this in two ways:

  1. We also check the error from poll_ready and don't even pop a message off from our buffer.
  2. If sending still fails, we re-queue it to the front of the buffer.

In certain scenarios as discovered in logs from #4058, this might have caused a loss of the "answer" message from a gateway to the client, resulting in a state mismatch where the gateway thinks the connection is established and the client times out on waiting for the answer.

Copy link

vercel bot commented Mar 25, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated (UTC)
firezone ⬜️ Ignored (Inspect) Mar 25, 2024 0:58am

Copy link

Terraform Cloud Plan Output

Plan: 9 to add, 8 to change, 9 to destroy.

Terraform Cloud Plan

Copy link

Performance Test Results

TCP

Test Name Received/s Sent/s Retransmits
direct-tcp-client2server 221.4 MiB (+1%) 223.0 MiB (+1%) 284 (+72%)
direct-tcp-server2client 229.4 MiB (+1%) 231.0 MiB (+1%) 160 (-63%)
relayed-tcp-client2server 147.8 MiB (+2%) 148.5 MiB (+3%) 163 (+24%)
relayed-tcp-server2client 152.4 MiB (-1%) 152.9 MiB (-1%) 172 (-7%)

UDP

Test Name Total/s Jitter Lost
direct-udp-client2server 50.0 MiB (-0%) 0.04ms (+14%) 0.00% (NaN%)
direct-udp-server2client 50.0 MiB (-0%) 0.02ms (+57%) 0.00% (NaN%)
relayed-udp-client2server 50.0 MiB (-0%) 0.09ms (+0%) 0.00% (NaN%)
relayed-udp-server2client 50.0 MiB (-0%) 0.05ms (-23%) 0.00% (NaN%)

tracing::trace!(target: "wire", to="portal", %message);
}
Err(e) => {
self.pending_messages.push_front(message);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Retrying failed messages seems like a good idea.

Copy link
Collaborator

@conectado conectado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm!

@jamilbk jamilbk added this pull request to the merge queue Mar 25, 2024
Merged via the queue into main with commit 7e68dff Mar 25, 2024
138 checks passed
@jamilbk jamilbk deleted the fix/phoenix-channel/send-errors branch March 25, 2024 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants