Skip to content

Improve after resume failure logic#906

Merged
ikbalkaya merged 19 commits intomainfrom
process_queueued_messages_after_reattach
Feb 1, 2023
Merged

Improve after resume failure logic#906
ikbalkaya merged 19 commits intomainfrom
process_queueued_messages_after_reattach

Conversation

@ikbalkaya
Copy link
Copy Markdown
Contributor

@ikbalkaya ikbalkaya commented Jan 26, 2023

This PR fixes two issues

  • When queueEvents is false, pending messages will also be cleared which was missing before.
  • Fixes the issue of reattachment when a connection resume failed.

Previously as soon as connected message was received by connection manager, channels would directly attempt to attach. I have changed the behaviour so that after a resume failure (connected event with a new connection id) I transfer all queued messages (if any) from the connection manager to associated channel. This sets a flag for later to be processed from setConnected. Previously when attempted to call directly to attach from reattach, that would also fail and retry would happen quite a bit later. Attach would time out and then a reattach with some delay would be invoked. I also removed a sync() call from setConnected() as recommended by @paddybyers .

There was also an issue with previously enters members wouldn't automatically reenter. There should be a fix for this and a test that confirms the behaviour.

There are also some test and test utilities improvements.

Fixes #905
Fixes #904

Previously when queued messages were cleared because the states requiring them to be (when queueEvents flag was false) pending message queue wasn't effected by that

This adds the functionality to clear pending queue with normal queued messages.
@ikbalkaya ikbalkaya changed the title Process queueued messages after reattach Improve queued messages logic Jan 26, 2023
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 26, 2023 18:20 Inactive
Comment thread lib/src/main/java/io/ably/lib/realtime/ChannelBase.java Outdated
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 26, 2023 20:56 Inactive
Comment thread lib/src/main/java/io/ably/lib/realtime/ChannelBase.java Outdated
Comment thread lib/src/main/java/io/ably/lib/realtime/AblyRealtime.java Outdated
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 27, 2023 10:54 Inactive
@ikbalkaya ikbalkaya force-pushed the process_queueued_messages_after_reattach branch from 81a5bf1 to 6762032 Compare January 27, 2023 11:19
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 27, 2023 11:20 Inactive
@ikbalkaya ikbalkaya force-pushed the process_queueued_messages_after_reattach branch from 6762032 to 3927bd8 Compare January 27, 2023 11:41
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 27, 2023 11:42 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 27, 2023 20:41 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 28, 2023 16:08 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 28, 2023 20:31 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 29, 2023 10:21 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 29, 2023 15:22 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 29, 2023 15:32 Inactive
@ikbalkaya ikbalkaya force-pushed the process_queueued_messages_after_reattach branch from 2b84ca0 to c6898b7 Compare January 29, 2023 15:53
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 29, 2023 15:54 Inactive
Message publishing on failed resume was happening prematurely before channel went into attached state, that would cause pending messages to fail.

Now in case of resume failure, queued messages will be transferred to their respective channels so that reattach can happen there. Directly calling attach from the method would cause the attach to get quite late as setConnected() from connectionManager wasn't called yet. So instead I created a new flag : reattachOnConnectionResume so that I could use it to reattach when setConencted event arrived. I also removed a previous sync call from this method with recommendation of @paddybyers . Now resume tests pass but there are some channel tests that fails - and related to continuity.
@ikbalkaya ikbalkaya force-pushed the process_queueued_messages_after_reattach branch from c6898b7 to 76a940c Compare January 29, 2023 15:56
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 29, 2023 15:57 Inactive
When sending reattach request to respective channel - it wasn't accounting for the case where there wouldn't be no queued messages. We need to support this so that reattach without queued messages will happen without a problem
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 29, 2023 16:36 Inactive
To enable message sending while disconnected - I added a block to connect message until the message is sent so the test flakiness is no longer the case
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 29, 2023 20:33 Inactive
@ikbalkaya ikbalkaya requested a review from paddybyers January 29, 2023 20:41
@ikbalkaya ikbalkaya self-assigned this Jan 29, 2023
@ikbalkaya ikbalkaya changed the title Improve queued messages logic Improve after resume failure logic Jan 29, 2023
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 30, 2023 16:56 Inactive
In case of a connection resume failure, the old keys for present members needs replacing
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 30, 2023 22:01 Inactive
Replace old presence when a new connection id has arrived
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 31, 2023 09:50 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc January 31, 2023 09:53 Inactive
Comment thread lib/src/main/java/io/ably/lib/transport/ConnectionManager.java Outdated
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc February 1, 2023 10:26 Inactive
@ikbalkaya ikbalkaya marked this pull request as draft February 1, 2023 11:05
@ikbalkaya
Copy link
Copy Markdown
Contributor Author

Moving this to draft state until tests pass / any new tests added

According to spec RTL6c2, queued messages must be resent as soon as the connection is CONNECTED.

This  commit does that by removing only presence messages from existing queuedMessages and transferring them onto the channel.

Also rename methods on channel and readjust to only add to pendingPresence
@ikbalkaya ikbalkaya force-pushed the process_queueued_messages_after_reattach branch from a03265f to dd9a49b Compare February 1, 2023 11:39
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc February 1, 2023 11:41 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc February 1, 2023 12:26 Inactive
As queued messages will remain on connection manager, there is no need to keep and process them on the channel level anymore
@ikbalkaya ikbalkaya force-pushed the process_queueued_messages_after_reattach branch from 30bb018 to 0af496d Compare February 1, 2023 12:27
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc February 1, 2023 12:29 Inactive
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc February 1, 2023 14:22 Inactive
As the retry wasn't suppressed, connection could go into connecting right after disconnection - this add suppress retries and also cleans up the test

Also removes unneeded logs
@ikbalkaya ikbalkaya force-pushed the process_queueued_messages_after_reattach branch from 83047a9 to 0b05d1e Compare February 1, 2023 14:38
@github-actions github-actions bot temporarily deployed to staging/pull/906/javadoc February 1, 2023 14:40 Inactive
Improve this test to block detached arriving so that we are more confident that we will stay in detaching state while this is blocked
Copy link
Copy Markdown
Member

@paddybyers paddybyers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@ikbalkaya ikbalkaya merged commit a1e2c46 into main Feb 1, 2023
@ikbalkaya ikbalkaya deleted the process_queueued_messages_after_reattach branch February 1, 2023 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

A failed resume incorrectly retries queued messages prior to reattachment Pending messages are not failed when transitioning to suspended

2 participants