Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][client] Avoid ack hole for chunk message #21101

Merged
merged 5 commits into from Sep 4, 2023
Merged

[fix][client] Avoid ack hole for chunk message #21101

merged 5 commits into from Sep 4, 2023

Conversation

liangyepianzhou
Copy link
Contributor

Motivation

Handle ack hole case:
For example:

                     Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
                     Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
                     Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
                     Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
                     Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5

Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk
message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
ChunkMessageIdImpl, so we should process it here.

Modification

Ack chunk-1 and chunk-2.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

## Motivation
                 Handle ack hole case:
                 For example:
                     Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
                     Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
                     Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
                     Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
                     Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
                 Consumer ack chunk message via ChunkMessageIdImpl that is consist of all the chunks in this chunk
                 message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
                 ChunkMessageIdImpl, so we should process here.
## Modification
Ack chunk-1 and chunk-2.
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Aug 31, 2023
@liangyepianzhou liangyepianzhou marked this pull request as draft August 31, 2023 06:44
@Technoboy-
Copy link
Contributor

@Technoboy- Technoboy- added this to the 3.2.0 milestone Aug 31, 2023
@liangyepianzhou
Copy link
Contributor Author

liangyepianzhou commented Aug 31, 2023

Is it related to this flaky test ? https://github.com/apache/pulsar/actions/runs/5934883964/job/16092673164?pr=21043

No, It should not be related to this flaky test. IMO, this test may be errored.

@liangyepianzhou
Copy link
Contributor Author

liangyepianzhou commented Aug 31, 2023

The root cause may be doAcknowledge(msgId, AckType.Individual, Collections.emptyMap(), null); it is an async method. I will open a PR to fix that flaky test.

@codecov-commenter
Copy link

codecov-commenter commented Aug 31, 2023

Codecov Report

Merging #21101 (f6595f8) into master (dab5b2f) will increase coverage by 0.18%.
Report is 16 commits behind head on master.
The diff coverage is 82.14%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #21101      +/-   ##
============================================
+ Coverage     73.03%   73.21%   +0.18%     
- Complexity    32363    32414      +51     
============================================
  Files          1887     1887              
  Lines        139860   139932      +72     
  Branches      15384    15404      +20     
============================================
+ Hits         102140   102445     +305     
+ Misses        29634    29392     -242     
- Partials       8086     8095       +9     
Flag Coverage Δ
inttests 24.13% <10.71%> (?)
systests 25.13% <3.57%> (+0.02%) ⬆️
unittests 72.47% <82.14%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
...va/org/apache/pulsar/client/impl/ConsumerImpl.java 76.65% <72.00%> (+0.10%) ⬆️
...he/pulsar/broker/resources/NamespaceResources.java 80.28% <75.00%> (-0.44%) ⬇️
...roker/service/persistent/MessageDeduplication.java 82.21% <89.47%> (+0.43%) ⬆️
...pache/pulsar/broker/resources/PulsarResources.java 83.72% <100.00%> (ø)
...g/apache/pulsar/client/impl/MessageIdAdvUtils.java 84.61% <100.00%> (+1.28%) ⬆️
...va/org/apache/pulsar/client/impl/ProducerImpl.java 83.88% <100.00%> (+0.60%) ⬆️
...ache/pulsar/client/impl/UnAckedMessageTracker.java 91.59% <100.00%> (+0.91%) ⬆️

... and 85 files with indirect coverage changes

@liangyepianzhou liangyepianzhou merged commit 59a8e72 into apache:master Sep 4, 2023
45 checks passed
liangyepianzhou added a commit that referenced this pull request Sep 4, 2023
## Motivation
Handle ack hole case:
For example:
```markdown
                     Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
                     Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
                     Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
                     Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
                     Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
```
 Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk
 message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
 ChunkMessageIdImpl, so we should process it here.
## Modification
Ack chunk-1 and chunk-2.

(cherry picked from commit 59a8e72)
liangyepianzhou added a commit that referenced this pull request Sep 4, 2023
## Motivation
Handle ack hole case:
For example:
```markdown
                     Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
                     Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
                     Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
                     Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
                     Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
```
 Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk
 message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
 ChunkMessageIdImpl, so we should process it here.
## Modification
Ack chunk-1 and chunk-2.

(cherry picked from commit 59a8e72)
@liangyepianzhou liangyepianzhou deleted the ack-hole branch September 4, 2023 01:41
Technoboy- pushed a commit that referenced this pull request Sep 5, 2023
## Motivation
Handle ack hole case:
For example:
```markdown
                     Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
                     Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
                     Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
                     Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
                     Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
```
 Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk
 message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
 ChunkMessageIdImpl, so we should process it here.
## Modification
Ack chunk-1 and chunk-2.
Technoboy- pushed a commit that referenced this pull request Sep 5, 2023
## Motivation
Handle ack hole case:
For example:
```markdown
                     Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
                     Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
                     Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
                     Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
                     Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
```
 Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk
 message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
 ChunkMessageIdImpl, so we should process it here.
## Modification
Ack chunk-1 and chunk-2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants