Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][broker] Duplicate LedgerOffloader creation when namespace/topic… #21591

Merged
merged 2 commits into from
Nov 20, 2023

Conversation

shibd
Copy link
Member

@shibd shibd commented Nov 17, 2023

Motivation

#21590

When namespace/topic policies are updated, will call this method to try to apply new policies of offload.

public LedgerOffloader getManagedLedgerOffloader(NamespaceName namespaceName, OffloadPoliciesImpl offloadPolicies) {
if (offloadPolicies == null) {
return getDefaultOffloader();
}
return ledgerOffloaderMap.compute(namespaceName, (ns, offloader) -> {
try {
if (offloader != null && Objects.equals(offloader.getOffloadPolicies(), offloadPolicies)) {
return offloader;
} else {
if (offloader != null) {
offloader.close();
}
return createManagedLedgerOffloader(offloadPolicies);
}
} catch (PulsarServerException e) {
LOG.error("create ledgerOffloader failed for namespace {}", namespaceName.toString(), e);
return new NullLedgerOffloader();
}
});
}

But, in this case(#21590), Never hit 1380 lines. offloader.getOffloadPolicies() and offloadPolicies are always unequal.

Reason:

  1. In [Bug] Duplicate LedgerOffloader creation when namespace/topic any policy is updated #21590 reproduce step, set a useless config: fileSystemProfilePath.
  2. This configuration is then ignored when converted to properties, then offloader.getOffloadPolicies() result not be included fileSystemProfilePath
  3. offloadPolicies is built from the broker configuration, which contains this configuration.
  4. So, the two are not equal

Note:
In v3.1 after. This pr: #20804 introduces a managedLedgerExtraConfigurations variable, which is also always unequal if the user does not set the value. (null vs empty map)

So after v3.1, In this case(#21590), this issue will appear regardless of whether you have fileSystemProfilePath set.

Modifications

  • Changed toProperties method: Regardless of the user's configuration, we should all convert.
  • The toProperties and create(Properties properties) methods should be complementary.
  • Fix managedLedgerExtraConfigurations conversion issue.

Verifying this change

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Nov 17, 2023
@shibd shibd self-assigned this Nov 17, 2023
@shibd shibd closed this Nov 17, 2023
@shibd shibd reopened this Nov 17, 2023
@Technoboy- Technoboy- added this to the 3.2.0 milestone Nov 20, 2023
@codecov-commenter
Copy link

Codecov Report

Merging #21591 (652999e) into master (403faa4) will decrease coverage by 0.09%.
Report is 1 commits behind head on master.
The diff coverage is 90.47%.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #21591      +/-   ##
============================================
- Coverage     73.35%   73.27%   -0.09%     
+ Complexity    32711    32685      -26     
============================================
  Files          1892     1892              
  Lines        140707   140663      -44     
  Branches      15483    15484       +1     
============================================
- Hits         103211   103066     -145     
- Misses        29394    29493      +99     
- Partials       8102     8104       +2     
Flag Coverage Δ
inttests 24.19% <28.57%> (-0.06%) ⬇️
systests 24.68% <71.42%> (-0.38%) ⬇️
unittests 72.58% <90.47%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...lsar/common/policies/data/OffloadPoliciesImpl.java 85.66% <90.47%> (+2.02%) ⬆️

... and 78 files with indirect coverage changes

@shibd shibd merged commit 98bf9dd into apache:master Nov 20, 2023
48 checks passed
shibd added a commit that referenced this pull request Nov 20, 2023
shibd added a commit that referenced this pull request Nov 20, 2023
shibd added a commit that referenced this pull request Nov 20, 2023
coderzc pushed a commit to coderzc/pulsar that referenced this pull request Nov 21, 2023
shibd added a commit to shibd/pulsar that referenced this pull request Nov 22, 2023
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants