[fix][broker] Fix get topic policies as null during clean cache #20763

coderzc · 2023-07-09T09:43:16Z

Motivation

Currently, we may get topic policies as null during the clean topic policies since there is a race condition between them.

time	thread1 `getTopicPolicies`	thread2 `cleanCacheAndCloseReader`
t1	check if policyCacheInitMap exists the value of this namespace	remove the reader of this namespace from readerCaches
t2	the value of this namespace in the policyCacheInitMap is `true`	remove the policies of this namespace from policiesCache
t3	gets the topicPolicies of this namespace from policiesCache as `null`
t4		remove the value of this namespace from policyCacheInitMap

Modifications

Use policyCacheInitMap.compute() to make get topic policies init flag and clean cache mutually exclusive.

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository:

...ker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java

codelipenghui · 2023-07-10T09:40:52Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java

@@ -3015,6 +3019,63 @@ public void testLoopCreateAndDeleteTopicPolicies() throws Exception {
            });
        }
    }
+
+    @Test
+    public void testGetTopicPoliciesWithCleanCache()


@coderzc Can you reproduce the issue with this test with the fix? I have tried it on my laptop without this fix. But can't reproduce the issue. The fix looks good, just the test can't protect the changes of this PR.

I have rewritten the test to make it can reproduce the issue.

@codelipenghui PTAL.

mattisonchao · 2023-07-12T04:17:37Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java

@@ -3015,6 +3015,7 @@ public void testLoopCreateAndDeleteTopicPolicies() throws Exception {
            });
        }
    }
+


can you delete the blank line?

codecov-commenter · 2023-07-12T10:38:12Z

Codecov Report

Merging #20763 (eae2a70) into master (0bac873) will increase coverage by 36.31%.
The diff coverage is 92.85%.

@@              Coverage Diff              @@
##             master   #20763       +/-   ##
=============================================
+ Coverage     36.80%   73.11%   +36.31%     
- Complexity    12073    32128    +20055     
=============================================
  Files          1689     1866      +177     
  Lines        129387   139041     +9654     
  Branches      14120    15294     +1174     
=============================================
+ Hits          47619   101661    +54042     
+ Misses        75482    29322    -46160     
- Partials       6286     8058     +1772

Flag	Coverage Δ
inttests	`24.17% <42.85%> (+0.08%)`	⬆️
systests	`25.01% <21.42%> (-0.11%)`	⬇️
unittests	`72.41% <92.85%> (+40.49%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../service/SystemTopicBasedTopicPoliciesService.java	`76.96% <92.85%> (+18.93%)`	⬆️

... and 1431 files with indirect coverage changes

(cherry picked from commit 3116abf)

…he#20763) (cherry picked from commit 3116abf)

(cherry picked from commit 3116abf)

…he (#20763)" This reverts commit 1a7accb.

…he (#20763)" This reverts commit 111c14d.

…he (#20763)" This reverts commit 644c11a.

…he (apache#20763)" This reverts commit 111c14d. (cherry picked from commit ab9384b)

Fix get topic policies is null

3c3e68c

coderzc self-assigned this Jul 9, 2023

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 9, 2023

coderzc added type/bug The PR fixed a bug or issue reported a bug area/broker and removed doc-not-needed Your PR changes do not impact docs labels Jul 9, 2023

coderzc added this to the 3.1.0 milestone Jul 9, 2023

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 9, 2023

coderzc requested a review from mattisonchao July 10, 2023 02:14

mattisonchao reviewed Jul 10, 2023

View reviewed changes

...ker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java Outdated Show resolved Hide resolved

mattisonchao reviewed Jul 10, 2023

View reviewed changes

...ker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java Outdated Show resolved Hide resolved

coderzc force-pushed the fix_topic_policy_null branch 3 times, most recently from 64d1f54 to 7d39b82 Compare July 10, 2023 04:29

Address comment

295ad12

coderzc force-pushed the fix_topic_policy_null branch from 7d39b82 to 295ad12 Compare July 10, 2023 04:32

codelipenghui reviewed Jul 10, 2023

View reviewed changes

Improve code

e3a66fb

coderzc requested review from codelipenghui and mattisonchao July 11, 2023 07:02

mattisonchao reviewed Jul 12, 2023

View reviewed changes

mattisonchao approved these changes Jul 12, 2023

View reviewed changes

coderzc closed this Jul 12, 2023

coderzc reopened this Jul 12, 2023

Fix Licence

eae2a70

codelipenghui approved these changes Jul 12, 2023

View reviewed changes

codelipenghui merged commit 3116abf into apache:master Jul 12, 2023
45 checks passed

coderzc added release/3.0.2 release/2.11.3 labels Jul 17, 2023

coderzc added the release/2.10.6 label Jul 17, 2023

Technoboy- pushed a commit that referenced this pull request Jul 17, 2023

[fix][broker] Fix get topic policies as null during clean cache (#20763)

668a4ee

Technoboy- added the cherry-picked/branch-2.11 label Jul 17, 2023

Technoboy- pushed a commit that referenced this pull request Aug 17, 2023

[fix][broker] Fix get topic policies as null during clean cache (#20763)

17a4bcf

Technoboy- added the cherry-picked/branch-2.10 label Aug 17, 2023

Technoboy- pushed a commit that referenced this pull request Aug 17, 2023

[fix][broker] Fix get topic policies as null during clean cache (#20763)

07aeffe

Technoboy- added the cherry-picked/branch-3.0 label Aug 17, 2023

coderzc added the release/2.9.5 label Aug 22, 2023

coderzc added a commit that referenced this pull request Aug 22, 2023

[fix][broker] Fix get topic policies as null during clean cache (#20763)

2389e65

(cherry picked from commit 3116abf)

coderzc added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Aug 22, 2023

coderzc added a commit that referenced this pull request Aug 28, 2023

[fix][broker] Fix get topic policies as null during clean cache (#20763)

1a7accb

(cherry picked from commit 3116abf)

touchkey pushed a commit to sajithsebastian/pulsar that referenced this pull request Aug 28, 2023

[fix][broker] Fix get topic policies as null during clean cache (apac…

d29c7a4

…he#20763) (cherry picked from commit 3116abf)

coderzc added a commit that referenced this pull request Aug 29, 2023

[fix][broker] Fix get topic policies as null during clean cache (#20763)

644c11a

(cherry picked from commit 3116abf)

coderzc added a commit that referenced this pull request Aug 29, 2023

[fix][broker] Fix get topic policies as null during clean cache (#20763)

111c14d

(cherry picked from commit 3116abf)

coderzc added a commit that referenced this pull request Aug 29, 2023

Revert "[fix][broker] Fix get topic policies as null during clean cac…

6e0ef70

…he (#20763)" This reverts commit 1a7accb.

coderzc added a commit that referenced this pull request Aug 29, 2023

Revert "[fix][broker] Fix get topic policies as null during clean cac…

ab9384b

…he (#20763)" This reverts commit 111c14d.

coderzc added a commit that referenced this pull request Aug 29, 2023

Revert "[fix][broker] Fix get topic policies as null during clean cac…

42e2ccd

…he (#20763)" This reverts commit 644c11a.

coderzc deleted the fix_topic_policy_null branch August 29, 2023 08:52

mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 13, 2024

Revert "[fix][broker] Fix get topic policies as null during clean cac…

2a435c1

…he (apache#20763)" This reverts commit 111c14d. (cherry picked from commit ab9384b)

mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 19, 2024

Revert "[fix][broker] Fix get topic policies as null during clean cac…

1d76f45

…he (apache#20763)" This reverts commit 111c14d. (cherry picked from commit ab9384b)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 23, 2024

Revert "[fix][broker] Fix get topic policies as null during clean cac…

1ccb17a

…he (apache#20763)" This reverts commit 111c14d. (cherry picked from commit ab9384b)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix][broker] Fix get topic policies as null during clean cache #20763

[fix][broker] Fix get topic policies as null during clean cache #20763

coderzc commented Jul 9, 2023 •

edited

Loading

codelipenghui Jul 10, 2023

coderzc Jul 10, 2023

coderzc Jul 12, 2023

mattisonchao Jul 12, 2023

codecov-commenter commented Jul 12, 2023

[fix][broker] Fix get topic policies as null during clean cache #20763

[fix][broker] Fix get topic policies as null during clean cache #20763

Conversation

coderzc commented Jul 9, 2023 • edited Loading

Motivation

Modifications

Verifying this change

Documentation

Matching PR in forked repository

codelipenghui Jul 10, 2023

Choose a reason for hiding this comment

coderzc Jul 10, 2023

Choose a reason for hiding this comment

coderzc Jul 12, 2023

Choose a reason for hiding this comment

mattisonchao Jul 12, 2023

Choose a reason for hiding this comment

codecov-commenter commented Jul 12, 2023

Codecov Report

coderzc commented Jul 9, 2023 •

edited

Loading