Skip to content

[fix] [test] fix flaky test MetadataStoreStatsTest.testMetadataStoreStats#19433

Merged
nicoloboschi merged 2 commits intoapache:masterfrom
poorbarcode:fix/testMetadataStoreStats
Feb 9, 2023
Merged

[fix] [test] fix flaky test MetadataStoreStatsTest.testMetadataStoreStats#19433
nicoloboschi merged 2 commits intoapache:masterfrom
poorbarcode:fix/testMetadataStoreStats

Conversation

@poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Feb 5, 2023

Motivation

We can see that this is a flaky test:

When the test is executed, it can be seen from the log that multiple metrics of class MetadataStoreStats are registered, and there is one named blank string. So I guess it is caused by other tests not closing MetadataStoreStats which is named blank string.


pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="",type="get",status="success"} 7.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="",type="get",status="success"} 0.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="",type="get",status="success"} 1.675430236619E9
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="1.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="3.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="5.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="10.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="20.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="50.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="100.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="200.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="500.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="1000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="2000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="5000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="fail",le="+Inf"} 0.0
pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="",type="put",status="fail"} 0.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="",type="put",status="fail"} 0.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="",type="put",status="fail"} 1.675430236619E9
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="1.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="3.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="5.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="10.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="20.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="50.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="100.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="200.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="500.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="1000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="2000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="5000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="fail",le="+Inf"} 0.0
pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="",type="del",status="fail"} 0.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="",type="del",status="fail"} 0.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="",type="del",status="fail"} 1.675430236619E9
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="1.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="3.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="5.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="10.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="20.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="50.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="100.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="200.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="500.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="1000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="2000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="5000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="fail",le="+Inf"} 0.0
pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="metadata-store",type="put",status="fail"} 0.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="metadata-store",type="put",status="fail"} 0.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="metadata-store",type="put",status="fail"} 1.675430399313E9
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="1.0"} 1.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="3.0"} 5.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="5.0"} 27.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="10.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="20.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="50.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="100.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="200.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="500.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="1000.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="2000.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="5000.0"} 28.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="success",le="+Inf"} 28.0
pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="metadata-store",type="get",status="success"} 28.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="metadata-store",type="get",status="success"} 124.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="metadata-store",type="get",status="success"} 1.675430399313E9
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="1.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="3.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="5.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="10.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="20.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="50.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="100.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="200.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="500.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="1000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="2000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="5000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="get",status="fail",le="+Inf"} 0.0
pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="metadata-store",type="get",status="fail"} 0.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="metadata-store",type="get",status="fail"} 0.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="metadata-store",type="get",status="fail"} 1.675430399313E9
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="1.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="3.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="5.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="10.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="20.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="50.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="100.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="200.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="500.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="1000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="2000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="5000.0"} 0.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="del",status="success",le="+Inf"} 0.0
pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="",type="del",status="success"} 0.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="",type="del",status="success"} 0.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="",type="del",status="success"} 1.675430236619E9
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="1.0"} 4.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="3.0"} 10.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="5.0"} 14.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="10.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="20.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="50.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="100.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="200.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="500.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="1000.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="2000.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="5000.0"} 15.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="metadata-store",type="put",status="success",le="+Inf"} 15.0
pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="metadata-store",type="put",status="success"} 15.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="metadata-store",type="put",status="success"} 41.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="metadata-store",type="put",status="success"} 1.675430399313E9
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="1.0"} 49.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="3.0"} 51.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="5.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="10.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="20.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="50.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="100.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="200.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="500.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="1000.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="2000.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="5000.0"} 52.0
pulsar_metadata_store_ops_latency_ms_bucket{cluster="test",name="",type="put",status="success",le="+Inf"} 52.0
pulsar_metadata_store_ops_latency_ms_count{cluster="test",name="",type="put",status="success"} 52.0
pulsar_metadata_store_ops_latency_ms_sum{cluster="test",name="",type="put",status="success"} 9.0
pulsar_metadata_store_ops_latency_ms_created{cluster="test",name="",type="put",status="success"} 1.675430236619E9

Modifications

Close MetadataStore after tests.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Feb 5, 2023
@Test
public void testCreate() throws MetadataStoreException{
public void testCreate() throws Exception{
MetadataStore instance = MetadataStoreFactoryImpl.create(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use try/finally block to ensure the store is closed even if the test won't pass

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already fixed

assertNotEquals(l2, l4);
assertNotEquals(l3, l4);

store2.close();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use try/finally block to ensure the store is closed even if the test won't pass

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already fixed

throw tearDownException;
}
// cleanup for metrics.
metadataStore.close();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be done before killing zookeeper, can you add it in stopZKCluster before zkUtil.killCluster(); ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already fixed

@nicoloboschi
Copy link
Contributor

/pulsarbot rerun-failure-checks

@poorbarcode
Copy link
Contributor Author

@poorbarcode
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@codecov-commenter
Copy link

Codecov Report

Merging #19433 (14afc11) into master (aa7af10) will decrease coverage by 1.27%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #19433      +/-   ##
============================================
- Coverage     64.20%   62.93%   -1.27%     
+ Complexity    26265    25815     -450     
============================================
  Files          1832     1832              
  Lines        134067   134113      +46     
  Branches      14753    14755       +2     
============================================
- Hits          86077    84408    -1669     
- Misses        40135    41970    +1835     
+ Partials       7855     7735     -120     
Flag Coverage Δ
inttests 24.73% <100.00%> (-0.19%) ⬇️
systests 25.46% <100.00%> (-0.01%) ⬇️
unittests 60.12% <100.00%> (-1.34%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java 78.05% <100.00%> (+0.12%) ⬆️
...ain/java/org/apache/pulsar/broker/rest/Topics.java 0.00% <0.00%> (-100.00%) ⬇️
.../stats/prometheus/AggregatedSubscriptionStats.java 0.00% <0.00%> (-100.00%) ⬇️
...metheus/AggregatedTransactionCoordinatorStats.java 0.00% <0.00%> (-100.00%) ⬇️
...broker/stats/prometheus/TransactionAggregator.java 0.00% <0.00%> (-96.06%) ⬇️
.../pulsar/broker/rest/RestMessagePublishContext.java 0.00% <0.00%> (-92.60%) ⬇️
...a/org/apache/pulsar/client/util/TypeCheckUtil.java 0.00% <0.00%> (-83.34%) ⬇️
.../apache/pulsar/utils/CmdGenerateDocumentation.java 0.00% <0.00%> (-76.93%) ⬇️
...ker/stats/prometheus/AggregatedNamespaceStats.java 5.20% <0.00%> (-76.05%) ⬇️
...apache/pulsar/websocket/data/ProducerMessages.java 0.00% <0.00%> (-75.00%) ⬇️
... and 167 more

@nicoloboschi nicoloboschi merged commit 329c8c0 into apache:master Feb 9, 2023
@nicoloboschi nicoloboschi added this to the 3.0.0 milestone Feb 9, 2023
@poorbarcode poorbarcode deleted the fix/testMetadataStoreStats branch February 9, 2023 15:24
@lhotari
Copy link
Member

lhotari commented Feb 9, 2023

thanks for fixing this @poorbarcode

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs ready-to-test type/flaky-tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants