RemoteConfigComponentTest.java: fix flaky test #6291

dconeybe · 2024-09-21T03:27:52Z

The flaky test is: registerRolloutsStateSubscriber_firebaseNamespace_callsSubscriptionHandler

The flaky failure is due to a race condition, and the error looks like this:

Argument passed to verify() is of type RolloutsStateSubscriptionsHandler and is not a mock!
Make sure you place the parenthesis correctly!
See the examples of correct verifications:
    verify(mock).someMethod();
    verify(mock, times(10)).someMethod();
    verify(mock, atLeastOnce()).someMethod();
org.mockito.exceptions.misusing.NotAMockException: 
Argument passed to verify() is of type RolloutsStateSubscriptionsHandler and is not a mock!

at com.google.firebase.remoteconfig.RemoteConfigComponentTest.registerRolloutsStateSubscriber_firebaseNamespace_callsSubscriptionHandler(RemoteConfigComponentTest.java:194)

…Subscriber_firebaseNamespace_callsSubscriptionHandler

github-actions · 2024-09-21T03:28:04Z

📝 PRs merging into main branch

Our main branch should always be in a releasable state. If you are working on a larger change, or if you don't want this change to see the light of the day just yet, consider using a feature branch first, and only merge into the main branch when the code complete and ready to be released.

google-oss-bot · 2024-09-21T03:28:37Z

	1 Warning
⚠️	Did you forget to add a changelog entry? (Add the 'no-changelog' label to the PR to silence this warning.)

Generated by 🚫 Danger

google-oss-bot · 2024-09-21T03:32:29Z

Coverage Report ¹

Affected Products

`firebase-config`

Overall coverage changed from ? (da0eefa) to 84.11% (23e21a0) by ?.

33 individual files with coverage change

Filename	Base (`da0eefa`)	Merge (`23e21a0`)	Diff
AutoValue_ConfigUpdate.java	?	29.41%	?
Code.java	?	0.00%	?
ConfigAutoFetch.java	?	87.74%	?
ConfigCacheClient.java	?	93.33%	?
ConfigContainer.java	?	94.29%	?
ConfigFetchHandler.java	?	92.86%	?
ConfigFetchHttpClient.java	?	87.33%	?
ConfigGetParameterHandler.java	?	96.45%	?
ConfigMetadataClient.java	?	92.31%	?
ConfigRealtimeHandler.java	?	41.38%	?
ConfigRealtimeHttpClient.java	?	68.93%	?
ConfigStorageClient.java	?	100.00%	?
ConfigUpdate.java	?	100.00%	?
ConfigUpdateListener.java	?	0.00%	?
ConfigUpdateListenerRegistration.java	?	0.00%	?
DefaultsXmlParser.java	?	0.00%	?
FirebaseRemoteConfig.java	?	89.52%	?
FirebaseRemoteConfigClientException.java	?	75.00%	?
FirebaseRemoteConfigException.java	?	95.65%	?
FirebaseRemoteConfigFetchThrottledException.java	?	100.00%	?
FirebaseRemoteConfigInfo.java	?	0.00%	?
FirebaseRemoteConfigInfoImpl.java	?	100.00%	?
FirebaseRemoteConfigServerException.java	?	68.42%	?
FirebaseRemoteConfigSettings.java	?	61.54%	?
FirebaseRemoteConfigValue.java	?	0.00%	?
FirebaseRemoteConfigValueImpl.java	?	84.62%	?
Personalization.java	?	91.43%	?
RemoteConfig.kt	?	31.58%	?
RemoteConfigComponent.java	?	90.70%	?
RemoteConfigConstants.java	?	0.00%	?
RemoteConfigRegistrar.java	?	100.00%	?
RolloutsStateFactory.java	?	95.24%	?
RolloutsStateSubscriptionsHandler.java	?	100.00%	?

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/Kal9SxkLLz.html

github-actions · 2024-09-21T03:33:27Z

Unit Test Results

  40 files +  32   40 suites +32 1m 27s ⏱️ +59s
309 tests +287 309 ✔️ +287 0 💤 ±0 0 ❌ ±0
630 runs +586 630 ✔️ +586 0 💤 ±0 0 ❌ ±0

Results for commit 0292af5. ± Comparison against base commit da0eefa.

This pull request removes 22 and adds 309 tests. Note that renamed tests count towards both.

com.google.firebase.crashlytics.internal.common.CrashlyticsControllerRobolectricTest ‑ testDoCloseSession_disabledAnrs_doesNotPersistsAppExitInfo
com.google.firebase.crashlytics.internal.common.CrashlyticsControllerRobolectricTest ‑ testDoCloseSession_enabledAnrs_doesNotPersistsAppExitInfoIfItDoesntExist
com.google.firebase.crashlytics.internal.common.CrashlyticsControllerRobolectricTest ‑ testDoCloseSession_enabledAnrs_persistsAppExitInfoIfItExists
com.google.firebase.crashlytics.internal.common.DataCollectionArbiterRobolectricTest ‑ testDefaultDataCollection_usedWhenNoOverrideOrManifestSetting
com.google.firebase.crashlytics.internal.common.DataCollectionArbiterRobolectricTest ‑ testManifestMetadata_respectedWhenNoOverride
com.google.firebase.crashlytics.internal.common.DataCollectionArbiterRobolectricTest ‑ testSetCrashlyticsDataCollectionEnabled_overridesOtherSettings
com.google.firebase.crashlytics.internal.common.SessionReportingCoordinatorRobolectricTest ‑ testAppExitInfoEvent_notPersistIfAnrBeforeSession
com.google.firebase.crashlytics.internal.common.SessionReportingCoordinatorRobolectricTest ‑ testAppExitInfoEvent_notPersistIfAppExitInfoNotAnrButWithinSession
com.google.firebase.crashlytics.internal.common.SessionReportingCoordinatorRobolectricTest ‑ testAppExitInfoEvent_persistIfAnrWithinSession
com.google.firebase.crashlytics.internal.common.SessionReportingCoordinatorRobolectricTest ‑ testAppExitInfoEvent_persistIfAnrWithinSession_multipleAppExitInfo
…

com.google.firebase.remoteconfig.ConfigTests ‑ Firebase#remoteConfig should delegate to FirebaseRemoteConfig#getInstance()
com.google.firebase.remoteconfig.ConfigTests ‑ Firebase#remoteConfig should delegate to FirebaseRemoteConfig#getInstance(FirebaseApp, region)
com.google.firebase.remoteconfig.ConfigTests ‑ FirebaseRemoteConfigSettings builder works
com.google.firebase.remoteconfig.ConfigTests ‑ Overloaded get() operator returns default value when key doesn't exist
com.google.firebase.remoteconfig.ConfigTests ‑ Overloaded get() operator returns value when key exists
com.google.firebase.remoteconfig.FirebaseRemoteConfigSettingsTest ‑ toBuilder_withFieldsSet_buildsObjectWithFieldsSet
com.google.firebase.remoteconfig.FirebaseRemoteConfigTest ‑ activate2p_hasAbtExperiments_doesNotCallAbt
com.google.firebase.remoteconfig.FirebaseRemoteConfigTest ‑ activate2p_hasNoAbtExperiments_doesNotCallAbt
com.google.firebase.remoteconfig.FirebaseRemoteConfigTest ‑ activate_callToAbtFails_activateStillSucceeds
com.google.firebase.remoteconfig.FirebaseRemoteConfigTest ‑ activate_configWithRolloutMetadata_storedInActivatedCacheSuccessfully
…

google-oss-bot · 2024-09-21T03:37:43Z

Size Report ¹

Affected Products

No changes between base commit (da0eefa) and merge commit (23e21a0).

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/c5n1cfwPZX.html

google-oss-bot · 2024-09-21T03:55:12Z

Startup Time Report ¹

Note: Layout is sometimes suboptimal due to limited formatting support on GitHub. Please check this report on GCS.

Notes

This report is for comparing the base commit (da0eefa) and the CI merge commit (23e21a0)
Please check below reports for each individual commit to find more details (Perfetto traces, histograms, detailed measurements)
- da0eefa: https://storage.googleapis.com/firebase-sdk-metric-reports/WhXg6epKO6/index.html
- 23e21a0: https://storage.googleapis.com/firebase-sdk-metric-reports/d76AKwxe37/index.html

Startup Times

`fire-rc`

Device Statistics Distributions

oriole-32

Percentile	`da0eefa`	`23e21a0`	Diff	Significant (?)
p10	104 ±53 μs	127 ±91 μs	+23.4 μs (+22.5%)	NO
p25	110 ±56 μs	134 ±95 μs	+24.5 μs (+22.4%)	NO
p50	119 ±59 μs	149 ±106 μs	+29.6 μs (+24.8%)	NO
p75	149 ±80 μs	182 ±137 μs	+32.2 μs (+21.5%)	NO
p90	202 ±90 μs	222 ±143 μs	+20.0 μs (+9.9%)	NO

20 test runs in comparison

Commit	Test Runs
`da0eefa`	2024-09-20_18:24:00.297572_vILN 2024-09-20_18:24:00.297604_ipvA 2024-09-20_18:24:00.297615_nvfW 2024-09-20_18:24:00.297625_kNjf 2024-09-20_18:24:00.297632_mBOF 2024-09-20_18:24:00.297640_AWfP 2024-09-20_18:24:00.297661_MTWi 2024-09-20_18:24:00.297669_BAoD 2024-09-20_18:24:00.297676_hDoG 2024-09-20_18:24:00.297683_nvmE
`23e21a0`	2024-09-21_03:41:15.467859_yXiS 2024-09-21_03:41:15.467896_bzEl 2024-09-21_03:41:15.467908_jHcZ 2024-09-21_03:41:15.467917_EQyZ 2024-09-21_03:41:15.467926_TqPk 2024-09-21_03:41:15.467935_CaqC 2024-09-21_03:41:15.467942_KqSn 2024-09-21_03:41:15.467955_StIX 2024-09-21_03:41:15.467963_JRNB 2024-09-21_03:41:15.467970_MPPZ

redfin-30

Percentile	`da0eefa`	`23e21a0`	Diff	Significant (?)
p10	349 ±552 μs	323 ±513 μs	-25.6 μs (-7.4%)	NO
p25	376 ±605 μs	362 ±606 μs	-14.6 μs (-3.9%)	NO
p50	426 ±699 μs	424 ±751 μs	-1.84 μs (-0.4%)	NO
p75	512 ±852 μs	516 ±865 μs	+3.81 μs (+0.7%)	NO
p90	638 ±966 μs	645 ±1048 μs	+7.81 μs (+1.2%)	NO

20 test runs in comparison

Commit	Test Runs
`da0eefa`	2024-09-20_18:24:00.297572_vILN 2024-09-20_18:24:00.297604_ipvA 2024-09-20_18:24:00.297615_nvfW 2024-09-20_18:24:00.297625_kNjf 2024-09-20_18:24:00.297632_mBOF 2024-09-20_18:24:00.297640_AWfP 2024-09-20_18:24:00.297661_MTWi 2024-09-20_18:24:00.297669_BAoD 2024-09-20_18:24:00.297676_hDoG 2024-09-20_18:24:00.297683_nvmE
`23e21a0`	2024-09-21_03:41:15.467859_yXiS 2024-09-21_03:41:15.467896_bzEl 2024-09-21_03:41:15.467908_jHcZ 2024-09-21_03:41:15.467917_EQyZ 2024-09-21_03:41:15.467926_TqPk 2024-09-21_03:41:15.467935_CaqC 2024-09-21_03:41:15.467942_KqSn 2024-09-21_03:41:15.467955_StIX 2024-09-21_03:41:15.467963_JRNB 2024-09-21_03:41:15.467970_MPPZ

`timeToInitialDisplay`

Device Statistics Distributions

oriole-32

Percentile	`da0eefa`	`23e21a0`	Diff	Significant (?)
p10	197 ±2 ms	203 ±3 ms	+5.99 ms (+3.0%)	NO
p25	204 ±3 ms	209 ±2 ms	+5.54 ms (+2.7%)	NO
p50	210 ±3 ms	217 ±3 ms	+6.23 ms (+3.0%)	NO
p75	218 ±3 ms	224 ±3 ms	+5.58 ms (+2.6%)	NO
p90	226 ±5 ms	234 ±5 ms	+8.69 ms (+3.9%)	NO

20 test runs in comparison

Commit	Test Runs
`da0eefa`	2024-09-20_18:24:00.297572_vILN 2024-09-20_18:24:00.297604_ipvA 2024-09-20_18:24:00.297615_nvfW 2024-09-20_18:24:00.297625_kNjf 2024-09-20_18:24:00.297632_mBOF 2024-09-20_18:24:00.297640_AWfP 2024-09-20_18:24:00.297661_MTWi 2024-09-20_18:24:00.297669_BAoD 2024-09-20_18:24:00.297676_hDoG 2024-09-20_18:24:00.297683_nvmE
`23e21a0`	2024-09-21_03:41:15.467859_yXiS 2024-09-21_03:41:15.467896_bzEl 2024-09-21_03:41:15.467908_jHcZ 2024-09-21_03:41:15.467917_EQyZ 2024-09-21_03:41:15.467926_TqPk 2024-09-21_03:41:15.467935_CaqC 2024-09-21_03:41:15.467942_KqSn 2024-09-21_03:41:15.467955_StIX 2024-09-21_03:41:15.467963_JRNB 2024-09-21_03:41:15.467970_MPPZ

redfin-30

Percentile	`da0eefa`	`23e21a0`	Diff	Significant (?)
p10	243 ±4 ms	273 ±23 ms	+30.2 ms (+12.5%)	NO
p25	249 ±4 ms	282 ±34 ms	+33.4 ms (+13.4%)	NO
p50	256 ±4 ms	291 ±38 ms	+35.0 ms (+13.7%)	NO
p75	264 ±5 ms	302 ±40 ms	+37.7 ms (+14.2%)	NO
p90	274 ±6 ms	315 ±45 ms	+41.6 ms (+15.2%)	NO

20 test runs in comparison

Commit	Test Runs
`da0eefa`	2024-09-20_18:24:00.297572_vILN 2024-09-20_18:24:00.297604_ipvA 2024-09-20_18:24:00.297615_nvfW 2024-09-20_18:24:00.297625_kNjf 2024-09-20_18:24:00.297632_mBOF 2024-09-20_18:24:00.297640_AWfP 2024-09-20_18:24:00.297661_MTWi 2024-09-20_18:24:00.297669_BAoD 2024-09-20_18:24:00.297676_hDoG 2024-09-20_18:24:00.297683_nvmE
`23e21a0`	2024-09-21_03:41:15.467859_yXiS 2024-09-21_03:41:15.467896_bzEl 2024-09-21_03:41:15.467908_jHcZ 2024-09-21_03:41:15.467917_EQyZ 2024-09-21_03:41:15.467926_TqPk 2024-09-21_03:41:15.467935_CaqC 2024-09-21_03:41:15.467942_KqSn 2024-09-21_03:41:15.467955_StIX 2024-09-21_03:41:15.467963_JRNB 2024-09-21_03:41:15.467970_MPPZ

https://storage.googleapis.com/firebase-sdk-metric-reports/2ONBDXGWqu/index.html

This reverts commit 5fccb61.

RemoteConfigComponentTest.java: fix flaky test: registerRolloutsState…

0292af5

…Subscriber_firebaseNamespace_callsSubscriptionHandler

dconeybe self-assigned this Sep 21, 2024

google-oss-bot added the size/XS label Sep 21, 2024

dconeybe requested a review from themiswang September 21, 2024 06:25

themiswang approved these changes Sep 23, 2024

View reviewed changes

dconeybe merged commit 5fccb61 into main Sep 23, 2024
41 checks passed

dconeybe deleted the dconeybe/RemoteConfigComponentTestFlakyTestFix branch September 23, 2024 14:24

dconeybe added a commit that referenced this pull request Sep 23, 2024

Revert "RemoteConfigComponentTest.java: fix flaky test (#6291)"

17b2fa4

This reverts commit 5fccb61.

dconeybe mentioned this pull request Sep 25, 2024

firebase-dataconnect merge into main branch #6290

Merged

firebase locked and limited conversation to collaborators Oct 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

RemoteConfigComponentTest.java: fix flaky test #6291

RemoteConfigComponentTest.java: fix flaky test #6291

Uh oh!

dconeybe commented Sep 21, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Sep 21, 2024 •

edited

Loading

Uh oh!

google-oss-bot commented Sep 21, 2024

Uh oh!

google-oss-bot commented Sep 21, 2024

`firebase-config`

Uh oh!

github-actions bot commented Sep 21, 2024

Uh oh!

google-oss-bot commented Sep 21, 2024

Uh oh!

google-oss-bot commented Sep 21, 2024

`fire-rc`

`timeToInitialDisplay`

Uh oh!

Uh oh!

Uh oh!

RemoteConfigComponentTest.java: fix flaky test #6291

RemoteConfigComponentTest.java: fix flaky test #6291

Uh oh!

Conversation

dconeybe commented Sep 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 PRs merging into main branch

Uh oh!

google-oss-bot commented Sep 21, 2024

Uh oh!

google-oss-bot commented Sep 21, 2024

Coverage Report 1

Affected Products

firebase-config

Test Logs

Uh oh!

github-actions bot commented Sep 21, 2024

Unit Test Results

Uh oh!

google-oss-bot commented Sep 21, 2024

Size Report 1

Affected Products

Test Logs

Uh oh!

google-oss-bot commented Sep 21, 2024

Startup Time Report 1

Notes

Startup Times

fire-rc

timeToInitialDisplay

Uh oh!

Uh oh!

Uh oh!

dconeybe commented Sep 21, 2024 •

edited

Loading

github-actions bot commented Sep 21, 2024 •

edited

Loading

Coverage Report ¹

`firebase-config`

Size Report ¹

Startup Time Report ¹

`fire-rc`

`timeToInitialDisplay`