Tool that calculate workflow/job failure rate and collect failure logs #4743

sunmou99 · 2023-03-03T03:49:14Z

A tool that calculate workflow/job failure rate and a tool that collect ci_test failure logs

see: https://github.com/firebase/firebase-android-sdk/tree/sunmou/ci_status/ci/fireci/workflow_summary

…ct ci_test failure logs

github-actions · 2023-03-03T03:49:32Z

📝 PRs merging into main branch

Our main branch should always be in a releasable state. If you are working on a larger change, or if you don't want this change to see the light of the day just yet, consider using a feature branch first, and only merge into the main branch when the code complete and ready to be released.

Add the 'main-merge-ack' label to your PR to confirm merging into the main branch is intended.

github-actions · 2023-03-03T04:05:55Z

Unit Test Results

  772 files +  766   772 suites +766 32m 9s ⏱️ + 31m 46s
4 806 tests +4 788 4 784 ✔️ +4 766 21 💤 +21 1 ❌ +1
9 409 runs +9 373 9 366 ✔️ +9 330 42 💤 +42 1 ❌ +1

For more details on these failures, see this check.

Results for commit 64c9d98. ± Comparison against base commit b010838.

♻️ This comment has been updated with latest results.

google-oss-bot · 2023-03-03T04:12:41Z

Size Report ¹

Affected Products

firebase-installations
Type Base (b010838) Merge (6242e42) Diff
apk (aggressive) 114 kB 118 kB +4.10 kB (+3.6%)

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/jE7YXA1OqK.html

google-oss-bot · 2023-03-03T05:15:48Z

Coverage Report ¹

Affected Products

firebase-database
Overall coverage changed from 50.18% (b010838) to 50.22% (6242e42) by +0.04%.
Filename Base (b010838) Merge (6242e42) Diff
ChildChangeAccumulator.java 83.33% 96.67% +13.33%

`firebase-inappmessaging`

Overall coverage changed from ? (b010838) to 0.00% (6242e42) by ?.

148 individual files with coverage change

Filename	Base (`b010838`)	Merge (`6242e42`)	Diff
AbtIntegrationHelper.java	?	0.00%	?
AbtIntegrationHelper_Factory.java	?	0.00%	?
AbtIntegrationHelper_MembersInjector.java	?	0.00%	?
Action.java	?	0.00%	?
Analytics.java	?	0.00%	?
AnalyticsConstants.java	?	0.00%	?
AnalyticsEventsManager.java	?	0.00%	?
AnalyticsEventsModule.java	?	0.00%	?
AnalyticsEventsModule_ProvidesAnalyticsConnectorEventsFactory.java	?	0.00%	?
AnalyticsEventsModule_ProvidesAnalyticsEventsManagerFactory.java	?	0.00%	?
AnalyticsListener.java	?	0.00%	?
ApiClient.java	?	0.00%	?
ApiClientModule.java	?	0.00%	?
ApiClientModule_ProvidesApiClientFactory.java	?	0.00%	?
ApiClientModule_ProvidesDataCollectionHelperFactory.java	?	0.00%	?
ApiClientModule_ProvidesFirebaseAppFactory.java	?	0.00%	?
ApiClientModule_ProvidesFirebaseInstallationsFactory.java	?	0.00%	?
ApiClientModule_ProvidesSharedPreferencesUtilsFactory.java	?	0.00%	?
ApiClientModule_ProvidesTestDeviceHelperFactory.java	?	0.00%	?
AppComponent.java	?	0.00%	?
AppForeground.java	?	0.00%	?
ApplicationModule.java	?	0.00%	?
ApplicationModule_DeveloperListenerManagerFactory.java	?	0.00%	?
ApplicationModule_ProvidesApplicationFactory.java	?	0.00%	?
AppMeasurementModule.java	?	0.00%	?
AppMeasurementModule_ProvidesAnalyticsConnectorFactory.java	?	0.00%	?
AppMeasurementModule_ProvidesSubsriberFactory.java	?	0.00%	?
AutoValue_InstallationIdResult.java	?	0.00%	?
AutoValue_RateLimit.java	?	0.00%	?
BannerMessage.java	?	0.00%	?
Button.java	?	0.00%	?
CampaignAnalytics.java	?	0.00%	?
CampaignAnalyticsOrBuilder.java	?	0.00%	?
CampaignCache.java	?	0.00%	?
CampaignCacheClient.java	?	0.00%	?
CampaignCacheClient_Factory.java	?	0.00%	?
CampaignImpression.java	?	0.00%	?
CampaignImpressionList.java	?	0.00%	?
CampaignImpressionListOrBuilder.java	?	0.00%	?
CampaignImpressionOrBuilder.java	?	0.00%	?
CampaignMetadata.java	?	0.00%	?
CampaignProto.java	?	0.00%	?
CardMessage.java	?	0.00%	?
ClientAppInfo.java	?	0.00%	?
ClientAppInfoOrBuilder.java	?	0.00%	?
Clock.java	?	0.00%	?
CommonTypesProto.java	?	0.00%	?
DaggerAppComponent.java	?	0.00%	?
DaggerUniversalComponent.java	?	0.00%	?
DataCollectionHelper.java	?	0.00%	?
DataCollectionHelper_Factory.java	?	0.00%	?
DeveloperListenerManager.java	?	0.00%	?
DismissType.java	?	0.00%	?
DisplayCallbacksFactory.java	?	0.00%	?
DisplayCallbacksFactory_Factory.java	?	0.00%	?
DisplayCallbacksImpl.java	?	0.00%	?
EventType.java	?	0.00%	?
ExecutorsModule.java	?	0.00%	?
ExecutorsModule_ProvidesBackgroundExecutorFactory.java	?	0.00%	?
ExecutorsModule_ProvidesBlockingExecutorFactory.java	?	0.00%	?
ExecutorsModule_ProvidesLightWeightExecutorFactory.java	?	0.00%	?
ExperimentPayloadProto.java	?	0.00%	?
FetchEligibleCampaignsRequest.java	?	0.00%	?
FetchEligibleCampaignsRequestOrBuilder.java	?	0.00%	?
FetchEligibleCampaignsResponse.java	?	0.00%	?
FetchEligibleCampaignsResponseOrBuilder.java	?	0.00%	?
FetchErrorReason.java	?	0.00%	?
FiamAnalyticsConnectorListener.java	?	0.00%	?
FiamFetchService.java	?	0.00%	?
FirebaseAppScope.java	?	0.00%	?
FirebaseInAppMessaging.java	?	0.00%	?
FirebaseInAppMessagingCampaignAnalyticsProto.java	?	0.00%	?
FirebaseInAppMessagingClickListener.java	?	0.00%	?
FirebaseInAppMessagingContextualTrigger.java	?	0.00%	?
FirebaseInAppMessagingDismissListener.java	?	0.00%	?
FirebaseInAppMessagingDisplay.java	?	0.00%	?
FirebaseInAppMessagingDisplayCallbacks.java	?	0.00%	?
FirebaseInAppMessagingDisplayErrorListener.java	?	0.00%	?
FirebaseInAppMessagingImpressionListener.java	?	0.00%	?
FirebaseInAppMessagingRegistrar.java	?	0.00%	?
FirebaseInAppMessaging_Factory.java	?	0.00%	?
ForegroundFlowableModule.java	?	0.00%	?
ForegroundFlowableModule_ProvidesAppForegroundEventStreamFactory.java	?	0.00%	?
ForegroundNotifier.java	?	0.00%	?
GrpcChannelModule.java	?	0.00%	?
GrpcChannelModule_ProvidesGrpcChannelFactory.java	?	0.00%	?
GrpcChannelModule_ProvidesServiceHostFactory.java	?	0.00%	?
GrpcClient.java	?	0.00%	?
GrpcClientModule.java	?	0.00%	?
GrpcClientModule_ProvidesApiKeyHeadersFactory.java	?	0.00%	?
GrpcClientModule_ProvidesInAppMessagingSdkServingStubFactory.java	?	0.00%	?
GrpcClient_Factory.java	?	0.00%	?
ImageData.java	?	0.00%	?
ImageOnlyMessage.java	?	0.00%	?
ImpressionStorageClient.java	?	0.00%	?
ImpressionStorageClient_Factory.java	?	0.00%	?
ImpressionStore.java	?	0.00%	?
InAppMessage.java	?	0.00%	?
InAppMessageStreamManager.java	?	0.00%	?
InAppMessageStreamManager_Factory.java	?	0.00%	?
InAppMessagingSdkServingGrpc.java	?	0.00%	?
InstallationIdResult.java	?	0.00%	?
Logging.java	?	0.00%	?
MessagesProto.java	?	0.00%	?
MessageType.java	?	0.00%	?
MetricsLoggerClient.java	?	0.00%	?
ModalMessage.java	?	0.00%	?
ProgramaticContextualTriggers.java	?	0.00%	?
ProgrammaticContextualTriggerFlowableModule.java	?	0.00%	?
ProgrammaticContextualTriggerFlowableModule_ProvidesProgramaticContextualTriggersFactory.java	?	0.00%	?
ProgrammaticContextualTriggerFlowableModule_ProvidesProgramaticContextualTriggerStreamFactory.java	?	0.00%	?
ProgrammaticTrigger.java	?	0.00%	?
ProtoMarshallerClient.java	?	0.00%	?
ProtoMarshallerClient_Factory.java	?	0.00%	?
ProtoStorageClient.java	?	0.00%	?
ProtoStorageClientModule.java	?	0.00%	?
ProtoStorageClientModule_ProvidesProtoStorageClientForCampaignFactory.java	?	0.00%	?
ProtoStorageClientModule_ProvidesProtoStorageClientForImpressionStoreFactory.java	?	0.00%	?
ProtoStorageClientModule_ProvidesProtoStorageClientForLimiterStoreFactory.java	?	0.00%	?
ProviderInstaller.java	?	0.00%	?
ProviderInstaller_Factory.java	?	0.00%	?
ProxyAnalyticsConnector.java	?	0.00%	?
RateLimit.java	?	0.00%	?
RateLimiterClient.java	?	0.00%	?
RateLimiterClient_Factory.java	?	0.00%	?
RateLimitModule.java	?	0.00%	?
RateLimitModule_ProvidesAppForegroundRateLimitFactory.java	?	0.00%	?
RateLimitProto.java	?	0.00%	?
RenderErrorReason.java	?	0.00%	?
SchedulerModule.java	?	0.00%	?
SchedulerModule_ProvidesComputeSchedulerFactory.java	?	0.00%	?
SchedulerModule_ProvidesIOSchedulerFactory.java	?	0.00%	?
SchedulerModule_ProvidesMainThreadSchedulerFactory.java	?	0.00%	?
Schedulers.java	?	0.00%	?
Schedulers_Factory.java	?	0.00%	?
SharedPreferencesUtils.java	?	0.00%	?
SharedPreferencesUtils_Factory.java	?	0.00%	?
SystemClock.java	?	0.00%	?
SystemClockModule.java	?	0.00%	?
SystemClockModule_ProvidesSystemClockModuleFactory.java	?	0.00%	?
SystemClock_Factory.java	?	0.00%	?
TestDeviceHelper.java	?	0.00%	?
TestDeviceHelper_Factory.java	?	0.00%	?
Text.java	?	0.00%	?
TransportClientModule.java	?	0.00%	?
TransportClientModule_ProvidesMetricsLoggerClientFactory.java	?	0.00%	?
TriggeredInAppMessage.java	?	0.00%	?
UniversalComponent.java	?	0.00%	?

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/z6zfZR36mq.html

google-oss-bot · 2023-03-03T05:24:26Z

Startup Time Report ¹

The report is too large (117,121 chars) to be displayed on GitHub. Please check this report on GCS.

https://storage.googleapis.com/firebase-sdk-metric-reports/wN3jgU4JnH/index.html

rlazo · 2023-03-04T01:12:42Z

The script is incide the fireci directory, but it doesn't integrate with fireci.

@yifanyang could you take a look at that part?

ci/fireci/workflow_summary/github.py

ci/fireci/workflow_summary/collect_ci_test_logs.py

yifanyang

The script is incide the fireci directory, but it doesn't integrate with fireci.

@yifanyang could you take a look at that part?

If the script is supposed to be standalone, maybe just place it inside ci but not ci/fireci. If you do want to integrate with fireci, use the decorator @ci_command, for example:

firebase-android-sdk/ci/fireci/fireciplugins/fireperf.py

Lines 27 to 40 in 46e7a69

    
           @click.option( 
        
             '--plugin_repo_dir', 
        
             help='The location of the fireperf plugin repository.', 
        
             required=True, 
        
           ) 
        
           @click.option( 
        
             '--target_environment', 
        
             type=click.Choice(['prod', 'autopush'], case_sensitive=False), 
        
             help='The target environment fireperf is built for.', 
        
             required=True, 
        
           ) 
        
           @ci_command() 
        
           def fireperf_e2e_test(target_environment, plugin_repo_dir): 
        
             """Run Firebase Performance end-to-end test."""

Another thing: the script depends on a couple of external libraries, such as argparse, requests. Do you want to add a setup instruction for your script?

rlazo · 2023-03-06T17:46:55Z

The script is incide the fireci directory, but it doesn't integrate with fireci.
@yifanyang could you take a look at that part?

If the script is supposed to be standalone, maybe just place it inside ci but not ci/fireci. If you do want to integrate with fireci, use the decorator @ci_command, for example:

firebase-android-sdk/ci/fireci/fireciplugins/fireperf.py

Lines 27 to 40 in 46e7a69

@click.option(

'--plugin_repo_dir',

help='The location of the fireperf plugin repository.',

required=True,

)

@click.option(

'--target_environment',

type=click.Choice(['prod', 'autopush'], case_sensitive=False),

help='The target environment fireperf is built for.',

required=True,

)

@ci_command()

def fireperf_e2e_test(target_environment, plugin_repo_dir):

"""Run Firebase Performance end-to-end test."""

Another thing: the script depends on a couple of external libraries, such as argparse, requests. Do you want to add a setup instruction for your script?

That's one of the benefits of integrating with fireci, since it already has it setup, right?

yifanyang · 2023-03-06T17:51:34Z

The script is incide the fireci directory, but it doesn't integrate with fireci.
@yifanyang could you take a look at that part?

If the script is supposed to be standalone, maybe just place it inside ci but not ci/fireci. If you do want to integrate with fireci, use the decorator @ci_command, for example:

firebase-android-sdk/ci/fireci/fireciplugins/fireperf.py

Lines 27 to 40 in 46e7a69

@click.option(

'--plugin_repo_dir',

help='The location of the fireperf plugin repository.',

required=True,

)

@click.option(

'--target_environment',

type=click.Choice(['prod', 'autopush'], case_sensitive=False),

help='The target environment fireperf is built for.',

required=True,

)

@ci_command()

def fireperf_e2e_test(target_environment, plugin_repo_dir):

"""Run Firebase Performance end-to-end test."""

Another thing: the script depends on a couple of external libraries, such as argparse, requests. Do you want to add a setup instruction for your script?

That's one of the benefits of integrating with fireci, since it already has it setup, right?

Yes. Here is the example:

firebase-android-sdk/ci/fireci/setup.cfg

Lines 5 to 17 in 8d99ce0

    
           [options] 
        
           install_requires = 
        
                   protobuf==3.19 
        
                   click==8.1.3 
        
                   google-cloud-storage==2.5.0 
        
                   mypy==0.991 
        
                   numpy==1.23.1 
        
                   pandas==1.5.1 
        
                   PyGithub==1.55 
        
                   pystache==0.6.0 
        
                   requests==2.23.0 
        
                   seaborn==0.12.1 
        
                   PyYAML==6.0.0

sunmou99 · 2023-03-06T18:35:54Z

The script is incide the fireci directory, but it doesn't integrate with fireci.
@yifanyang could you take a look at that part?

If the script is supposed to be standalone, maybe just place it inside ci but not ci/fireci. If you do want to integrate with fireci, use the decorator @ci_command, for example:

firebase-android-sdk/ci/fireci/fireciplugins/fireperf.py

Lines 27 to 40 in 46e7a69

@click.option(

'--plugin_repo_dir',

help='The location of the fireperf plugin repository.',

required=True,

)

@click.option(

'--target_environment',

type=click.Choice(['prod', 'autopush'], case_sensitive=False),

help='The target environment fireperf is built for.',

required=True,

)

@ci_command()

def fireperf_e2e_test(target_environment, plugin_repo_dir):

"""Run Firebase Performance end-to-end test."""

Another thing: the script depends on a couple of external libraries, such as argparse, requests. Do you want to add a setup instruction for your script?

Standalone for now, thus I moved it to ci.

yifanyang

I see that you moved them to a new folder. But still, the script is using external libraries. We cannot assume it is available anywhere. We will need to clarify which version of those dependencies to use and how to install them. The current practice (such as the setup for fireci) is to use Python virtual environment to isolate installations for different purposes.

Another nit: can you make the commit title more concise?

ci/workflow_summary/workflow_information.py

…se-android-sdk into sunmou/ci_status

ci/workflow_summary/github.py

ci/workflow_summary/workflow_information.py

ci/workflow_summary/github.py

ci/workflow_summary/workflow_information.py

ci/workflow_summary/collect_ci_test_logs.py

…se-android-sdk into sunmou/ci_status

sunmou99 · 2023-03-08T18:36:32Z

I see that you moved them to a new folder. But still, the script is using external libraries. We cannot assume it is available anywhere. We will need to clarify which version of those dependencies to use and how to install them. The current practice (such as the setup for fireci) is to use Python virtual environment to isolate installations for different purposes.

Another nit: can you make the commit title more concise?

I think it's fine that not specifying the python library version.

We are using the very basic functions from the external libraries, thus it should work in most versions.
Install library with multiple versions can be a headache. This tool is not a Python program like fireci, thus unable to leverage the config file
This tool doesn't have a lot of customers
It take time to find out and test out the minimum support version.

Once we have plan to advertise the tool, let's take the library version in to consideration.

yifanyang · 2023-03-08T21:33:20Z

We are using the very basic functions from the external libraries, thus it should work in most versions.

The script may not work with future versions of these libraries, if there is a backward incompatible change. The purpose of specifying a version is to make sure the script always resolve to the version we have verified here to be working.

Install library with multiple versions can be a headache. This tool is not a Python program like fireci, thus unable to leverage the config file

I recommend Python virtual environments to isolate installations of different libraries.

This tool doesn't have a lot of customers

Maybe it's just my opinion, I'd prefer whatever we deliver to be with a good quality, regardless of the customer base.

It take time to find out and test out the minimum support version.

See the first reply above. We only need a version that we are sure everything is working right?

Once we have plan to advertise the tool, let's take the library version in to consideration.

Nevertheless, it is up to you whether you want to incorporate these feedback.

see comments

a tool that calculate workflow/job failure rate and a tool that colle…

c55bb63

…ct ci_test failure logs

google-oss-bot added the size/XL label Mar 3, 2023

Merge branch 'master' into sunmou/ci_status

91b1c52

sunmou99 added the main-merge-ack label Mar 3, 2023

fix typo

1921f7e

sunmou99 added 3 commits March 3, 2023 10:39

update Readme

84254ee

update Readme

35420a9

update Readme

e2b8343

sunmou99 requested a review from rlazo March 3, 2023 19:12

rlazo requested a review from yifanyang March 4, 2023 01:13

rlazo reviewed Mar 4, 2023

View reviewed changes

ci/fireci/workflow_summary/github.py Outdated Show resolved Hide resolved

ci/fireci/workflow_summary/collect_ci_test_logs.py Outdated Show resolved Hide resolved

update format string based on comment

94f0249

yifanyang suggested changes Mar 6, 2023

View reviewed changes

change the file path

046852e

sunmou99 requested review from rlazo and yifanyang March 6, 2023 18:41

Merge branch 'master' into sunmou/ci_status

8d90fa6

yifanyang previously requested changes Mar 6, 2023

View reviewed changes

ci/workflow_summary/workflow_information.py Outdated Show resolved Hide resolved

sunmou99 added 2 commits March 6, 2023 11:39

remove space

2becbe9

Merge branch 'sunmou/ci_status' of https://github.com/firebase/fireba…

284f9b4

…se-android-sdk into sunmou/ci_status

sunmou99 changed the title ~~A tool that calculate workflow/job failure rate and a tool that colle…~~ Tool that calculate workflow/job failure rate and collect failure logs Mar 6, 2023

rlazo reviewed Mar 7, 2023

View reviewed changes

sunmou99 and others added 3 commits March 7, 2023 15:54

update based on comments

1c3080a

collecting logs

d935255

Merge branch 'master' into sunmou/ci_status

06510e8

sunmou99 requested a review from rlazo March 8, 2023 00:10

rlazo reviewed Mar 8, 2023

View reviewed changes

ci/workflow_summary/github.py Outdated Show resolved Hide resolved

ci/workflow_summary/workflow_information.py Outdated Show resolved Hide resolved

ci/workflow_summary/collect_ci_test_logs.py Outdated Show resolved Hide resolved

sunmou99 added 2 commits March 8, 2023 10:13

fix based on comments

5062c33

Merge branch 'sunmou/ci_status' of https://github.com/firebase/fireba…

df4bf7a

…se-android-sdk into sunmou/ci_status

sunmou99 requested review from rlazo and yifanyang March 8, 2023 18:37

rlazo approved these changes Mar 9, 2023

View reviewed changes

Merge branch 'master' into sunmou/ci_status

5df6826

sunmou99 enabled auto-merge (squash) March 9, 2023 18:12

Merge branch 'master' into sunmou/ci_status

64c9d98

sunmou99 merged commit dfe71a9 into master Mar 9, 2023

sunmou99 deleted the sunmou/ci_status branch March 9, 2023 22:23

firebase locked and limited conversation to collaborators Apr 9, 2023

	@click.option(
	'--plugin_repo_dir',
	help='The location of the fireperf plugin repository.',
	required=True,
	)
	@click.option(
	'--target_environment',
	type=click.Choice(['prod', 'autopush'], case_sensitive=False),
	help='The target environment fireperf is built for.',
	required=True,
	)
	@ci_command()
	def fireperf_e2e_test(target_environment, plugin_repo_dir):
	"""Run Firebase Performance end-to-end test."""

Tool that calculate workflow/job failure rate and collect failure logs #4743

Tool that calculate workflow/job failure rate and collect failure logs #4743

Uh oh!

Conversation

sunmou99 commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 PRs merging into main branch

Uh oh!

github-actions bot commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results

Uh oh!

google-oss-bot commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Size Report 1

Affected Products

firebase-installations

Test Logs

Uh oh!

google-oss-bot commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report 1

Affected Products

firebase-database

firebase-inappmessaging

Test Logs

Uh oh!

google-oss-bot commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Startup Time Report 1

Uh oh!

rlazo commented Mar 4, 2023

Uh oh!

Uh oh!

Uh oh!

yifanyang left a comment

Choose a reason for hiding this comment

Uh oh!

rlazo commented Mar 6, 2023

Uh oh!

yifanyang commented Mar 6, 2023

Uh oh!

sunmou99 commented Mar 6, 2023

Uh oh!

yifanyang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sunmou99 commented Mar 8, 2023

Uh oh!

yifanyang commented Mar 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sunmou99 commented Mar 3, 2023 •

edited

Loading

github-actions bot commented Mar 3, 2023 •

edited

Loading

github-actions bot commented Mar 3, 2023 •

edited

Loading

google-oss-bot commented Mar 3, 2023 •

edited

Loading

Size Report ¹

`firebase-installations`

google-oss-bot commented Mar 3, 2023 •

edited

Loading

Coverage Report ¹

`firebase-database`

`firebase-inappmessaging`

google-oss-bot commented Mar 3, 2023 •

edited

Loading

Startup Time Report ¹