Skip to content

Add --date parameter to load_bigquery_stats job - #5369

Merged
stormKx18 merged 5 commits into
masterfrom
load_bigquery_stats_by_date_v1
Jul 30, 2026
Merged

Add --date parameter to load_bigquery_stats job#5369
stormKx18 merged 5 commits into
masterfrom
load_bigquery_stats_by_date_v1

Conversation

@stormKx18

@stormKx18 stormKx18 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

This PR updates load_bigquery_stats.py to accept an optional --date flag (in YYYY-MM-DD format), allowing cluster operators and developers to execute the BigQuery stats loading job against an arbitrary target date (useful for backfilling or re-processing past stats).

Key Highlights

  • Optional --date Flag: Supports loading stats for a specified date (e.g. --date 2026-06-20).
  • Backward Compatible: Defaults to yesterday's date (_utc_now().date() - datetime.timedelta(days=1)), maintaining standard daily cron execution behavior.
  • Design Parity: Follows the existing command-line argument pattern established in aggregate_fuzzer_stats.py.
  • Unit Test Coverage: Added test_execute_with_date to verify that target partition tables ($YYYYMMDD) and GCS source URIs (/date/YYYYMMDD/) are correctly generated when a custom date is provided.
  • Unit Test Refactoring: Extracted BigQuery dataset, table, and load job assertions into a shared _assert_load_calls(expected_date_str) helper method to avoid code duplication across default and date-specific test cases.

Testing Instructions

  • Unit Tests:
    python butler.py py_unittest -t appengine -p load_bigquery_stats_test.py
    
  • Unit Tests results:
image

@stormKx18
stormKx18 requested a review from dylanjew July 16, 2026 17:14
@stormKx18 stormKx18 self-assigned this Jul 16, 2026
@stormKx18
stormKx18 requested a review from a team as a code owner July 16, 2026 17:14

@dylanjew dylanjew left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! Mostly LGTM. Just one nit.

I think you can go ahead and test this on dev by running the cron job with butler. You may need to verify that the args are passed in correctly, I remember that being a bit annoying when trying to run the cron

@dylanjew dylanjew left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks!

@vitaliset vitaliset left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I think you can go ahead and test this on dev by running the cron job with butler.

+1. Please provide some evidence that it worked on dev before merging! :)

@stormKx18

Copy link
Copy Markdown
Contributor Author

Please see attached screenshots of the script being validated on dev.

Before running the script there was missing data on 2025-12-29 for a specific fuzzer.

image

Run script on Dev with the target date 2025-12-29

python butler.py --local-logging run -c ../clusterfuzz-config/configs/@@@@-development run_cron 
--script_args load_bigquery_stats --date 2025-12-29

Results after running the script:
image

Data was loaded for the target date 2025-12-29 as it was intended.

def test_execute(self):
"""Tests executing of cron job."""
load_bigquery_stats.main()
self._assert_load_calls('20160907')

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i would add a comment or use a variable to help make it clear that this is one day before the mock in the test

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback.
I have added a variable (expected_yesterday) and a comment to clarify the unit tests.

@stormKx18
stormKx18 force-pushed the load_bigquery_stats_by_date_v1 branch from 689334d to 79b492a Compare July 30, 2026 21:39
@stormKx18
stormKx18 merged commit f9b280e into master Jul 30, 2026
15 checks passed
@stormKx18
stormKx18 deleted the load_bigquery_stats_by_date_v1 branch July 30, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants