Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix: Onboard HRRR processes in NOAA ETL #549

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

Fix: Onboard HRRR processes in NOAA ETL #549

wants to merge 24 commits into from

Commits on Nov 29, 2022

  1. Feat: Added GHCN_M.

    @nlarge-google
    nlarge-google committed Nov 29, 2022
    Configuration menu
    Copy the full SHA
    ec09554 View commit details
    Browse the repository at this point in the history

  2. fix: Flake.

    @nlarge-google
    nlarge-google committed Nov 29, 2022
    Configuration menu
    Copy the full SHA
    ad1d0fe View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2022

  1. feat: Adding HRRR Failover pipeline.

    @nlarge-google
    nlarge-google committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    d097786 View commit details
    Browse the repository at this point in the history

  2. fix: Resolution to graph path.

    @nlarge-google
    nlarge-google committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    38f35ac View commit details
    Browse the repository at this point in the history

  3. fix: Attempting to make url_directory_list recurse through directorie… 

    …s so that copy process for NOAA HRRR Failover will recurse. Not ready for production.
    @nlarge-google
    nlarge-google committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    0a0e860 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2022

  1. fix: Still Testing HRRR Failover. Not Production Ready.

    @nlarge-google
    nlarge-google committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    4c6c275 View commit details
    Browse the repository at this point in the history

  2. fix: Still Testing HRRR Failover. Not Production Ready.

    @nlarge-google
    nlarge-google committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    f0d92ef View commit details
    Browse the repository at this point in the history

  3. fix: Still Testing HRRR Failover.

    @nlarge-google
    nlarge-google committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    c6f3235 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2022

  1. fix: Pre-testing.

    @nlarge-google
    nlarge-google committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    685e7a6 View commit details
    Browse the repository at this point in the history

  2. fix: Configure quiet mode when processing HRRR Failover files.

    @nlarge-google
    nlarge-google committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    c0a287b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into noaa

    @nlarge-google
    nlarge-google committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    9d20b3d View commit details
    Browse the repository at this point in the history

  4. fix: Configure quiet mode in upload to GCS function when processing H… 

    …RRR Failover files.
    @nlarge-google
    nlarge-google committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    5847669 View commit details
    Browse the repository at this point in the history

  5. fix: Flake issues.

    @nlarge-google
    nlarge-google committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    5aa24d2 View commit details
    Browse the repository at this point in the history

  6. fix: Added count of files transferred.

    @nlarge-google
    nlarge-google committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    b6a5861 View commit details
    Browse the repository at this point in the history

  7. fix: Modified HRRR Failover transfer process to NOT recurse into http… 

    … subfolders.
    @nlarge-google
    nlarge-google committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    3c200d8 View commit details
    Browse the repository at this point in the history

  8. fix: Implement code-regions.

    @nlarge-google
    nlarge-google committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    4fb550b View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2022

  1. fix: Added step to remove staged source file from pod after transferr… 

    …ing to GCS in order to resolve ephimeral storage limitations.
    @nlarge-google
    nlarge-google committed Dec 8, 2022
    Configuration menu
    Copy the full SHA
    5a6d644 View commit details
    Browse the repository at this point in the history

  2. fix: Clean up code.

    @nlarge-google
    nlarge-google committed Dec 8, 2022
    Configuration menu
    Copy the full SHA
    b31e510 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2022

  1. fix: Testing NOAA HRRR ARL Formatting in AF.

    @nlarge-google
    nlarge-google committed Dec 9, 2022
    Configuration menu
    Copy the full SHA
    9a029e5 View commit details
    Browse the repository at this point in the history

  2. fix: Cleanup

    @nlarge-google
    nlarge-google committed Dec 9, 2022
    Configuration menu
    Copy the full SHA
    5f03f4b View commit details
    Browse the repository at this point in the history

  3. fix: Extended ephemeral storage.

    @nlarge-google
    nlarge-google committed Dec 9, 2022
    Configuration menu
    Copy the full SHA
    5f2cf16 View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2022

  1. fix: ARL Formattiong - attempting to incorporate gsutil.

    @nlarge-google
    nlarge-google committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    8e89384 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin' into noaa

    @nlarge-google
    nlarge-google committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    1651fef View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2022

  1. fix: Modified Dockerfile to install gsutil.

    @nlarge-google
    nlarge-google committed Dec 14, 2022
    Configuration menu
    Copy the full SHA
    362e69a View commit details
    Browse the repository at this point in the history