Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat: Onboard EPA Historical Air Quality dataset #301

Merged
merged 68 commits into from
Apr 8, 2022
Merged

Feat: Onboard EPA Historical Air Quality dataset #301

merged 68 commits into from
Apr 8, 2022

Commits on Oct 27, 2021

  1. feat: Added annual_summaries, tested locally. Errors in AF

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    b717e9d View commit details
    Browse the repository at this point in the history

  2. feat: Added co_daily_summaries. Not ready for production

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    bb846aa View commit details
    Browse the repository at this point in the history

  3. feat: Added co_hourly_summary. Not ready for production.

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    953d1cd View commit details
    Browse the repository at this point in the history

  4. fix: Changed dataset name

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    e3966b4 View commit details
    Browse the repository at this point in the history

  5. fix: Attempt to resolve AF load_to_bq errors

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    71a9d26 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2021

  1. fix: Resolves issues with AF failure to execute DAG, also, some datat… 

    …ype fixes.
    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    5efa0b5 View commit details
    Browse the repository at this point in the history

  2. feat: Added HAP Hourly Summary. Fixed schema issues in HAP Daily Summ… 

    …ary.
    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    7a64407 View commit details
    Browse the repository at this point in the history

  3. fix: datatype fixes

    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    27d962e View commit details
    Browse the repository at this point in the history

  4. feat: Added no2 daily and hourly.

    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    38d4e6a View commit details
    Browse the repository at this point in the history

  5. feat: Added NONOxNOy pipelines daily and hourly

    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    ac99d56 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2021

  1. feat: Added multiple pipelines and assigned destination table to a va… 

    …riable value for each.
    @nlarge-google
    nlarge-google committed Oct 29, 2021
    Configuration menu
    Copy the full SHA
    f4c09c8 View commit details
    Browse the repository at this point in the history

  2. fix: Added terraform files for new pipelines.

    @nlarge-google
    nlarge-google committed Oct 29, 2021
    Configuration menu
    Copy the full SHA
    9625f2d View commit details
    Browse the repository at this point in the history

  3. fix: Regenerated some dags.

    @nlarge-google
    nlarge-google committed Oct 29, 2021
    Configuration menu
    Copy the full SHA
    c2fb025 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2021

  1. fix: Resolved variable issues in pipeline.yaml files so that they ope… 

    …rate as intended in AF
    @nlarge-google
    nlarge-google committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    9535847 View commit details
    Browse the repository at this point in the history

  2. fix: clean-up code

    @nlarge-google
    nlarge-google committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    54c3d7f View commit details
    Browse the repository at this point in the history

  3. fix: Reduced CHUNKSIZE in order to prevent memory outage in AF, preve… 

    …nting some pipelines to complete successfully.
    @nlarge-google
    nlarge-google committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    7d93681 View commit details
    Browse the repository at this point in the history

  4. fix: Resolved incorrect path entry in ozone daily summary pipeline.ya… 

    …ml; Fixed flake8 issue in csv_transform
    @nlarge-google
    nlarge-google committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    7d0f21e View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2021

  1. fix: Requested changes as per PR code review

    @nlarge-google
    nlarge-google committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    3869b2d View commit details
    Browse the repository at this point in the history

  2. fix: Resolved black hook issue

    @nlarge-google
    nlarge-google committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    9caa687 View commit details
    Browse the repository at this point in the history

  3. fix: Reduced resources used in both lead daily summary and pressure d… 

    …aily summary
    @nlarge-google
    nlarge-google committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    642ab67 View commit details
    Browse the repository at this point in the history

  4. fix: Tiered start time for each DAG by converting start time to chron… 

    … expression
    @nlarge-google
    nlarge-google committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    4636273 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2021

  1. fix: Resolved invalid folder path in pipeline.yaml

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    6a5b3f1 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into EPA

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    cdd1ca7 View commit details
    Browse the repository at this point in the history

  3. fix: Removed out of date terraform file

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    9d7c753 View commit details
    Browse the repository at this point in the history

  4. fix: attempting to resolve code check issues.

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    fcb1503 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'upstream/main' into EPA

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    9e73eee View commit details
    Browse the repository at this point in the history

  6. fix: Missed one change specified in code review.

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    34d67fa View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2021

  1. fix: Increase memory and CPU for pressure_daily_summary and reduced b… 

    …atch size in order to resolve memory pressure resulting in DAG failure.
    @nlarge-google
    nlarge-google committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    5bf3062 View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2021

  1. fix: Accidental check-in resolved

    @nlarge-google
    nlarge-google committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    ba4bff6 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2022

  1. Fix: Modified to run in AF 2 environment. Fixed issue with source fil… 

    …e not available on source URL.
    @nlarge-google
    nlarge-google committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    ebd268f View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2022

  1. fix: checkin. incomplete

    @nlarge-google
    nlarge-google committed Feb 14, 2022
    Configuration menu
    Copy the full SHA
    a21e94d View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2022

  1. fix: modified to new structure and added schema files

    @nlarge-google
    nlarge-google committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    f2acf41 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2022

  1. feat: Added schema JSON files

    @nlarge-google
    nlarge-google committed Feb 22, 2022
    Configuration menu
    Copy the full SHA
    d7f7f87 View commit details
    Browse the repository at this point in the history

  2. fix: need to push to git for adlers debug

    @nlarge-google
    nlarge-google committed Feb 22, 2022
    Configuration menu
    Copy the full SHA
    fe1c15e View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2022

  1. Merge branch 'GoogleCloudPlatform:main' into main

    @nlarge-google
    nlarge-google committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    f29a843 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into EPA

    @nlarge-google
    nlarge-google committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    26e2473 View commit details
    Browse the repository at this point in the history

  3. fix: added schema files

    @nlarge-google
    nlarge-google committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    3194464 View commit details
    Browse the repository at this point in the history

  4. fix: Submitting changes to structure while trying to test new version… 

    … of annual summary.
    @nlarge-google
    nlarge-google committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    80b09f7 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2022

  1. fix: Annual summaries now works with new structure

    @nlarge-google
    nlarge-google committed Feb 24, 2022
    Configuration menu
    Copy the full SHA
    8e6451f View commit details
    Browse the repository at this point in the history

  2. fix: working on cluster changes in each pipeline

    @nlarge-google
    nlarge-google committed Feb 24, 2022
    Configuration menu
    Copy the full SHA
    81d19e8 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2022

  1. fix: (1) changed pipeline.yaml for all pipelines to use clusters (2) … 

    …incorporated reorder headers
    @nlarge-google
    nlarge-google committed Feb 25, 2022
    Configuration menu
    Copy the full SHA
    ba95f41 View commit details
    Browse the repository at this point in the history

  2. fix: resolved flake issues

    @nlarge-google
    nlarge-google committed Feb 25, 2022
    Configuration menu
    Copy the full SHA
    0b06fe1 View commit details
    Browse the repository at this point in the history

  3. fix: resolved cluster reference in pipeline.yaml for co_hourly

    @nlarge-google
    nlarge-google committed Feb 25, 2022
    Configuration menu
    Copy the full SHA
    1697757 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2022

  1. fix: Added WRITE_TRUNCATE to the load to bq function. Fixed variable … 

    …reference in nonoxnoy hourly summary pipeline yaml file.
    @nlarge-google
    nlarge-google committed Feb 28, 2022
    Configuration menu
    Copy the full SHA
    4eabd38 View commit details
    Browse the repository at this point in the history

  2. fix: fixes for pipeline yaml in no2 hourly and nonoxnoy daily

    @nlarge-google
    nlarge-google committed Feb 28, 2022
    Configuration menu
    Copy the full SHA
    dc1a3af View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2022

  1. fix: Re-engineered code to process each file individsually instead of… 

    … concatenating files. Issue was due to lack of resources files cannot always be resident in memory for processing.
    @nlarge-google
    nlarge-google committed Mar 1, 2022
    Configuration menu
    Copy the full SHA
    f717622 View commit details
    Browse the repository at this point in the history

  2. fix: misc fixes to csv transform, plus, changed pipeline yaml files t… 

    …o use 50 nodes with 1 cluster
    @nlarge-google
    nlarge-google committed Mar 1, 2022
    Configuration menu
    Copy the full SHA
    1ff2ac6 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2022

  1. fix: checks table has data before loading. If table has data for the … 

    …respective year then no data is loaded for that year.
    @nlarge-google
    nlarge-google committed Mar 2, 2022
    Configuration menu
    Copy the full SHA
    53602f6 View commit details
    Browse the repository at this point in the history

  2. fix: resolved typo in pipeline.yaml.

    @nlarge-google
    nlarge-google committed Mar 2, 2022
    Configuration menu
    Copy the full SHA
    8c3f55d View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2022

  1. fix: Updated upload function to include file path of source file

    @nlarge-google
    nlarge-google committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    c0e0eae View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2022

  1. Merge branch 'GoogleCloudPlatform:main' into main

    @nlarge-google
    nlarge-google committed Mar 11, 2022
    Configuration menu
    Copy the full SHA
    8594162 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2022

  1. Merge branch 'GoogleCloudPlatform:main' into main

    @nlarge-google
    nlarge-google committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    d7ac308 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2022

  1. fix: Updated code

    @nlarge-google
    nlarge-google committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    2f526b6 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2022

  1. fix: Integrated all pipelines into new pipeline

    @nlarge-google
    nlarge-google committed Mar 28, 2022
    Configuration menu
    Copy the full SHA
    b0ca51e View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2022

  1. fix: Misc fixes also removed individual pipelines from project.

    @nlarge-google
    nlarge-google committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    275ea09 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'GoogleCloudPlatform:main' into main

    @nlarge-google
    nlarge-google committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    e335fb3 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2022

  1. Merge branch 'GoogleCloudPlatform:main' into main

    @nlarge-google
    nlarge-google committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    94b7cfa View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into EPA

    @nlarge-google
    nlarge-google committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    fe3287e View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/main' into EPA

    @nlarge-google
    nlarge-google committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    8b25b3d View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into EPA

    @nlarge-google
    nlarge-google committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    45eeddc View commit details
    Browse the repository at this point in the history

  5. fix: black issue

    @nlarge-google
    nlarge-google committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    48c7af9 View commit details
    Browse the repository at this point in the history

  6. fix: clean-up.

    @nlarge-google
    nlarge-google committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    8c8da9f View commit details
    Browse the repository at this point in the history

  7. fix: black issue

    @nlarge-google
    nlarge-google committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    72bf125 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2022

  1. fix: added table ids to pipeline.yaml as per code review.

    @nlarge-google
    nlarge-google committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    d228a0c View commit details
    Browse the repository at this point in the history

  2. fix: Resolved isort issue

    @nlarge-google
    nlarge-google committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    b6d66ab View commit details
    Browse the repository at this point in the history

  3. fix: misc

    @nlarge-google
    nlarge-google committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    c37d071 View commit details
    Browse the repository at this point in the history

  4. fix: Resolved isort issue

    @nlarge-google
    nlarge-google committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    dfc9ba9 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2022

  1. fix: changes as-per code review

    @nlarge-google
    nlarge-google committed Apr 8, 2022
    Configuration menu
    Copy the full SHA
    f856cde View commit details
    Browse the repository at this point in the history