Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Load Elavon Data to BigQuery #2376

Merged
merged 46 commits into from
Apr 7, 2023
Merged

Load Elavon Data to BigQuery #2376

merged 46 commits into from
Apr 7, 2023

Commits on Mar 7, 2023

  1. Beginnings of Elavon SFTP data extraction

    @SorenSpicknall
    SorenSpicknall committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    b05ca05 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2023

  1. Write Elavon data successfully to GCS

    @SorenSpicknall
    SorenSpicknall committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    7868f53 View commit details
    Browse the repository at this point in the history

  2. Only get files from task date

    @SorenSpicknall
    SorenSpicknall committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    491003a View commit details
    Browse the repository at this point in the history

  3. Inline comments

    @SorenSpicknall
    SorenSpicknall committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    8bc4a3e View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. Recreate work as plugin operator

    @SorenSpicknall
    SorenSpicknall committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    3408d95 View commit details
    Browse the repository at this point in the history

  2. Update argument handling

    @SorenSpicknall
    SorenSpicknall committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    001d459 View commit details
    Browse the repository at this point in the history

  3. Align date formats

    @SorenSpicknall
    SorenSpicknall committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    565246f View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2023

  1. Remove datestamp from consideration

    @SorenSpicknall
    SorenSpicknall committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    94913b6 View commit details
    Browse the repository at this point in the history

  2. Cleanup references to source + local filename

    @SorenSpicknall
    SorenSpicknall committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    cdb37cc View commit details
    Browse the repository at this point in the history

  3. Prototype writing raw data to bucket

    @SorenSpicknall
    SorenSpicknall committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    cf1f217 View commit details
    Browse the repository at this point in the history

  4. Correctly write raw files to GCS

    @SorenSpicknall
    SorenSpicknall committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    6935825 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2023

  1. Remove unused elavon_data_loader job files

    @SorenSpicknall
    SorenSpicknall committed Mar 13, 2023
    Configuration menu
    Copy the full SHA
    22170cf View commit details
    Browse the repository at this point in the history

  2. Split extract process into two jobs

    @SorenSpicknall
    SorenSpicknall committed Mar 13, 2023
    Configuration menu
    Copy the full SHA
    df1180b View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. Get initial two-step process working

    @SorenSpicknall
    SorenSpicknall committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    46215c4 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2023

  1. Add to external tables

    @SorenSpicknall
    SorenSpicknall committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    46cfbb9 View commit details
    Browse the repository at this point in the history

  2. Update dates not automatically interpreted by BQ

    @SorenSpicknall
    SorenSpicknall committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    e8f6bda View commit details
    Browse the repository at this point in the history

  3. Initial models

    @SorenSpicknall
    SorenSpicknall committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    c06b263 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2023

  1. Small model fixes

    @SorenSpicknall
    SorenSpicknall committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    1246670 View commit details
    Browse the repository at this point in the history

  2. Update DAG start date

    @SorenSpicknall
    SorenSpicknall committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    e791b1d View commit details
    Browse the repository at this point in the history

  3. PR feedback round 1

    @SorenSpicknall
    SorenSpicknall committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    cb76985 View commit details
    Browse the repository at this point in the history

  4. Initial one-day lookback when missing schedule data at time of RT val… 

    …idation
    @SorenSpicknall
    SorenSpicknall committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    932266e View commit details
    Browse the repository at this point in the history

  5. Revert 932266e

    @SorenSpicknall
    SorenSpicknall committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    2fbd455 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2023

  1. Make all external table fields strings

    @SorenSpicknall
    SorenSpicknall committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    a21ab0f View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2023

  1. Merge branch 'main' into soren-load_elavon_data_to_bigquery

    @SorenSpicknall
    SorenSpicknall committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    99f1f86 View commit details
    Browse the repository at this point in the history

  2. Switch to multi-DAG approach

    @SorenSpicknall
    SorenSpicknall committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    3abde6b View commit details
    Browse the repository at this point in the history

  3. Move execution into PythonOperators

    @SorenSpicknall
    SorenSpicknall committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    bc0f82d View commit details
    Browse the repository at this point in the history

  4. Switch staging table naming scheme

    @SorenSpicknall
    SorenSpicknall committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    4a1825f View commit details
    Browse the repository at this point in the history

  5. Add uniqueness test on trn_ref_num

    @SorenSpicknall
    SorenSpicknall committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    37c2305 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2023

  1. Switch to using get_fs instead of initializing using imported constru… 

    …ctor
    @SorenSpicknall
    SorenSpicknall committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    e940285 View commit details
    Browse the repository at this point in the history

  2. Switch to endswith for filename string filter

    @SorenSpicknall
    SorenSpicknall committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    004e81b View commit details
    Browse the repository at this point in the history

  3. Add ts to raw file export path

    @SorenSpicknall
    SorenSpicknall committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    2dac250 View commit details
    Browse the repository at this point in the history

  4. Fix filepath reference

    @SorenSpicknall
    SorenSpicknall committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    5267098 View commit details
    Browse the repository at this point in the history

  5. Remove unused operator types

    @SorenSpicknall
    SorenSpicknall committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    a85535b View commit details
    Browse the repository at this point in the history

  6. Read in files more consistently

    @SorenSpicknall
    SorenSpicknall committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    19c9960 View commit details
    Browse the repository at this point in the history

  7. Switch database reference in source table YAML

    @SorenSpicknall
    SorenSpicknall committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    9f80ee5 View commit details
    Browse the repository at this point in the history

  8. Update dbt for updated source behavior

    @SorenSpicknall
    SorenSpicknall committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    d637e0d View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2023

  1. Map connection information to variables

    @SorenSpicknall
    SorenSpicknall committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    56cd88e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into soren-load_elavon_data_to_bigquery

    @SorenSpicknall
    SorenSpicknall committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    855b3f8 View commit details
    Browse the repository at this point in the history

  3. Additional comment

    @SorenSpicknall
    SorenSpicknall committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    609c0c1 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. Coerce strings in CSV reading

    @SorenSpicknall
    SorenSpicknall committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    8a8ae54 View commit details
    Browse the repository at this point in the history

  2. Change interpretation of ts string

    @SorenSpicknall
    SorenSpicknall committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    1be64ea View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2023

  1. Add a couple env variables

    @SorenSpicknall
    SorenSpicknall committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    8563ed3 View commit details
    Browse the repository at this point in the history

  2. Update docker-compose with new variables

    @SorenSpicknall
    SorenSpicknall committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    51e6204 View commit details
    Browse the repository at this point in the history

  3. Review fixes

    @SorenSpicknall
    SorenSpicknall committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    c11411b View commit details
    Browse the repository at this point in the history

  4. Parsed file bucket rename

    @SorenSpicknall
    SorenSpicknall committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    5774f01 View commit details
    Browse the repository at this point in the history

  5. Modify source bucket for external table

    @SorenSpicknall
    SorenSpicknall committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    d0bfbe9 View commit details
    Browse the repository at this point in the history