Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extract raw PHMSA distribution and start of transmission data (Table A-D, H, I) #2932

Merged
merged 57 commits into from
Jan 19, 2024
Merged

Extract raw PHMSA distribution and start of transmission data (Table A-D, H, I) #2932

merged 57 commits into from
Jan 19, 2024

Commits on Sep 11, 2023

  1. Start adding PHMSA extraction functionality

    @e-belfer
    e-belfer committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    e7db559 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. WIP Add CSV maps for distribution data

    @e-belfer
    e-belfer committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    2534c40 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. WIP add raw assets, need to fix partition

    @e-belfer
    e-belfer committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    57e6a9f View commit details
    Browse the repository at this point in the history

  2. fix merge conflicts

    @e-belfer
    e-belfer committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    f7f040d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dev' into phmsa-extractor

    @e-belfer
    e-belfer authored Oct 11, 2023
    Configuration menu
    Copy the full SHA
    c0a4f1c View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. Merge branch 'dev' into phmsa-extractor

    @e-belfer
    e-belfer authored Oct 26, 2023
    Configuration menu
    Copy the full SHA
    33220e9 View commit details
    Browse the repository at this point in the history

  2. WIP migrate to start_year partition

    @e-belfer
    e-belfer committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    b06585e View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2023

  1. Resolve merge conflicts and fix phmsa partitions and settings

    @e-belfer
    e-belfer committed Dec 18, 2023
    Configuration menu
    Copy the full SHA
    d9916ab View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2023

  1. WIP: add list partitions to _matches

    @e-belfer
    e-belfer committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    4fb33e2 View commit details
    Browse the repository at this point in the history

  2. Fix csv file name

    @e-belfer
    e-belfer committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    a4cbf89 View commit details
    Browse the repository at this point in the history

  3. revert fast etl settings

    @e-belfer
    e-belfer committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    aaecdfa View commit details
    Browse the repository at this point in the history

  4. update the 860m doi 

    it seems to all just work which is tres fun but makes sense after looking at it
    @cmgosnell
    cmgosnell committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    ee9b3bf View commit details
    Browse the repository at this point in the history

Commits on Dec 26, 2023

  1. Fix docs build

    @e-belfer
    e-belfer committed Dec 26, 2023
    Configuration menu
    Copy the full SHA
    61bc757 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #3189 from catalyst-cooperative/eia860m-extraction 

    update the 860m doi
    @e-belfer
    e-belfer authored Dec 26, 2023
    Configuration menu
    Copy the full SHA
    542ce85 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dev' into cems-extraction

    @e-belfer
    e-belfer authored Dec 26, 2023
    Configuration menu
    Copy the full SHA
    cf0454d View commit details
    Browse the repository at this point in the history

  4. Update to non-broken CEMS archive

    @e-belfer
    e-belfer committed Dec 26, 2023
    Configuration menu
    Copy the full SHA
    d26b4aa View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2023

  1. Try adding datastore to CI

    @e-belfer
    e-belfer committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    0b08162 View commit details
    Browse the repository at this point in the history

  2. Update docker to point at actually right year

    @e-belfer
    e-belfer committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    e1215fe View commit details
    Browse the repository at this point in the history

  3. Actually fix in GH action

    @e-belfer
    e-belfer committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    4f50183 View commit details
    Browse the repository at this point in the history

  4. Move pudl_datastore call

    @e-belfer
    e-belfer committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    65af95d View commit details
    Browse the repository at this point in the history

  5. Fix typo

    @e-belfer
    e-belfer committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    2cbd7dd View commit details
    Browse the repository at this point in the history

  6. Fix partition option

    @e-belfer
    e-belfer committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    5587b2e View commit details
    Browse the repository at this point in the history

Commits on Jan 2, 2024

  1. Merge branch 'dev' into phmsa-extractor

    @e-belfer
    e-belfer authored Jan 2, 2024
    Configuration menu
    Copy the full SHA
    8b7bbc9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into cems-extraction

    @e-belfer
    e-belfer authored Jan 2, 2024
    Configuration menu
    Copy the full SHA
    36752c3 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'cems-extraction' into phmsa-extractor

    @e-belfer
    e-belfer authored Jan 2, 2024
    Configuration menu
    Copy the full SHA
    1994527 View commit details
    Browse the repository at this point in the history

  4. Add temp form extraction method, fix matching to take multiple partit… 

    …ions of mixed type
    @e-belfer
    e-belfer committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    cce25e2 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2024

  1. First attempt at column checking notebook

    @e-belfer
    e-belfer committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    60785da View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. Stash transmission csvs

    @e-belfer
    e-belfer committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    128e9fa View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2024

  1. first pass of adding an additional layer in the excel extractor that … 

    …maps pages to another part
    @cmgosnell
    cmgosnell committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    2f38aa6 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into phmsa-extractor

    @cmgosnell
    cmgosnell committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    c04dff5 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. Merge branch 'phmsa-extractor' into phmsa_transmission

    @e-belfer
    e-belfer authored Jan 11, 2024
    Configuration menu
    Copy the full SHA
    d9dc053 View commit details
    Browse the repository at this point in the history

  2. Update notebook and fix column maps for transmission a-d and distribu… 

    …tion
    @e-belfer
    e-belfer committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    56930be View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2024

  1. Add mappings for form I

    @e-belfer
    e-belfer committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    480d656 View commit details
    Browse the repository at this point in the history

  2. Rename transmission tables and map part H

    @e-belfer
    e-belfer committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    3d90e21 View commit details
    Browse the repository at this point in the history

  3. resolve merge conflicts

    @e-belfer
    e-belfer committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    41ce79b View commit details
    Browse the repository at this point in the history

  4. Merge pull request #3235 from catalyst-cooperative/phmsa_transmission 

    WIP: Extract raw PHMSA transmission data (A-D, H, I)
    @e-belfer
    e-belfer authored Jan 12, 2024
    Configuration menu
    Copy the full SHA
    ab93e06 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'phmsa-extractor' of https://github.com/catalyst-coopera… 

    …tive/pudl into phmsa-extractor
    @e-belfer
    e-belfer committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    91bf98c View commit details
    Browse the repository at this point in the history

  6. Actually generate dagster assets!

    @e-belfer
    e-belfer committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    405776c View commit details
    Browse the repository at this point in the history

  7. Fix asset name

    @e-belfer
    e-belfer committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    16b8641 View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2024

  1. Fix notebook to use transformed column names, update column maps, upd… 

    …ate distribution table name, add method to drop columns mapped to other tables
    @e-belfer
    e-belfer committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    db68ed9 View commit details
    Browse the repository at this point in the history

  2. Only log drop when dropping

    @e-belfer
    e-belfer committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    4122299 View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. Update docs and release notes, add jinja template and sources, clarif… 

    …y docstrings
    @e-belfer
    e-belfer committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    f8ab0e0 View commit details
    Browse the repository at this point in the history

  2. Resolve merge conflict with main

    @e-belfer
    e-belfer committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    90220be View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into phmsa-extractor

    @e-belfer
    e-belfer authored Jan 17, 2024
    Configuration menu
    Copy the full SHA
    bfe5dd2 View commit details
    Browse the repository at this point in the history

  4. Fix docs build issue

    @e-belfer
    e-belfer committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    6871b92 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'phmsa-extractor' of https://github.com/catalyst-coopera… 

    …tive/pudl into phmsa-extractor
    @e-belfer
    e-belfer committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    7b0aa39 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into phmsa-extractor

    @e-belfer
    e-belfer authored Jan 17, 2024
    Configuration menu
    Copy the full SHA
    f5b4fbb View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. Minor docstring and jinja cleanup, add .txt documentation files to jinja

    @e-belfer
    e-belfer committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    5b37e94 View commit details
    Browse the repository at this point in the history

  2. Fix docs build

    @e-belfer
    e-belfer committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    aee4548 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into phmsa-extractor

    @e-belfer
    e-belfer authored Jan 18, 2024
    Configuration menu
    Copy the full SHA
    1ac438e View commit details
    Browse the repository at this point in the history

  4. Rename pdf and txt to sort better, split distribution and transmissio… 

    …n in jinja
    @e-belfer
    e-belfer committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    e98c456 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'phmsa-extractor' of https://github.com/catalyst-coopera… 

    …tive/pudl into phmsa-extractor
    @e-belfer
    e-belfer committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    37c4dd0 View commit details
    Browse the repository at this point in the history

  6. A bit more cleaning of file names

    @e-belfer
    e-belfer committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    c5d22e6 View commit details
    Browse the repository at this point in the history

  7. Update docstring for part-partition method

    @e-belfer
    e-belfer committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    20f05b1 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'main' into phmsa-extractor

    @e-belfer
    e-belfer authored Jan 18, 2024
    Configuration menu
    Copy the full SHA
    ffaa8da View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. Move units to end of column names and clean up 192.710

    @e-belfer
    e-belfer committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    30d237b View commit details
    Browse the repository at this point in the history

  2. Clean up CSVs

    @e-belfer
    e-belfer committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    27dab3e View commit details
    Browse the repository at this point in the history