Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extract for DCAT-US #7

Merged
merged 73 commits into from
Jun 2, 2023
Merged

Extract for DCAT-US #7

merged 73 commits into from
Jun 2, 2023

Commits on May 8, 2023

  1. add nginx and aws/s3 mock services.

    @rshewitt
    rshewitt committed May 8, 2023
    Configuration menu
    Copy the full SHA
    a259759 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2023

  1. index to grab json in return tuple.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    5dbf9b1 View commit details
    Browse the repository at this point in the history

  2. add file.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    bcb3711 View commit details
    Browse the repository at this point in the history

  3. exclude openstack dir.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    9606af0 View commit details
    Browse the repository at this point in the history

  4. update file.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    ec2a985 View commit details
    Browse the repository at this point in the history

  5. add new libraries.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    8d72eab View commit details
    Browse the repository at this point in the history

  6. add docker compose.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    d5ff893 View commit details
    Browse the repository at this point in the history

  7. add extract utilities.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    b3d6cce View commit details
    Browse the repository at this point in the history

  8. add try/catch and update return.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    217df1d View commit details
    Browse the repository at this point in the history

  9. create module for s3.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    3a8cbdb View commit details
    Browse the repository at this point in the history

  10. add nginx config.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    0091fd2 View commit details
    Browse the repository at this point in the history

  11. create fixures for dcatus extract.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    9d0840f View commit details
    Browse the repository at this point in the history

  12. add tests for dcatus extract.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    2a3c71e View commit details
    Browse the repository at this point in the history

  13. add harvest source for nginx.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    420b14f View commit details
    Browse the repository at this point in the history

  14. isolate processes as functions.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    2808ca6 View commit details
    Browse the repository at this point in the history

  15. refactor.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    445c81c View commit details
    Browse the repository at this point in the history

  16. move processes to package scripts.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    b976435 View commit details
    Browse the repository at this point in the history

  17. remove unused imports.

    @rshewitt
    rshewitt committed May 9, 2023
    1 Configuration menu
    Copy the full SHA
    aacb956 View commit details
    Browse the repository at this point in the history

  18. test.

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    3404d23 View commit details
    Browse the repository at this point in the history

  19. updated renamed function

    @rshewitt
    rshewitt committed May 9, 2023
    Configuration menu
    Copy the full SHA
    786e92b View commit details
    Browse the repository at this point in the history

  20. simplified bucket creation.

    @rshewitt
    rshewitt committed May 9, 2023
    1 Configuration menu
    Copy the full SHA
    4f538ce View commit details
    Browse the repository at this point in the history

  21. clean up.

    @rshewitt
    rshewitt committed May 9, 2023
    1 Configuration menu
    Copy the full SHA
    3e393f9 View commit details
    Browse the repository at this point in the history

  22. replace catalog with record upload.

    @rshewitt
    rshewitt committed May 9, 2023
    1 Configuration menu
    Copy the full SHA
    2b3047e View commit details
    Browse the repository at this point in the history

  23. Merge remote-tracking branch 'origin' into extract-dcatus

    @rshewitt
    rshewitt committed May 9, 2023
    1 Configuration menu
    Copy the full SHA
    e48cf18 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2023

  1. omit ruff cache.

    @rshewitt
    rshewitt committed May 10, 2023
    Configuration menu
    Copy the full SHA
    c5b8f0f View commit details
    Browse the repository at this point in the history

  2. refactor.

    @rshewitt
    rshewitt committed May 10, 2023
    1 Configuration menu
    Copy the full SHA
    8c4dbda View commit details
    Browse the repository at this point in the history

  3. added yamllint rules

    @robert-bryson
    robert-bryson committed May 10, 2023
    Configuration menu
    Copy the full SHA
    0ad957f View commit details
    Browse the repository at this point in the history

  4. bumped python action to v4

    @robert-bryson
    robert-bryson committed May 10, 2023
    Configuration menu
    Copy the full SHA
    ea38fa3 View commit details
    Browse the repository at this point in the history

  5. changed to poetry action 

    Any reason not to use an action? Or pin to a poetry version? I'm very happy to be overridden on this.
    @robert-bryson
    robert-bryson committed May 10, 2023
    1 Configuration menu
    Copy the full SHA
    b173511 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2023

  1. conform to PEP8 on constants.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    835d682 View commit details
    Browse the repository at this point in the history

  2. glob all.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    dea7ef2 View commit details
    Browse the repository at this point in the history

  3. remove previous versions of schemas.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    2b6dc68 View commit details
    Browse the repository at this point in the history

  4. remove unneeded env vars for localstack.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    72d41ec View commit details
    Browse the repository at this point in the history

  5. add features section and content.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    882f4cd View commit details
    Browse the repository at this point in the history

  6. for python package.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    c976d67 View commit details
    Browse the repository at this point in the history

  7. move to own module from utils.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    dac8b94 View commit details
    Browse the repository at this point in the history

  8. moved dcatus extract to own module.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    eba9988 View commit details
    Browse the repository at this point in the history

  9. rename upload function.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    70f2cd5 View commit details
    Browse the repository at this point in the history

  10. omit default port 80 from nginx endpoint.

    @rshewitt
    rshewitt committed May 12, 2023
    Configuration menu
    Copy the full SHA
    f26d89c View commit details
    Browse the repository at this point in the history

  11. refactor.

    @rshewitt
    rshewitt committed May 12, 2023
    1 Configuration menu
    Copy the full SHA
    4f59571 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2023

  1. confirm to <service>__<variable> format.

    @rshewitt
    rshewitt committed May 15, 2023
    Configuration menu
    Copy the full SHA
    4b25b0d View commit details
    Browse the repository at this point in the history

  2. add default debugger.

    @rshewitt
    rshewitt committed May 15, 2023
    Configuration menu
    Copy the full SHA
    cadb9d1 View commit details
    Browse the repository at this point in the history

  3. refactor and add dataset key check.

    @rshewitt
    rshewitt committed May 15, 2023
    Configuration menu
    Copy the full SHA
    d4595ad View commit details
    Browse the repository at this point in the history

  4. refactor by combining functions.

    @rshewitt
    rshewitt committed May 15, 2023
    Configuration menu
    Copy the full SHA
    35670b5 View commit details
    Browse the repository at this point in the history

  5. may not be needed.

    @rshewitt
    rshewitt committed May 15, 2023
    Configuration menu
    Copy the full SHA
    2fc11ec View commit details
    Browse the repository at this point in the history

  6. refactor.

    @rshewitt
    rshewitt committed May 15, 2023
    1 Configuration menu
    Copy the full SHA
    8c2acaf View commit details
    Browse the repository at this point in the history

Commits on May 26, 2023

  1. refactor and add docstrings.

    @rshewitt
    rshewitt committed May 26, 2023
    Configuration menu
    Copy the full SHA
    60fa67b View commit details
    Browse the repository at this point in the history

  2. fix linting issues.

    @rshewitt
    rshewitt committed May 26, 2023
    1 Configuration menu
    Copy the full SHA
    2475a35 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. updated s3 file key name.

    @rshewitt
    rshewitt committed May 30, 2023
    1 Configuration menu
    Copy the full SHA
    54226c7 View commit details
    Browse the repository at this point in the history

  2. update requests to 2.31 to fix snyk issue.

    @rshewitt
    rshewitt committed May 30, 2023
    1 Configuration menu
    Copy the full SHA
    869af8c View commit details
    Browse the repository at this point in the history

  3. updating s3 path to match the key in store_record_in_s3

    @rshewitt
    rshewitt committed May 30, 2023
    1 Configuration menu
    Copy the full SHA
    41209f8 View commit details
    Browse the repository at this point in the history

  4. removed unnecessary comments.

    @rshewitt
    rshewitt committed May 30, 2023
    Configuration menu
    Copy the full SHA
    eec840c View commit details
    Browse the repository at this point in the history

  5. delete debugger.

    @rshewitt
    rshewitt committed May 30, 2023
    Configuration menu
    Copy the full SHA
    b359ab0 View commit details
    Browse the repository at this point in the history

  6. ignore vscode debugger.

    @rshewitt
    rshewitt committed May 30, 2023
    Configuration menu
    Copy the full SHA
    8b09e81 View commit details
    Browse the repository at this point in the history

  7. use only 1 reference to the s3 key name.

    @rshewitt
    rshewitt committed May 30, 2023
    1 Configuration menu
    Copy the full SHA
    7049e38 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2023

  1. add configuration data to base harvester package.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    6c83b6a View commit details
    Browse the repository at this point in the history

  2. create main extract function.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    967c76d View commit details
    Browse the repository at this point in the history

  3. refactor. remove store_record_in_s3 function entirely.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    0a6c086 View commit details
    Browse the repository at this point in the history

  4. rename function. add raise exceptions.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    f52a8c1 View commit details
    Browse the repository at this point in the history

  5. refactor and update docstrings. use config bucket name.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    8c71b89 View commit details
    Browse the repository at this point in the history

  6. add dcatus schema check.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    9660082 View commit details
    Browse the repository at this point in the history

  7. delete because tests are py packages.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    e0828b6 View commit details
    Browse the repository at this point in the history

  8. delete becaused tests arent py packages.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    03575a8 View commit details
    Browse the repository at this point in the history

  9. add fixtures for more tests.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    ca86e9d View commit details
    Browse the repository at this point in the history

  10. add tests: 404, invalid json, invalid dcatus.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    cd5610f View commit details
    Browse the repository at this point in the history

  11. rename.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    1110bf2 View commit details
    Browse the repository at this point in the history

  12. renamed.

    @rshewitt
    rshewitt committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    b9a05ed View commit details
    Browse the repository at this point in the history

  13. add for tests.

    @rshewitt
    rshewitt committed Jun 1, 2023
    1 Configuration menu
    Copy the full SHA
    ba005ad View commit details
    Browse the repository at this point in the history

  14. version number in pyproject.toml not setup.py.

    @rshewitt
    rshewitt committed Jun 1, 2023
    1 Configuration menu
    Copy the full SHA
    aa21c5a View commit details
    Browse the repository at this point in the history

  15. the magic of isort

    @robert-bryson
    robert-bryson committed Jun 1, 2023
    1 Configuration menu
    Copy the full SHA
    4f6554a View commit details
    Browse the repository at this point in the history

  16. black formatting

    @robert-bryson
    robert-bryson committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    6426179 View commit details
    Browse the repository at this point in the history

  17. fstringed key_name 

    love me some fstrings
    @robert-bryson
    robert-bryson committed Jun 1, 2023
    1 Configuration menu
    Copy the full SHA
    5449a17 View commit details
    Browse the repository at this point in the history

  18. ruff ignore line length 

    i think this makes more sense being one line
    @robert-bryson
    robert-bryson committed Jun 1, 2023
    1 Configuration menu
    Copy the full SHA
    a05ec8d View commit details
    Browse the repository at this point in the history