Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extract for DCAT-US #7

Merged
merged 73 commits into from
Jun 2, 2023
Merged

Extract for DCAT-US #7

merged 73 commits into from
Jun 2, 2023

Conversation

rshewitt
Copy link
Contributor

@rshewitt rshewitt commented May 9, 2023 •
edited
Loading

Pull Request

Related to Develop extract for DCAT-US ticket

About

  • Added DCAT-US extraction functionality.
  • Added tests to verify DCAT-US extraction works.
  • Added nginx and localstack services to docker compose yml.
  • Added 3 python libraries to poetry dependencies.

PR TASKS

  • The actual code changes.
  • Tests written and passed.
  • Any changes to docs?
  • Bumped version number in setup.py (also checked on PyPi).

rshewitt added 24 commits May 8, 2023 12:20
@rshewitt
add nginx and aws/s3 mock services.
a259759
@rshewitt
index to grab json in return tuple.
5dbf9b1
@rshewitt
add file.
bcb3711
@rshewitt
exclude openstack dir.
9606af0
@rshewitt
update file.
ec2a985
@rshewitt
add new libraries.
8d72eab
@rshewitt
add docker compose.
d5ff893
@rshewitt
add extract utilities.
b3d6cce
@rshewitt
add try/catch and update return.
217df1d
@rshewitt
create module for s3.
3a8cbdb
@rshewitt
add nginx config.
0091fd2
@rshewitt
create fixures for dcatus extract.
9d0840f
@rshewitt
add tests for dcatus extract.
2a3c71e
@rshewitt
add harvest source for nginx.
420b14f
@rshewitt
isolate processes as functions.
2808ca6
@rshewitt
refactor.
445c81c
@rshewitt
move processes to package scripts.
b976435
@rshewitt
remove unused imports.
aacb956
@rshewitt
test.
3404d23
@rshewitt
updated renamed function
786e92b
@rshewitt
simplified bucket creation.
4f538ce
@rshewitt
clean up.
3e393f9
@rshewitt
replace catalog with record upload.
2b3047e
@rshewitt
Merge remote-tracking branch 'origin' into extract-dcatus
e48cf18
@github-actions
Copy link

github-actions bot commented May 9, 2023 •
edited
Loading

Coverage

Coverage Report
FileStmtsMissCoverMissing
datagovharvester
   __init__.py30100% 
datagovharvester/extract
   __init__.py1922 89%
   dcatus.py1122 82%
   utils.py1644 75%
datagovharvester/utils
   __init__.py00100% 
   json_utilities.py722 71%
   s3_utilities.py1544 73%
datagovharvester/validate
   __init__.py00100% 
   dcat_us.py240100% 
TOTAL951485% 

Tests Skipped Failures Errors Time
16 0 💤 0 ❌ 0 🔥 21.204s ⏱️

@jbrown-xentity jbrown-xentity requested a review from a team May 10, 2023 18:29
nickumia-reisys
nickumia-reisys previously requested changes May 10, 2023
View reviewed changes
Copy link
Contributor

@nickumia-reisys nickumia-reisys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the new features to the README file or some other doc file.

.env Outdated Show resolved Hide resolved
datagovharvester/extract/utils.py Outdated Show resolved Hide resolved
datagovharvester/utils/s3_utilities.py Outdated Show resolved Hide resolved
tests/extract/test_dcatus.py Outdated Show resolved Hide resolved
datagovharvester/extract/utils.py Outdated Show resolved Hide resolved
btylerburton
btylerburton reviewed May 10, 2023
View reviewed changes
.gitignore Outdated Show resolved Hide resolved
datagovharvester/extract/utils.py Outdated Show resolved Hide resolved
datagovharvester/extract/utils.py Outdated Show resolved Hide resolved
datagovharvester/utils/s3_utilities.py Outdated Show resolved Hide resolved
docker-compose.yml Outdated Show resolved Hide resolved
tests/extract/conftest.py Outdated Show resolved Hide resolved
nickumia-reisys
nickumia-reisys reviewed May 30, 2023
View reviewed changes
tests/nginx.conf Outdated Show resolved Hide resolved
rshewitt and others added 18 commits June 1, 2023 09:56
@rshewitt rshewitt merged commit 21a0252 into main Jun 2, 2023
4 checks passed
@rshewitt rshewitt deleted the extract-dcatus branch June 2, 2023 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@robert-bryson robert-bryson robert-bryson approved these changes

@btylerburton btylerburton btylerburton left review comments

@jbrown-xentity jbrown-xentity jbrown-xentity approved these changes

@nickumia-reisys nickumia-reisys Awaiting requested review from nickumia-reisys

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@rshewitt @nickumia-reisys @robert-bryson @jbrown-xentity @btylerburton