Skip to content
This repository has been archived by the owner on Aug 4, 2023. It is now read-only.

Commit

Permalink
Refactor Phylopic to use ProviderDataIngester (#747)
Browse files Browse the repository at this point in the history
* Add phylopic to provider module

* Initial phylopic refactor

* Finish provider refactor

* Only use endpoint conditionally, add should_continue & fix params

* Correct get_response_json logic

* Initial test changes

* Update tests

* Add provider data ingester to list

* Improve documentation

Co-authored-by: sarayourfriend <24264157+sarayourfriend@users.noreply.github.com>

* Use computed endpoint rather than get_response_json override

* Rename _uuid to uid

* Remove scratch code

Co-authored-by: sarayourfriend <24264157+sarayourfriend@users.noreply.github.com>

* Remove _get_meta_data function

* Make Phylopic daily

* Move start date to first day data is actually made available

Co-authored-by: sarayourfriend <24264157+sarayourfriend@users.noreply.github.com>
  • Loading branch information
AetherUnbound and sarayourfriend authored Oct 24, 2022
1 parent a3f4a42 commit 1506d6d
Show file tree
Hide file tree
Showing 6 changed files with 366 additions and 395 deletions.
2 changes: 1 addition & 1 deletion DAGs.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ The following are DAGs grouped by their primary tag:
| [`metropolitan_museum_workflow`](#metropolitan_museum_workflow) | `@daily` | `True` | image |
| `museum_victoria_workflow` | `@monthly` | `False` | image |
| `nypl_workflow` | `@monthly` | `False` | image |
| [`phylopic_workflow`](#phylopic_workflow) | `@weekly` | `True` | image |
| [`phylopic_workflow`](#phylopic_workflow) | `@daily` | `True` | image |
| `rawpixel_workflow` | `@monthly` | `False` | image |
| `science_museum_workflow` | `@monthly` | `False` | image |
| [`smithsonian_workflow`](#smithsonian_workflow) | `@weekly` | `False` | image |
Expand Down
1 change: 1 addition & 0 deletions openverse_catalog/dags/common/loader/provider_details.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
WORDPRESS_DEFAULT_PROVIDER = "wordpress"
FREESOUND_DEFAULT_PROVIDER = "freesound"
INATURALIST_DEFAULT_PROVIDER = "inaturalist"
PHYLOPIC_DEFAULT_PROVIDER = "phylopic"

# Finnish parameters
FINNISH_SUB_PROVIDERS = {
Expand Down
Loading

0 comments on commit 1506d6d

Please sign in to comment.