Skip to content
This repository was archived by the owner on Jan 13, 2022. It is now read-only.

Production release

Latest
Latest

Choose a tag to compare

View all tags
@mathemancer mathemancer released this 20 Oct 18:21
v0.9.0
727e355
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

This release adds the following features:

  • Added greatly improved logic to clean and verify URLs to the ImageStore class.
  • Replaced a number of str.format() occurrences with f-strings.
  • Fixed broken links in the README.
  • Added logic to sync_commoncrawl_workflow that feeds all CommonCrawl-sourced metadata through the ImageStore class for cleaning.
  • Added Provider API Script and associated Apache Airflow DAG to pull image metadata from the Walters Art Museum API.
  • Added function to collect the description field from the RawPixel API.
  • Split the python requirements.txt file into prod and dev versions.
  • Added functionality to save full UTF-8 into the intermediary DB rather than just ASCII.
  • Added Apache Airflow DAGs to clean preexisting data from the intermediary database by running it through the ImageStore class for cleaning.
Assets 2
Loading