Production release

This release adds the following features:

• Added greatly improved logic to clean and verify URLs to the ImageStore class.
• Replaced a number of str.format() occurrences with f-strings.
• Fixed broken links in the README.
• Added logic to sync_commoncrawl_workflow that feeds all CommonCrawl-sourced metadata through the ImageStore class for cleaning.
• Added Provider API Script and associated Apache Airflow DAG to pull image metadata from the Walters Art Museum API.
• Added function to collect the description field from the RawPixel API.
• Split the python requirements.txt file into prod and dev versions.
• Added functionality to save full UTF-8 into the intermediary DB rather than just ASCII.
• Added Apache Airflow DAGs to clean preexisting data from the intermediary database by running it through the ImageStore class for cleaning.

Production release

This release adds the following features:

• Increased completeness and correctness of metadata collected by the Smithsonian Institution Provider API Script:
  • Increased coverage of creator and description fields
• Automated linting of PRs
• An increased timeout for Europeana ingestion tasks
• A function to collect categories info for Wikimedia Commons media
• A script to merge old data generated for metadata gathered from Common Crawl with new data gathered from Provider API Scripts
• An Apache Airflow DAG to mark images which have old metadata as 'expired'

Production Release

This release adds the following features:

• Rewritten Brooklyn Museum Provider API Script
• DAG to implement reingestion strategy for Europeana
• DAG to automatically check for new Smithsonian Unit Codes
• Three DAGs to define and refresh data pipeline for Popularity Metric Calculation

Production Release

This release adds the following features:

• Add WOCinTech as a sub provider of Flickr
• Split all Smithsonian Museums out as sub-providers (sources)
• Ingest NYPL using a Provider API Script

This release fixes the following bug:

• Change provider value of Museums Victoria to include 's'.

Production Release

This release adds two new features:

• There is a new Provider API Script and associated Apache Airflow DAG to get data from Museums Victoria
• The Europeana Provider API Script can now separate out Wellcome Collection as a subprovider.

Production Release

This release fixes a bug in the Wikimedia Commons Provider API Script which caused non-image types' metadata to be saved. It also reduces the number of records per request in the same script, to lower the load on the Wikimedia Commons infrastructure.

Production Release

This release has a few minor changes and bug fixes.

• Fix bugs in the popularity calculation workflow.
• Fix bug in MET Museum Provider API Script.
• Add total_images property to ImageStore class.

Production Release

• Fixes bugs in Popularity Metric calculation Workflow
• Adds subproviders of Flickr:
  • Biodiversity Heritage Library
  • NASA
  • SpaceX
• Adds Statens Museum for Kunst (National Gallery of Denmark) provider
• Minor bug fixes

Production Release

• Changes schema of main image table to enforce tighter uniqueness constraints.
• Adds calculation of new standardized popularity metric whenever popularity data (e.g., views) is available for a set of images.
• Improves Loader Workflow so that we can now load TSV with a limited number of malformed rows.
• Adds Europeana (https://www.europeana.eu/) as an image provider.
• Migrates the Science Museum Group (https://www.sciencemuseumgroup.org.uk/) to a Provider API Script image provider.
• Fixes bug in Wikimedia Commons Provider API Script so that we now handle empty API responses quietly.

Production Release

• All Provider API Scripts using the new ImageStore class to store their images.
• Apache Airflow DAGs have been made more uniform, and use Airflow's date templating.
• We store all TSV files generated by Provider API Scripts to AWS S3 as well as loading them to PostgreSQL.
• Greatly increased test coverage.