Skip to content
This repository has been archived by the owner on Jan 13, 2022. It is now read-only.

Expiration of outdated images in the database #483

Merged
merged 10 commits into from Aug 25, 2020
Merged

Conversation

ChariniNana
Copy link
Contributor

Fixes

Fixes #482 by @ChariniNana

Description

This pull request handles the image expiration logic, where the updated_on value associated with an image is older than what it can be as per the re-ingestion logic of the corresponding provider.

Technical details

We specify the oldest possible value the updated_on attribute can take per each provider (based on the re-ingestion strategy) in a dictionary. If certain images in the database are deemed to be outdated as per this logic, we set the corresponding removed_from_source value to True.

Tests

The test_image_expiration function in the test_sql test suite checks whether the image expiration logic works as expected.

Checklist

  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@ChariniNana ChariniNana requested review from a team, kss682 and mathemancer and removed request for a team August 13, 2020 22:52
@kgodey kgodey added this to In Progress in Active Sprint Aug 13, 2020
@ChariniNana ChariniNana requested a review from a team August 17, 2020 10:37
Copy link
Contributor

@mathemancer mathemancer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made one stylistic suggestion.

Overall, this looks great though, thank you! I'm really excited to have this functionality.

src/cc_catalog_airflow/dags/image_expiration_workflow.py Outdated Show resolved Hide resolved
Co-authored-by: Brent Moran <brent@creativecommons.org>
Copy link
Contributor

@mathemancer mathemancer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thank you!

@mathemancer mathemancer merged commit c4ac873 into master Aug 25, 2020
Active Sprint automation moved this from In Progress to Done Aug 25, 2020
@mathemancer mathemancer deleted the image_expiration branch August 25, 2020 08:14
@TimidRobot TimidRobot removed this from Done in Active Sprint Jan 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

Successfully merging this pull request may close these issues.

[Infrastructure] Expire outdated images in the database
2 participants