mature -> sensitive code changes for the API

[AetherUnbound]

Fixes

Fixes #3723 by @AetherUnbound

Description

This PR applies the code changes for the API necessary to use sensitive internally rather than mature. Per the implementation plan linked in #3723, the mature property on the serializer and the mature property of Elasticsearch hits are unaffected.

This PR requires 2 migrations in order to appease Django. Both are no-ops:

1. Make explicit the tables which the mature media records are using
2. Rename the models and fields from mature to sensitive

Below are the sqlmigrate outputs for each migration:

j api/dj sqlmigrate api 0055

BEGIN;
--
-- Rename table for matureaudio to api_matureaudio
--
-- (no-op)
--
-- Rename table for matureimage to api_matureimage
--
-- (no-op)
COMMIT;

j api/dj sqlmigrate api 0056

BEGIN;
--
-- Rename model MatureImage to SensitiveImage
--
-- (no-op)
--
-- Rename model MatureAudio to SensitiveAudio
--
-- (no-op)
--
-- Alter field media_obj on sensitiveaudio
--
-- (no-op)
--
-- Alter field media_obj on sensitiveimage
--
-- (no-op)
COMMIT;

Warning:
I am no DRF aficionado, so I'm really not sure if everything I've done here is idiomatic or makes sense! Please let me know if there's a better approach to tackling this issue 😄

Testing Instructions

Tests should pass, and all routes should also work locally!

Checklist

• My pull request has a descriptive title (not a vague title likeUpdate index.md).
• My pull request targets the default branch of the repository (main) or a parent feature branch.
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no visible errors.
• I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[github-actions]

This PR has migrations. Please rebase it before merging to ensure that conflicting migrations are not introduced.

[obulat]

The code looks good and seems to work well.

I was testing this locally, and think I found an error with our local setup, or with this PR.
If you open http://0.0.0.0:50280/v1/images/41c8f114-f77d-45bc-b1ec-55c205f96e9f/, you should see the an item with sensitive_text, but the value there is an empty array. And this item is in both filtered and main indices, even though it's supposed to only be in the main one (due to sensitive term).

I don't remember how we set up the filtered index locally, and how we add the sensitivities in the API: is that the local setup problem?

[AetherUnbound]

@obulat I tested that result on main and it also shows no value for unstable__sensitivity. I'll look into why that's the case, but given that it's on main I don't think it should block this PR!

[AetherUnbound]

Ah, right! It's because we use a mock sensitive terms list: https://github.com/WordPress/openverse/blob/7427bbd4a8178d05a27e6fef07d70905ec7ef16b/ingestion_server/ingestion_server/static/mock_sensitive_terms.txt

[krysal]

The code looks good. I have a question similar to Olga's.

we use a mock sensitive terms list: 7427bbd/ingestion_server/ingestion_server/static/mock_sensitive_terms.txt

Does that means that items with those terms should be marked as sensitive? I searched for them but I couldn't find any. How can we check for an item where "mature": true?

[AetherUnbound]

we use a mock sensitive terms list: 7427bbd/ingestion_server/ingestion_server/static/mock_sensitive_terms.txt

Does that means that items with those terms should be marked as sensitive? I searched for them but I couldn't find any. How can we check for an item where "mature": true?

I'm not sure! I believe it's separate from the work on this PR though; @krysal or @obulat would either of you be willing to look into it this week to see if/how that mock list is being used?

[sarayourfriend]

Does that means that items with those terms should be marked as sensitive? I searched for them but I couldn't find any.

Yes, anything with those terms locally will get marked with having sensitive text. Compare the following two requests:

http://0.0.0.0:50280/v1/images/?q=water&unstable__include_sensitive_results=True
http://0.0.0.0:50280/v1/images/?q=water

All results with water are marked sensitive text with correctly populated sensitivity list.

How can we check for an item where "mature": true?

None of these exist in the testing data. That's why the one @obulat brought up isn't matched, it has none of the mock terms, and isn't marked mature in the index.

To test for mature: true locally, you need to add a confirmed sensitivity report via your local Django admin. Go to http://localhost:50280/admin/api/imagereport/add/?reason=mature&status=mature_filtered and select a media object to mark as "mature" (just choose any one of them you want) and save the report. Then you'll have one result with mature: true in the index.

[obulat]

Tested locally: added a mature report for an image, searched for it, checked that it has correct values in the sensitivities array. Tested that both mature and unstable__include_sensitive_results query parameters work.

@AetherUnbound, are there any special considerations for deploying this PR? If so, could you add a note for the next deployer?

Changes were addressed

[AetherUnbound]

No special considerations here! It will run the two no-op migrations on deployment, but nothing about the application itself should change 😄 I'll confirm it works in staging before deploying though.

[openverse-bot]

Based on the medium urgency of this PR, the following reviewers are being gently reminded to review this PR:

@krysal
This reminder is being automatically generated due to the urgency configuration.

Excluding weekend¹ days, this PR was ready for review 13 day(s) ago. PRs labelled with medium urgency are expected to be reviewed within 4 weekday(s)².

@AetherUnbound, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings.

Footnotes

1. Specifically, Saturday and Sunday. ↩

2. For the purpose of these reminders we treat Monday - Friday as weekdays. Please note that the operation that generates these reminders runs at midnight UTC on Monday - Friday. This means that depending on your timezone, you may be pinged outside of the expected range. ↩

[krysal]

Continues to work as expected! Thanks for reminding us of the testing instructions, @sarayourfriend, and great work, @AetherUnbound!

[AetherUnbound]

Going to merge and make sure it works in staging!