Contentfile archive comparison fix

[mbertrand]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/4567

Description (What does it do?)

Uses a different method of comparing tarfiles that seems more accurate in distinguishing different vs identical archives.

How can this be tested?

• Set .env variables for AWS_* and MITX_ONLINE_* using RC values
• Run ./manage.py backpopulate_mitxonline_data if you haven't done so before.

from learning_resources.models import LearningResourceRun
from learning_resources.etl.edx_shared import *

run = LearningResourceRun.objects.get(run_id="course-v1:MITxT+8.01.4x+1T2024")
sync_edx_course_files(
    "mitxonline", 
    [run.learning_resource.id], 
    ["20231213/courses/course-v1:MITxT+8.01.4x+1T2024.tar.gz"]
)

This should take a little while, and a bunch of contentfiles should be imported. You will see the error "Could not read verticals from path /tmp/course-v1:MITxT+8j2tzjuox/course" at one point, that's okay.

Now run sync_edx_course_files again, slightlly altering the last argument value to the next day's archive:

sync_edx_course_files(
    "mitxonline", 
    [run.learning_resource.id], 
    ["20231214/courses/course-v1:MITxT+8.01.4x+1T2024.tar.gz"]
)

It should finish much more quickly and you should see the log message "Checksums match for 20231214/courses/course-v1:MITxT+8.01.4x+1T2024.tar.gz, skipping"

Run again, with a more recent archive that has changes:

sync_edx_course_files(
    "mitxonline", 
    [run.learning_resource.id], 
    ["20240613/courses/course-v1:MITxT+8.01.4x+1T2024.tar.gz"]
)

It should take a little while again and you should not see the same log message above.

[jkachel]

LGTM 👍