Remove micromasters ETL - #3663
Merged
Merged
Conversation
OpenAPI ChangesNo changes detected Unexpected changes? Ensure your branch is up-to-date with |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR removes the deprecated MicroMasters ETL pipeline and its operational hooks (Celery task + beat schedule, management command, settings/env wiring), and adds a cleanup migration to unpublish any remaining etl_source=micromasters resources while triggering the standard unpublish/deindex plugin flow.
Changes:
- Removed MicroMasters ETL pipeline code (
learning_resources/etl/micromasters.py), pipeline composition, Celery task + beat schedule entry, and related tests. - Removed
MICROMASTERS_CATALOG_API_URLconfiguration and associated dev/env references, plus the devcontainer startup command that invoked the removed backpopulate command. - Added a data migration to unpublish remaining MicroMasters resources and trigger deindexing via
resource_unpublished_actions.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
main/settings_course_etl.py |
Removes the MicroMasters catalog API setting. |
main/settings_celery.py |
Removes the Celery beat schedule entry for MicroMasters updates. |
learning_resources/tasks.py |
Removes the get_micromasters_data Celery task. |
learning_resources/tasks_test.py |
Updates cache-clearing test expectations and removes the MicroMasters task test. |
learning_resources/migrations/0117_unpublish_micromasters_resources.py |
Adds a cleanup migration to unpublish/deindex remaining MicroMasters resources. |
learning_resources/management/commands/backpopulate_micromasters_data.py |
Removes the backpopulate management command. |
learning_resources/etl/pipelines.py |
Removes micromasters import and micromasters_etl pipeline composition. |
learning_resources/etl/pipelines_test.py |
Removes micromasters_etl pipeline test. |
learning_resources/etl/micromasters.py |
Removes the MicroMasters ETL implementation. |
learning_resources/etl/micromasters_test.py |
Removes tests for the MicroMasters ETL module. |
env/codespaces.env |
Removes MicroMasters catalog/course env vars from codespaces env. |
env/app_vars.sh |
Removes MicroMasters catalog API var from app var generation script. |
.devcontainer/devcontainer.json |
Removes the MicroMasters backpopulate command from the devcontainer post-start bootstrap. |
abeglova
requested changes
Jul 23, 2026
abeglova
left a comment
abeglova
left a comment
Contributor
There was a problem hiding this comment.
Should you remove references to micromasters in learning/resources/etl/constants.py now or will they be removed in a followup
Member
Author
|
Leaving EtlSource.micromasters because it is referenced by a migration and existing LearningResource records |
Comment on lines
+4
to
+8
| def delete_micromasters_resources(apps, schema_editor): | ||
| """Deindex and delete any remaining micromasters resources""" | ||
| # Deindex synchronously rather than via the async plugin hooks: the | ||
| # OpenSearch/Qdrant removal serializers query the DB by id, so the rows | ||
| # must still exist when they run |
Comment on lines
80
to
84
| class ETLSource(ExtendedEnum): | ||
| """Enum of ETL sources""" | ||
|
|
||
| micromasters = "micromasters" | ||
| mit_edx = "mit_edx" | ||
| mitpe = "mitpe" |
mbertrand
force-pushed
the
remove-micromasters-etl
branch
from
July 23, 2026 18:36
6fae9e6 to
43bda3d
Compare
MicroMasters programs are no longer ingested from the micromasters catalog API. Removes the ETL module, celery task, beat schedule entry, backpopulate command, settings, and tests. Adds a data migration that unpublishes any remaining micromasters resources and deindexes them from OpenSearch/Qdrant via the standard unpublished plugin hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Address review: query/update via apps.get_model so the migration stays valid as the schema evolves, and dispatch deindexing through the id-based bulk_resources_unpublished_actions instead of passing model instances to the per-resource hook. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mbertrand
force-pushed
the
remove-micromasters-etl
branch
from
July 23, 2026 18:36
43bda3d to
dd2f598
Compare
abeglova
approved these changes
Jul 23, 2026
Member
Author
|
will delete existing ETLSource.micromasters records, remove that constant, update affected earlier migrations in a followup PR |
This was referenced Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
N/A - all micromasters programs are now ingested via the mitxonline ETL
Description (What does it do?)
Removes all code related to the MicroMasters ETL:
micromastersETL module and its testsmicromasters_etlpipelineget_micromasters_datacelery task and its celerybeat schedule entrybackpopulate_micromasters_datamanagement commandMICROMASTERS_CATALOG_API_URLsetting and related env/devcontainer entriesAdds a data migration that unpublishes any remaining learning resources with
etl_source=micromastersand deindexes them from OpenSearch/Qdrant via the standardresource_unpublished_actionsplugin hooks.Kept (not part of the ETL):
ETLSource.micromastersenum (referenced by migration 0052),CertificationType.micromasters(used by the mitxonline ETL), andMICROMASTERS_CMS_API_URL(used for program letters).How can this be tested?
docker compose run --rm web python manage.py migrate learning_resources— any published micromasters resources should be unpublished and deindex tasks dispatched.docker compose run --rm web uv run pytest learning_resources/etl/pipelines_test.py learning_resources/tasks_test.py -n logical