Professional Education ETL Pipeline

[mbertrand]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/4731

Related ol-infrastructure PR: mitodl/ol-infrastructure#2536

Description (What does it do?)

• Adds a new ETL pipeline to ingest courses and programs directly from Professional Education's own API feed instead of Prolearn's.
• Excludes PE courses from being ingested by the Prolearn ETL pipeline.

Screenshots (if appropriate):

How can this be tested?

• Set the following values in your backend .env file:

   PROLEARN_CATALOG_API_URL=https://prolearn.mit.edu/graphql
   MITPE_BASE_API_URL=https://professional.mit.edu/
  

• Run ./manage.py backpopulate_prolearn_data

• When it is done, there should be roughly >130 learning resources with etl_source=prolearn and offered_by=mitpe. They will all be courses and not programs. http://api.open.odl.local:8063/admin/learning_resources/learningresource/?o=2&offered_by__code__exact=mitpe

• If you don't already have a populated featured learning path for Professional Education, run ./manage.py populate_featured_lists to create one

• Switch to this branch.

• Set MITPE_API_ENABLED=True in your backend .env file, restart containers.

• Run ./manage.py backpopulate_mitpe_data followed by ./manage.py backpopulate_prolearn_data. When it is done, most/all the same courses returned above should now have etl_source=mitpe and have new readable_id values. Some that were courses in prolearn should be unpublished, and created as new programs instead of courses. There should not be any published dupes from different sources (prolearn and mitpe). Some will be unpublished now due to changed criteria (past enrollment end date or past end date).

• If you go to http://open.odl.local:8062/learningpaths and open the Professional Education featured learning path, it will be empty. Run the following to reassign the old featured courses from prolearn to their new equivalents

./manage.py transfer_list_resources course course title prolearn mitpe
./manage.py transfer_list_resources course program title prolearn mitpe

• Run ./manage.py update_index --all
• Once complete, you should have courses/programs in the MITPE featured learning path again. These commands will need to be run on RC/production as well to retain the current featured lists.

[abeglova]

Works great! Back to you with a request for some variable name changes and a question

[abeglova]

I am a little confused by this logic. Why is the delete necessary? And does that mean that courses that are imported as unpublished, such as mitpe courses with no future runs in the logic below are deleted and recreated every run?

[mbertrand]

This code should be run only when the readable_id is not the unique_field value, and in that case if there may be multiple unpublished resources with the same value for unique_field that need to be removed to avoid database duplicate errors.

Good catch on the mitpe courses with no future runs though, those probably should just be skipped altogether instead of (re)created as unpublished. Not much point in importing unpublished resources.

[abeglova]

should this be called image_data_url or something like that?

[abeglova]

Can this be

topics_exist = resource_data["relationships"]["field_course_topics"]["data"] 

instead of reusing the variable name topic_data to mean different things

[mbertrand]

@abeglova Peter and I are going to have a meeting at some point with the Professional Education API developer to discuss possible enhancements like clearly delineating programs vs courses. So I'm going to label this as "Blocked" until after that meeting.

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[abeglova]

work great one minor fix required

[abeglova]

This duplicates settings.MITPE_BASE_API_URL. You should get rid of either the constant or the setting

[mbertrand]

Good catch, at first the value was different while the API was in development (professional-dev.mit.edu) but now they're the same and that's unlikely to ever change.

[mbertrand]

There's already a settings.MITPE_BASE_URL, going to use that here instead and remove both the above constant BASE_URL as well as settings.MITPE_BASE_API_URL