Switch MyLA to pull data from UDP instead of UDW

[jennlove-um]

The pilot versions of MyLA used data from the UDW. With the UDP available, we need to switch to pulling the data from UDP instead.

[zqian]

FYI: here updated link of UDP Documentation:

https://docs.udp.unizin.org/ucdm/canvas-to-ucdm.html

[ssciolla]

Okay, I have put in some initial work that will hopefully help get us organized, or at least start thinking about this more. I've identified in the following queries from cron.py that will need to be re-written to use data from the UDP.

• A term query from update_term
• A course query from verify_course_ids
• A user query from update_user
• An assignment group query from update_groups
• An assignment query from update_assignment
• A submission query from update_submission
• A weight consideration query from weight_consideration
• A resource query from update_canvas_resource

To help with this effort, I have started work in my issue-489-set-up-cron-queries-refactor branch. There I've included reformatted versions of these eight queries, along with the skeleton of a JSON file called cron_queries.json. This is mostly still the brainstorming stage (nothing is hooked up yet to the project), so if you like, don't like, or have ideas about this approach, let me know.

[ssciolla]

In speaking with @jonespm, I'm now thinking that this may be the right (if not the ideal) time to talk about our current MySQL data model. I will create an EER diagram for us to look at in our next meeting.

[jonespm]

Some thoughts on this from the email. It's been mentioned the "Our hope is that all institutions still utilizing the UDW will be able to migrate their processes over to the UDP by the end of this calendar year, 2021."

Minimally:

• Decompose and understand each of the current queries that exist as part of the cron. These have evolved over time and some are likely inefficient and could be minimally improved. Identify current cron queries and understand what each one does #1266

• Sam identified 8 queries that in 2019. I'd expect more might have been added since then.
• Break up the parts a bit more.

• Rewrite each of the queries and ensure the data retrieved is identical for a select group of courses. The BigQuery can be excluded as it won't be needed to change Rewrite existing cron queries to get data from UDP and validate data matches UDW data queries #1267
• Add some process somewhere (another job?) that will compare the databases since we will be running both processes and need to ensure the results are identical. This could be just ensuring table counts are close and spot checking some rows in each table. Add a process better than 👀 to compare databases using UDW and UDP data #1268

Nice to have:

• Improve the cron process to make it run as a management command rather than via "cron" module. This is issue Change django-cron over to just a management command #664

• Retain the internal job log we currently have and fix the issue with the length of the log message.
• We may also want a process for cleaning up old job logs

• Ideally improve the data model as, no currentissue but mentioned in Switch MyLA to pull data from UDP instead of UDW #489

• This would have to be done as a second phase if we do it after we ensure the data is equilivant
• It would take some of the load off of the API and improve performance

• Select more in batch Issue in cron job, use batch select for artifacts across all courses, instead of doing one course at a time #827 or think of other ways of optimization. The current process takes 10 minutes and just increases in length.
• Fix the various bugs that exist in the cron CronJob is failing silently; AcademicTerms matching query does not exist #770, event data is lost for new students #1012, Out of bound errors on dates from import job  #1242
• See if we can take advantage of upsert on any other tables to avoid delete/insert like we are for Bigquery. This would allow us to run the job whenever we wanted.

• The biggest challenge I can think of here is removing old data, like deleted assignments
• Also upsert still isn't in pandas

• Provide visibility of the job log runs to the admin console
• Allow running the job from the admin console

[rpfernandezjr]

I see that this is in progress, and is labeled as priority, but is there an ETA on when this will get released?

[jennlove-um]

Still waiting for the metadata table from Unizin. The June 2022 release was delayed.

Unizin release notes - https://resources.unizin.org/display/UDP/Release+Notes

[jennlove-um]

Research whether we need the first name and last name from the UDP. Can we trust the LTI launch data instead and use that data for the profile?