in cron job, use batch select for artifacts across all courses, instead of doing one course at a time

[zqian]

Expected behavior (A clear and concise description of what you expected to happen) :

There is still room for improvement for database select in cron.py

Describe the bug (Tell us what happens instead of the expected behavior) :

It looks like the util_function for executing database query is called for each course separately, for various objects.

For example, https://github.com/tl-its-umich-edu/my-learning-analytics/blob/master/dashboard/cron.py line 370 is executed for every course to get the course assignment data.

We could do a batch select for all registered courses

[jonespm]

The objects that this could be improved in are

However looks like the only slow process in the cron currently is the update_submission process which currently takes 13 minutes (more than half of the time of a typical 23 minute run run). That might be the only part to focus on. Though this seems like low priority.

• assignment and co.lms_int_id = '{course_id}'
• assignment_groups and co.lms_int_id = '{course_id}'
• assignment_weight and co_km.lms_int_id = '{course_id}'
• course WHERE co2.lms_int_id = '{course_id}'
• submission co.lms_int_id = '{course_id}'
• user co.lms_int_id ='{course_id}'

It's possible changing some of these would require more processing in the backend job.

It looks like the resource query is already does this efficiently with and co_km.lms_int_id in %(course_ids)s

[pushyamig]

I will QA this