Course/Program platform should not be nullable #2766

mbertrand · 2023-09-22T15:52:41Z

Every course and program should have a Platform assigned to it. The schema should be changed to reflect this:

    platform = models.ForeignKey(
        Platform, on_delete=models.PROTECT, null=False, blank=False
    )

arslanashraf7 · 2023-10-02T08:31:15Z

Some clarifications are needed before doing this.

Context:

Initially, the plan was to add Partner instead of Platform. But for the sake of consistency within multiple apps in mit-open we decided to add the Platform field instead. This was done as part of Add platform model association with Courses and Programs #2698.
The field was left nullable/blankable because we did not have partners for all the courses and thought the platform would only be set for external courses and not internal.

Problem: If we are to make it non nullable/blankable we need to decide how we will populate the existing data for courses and programs. Possible problems are:

How many Platform entries are to be created in the system before we do this change?
How do we decide which Course/Program to associate with which Platform?

A possible Solution: We can create a default platform in xPRO named MITxPRO and in the migration we set all the courses and programs to it by default. Later on, we can change the platform manually where needed. I believe manual intervention will be required for external courses only.

Ferdi · 2023-10-02T15:50:25Z

@arslanashraf7 how many courses/program do we have where platform is not set ?

arslanashraf7 · 2023-10-03T08:15:43Z

@Ferdi I looked at the data and here are the stats:

On Production:

Total Course	Courses Without Platform	Total Programs	Courses Without Program
154	77	26	19

Details (readable_id, live, is_external) about courses and programs with missing platforms can be seen below:

Courses list without platfrom

[
   {'readable_id': 'course-v1:xPRO+SysEngxB2', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE4B', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE3B', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE2B', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE1B', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM+YDL2', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM+YDL3', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM-FDL1', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM+FDL2', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM+FDL3', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-LASER-SL-Course2', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO-LASER-SL-Course3', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO-LASER-SL-Course4', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+CDAT2_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+CDAT3_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+CDAT1_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+AMxB', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+AMx_CROWN', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+AMxF+R1', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+ADETxB', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+ADSTxB', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+ADVTxB', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+DECA_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+DSCA_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+DVCA_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngx1_Raytheon', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCx3+R1', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+QCFx1TouchEdu', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCYx2+R1', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+PCCYx1+R1', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+DSx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+AMxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+MLx2DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+SysEng1xDEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERE4xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+DSxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERx4DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+ENxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+HNxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+QCFx1DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERx3DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERE3xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+MLx1DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+SysEng3xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+SysEng2xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERx2DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+GNxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+QCRx1DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+SysEng4xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+QCFx2DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+QCRx2DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:MITxPRO+LASERxE1DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERE2xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASER1xDEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCx2+R1', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+AMx_CROWN_FUNDAMENTALS', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+NHx', 'live': True, 'is_external': False},
   {'readable_id': 'v1-bootcamp-innovation-leadership', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+MBDTx', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+EngLeadxDEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:MITxPRO+LASERMC+R1', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+MLx1+R0', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+GNx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+SPOC_MIT_PE', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCx+R1+Course2_DO_NOT_USE', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCx+R1+Course3_DO_NOT_USE', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCYx-DONOTUSE', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCDEx+Course2', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+PCDGx+Course2', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+PCTPMx+Course3', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+ENxSE', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE1_FreddieMac', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-WHU+TTLS', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-WHU+TTLS+R0', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngxNAV', 'live': False, 'is_external': False},
   {'readable_id': 'v1-bootcamp-venture', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+PCCx1+R1', 'live': False, 'is_external': True}
]

Programs list without platfrom

[
   {'readable_id': 'program-v1:xPRO-SL+LASERx-SL', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO-ADGM+FDL', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO-ADGM+YDL', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+CB_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+LASERx', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+SysEngxB', 'live': False, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+LASERxB', 'live': False, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+LASERxEB',  'live': False, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+PCCx+R1', 'live': False,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+PCCxW+R1',  'live': False,  'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCCYx', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+TIAPx+R1',  'live': True,  'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCDEx', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCCFTx', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCTPMx', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+LASER-SL',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+PCGDx', 'live': True, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+DONOTUSE',  'live': False,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+CDAT_Boeing',  'live': True,  'is_external': False}
]

On RC:

Total Course	Courses Without Platform	Total Programs	Courses Without Program
55	65	16	15

Courses list without platfrom

[
   {'readable_id': 'course-v1:xPRO+SysEngxB1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngxB3',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngxB2',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngxB4',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+DgtlLearn1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+DgtlLearn2',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+DgtlLearn3',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+AnlgLearn1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+AnlgLearn2',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+AnlgLearn3',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERx4', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERx3', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERx2', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERx1', 'live': True, 'is_external': False},
   {'readable_id': 'SysEngxB3', 'live': False, 'is_external': False},
   {'readable_id': 'SysEngBx2', 'live': False, 'is_external': False},
   {'readable_id': 'SysEngBx4', 'live': False, 'is_external': False},
   {'readable_id': 'SysEngBx3', 'live': False, 'is_external': False},
   {'readable_id': 'TestX1', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:TESTx+hidden2x',  'live': False,  'is_external': False},
   {'readable_id': 'course-v1:TESTx+hidden1x',  'live': False,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+TestCourse1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+TestCourse2',  'live': True,  'is_external': False},
   {'readable_id': 'ATCx', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+Bonx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+Bonx+SPOC',  'live': False,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+external1+R1',  'live': True,  'is_external': True},
   {'readable_id': 'course-v1:xPRO+external2+R1',  'live': True,  'is_external': True},
   {'readable_id': 'course-v1:xPRO+external3+R1',  'live': True,  'is_external': True},
   {'readable_id': 'course-v1:arslan_external_1',  'live': True,  'is_external': True},
   {'readable_id': 'ATP_external_2', 'live': True, 'is_external': True},
   {'readable_id': 'arslan_test_internal_externl_1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+BONx2', 'live': True, 'is_external': False},
   {'readable_id': 'course_without_cms_page',  'live': True,  'is_external': False},
   {'readable_id': 'AMxB', 'live': True, 'is_external': False},
   {'readable_id': 'AMxBIntro', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:MITx+Ali-Test-Course',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:MITx+101', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ARSLAN2', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:edx+BlockchainBasics',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+ExtCourse',  'live': True,  'is_external': True},
   {'readable_id': 'brian_test_external_page_2',  'live': False,  'is_external': True},
   {'readable_id': 'DSx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+ENx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+MLx1+R0', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:MAPLE+MAPLE', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+EngWeb', 'live': True, 'is_external': True},
   {'readable_id': 'QCFx2', 'live': True, 'is_external': False},
   {'readable_id': 'RCHNTC', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:edX+DemoX+Demo_Course',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+NotLive1',  'live': False,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+QuantCompx1',  'live': True,  'is_external': False},
   {'readable_id': 'external-readable-id', 'live': True, 'is_external': True},
   {'readable_id': 'TC3', 'live': True, 'is_external': False}
]

Programs list without platfrom

[
   {'readable_id': 'program-v1:ARSLAN+TestProgram',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+LASERx', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:MITxPro+SysEngxB',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+SysEngx',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+DgtlLearn',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+AnlgLearn',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+SysEngBx',  'live': True,  'is_external': False},
   {'readable_id': 'TestX', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:TESTx+SysEngBx',  'live': False,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+TestProgram',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+Bonx', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+externalprogram',  'live': True,  'is_external': True},
   {'readable_id': 'ATP_external_1', 'live': True, 'is_external': True},
   {'readable_id': 'test_internal_program_external',  'live': True,  'is_external': False},
   {'readable_id': 'progam_without_cms_page',  'live': True,  'is_external': False}
]

Let me know if you require further details.

Ferdi · 2023-10-03T13:06:43Z

Thanks Arslan, why don't I see these courses here https://xpro.mit.edu/api/courses/?

arslanashraf7 · 2023-10-03T13:37:55Z

Thanks Arslan, why don't I see these courses here https://xpro.mit.edu/api/courses/?

That would be because the /api/courses only returns those courses that have:

live=True (This is from Django)
coursepage.live=True (This is CMS page for the course)
externalcoursepage.live=True (Again, from CMS but for external courses)

*Edit
In the list that I provided above, I didn't filter the data based on any of these conditions because we will need a platform for all of those irrespective of the above conditions.

Ferdi · 2023-10-03T13:51:51Z

ok, should we fix the data before making the field non-nullable or after. Most of them seem to be external = false, so we can set the platform to mitxpro.

arslanashraf7 · 2023-10-03T14:00:39Z

ok, should we fix the data before making the field non-nullable or after. Most of them seem to be external = false, so we can set the platform to mitxpro.

We can do both ways.

If we know what platform we want for all the remaining courses and programs, we can automatically fix it in the Django migration that we will generate to make this field non-nullable.
If not, Then we need to fix the data before we make it non-nullable.

Ferdi · 2023-10-03T14:14:02Z

it seems like external courses/programs need to be fixed before. Internal ones (platform = mitxpro) can be done during migration

arslanashraf7 · 2023-10-03T14:24:49Z

it seems like external courses/programs need to be fixed before. Internal ones (platform = mitxpro) can be done during migration

I agree. I suppose the course team would fix that data? Or I can but I will require details about the external courses. In any case, we can put up the PR as soon as the data is fixed.

Ferdi · 2023-10-04T02:05:51Z

Can we get a list of the external courses/programs for @cachob to look at ?

arslanashraf7 · 2023-10-04T10:42:22Z

Can we get a list of the external courses/programs for @cachob to look at ?

Sure.

@cachob Please take a look below for the list of external courses and programs without the platform. For ease, I've convertedthe lists to easily clickable links.

Production:

Here is the list of external courses without a platform

Here is the list of external programs without a platform

https://xpro.mit.edu/admin/courses/program/20/change/
https://xpro.mit.edu/admin/courses/program/21/change/
https://xpro.mit.edu/admin/courses/program/22/change/
https://xpro.mit.edu/admin/courses/program/23/change/
https://xpro.mit.edu/admin/courses/program/24/change/
https://xpro.mit.edu/admin/courses/program/25/change/
https://xpro.mit.edu/admin/courses/program/27/change/

RC: (Changing this doesn't matter much since this is just test data but in case you would like to take a look)

Here is the list of external courses without a platform

Here is the list of external programs without a platform

https://rc.xpro.mit.edu/admin/courses/program/13/change/
https://rc.xpro.mit.edu/admin/courses/program/14/change/

cachob · 2023-10-04T13:29:45Z

@arslanashraf7 - I noticed that a lot of the courses are the dummy courses that we had to create to correctly display the amount of courses for a program in the catalog. The others are also not set as live. I will clean up and update both courses and programs on the lists as needed, and set the default value as xPRO if applicable

arslanashraf7 · 2023-10-04T13:31:23Z

@arslanashraf7 - I noticed that a lot of the courses are the dummy courses that we had to create to correctly display the amount of courses for a program in the catalog. The others are also not set as live. I will clean up and update both courses and programs on the lists as needed, and set the default value as xPRO if applicable

@cachob Thanks. Let us know when you are done.

cachob · 2023-10-04T13:57:54Z

@arslanashraf7 - production data is all set

arslanashraf7 · 2023-10-18T08:38:15Z

Done via #2786. This has been deployed on production. The data migration for this was successful and there are no more courses or programs without a platform.

mbertrand added the Enhancement New feature or request label Sep 22, 2023

cachob added the product:xpro label Sep 22, 2023

arslanashraf7 self-assigned this Oct 6, 2023

arslanashraf7 mentioned this issue Oct 6, 2023

fix: make platform a required field for Courses/Programs #2786

Merged

8 tasks

arslanashraf7 closed this as completed Oct 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Course/Program platform should not be nullable #2766

Course/Program platform should not be nullable #2766

mbertrand commented Sep 22, 2023

arslanashraf7 commented Oct 2, 2023

Ferdi commented Oct 2, 2023

arslanashraf7 commented Oct 3, 2023

Ferdi commented Oct 3, 2023

arslanashraf7 commented Oct 3, 2023 •

edited

Loading

Ferdi commented Oct 3, 2023

arslanashraf7 commented Oct 3, 2023 •

edited

Loading

Ferdi commented Oct 3, 2023

arslanashraf7 commented Oct 3, 2023

Ferdi commented Oct 4, 2023

arslanashraf7 commented Oct 4, 2023

cachob commented Oct 4, 2023

arslanashraf7 commented Oct 4, 2023

cachob commented Oct 4, 2023

arslanashraf7 commented Oct 18, 2023

Course/Program platform should not be nullable #2766

Course/Program platform should not be nullable #2766

Comments

mbertrand commented Sep 22, 2023

arslanashraf7 commented Oct 2, 2023

Ferdi commented Oct 2, 2023

arslanashraf7 commented Oct 3, 2023

Ferdi commented Oct 3, 2023

arslanashraf7 commented Oct 3, 2023 • edited Loading

Ferdi commented Oct 3, 2023

arslanashraf7 commented Oct 3, 2023 • edited Loading

Ferdi commented Oct 3, 2023

arslanashraf7 commented Oct 3, 2023

Ferdi commented Oct 4, 2023

arslanashraf7 commented Oct 4, 2023

cachob commented Oct 4, 2023

arslanashraf7 commented Oct 4, 2023

cachob commented Oct 4, 2023

arslanashraf7 commented Oct 18, 2023

arslanashraf7 commented Oct 3, 2023 •

edited

Loading

arslanashraf7 commented Oct 3, 2023 •

edited

Loading