[BUG-231283] Implement internal process to keep root CA store updated

[sl-service-account]

This may come across as a bit of a moan, but I hope that it is perceived in the constructive way that it is intended.

I'm rather concerned by the language on the Second Life status page, which implies that the expiry of the LetsEncrypt DST X3 root certificate was not really fully understood ("Our engineers are working hard to figure out why this certificate expiring had this effect").

This issue (BUG-231280) could have completely avoided. You were first alerted to this issue with BUG-230135 back in January. This was the point that LetsEncrypt introduced the cross-signed root certificate intended as a temporary stopgap to allow vendors time to update their root certificate stores. Essentially, you had 9 months in which to update your root certificate store (including the ISRG Root X1 certificate), which has been in the Mozilla root CA bundle since 2016 (https://bugzilla.mozilla.org/show_bug.cgi?id=1289889)

I also raised a ticket back in May 2020 which was related to a similar issue with Sectigo certificates (BUG-228848) which was never actually addressed (again the advice was to use a temporary cross-signed bundle but I ended up just moving to a different vendor because the root was never updated)

I also explicitly asked Oz in that ticket if LL had implemented internal procedures to get root certificates update regularly, to which he replied "Yes".

I do understand completely that such things can easily be overlooked, but there's an overwhelming amount of in-world content which depends on HTTPS. So, this ticket is an explicit request to implement internal policies and procedures to make sure that your root certificate bundle is updated on a regular basis. This really needs to be done proactively, not reactively, to avoid such situations in the future - because all root certificates will eventually expire.

Links

Related

• [BUG-228848] Legitimate SSL certificates being rejected #6834

Original Jira Fields

Field	Value
Issue	BUG-231283
Summary	Implement internal process to keep root CA store updated
Type	Bug
Priority	Unset
Status	Closed
Resolution	Not Applicable
Reporter	Casper Warden (casper.warden)
Created at	2021-10-01T10:15:56Z
Updated at	2021-10-04T17:02:28Z

{
  'Build Id': 'unset',
  'Business Unit': ['Platform'],
  'Date of First Response': '2021-10-01T10:38:53.404-0500',
  'ReOpened Count': 0.0,
  'Severity': 'Unset',
  'System': 'SL Simulator',
  'Target Viewer Version': 'viewer-development',
  'What just happened?': 'This may come across as a bit of a moan, but I hope that it is perceived in the constructive way that it is intended.\r\n\r\nI\'m rather concerned by the language on the Second Life status page, implying that the expiry of the LetsEncrypt DST X3 root certificate was not fully understood.\r\n\r\nThis issue could have completely avoided. You were first alerted to this issue with BUG-230135 back in January. This was the point that LetsEncrypt introduced the cross-signed root certificate intended as a **temporary stopgap** to allow vendors time to update their root certificate. Essentially, you had 9 months in which to update your root certificate store (the ISRG Root X1), which has been in the Mozilla root CA bundle since 2016 (https://bugzilla.mozilla.org/show_bug.cgi?id=1289889)\r\n\r\nI also raised a ticket back in May 2020 which was related to a similar issue with Comodo certificates  ( https://jira.secondlife.com/browse/BUG-228848 )  which was never actually addressed (again the advice was to use a temporary cross-signed bundle but I ended up just moving to a different vendor because the root was never updated)\r\n\r\nI also explicitly asked Oz in that ticket if LL had implemented internal procedures to get root certificates update reqularly, to which he replied "Yes".\r\n\r\nThis ticket is an explicit request to implement internal policies and procedures to make sure that your root certificate bundle is updated on a regular basis.',
  'What were you doing when it happened?': 'N/A',
  'What were you expecting to happen instead?': 'N/A',
}

[sl-service-account]

Mazidox Linden commented at 2021-10-01T15:38:53Z

We have a procedure to keep our root store up to date, and it was. The issue (and why it took so long to fix) is that we encountered a bug in a library we rely on which meant that even though we had a valid root certificate we had an invalid intermediate certificate which improperly caused the entire chain to be discarded, even though there was a different path to the valid root that would have succeeded.

[sl-service-account]

JIRAUSER338728 commented at 2021-10-01T23:30:01Z

sounds fair, those pesky intermediates.