SRI hashes changed on multiple libraries yesterday and some SRI hashes are wrong

[ssigwart]

Yesterday (2023-07-31), I noticed that the SRI hash for https://cdnjs.cloudflare.com/ajax/libs/vue/3.3.2/vue.global.min.js (https://cdnjs.com/libraries/vue/3.3.2) changed. It used to be sha512-6mO8pNkTyFMzOwbajocp9NbJzQUVV+zBFPZW8pKpIKnjhHYY19ez+tISSNkkvaBsj/nHq/E2FANyg6YlB7A+dQ==. Now it's sha512-twbmYKE2H3LPbArlkbjDppF/XJ+GVTM/RvEceYhFMEl6xi1PF1Q5004mNU8BFp3WKatWanAFKU2SQbdk4PdjXg==.

Also, I noticed that the SRI hash for https://cdnjs.cloudflare.com/ajax/libs/vue/3.3.4/vue.global.prod.min.js is wrong. https://cdnjs.com/libraries/vue/3.3.4 says it should be sha512-pHXx64U1UkFexsKz00jfapuz0FgKq+a+5dBTuitirYVcJzaxPqTbNbCWsUiFyTYVYmHMiql9zLCeod+IpAJUew==. However, https://www.srihash.org/ says it should be sha512-39BSQXI5q1XlvVhLfFRidKG8KM6Tr6VS/XSnNo6N/A0ZXexHCeoUI/s+ulujQy3UREjoLNrMnFat8VI0mMugWA==.

I did some debugging and found these logs:

• Processing logs: cdnjs/logs@cb82d2b
• KV Publish logs: cdnjs/logs@1feb270

From that, it looks like version 3.3.2 was published on 5/12, but was updated yesterday at 12:07:49. You can also see that the SRI hashes changed. I'm not sure if it's normal that old packages will be regenerated periodically.

Based on comparing an old copy of vue.global.min.js with the new one, it seems like something changed in the minification process, so different variable names were used.

I searched commits in https://github.com/cdnjs/logs for "update kv-publish". From that, I found several other libraries updated yesterday where the SRI tags changed:

• cdnjs/logs@10c27fa
• cdnjs/logs@c1d8d7c
• cdnjs/logs@8a46240
• cdnjs/logs@b7606ff

From them, it definitely seems like the issue is that:

1. Existing versions were regenerated.
2. If those versions had minified versions, those ended up with their SRI hash changing.

I still don't know if that explains why vue/3.3.4/vue.global.prod.min.js is wrong.

[MattIPv4]

Hey!

It looks like a backfill was run yesterday against a large set of libraries to populate them in new infrastructure -- this backfill was not meant to update the files on the CDN, nor the SRI hashes, as we consider all files on the CDN to be immutable, it was only meant to populate the libraries in the new infrastructure.

However, due to a bug in the processing logic, that being that the minification dependencies used weren't pinned to a specific version, many of the minified files were regenerated during the backfill processing and the new versions were pushed to the CDN with slightly different contents and new SRI hashes.

I flagged this internally this morning when I discovered that the backfill had caused SRI hashes to change, and the folks involved in the backfill have put a fix in place to pin the minification dependencies and re-run the backfill again, which should hopefully result in all the changes files and SRI hashes being updated again back to what they were before (this second backfill run is actively processing as we speak, it just takes a while to get through everything).

I have an open question for the folks running this logic around how we plan to verify that we are back to the state we were in before this backfill was run, so that folks' sites aren't broken with SRI hash mismatches going forward. I will update here as I hear more.

Apologies to anyone whose site this has broken!

[ssigwart]

Thanks for the quick reply, @MattIPv4!

For verifying the old SRI tags, perhaps you can find any kv-publish.log that was updated yesterday and grab the original SRI hashes from there.

[MattIPv4]

https://status.cdnjs.com/incidents/j85nrhmbcbc5 has been published to track this as an incident impacting the CDN.

[xtuc]

• https://cdnjs.cloudflare.com/ajax/libs/vue/3.3.2/vue.global.min.js is 6mO8pNkTyFMzOwbajocp9NbJzQUVV+zBFPZW8pKpIKnjhHYY19ez+tISSNkkvaBsj/nHq/E2FANyg6YlB7A+dQ==
• https://cdnjs.cloudflare.com/ajax/libs/vue/3.3.4/vue.global.prod.min.js is 39BSQXI5q1XlvVhLfFRidKG8KM6Tr6VS/XSnNo6N/A0ZXexHCeoUI/s+ulujQy3UREjoLNrMnFat8VI0mMugWA==

Those files are back to normal. I will invalidate the cache globally, please let me know if this fixes the issue for you.

[ssigwart]

Thanks, @xtuc.

For the 3.3.2 update, that did solve it. However, I had to do a hard browser refresh to get the corrected version because it looks like the caching headers allow the browser to reuse the same file for up to 355 days. I think that's going to have lingering effects for users that happened to visit websites in the last day or so and have the incorrect version cached. I don't think there's much that can be done about that though.

For the 3.3.4 update, https://cdnjs.com/libraries/vue/3.3.4 is showing the correct SRI hash now, but https://cdnjs.com/libraries/vue still has sha512-pHXx64U1UkFexsKz00jfapuz0FgKq+a+5dBTuitirYVcJzaxPqTbNbCWsUiFyTYVYmHMiql9zLCeod+IpAJUew==.

[AKHorizon]

still having issue with vue 3.3.4
https://cdnjs.cloudflare.com/ajax/libs/vue/3.3.4/vue.global.min.js is showing:
sha512-fcdKKm7U9XeFv9U8RsC9vwQfGoLk2vnUXhFAzJk1bCtWUvNxJP2HaSSW1+RopF16gpk4MLkYFbYukbuQX7e4Ug==

[MattIPv4]

cdnjs/logs@8a96350 indicates the asset in KV has been rolled back to the prior version and matches the origin SRI hash (cdnjs/logs@46f0f30). I suspect you're still seeing a cached version of the bad state from yesterday?

@xtuc has there been any thought given to purging the CDN cache as a whole, given how many libraries and files were impacted by this incident? Of course, end-users are going to be suffering from this for roughly a year with the cache headers we have in place, but at least we can clear out any bad responses from the CDN itself?

[MattIPv4]

@xtuc Could you take a look at this? https://github.com/cdnjs/logs/commits/prod/packages/c/codemirror/5.48.4 doesn't have any of the kv-publish logs I'd expect to verify whether the SRI hashes were changed/reverted as part of this incident?

[Martin-F]

I just ran into this problem, and found this:

We'll continue to monitor for reports of any hash mismatches, which should be reported in the issue on GitHub

...so, here I am.

My affected code uses https://cdnjs.com/libraries/codemirror/5.38.0:

URL	Expected SHA-256	Current SHA-256
https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/codemirror.min.css	I8NyGs4wjbMuBSUE40o55W6k6P7tu/7G28/JGUUYCIs=	9T+2aDjQSkwR+EUy1CF0CdcHIcSvYBLTIcE0bIfGRvg=
https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/codemirror.min.js	/fNlNrKCT1Q8cEd3M2yaup/lkUpN1LQEG109ZPNskYk=	GTw5k2DzzSlHiUfE7e9WjJDJYfmRjieIe1laeXbQGFU=
https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/mode/xml/xml.min.js	QmQ8vh5c1+UzpnyWXYB7nnY1aqpJmXPVkFWtJHJkLnE=	G6wKF6cDqgsd/06cqf2ilYycFusA8WE+pnc5K9Q/8LE=
https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/mode/javascript/javascript.min.js	ylo1K/gUyfpfh6s3tKjY8Q6voK8/TCnx8eZaZeoMldo=	84b94SFsvUq1IsfeaC/aVoLY2KZ5WSx6RabGswnga8s=
https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/mode/css/css.min.js	YVb7n24DLwrjN0DghcwJYL17MI/X7e3c4EJZr6PwNdY=	oc1I6s4YVpw+rzBOMU7PGtwtXfnQrweg0vm7009U+3g=
https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/mode/htmlmixed/htmlmixed.min.js	CcVYnZJocbbLCyjOo/tA2QEH56hhTBEPZX+che0RYSo=	CzgL3sVoqXsZU1T+JHgK8qZnnB3OKdvGcbb+n+KPMkU=

[xtuc]

Our cdnjs.cloudflare.com main storage got out of sync with the cdnjs/cdnjs git repo, after syncing:

@Martin-F:

• cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/codemirror.min.css: sha256-I8NyGs4wjbMuBSUE40o55W6k6P7tu/7G28/JGUUYCIs=
• cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/codemirror.min.js: sha256-/fNlNrKCT1Q8cEd3M2yaup/lkUpN1LQEG109ZPNskYk=
• cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/mode/xml/xml.min.js: sha256-QmQ8vh5c1+UzpnyWXYB7nnY1aqpJmXPVkFWtJHJkLnE=
• cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/mode/javascript/javascript.min.js: sha256-ylo1K/gUyfpfh6s3tKjY8Q6voK8/TCnx8eZaZeoMldo=
• cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/mode/css/css.min.js: sha256-YVb7n24DLwrjN0DghcwJYL17MI/X7e3c4EJZr6PwNdY=
• cdnjs.cloudflare.com/ajax/libs/codemirror/5.38.0/mode/htmlmixed/htmlmixed.min.js: sha256-CcVYnZJocbbLCyjOo/tA2QEH56hhTBEPZX+che0RYSo=

@AKHorizon:

• cdnjs.cloudflare.com/ajax/libs/vue/3.3.4/vue.global.min.js: sha512-Wbf9QOX8TxnLykSrNGmAc5mDntbpyXjOw9zgnKql3DgQ7Iyr5TCSPWpvpwDuo+jikYoSNMD9tRRH854VfPpL9A==

@wh1t3t1p :

• cdnjs.cloudflare.com/ajax/libs/codemirror/5.48.4/addon/dialog/dialog.min.js: sha256-G+QhvxjUNi5P5cyQqjROwriSUy2lZtCFUQh+8W1o6I0=
• cdnjs.cloudflare.com/ajax/libs/codemirror/5.48.4/mode/xml/xml.min.js; cphnEddX56MtGJsi1PoCPLds+dlnDj1QQkAlCWeJYDo=

These libraries SRIs went back to normal. Also, I globally invalidated the cache for them.

[wh1t3t1p]

thanks for the resolution, much appreciated!

[Martin-F]

@xtuc can confirm that's fixed for me now, thank you!

[drewfreyling]

Thanks to everyone involved with fixing this. Is there anyway to understand the number of impacted libraries so we can proactively check things?

[SergeyTsalkov]

I just ran into another wrong hash: https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.29.4/moment.min.js

[MattIPv4]

Hm, that seems odd, we're back to serving assets from KV now (rather than R2 that caused the hashes to be different for a while), so everything should be back to how it was before that change. 🤔

@xtuc thoughts?

It looks like we re-processed it in https://github.com/cdnjs/logs/tree/prod/packages/c/codemirror/5.48.4, and it looks like that didn't write to KV? So my thinking would be KV has the original assets (and hashes) from when this was added to cdnjs years ago, and R2 had a pre-processed version for a bit that was different?

[MattIPv4]

@njzjz The hash being different is expected, when compared to other CDNs, as we compress libraries ourselves.

Please only report issues here if you are seeing that the hash from cdnjs has changed over tine.

[MattIPv4]

For the sake of complete transparency here, I've heard nothing further from folks that I have contact with at Cloudflare. As such, I've filed a formal security report through HackerOne for the fact that SRI hashes are mismatching, indicating that files on the CDN (that are meant to be immutable) have changed. Hopefully, that'll finally get some eyes on the issue. I've referenced this issue as well as #14080.

[xtuc]

@jschwartzentruber prism 1.16.0 has been fixed:

$ curl https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/prism.min.js -vvv | sri-sha512
b6x3q+bsDBUcTzuCP9rsvFzqb/LJgkSxy7J0K3M+BFubzcLFkJTOoLG/GYfZr4W65qFV25PIxn+KmlQnoZPAWw==

$ curl https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/themes/prism.min.css | sri-sha512 
zpEXay1ygEa8Bjw1KCXJwsP2BvHNwiB9kYH+fnZmesHI6ixnaN49rcR8X9gx7SfzOW1lybh/TKvcqldC+SLc2w==

[xtuc]

codemirror 5.48.4 has been updated. @wh1t3t1p would you be able to double check?

[xtuc]

Closing this issue as all the affected libraries have now been restored.

[Ouroboros2460]

Hey guys, I am not sure if this is the same cause, but I am pretty sure the SRI hash for https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.30.1/moment.min.js is wrong.

https://cdnjs.com/libraries/moment.js gives sha512-hUhvpC5f8cgc04OZb55j0KNGh4eh7dLxd/dPSJ5VyzqDWxsayYbojWyl5Tkcgrmb/RVKCRJI1jNlRbVP4WWC4w== but

https://www.srihash.org gives sha512-QoJS4DOhdmG8kbbHkxmB/rtPdN62cGWXAdAFWWJPvUFF1/zxcPSdAnn4HhYZSIlVoLVEJ0LesfNlusgm2bPfnA==

Can someone look into this? Thanks!

[john-aws]

Unclear why the CDN owner is not detecting and blocking publication of these SRI hash failures. Don't you have any automated testing? People are working around this by modifying the hashes when they source the library from CDN, which is obviously not great.

[MattIPv4]

I'm re-opening this as it sounds like there are still a few hashes in the wild that don't match what is expected, and I agree that a way to detect these would be great (both at the time of publish as an audit step, and as an audit of existing published assets given we know there are bad hashes in the wild). This has been flagged to the Cloudflare team.

[jackielii]

htmx.org 1.9.11 also has a problem:

<script src="https://cdnjs.cloudflare.com/ajax/libs/htmx/1.9.11/htmx.min.js" integrity="sha512-nVsCpUASgFO/oi5aDUFe5DApV6KS9CIhEcAluIgTiDnWvi8H39f/0H0cxJvsTlGppLapsbIHRuaTXoHGpm35Yg==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

https://cdnjs.com/libraries/htmx

[xPaw]

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.7.1/jquery.min.js" integrity="sha512-qFOQ9YFAeGj1gDOuUD61g3D+tLDv3u1ECYWqT82WQoaWrOhAY+5mRMTTVsQdWutbA5FORCnkEPEgU0OF8IzGvA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

Failed to find a valid digest in the 'integrity' attribute for resource 'https://cdnjs.cloudflare.com/ajax/libs/jquery/3.7.1/jquery.min.js' with computed SHA-512 integrity 'v2CJ7UaYy4JwqLDIrZUI/4hqeoQieOmAZNXBeQyjo21dadnwR+8ZaIJVT8EE2iyI61OV8e6M8PP2/4hpQINQ/g=='. The resource has been blocked.

[dylanvdmerwe]

Failed to find a valid digest in the 'integrity' attribute for resource 'https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.30.1/moment.min.js' with computed SHA-512 integrity 'QoJS4DOhdmG8kbbHkxmB/rtPdN62cGWXAdAFWWJPvUFF1/zxcPSdAnn4HhYZSIlVoLVEJ0LesfNlusgm2bPfnA=='. The resource has been blocked.