New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ingest https://github.com/psf/advisory-database/tree/main/advisories #1552
Comments
@oliverchang Would be happy to help out implementing this. I saw in the contributing guide it mentioned creating an issue but not much else, is there anything I can help with? |
Thanks for reaching out! https://github.com/psf/advisory-database/tree/main/advisories appears to all be keyed on commits, so there's not much to do here other than for us to set up the infra config to start importing. We'll get to it this week :) |
https://oss-vdb-test.wl.r.appspot.com/vulnerability/PSF-2023-9 got on our radar because it's invalid to have an OSV.dev will attempt to enumerate |
+1 see the example linked: These are dervied from git tags that match the provided git ranges. |
Got it, one hitch to that is it seems that historical git branches get deleted from the GitHub repository after they're no longer supported, so advisories affecting versions before 3.8.0 might not get populated? |
Also looks like the advisory https://oss-vdb-test.wl.r.appspot.com/vulnerability/PSF-2023-8 isn't rendering correctly, I'm assuming that I'm encoding the newlines incorrectly. |
Hmm, we managed to extract older versions (including 2.x) for https://oss-vdb-test.wl.r.appspot.com/vulnerability/PSF-2023-8. Perhaps that's a non-issue?
https://github.com/psf/advisory-database/blob/4d5ebbc62f10b00e10f10e165587aedfd85134da/advisories/python/PSF-2023-8.json#L10 looks to have double quoted newlines. |
That is missing a lot of tags for v2.x so it's not a complete import, the 2.7.x series went until v2.7.18. |
Hmm, that's rather unfortunate. Without historical branches the analysis is purely on the In any case, having |
More to the point it currently doesn't break our import and we're winding up with vulnerabilities that don't meet certain assumptions and that causes all sorts of issues :-( |
@sethmlarson any chance we can remove the Assuming that the branch heads to all the historical released versions of cpython still exist (e.g. via tags), there's likely a technical solution that will enable us to enumerate every version correctly (i.e. correctly enumerating all the way up to v2.7.18 per #1552 (comment)). The challenge for us is doing this efficiently, but I've filed #1590 to track this. |
I can remove the ecosystem version ranges, it would be good to somehow capture the correct affected versions though. Would listing out the tags in "affected[].versions" work for that purpose? |
I've updated the records to remove the |
https://ossf.github.io/osv-schema/#affectedversions-field OSV.dev, as part of the importing process will do its best to enumerate the versions between the |
This would work. Our own tag enumeration would only add new tags that we match that don't already exist to the original record. We don't ever delete things already in |
@oliverchang I talked with Release Manager Łukasz and he mentioned that while branches are deleted all of the commits are kept in the repo, so in theory by cloning the entire repository all of the tags should be discoverable. Maybe this is causing an issue in the discovery process? |
Thanks for checking! Yep, this is what we will be trying as part of #1590 |
These are now ingested in our prod instance. e.g. https://osv.dev/vulnerability/PSF-2019-7 #1590 tracks the remaining git enumeration improvement |
Awesome, thanks @oliverchang and @andrewpollock! 💪 |
No description provided.
The text was updated successfully, but these errors were encountered: