use github api to find github releases

[tardyp]

previous version with svn uses blocking synchronous api and do not work well with proxies

the tag api do not contain the commit infos and thus we don't have the release date.
I think what we really want in vulnerable code is the list of actual releases, and not really the tags.

Fix #554

[pombredanne]

Thanks! I would rather not remove the SVN approach. Rather we can keep both?
Using svn on GH feels klunky BUT it has benefits: you can get tags with their dates without auth and rate limiting.
So I would rather not drop yet I get your use case too of using an authenticated.

[tardyp]

@pombredanne thanks for the feedback.

I restored the svn approach as a fallback.

note that the svn also triggers some error 500 for big repos

svn ls --xml https://github.com/torvalds/linux/tags
<?xml version="1.0" encoding="UTF-8"?>
<lists>
<list
   path="https://github.com/torvalds/linux/tags">
svn: E170013: Unable to connect to a repository at URL 'https://github.com/torvalds/linux/tags'
svn: E175002: Unexpected server error 500 'Internal Server Error' on '/torvalds/linux/tags'

and we have no garantee github will not rate limit it once vulnerablecode is deployed everywhere. ;-)

[tardyp]

even a moderately tagged project (50 tags) creates a 504 error after 2 minutes of github hard working:

svn ls --xml https://github.com/nexB/scancode-toolkit/tags
<?xml version="1.0" encoding="UTF-8"?>
<lists>
<list
   path="https://github.com/nexB/scancode-toolkit/tags">
svn: E175002: Unexpected HTTP status 504 'Gateway Timeout' on '/nexB/scancode-toolkit/tags'

[tardyp]

I did some more research to see if we can use the graphql api to get the tag's commit date. it is possible:

It takes less than a second to get the first 100 tags in linux repo

https://docs.github.com/en/graphql/overview/explorer

{
  repository(name: "linux", owner: "torvalds") {
    refs(refPrefix: "refs/tags/", first: 100) {
      totalCount
      pageInfo {
        endCursor
      }
      nodes {
        name
        target {
          oid
          """ in case it is a standard tag"""
          ... on Commit {
            committedDate
          }
          """ in case it is a signed tag"""
          ... on Tag {
            target {
              ... on Commit {
                committedDate
              }
            }
          }
        }
      }
    }
  }
}

[pombredanne]

Thanks for the research!
It makes sense to have the GH api first, attempting to have something authed first and second a fallback to unauthed and then fallback to SVN?

And yes, $ svn ls --xml https://github.com/torvalds/linux/tags is getting me repeated 500 errors so that's not a great solution, but on the other hand, not every repo is like Linux ;)

[tardyp]

like I said in second comment scancode repo with its 50 tags is also failing. Thats why I think it is quite a dead end.

[tardyp]

updated with Graphql version.

There is a 0 rate limit when not authenticated with graphql, means I directly hit a rate limit error without GH_TOKEN.
This is why I didn't implement that fallback.

[pombredanne]

Thank you ++
LGTM... merging shortly