GithubRelease assets are not found via autobump #895

holtjma · 2023-05-25T15:59:54Z

Hello,

I've having issues getting autobump to correctly identify a version change for this recipe: https://github.com/bioconda/bioconda-recipes/blob/master/recipes/hiphase/meta.yaml. It appears to be an issue between the GithubRelease hoster and what GitHub returns via HTML now. Couple notes:

I verified that autobump is correctly trying to use GithubRelease
I verified that it's fetching the correct URL in this instance
When I checked the downloaded HTML, I noticed that the links required (of this form https://github.com/PacificBiosciences/HiPhase/releases/download/v{{ version }}/hiphase-v{{ version }}-x86_64-unknown-linux-gnu.tar.gz) don't actually show up in the HTML.
When I reviewed it with web, it appears to be lazily loaded via a second URL of this pattern: https://github.com/PacificBiosciences/HiPhase/releases/expanded_assets/v0.10.0
That pattern does show up in the autobump download

Below are some logs with things I added while trying to debug:

$ bioconda-utils autobump recipes/ config.yml --packages hiphase --loglevel debug
...
08:09:03 BIOCONDA DEBUG Matching url 'https://github.com/PacificBiosciences/HiPhase/releases/download/v0.8.1/hiphase-v0.8.1-x86_64-unknown-linux-gnu.tar.gz'                                   
08:09:03 BIOCONDA DEBUG GithubRelease matched https://github.com/PacificBiosciences/HiPhase/releases/download/v0.8.1/hiphase-v0.8.1-x86_64-unknown-linux-gnu.tar.gz with {'account': 'PacificBiosciences', 'project': 'HiPhase', 'prefix': '', 'version': '0.8.1', 'fname': 'hiphase-v0.8.1-x86_64-unknown-linux-gnu.tar.gz', 'ext': ''}
08:09:03 BIOCONDA DEBUG Scanning with GithubRelease                                                                                                                                            
08:09:03 BIOCONDA DEBUG before: <bioconda_utils.aiopipe.AsyncRequests object at 0x7f9b61f85eb0>, 0.8.1                                                                                         
08:09:03 BIOCONDA DEBUG OHH start: <bioconda_utils.aiopipe.AsyncRequests object at 0x7f9b61f85eb0>, 0.8.1                                                                                      
08:09:03 BIOCONDA DEBUG HH exclude: {'version', 'fname'}                                                                                                                                       
08:09:03 BIOCONDA DEBUG HH vals: {'account': 'PacificBiosciences', 'project': 'HiPhase', 'prefix': '', 'ext': ''}                                                                              
08:09:03 BIOCONDA DEBUG HH link_pattern: /PacificBiosciences/HiPhase/releases/download/??(?:(?<=[/._-])[rv])?(?P<version>\d[\da-zA-Z\-+\.:\~_]{0,30})/(?P<fname>[^/]+)?                        
08:09:03 BIOCONDA DEBUG HH link_re: regex.Regex('/PacificBiosciences/HiPhase/releases/download/??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})/(?P<fname>[^/]+)?', flags=regex.V0)
08:09:03 BIOCONDA DEBUG HH url: https://github.com/PacificBiosciences/HiPhase/releases

So my questions are basically how do we fix this? Unless I'm mistaken, this should be broken for every recipe that is pulling from the assets. I tried using the GitHub API via https://api.github.com/repos/PacificBiosciences/HiPhase/releases. However, I think the parser is expecting HTML so it fails to parse out the URLs.

Hopefully, I'm not missing something obvious here. Any suggestion?

The text was updated successfully, but these errors were encountered:

This is a first pass attempt at fixing #895. Main changes are updating the GithubRelease hoster to look for alternate paths via the GitHub API. The API sends a JSON response (example: https://api.github.com/repos/PacificBiosciences/HiPhase/releases) that contains each release along with a list of assets. The assets are searched for files that match the provides pattern. Currently, this first pass puts this as a backup to the main one (which I think is always failing now). I also did not update the tests, I think someone may have a better idea of how to do that. Happy to help push this further, but wanted to get the ball rolling. --------- Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>

@dpryan79

This is basically a patch to fix a new issue (described here: bioconda/bioconda-recipes#42176) that was introduced by myself here: #896. The primary cause is that the GitHub API is rate limited (which I was not aware of), so autobump basically blows through the limits quickly and then subsequent calls automatically fail. Of course, this means that the tools we were hoping to see get autobumped are not. This patch basically removes the original patch (which uses a single GitHub API call) and replaces it with an approach that basically "expands" the release webpage to find the assets. See #895 for more details on how the URLs are masked and the expansion pattern. Couple other notes: * I added an `IncludeFragmentParser` that is basically identical to the `HrefParser` except it is looking for tags with this pattern: `<include-fragment src="http://url/of/interest" ...>` * The general expansion logic is: * Pull the full release page * Search for `<include-fragment ...>` tags * Iteratively pull these looking for the main link URL of interest * Break the above iteration early if the current version is identified * This should no longer use the GitHub API, I'm unaware of other rate limiters for these URLs (🤞that none are hidden...) Pinging @dpryan79 and @johanneskoester as you both reviewed the last related PR. Sorry that the previous PR injected a rate limit issue! EDIT: I forgot, I have tested these on both `hiphase` and `pbfusion` recipes, the diff of the results are here and match our current expectations: ``` (bioconda)$ git diff diff --git a/recipes/hiphase/meta.yaml b/recipes/hiphase/meta.yaml index 38b939bf9f..b5d2503597 100644 --- a/recipes/hiphase/meta.yaml +++ b/recipes/hiphase/meta.yaml @@ -1,6 +1,6 @@ {% set name = "hiphase" %} -{% set version = "0.10.0" %} -{% set hiphase_sha256 = "e22d3530a395bd218bd4634537f5b3c31a82cd8155849291c69d81c48f0c58fe" %} +{% set version = "0.10.2" %} +{% set hiphase_sha256 = "2a20fc437ab5c0f2437246e8a36e7590c68f5ae5caa0cecb8e85a371f5357c6c" %} package: name: {{ name }} diff --git a/recipes/pbfusion/meta.yaml b/recipes/pbfusion/meta.yaml index 781ada3413..ca69e87d0f 100644 --- a/recipes/pbfusion/meta.yaml +++ b/recipes/pbfusion/meta.yaml @@ -1,6 +1,6 @@ {% set name = "pbfusion" %} -{% set version = "0.2.2" %} -{% set pbfusion_sha256 = "3390adbeb8550265fcf4fad89dad80740cc368e96d10f3b23bb19ef001ffd081" %} +{% set version = "0.3.0" %} +{% set pbfusion_sha256 = "ef9767af7e82bd1e664711e7521d7a8370a41f101cccf1db4a58895d9f88934c" %} package: name: {{ name }} ```

aliciaaevans · 2023-09-05T21:40:48Z

A variation on this issue came up from @AndreaGuarracino in Gitter with https://github.com/pangenome/smoothxg/releases. The previous release descriptions had a link to the tarball, so that's getting found and the Assets section of the most recent release is never checked.

Would it make sense to to return all versions found with either HTML method? Or give the Assets section priority over the static HTML.

holtjma · 2023-09-05T22:59:58Z

This isn’t fully functional yet. We’re still waiting on #912 to make it into prod. I don’t know if that will resolve the issue, but it’s probably worth waiting to see if it does.

holtjma · 2023-10-09T02:39:29Z

As of #920, the original issue is fixed.

holtjma mentioned this issue May 25, 2023

fix: adds alternate lookup if base lookup fails #896

Merged

holtjma mentioned this issue Jul 28, 2023

fix: swaps from github API to expanding the assets #907

Merged

holtjma closed this as completed Oct 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GithubRelease assets are not found via autobump #895

GithubRelease assets are not found via autobump #895

holtjma commented May 25, 2023

aliciaaevans commented Sep 5, 2023

holtjma commented Sep 5, 2023 •

edited

Loading

holtjma commented Oct 9, 2023 •

edited

Loading

GithubRelease assets are not found via autobump #895

GithubRelease assets are not found via autobump #895

Comments

holtjma commented May 25, 2023

aliciaaevans commented Sep 5, 2023

holtjma commented Sep 5, 2023 • edited Loading

holtjma commented Oct 9, 2023 • edited Loading

holtjma commented Sep 5, 2023 •

edited

Loading

holtjma commented Oct 9, 2023 •

edited

Loading