Fix crawler due to bintray changes

[lightpriest]

I guess bintray changed their URL prefix to be with % instead of #.
Everything else looks the same.

[jenkinsadmin]

Thank you for a pull request! Please check this document for how the Jenkins project handles pull requests

[jdamick]

agreed, please merge. cc/ @ndeloof

[jdamick]

other option is to change like the groovy crawler:

def url = baseUrl + e.getHrefAttribute()[1..-1]

Either way is not very pretty and fragile..

[aheritier]

and now it is broken because it is a : character :(
I don't know what is doing bintray cc @freddy33

[jbaruch]

We're looking into it.

[freddy33]

% is a big issue in URL. The : is better.
Anyway, crawling is not the way to go.

[aheritier]

Hi @jbaruch @freddy33, Nihal from your support team explained me that the character was changing regularly to avoid crawling (which I can understand) and thus we should use bintray APIs instead.
We need to rework our scripts which are creating catalogs from bintray. For now, I found 3 of them:

• https://github.com/jenkinsci/backend-crawler/blob/master/groovy.groovy
• https://github.com/jenkinsci/backend-crawler/blob/master/packer.groovy
• https://github.com/jenkinsci/backend-crawler/blob/master/sbt.groovy

SBT script is already using api.bintray.com. I will see to propose a fix to unify them and be sure they are using API instead of doing crawling

[aheritier]

@freddy33 @jbaruch I created the issue https://issues.jenkins-ci.org/browse/JENKINS-30708 to try to solve the crawling issue but the fact that the URL changes regularly will continue to break our downloads at least temporarily because of our Jenkins current architecture to manage tools installers. I will see with others teams member if we can improve/fix this.

[ydubreuil]

I'm trying to fix this issue but Bintray API seems to require an API token to list available files. Listing version with https://api.bintray.com/v1/packages/mitchellh/packer/packer is possible without authentication but getting files list is not: https://api.bintray.com/v1/packages/mitchellh/packer/packer/files

For SBT, I was able to workaround this by crafting the good URL from version list, but for packer it's not possible.

@freddy33 @jbaruch Do we really need to have an API token to list files? Do you have a workaround?

[lightpriest]

As a quick workaround, maybe we can use the text inside the anchor tag - it seems to match and be consistent between changes they make to the href attribute.