Resource does not work with Bitbucket >7.0

[holgerstolzenberg]

Today our pipelines broke without any change to our PR builds. It turns out that our on-prem Bitbucket Server was updated to 7.2.x. It looks like this resource is not compatible with Bitbucket 7.2+.

The following error occurs during the GET step:

Cloning into '/tmp/build/get'...
fatal: Couldn't find remote ref refs/pull-requests/24/merge

Screenshot:

Googling around for the error, the assumption arises that this might also effect Bitbucket Server 7.0/7.1, but I am not sure about this.

This is a nasty one, as this affects a lot of pipelines…

[holgerstolzenberg]

The problem might be related with the following line in the IN script:

concourse-bitbucket-pullrequest-resource/assets/in

Line 68 in d283701

branch="pull-requests/${prq_id}/merge"

Listing all refs in the given repository via git ls-remote only shows a ref:

c10913bff4e91a511685a29d28e98ce06e44eeaf	refs/pull-requests/24/from

Please note that its suffixed with /from whereas the resource is searching for /merge.

The commit SHA of the ref matched the given SHA found by the check script:

[holgerstolzenberg]

I did some further investigation and it looks like the whole world was beaten by this one. The relating information can be found here:

• https://community.atlassian.com/t5/Bitbucket-questions/Difference-of-refs-pull-requests-lt-ID-gt-merge-and-refs-pull/qaq-p/772142
• https://jira.atlassian.com/browse/BSERV-12284
• Pull-Requests no longer working on BitBucket Server 7.0 jenkinsci/bitbucket-branch-source-plugin#287

It turns out that the internal Git refs everyone is relying on is only internal API of Bitbucket which has been removed in Bitbucket 7.0. As stated by Atlassian, this is not going to come back and the REST API should be used for that matter.

I tried to work around by using the following Bitbucket plugin for eagerly updating the refs: https://github.com/Cyanoth/Bitbucket-EagerPR-Updates
Unfortunately this updates the /from ref, so this is no help with this resource:
Cyanoth/Bitbucket-EagerPR-Updates#2

So it looks like the only chance we have is to modify the in script to use the API.
I looked into the source, but my Git knowledge is not that deep, so I do not fully understand what is going on in lines 68-120.

In the end, what is used is the source_commit and target_commit values. There is an old pending PR (#11 ) in this repo for using the REST API for the check script that already creates the meta data for these commits in the check step. The PR needs some refresh but might already provide what is needed to fix this one.

If there are any other solutions or workarounds to this one, please share 'em.

[sjoerdschouten]

Any updates on this one? I'm trying to setup a pipeline in combination with this resource but I'm getting blocked by this issue.

[holgerstolzenberg]

We found another fork of this resource that seems to be maintained lately. It still has the same problems. We did a initial PR on that that fixes some of the problems and is working for us, but is not final:

• https://github.com/emerald-squad/concourse-bitbucket-pullrequest-resource
• https://github.com/ewerk/concourse-bitbucket-pullrequest-resource

We also had to install the plugin because in the out script refs/pull-request/*/from is still used and does not get created by Bitbucket until you look at the PR diff for the first time.

I´ll try to fix that also in the resource, but cannot give any timeline by now.

[sjoerdschouten]

Hi @holgerstolzenberg, thanks for replying. This is the first time I'm checking again. Did you manage to fix the resource in the fork by any chance?

[holgerstolzenberg]

No not finally. At emeraldsquad, there is a pending code review for our PR that is missing a final comment.
It stillt might be useful to merge all the stuff to this repo as your resource has the highest download rates at Dockerhub.
Maybe we can get some pace on here?

[mmb]

We have recently run into similar issues with the pullrequest refs. There are race conditions where sometimes the ref isn't there by the time Concourse checks. We also have issues with bitbucket mirrors not having all the refs. Bitbucket admins and Atlassian claim this is an internal API and it sounds like they've finally removed it.

I also started working on some changes to use the API instead of ls-remote with pullrequest refs in this branch:

https://github.com/mmb/concourse-bitbucket-pullrequest-resource/tree/list-prs-api

The main issue I'm seeing is a lot of this resource is based on the concept of a "merge" ref and that doesn't seem to exist in the API. From the API you can only get a "from" commit and a "to" commit for a PR.

We set "rebuild_when_target_changed" to false so we don't need the merge refs but other use cases might. I'll work on it but I wanted to get some community feedback about making this resource use only what is available in the REST API, which could mean removal of some features.

[sjoerdschouten]

This might be of interest for you if you want to continue this effort:

https://www.atlassian.com/blog/announcements/journey-to-cloud

i.e. Atlassian pulls the plug on server licences, drags customers to the cloud

[vijayantprakash]

@mmb Is this fixed or is there a workaround for this ?

[vijayantprakash]

@holgerstolzenberg @sjoerdschouten how did you all fix this issue ?