feat(bitbucket): Bitbucket Server platform support #2774

iamstarkov · 2018-11-09T15:23:54Z

iamstarkov · 2018-11-09T15:24:26Z

addresses #2482

kayoub5 · 2018-11-10T15:03:06Z

Is it just me, or bitbucket API for cloud and server diverge too much, to the point where we should consider it a different platform?

rarkins · 2018-11-10T16:39:41Z

It does start to seem like they could be separate platforms. I think the current approach to the PR is good - just change everything necessary and if we decide it’s not worth “unifying” them again then we can split into two platforms with a little shared code.

iamstarkov · 2018-11-10T22:01:05Z

yeah, basic concepts are the same, but might have new names, also set of endpoints is different a little

iamstarkov · 2018-11-10T22:02:19Z

we thought we would start with existing adapter to see how big are the moving parts

rarkins · 2018-11-11T05:44:43Z

@iamstarkov what do you think about using git for the FS commands? @kayoub5 has built this functionality for GitHub and GitLab so far - where we support both API as well as git approaches to files - and he could also do the same with Bitbucket. The problem is that if he does it now it will require some refactoring that will then cause conflicts with this branch, so it's up to you whether you think it will save time or require extra time.

FYI here is the list of commands that can be supported via gitFs:

renovate/lib/platform/git/storage.js

Lines 18 to 30 in 9534e6a

    
           setBaseBranch, 
        
           branchExists, 
        
           commitFilesToBranch, 
        
           createBranch, 
        
           deleteBranch, 
        
           getAllRenovateBranches, 
        
           getBranchCommit, 
        
           getBranchLastCommitTime, 
        
           getCommitMessages, 
        
           getFile, 
        
           getFileList, 
        
           isBranchStale, 
        
           mergeBranch,

So basically you could remove all the API equivalents of these and not have to worry about porting them to Bitbucket Server.

iamstarkov · 2018-11-12T09:14:24Z

@rarkins @kayoub5 we are fine with cloud adapter rewrite. in the end of the day it means that only part of our adjustments to BB server will be needed

CLAassistant · 2018-11-12T09:54:49Z

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 6 committers have signed the CLA.

✅ JCB-K
✅ tommysdk
✅ JacobBlomgren
✅ rarkins
✅ iamstarkov
❌ Ricky Patel

Ricky Patel seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

iamstarkov · 2018-11-12T11:37:38Z

ahaha, bitbucket server allows to delete endpoint only through non-documented api endpoint.

Also if you want to reset the branch it will throw an exception because the file used to be in that branch already

iamstarkov · 2018-11-12T11:37:52Z

we are going to try gitFs

rarkins · 2018-11-12T12:16:26Z

we are going to try gitFs

Integrating it is a little tough, although of course you can follow the examples from GitHub and GitLab, which are already migrated.

However if you can wait a little then @kayoub5 hopefully has time to submit a PR migrating BitBucket Cloud and you could rebase based on that (with some conflicts unfortunately, but unavoidable).

JCB-K · 2018-11-12T17:07:49Z

@rarkins we setup gitFs for BB server now and it works a charm. This way we avoid migrating a lot of API calls to obscure BB Server equivalents :). I think we covered most of the happy path, tomorrow we'll focus on error handling + refactoring the server code into its own module.

JCB-K · 2018-11-12T17:09:47Z

@kayoub5 not sure if you started on migrating BB Cloud to gitFs, but we should be able to cover that in this PR too if you haven't started.

rarkins · 2018-11-12T17:16:55Z

Amazing work @JCB-K !

I’m pretty sure @kayoub5 hasn’t started yet, so any work in this PR would be great.

Does it work using the same token for both git as well as API? For GitLab we found it has a limitation of only supporting git write operations via SSH, which means you need both an SSH key as well as API token.

I think we could even make both Bitbucket Cloud and Server “gitFs-only” if it simplifies maintenance. I still classified BB support in Renovate as “beta” until now so have no problem deprecating the API-based functions if you think that’s a good idea.

JCB-K · 2018-11-12T18:14:48Z

Does it work using the same token for both git as well as API?

For now we do git operations on http://username:password@host. BB Server allows you to generate tokens which then can be used as passwords (so "single-use password" would be a more correct term), so I don't think we'll need SSH keys.

I still classified BB support in Renovate as “beta” until now so have no problem deprecating the API-based functions if you think that’s a good idea.

Aren't all these API methods internal anyway? Or are some of them exposed to users? Don't mind spending some time adopting the functions that aren't available in gitFs to the Server API if they're useful.

rarkins · 2018-11-12T18:25:28Z

Aren't all these API methods internal anyway?

Yes, you could reasonably argue that, which is even more reason why making the change is not a big deal.

Or are some of them exposed to users? Don't mind spending some time adopting the functions that aren't available in gitFs to the Server API if they're useful.

There should be no need to expose any. I have about 5% of the hosted GitHub App running on gitFs and 100% of the GitLab hosted app on gitFs so far. I think once we support gitFs on all platforms - includes VSTS/Azure DevOps - then I might drop non-git support in the next major release.

JCB-K · 2018-11-12T18:31:55Z

OK cool, we'll remove methods in lib/platform/bitbucket/index.js as we see fit then.

* Reordered imports * Removed unused variable * Removed unexpected console/logging statements * Temporarily suppressed async functions without await expressions (to be fixed properly when the underlying issue is addressed)

rarkins · 2019-02-05T06:25:37Z

FYI, I have now refactored the existing Bitbucket Cloud platform code to use gitFs-only, like this PR. So we can align the codebases and tests a lot easier, I hope.

rarkins · 2019-02-05T06:39:29Z

test/platform/bitbucket-server/index.spec.js

+    }
+  }
+  async function mockedGet(path) {
+    let body = responses[URL.parse(path).pathname] || { values: [] };


URL.parse() returns results in lower case while the fixtures file uses uppercase SOME in its response keys

rarkins · 2019-02-05T13:44:26Z

Test coverage is now up to 67% of this PR. Once we hit 100%, we can merge this to master. Can anyone help?

iamstarkov · 2019-02-06T00:05:34Z

@k2snowman69 @PixnBits we used api docs for v5.11.1 , URL is here https://docs.atlassian.com/bitbucket-server/rest/5.11.1/bitbucket-rest.html

because that is the BB version we have in prod. With that said, Feel free to adjust the code for latest version available.

iamstarkov · 2019-02-06T00:06:36Z

Also, Ricky Patel please sign CLA =) is it you, @k2snowman69?

k2snowman69 · 2019-02-06T00:10:57Z

Also, Ricky Patel please sign CLA =) is it you, @k2snowman69?

Not sure what's going on here but I believe I signed the CLA request as k2snowman69 but it doesn't seem to sync up with my name. But yes that is me :-)

rarkins · 2019-02-07T10:48:43Z

Hi all, I'm planning to merge this PR as-is so that it is easier for everyone to test out locally as well as submit PRs against the master branch whenever you find something needing improvement.

I've been testing it myself with a local Bitbucket Server instance running inside Docker and it onboards, creates and updates PRs, etc.

renovate-bot · 2019-02-07T19:09:44Z

🎉 This PR is included in version 14.12.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

iamstarkov · 2019-02-08T22:29:03Z

This is amazing news.

@rarkins thank you very much for this

WIP

c3e9e57

iamstarkov mentioned this pull request Nov 9, 2018

Support for Bitbucket Server #2482

Closed

rarkins changed the title ~~[WIP] bitbucket-server~~ feat(bitbucket): Support BitBucket Server (WIP) Nov 9, 2018

Merge branch 'master' into feat/bitbucket-server

783b689

rarkins added the in progress label Nov 11, 2018

create branch on server before committing files

a8591a6

Jacob Klapwijk and others added 5 commits November 12, 2018 14:06

wip: onboarding branch, setup gitFs for Bitbucket server

7370a35

wip isBranchStale

ef0fa54

adopt getAllRenovateBranches to BBserver

ae05cca

keep storage.getUrl backwards compatible

bcf6125

decline PR before deleting branch on BB server

b1f18ac

tommysdk added 2 commits November 12, 2018 20:16

Addressed various lint issues

90cd799

* Reordered imports * Removed unused variable * Removed unexpected console/logging statements * Temporarily suppressed async functions without await expressions (to be fixed properly when the underlying issue is addressed)

Code style: Obey line length

5df1f82

vulnerabilityalerts test

d861c44

rarkins reviewed Feb 5, 2019

View reviewed changes

rarkins added 3 commits February 5, 2019 09:32

get tests working

de9b20d

more tests

b33e77d

update tests

8ca7178

fix setBaseBranch

73771cf

Update storage.js

e8e811f

rarkins added 7 commits February 7, 2019 11:48

Merge branch 'master' into feat/bitbucket-server

bc79221

trim unused

688463e

prBody replace

f07825f

fix rebase

3ccb861

add readme

f46084b

fix getPr test

e04e92b

coverage

48bbedf

rarkins changed the title ~~feat(bitbucket): Support BitBucket Server (WIP)~~ feat(bitbucket): Bitbucket Server platform support Feb 7, 2019

rarkins added 2 commits February 7, 2019 17:57

reduce length to 30000

319128e

update self-hosting instructions

6f59992

rarkins merged commit 8a841a7 into renovatebot:master Feb 7, 2019

rarkins deleted the feat/bitbucket-server branch February 7, 2019 19:04

rarkins removed the in progress label Feb 7, 2019

renovate-bot added the released label Feb 7, 2019

github-actions bot locked as resolved and limited conversation to collaborators Dec 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(bitbucket): Bitbucket Server platform support #2774

feat(bitbucket): Bitbucket Server platform support #2774

iamstarkov commented Nov 9, 2018 •

edited by rarkins

iamstarkov commented Nov 9, 2018

kayoub5 commented Nov 10, 2018

rarkins commented Nov 10, 2018

iamstarkov commented Nov 10, 2018

iamstarkov commented Nov 10, 2018

rarkins commented Nov 11, 2018

iamstarkov commented Nov 12, 2018

CLAassistant commented Nov 12, 2018 •

edited

iamstarkov commented Nov 12, 2018

iamstarkov commented Nov 12, 2018

rarkins commented Nov 12, 2018

JCB-K commented Nov 12, 2018

JCB-K commented Nov 12, 2018

rarkins commented Nov 12, 2018

JCB-K commented Nov 12, 2018

rarkins commented Nov 12, 2018

JCB-K commented Nov 12, 2018

rarkins commented Feb 5, 2019

rarkins Feb 5, 2019

rarkins commented Feb 5, 2019

iamstarkov commented Feb 6, 2019

iamstarkov commented Feb 6, 2019

k2snowman69 commented Feb 6, 2019

rarkins commented Feb 7, 2019

renovate-bot commented Feb 7, 2019

iamstarkov commented Feb 8, 2019

feat(bitbucket): Bitbucket Server platform support #2774

feat(bitbucket): Bitbucket Server platform support #2774

Conversation

iamstarkov commented Nov 9, 2018 • edited by rarkins

iamstarkov commented Nov 9, 2018

kayoub5 commented Nov 10, 2018

rarkins commented Nov 10, 2018

iamstarkov commented Nov 10, 2018

iamstarkov commented Nov 10, 2018

rarkins commented Nov 11, 2018

iamstarkov commented Nov 12, 2018

CLAassistant commented Nov 12, 2018 • edited

iamstarkov commented Nov 12, 2018

iamstarkov commented Nov 12, 2018

rarkins commented Nov 12, 2018

JCB-K commented Nov 12, 2018

JCB-K commented Nov 12, 2018

rarkins commented Nov 12, 2018

JCB-K commented Nov 12, 2018

rarkins commented Nov 12, 2018

JCB-K commented Nov 12, 2018

rarkins commented Feb 5, 2019

rarkins Feb 5, 2019

Choose a reason for hiding this comment

rarkins commented Feb 5, 2019

iamstarkov commented Feb 6, 2019

iamstarkov commented Feb 6, 2019

k2snowman69 commented Feb 6, 2019

rarkins commented Feb 7, 2019

renovate-bot commented Feb 7, 2019

iamstarkov commented Feb 8, 2019

iamstarkov commented Nov 9, 2018 •

edited by rarkins

CLAassistant commented Nov 12, 2018 •

edited