Error when parsing Git URL on Bitbucket Server

[AntoineFr]

Summary

Some commands like jx import or jx create devpod need to clone a Git repo in order to work.
If you use Bitbucket Server, this repo URL isn't parsed correctly in this step.

In the Bitbucket web interface, the clone URL is something like https://git.***.fr/scm/<PROJECT>/<REPO>.git.
These commands are failing because they don't have the right clone URL (it is missing the /scm part) : fatal: repository 'https://git.***.fr/<PROJECT>/<REPO>.git/' not found.

I think it is related to this line in the code :

jx/pkg/gits/git_url.go

Lines 203 to 204 in 1976d34

	// This is necessary for Bitbucket Server in some cases.
	trimPath := strings.TrimPrefix(path, "/scm")

Steps to reproduce the behavior

I tested these cases : install Jenkins X with Bitbucket Server and either :

• run jx create devpod
• run jx import with webhook: lighthouse

Expected behavior

The repository is correctly cloned.

Actual behavior

It fails with an error like

Cloning into '<REPO>'...
fatal: repository 'https://git.***.fr/<PROJECT>/<REPO>.git/' not found
command terminated with exit code 128

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.1202
Kubernetes cluster v1.17.2
kubectl            v1.16.0
helm client        Client: v2.16.3+g1ee0254
git                2.17.1
Operating System   Ubuntu 18.04.4 LTS

Jenkins type

• Serverless Jenkins X Pipelines (Tekton + Prow)
• Classic Jenkins

Kubernetes cluster

On premise

Operating system / Environment

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic

[abayer]

So I believe that the problem is not that we're trimming /scm in parsePath, it's that we're not putting it back when getting the clone URL from the SourceRepository (and possibly other places).

[salvag-ntt]

Still reproducible on jx version 2.0.1240. Interestingly enough I tried jx import of an existing
maven project and it failed but it worked with a react project one of my colleagues imported/created.

[AntoineFr]

Hello, I retried today with the jx version 2.0.1258 and the command jx create quickstart with the php-helloworld template.

It is working with Traditional Jenkins (webhook: jenkins in jx-requirements) but it fails if I use LightHouse (webhook: lighthouse in jx-requirements).

If it could help, here is an error that is shown in the command jx get build logs <PROJECT>/<REPO>/master :

{"level":"error","ts":1585643585.0040672,"logger":"fallback-logger","caller":"git/git.go:40","msg":"Error running git [fetch --depth=1 --recurse-submodules=yes origin master]: exit status 128\nfatal: repository 'https://git.***.fr/<PROJECT>/<REPO>.git/' not found\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/Users/abayer/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:40\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/Users/abayer/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:91\nmain.main\n\t/Users/abayer/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:39\nruntime.main\n\t/usr/local/Cellar/go/1.12.4/libexec/src/runtime/proc.go:200"}

[salvag-ntt]

Just tried this with 2.1.31 and it's still failing. Getting the sourcerepository resource, cloneUrl is still missing scm. Tried this with jx create quickstart and jx import a local folder

[AntoineFr]

I found some functions in the code that could be related to this issue.

jx/pkg/gits/git_url.go

Lines 28 to 31 in 22ce213

	// HttpCloneURL returns the HTTPS git URL this repository
	func (i *GitRepository) HttpCloneURL() string {
	return i.HttpsURL() + ".git"
	}

calls this function

jx/pkg/gits/git_url.go

Lines 42 to 49 in 22ce213

	// HttpsURL returns the URL to browse this repository in a web browser
	func (i *GitRepository) HttpsURL() string {
	host := i.Host
	if !strings.Contains(host, ":/") {
	host = "https://" + host
	}
	return util.UrlJoin(host, i.Organisation, i.Name)
	}

but it doesn't add the /scm between the host and the organisation.

[salvag-ntt]

Nicely spotted @AntoineFr, meanwhile I am using a bit of kubectl and scripting to workaround it

kubectl get sourcerepositories.jenkins.io ${PROJECT_NAME}-${REPO_NAME} -o yaml > ${PROJECT_NAME}-${REPO_NAME}.yaml
INPUT_STRING="httpCloneURL: ${BITBUCKET_SERVER}/${PROJECT_NAME}/"
FIX_STRING="httpCloneURL: ${BITBUCKET_SERVER}/scm/${PROJECT_NAME}/"
sed -i "s|${INPUT_STRING}|${FIX_STRING}|g" ${PROJECT_NAME}-${REPO_NAME}.yaml
kubectl apply -f ${PROJECT_NAME}-${REPO_NAME}.yaml

Edit: although I have to say that the first meta-pipeline will fail anyway because it will start before I can possibly update that resource.

In fact it fails while upserting the Pipeline resource and a bit of digging brought me here:

jx/pkg/tekton/pipelines.go

Lines 433 to 438 in a0d4c5a

	if resource.Spec.Type == pipelineapi.PipelineResourceTypeGit {
	gitURL := activityKey.GitInfo.HttpCloneURL()
	log.Logger().Infof("upserted PipelineResource %s for the git repository %s", info(resource.Name), info(gitURL))
	} else {
	log.Logger().Infof("upserted PipelineResource %s", info(resource.Name))
	}

[abayer]

I'm hoping #7173 deals with this, but I'm going to test it against BitBucket Server tomorrow. The challenge is that we can generate the GitRepository in a few different ways, so HttpCloneURL() doesn't necessarily know what the underlying server is.