Gitlab credential issues. Git clone container throws error.

[EliasGabrielsson]

This a summary of discussions made in Gitter to keep track of progress.
Leave this issue to verified information. Keep the discussion to Gitter

Error:
Within the git plugin container following error is throwed while cloning a repo from gitlab. This error have begun to appear with drone integration with recent versions of GitLab.

$ git init
Initialized empty Git repository in /drone/src/git.company.se/company/placeholder-prototype/.git/
$ git remote add origin https://git.company.se/company/sleepy-prototype.git
$ git fetch --depth=50 origin +refs/heads/master:
fatal: could not read Username for 'https://git.company.se': No such device or address

Versions:
The error have been reproduced using: gitlab-8.0.4-0, gitlab-8.0.4-1
Anton Ikonovalov cant reproduce this error on version gitlab-8.0.3-1. Something have probably happen in later version which breaks Drone from working.

Older versions of GitLab CE could be found here: https://packages.gitlab.com/gitlab/gitlab-ce

Next step:

• Try clone_mode=oauth in your drone gitlab configuration string
• Try to manually create netrc locally and test it with GitLab.

[AKoetsier]

@bobbarebygg any news on this? I'm having the same issue.

[bradrydzewski]

@AKoetsier are you using clone_mode=oauth configuration setting? This will clone the repository with the oauth token instead of the repository token.

I'm probably going to alter GitLab to default to clone_mode=oauth since, this is how GitHub and Bitbucket work by default as well. I'm also told there are issues with cloning using the repository token. I don't have the reference to the GitLab issue, but someone posted it in the Gitter room a few weeks back, and @bobbarebygg references above that something broke in GitLab between versions.

[bradrydzewski]

also make sure your GitLab instance is capable of cloning with a token and git+http(s). We use a netrc file to clone private repositories to inject the username and password:

machine: <mydomain.com>
login: oauth2
password: <my oauth token>

Also, if you are using a self-signed certificate, make sure you configure skip_verify in your yaml file. See http://addons.drone.io/git/ for details.

[AKoetsier]

@bradrydzewski I was not using the oauth-setting but the default (token). I did not set anything in gitlab (apart from the defaults) to be able to clone using a token so I will take another look into that. Tnx for the quick response!

[EliasGabrielsson]

No clear news, no.

I tried a fresh installation (8.0.4.) in hope of the error occured when upgraded from 7.x.x to 8.0.x.
Unfortunately I got the same error. I am aint a ruby developer so i don´t know where to start looking in the gitlab repo. I guess we need a precise log for gitlab and drone to nail down the error.

The oauth is working for me. The problem is to clone repos with git.

We are currently testing out GOGS to see if it can replace gitlab. It is really fast and feels solid, but it doesn't solve the problem.

[suquant]

Gitlab use

upstream gitlab {
  server unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fail_timeout=0;
}

upstream gitlab-workhorse {
  server unix:/var/opt/gitlab/gitlab-workhorse/socket;
}

and some uri paths routed to different upstream
use nginx configuration https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md

[davidak]

i have the same issue with gitlab 8.0.1.
also tried clone_mode=oauth and fail_timeout=0 in nginx vhost without success.

[kzaitsev]

@davidak Did you try skip_verify?

[davidak]

@Bugagazavr yes, it helps with the self-signed certificate.

[blandes]

Did anyone find a solution to this problem? I'm currently having the same issue and I have tried the clone_mode=oauth but not success. It looks like drone needs it's own login for Gitlab? Anything would be greatly appreciated.

[davidak]

@blandes what gitlab version do you have? i havn't found the time to update, that might help.

[blandes]

@davidak So I'm actually using Rancher to deploy my CI (which may be the problem) so the version is rancher/drone:0.4

[davidak]

@blandes what gitlab version?

[blandes]

@davidak drone/drone:0.4

[davidak]

@blandes gitlab not drone.

[blandes]

@davidak Sorry, I wasn't paying attention. gitlab:8.4.4

[davidak]

Okay, that is the newest version. Then updating gitlab also don't help. :/

Someone has to debug this.

[blandes]

Yeah, it's been stressing me out too much. Iv'e tried so many things such as deployment keys, tokens, oauth, and so on. I was thinking possibly a port issue because of the git+http but no dice. Thanks for the help.

[kzaitsev]

@blandes do you have self signed certificate?

[blandes]

@Bugagazavr No

[blandes]

So Gitlab is connected to Drone and is showing repos, builds, and so on. When a build is triggered or a commit is made Drone then spits out the original error at the top. I'm not sure where the .netrc is so I haven't tried that option yet.

I did however note that when I had a co-worker clone the repo from his computer that he had to login. I haven't set up his ssh key yet but the permission looked the same as above. Could I possibly make a user for drone?

[bradrydzewski]

Drone clones the repository using the git+http(s) url and automatically authenticates with a token, or if you use clone_mode=oauth with an oauth token. I personally prefer clone_mode=oauth. Drone clones the repository the same way for every remote system (GitHub, Bitbucket, GitLab and Gogs) and we only ever seem to have this issue with GitLab. Worse, some GitLab installations have this issue and others don't. I'm honestly not sure how to proceed with troubleshooting

[kzaitsev]

@blandes .netrc placed in user home directory

@blandes @davidak if you can help me:

1. Do you use gitlab on relative path /gitlab/ or etc?
2. Do you use certificate
   1. If certificate not self signed, i would to know what a root certificate used in your cert
3. Can you reproduce this error? Launch drone tests, with command cat ~/.netrc, and create this file locally on your own machine with the same content, and try to clone your repository locally via https.

[davidak]

@Bugagazavr 1. No 2. self signed 3. how should the command get executed if the repo is not cloned?

[kzaitsev]

@davidak sorry, you can create netrc file manually

machine your.gitlab.com
        login drone-ci-token
        password your_drone_repo_token

your_drone_repo_token you can get from gitlab in drone service settings

http://www.mavetju.org/unix/netrc.php

[bradrydzewski]

@davidak if you are using a self-signed certificate you might need to configure your yaml to clone the repository and ignore tls verification:

clone:
  skip_verify: true

this executes git config --global http.sslVerify false prior to cloning the repository

[kzaitsev]

@bradrydzewski I think this is only certificates problem:

1. Self-signed cerificates
2. Unknow authority.

@blandes can you try add this to your .drone.yml ?

clone:
  skip_verify: true

[davidak]

@bradrydzewski i have that already.

@Bugagazavr it asks for a username like without the .netrc, i have set the right permission. tested on ubuntu 14.04

[blandes]

@bradrydzewski I have it on my remote config, does this do the same thing?

[blandes]

@davidak When I turn down the settings from Private to Public, it will actually try to pull the repo but does a hard reset?

[davidak]

I set the repo to public and get exactly the same error.

As the message "fatal: could not read Username" implies, git tries to read the username (and password) from keyboard input, the .netrc is ignored.

[bradrydzewski]

I have it on my remote config, does this do the same thing?

This is not the same thing. This value in the remote config defines how Drone server interacts with GitLab. The yaml configuration defines how the Git plugin should clone the repository.

[bradrydzewski]

I was able to verify that I can clone a repository by configuring the appropriate drone integration for my private gitlab repository (via gitlab repository settings), and then manually invoking the git clone plugin. I don't have a drone+gitlab installation, so I chose to perform this manually:

docker run -i plugins/drone-git <<EOF
{
    "repo": {
        "clone_url": "https://gitlab.com/bradrydzewski/test.git"
    },
    "build": {
        "event": "push",
        "branch": "master",
        "commit": "636b4f33dbe819f65a06036cfad619afa2812503",
        "ref": "refs/heads/master"
    },
    "workspace": {
        "root": "/drone/src",
        "path": "/drone/src/gitlab.com/bradrydzewski/test",
        "netrc": {
            "machine": "gitlab.com",
            "user": "XXXXXXXXXXXXX",
            "login": "drone-ci-token"
        }
    }
}
EOF

This results in the following successful output:

$ git init
Initialized empty Git repository in /drone/src/gitlab.com/bradrydzewski/test/.git/
$ git remote add origin https://gitlab.com/bradrydzewski/test.git
$ git fetch --no-tags --depth=50 origin +refs/heads/master:
From https://gitlab.com/bradrydzewski/test
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
$ git reset --hard -q 636b4f33dbe819f65a06036cfad619afa2812503

While it is true I didn't test this wing-to-wing with a full drone+gitlab installation, my understanding is there are individuals successfully using drone+gitlab. This leads me to believe there is no known issues with cloning gitlab repositoires with a netrc.

@Bugagazavr can you confirm? Are you still using Drone with GitLab?

** note that the netrc user field is actually the password. long story, but this is the result of a typo that, for backward compatibility reasons, has not been updated yet.

[kzaitsev]

@bradrydzewski yep it working for me, looks like this depends on certificate.

[davidak]

@bradrydzewski with your code i get:

fatal: unable to access 'https://...': SSL certificate problem: self signed certificate

i tried to put "skip_verify": "true" to repo but get the same result. where to put it?

[bradrydzewski]

plugin-specific configuration goes in the vargs section of the payload like this:

docker run -i plugins/drone-git <<EOF
{
    "repo": {
        "clone_url": "https://gitlab.com/bradrydzewski/test.git"
    },
    "build": {
        "event": "push",
        "branch": "master",
        "commit": "636b4f33dbe819f65a06036cfad619afa2812503",
        "ref": "refs/heads/master"
    },
    "workspace": {
        "root": "/drone/src",
        "path": "/drone/src/gitlab.com/bradrydzewski/test",
        "netrc": {
            "machine": "gitlab.com",
            "user": "XXXXXXXXXXXXX",
            "login": "drone-ci-token"
        }
-   }
+   },
+    "vargs": {
+        "skip_verify": true
+    }
}
EOF

[davidak]

thanks.

with that the certificate issue is solved, but i get that issue of this ticket:

fatal: could not read Username for 'https://...': No such device or address

[bradrydzewski]

@davidak can you clone with git clone https://drone-ci-token:{token}@{hostname}/{owner}/{repo} for your repository? These are the questions we should be able to answer:

• can you git clone inside a docker container with git+https and your username and password
• can you git clone inside a docker container with git+https using the drone token (as seen above)
• can you git clone inside a docker container with git+https using a .netrc file
• can you git clone by manually invoking the plugins/drone-git container (as seen above)

This will help us isolate the problem. I can complete all four of these tasks using gitlab.com. Note that the first three don't even involve drone. So if the first three tasks don't work, perhaps there is something wrong with the gitlab install, given these commands all work with gitlab.com

[davidak]

this is strange:

root@9c956968ca92:/# git config --global http.sslVerify false
root@9c956968ca92:/# git clone https://gitlab.company.de/dkl/tekemio.git
Cloning into 'tekemio'...
Username for 'https://gitlab.company.de': dkl
Password for 'https://dkl@gitlab.company.de':
warning: You appear to have cloned an empty repository.
Checking connectivity... done.
root@9c956968ca92:/# cd tekemio/
root@9c956968ca92:/tekemio# git status
On branch master

Initial commit

nothing to commit (create/copy files and use "git add" to track)
root@9c956968ca92:/tekemio# git pull
Username for 'https://gitlab.company.de': dkl
Password for 'https://dkl@gitlab.company.de':
Your configuration specifies to merge with the ref 'master'
from the remote, but no such ref was fetched.

could be this issue https://gitlab.com/gitlab-org/gitlab-ce/issues/2669
https://gitlab.com/gitlab-org/gitlab-ce/issues/2727
i will look into this later.

[blandes]

@bradrydzewski I did add the following lines to my .drone.yml but nothing still. Sorry for the long delay.

clone:
skip_verify: true

[bradrydzewski]

@blandes @davidak my recommendation is to test this against gitlab.com ... if you can demonstrate the same error with gitlab.com then we know it is a drone error. If you cannot demonstrate with gitlab.com, then we know it is a configuration error, at some level, outside of drone.

The fact that I was able to clone from gitlab.com with a netrc and the drone git plugin, and that @Bugagazavr is using drone with omnibus gitlab, leads me to believe this is a configuration error outside of drone. If that is the case, our ability to help troubleshoot will be very limited.

[venoma333]

I'm getting the same errors as above on my self-hosted GitLab instance.
Testing on gitlab.com and pushes don't even trigger builds.
I believe Drone is registered and configured properly with GitLab.

Screens:

[kzaitsev]

@venoma333 thanks for report

Can you provide some drone logs?
did you add a docker host to your config?

[venoma333]

I believe these lines are relevant from the logs:

[GIN] 2016/03/28 - 09:03:29 | 200 |  402.389427ms | 217.145.92.194 |   GET     /api/stream/venoma333/test-drone
time="2016-03-28T09:04:15Z" level=error msg="failure to calculate matrix for venoma333/test-drone. yaml: line 1: found character that cannot start any token" 
[GIN] 2016/03/28 - 09:04:15 | 400 |  4.823014838s | 104.210.3.246 |   POST    /api/hook
Error #01: yaml: line 1: found character that cannot start any token
http: multiple response.WriteHeader calls
time="2016-03-28T09:04:26Z" level=info msg="closed event stream" 

This is my .drone.yml:

build:
  image: node:5.5.0
  commands:
    - echo "done"

The drone server is behind a reverse nginx proxy and this is the nginx config:

server {
        listen 80;
        server_name teeyankov.co;

        location / {
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header Host $http_host;
                proxy_set_header Origin "";

                proxy_pass http://teeyankov.co:8000;
                proxy_redirect off;
                proxy_http_version 1.1;
                proxy_buffering off;

                chunked_transfer_encoding off;
        }
}

[kzaitsev]

@venoma333 this is not a gitlab bug, your yaml is invalid.

try to replace echo "done" to echo 'done'

[venoma333]

@Bugagazavr I'm really sorry, Not familiar with yaml yet.

The private project was pulled, tested and the build was a success. I guess it really is a configuration issue.

[ColtonProvias]

Sorry to bring an old issue alive again, but I have a workaround that works for self-hosted GitLab solutions with private repositories. However, be aware that drone should not be in public mode as this will reveal the token used.

After activating the repository with GitLab, go into the repository settings in GitLab, then Services, then Drone CI. Copy the token on that page. Now go into the database for Drone and run the following:

UPDATE repos SET repo_clone='{scheme}://drone-ci-token:{token}@{gitlab-domain}/{owner}/{project}.git' WHERE repo_id={repo_id};

scheme will be either http or https, dependent on your installation. And the repo_id can easily be figured out by running a select * from repos; first. Now when you attempt to push or trigger a build again, it should run successfully.

The downside now is that Drone will show the full clone URL in its output, including the token. This could be a security issue if you aren't careful.

The root cause of the problem, in the end, appears to be Drone not attaching the credentials provided for private repositories. It may be beneficial to allow a toggle for private repositories.

[bradrydzewski]

@ColtonProvias for private repositories drone adds a netrc file which sets drone-ci-token:{token} for all outbound http requests to GitLab. We do the same for GitHub, Bitbucket, Gogs, as well. Again, assuming you are cloning private repositories this should be handled by the netrc. Also, have you tried starting drone with clone_mode=oauth

see http://readme.drone.io/setup/remotes/gitlab/#gitlab-options:dd87d86ac28472949397c68a306dd605

The root cause of the problem, in the end, appears to be Drone not attaching the credentials provided for private repositories. It may be beneficial to allow a toggle for private repositories.

I am able to clone private repositories with Drone and GitLab (gitlab.com) without issue

[ColtonProvias]

I am using clone_mode=oauth. It still wasn't setting up an appropriate netrc file. I'm using a self-hosted GitLab-CE instead of gitlab.com, so that might have something to do with it.

[bradrydzewski]

It still wasn't setting up an appropriate netrc file.

Are you sure this is the root cause? the git plugin is responsible for creating the netrc file and the git plugin is agnostic to the service provider (it doesn't know anything about GitHub, GitLab, etc). It creates the netrc for all private repositories, regardless of service provider.

We can fully isolate this test by running the git plugin locally. You can get a copy of the git plugin by running docker pull plugins/git and execute like this:

docker run --rm \
  -e DRONE_NETRC_MACHINE=gitlab.com \
  -e DRONE_NETRC_USERNAME=drone-ci-token \
  -e DRONE_NETRC_PASSWORD={token} \
  -e DRONE_COMMIT_SHA=636b4f33dbe819f65a06036cfad619afa2812503 \
  -e DRONE_REMOTE_URL=https://gitlab.com/bradrydzewski/test.git \
  plugins/git

I ran the above command and it succeeded with the below output.

git init
Initialized empty Git repository in /.git/
git remote add origin https://gitlab.com/bradrydzewski/test.git
git fetch --no-tags origin +refs/heads/master:</command>
From https://gitlab.com/bradrydzewski/test
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
git reset --hard -q 636b4f33dbe819f65a06036cfad619afa2812503
git submodule update --init --recursive

[ColtonProvias]

If I run the plugins/git image directly in rancher with the appropriate values as you showed, there is no error. However, running it normally via drone produces the following JSON in the build container's command:

{
  "repo": {
    "id": 3,
    "owner": "Zeitghost",
    "name": "Webmail",
    "full_name": "Zeitghost/Webmail",
    "avatar_url": "",
    "link_url": "https://git.zeitghost.io/Zeitghost/Webmail",
    "scm": "",
    "clone_url": "http://git.zeitghost.io/Zeitghost/Webmail.git",
    "default_branch": "master",
    "timeout": 60,
    "private": true,
    "trusted": false,
    "allow_pr": true,
    "allow_push": true,
    "allow_deploys": true,
    "allow_tags": true
  },
  "build": {
    "id": 3,
    "number": 1,
    "event": "push",
    "status": "running",
    "enqueued_at": 1461694271,
    "created_at": 1461694271,
    "started_at": 1461694271,
    "finished_at": 0,
    "deploy_to": "",
    "commit": "ed45a85f463db8647e56b30a8f3f4514add2a421",
    "branch": "master",
    "ref": "refs/heads/master",
    "refspec": "",
    "remote": "",
    "title": "",
    "message": "Update README.md to trigger build",
    "timestamp": 0,
    "author": "Colton Provias",
    "author_avatar": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61.jpg?s=128",
    "author_email": "admin@example.com",
    "link_url": ""
  },
  "build_last": {
    "id": 0,
    "number": 0,
    "event": "",
    "status": "",
    "enqueued_at": 0,
    "created_at": 0,
    "started_at": 0,
    "finished_at": 0,
    "deploy_to": "",
    "commit": "",
    "branch": "",
    "ref": "",
    "refspec": "",
    "remote": "",
    "title": "",
    "message": "",
    "timestamp": 0,
    "author": "",
    "author_avatar": "",
    "author_email": "",
    "link_url": ""
  },
  "job": {
    "id": 3,
    "number": 1,
    "error": "",
    "status": "running",
    "exit_code": 0,
    "enqueued_at": 1461694271,
    "started_at": 1461694271,
    "finished_at": 0,
    "environment": {}
  },
  "keys": {
    "public": "REDACTED",
    "private": "REDACTED"
  },
  "netrc": {
    "machine": "git.zeitghost.io:443",
    "login": "oauth2",
    "user": "REDACTED"
  },
  "config": "publish:\\n  docker:\\n    registry: docker.zeitghost.io\\n    insecure: true\\n    repo: zeitghost/webmail\\n    tag: latest\\n    username: $$REGISTRY_USERNAME\\n    password: $$REGISTRY_PASSWORD\\n    email: $$REGISTRY_EMAIL\\n\\ndeploy:\\n  rancher:\\n    url: http://rancher.zeitghost.io/v1/projects/1a5\\n    access_key: $$RANCHER_ACCESS\\n    secret_key: $$RANCHER_SECRET\\n    service: mail/webmail\\n    docker_image: docker.zeitghost.io/zeitghost/webmail:latest\\n    start_first: true\\n    confirm: true\\n    timeout: 300\\n\\nnotify:\\n  slack:\\n    channel: development\\n    webhook_url: $$SLACK_WEBHOOK",
  "secret": "",
  "system": {
    "version": "",
    "link_url": "http://drone:8000",
    "plugins": [
      ""
    ],
    "globals": [
      ""
    ],
    "privileged_plugins": [
      ""
    ]
  }
}

The link_url is http://drone:8000 since the SSL is self-signed for drone (thus blocks GitLab from contacting drone normal), whereas the SSL for GitLab is Let's Encrypt.

The docker-compose.yml for my Drone CI service is as follows:

drone:
  environment:
    REMOTE_DRIVER: gitlab
    REMOTE_CONFIG: https://git.zeitghost.io:443?client_id=REDACTED&client_secret=REDACTED&skip_verify=true&clone_mode=oauth
    DATABASE_DRIVER: postgres
    DATABASE_CONFIG: postgres://REDACTED:REDACTED@postgres:5432/drone?sslmode=disable
  log_driver: ''
  labels:
    io.rancher.container.pull_image: always
  tty: true
  log_opt: {}
  image: drone/drone:latest
  links:
  - 'postgres:'
  volumes:
  - drone-data:/var/lib/drone
  - /var/run/docker.sock:/var/run/docker.sock
  stdin_open: true
  volume_driver: convoy-gluster
postgres:
  environment:
    POSTGRES_PASSWORD: REDACTED
    POSTGRES_USER: REDACTED
    POSTGRES_DB: drone
  log_driver: ''
  labels:
    io.rancher.container.pull_image: always
  tty: true
  log_opt: {}
  image: postgres:9.5.2
  volumes:
  - drone-db:/var/lib/postgresql/data
  stdin_open: true
  volume_driver: convoy-gluster

[bradrydzewski]

I am guessing the issue is the port included in the machine:

  "netrc": {
-    "machine": "git.zeitghost.io:443",
+    "machine": "git.zeitghost.io",
    "login": "oauth2",
    "user": "REDACTED"
  },

I would expect the 443 to be implicit when using https. Perhaps when you setup drone you should omit 443 from the configuration? This would prevent it from being included in the netrc.

Reference https://github.com/drone/drone/blob/master/remote/gitlab/gitlab.go#L287

[ColtonProvias]

That was the issue. Thanks!

I added the :443 in as the documentation at http://readme.drone.io/setup/remotes/gitlab/#gitlab-configuration:dd87d86ac28472949397c68a306dd605 stated that it defaults to :80 otherwise. That fixed the issue.

[bradrydzewski]

Thanks, we can get that fixed in the docs. We also added code to the Gogs implementation to strip the port, which probably makes sense for GitLab as well https://github.com/drone/drone/blob/master/remote/gogs/gogs.go#L178

I'll make those changes for good measure and then we can close this issue.

[bradrydzewski]

I'm closing this issue because we have now verified with multiple GitLab installs that Drone is able to clone public and private repositories when correctly configured.

I recently patched the issue described above where the netrc was unable to clone private repositories when the gitlab url included port number: #1223 (comment)

Note that in addition to my own testing that shows Drone can clone from a properly configured GitLab instance (at GitLab.com) this was also confirmed by @ColtonProvias on his private GitLab instance once we fixed the netrc and port issue: #1223 (comment)

So in conclusion, if you are coming to this issue because you think there is a bug with Drone please take a look at your GitLab configuration. Also note that you can test the Git clone capabilities in isolation per this comment: #1223 (comment)