Gitlab can't clone or push #3384

Closed
proverbface opened this Issue Mar 25, 2013 · 29 comments

Comments

Projects
None yet

I tried to find the solution from many websites, but I couldn't.

I can't push or clone repositories. I try to clone https://github.com/gitlabhq/gitlabhq.git but gitlab stays like the next image

CloningLoop3-25-2013 5-36-15 PM

Then when I tried push something into another repo called 'GEOSurvey', I get the next trouble.

debug-3-25-2013 5-43-27 PM

Which step of the installation guide (https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md) could make this error or what I can do?

I meet the same or similar problem. GitLab 5.0 and mater both.

The check logs are:

shiquanwang@las-project:/home/git/gitlab$ sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production

System information
System:     Ubuntu 12.10
Current User:   git
Using RVM:  no
Ruby Version:   1.9.3p392
Gem Version:    1.8.23
Bundler Version:1.3.4
Rake Version:   10.0.3

GitLab information
Version:    5.1.0pre
Revision:   8c91239
Directory:  /home/git/gitlab
DB Adapter: mysql2
URL:        http://las-project.cn-093.lan.philips.com
HTTP Clone URL: http://las-project.cn-093.lan.philips.com/some-project.git
SSH Clone URL:  git@las-project.cn-093.lan.philips.com:some-project.git
Using LDAP: no
Using Omniauth: no

GitLab Shell
Version:    1.2.0
Repositories:   /home/git/repositories/
Hooks:      /home/git/gitlab-shell/hooks/
Git:        /usr/bin/git

and

shiquanwang@las-project:/home/git/gitlab$ sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
Checking Environment ...

Git configured for git user? ... yes
Has python2? ... yes
python2 is supported version? ... yes

Checking Environment ... Finished

Checking Gitlab Shell ...

GitLab Shell version? ... OK (1.2.0)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
post-receive hook up-to-date? ... yes
post-receive hooks in repos are links: ... 
Shiquan Wang / GitLab ... repository is empty

Checking Gitlab Shell ... Finished

Checking Sidekiq ...

Running? ... yes

Checking Sidekiq ... Finished

Checking GitLab ...

Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
Projects have satellites? ... 
Shiquan Wang / GitLab ... can't create, repository is empty

Checking GitLab ... Finished

When I clone:

shiquanwang@las-z820:~/code/test$ git clone git@las-project.cn-093.lan.philips.com:shiquanwang/gitlab.git
Cloning into 'gitlab'...
fatal: The remote end hung up unexpectedly

When I ssh:

shiquanwang@las-z820:~/code/test$ ssh -vT git@las-project.cn-093.lan.philips.com 
OpenSSH_6.0p1 Debian-3ubuntu1, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /home/shiquanwang/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to las-project.cn-093.lan.philips.com [161.92.164.172] port 22.
debug1: Connection established.
debug1: identity file /home/shiquanwang/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/shiquanwang/.ssh/id_rsa-cert type -1
debug1: identity file /home/shiquanwang/.ssh/id_dsa type -1
debug1: identity file /home/shiquanwang/.ssh/id_dsa-cert type -1
debug1: identity file /home/shiquanwang/.ssh/id_ecdsa type -1
debug1: identity file /home/shiquanwang/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-3ubuntu1
debug1: match: OpenSSH_6.0p1 Debian-3ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-3ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 0e:24:55:d8:6b:0a:d3:cf:36:20:2e:7d:d6:9f:a4:a9
debug1: Host 'las-project.cn-093.lan.philips.com' is known and matches the ECDSA host key.
debug1: Found key in /home/shiquanwang/.ssh/known_hosts:2
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/shiquanwang/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to las-project.cn-093.lan.philips.com ([161.92.164.172]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Remote: Forced command.
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Pty allocation disabled.
debug1: Remote: Forced command.
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Pty allocation disabled.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
/usr/local/lib/ruby/1.9.1/json/common.rb:148:in `parse': 746: unexpected token at '<html> (JSON::ParserError)
<head>
<title>Welcome to nginx!</title>
</head>
<body bgcolor="white" text="black">
<center><h1>Welcome to nginx!</h1></center>
</body>
</html>
'
    from /usr/local/lib/ruby/1.9.1/json/common.rb:148:in `parse'
    from /home/git/gitlab-shell/lib/gitlab_net.rb:25:in `discover'
    from /home/git/gitlab-shell/lib/gitlab_shell.rb:28:in `exec'
    from /home/git/gitlab-shell/bin/gitlab-shell:16:in `<main>'
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2664, received 3264 bytes, in 0.5 seconds
Bytes per second: sent 5275.5, received 6463.7
debug1: Exit status 1

leopku commented Mar 26, 2013

It seems your ngix + unicorn + gitlab not configure correctly. git-shell can't parser incorrect html page.
`

<title>Welcome to nginx!</title>

Welcome to nginx!

`

It seems something wrong with gitlab-shell.

Contributor

akzhan commented Mar 26, 2013

Gitlab-shell send request to Gitlab itself through http protocol.

Please check your gitlab-shell config. file. There must be an url to your Gitlab website with trailing slash.

im having the exact same problem.

i checked the gitlab-shell config file, with or without trailing slash works the same, nginx was installed with the simple instructions in the configuration guide

/usr/local/lib/ruby/1.9.1/json/common.rb:148:in `parse': 746: unexpected token at '<html> (JSON::ParserError)
<head>
<title>Welcome to nginx!</title>
</head>
<body bgcolor="white" text="black">
<center><h1>Welcome to nginx!</h1></center>
</body>
</html>
'
    from /usr/local/lib/ruby/1.9.1/json/common.rb:148:in `parse'
    from /home/git/gitlab-shell/lib/gitlab_net.rb:25:in `discover'
    from /home/git/gitlab-shell/lib/gitlab_shell.rb:28:in `exec'
    from /home/git/gitlab-shell/bin/gitlab-shell:16:in `<main>'

the problem is that the api is returning html rather than some json, but why ? i can't figure out what's the problem with nginx or unicorn configuration

invoking this on the command line

curl -v http://server/api/v3/internal/allowed?key_id=2&action=git-upload-pack&ref=_any&project=user/project

yields this

$ * About to connect() to server port 80 (#0)
*   Trying 127.0.1.1... connected
> GET /api/v3/internal/allowed?key_id=2 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: server
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.1.19
< Date: Tue, 26 Mar 2013 10:26:24 GMT
< Content-Type: text/html
< Content-Length: 151
< Last-Modified: Mon, 04 Oct 2004 15:04:06 GMT
< Connection: keep-alive
< Accept-Ranges: bytes
<
<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body bgcolor="white" text="black">
<center><h1>Welcome to nginx!</h1></center>
</body>
</html>
* Connection #0 to host server left intact
* Closing connection #0

the logs when invoked from the ssh -vT git@server, show

/var/log/auth.log

127.0.0.1 - - [25/Mar/2013:19:52:53 -0500] "GET //api/v3/internal/allowed?key_id=2&action=git-upload-pack&ref=_any&project=user/project HTTP/1.1" 200 151 "-" "Ruby"

Currently experiencing similar symtoms. In-fact the symtoms were the same as above until I updated all references to the hostname in the gitlab-shell, gitlab and nginx configs to the same address (we use a local dns entry so things were working before the update using just the machines local IP in the nginx config and gitlab.domain.lcl in the gitlab settings).

At the moment we're unable to clone or push or edit files straight from the gitlab web interface (the page reloads but the changes aren't saved). On cloning or pushing the following error is returned.

fatal: protocol error: bad line length character: http
fatal: The remote end hung up unexpectedly

Invoking the command @apertoire suggested (curl -v http://server/api/v3/internal/allowed?key_id=2&action=git-upload-pack&ref=_any&project=user/project) results in the following.

pilot@mm:~$ curl -v http://<host>/api/v3/internal/allowed?key_id=2&action=git-upload-pack&ref=_any&project=user/project
[1] 6344
[2] 6345
[3] 6346
pilot@mm:~$ * About to connect() to <host> port 80 (#0)
*   Trying 127.0.0.1... connected
> GET /api/v3/internal/allowed?key_id=2 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: <host>
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Server: nginx/1.1.19
< Date: Tue, 26 Mar 2013 11:24:42 GMT
< Content-Type: application/json
< Content-Length: 40
< Connection: keep-alive
< Status: 500 Internal Server Error
< X-UA-Compatible: IE=Edge,chrome=1
< Cache-Control: no-cache
< X-Request-Id: f7ab6cbf90aa143d34b27b7cac77eeac
< X-Runtime: 0.031105
< X-Rack-Cache: miss
< 
* Connection #0 to host <host> left intact
* Closing connection #0
{"message"=>"500 Internal Server Error"}

Requesting the above URI from another machine on the network however results 'false' and requesting http:///api/v3/internal/check respondes with the correct JSON string.

{"api_version":"v3","gitlab_version":"5.0.0","gitlab_rev":"0c7f426"}

Inside the server I tried to clone GEOSurvey and I get error again.

root@proverbio-debian:/home/git# git clone git@192.168.XXX.128:geosurvey.git
Cloning into 'geosurvey'...
git@192.168.XXX.128's password:
fatal: 'geosurvey.git' does not appear to be a git repository
fatal: The remote end hung up unexpectedly

proverbface referenced this issue in gitlabhq/gitlab-shell Mar 26, 2013

Closed

No push, pull, clone, .. possible #20

I found out why.

When invoking the internal api like http://server/api/v3/internal/allowed?key_id=2&action=git-upload-pack&ref=_any&project=user/project from other machines, the nginx config will give gitlab as default web so it works.

However when actually doing clone, push and pull. The internal api is invoked by gitlab-shell on the same machine of the gitlab server. Now since the gitlab config for nginx /etc/nginx/sites-enabled/gitlab is set as:

server {
  listen YOUR_SERVER_IP:80 default_server;         # e.g., listen 192.168.1.1:80;
  server_name YOUR_SERVER_FQDN;     # e.g., server_name source.example.com;
  root /home/git/gitlab/public;

user git on the server invoked gitlab-shell to access web api at YOUR_SERVER_FQDN, but now the server IP to user git is 127.0.0.1. So gitlab is not valid website now. But the default nginx site listens to any IP at port 80 so the

<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body bgcolor="white" text="black">
<center><h1>Welcome to nginx!</h1></center>
</body>
</html>

is given out. Because it is the index page of default nginx site.

Solution is to set the gitlab nginx config file /etc/nginx/sites-enabled/gitlab as:

server {
  listen *:80 default_server;         # e.g., listen 192.168.1.1:80;
  server_name YOUR_SERVER_FQDN;     # e.g., server_name source.example.com;
  root /home/git/gitlab/public;

Hope this solves problems.

If don't use FQDN only use IP, then the mentioned problem would not appear.

shiquanwang referenced this issue in gitlabhq/gitlab-shell Mar 26, 2013

Closed

redundant slash in api host #29

The trouble was solved by @shiquanwang. Please see the previous instructions.

Pull request -> #3403

paulvi commented Apr 9, 2013

Solution for Nginx is above. What about Apache configuration for Bitnami GitLab
Bitnami GitLab 5.0 and git & Eclipse EGit quick start?

I have my server config like @shiquanwang said:

listen *:80 default_server;         # e.g., listen 192.168.1.1:80; In most cases *:80 is a good idea
  server_name git.mysite.pt;     # e.g., server_name source.example.com;
  root /home/git/gitlab/public;

And I cannot do any git pull/clone/push or ssh connection to my server

wanxsb commented Jun 14, 2013

I also have this kind of problem. When I clone a project, the new created user was asked to input password for git@gitlab.xxx.com, after I input the password for the git account of OS, it shows the following error:
fatal: 'erp.git' does not appear to be a git repository fatal: The remote end hung up unexpectedly
BTW, I have already import the SSH key by web interface and it is OK for root user.

paulvi commented Jun 14, 2013

This question is already answered on Stackoverflow
(ask there, not in this thread)

To put it simply asking for git password shows that GitLab is not configured well.

yhostc commented Jul 10, 2013

I also encountered the same problem, the solution is static IP in nginx

nginx.conf add:

client_max_body_size 100m;

@inetfuture inetfuture pushed a commit to inetfuture/gitlab-tweak that referenced this issue Nov 2, 2013

@frlan frlan Adding a short hint to *:80 for default listen address on nginx examp…
…le config

Based on posting gitlabhq/gitlabhq#3384 (comment) I was able to solved an issue here. Pointing out *:80 might could prevent others for struggeling here
bfd5c1a

@inetfuture inetfuture pushed a commit to inetfuture/gitlab-tweak that referenced this issue Nov 2, 2013

@frlan frlan Adding a short hint to *:80 for default listen address on nginx examp…
…le config

Based on posting gitlabhq/gitlabhq#3384 (comment) I was able to solved an issue here. Pointing out *:80 might could prevent others for struggeling here
19370de

miko referenced this issue in sameersbn/docker-gitlab Feb 25, 2014

Closed

git push/pull: Access denied #26

fibo99 commented Sep 3, 2014

I have myself found in this crazy bug.
My case was only when creating a new repository thru gitlab: all semed to go fine... but is not really.
You can spot this problem at once by changing "settings" in the project web interface: if you get an error 500, then this is the same case than the one I had.

  • Usual step of restarting Gitlab is NOT enough
  • In my case (your mileage may differ) the owners of the project (directory /home/git/repositories/user/project) were root:root while all the others were git:git.
  • once this was changed, all was back to normal

I had the same issue while following this document.
It says:

sudo -u git -H bundle exec rake gitlab🐚install[v2.0.1] REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production

As result rake gitlab:check fails. To make it green:

  • Remove [v2.0.1] and reload GitLab

Also

Upd: nice 🐚

welly commented Feb 25, 2015

I'm having a similar problem of git push resulting in:

fatal: protocol error: bad line length character: API

I'm using Gitlab 7.8 which was installed using the omnibus installer. I'm also using my own nginx server rather than the bundled ominbus nginx server. I'm assuming something has cropped up when I made the amends for it to run like that.

welly commented Feb 25, 2015

Appears the problem is in the config.yml file in gitlab-shell - the URL was slightly different. I've changed that and am trying to rebuild/reconfigure now.

Rushor commented Feb 28, 2015

does this worked for you @welly?

welly commented Mar 5, 2015

All working now, thank you!

In case anyone else finds this helpful, the issue for me was that my .gitlab_shell_secret file was symlinked to an older release of our site's file.

./bin/check now passes.

for any one having this issue with ( bitnami configuration or any configuration) and want to resolved fast just use full path instead..
exp: git clone git@server_adress:/full_path_adress_of_the_projact_on_the_server.git
exp: git clone /opt/bitnami/apps/gitlab/repositories/test/project.git
exp: git push -u origin master
done:

edited: I forgot to mention , it is very important to check if you have added the SSH keys to git-lab via the web page.
try adding the remote machine keys and your workstation keys to the gitlab.

Just installed GitLab CE 7.12.2 using Omnibus installer on Ubuntu 14.04. I created a project in GitLab, added my SSH key from GitLab and tried to clone it to my Mac.

git clone ssh://git@gitlab.xxxx.com:2022/userxxxx/test-project.git
Cloning into 'test-project'...
GitLab: Failed to authorize your Git request: internal API unreachable
fatal: Could not read from remote repository.

Initially, I thought it might have been my non-standard server setup. My server uses a non-standard SSH port 2022. I also installed an external Nginx because I'm going to host Redmine on this server as well. Neither of those were the issue. It turns out, since 7.10, something changed in the default /etc/gitlab/gitlab.rb and now a key configuration attribute for shell commands is missing from the config file.

I found this issue on the GitLab issue tracker.

https://gitlab.com/gitlab-org/gitlab-ce/issues/1486

When I opened up /opt/gitlab/embedded/service/gitlab-shell/config.yml I found something wrong.

gitlab_url: "http://127.0.0.1:8080"

That is not correct. As a quick test, I changed it to (obviously xxxx is a fake host name for illustration purposes)

gitlab_url: "http://gitlab.xxxx.com"

Guess what, it worked!

So, knowing I'd need to reconfigure GitLab again at some point, I used the solution found in the GitLab issue tracker.

Add this line to /etc/gitlab/gitlab.rb

gitlab_rails['internal_api_url'] = 'http://gitlab.xxxx.com'

I hope this helps someone else. This configuration option needs to be documented.

Yes, helped me. Thanks a lot.
Don't forget to sudo gitlab-ctl reconfigure

On entering the git repo URL im getting error as : Failed to connect to repository : Remote host closed connection during handshake.
Tried a lot but did not find a solution yet.
git

I had this issue:

GitLab: Failed to authorize your Git request: internal API unreachable
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The cause was the firewall, after adding the following rule everything worked OK:

-A INPUT -p tcp -s localhost --dport 8080 -j ACCEPT

Leaving this here in case it helps anyone else in the future and because a suggestion above, using the public URL for the internal API requests, might have unexpected side effects?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment