Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LFS does not transfer files. file already closed #1418

Open
AlexandreRio opened this issue Nov 24, 2017 · 28 comments
Open

LFS does not transfer files. file already closed #1418

AlexandreRio opened this issue Nov 24, 2017 · 28 comments

Comments

@AlexandreRio
Copy link

@AlexandreRio AlexandreRio commented Nov 24, 2017

I tried to use git lfs on my gitlab but it doesn't work, even for really small files.
LFS is enabled both in gitlab settings and project settings.

I'm using git-lfs/2.3.4 and git version 2.15.0 with gitlab 10.1.4.

Here is what happens with a basic text file:

Git LFS: (0 of 1 files) 45 B / 5 B                                                                                                                  
LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
error: failed to push some refs to 'ssh://git@git.alexrio.fr:10022/orbital/lfs-test.git'

With a bigger project, and bigger commit, I have the same issue around 24MB / 324MB with extra 413 errors.
If I look in the data folder of the container the lfs directory is empty.
The web interface of gitlab is behind a nginx proxy but the ssh port is directly binded to my host machine.

Here is a full trace of the error:

12:01:03.175203 git.c:344               trace: built-in: git 'push' 'origin' 'master'
12:01:03.175662 run-command.c:626       trace: run_command: 'ssh' '-p' '10022' 'git@git.alexrio.fr' 'git-receive-pack '\''/orbital/lfs-test.git'\'''
Enter passphrase for key '/home/ario/.ssh/id_rsa': 
12:01:16.000223 run-command.c:626       trace: run_command: '.git/hooks/pre-push' 'origin' 'ssh://git@git.alexrio.fr:10022/orbital/lfs-test.git'
12:01:16.011365 git.c:576               trace: exec: 'git-lfs' 'pre-push' 'origin' 'ssh://git@git.alexrio.fr:10022/orbital/lfs-test.git'
12:01:16.011436 run-command.c:626       trace: run_command: 'git-lfs' 'pre-push' 'origin' 'ssh://git@git.alexrio.fr:10022/orbital/lfs-test.git'
12:01:16.016594 trace git-lfs: run_command: 'git' version
12:01:16.018416 trace git-lfs: run_command: 'git' config -l
12:01:16.020568 trace git-lfs: tq: running as batched queue, batch size of 100
Enter passphrase for key '/home/ario/.ssh/id_rsa': 
12:01:29.396981 trace git-lfs: pre-push: refs/heads/master eb3b684a5e156214eccef052c88bac70550a97a8 refs/heads/master 0000000000000000000000000000000000000000
12:01:29.397291 trace git-lfs: run_command: git rev-list --objects eb3b684a5e156214eccef052c88bac70550a97a8 --not --remotes=origin --
12:01:29.398457 trace git-lfs: run_command: git cat-file --batch
12:01:29.401794 trace git-lfs: tq: sending batch of size 1
12:01:29.402137 trace git-lfs: run_command: ssh -p 10022 -- git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
Enter passphrase for key '/home/ario/.ssh/id_rsa': 
12:01:41.600471 trace git-lfs: api: batch 1 files
12:01:41.600743 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:01:43.565694 trace git-lfs: HTTP: 301
12:01:43.565774 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:01:43.565799 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:01:52.917704 trace git-lfs: HTTP: 200
12:01:52.920837 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:01:52.921069 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 0 B / 5 B                                                                                                                            12:01:52.921635 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 5 B / 5 B                                                                                                                            12:01:55.194689 trace git-lfs: HTTP: 301
12:01:55.194781 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:01:55.194812 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:01:55.195256 trace git-lfs: tq: retrying object f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
12:01:55.195323 trace git-lfs: tq: enqueue retry #1 for "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2" (size: 5)
12:01:55.195387 trace git-lfs: tq: sending batch of size 1
12:01:55.195612 trace git-lfs: ssh cache: git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
12:01:55.195669 trace git-lfs: api: batch 1 files
12:01:55.195835 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:01:57.294234 trace git-lfs: HTTP: 301
12:01:57.294297 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:01:57.294314 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:01:59.446129 trace git-lfs: HTTP: 200
12:01:59.448863 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:01:59.449003 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 5 B / 5 B                                                                                                                            12:01:59.449411 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 10 B / 5 B                                                                                                                           12:02:00.171424 trace git-lfs: HTTP: 301
12:02:00.171490 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:00.171513 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:00.171804 trace git-lfs: tq: retrying object f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
12:02:00.171898 trace git-lfs: tq: enqueue retry #2 for "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2" (size: 5)
12:02:00.171961 trace git-lfs: tq: sending batch of size 1
12:02:00.172187 trace git-lfs: ssh cache: git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
12:02:00.172243 trace git-lfs: api: batch 1 files
12:02:00.172405 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:01.396428 trace git-lfs: HTTP: 301
12:02:01.396452 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:01.396460 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:10.193762 trace git-lfs: HTTP: 200
12:02:10.196442 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:02:10.196573 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 10 B / 5 B                                                                                                                           12:02:10.197000 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 15 B / 5 B                                                                                                                           12:02:11.755360 trace git-lfs: HTTP: 301
12:02:11.755457 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:11.755493 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:11.755792 trace git-lfs: tq: retrying object f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
12:02:11.755869 trace git-lfs: tq: enqueue retry #3 for "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2" (size: 5)
12:02:11.755934 trace git-lfs: tq: sending batch of size 1
12:02:11.756160 trace git-lfs: ssh cache: git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
12:02:11.756217 trace git-lfs: api: batch 1 files
12:02:11.756386 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:18.594296 trace git-lfs: HTTP: 301
12:02:18.594362 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:18.594391 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:21.144779 trace git-lfs: HTTP: 200
12:02:21.147695 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:02:21.147843 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 15 B / 5 B                                                                                                                           12:02:21.148340 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 20 B / 5 B                                                                                                                           12:02:21.511176 trace git-lfs: HTTP: 301
12:02:21.511247 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:21.511287 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:21.511616 trace git-lfs: tq: retrying object f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
12:02:21.511686 trace git-lfs: tq: enqueue retry #4 for "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2" (size: 5)
12:02:21.511742 trace git-lfs: tq: sending batch of size 1
12:02:21.512009 trace git-lfs: ssh cache: git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
12:02:21.512071 trace git-lfs: api: batch 1 files
12:02:21.512240 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:21.920760 trace git-lfs: HTTP: 301
12:02:21.920820 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:21.920840 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:22.863083 trace git-lfs: HTTP: 200
12:02:22.864274 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:02:22.864348 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 20 B / 5 B                                                                                                                           12:02:22.864540 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 25 B / 5 B                                                                                                                           12:02:23.721290 trace git-lfs: HTTP: 301
12:02:23.721358 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:23.721381 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:23.721670 trace git-lfs: tq: retrying object f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
12:02:23.721740 trace git-lfs: tq: enqueue retry #5 for "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2" (size: 5)
12:02:23.721791 trace git-lfs: tq: sending batch of size 1
12:02:23.722056 trace git-lfs: ssh cache: git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
12:02:23.722123 trace git-lfs: api: batch 1 files
12:02:23.722303 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:24.034440 trace git-lfs: HTTP: 301
12:02:24.034503 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:24.034525 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:24.672625 trace git-lfs: HTTP: 200
12:02:24.675637 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:02:24.675829 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 25 B / 5 B                                                                                                                           12:02:24.676361 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 30 B / 5 B                                                                                                                           12:02:24.978193 trace git-lfs: HTTP: 301
12:02:24.978261 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:24.978286 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:24.978637 trace git-lfs: tq: retrying object f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
12:02:24.978707 trace git-lfs: tq: enqueue retry #6 for "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2" (size: 5)
12:02:24.978748 trace git-lfs: tq: sending batch of size 1
12:02:24.978934 trace git-lfs: ssh cache: git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
12:02:24.978985 trace git-lfs: api: batch 1 files
12:02:24.979128 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:25.288009 trace git-lfs: HTTP: 301
12:02:25.288080 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:25.288101 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:27.371194 trace git-lfs: HTTP: 200
12:02:27.374290 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:02:27.374469 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 30 B / 5 B                                                                                                                           12:02:27.374950 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 35 B / 5 B                                                                                                                           12:02:31.375567 trace git-lfs: HTTP: 301
12:02:31.375638 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:31.375661 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:31.375987 trace git-lfs: tq: retrying object f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
12:02:31.376046 trace git-lfs: tq: enqueue retry #7 for "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2" (size: 5)
12:02:31.376098 trace git-lfs: tq: sending batch of size 1
12:02:31.376329 trace git-lfs: ssh cache: git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
12:02:31.376387 trace git-lfs: api: batch 1 files
12:02:31.376551 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:33.276349 trace git-lfs: HTTP: 301
12:02:33.276408 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:33.276438 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:02:48.351981 trace git-lfs: HTTP: 200
12:02:48.354974 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:02:48.355122 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 35 B / 5 B                                                                                                                           12:02:48.355613 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 40 B / 5 B                                                                                                                           12:02:50.639273 trace git-lfs: HTTP: 301
12:02:50.639344 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:50.639378 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:02:50.639727 trace git-lfs: tq: retrying object f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
12:02:50.639797 trace git-lfs: tq: enqueue retry #8 for "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2" (size: 5)
12:02:50.639850 trace git-lfs: tq: sending batch of size 1
12:02:50.640061 trace git-lfs: ssh cache: git@git.alexrio.fr git-lfs-authenticate orbital/lfs-test.git upload
12:02:50.640125 trace git-lfs: api: batch 1 files
12:02:50.640289 trace git-lfs: HTTP: POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:03:01.850585 trace git-lfs: HTTP: 301
12:03:01.850621 trace git-lfs: api: redirect POST http://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch to https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:03:01.850628 trace git-lfs: HTTP: POST https://git.alexrio.fr/orbital/lfs-test.git/info/lfs/objects/batch
12:03:06.227627 trace git-lfs: HTTP: 200
12:03:06.230687 trace git-lfs: HTTP: {"objects":[{"oid":"f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2","size":5,"actions":{"upload":{"href":"http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5","header":{"Authorization":"Basic b3JiaXRhbDo1aHN0UkNRQ2NVNWNNOVdWVHNyV3hzUXpOOXYtM1dwU2pmdi14WDNyUlBqVl8tMXdhdw=="}}}}]}
12:03:06.230876 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 40 B / 5 B                                                                                                                           12:03:06.231345 trace git-lfs: HTTP: PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
Git LFS: (0 of 1 files) 45 B / 5 B                                                                                                                           12:03:08.667376 trace git-lfs: HTTP: 301
12:03:08.667402 trace git-lfs: api: redirect PUT http://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5 to https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:03:08.667411 trace git-lfs: HTTP: PUT https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5
12:03:08.667522 trace git-lfs: tq: refusing to retry "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2", too many retries (8)
Git LFS: (0 of 1 files) 45 B / 5 B                                                                                                                           
LFS: Put https://git.alexrio.fr/orbital/lfs-test.git/gitlab-lfs/objects/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5: read /home/ario/lfs-test/.git/lfs/objects/f2/ca/f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2: file already closed
error: failed to push some refs to 'ssh://git@git.alexrio.fr:10022/orbital/lfs-test.git'
@freakingawesome

This comment has been minimized.

Copy link

@freakingawesome freakingawesome commented Mar 23, 2018

Did you ever find a resolution to this? We've started experiencing the same problem.

@AlexandreRio

This comment has been minimized.

Copy link
Author

@AlexandreRio AlexandreRio commented Mar 23, 2018

I'm sorry to hear that, I still have this error.
Did it work before with your instance? What changes have you made?

@freakingawesome

This comment has been minimized.

Copy link

@freakingawesome freakingawesome commented Mar 23, 2018

LFS was working fine up until a day or two ago. We're on an installed version of Visual Studio Team System and we're about to contact Support. The IT team says they've only restarted the TFS instance in the past day, indicating that no other maintenance has occured.

I'll post back here with a resolution (though I'm not sure whether or not it will be suitable for your Gitlab experience) when we get this figured out.

@freakingawesome

This comment has been minimized.

Copy link

@freakingawesome freakingawesome commented Mar 23, 2018

Turning on tracing for Git commands indicated that the LFS server was returning HTTP 401 (Unauthorized). From a bash terminal:

GIT_TRACE=1 git lfs push origin <branch_name>

omitting lots of output

13:55:57.549907 trace git-lfs: HTTP: 401
13:55:57.549907 trace git-lfs: tq: refusing to retry 
@krischik

This comment has been minimized.

Copy link

@krischik krischik commented Apr 11, 2018

“Visual Studio Team System” — we use that as well. Was there any fix?

@directrix1

This comment has been minimized.

Copy link

@directrix1 directrix1 commented Jun 11, 2018

I'm still having this error. Is there a fix?

@MarkKharitonov

This comment has been minimized.

Copy link

@MarkKharitonov MarkKharitonov commented Oct 19, 2018

We have this problem too. Anyone?

@MisterJD

This comment has been minimized.

Copy link

@MisterJD MisterJD commented Nov 2, 2018

Yes same here, anyone an idea?

@sprayzcs

This comment has been minimized.

Copy link

@sprayzcs sprayzcs commented Dec 8, 2018

I faced the same problem and solved it by executing git lfs fetch

@andriy-guzenko

This comment has been minimized.

Copy link

@andriy-guzenko andriy-guzenko commented Dec 20, 2018

Fixing the domain url for lfs did the job for me

@MarkKharitonov

This comment has been minimized.

Copy link

@MarkKharitonov MarkKharitonov commented Dec 24, 2018

We worked around it by fixing the URLs we use to communicate with the TFS. Our TFS server is configured to return URLs without the domain suffix, so removing the domain suffix from all the URLs in the git configuration fixes the problem.

@Zanndorin

This comment has been minimized.

Copy link

@Zanndorin Zanndorin commented Jan 8, 2019

I'm stuck here in 2019. Nothing written above fixed my issue.

Mine never explicitly fails though. It just keeps running

@AlexandreRio

This comment has been minimized.

Copy link
Author

@AlexandreRio AlexandreRio commented Jan 16, 2019

@Zanndorin even with GIT_TRACE=1 ?

Same, I still have my original issue

@Zanndorin

This comment has been minimized.

Copy link

@Zanndorin Zanndorin commented Jan 18, 2019

@AlexandreRio Well Git_trace only shows us that we have the file already closed issue. so yeah, even "with GIT_TRACE=1".

@shadow7412

This comment has been minimized.

Copy link

@shadow7412 shadow7412 commented Jan 29, 2019

Fixing the domain url for lfs did the job for me

@andriy-guzenko fixing it from what, to what?

@andriy-guzenko

This comment has been minimized.

Copy link

@andriy-guzenko andriy-guzenko commented Feb 1, 2019

@shadow7412 Mark Kharitonov explained this better above. Removing domain suffix from url fixed the issue.

@shadow7412

This comment has been minimized.

Copy link

@shadow7412 shadow7412 commented Feb 1, 2019

For me the solution was a little different.
I'm using nginx as a proxy to the gitlab nginx as I have other stuff running on the server.

When I first installed it, I had issues with an infinite redirect. The solution I found was to change the configuration file to http instead of https. This solution works for everything except lfs.

Instead, leaving the config file as https and forwarding nginx to the https port stops the infinite redirect as well as letting lfs do its thing.

@Adambean

This comment has been minimized.

Copy link

@Adambean Adambean commented Feb 7, 2019

I'm not sure this is a transport issue. It fails regardless of whether I use Apache's reverse proxy (intended) or the Omnibus supplied NGINX. Exact same issue.

Configurations I've tried:

Repository -> Apache (HTTPS) -> Gitlab Workhorse (HTTP) = Fails on push
Repository -> Apache (HTTPS) -> Gitlab Workhorse (HTTPS) = Fails to connect at all (both the web UI and Git access)
Repository -> Apache (HTTPS) -> Omnibus NGINX (HTTP) -> Gitlab Workhorse = Fails on push
Repository -> Apache (HTTPS) -> Omnibus NGINX (HTTPS) -> Gitlab Workhorse = Fails on push
Repository -> Omnibus NGINX (HTTP) -> Gitlab Workhorse = Fails on push
Repository -> Omnibus NGINX (HTTPS) -> Gitlab Workhorse = Fails on push

Relevant Gitlab configuration:

external_url                                                                    = "https://git.example.com"
registry_external_url                                                           = "https://git.example.com"
mattermost_external_url                                                         = "https://git.example.com"

nginx['enable']                                                                 = true
nginx['listen_addresses']                                                       = ["0.0.0.0", "[::]"]
nginx['listen_port']                                                            = 28443
nginx['listen_https']                                                           = true
nginx['real_ip_trusted_addresses']                                              = ['ipv4-of-this-server', 'ipv6-of-this-server']
nginx['real_ip_header']                                                         = 'X-Real-IP'
nginx['real_ip_recursive']                                                      = 'on'
nginx['redirect_http_to_https']                                                 = true
nginx['redirect_http_to_https_port']                                            = 28080
nginx['ssl_certificate']                                                        = "/etc/gitlab/ssl/git.example.com.crt"
nginx['ssl_certificate_key']                                                    = "/etc/gitlab/ssl/git.example.com.key"
nginx['proxy_set_headers'] = {
    "X-Forwarded-Proto" => "http",
    "X-Forwarded-Ssl" => "on"
}

web_server['external_users']                                                    = ['www-data']
letsencrypt['enable']                                                           = false # Copied from certbot to /etc/gitlab/ssl by cron
letsencrypt['contact_emails']                                                   = ['mgmt@example.com']
registry_nginx['redirect_http_to_https']                                        = true
mattermost_nginx['redirect_http_to_https']                                      = true

gitlab_workhorse['listen_network']                                              = "tcp"
gitlab_workhorse['listen_addr']                                                 = "127.0.0.1:8181"

gitlab_rails['gitlab_host']                                                     = "git.example.com"
gitlab_rails['gitlab_ssh_host']                                                 = "git.example.com"
gitlab_rails['trusted_proxies']                                                 = ['ipv4-of-this-server', 'ipv6-of-this-server']

Apache VHOST:

<VirtualHost *:80>
	ServerName	git.example.com
	ServerAdmin	${APACHE_SERVER_ADMIN}

	DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public
	<Directory "/opt/gitlab/embedded/service/gitlab-rails/public">
		Options -Indexes +FollowSymLinks +MultiViews +ExecCGI
		AllowOverride all
		Require all granted
	</Directory>

	ErrorLog ${APACHE_LOG_DIR}/git.example.com.error.log
	CustomLog ${APACHE_LOG_DIR}/git.example.com.access.log combined

	RewriteEngine on

	RewriteRule "/\.|^\.(?!well-known/)" - [NC,L]

	RewriteCond %{HTTPS} !=on
	RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]
	#RedirectMatch permanent "^/(.*)" "https://git.example.com/$1"

	RedirectMatch permanent "^/favicon.ico" "https://www.example.com/favicon.ico"
</VirtualHost>

<IfModule mod_ssl.c>
	<VirtualHost *:443>
		ServerName	git.example.com
		ServerAdmin	${APACHE_SERVER_ADMIN}

		DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public
		<Directory "/opt/gitlab/embedded/service/gitlab-rails/public">
			Options -Indexes +FollowSymLinks +MultiViews +ExecCGI
			AllowOverride all
			Require all granted
		</Directory>

		ErrorLog ${APACHE_LOG_DIR}/git.example.com.error.log
		CustomLog ${APACHE_LOG_DIR}/git.example.com.access.log combined

		SSLEngine		on
		SSLVerifyClient		none
		SSLCertificateFile	/etc/letsencrypt/live/git.example.com/fullchain.pem
		SSLCertificateKeyFile	/etc/letsencrypt/live/git.example.com/privkey.pem

		RewriteEngine on

		RedirectMatch permanent "^/favicon.ico" "https://www.example.com/favicon.ico"

		ProxyPreserveHost	On
		AllowEncodedSlashes	NoDecode
		SSLProxyEngine		On

		<Location />
			Require all granted

			ProxyPassReverse http://127.0.0.1:8181
			ProxyPassReverse https://git.example.com/
		</Location>

		RewriteEngine on

		RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
		RewriteCond %{REQUEST_URI} ^/uploads/.*
		RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE]

		RequestHeader set X_FORWARDED_PROTO 'https'
		RequestHeader set X-Forwarded-Ssl on

		ErrorDocument 404 /404.html
		ErrorDocument 422 /422.html
		ErrorDocument 500 /500.html
		ErrorDocument 502 /502.html
		ErrorDocument 503 /503.html
	</VirtualHost>
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

With the http://127.0.0.1:8181 reverse proxy destination I've tried:

Also tried pushing to the Omnibus NGINX ports directly excluding Apache from the loop completely, exact same issue.

@Cnidarias

This comment has been minimized.

Copy link

@Cnidarias Cnidarias commented May 11, 2019

For me the solution was a little different.
I'm using nginx as a proxy to the gitlab nginx as I have other stuff running on the server.

When I first installed it, I had issues with an infinite redirect. The solution I found was to change the configuration file to http instead of https. This solution works for everything except lfs.

Instead, leaving the config file as https and forwarding nginx to the https port stops the infinite redirect as well as letting lfs do its thing.

@shadow7412 Can you explain a little more what you did exactly? because I think I am running into exactly the same problem but I am usure what you mean. You left apache to https? And what about the nginx setup?

@shadow7412

This comment has been minimized.

Copy link

@shadow7412 shadow7412 commented May 12, 2019

@Cnidarias - I didn't use apache.

So on my server I had nginx already. I some other sites, so letting GitLab take over port 80 or 443 wasn't an option. So I mapped the docker container's port 80 to an arbitrary port number, and set nginx up to reverse proxy to this port on a specific server name.

My nginx install redirects all http traffic to the https equivalent. GitLab seemed to detect that I was connecting on 'port 80' (as that is what I was forwarding to) and would try to redirect the user to https (even though they were technically already using it) thus the infinite loop.

So instead of exposing port 80 - expose port 443, and reverse proxy to that instead.

Is that clearer?

@Cnidarias

This comment has been minimized.

Copy link

@Cnidarias Cnidarias commented May 15, 2019

@shadow7412 yes that was very clear and I was able to get it to work now as well thank you very much

@Adambean

This comment has been minimized.

Copy link

@Adambean Adambean commented Jun 15, 2019

I guess Gitlab LFS doesn't work at all over HTTPS anymore then?

@shadow7412

This comment has been minimized.

Copy link

@shadow7412 shadow7412 commented Jun 15, 2019

@Adambean no, it works...

@Adambean

This comment has been minimized.

Copy link

@Adambean Adambean commented Jun 16, 2019

Hm ok, what would be causing the "file already closed" issue? I've tried a lot of combinations of both with and without the supplied NGINX.

@shadow7412

This comment has been minimized.

Copy link

@shadow7412 shadow7412 commented Jun 16, 2019

Did you read my comment above? It solved the issue for me...

@Adambean

This comment has been minimized.

Copy link

@Adambean Adambean commented Jun 16, 2019

Yes, I read that.

The servers I have running Gitlab did not have NGINX already, only the one that arrived with Gitlab's omnibus package. I had turned this off as I intended to use the already involved Apache to do the reverse proxying (by FQDN virtual host) so we could continue using SVN repositories over HTTPS+WebDAV.

The Gitlab workhorse listens fine on port 8181 for HTTP requests (but not HTTPS requests) and accessing the Gitlab web UI works fine. Just pushing a repository to it does not. I therefore tried 4 other combinations involving the omnibus provided NGINX, both directly and behind Apache, exact same push failure.

@Adambean

This comment has been minimized.

Copy link

@Adambean Adambean commented Jun 22, 2019

Resolved it.

I noticed that Gitlab was using "http://" instead of "https://" in its clone URLs, which was the cause for concern regarding all the https->http redirects when pushing.

Turns out a ruddy inconsistent configuration language caused this issue for us.

external_url = "https://git.example.com" needed to be external_url "https://git.example.com". Just drop the =, recompiled configuration, then pushing worked first time with Apache (no NGINX).

@YawJatah

This comment has been minimized.

Copy link

@YawJatah YawJatah commented Nov 13, 2019

For those that are still having this issue and are using Azure/TFS brand of git, there is a work around, but it requires admin access to the server (and being comfortable with regedits).
The main problem stems from git lfs and NTLM, with a secondary issue being HTTP/2. There is an open issue to try to fix that : git-lfs/git-lfs#3875.

The work around is to:
Step 1:
Turn of HTTP/2 on your windows server by following these step listed in this link:
https://docs.vmware.com/en/VMware-Horizon-7/7.0/com.vmware.view-agent.directconnectionplugin.doc/GUID-BE0738EC-E35E-4215-8E1E-1F08F911D592.html.

Basically you have to add 2 new REG_DWORD values, EnableHttp2Tls and EnableHttp2Cleartext, to this registry key - HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters.

You must reboot for the changes to take effect.

Step 2:
(Warning only use https with Basic Authentication, or else your credentials is exposed over the wire when using plain http)
Turn on Basic Authentication for your TFS webservice in IIS (important: the tfs webservice is inside the Azure Devops/ Team Foundation web site, that is the one you want to change, not the website itself).

Step 3:

Force you git lfs client to use basic authenticaiton, by adding (modifying the following entries in you workspace (the .git folder) git config.

[lfs "https://azuredevopserver/tfs/collection/_git/Project.git/info/lfs"]
	access = basic
[lfs "https://azuredevopserver/tfs/collection/Project/_git/Project/info/lfs/objects/"]
	access = basic
[lfs]
	tlstimeout = 300
    activitytimeout = 60
    dialtimeout = 600444
    concurrenttransfers = 10
    downloadtransfers = basic
    uploadtransfers = basic
    skipdownloaderrors = true

Replace collection, Project and azuredevopserver with the actually url of your server.
Those two git lfs entries are important and without that git lfs will still attempt to use ntlm and fail.
the other git lfs options are optional except for uploadtransfers, and downloadtransfers, those must be set to basic to force git lfs to use basic authentication during batch transfer.

This was tested on git lfs 2.8.0 and 2.9.0, you mileage might vary if you are using and earlier version or newer version of git lfs.

Hope this work around helps , it isn't trivial (and might not be feasible if you don't have access to the TFS server).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.