lfs files failing to download

[lbergelson]

I'm having an issue where git-lfs pull is failing to download all of the files it should be:

It seems to output a - on files that fail. I'm running this on travis-ci as part of our tests suit, so debugging is made more difficult. The failures don't seem to happen locally. The failures only seem to happen when we add new files to git-lfs. We're using v1.1.0 built from source.

This is the output of git lfs pull followed by git lfs ls-files

pull
Git LFS: (9 of 22 files) 67.15 MB / 442.28 MB                                  ls-files
6b1304800e - src/test/resources/large/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.bam
06d71f4306 * src/test/resources/large/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.bam.bai
04b5860ec2 * src/test/resources/large/NA12878.RNAseq.bai
0507bbbe36 * src/test/resources/large/NA12878.RNAseq.bam
297a01dc33 * src/test/resources/large/NA12878.RNAseq.bam.bai
8947c455ba - src/test/resources/large/dbsnp_138.b37.1.1-65M.vcf
0dc0ef4be8 * src/test/resources/large/dbsnp_138.b37.1.1-65M.vcf.idx
e0c7060c55 * src/test/resources/large/dbsnp_138.b37.20.21.vcf
5b62a51eba * src/test/resources/large/dbsnp_138.b37.20.21.vcf.idx
ca5099e45c * src/test/resources/large/exampleLargeFile.txt
12184a8d50 * src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.bai
ea11f3fceb * src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.bam
192fee224c * src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.doNotFixOverhangs.bai
84048fc009 - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.doNotFixOverhangs.bam
e3e479fd66 - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxBasesInOverhang5.bai
b7a3ffe18f - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxBasesInOverhang5.bam
1412788c2e - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxMismatchesInOverhang0.bai
07a5ed8dfe - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxMismatchesInOverhang0.bam
671c073f3a - src/test/resources/large/human_g1k_v37.20.21.2bit
233c4c703a - src/test/resources/large/human_g1k_v37.20.21.dict
bafe4b1274 - src/test/resources/large/human_g1k_v37.20.21.fasta
8e5daa85c5 - src/test/resources/large/human_g1k_v37.20.21.fasta.fai

for some reason many files show as - which seems to mean they are not downloaded.

Here's the .gitattributes file which is located in src/test/resources/large

CEUTrio.HiSeq.WGS.b37.NA12878.20.21.bam filter=lfs diff=lfs merge=lfs -text
CEUTrio.HiSeq.WGS.b37.NA12878.20.21.bam.bai filter=lfs diff=lfs merge=lfs -text
dbsnp_138.b37.20.21.vcf filter=lfs diff=lfs merge=lfs -text
dbsnp_138.b37.20.21.vcf.idx filter=lfs diff=lfs merge=lfs -text
human_g1k_v37.20.21.dict filter=lfs diff=lfs merge=lfs -text
human_g1k_v37.20.21.fasta filter=lfs diff=lfs merge=lfs -text
human_g1k_v37.20.21.fasta.fai filter=lfs diff=lfs merge=lfs -text
exampleLargeFile.txt filter=lfs diff=lfs merge=lfs -text
dbsnp_138.b37.1.1-65M.vcf filter=lfs diff=lfs merge=lfs -text
dbsnp_138.b37.1.1-65M.vcf.idx filter=lfs diff=lfs merge=lfs -text
human_g1k_v37.20.21.2bit filter=lfs diff=lfs merge=lfs -text
*.bam filter=lfs diff=lfs merge=lfs -text
*.bai filter=lfs diff=lfs merge=lfs -text

The last two lines were added in the pull request that is failing to check out files.

I don't understand what's causing the failures. Can you help?

[technoweenie]

It's hard to say without any error output. Can you try adding GIT_TRACE=1 to the front of the git lfs pull command?

[lbergelson]

Here's the output with `GIT_TRACE=1

pull
trace git-lfs: run_command: 'git' rev-parse --abbrev-ref HEAD
trace git-lfs: run_command: 'git' config branch.HEAD.remote
trace git-lfs: run_command: 'git' rev-parse HEAD --symbolic-full-name HEAD
trace git-lfs: run_command: 'git' config -l
trace git-lfs: run_command: git rev-list --objects --no-walk ffc71022d54b337529939faee41438bc52843dde
trace git-lfs: run_command: git cat-file --batch-check
trace git-lfs: run_command: git cat-file --batch
trace git-lfs: run_command: 'git' rev-parse HEAD --symbolic-full-name HEAD
trace git-lfs: tq: starting 3 transfer workers
trace git-lfs: tq: running as batched queue, batch size of 100
trace git-lfs: fetch src/test/resources/large/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.bam [6b1304800e60c0ac0358df137bdad48b7857a36465b04fef3fbbb09380f04746]
trace git-lfs: fetch src/test/resources/large/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.bam.bai [06d71f4306957dea0f46adf78fce2969e80c37ecb070c2d51088b8db88ef9ab1]
trace git-lfs: fetch src/test/resources/large/NA12878.RNAseq.bai [04b5860ec2fce7bb2ae3d5c8429b36462d26e306239de494e6f1e42ba98e23a7]
trace git-lfs: fetch src/test/resources/large/NA12878.RNAseq.bam [0507bbbe367c26e328e6ec733b90acd1e240c54470b9d3dbf8b1d2ba63141e14]
trace git-lfs: fetch src/test/resources/large/NA12878.RNAseq.bam.bai [297a01dc331f21c358b61793b6b55a2831e36ea2b3df686251fa0d8b7f140010]
trace git-lfs: fetch src/test/resources/large/dbsnp_138.b37.1.1-65M.vcf [8947c455bad5761570e1c70fc972aff5432b07061f085ef5ad123b3b741d7891]
trace git-lfs: fetch src/test/resources/large/dbsnp_138.b37.1.1-65M.vcf.idx [0dc0ef4be816c2daca0045668d8143cbf522b723b38e4fa33f1aec0cebab50f5]
trace git-lfs: fetch src/test/resources/large/dbsnp_138.b37.20.21.vcf [e0c7060c55e35cc879da11b0ac9f3f6772fe95dc8b927fafb80f0139e6c4c065]
trace git-lfs: fetch src/test/resources/large/dbsnp_138.b37.20.21.vcf.idx [5b62a51ebafae25cc735b16a90c4233a13372e66aa75c2410c4819c4140d5f94]
trace git-lfs: fetch src/test/resources/large/exampleLargeFile.txt [ca5099e45c685208cb4c99ddc2b2157bf3eb7bee7e728b86c09afc58514a8bcd]
trace git-lfs: fetch src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.bai [12184a8d5046fc16c932e52fc2598642b41dc69e7c8bf897dfc313ce777a6d46]
trace git-lfs: fetch src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.bam [ea11f3fceb2108d9097ef7222f4725c9f02c13007c1ecd12cc49121dd5dbdafb]
trace git-lfs: fetch src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.doNotFixOverhangs.bai [192fee224cd7a92d68ce8ac31a8e3873f99ec99f11cecc9af867bd41a5ab4a1d]
trace git-lfs: fetch src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.doNotFixOverhangs.bam [84048fc009bdfee05561aeb3b5e8054c539184c30ecf8bec50f8388546707758]
trace git-lfs: fetch src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxBasesInOverhang5.bai [e3e479fd666963b173977ab08d2aca5522b4ee3f64e63e0d5ddb264d7d371abc]
trace git-lfs: fetch src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxBasesInOverhang5.bam [b7a3ffe18f105ca51dd219c6926ec75bed6566f602e008a572254216b0f82aa5]
trace git-lfs: fetch src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxMismatchesInOverhang0.bai [1412788c2e52de6588a14da65357dc0f6013ae396fa70b13018d571cebd6260b]
trace git-lfs: fetch src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxMismatchesInOverhang0.bam [07a5ed8dfe586238bb6226df054438f55b753620fbeacbfd1869956cfe653d9f]
trace git-lfs: fetch src/test/resources/large/human_g1k_v37.20.21.2bit [671c073f3aedded63b187dbee993d1e441af74d21184135add78f497b7f9543b]
trace git-lfs: fetch src/test/resources/large/human_g1k_v37.20.21.dict [233c4c703ac071c37ff052c12cd5ad2016a0e84021e9ca3d01764b7165821aa7]
trace git-lfs: fetch src/test/resources/large/human_g1k_v37.20.21.fasta [bafe4b1274a4ee6f9b5d9348a6a5f31d49781595bcda25cc452ad6fd6ad3cb34]
trace git-lfs: fetch src/test/resources/large/human_g1k_v37.20.21.fasta.fai [8e5daa85c50bc06de53d26794dc02ed3bddcad083badae7a601e524982eeb58d]
trace git-lfs: tq: sending batch of size 22
trace git-lfs: ssh: git@github.com git-lfs-authenticate broadinstitute/gatk.git download 
trace git-lfs: run_command: git ls-tree -r -l --full-tree ffc71022d54b337529939faee41438bc52843dde
trace git-lfs: run_command: git cat-file --batch
trace git-lfs: ssh: download attempted with git@github.com.  Error: exit status 255
trace git-lfs: api: batch 22 files
trace git-lfs: HTTP: POST https://github.com/broadinstitute/gatk.git/info/lfs/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: {"objects":[{"oid":"6b1304800e60c0ac0358df137bdad48b7857a36465b04fef3fbbb09380f04746","size":79856849,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/6b/13/6b1304800e60c0ac0358df137bdad48b7857a36465b04fef3fbbb09380f04746?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=c11259e15b199cacdc43c57340ebd064686aea0bbeaa0bc682407b4b2082cce2&X-A
trace git-lfs: HTTP: mz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"06d71f4306957dea0f46adf78fce2969e80c37ecb070c2d51088b8db88ef9ab1","size":11520,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/06/d7/06d71f4306957dea0f46adf78fce2969e80c37ecb070c2d51088b8db88ef9ab1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=74a3af149d1ed193d9a792026527c59cd9e4773f42a18f003a3a4759bbe1da21&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"04b5860ec2fce7bb2ae3d5c8429b36462d26e306239de494e6f1e42ba98e23a7","size":2344,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/04/b5/04b5860ec2fce7bb2ae3d5c8429b36462d26e306239de494e6f1e42ba98e23a7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X
trace git-lfs: HTTP: -Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=61108aa369d63b4afe75a03b1621908484cfcc943a7ad5844a59fea966f1f259&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"0507bbbe367c26e328e6ec733b90acd1e240c54470b9d3dbf8b1d2ba63141e14","size":668054,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/05/07/0507bbbe367c26e328e6ec733b90acd1e240c54470b9d3dbf8b1d2ba63141e14?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=b4804c75b139374e828df4409dba988d7df35e8f73e9c1436bb5475faf0cced6&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"297a01dc331f21c358b61793b6b55a2831e36ea2b3df686251fa0d8b7f140010","size":1600,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/29/7a/297a01dc331f21c358b61793b6b55a2831e36ea2b3df686251fa0d8b7f140010?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=6c82bf43e779bd31bfc75c8e177611ad3241f6ec6fe8c25e5ff019767b2f0267&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"8947c455bad5761570e1c70fc972aff5432b07061f085ef5ad123b3b741d7891","size":237432747,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/89/47/8947c455bad5761570e1c70fc972aff5432b07061f085ef5ad123b3b741d7891?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=38a6dc6f873640a52fc6b78f14f86fde461388ad9802d90ea7127d5a1da7a1a5&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"0dc0ef4be816c2daca0045668d8143cbf522b723b38e4fa33f1aec0cebab50f5","size":262025,"actions":
trace git-lfs: HTTP: {"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/0d/c0/0dc0ef4be816c2daca0045668d8143cbf522b723b38e4fa33f1aec0cebab50f5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=c567730d6549c5e5754cd7d914a6678990ff2f57298e6bbe8b0ee0041dc028db&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"e0c7060c55e35cc879da11b0ac9f3f6772fe95dc8b927fafb80f0139e6c4c065","size":1575950,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/e0/c7/e0c7060c55e35cc879da11b0ac9f3f6772fe95dc8b927fafb80f0139e6c4c065?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=c1421291109213fac97d0548ccb7d09f712270241b205ad42f67b364f092a1d3&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"5b62a51ebafae25cc735b16a90c4233a13372e66aa75c2410c4819c4140d5f94","size":83916,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/5b/62/5b62a51ebafae25cc735b16a90c4233a13372e66aa75c2410c4819c4140d5f94?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=bd162751b62b6d9c8d13c94ec8aaf5f5999c1796a6586d0bca89b6db36545477&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"ca5099e45c685208cb4c99ddc2b2157bf3eb7bee7e728b86c09afc58514a8bcd","size":78,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/ca/50/ca5099e45c685208cb4c99ddc2b2157bf3eb7bee7e728b86c09afc58514a8bcd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=f562b7eae8dd5a5dcebd80696f4d352f9d4590e55d8b5566ca1a4cbe7fbe4d84&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"12184a8d5046fc16c932e52fc2598642b41dc69e7c8bf897dfc313ce777a6d46","size":1656,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/12/18/12184a8d5046fc16c932e52fc2598642b41dc69e7c8bf897dfc313ce777a6d46?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=213cb3fb38b09dee74fc0352f805c31288c2f40b9989ecfb545ee6cc9e35658b&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"ea11f3fceb2108d9097ef7222f4725c9f02c13007c1ecd12cc49121dd5dbdafb","size":766182,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/ea/11/ea11f3fceb2108d9097ef7222f4725c9f02c13007c1ecd12cc49121dd5dbdafb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=fa3a3ec225920c8250993bcf46fef00ee41f705c4221e206150f680f3e82f681&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"192fee224cd7a92d68ce8ac31a8e3873f99ec99f11cecc9af867bd41a5ab4a1d","size":1656,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/19/2f/192fee224cd7a92d68ce8ac31a8e3873f99ec99f11cecc9af867bd41a5ab4a1d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=1fd7f9aba5da1e94f59e771c06b9b4c0d0dd244b8762215c8398a9d0b68974fc&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"84048fc009bdfee05561aeb3b5e8054c539184c30ecf8bec50f8388546707758","size":768854,"actions":{"download":{"href":"https://github-cloud.s3.amazon
trace git-lfs: HTTP: aws.com/alambic/media/107463621/84/04/84048fc009bdfee05561aeb3b5e8054c539184c30ecf8bec50f8388546707758?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=d19af32c58051b4223476ce799f8745d4e7ebfebf43f035a30603fcaf9b92326&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"e3e479fd666963b173977ab08d2aca5522b4ee3f64e63e0d5ddb264d7d371abc","size":1656,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/e3/e4/e3e479fd666963b173977ab08d2aca5522b4ee3f64e63e0d5ddb264d7d371abc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=d235ba4e741749d01f677dc3be93c60a4eafd0a821db166afd2e02205f019315&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:02Z"}}},{"oid":"b7a3ffe18f105ca51dd219c6926ec75bed6566f602e008a572254216b0f82aa5","size":768836,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/b7/a3/b7a3ffe18f105ca51dd219c6926ec75bed6566f602e008a572254216b0f82aa5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162803Z&X-Amz-Expires=86400&X-Amz-Signature=b052fa5bd88b5f28b8971efed49c5fbe614feb4c22b833c74301d9c162400abe&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:03Z"}}},{"oid":"1412788c2e52de6588a14da65357dc0f6013ae396fa70b13018d571cebd6260b","size":1656,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/14/12/1412788c2e52de6588a14da65357dc0f6013ae396fa70b13018d571cebd6260b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162803Z&X-Amz-Expires=86400&X-Amz-Signature=55ec18e9d0175966998fdd90c5ce0748a63fd437739156d0392280ab5801bda1&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:03Z"}}},{"oid":"07a5ed8dfe586238bb6226df054438f55b753620fbeacbfd1869956cfe653d9f","size":765549,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/07/a5/07a5ed8dfe586238bb6226df054438f55b753620fbeacbfd1869956cfe653d9f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162803Z&X-Amz-Expires=86400&X-Amz-Signature=22baa7a43e465984612790e3341176dfe280957cb5b631b9712ef9fb28383fa8&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:03Z"}}},{"oid":"671c073f3aedded63b187dbee993d1e441af74d21184135add78f497b7f9543b","size":27789108,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/67/1c/671c073f3aedded63b187dbee993d1e441af74d21184135add78f497b7f9543b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162803Z&X-Amz-Expires=86400&X-Amz-Signature=dd6de72b962b4af00d795863a58ec83563ebf08b600d82eb667575de4047fef4&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:03Z"}}},{"oid":"233c4c703ac071c37ff052c12cd5ad2016a0e84021e9ca3d01764b7165821aa7","size":317,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/23/3c/233c4c703ac071c37ff052c12cd5ad2016a0e84021e9ca3d01764b7165821aa7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162803Z&X-Amz-Expires=86400&X-Amz-Signature=3fe7589be3f6b5b2b1dcd259cd5f91efa8b28c83e6cf326d57dfcba230b203aa&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:03Z"}}},{"oid":"bafe4b1274a4ee6f9b5d9348a6a5f31d49781595bcda25cc452ad6fd6ad3cb34","size":113008112,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/ba/fe/bafe4b1274a4ee6f9b5d9348a6a5f31d49781595bcda25cc452ad6fd6ad3cb34?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162803Z&X-Amz-Expires=86400&X-Amz-Signature=a04ce6dd8c6e1eeb13928cebfe83cb35506f4999f4532be4ca2aee18ca35affe&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:03Z"}}},{"oid":"8e5daa85c50bc06de53d26794dc02ed3bddcad083badae7a601e524982eeb58d","size":48,"actions":{"download":{"href":"https://github-cloud.s3.amazonaws.com/alambic/media/107463621/8e/5d/8e5daa85c50bc06de53d26794dc02ed3bddcad083badae7a601e524982eeb58d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162803Z&X-Amz-Expires=86400&X-Amz-Signature=89e07519ae29be6787c67d419ed783b43a309fd3cd631432f0b6f72bb0aaeb88&X-Amz-SignedHeaders=host&actor_id=0&token=1","expires_at":"2015-12-23T16:28:03Z"}}}]}
trace git-lfs: HTTP: 
Git LFS: (0 of 22 files) 0 B / 442.28 MB                                       trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/6b/13/6b1304800e60c0ac0358df137bdad48b7857a36465b04fef3fbbb09380f04746?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=c11259e15b199cacdc43c57340ebd064686aea0bbeaa0bc682407b4b2082cce2&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/06/d7/06d71f4306957dea0f46adf78fce2969e80c37ecb070c2d51088b8db88ef9ab1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=74a3af149d1ed193d9a792026527c59cd9e4773f42a18f003a3a4759bbe1da21&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/04/b5/04b5860ec2fce7bb2ae3d5c8429b36462d26e306239de494e6f1e42ba98e23a7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=61108aa369d63b4afe75a03b1621908484cfcc943a7ad5844a59fea966f1f259&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/05/07/0507bbbe367c26e328e6ec733b90acd1e240c54470b9d3dbf8b1d2ba63141e14?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=b4804c75b139374e828df4409dba988d7df35e8f73e9c1436bb5475faf0cced6&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/29/7a/297a01dc331f21c358b61793b6b55a2831e36ea2b3df686251fa0d8b7f140010?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=6c82bf43e779bd31bfc75c8e177611ad3241f6ec6fe8c25e5ff019767b2f0267&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/89/47/8947c455bad5761570e1c70fc972aff5432b07061f085ef5ad123b3b741d7891?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=38a6dc6f873640a52fc6b78f14f86fde461388ad9802d90ea7127d5a1da7a1a5&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: 200
Git LFS: (3 of 22 files) 122.69 KB / 442.28 MB                                 trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/0d/c0/0dc0ef4be816c2daca0045668d8143cbf522b723b38e4fa33f1aec0cebab50f5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=c567730d6549c5e5754cd7d914a6678990ff2f57298e6bbe8b0ee0041dc028db&X-Amz-SignedHeaders=host&actor_id=0&token=1
Git LFS: (4 of 22 files) 5.51 MB / 442.28 MB                                   trace git-lfs: HTTP: 200
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/e0/c7/e0c7060c55e35cc879da11b0ac9f3f6772fe95dc8b927fafb80f0139e6c4c065?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=c1421291109213fac97d0548ccb7d09f712270241b205ad42f67b364f092a1d3&X-Amz-SignedHeaders=host&actor_id=0&token=1
Git LFS: (5 of 22 files) 15.59 MB / 442.28 MB                                  trace git-lfs: HTTP: 200
Git LFS: (5 of 22 files) 23.30 MB / 442.28 MB                                  trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/5b/62/5b62a51ebafae25cc735b16a90c4233a13372e66aa75c2410c4819c4140d5f94?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=bd162751b62b6d9c8d13c94ec8aaf5f5999c1796a6586d0bca89b6db36545477&X-Amz-SignedHeaders=host&actor_id=0&token=1
Git LFS: (6 of 22 files) 24.62 MB / 442.28 MB                                  trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/ca/50/ca5099e45c685208cb4c99ddc2b2157bf3eb7bee7e728b86c09afc58514a8bcd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=f562b7eae8dd5a5dcebd80696f4d352f9d4590e55d8b5566ca1a4cbe7fbe4d84&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/12/18/12184a8d5046fc16c932e52fc2598642b41dc69e7c8bf897dfc313ce777a6d46?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=213cb3fb38b09dee74fc0352f805c31288c2f40b9989ecfb545ee6cc9e35658b&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/ea/11/ea11f3fceb2108d9097ef7222f4725c9f02c13007c1ecd12cc49121dd5dbdafb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=fa3a3ec225920c8250993bcf46fef00ee41f705c4221e206150f680f3e82f681&X-Amz-SignedHeaders=host&actor_id=0&token=1
Git LFS: (9 of 22 files) 25.09 MB / 442.28 MB                                  trace git-lfs: HTTP: 200
Git LFS: (9 of 22 files) 29.56 MB / 442.28 MB                                  trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/19/2f/192fee224cd7a92d68ce8ac31a8e3873f99ec99f11cecc9af867bd41a5ab4a1d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=1fd7f9aba5da1e94f59e771c06b9b4c0d0dd244b8762215c8398a9d0b68974fc&X-Amz-SignedHeaders=host&actor_id=0&token=1
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/107463621/84/04/84048fc009bdfee05561aeb3b5e8054c539184c30ecf8bec50f8388546707758?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20151222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151222T162802Z&X-Amz-Expires=86400&X-Amz-Signature=d19af32c58051b4223476ce799f8745d4e7ebfebf43f035a30603fcaf9b92326&X-Amz-SignedHeaders=host&actor_id=0&token=1
ls-files
6b1304800e - src/test/resources/large/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.bam
06d71f4306 * src/test/resources/large/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.bam.bai
04b5860ec2 * src/test/resources/large/NA12878.RNAseq.bai
0507bbbe36 * src/test/resources/large/NA12878.RNAseq.bam
297a01dc33 * src/test/resources/large/NA12878.RNAseq.bam.bai
8947c455ba - src/test/resources/large/dbsnp_138.b37.1.1-65M.vcf
0dc0ef4be8 * src/test/resources/large/dbsnp_138.b37.1.1-65M.vcf.idx
e0c7060c55 * src/test/resources/large/dbsnp_138.b37.20.21.vcf
5b62a51eba * src/test/resources/large/dbsnp_138.b37.20.21.vcf.idx
ca5099e45c * src/test/resources/large/exampleLargeFile.txt
12184a8d50 * src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.bai
ea11f3fceb * src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.bam
192fee224c * src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.doNotFixOverhangs.bai
84048fc009 - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.doNotFixOverhangs.bam
e3e479fd66 - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxBasesInOverhang5.bai
b7a3ffe18f - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxBasesInOverhang5.bam
1412788c2e - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxMismatchesInOverhang0.bai
07a5ed8dfe - src/test/resources/large/expected.NA12878.RNAseq.splitNcigarReads.maxMismatchesInOverhang0.bam
671c073f3a - src/test/resources/large/human_g1k_v37.20.21.2bit
233c4c703a - src/test/resources/large/human_g1k_v37.20.21.dict
bafe4b1274 - src/test/resources/large/human_g1k_v37.20.21.fasta
8e5daa85c5 - src/test/resources/large/human_g1k_v37.20.21.fasta.fai

I see an error here, but I don't know how to interpret it:

trace git-lfs: run_command: git cat-file --batch
trace git-lfs: ssh: download attempted with git@github.com.  Error: exit status 255

[larsxschneider]

I (probably) ran into the same error and here is how you can reproduce it:

1.) Clone my public test repo without fetching LFS files:
GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/larsxschneider/lfstest.git

2.) Pull LFS files:
cd lfstest; git lfs pull

3.) List files that were not downloaded:
git lfs ls-files | grep -v "*"

This is my local setup:

$ git --version
git version 2.5.4 (Apple Git-61)

$ git lfs version
git-lfs/1.1.0 (GitHub; darwin amd64; go 1.5.1)

$ git config -l | grep lfs
lfs.batch=true
lfs.concurrenttransfers=10
filter.lfs.clean=git-lfs clean %f
filter.lfs.smudge=git-lfs smudge %f
filter.lfs.required=true

[lbergelson]

@larsxschneider Thanks for putting together a clean way to reproduce this. It looks like the same behavior to me. I don't see the same ssh error when I run your test case with GIT_TRACE=1, so it's likely that that's not the cause.

[technoweenie]

ssh: download attempted with git@github.com. Error: exit status 255

That's an intermittent error talking to our SSH server. Apparently it still worked though, as it downloaded a bunch of stuff successfully.

Have you tried tracking files by extension, and not filename? That seems more foolproof. Sometimes the patterns don't match, and you're not tracking what you think you're tracking with LFS sometimes.

[lbergelson]

Hmn. I haven't tried switching to tracking by extension, but @larsxschneider's example does track by extension and exhibits the same problem. It also only occurs when you do the first clone of the repo with git lfs disabled. So I'm pretty sure I'm tracking the files I expect to be, since it works when I clone with git lfs enabled.

I've found that if I run git lfs pull a second time it seems to fix the problem. It's confusing and ugly though to have to repeat it.

Also, git lfs fsck reports Git LFS fsck OK when a bunch of files were not successfully checked out. Although I may not understand what fsck is meant to check exactly, I assumed it checked "is this repository in a valid state with all it's files present".

[larsxschneider]

@technoweenie can you replicate the problem? Thanks 😄

[larsxschneider]

@technoweenie Does the test case work for you?

[technoweenie]

I can replicate the problem. After running pull the first time, tree .git/lfs/objects returned "727 directories, 501 files". So it's downloading everything properly, but not checking everything out to the working directory.

Also, running git lfs fetch and then git lfs checkout worked, so it seems to be related to just git lfs pull.

[larsxschneider]

Great, I am glad you can reproduce the error. Is this something you can/want to look at? git lfs pull is essential for the fast clone approach we are using.

[MichPerry-GG]

Greetings all. My team has been having similar issues with smudge filter failing or not being able to access media. Google search led me to this issue and #911. After trying everything, the one workaround that got me to a fully functional production state was technoweenie's last reply. Simple git clone and checkout of branches fails immediately, so I started trying the suggested workarounds.

My current process is:

// Skip smudge
git lfs install --skip-smudge

// Fresh clone of repo (master branch)
git clone my_repo_url

// Switch to development branch
git checkout development

// Fetch lfs files
git lfs fetch

// Really get the files
git lfs checkout

// Fix smudge
git lfs install --force

If I try git lfs pull instead of fetch + checkout, I get media access errors or my files get marked as modified (when there have been no changes).

This work around at least has me moving forward, but is not ideal for the rest of my team who are using visual clients like SmartGit and GitHub Desktop App.

[strich]

I can confirm that I've also seen this issue in the wild. As I couldn't ever pin down what was going on I never noted it down. However in some scripts I have internally I actively have it perform git lfs pull twice in a row to ensure it succeeds.

There is definitely some weird checkout bug causing files to fail on first try.