Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pantry does not work with git annex #4579

Closed
dbaynard opened this issue Feb 10, 2019 · 12 comments

Comments

Projects
3 participants
@dbaynard
Copy link
Contributor

commented Feb 10, 2019

General summary/comments (optional)

Pantry does not support a git repo which uses git-annex.

Steps to reproduce

For example, using master:

  1. stack new no-annex --resolver https://gist.github.com/dbaynard/9693107a883a6282f52f22785d0072e2/raw/edd32d61464cf0e19236081e250aebbe8cea1112

The resolver includes a git repository which uses git-annex.

Expected

Stack to create the directory and populate with a stack.yaml file.

Actual

Stack creates the directory, then attempts to resolve the git repos in the snapshot, fails with https://github.com/dbaynard/cascade, and so doesn't create the stack.yaml file. The command succeeds on 1.9.3.

Downloading template "new-template" to create project "no-annex" in no-annex/ ...
Looking for .cabal or package.yaml files to use to init the project.
Using cabal packages:
- no-annex/

Selected resolver: 8.6.3-13.5-baynard-snapshot
Loading a snapshot from a SnapshotDef: 8.6.3-13.5-baynard-snapshot
Loading a snapshot from a SnapshotDef: lts-13.5
Loading a snapshot from a SnapshotDef: ghc-8.6.3
Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
Unsupported tarball from /tmp/get-repo18829/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader
Verbose log
Version 1.10.0 (6902 commits) x86_64 hpack-0.31.1
2019-02-10 17:30:20.358305: [debug] Opening local template: "new-template.hsfiles"
2019-02-10 17:30:20.358606: [info] Downloading template "new-template" to create project "no-annex" in no-annex/ ...
2019-02-10 17:30:20.358685: [debug] Downloading /commercialhaskell/stack-templates/master/new-template.hsfiles
2019-02-10 17:30:20.795431: [debug] Opening local template: "/home/db/.stack/templates/new-template.hsfiles"
2019-02-10 17:30:20.809348: [info] Looking for .cabal or package.yaml files to use to init the project.
2019-02-10 17:30:20.811577: [info] Using cabal packages:
2019-02-10 17:30:20.812241: [info] - no-annex/

2019-02-10 17:30:20.812622: [debug] Running hpack on /home/db/Projects/stack-triage/no-annex/package.yaml
2019-02-10 17:30:20.816604: [debug] hpack generated a modified version of /home/db/Projects/stack-triage/no-annex/no-annex.cabal
2019-02-10 17:30:20.967687: [info] Selected resolver: 8.6.3-13.5-baynard-snapshot
2019-02-10 17:30:20.970425: [debug] Trying to decode /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache
2019-02-10 17:30:20.970564: [debug] Exception ignored when attempting to load /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache: /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache: openBinaryFile: does not exist (No such file or directory)
2019-02-10 17:30:20.970653: [debug] Failure decoding /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache
2019-02-10 17:30:20.970713: [info] Loading a snapshot from a SnapshotDef: 8.6.3-13.5-baynard-snapshot
2019-02-10 17:30:20.970814: [debug] Trying to decode /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/d7f31aacdd7580839a49a29174d1025edd557547229c8e84bdc74bf8aee94c56.cache
2019-02-10 17:30:21.103588: [debug] Success decoding /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/d7f31aacdd7580839a49a29174d1025edd557547229c8e84bdc74bf8aee94c56.cache
2019-02-10 17:30:21.103745: [debug] Parsing cabal file for Repo from https://github.com/jameysharp/corrode, commit 34053342c2f1ca04f23ad94d67057f14e74d9fb9
2019-02-10 17:30:21.106157: [debug] Parsing cabal file for Repo from https://github.com/mgattozzi/curryrs, commit 6d3ecbc3d1786ccc16e2ff20c6d6da53734f6b03
2019-02-10 17:30:21.108578: [debug] Parsing cabal file for Repo from https://github.com/smallhadroncollider/taskell, commit 3c2b5bb272df616670202bb3da42e1dc0606015a
2019-02-10 17:30:21.114289: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict
2019-02-10 17:30:21.118805: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-db
2019-02-10 17:30:21.120918: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-html
2019-02-10 17:30:21.122066: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-json
2019-02-10 17:30:21.123896: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-quickcheck
2019-02-10 17:30:21.125179: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay
2019-02-10 17:30:21.127342: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay-data
2019-02-10 17:30:21.129108: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay-serial
2019-02-10 17:30:21.130675: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir readp
2019-02-10 17:30:21.132285: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/hserv, commit 736555ca4869da6d8b3d818df5d32a0831a58488
2019-02-10 17:30:21.133937: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/htidy, commit a4139a34f128a858169218211565cae7e1ba5752
2019-02-10 17:30:21.135793: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/pandoc-include, commit ad1f72c485c4c00908326a90efe94ead9bb73c77
2019-02-10 17:30:21.137710: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/lucid-from-html, commit 901bb8edf5e77f227cde847763ee429c620ef62b in subdir lucid-from-html
2019-02-10 17:30:21.140073: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/cascade, commit 9699b1a6f61c28ce154f271aef9bb784f65fd47e in subdir cascade
2019-02-10 17:30:21.140958: [info] Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
2019-02-10 17:30:21.141851: [debug] Run process within /tmp/get-repo19260: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-10 17:30:22.705593: [debug] Process finished in 1563ms: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-10 17:30:22.705788: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-10 17:30:22.714857: [debug] Process finished in 9ms: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-10 17:30:22.715085: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git submodule update --init --recursive
2019-02-10 17:30:22.785615: [debug] Process finished in 70ms: /usr/bin/git submodule update --init --recursive
2019-02-10 17:30:22.785991: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo19260/foo.tar HEAD
2019-02-10 17:30:22.798792: [debug] Process finished in 12ms: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo19260/foo.tar HEAD
2019-02-10 17:30:22.802218: [debug] parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo19260/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

Stack version

$ stack --version
Version 1.10.0 (6902 commits) x86_64 hpack-0.31.1

I've no idea why it doesn't show the git ref.

Method of installation

Built from git repo.

@dbaynard

This comment has been minimized.

Copy link
Contributor Author

commented Feb 10, 2019

Note that if the directory already exists, stack build fails for the same reason.

@dbaynard

This comment has been minimized.

Copy link
Contributor Author

commented Feb 10, 2019

I'd like stack to support git-annex, but I'm happy to consider it out of scope. This error ought to be handled, though. Would you mind taking a look, @qrilka?

@dbaynard

This comment has been minimized.

Copy link
Contributor Author

commented Feb 10, 2019

Actually, in this case I don't need any git-annex files to build the package so it ought to work.

@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 10, 2019

Sure, I will try to find time to investigate this

@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2019

@dbaynard this doesn't appear to be related to tar problems, e.g. I see the following verbose output:

2019-02-13 00:02:05.712578: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/cascade, commit 9699b1a6f61c28ce154f271aef9bb784f65fd47e in subdir cascade
2019-02-13 00:02:05.712919: [info] Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
2019-02-13 00:02:05.713000: [debug] Run process within /tmp/get-repo10938: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-13 00:02:07.521022: [debug] Process finished in 1808ms: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-13 00:02:07.521206: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-13 00:02:07.526820: [debug] Process finished in 6ms: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-13 00:02:07.526981: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git submodule update --init --recursive
2019-02-13 00:02:07.554099: [debug] Process finished in 27ms: /usr/bin/git submodule update --init --recursive
2019-02-13 00:02:07.554269: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo10938/foo.tar HEAD
2019-02-13 00:02:07.557223: [debug] Process finished in 3ms: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo10938/foo.tar HEAD
2019-02-13 00:02:07.558883: [debug] parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo10938/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

See git archive being used - that's the way Pantry currently deals with Git repos - it just gets an archive and uses archive handling code and as your archive contains annex files and Pantry doesn't do anything about them you get broken links in tar file and Pantry fails.
@snoyberg could you give more details why we go through archives for Git repos?

@snoyberg

This comment has been minimized.

Copy link
Contributor

commented Feb 13, 2019

Yes:

  1. It can be more efficient than asking git to populate the filesystem
  2. On filesystems without proper symbolic link support or case insensitivity, it's the only way to ensure we get reproducible repo contents, otherwise the tree hashes would be different on different OSes

We could theoretically make Pantry simply ignore broken symlinks, but the error we're getting here is due to more rigorous error checking.

@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 13, 2019

@dbaynard the only solution I could propose in this case is to add fonts export-ignore to .gitattributes of your git-annex repo. If that sounds as a good workaround then it makes sense to document it.

@dbaynard

This comment has been minimized.

Copy link
Contributor Author

commented Feb 13, 2019

Yes, fixing the archive seems like the right way to do this. I'll try your suggestion. I wonder if there are other cases where archiving fails, too?


Meanwhile:

parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo10938/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

This snippet shows the code which is throwing this error (the upper case is the end of the definition of toSimple).

case Map.lookup dest files of
Nothing -> Left $ "Symbolic link dest not found from " ++ mePath me ++ " to " ++ relDest ++ ", looking for " ++ dest
Just me' ->
case meType me' of
METNormal -> Right $ SimpleEntry dest FTNormal
METExecutable -> Right $ SimpleEntry dest FTExecutable
METLink _ -> Left $ "Symbolic link dest cannot be a symbolic link, from " ++ mePath me ++ " to " ++ relDest
case traverse toSimple files of
Left e -> throwIO $ UnsupportedTarball loc $ T.pack e

Would line 385 be a good place to add that stack doesn't support git-annex, and provide a link to the workaround?

@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 19, 2019

@dbaynard that code deals with archives in general so I'm not sure we could talk explicitly about git-annex here, probably we could give some hint that it could be the reason behind such a failure but also there are other options - e.g. archive could come not from git at all and bad symlinks could get put into it my some accident.

@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2019

@dbaynard what do you think could be a proper resolution for this issue?

@dbaynard

This comment has been minimized.

Copy link
Contributor Author

commented Mar 3, 2019

give some hint that it could be the reason behind such a failure but also there are other options - e.g. archive could come not from git at all and bad symlinks could get put into it my some accident.

This sounds like an improvement.

dbaynard added a commit to dbaynard/cascade that referenced this issue Mar 7, 2019

@dbaynard

This comment has been minimized.

Copy link
Contributor Author

commented Mar 7, 2019

the only solution I could propose in this case is to add fonts export-ignore to .gitattributes of your git-annex repo.

This works. Thanks @qrilka! I'll PR with some documentation.

@dbaynard dbaynard self-assigned this Mar 9, 2019

Pantry automation moved this from To do to Done Mar 26, 2019

ashvds added a commit to ashvds/adl that referenced this issue Jul 4, 2019

Adding 'fonts export-ignore' as per haskel doco
running into issues building hx-deploy-tool , according to the error message, this fixes the problem
commercialhaskell/stack#4579
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.