New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Right Nothing" printed to the console during setup #2175

Closed
silky opened this Issue May 22, 2016 · 41 comments

Comments

Projects
None yet
@silky
Contributor

silky commented May 22, 2016

with

>stack --version
Version 1.1.3, Git revision 8f7e62fbe3620a4b0ae5eb833cebfa50820f5050 x86_64 hpack-0.14.0

when i attempt a setup i see:

⚡ 05:37 PM noon ∈ ~>stack --resolver nightly-2016-05-13 setup
Run from outside a project, using implicit global project config
Using resolver: nightly-2016-05-13 specified on command line
Downloaded nightly-2016-05-13 build plan.    
Did not find .cabal file for servant-yaml-0.1.0.0 with Git SHA of d7c8a8eaf9e7c215433f8f39daac7fc6cf116325
Right Nothing
Did not find .cabal file for reducers-3.12.1 with Git SHA of bcebf11be1770ac480d9d0ee6238044f143a57f8
Right Nothing
Did not find .cabal file for math-functions-0.1.7.0 with Git SHA of 5870cda6259e8a838c8cb22043b9d3cbf55d9a8e
Right Nothing
Did not find .cabal file for hw-prim-0.0.0.10 with Git SHA of 3658e799df0b7fceb045899b7a3d686b960b5d01
Right Nothing
Did not find .cabal file for hspec-megaparsec-0.1.1 with Git SHA of 116c887f005a6e599dc9b6ce5627f1b69782f44b
Right Nothing
Did not find .cabal file for dixi-0.6.9.1 with Git SHA of 17cba908e23588d9e5b67541f53afd8590c5c30f
Right Nothing
Did not find .cabal file for cron-0.4.0 with Git SHA of d485877cb0e43ee1b8fcdaa0a7224acbd753186c
Right Nothing
stack will use a locally installed GHC
For more information on paths, see 'stack path' and 'stack exec env'
To use this GHC and packages outside of a project, consider using:
stack ghc, stack ghci, stack runghc, or stack exec

it's mildly unfortunate that Right Nothing is printed here.

introduced a few weeks ago by f483e82#diff-2068556e565acc62de252ac25ed550c4R288

mgsloan added a commit that referenced this issue May 23, 2016

@mgsloan

This comment has been minimized.

Collaborator

mgsloan commented May 23, 2016

Good point, fixed!

@mgsloan mgsloan closed this May 23, 2016

@mgsloan

This comment has been minimized.

Collaborator

mgsloan commented May 30, 2016

Then again, perhaps the root issue is not addressed - why are these SHAs missing?

@mgsloan mgsloan reopened this May 30, 2016

@mgsloan mgsloan added the type: bug label May 31, 2016

@mgsloan mgsloan added this to the P2: Should milestone May 31, 2016

@silky

This comment has been minimized.

Contributor

silky commented Jun 1, 2016

i'm not too sure about that one.

let me know if you want more information from me.

@mgsloan

This comment has been minimized.

Collaborator

mgsloan commented Jun 1, 2016

Oh yeah, to be clear that wasn't a request for more information. More like "this needs further investigation" :)

The failure case here is just a fallback to the old behavior where cabal metadata wasn't fixed to a particular revision. So thankfully this shouldn't be very problematic in practice. Good to fix, though!

@schell

This comment has been minimized.

schell commented Jun 1, 2016

I seem to be running into a similar issue, though the output is a bit different - notably the "Left ..." result:

> stack --version
Version 1.1.2, Git revision cebe10e845fed4420b6224d97dcabf20477bbd4b (3646 commits) x86_64 hpack-0.14.0

> stack build
Downloading nightly-2016-05-20 build plan ...���������������������������������������������                                             ���������������������������������������������Downloaded nightly-2016-05-20 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...����������������������������������������������������������������������������������������������������������������                                                                                                                ����������������������������������������������������������������������������������������������������������������Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...
����������������������������������������������������������������������������������������������������������������                                                                                                                ����������������������������������������������������������������������������������������������������������������Populating index cache ...��������������������������                          ��������������������������Populated index cache.
Did not find .cabal file for zot-0.0.3 with Git SHA of f9092bae03d0044d2b4e3f33609c9544e8c46b19
Left /root/.stack/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-lens-0.1.2.1 with Git SHA of b5797267b30c16afcccbd42fa0ce4b9fa58a7ca9
Left /root/.stack/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-enum-0.2.3.1 with Git SHA of 5bb46141c62ea8a473a2cfac9293a018c804189a
...

I'm using stack on bitbucket's new "pipelines" feature which executes stack from the haskell:7.10.3 docker image.

@schell

This comment has been minimized.

schell commented Jun 1, 2016

The result of using lts-5.2 is similar but different:

stack build
Downloading lts-5.2 build plan ...����������������������������������                                  ����������������������������������Downloaded lts-5.2 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...����������������������������������������������������������������������������������������������������������������                                                                                                                ����������������������������������������������������������������������������������������������������������������Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...����������������������������������������������������������������������������������������������������������������                                                                                                                ����������������������������������������������������������������������������������������������������������������Populating index cache ...��������������������������                          ��������������������������Populated index cache.
/opt/atlassian/bitbucketci/agent/build/.stack-work/downloaded/NC9VHrq5ky0p/: getDirectoryContents: does not exist (No such file or directory)
@simonmichael

This comment has been minimized.

Contributor

simonmichael commented Jun 12, 2016

Another data point: I saw a bunch of these messages after upgrading stack (from 1.1.0 x86_64 hpack-0.13.0 to 1.1.2 x86_64 hpack-0.14.0) and running stack install. I did stack clean and they went away.

@sjakobi

This comment has been minimized.

Contributor

sjakobi commented Jul 6, 2016

I see the Did not find .cabal file messages every time I build a project with a different (dev) version of stack.

Edit: I actually see these messages also for other commands, like stack path.

@DanBurton

This comment has been minimized.

Contributor

DanBurton commented Jul 11, 2016

I've seen these messages lately as well, seemingly related to using the latest stack in environments that don't have a ~/.stack directory yet. For example:

(Modified instructions from trying to reproduce this bug)

docker run -it danburton/stackage-build-server:2016-07-11
cd ~
stack unpack gi-gtk-3.0.4
cd gi-gtk-3.0.4
stack init --resolver nightly-2016-06-08
Right Nothing
Did not find .cabal file for text-show-3.2.2 with Git SHA of dd7d87942bbe93e92db8be47a37172ac12be63fa
Right Nothing
Did not find .cabal file for socks-0.5.5 with Git SHA of a4fe3dade53c652f5ef4d04218d17a989e8414e1
Right Nothing
Did not find .cabal file for servant-yaml-0.1.0.0 with Git SHA of 2ef4721a30ef0c982f26f67688cfa09a307a62fc
Right Nothing
Did not find .cabal file for servant-swagger-1.1 with Git SHA of a35531ce5ae19faf483b4a7c5ceb73b67b0612e6
Right Nothing
Did not find .cabal file for servant-lucid-0.7.1 with Git SHA of cf726c02fbee0a185c0b861fcd8cbf1863c7b414

Last month when I ran this with snoyberg/stackage:nightly, these things were not printed to the console.

@yacinehmito

This comment has been minimized.

yacinehmito commented Jul 27, 2016

I still experience the same bug.

@rainbyte

This comment has been minimized.

rainbyte commented Aug 4, 2016

I'm experiencing the same issue inside a docker container:

When I run the cmd

stack install --stack-root $PWD/.stack-root/ --work-dir .stack-work-docker/ --local-bin-path artifacts/

It shows this

Downloading lts-6.10 build plan ...
Downloaded lts-6.10 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...
Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Populating index cache ...
Populated index cache.
Did not find .cabal file for zot-0.0.3 with Git SHA of f9092bae03d0044d2b4e3f33609c9544e8c46b19
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-lens-0.1.2.1 with Git SHA of b5797267b30c16afcccbd42fa0ce4b9fa58a7ca9
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)

And after various similar lines, I get

Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for AC-Vector-2.3.2 with Git SHA of 32d97a60969fe6912b33597e7399c407b5f2e39a
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
/src/.stack-work-docker/downloaded/9rVosC0PeQVv/: getDirectoryContents: does not exist (No such file or directory)
@lehtoj

This comment has been minimized.

lehtoj commented Aug 4, 2016

I am new to Haskell and today I attempted to install Haskell for Windows using Stack. However, I get similar errors as in this issue. Number of errors is too long to be copied here, so I provide you with a link instead.

While I have clearly run stack install outside a project, I get same errors on the first time I run stack setup.

Version 1.1.2, Git revision c6dac65 (3647 commits) x86_64 hpack-0.14.0
Downloaded from https://www.stackage.org/stack/windows-x86_64-installer

@mgsloan mgsloan modified the milestones: P1: Must, P2: Should Aug 4, 2016

@mgsloan

This comment has been minimized.

Collaborator

mgsloan commented Aug 4, 2016

I was working on getting https://github.com/lukexi/rumpus built on windows yesterday, and also ran into this. I think it just happens when you don't have git. We need to identify when we shouldn't expect to find any SHAs for snapshot packages.

@mgsloan mgsloan self-assigned this Aug 4, 2016

@Blaisorblade

This comment has been minimized.

Collaborator

Blaisorblade commented Aug 5, 2016

What I got from looking at the code and hadn't got from reading here: master won't print Right Nothing or Left ..., but that fix is not yet released.
The "Did not find .cabal file" message is still there and not fully explained.

@lehtoj

This comment has been minimized.

lehtoj commented Aug 5, 2016

Another observation is that when I use Stack to download contents of stack_root, mirror is https://s3.amazonaws.com/hackage.fpcomplete.com, size ends up being 262MB. But when someone else in another country uses Stack, download mirror is Github and contents end up being 451MB. Is the size difference explained by missing cabal/SHA files, or is there something else amiss?

@Blaisorblade

This comment has been minimized.

Collaborator

Blaisorblade commented Aug 5, 2016

@KeeperB5 interesting catch! Do you have the two actual URLs for one to look at?
If mirrors aren't mirrors, debugging this issue isn't going to be fun.

@lehtoj

This comment has been minimized.

lehtoj commented Aug 5, 2016

Well, my index is fetched from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz, but I believe the GitHub mirror had https://github.com/commercialhaskell/all-cabal-hashes.

How does Stack pick a mirror and do we have any way to manually select a mirror?

@sjakobi

This comment has been minimized.

Contributor

sjakobi commented Aug 9, 2016

I've had the "Did not find .cabal file" message pop up in the middle of a build while trying to reproduce #2422:

~/t/postgres-tmp> stack build --trace
...
attoparsec-0.13.0.2: copy/register
Did not find .cabal file for aeson-0.11.2.0 with Git SHA of 2057e715a488cd9d2456fdb1c6c4b0d6142c539e
aeson-0.11.2.0: configure
aeson-0.11.2.0: build
aeson-0.11.2.0: copy/register
...

I can't rule out that I had just started using another stack version in another thread.

@mgsloan:

I was working on getting https://github.com/lukexi/rumpus built on windows yesterday, and also ran into this. I think it just happens when you don't have git.

Are you saying you got the "Right Nothing" lines with stack-HEAD? Or do you only see the "Did not find .cabal file" lines?

The "Did not find .cabal file" lines certainly appear on Linux with git installed.

@mgsloan mgsloan removed their assignment Aug 9, 2016

@Blaisorblade

This comment has been minimized.

Collaborator

Blaisorblade commented Aug 9, 2016

Further experiments (below) suggest the hashes are part of the snapshot, which makes sense to ensure reproducibility. Hence, we get a log entry for each cabal file in the snapshot considered which was updated since then. Furthermore, I understand the fallback uses a newer cabal file, limiting build reproducibility in the sense of #2217.

Under these circumstances, I'd propose to clone the entirety of all-cabal-hashes as a quick fix (currently 215M). This is not fine long-term since bandwidth is limited and since the repo will grow indefinitely, but a better fix wouldn't be immediate.

I imagine one could add tags for each snapshot, and stack could fetch those tags when needed, but the tags would be needed remotely.

Experiment Results

Using lts-6.0, we request for instance a package at revision 5, while git log reveals we're at revision 7.
Trying out the snapshot in the OP shows the same hash (revision 6) for servant-yaml, while we're now at revision 8. Generally, all old messages reappear, and there are new ones.

Did not find .cabal file for opaleye-0.4.2.0 with Git SHA of 9b5608ac701b60a82f5cf985887d0d1450757864
Right Nothing
$ git show 9b5608ac701b60a82f5cf985887d0d1450757864|grep x-revi
x-revision: 5
$ git log -p current-hackage -- opaleye/0.4.2.0/opaleye.cabal
[...]
-x-revision: 6
+x-revision: 7
@Blaisorblade

This comment has been minimized.

Collaborator

Blaisorblade commented Aug 9, 2016

@rainbyte In your new issue, can you also add whether stack update fixes it?
I've just gotten that after rm -rf ~/.stack/indices/Hackage/git-update—somehow, stack did not figure out it needed to refetch the repo.

@Blaisorblade

This comment has been minimized.

Collaborator

Blaisorblade commented Aug 10, 2016

@KeeperB5: mirroring turns out to not be the problem, but here are some answers.
But since you were installing from scratch on Windows, here's the short of it:
TL;DR.: Install Git and ensure it is on the PATH so Stack can use it to update the index faster. Otherwise, Stack will fall back to downloading an index snapshot via HTTP from scratch each time.

Below's more info than you'd want, saving it for me.

Well, my index is fetched from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz, but I believe the GitHub mirror had https://github.com/commercialhaskell/all-cabal-hashes.

Those are the correct URLs, and they're related but not exactly mirrors, so the size difference is expected. https://github.com/commercialhaskell/all-cabal-hashes has extra metadata on top of https://github.com/commercialhaskell/all-cabal-files, which is a mirror in Git format of the tarball (which is a mirror of the Hackage index on the Amazon cloud).

How does Stack pick a mirror and do we have any way to manually select a mirror?

Basically, mirroring is not geographically based (except whatever optimizations Amazon does internally in S3). Make sure your stack can find your Git and you don't change your config, and stack will use git to update the index incrementally and faster (since you can download only the changes rather than a new snapshot), as described in these docs.

The relevant config is described here, https://docs.haskellstack.org/en/latest/yaml_configuration/#package-indices, but I'd avoid changing it usually.

With the default config, on both master and 1.1.2, the HTTP download (which is slower) is not used and the Git one is preferred, as long as Git is installed. So either stack doesn't find your Git or you changed your config.

Sources

Updating logic:

updateIndex menv index =
do let name = indexName index
logUpdate mirror = $logSticky $ "Updating package index " <> indexNameText (indexName index) <> " (mirrored at " <> mirror <> ") ..."
git <- isGitInstalled menv
case (git, indexLocation index) of
(True, ILGit url) -> logUpdate url >> updateIndexGit menv name index url
(True, ILGitHttp url _) -> logUpdate url >> updateIndexGit menv name index url
(_, ILHttp url) -> logUpdate url >> updateIndexHTTP name index url
(False, ILGitHttp _ url) -> logUpdate url >> updateIndexHTTP name index url
(False, ILGit url) -> logUpdate url >> throwM (GitNotAvailable name)

Default config (matching docs):

stack/src/Stack/Config.hs

Lines 219 to 228 in aac2c81

configPackageIndices = fromFirst
[PackageIndex
{ indexName = IndexName "Hackage"
, indexLocation = ILGitHttp
"https://github.com/commercialhaskell/all-cabal-hashes.git"
"https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz"
, indexDownloadPrefix = "https://s3.amazonaws.com/hackage.fpcomplete.com/package/"
, indexGpgVerify = False
, indexRequireHashes = False
}]

@lehtoj

This comment has been minimized.

lehtoj commented Aug 10, 2016

I've been using Visual Studio and SourceTree which don't need Git to be installed separately. Anyhow, I installed Git for Windows, deleted my stack_root directory, ran stack setup. Now it seems to work as intended, no more errors.

Perhaps install documentation should mention requirement for Git?

@Blaisorblade

This comment has been minimized.

Collaborator

Blaisorblade commented Aug 10, 2016

Perhaps install documentation should mention requirement for Git?

#2469

I've been using Visual Studio and SourceTree which don't need Git to be installed separately. Anyhow, I installed Git for Windows, deleted my stack_root directory, ran stack setup. Now it seems to work as intended, no more errors.

Note these were warnings. Should you get further warnings of this kind, you can mostly ignore them—and we'll have a fix by next release.
The real advantage with Git is that index updates will be much faster.

@lehtoj

This comment has been minimized.

lehtoj commented Aug 10, 2016

Noted. :) But the log does not really specify either way, Usually "The system cannot find the path specified." is treated as an error, so I just followed the suit since I didn't know better.

mgsloan added a commit that referenced this issue Aug 12, 2016

When marking exe installed, remove old vers #2175
+ Ignore installed exe info when it's ambiguous

Blaisorblade added a commit that referenced this issue Aug 13, 2016

Stop truncating all-cabal-hashes repo
Fix #2175.

* Fetch full history of tags (in particular, `current-hackage`).
* Before fetching tags, transition previously shallow repos to be
  non-shallow with `fetch --unshallow`.
* Fetch full history in initial clone, otherwise, we immediately
  afterwards use `fetch --unshallow`.

This means that the initial fetch and later updates require more data,
proportional to the entire repository history; however, reducing data
usage again is not trivial and would require changes in the layout of
all-cabal-hashes, as discussed in #2175.

Blaisorblade added a commit that referenced this issue Aug 13, 2016

@borsboom borsboom removed the in progress label Aug 13, 2016

@snoyberg snoyberg removed the in progress label Aug 13, 2016

@wiz

This comment has been minimized.

wiz commented Sep 16, 2016

Receiving objects: 0% (3467/576096), 1.97 MiB | 19.00 KiB/s

Ouch! I never experienced the problems that prompted this fix but can't proceed now.
You'd better add something like --do-no-harm or --fix-2175 next time. Until then, gotta return to previous stack... 😔

@Blaisorblade

This comment has been minimized.

Collaborator

Blaisorblade commented Sep 16, 2016

@wiz Is that your usual connection speed?

@wiz

This comment has been minimized.

wiz commented Sep 19, 2016

I hope not. It downloaded eventually, but in a pinch that got me nervous.

@wiz

This comment has been minimized.

wiz commented Oct 20, 2016

Now there's real pain. My CI have this restricted uplink and should start from scratch each time 😔

@sjakobi

This comment has been minimized.

Contributor

sjakobi commented Oct 20, 2016

@wiz: Can't you cache the ~/.stack directory?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment