Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

shards keeps downloading dependencies #288

Closed
peterhoeg opened this issue Jul 10, 2019 · 9 comments · Fixed by #315
Closed

shards keeps downloading dependencies #288

peterhoeg opened this issue Jul 10, 2019 · 9 comments · Fixed by #315

Comments

@peterhoeg
Copy link

Running shards install takes a very long time on a project with a number of dependencies, as shards keep refetching the same sources.

Here's an example from a lucky project (just an example, but could be anything really):

$ shards install
  Resolving dependencies                                                                                                                                          [19/9143]
  Fetching https://github.com/luckyframework/lucky.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/mosop/teeplate.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/avram.git
  Fetching https://github.com/kostya/blank.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/mosop/teeplate.git
  Fetching https://github.com/mosop/teeplate.git
  Fetching https://github.com/mosop/teeplate.git
  Fetching https://github.com/mosop/teeplate.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/will/crystal-pg.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/luckyframework/lucky_router.git
  Fetching https://github.com/luckyframework/shell-table.cr.git
  Fetching https://github.com/paulcsmith/cry.git
  Fetching https://github.com/mosop/cli.git
  Fetching https://github.com/mosop/optarg.git
  Fetching https://github.com/mosop/callback.git
  Fetching https://github.com/mosop/string_inflection.git
  Fetching https://github.com/mosop/string_inflection.git
  Fetching https://github.com/crystal-loot/exception_page.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/avram.git
  Fetching https://github.com/luckyframework/lucky_router.git
  Fetching https://github.com/luckyframework/shell-table.cr.git
  Fetching https://github.com/paulcsmith/cry.git
  Fetching https://github.com/crystal-loot/exception_page.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/luckyframework/authentic.git
  Fetching https://github.com/luckyframework/lucky.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/mosop/teeplate.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/avram.git
  Fetching https://github.com/kostya/blank.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/will/crystal-pg.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/kostya/blank.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/will/crystal-pg.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/kostya/blank.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/will/crystal-pg.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/kostya/blank.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/will/crystal-pg.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/kostya/blank.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/crystal-lang/crystal-db.git
  Fetching https://github.com/will/crystal-pg.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/luckyframework/lucky_router.git
  Fetching https://github.com/luckyframework/shell-table.cr.git
  Fetching https://github.com/paulcsmith/cry.git
  Fetching https://github.com/crystal-loot/exception_page.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/luckyframework/lucky_cli.git
  Fetching https://github.com/luckyframework/habitat.git
  Fetching https://github.com/luckyframework/wordsmith.git
  Fetching https://github.com/luckyframework/avram.git
  Fetching https://github.com/luckyframework/lucky_router.git
  Fetching https://github.com/luckyframework/shell-table.cr.git
  Fetching https://github.com/paulcsmith/cry.git
  Fetching https://github.com/crystal-loot/exception_page.git
  Fetching https://github.com/luckyframework/dexter.git
  Fetching https://github.com/luckyframework/lucky_cli.git

And so on and so on until eventually it finishes. But it takes ages.

shards 0.9.0 and crystal 0.29.0 but also experienced on older versions
@peterhoeg
Copy link
Author

A few more things.

~/.cache/shards is being populated but running with --local fails:

$ shards install --local
Resolving dependencies
Missing repository cache for "lucky". Please run without --local to fetch it.

@ysbaddaden
Copy link
Contributor

There is something wrong with either the shard.yml or your disk, because the same set of dependencies is being fetched again and again in a loop.

@peterhoeg
Copy link
Author

peterhoeg commented Jul 10, 2019 •
edited
Loading

It's not a never-ending loop though - for whatever reason it is just processing the various dependencies several times . Here is a much smaller example:

$ crystal init app woot
...
$ cd woot
$ cat shards.yml
$ cat shard.yml
name: woot
version: 0.1.0

authors:
  - Peter Hoeg <peter@hoeg.com>

targets:
  woot:
    main: src/woot.cr

crystal: 0.29.0

dependencies:
  emoji:
    github: veelenga/emoji.cr
    branch: master

license: MIT
$ shards install
Resolving dependencies
Fetching https://github.com/veelenga/emoji.cr.git
Installing emoji (0.3.1 at 22c147f37333a94249eac7ce54e7e9f1c4999838)
Fetching https://github.com/veelenga/emoji.cr.git
Fetching https://github.com/veelenga/emoji.cr.git
Writing shard.lock

So it finishes, but it fetches the emoji repo 3 times.

@ysbaddaden
Copy link
Contributor

I can't reproduce.

From your logs, I guess the repository is cloned over and over and never put in the cache. Run shards install --verbose and you'll see numerous git clone ... commands.

My guess is you have a problem on your disk or your shard's cache folder. There should be a github.com/veelenga/emoji.cr.git folder with a proper Git bare clone. Shards' cache is either at:

  • $SHARDS_CACHE_PATH
  • $XDG_CACHE_HOME/shards
  • ~/.cache/shards
  • ~/.cache/.shards
  • .shards

@peterhoeg
Copy link
Author

The shards command removes the repository again:

$ shards install --verbose
Resolving dependencies
Fetching https://github.com/veelenga/emoji.cr.git
git clone --mirror --quiet -- 'https://github.com/veelenga/emoji.cr.git' /home/peter/.cache/shards/github.com/veelenga/emoji.cr.git
git show master:shard.yml
git log -n 1 --pretty=%H master
git show 22c147f37333a94249eac7ce54e7e9f1c4999838:shard.yml
Installing emoji (0.3.1 at 22c147f37333a94249eac7ce54e7e9f1c4999838)
git ls-remote --get-url origin
rm -rf '/home/peter/.cache/shards/github.com/veelenga/emoji.cr.git'
Fetching https://github.com/veelenga/emoji.cr.git
git clone --mirror --quiet -- 'https://github.com/veelenga/emoji.cr.git' /home/peter/.cache/shards/github.com/veelenga/emoji.cr.git
rm -rf ''/home/peter/src/active/woot/peter/lib/emoji''
git ls-tree -r --full-tree --name-only 22c147f37333a94249eac7ce54e7e9f1c4999838 -- shard.yml
git archive --format=tar --prefix= 22c147f37333a94249eac7ce54e7e9f1c4999838 | tar -x -f - -C '/home/peter/src/active/woot/peter/lib/emoji'
git log -n 1 --pretty=%H 22c147f37333a94249eac7ce54e7e9f1c4999838
Link /home/peter/src/active/woot/peter/lib to /home/peter/src/active/woot/peter/lib/emoji/lib
git ls-remote --get-url origin
rm -rf '/home/peter/.cache/shards/github.com/veelenga/emoji.cr.git'
Fetching https://github.com/veelenga/emoji.cr.git
git clone --mirror --quiet -- 'https://github.com/veelenga/emoji.cr.git' /home/peter/.cache/shards/github.com/veelenga/emoji.cr.git
git ls-tree -r --full-tree --name-only 22c147f37333a94249eac7ce54e7e9f1c4999838 -- shard.yml
git show 22c147f37333a94249eac7ce54e7e9f1c4999838:shard.yml
Writing shard.lock

@peterhoeg
Copy link
Author

Ahhhh, I know why

@peterhoeg
Copy link
Author

I have this in ~/.gitconfig:

[url "git@github.com:"]
	insteadOf = git://github.com/
	insteadOf = https://github.com/

So that makes origin_changed? return true and the repository gets deleted.

@peterhoeg
Copy link
Author

I'm thinking that the protocol shouldn't matter - just the host and path. If you agree I'll prep a PR.

@ysbaddaden
Copy link
Contributor

I suppose it's acceptable.

@remydev remydev mentioned this issue Jul 11, 2019
Closed
@kalafut kalafut mentioned this issue Jan 20, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update origin_changed? to ignore protocol differences
2 participants
@ysbaddaden @peterhoeg