New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Global cache issues #211

Closed
paulcsmith opened this Issue Jun 15, 2018 · 8 comments

Comments

Projects
None yet
4 participants
@paulcsmith

paulcsmith commented Jun 15, 2018

I just updated to 0.25 of Crystal which uses shards 0.8 with the new global cache.

The shard cache failed to update under these circumstances:

  • Defined a shard (lucky_cli) in shard.yml using version
  • Then changed the definition to use branch: master
  • Ran shards update
  • Then made some changes to the lucky_cli
  • Ran shards update
  • Cache still used the old version of branch: master instead of updating

Temporary fix

rm -rf ~/.cache/shards and re-run `shards update.

Possible fix

  • When running shards update always update cache if using branch or tag?
@kostya

This comment has been minimized.

kostya commented Jun 15, 2018

oh, i just try to find why my shard not updated, rm -rf ~/.cache/shards helps. it not update even for version not for master only. (after update crystal and shard to 0.8.0)

@paulcsmith paulcsmith changed the title from 'branch: master' and global cache issues to Global cache issues Jun 15, 2018

@wontruefree

This comment has been minimized.

wontruefree commented Jun 15, 2018

It seems like if you are using a branch for the version it should always have a cache miss.
It seems like something would have to be added here https://github.com/crystal-lang/shards/blob/master/src/commands/check.cr#L41 to always return false for installed.

@kostya

This comment has been minimized.

kostya commented Jun 15, 2018

dependencies:
  myhtml:
    github: kostya/myhtml
    version: ">= 0.29"
shards update
Fetching https://github.com/kostya/myhtml.git
Error resolving myhtml (>= 0.29)

but 0.29 there is, i just push it.
i think this is quite critical issue.

rm -rf ~/.cache/shards
shards update
Fetching https://github.com/kostya/myhtml.git
Installing myhtml (0.29)
Postinstall cd src/ext && make package
@ysbaddaden

This comment has been minimized.

Member

ysbaddaden commented Jun 16, 2018

Sigh. Can someone reproduce and run shards commands with --verbose, look at the global cache structure, and start investigating?

I won't be able to look at this anytime soon.

@ysbaddaden

This comment has been minimized.

Member

ysbaddaden commented Jun 16, 2018

It's probably not the global cache, but not mirroring the Git repository anymore. A simple --bare and fetch isn't enough to pull Git tags. Without them, Shards is incapable to resolve versions.

@ysbaddaden

This comment has been minimized.

Member

ysbaddaden commented Jun 16, 2018

If you type git commands in the cached git folder, you probably won't see the new tags, even after a shards update.

@kostya

This comment has been minimized.

kostya commented Jun 16, 2018

shards update --verbose
git ls-remote --get-url origin
Fetching https://github.com/kostya/stuffs.git
git fetch --all --quiet
git tag --list --column=never
versions: 0.1
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
Installing stuffs (0.1)
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git archive --format=tar --prefix= v0.1 | tar -x -f - -C '/Users/kostya/projects/blah5/lib/stuffs'
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml

after pushing v0.2

shards update --verbose
git ls-remote --get-url origin
Fetching https://github.com/kostya/stuffs.git
git fetch --all --quiet
git tag --list --column=never
versions: 0.1
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
Using stuffs (0.1)
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml

after rm -rf ~/.cache/shards/

shards update --verbose
Fetching https://github.com/kostya/stuffs.git
git clone --bare --quiet -- 'https://github.com/kostya/stuffs.git' /Users/kostya/.cache/shards/github.com/kostya/stuffs.git
git tag --list --column=never
versions: 0.2, 0.1
git ls-remote --get-url origin
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git show v0.2:shard.yml
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git show v0.2:shard.yml
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git show v0.2:shard.yml
Installing stuffs (0.2)
rm -rf '/Users/kostya/projects/blah5/lib/stuffs'
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git archive --format=tar --prefix= v0.2 | tar -x -f - -C '/Users/kostya/projects/blah5/lib/stuffs'
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git show v0.2:shard.yml
@kostya

This comment has been minimized.

kostya commented Jun 17, 2018

i just test it, it works if replace --bare with --mirror (from https://stackoverflow.com/questions/3382679/how-do-i-update-my-bare-repo).
Git fetch not update in bare repo, but update in mirror repo.

ysbaddaden added a commit that referenced this issue Jun 17, 2018

ysbaddaden added a commit that referenced this issue Jun 17, 2018

Fix: Git resolver doesn't pull tags anymore
Creating bare Git repositories doesn't configure the local
repository to fetch remote refs, such as branches and tags, which
totally broke the update command.

This patch reverts back to mirroring Git repositories, and verifies
whether a cloned repository is valid, or not before fetching new
refs.

fixes #211

@ysbaddaden ysbaddaden self-assigned this Jun 17, 2018

@ysbaddaden ysbaddaden added the bug label Jun 17, 2018

@ysbaddaden ysbaddaden added this to the v0.8.1 milestone Jun 17, 2018

ysbaddaden added a commit that referenced this issue Jun 17, 2018

Fix: Git resolver doesnt pull tags anymore (#212)
Creating bare Git repositories doesn't configure the local
repository to fetch remote refs, such as branches and tags, which
totally broke the update command.

This patch reverts back to mirroring Git repositories, and verifies
whether a cloned repository is valid, or not before fetching new
refs.

fixes #211
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment