New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use transitive dependency's version as top-level version #1236

Open
benmai opened this Issue Oct 4, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@benmai

benmai commented Oct 4, 2017

What version of dep are you using (dep version)?

v0.3.1

What dep command did you run?

cd $GOPATH/src/github.com/benmai/dep-test-c && dep init

What did you expect to see?

I've set up three example repositories, benmai/dep-test-a, benmai/dep-test-b, benmai/dep-test-c to demonstrate this:

  • dep-test-a has no dependencies and two revisions, bfe5ef (the earlier revision) and 790c1e4 (the later revision).
  • dep-test-b imports dep-test-a and has a constraint to set its version of dep-test-a to bfe5ef (the earlier revision).
  • dep-test-c imports dep-test-a and dep-test-b. dep-test-c's Gopkg.toml has not yet been created, since we're running dep init now.
   A
 / |
B  |
 \ |
   C

I expect to see:

  • dep init to finish without errors.
  • Gopkg.toml to be empty (the dependencies should be inferred from imports).
  • Gopkg.lock to contain:
    • Lock of dep-test-a at bfe5ef (the earlier revision).
    • Lock of dep-test-b at 7a70e4a (its only revision).

What did you see instead?

No versions of github.com/benmai/dep-test-b met constraints:
	master: Could not introduce github.com/benmai/dep-test-b@master, as it has a dependency on github.com/benmai/dep-test-a with constraint bfe5ef24b86ad8946c91275fcde1739f0d4bf12b, which does not allow the currently selected version of master

It appears that since dep-test-c imports dep-test-a, dep init begins by forcing dep-test-a to the tip of master instead of deferring to dep-test-b's constraint.

Is this expected behavior? It seems I would have to create a Gopkg.toml by hand here, which I don't want to do. I would like to use the version of dep-test-a which dep-test-b specified in its Gopkg.toml by default.

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Oct 4, 2017

Member

hi, welcome! thanks for the detailed the issue.

i think this might be a small solver bug. we sometimes have a bit of a blind spot around plain revisions in Gopkg.toml, as we generally discourage it as an approach. let's quickly validate before i say more - push a tag (doesn't matter what it's called) to the tip of dep-test-a, then try the same dep init again, and see if it magically works.

Member

sdboyer commented Oct 4, 2017

hi, welcome! thanks for the detailed the issue.

i think this might be a small solver bug. we sometimes have a bit of a blind spot around plain revisions in Gopkg.toml, as we generally discourage it as an approach. let's quickly validate before i say more - push a tag (doesn't matter what it's called) to the tip of dep-test-a, then try the same dep init again, and see if it magically works.

@benmai

This comment has been minimized.

Show comment
Hide comment
@benmai

benmai Oct 4, 2017

@sdboyer Thanks for the quick response!

I added a tag on dep-test-a called ok-version for commit bfe5ef, and your hypothesis is proven correct:

$ dep init
Locking in  (bfe5ef2) for direct dep github.com/benmai/dep-test-a
Using master as constraint for direct dep github.com/benmai/dep-test-b
Locking in master (7a70e4a) for direct dep github.com/benmai/dep-test-b

Additional questions I have now:

  • Why does dep-test-b but not dep-test-a show up as a dependency in dep-test-c's Gopkg.toml by default? They are both dependencies of dep-test-c.
  • Why would adding a tag fix this while I'm still using a commit hash in dep-test-b's Gopkg.toml?

And most importantly:

  • In the real-life case I'm dealing with, the equivalent of dep-test-a is not in my org, but dep-test-b and dep-test-c are. I want to be able to set an org-wide dependency on a specific version of dep-test-a, but that repository doesn't use tags. What do you recommend?

Also, even though I'm digging into this revision issue, I really do appreciate that dep discourages using specific hashes, and I think that's a great approach. It sets a good tone for dependency hygiene in the community. 👍

benmai commented Oct 4, 2017

@sdboyer Thanks for the quick response!

I added a tag on dep-test-a called ok-version for commit bfe5ef, and your hypothesis is proven correct:

$ dep init
Locking in  (bfe5ef2) for direct dep github.com/benmai/dep-test-a
Using master as constraint for direct dep github.com/benmai/dep-test-b
Locking in master (7a70e4a) for direct dep github.com/benmai/dep-test-b

Additional questions I have now:

  • Why does dep-test-b but not dep-test-a show up as a dependency in dep-test-c's Gopkg.toml by default? They are both dependencies of dep-test-c.
  • Why would adding a tag fix this while I'm still using a commit hash in dep-test-b's Gopkg.toml?

And most importantly:

  • In the real-life case I'm dealing with, the equivalent of dep-test-a is not in my org, but dep-test-b and dep-test-c are. I want to be able to set an org-wide dependency on a specific version of dep-test-a, but that repository doesn't use tags. What do you recommend?

Also, even though I'm digging into this revision issue, I really do appreciate that dep discourages using specific hashes, and I think that's a great approach. It sets a good tone for dependency hygiene in the community. 👍

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Oct 7, 2017

Member

i do love guessing right 😄

Why does dep-test-b but not dep-test-a show up as a dependency in dep-test-c's Gopkg.toml by default? They are both dependencies of dep-test-c.

This is another judgment call we made in pursuit of that same goal - not having revisions in Gopkg.toml. Given the number of existing tools that supported, or even required, the use of SHA1s in their metadata files, we were concerned that such values into Gopkg.toml by default would just cause people to never question those habits, even though it's a deeply suboptimal way of doing things. And, because the same reproducibility assurances are achieved through Gopkg.lock, we weren't really losing anything by doing it.

Why would adding a tag fix this while I'm still using a commit hash in dep-test-b's Gopkg.toml?

The detailed answer reaches into the depths of dep's solver, and I don't know where the problem is exactly (if I did, we wouldn't have this bug!), as it's a subtle order-of-operations issue. But, the problem arises from solver mechanics: when the solver is trying to figure out which version of a dependency to use, it creates an ordered list of versions to try.

That list does not, by default, contain revisions, because that would mean the solver would have to search every revision in every dependency to see if they work. Instead, when we encounter a revision constraint, we hack that version in to the list at the front. My hunch was that the problem arose when that hack is applied on an already-empty (or perhaps just a single-item) queue, as it's basically the only time that we mess with the version list after initial construction (like I said, it's a hack), so it was the only plausible way that i could imagine the solver missing a version it needed to try.

And that's why adding another to dep-test-a fixed the problem - the queue was not exhausted, or whatever, by the time the solver arrived at the new revision constraint it wanted to apply.

In the real-life case I'm dealing with

Apart from waiting for a fix, which may take me a little while to get to, I think your best bet is likely to be picking out the revision of dep-test-a that you know dep-test-b will eventually want, and hand-populating it into Gopkg.lock. Having a revision in Gopkg.lock already causes that revision to be the first item in the version queue, and obviate the need for this entire code path. That may mean you can't use dep init to set up your projects at first, unfortunately.

Alternatively, ask the maintainers of dep-test-a if they might be willing to push a dummy branch or non-semver tag (it could just point to the initial commit of the repo).

Also, even though I'm digging into this revision issue, I really do appreciate that dep discourages using specific hashes, and I think that's a great approach. It sets a good tone for dependency hygiene in the community.

i'm glad you agree! 🎉 🎉

Member

sdboyer commented Oct 7, 2017

i do love guessing right 😄

Why does dep-test-b but not dep-test-a show up as a dependency in dep-test-c's Gopkg.toml by default? They are both dependencies of dep-test-c.

This is another judgment call we made in pursuit of that same goal - not having revisions in Gopkg.toml. Given the number of existing tools that supported, or even required, the use of SHA1s in their metadata files, we were concerned that such values into Gopkg.toml by default would just cause people to never question those habits, even though it's a deeply suboptimal way of doing things. And, because the same reproducibility assurances are achieved through Gopkg.lock, we weren't really losing anything by doing it.

Why would adding a tag fix this while I'm still using a commit hash in dep-test-b's Gopkg.toml?

The detailed answer reaches into the depths of dep's solver, and I don't know where the problem is exactly (if I did, we wouldn't have this bug!), as it's a subtle order-of-operations issue. But, the problem arises from solver mechanics: when the solver is trying to figure out which version of a dependency to use, it creates an ordered list of versions to try.

That list does not, by default, contain revisions, because that would mean the solver would have to search every revision in every dependency to see if they work. Instead, when we encounter a revision constraint, we hack that version in to the list at the front. My hunch was that the problem arose when that hack is applied on an already-empty (or perhaps just a single-item) queue, as it's basically the only time that we mess with the version list after initial construction (like I said, it's a hack), so it was the only plausible way that i could imagine the solver missing a version it needed to try.

And that's why adding another to dep-test-a fixed the problem - the queue was not exhausted, or whatever, by the time the solver arrived at the new revision constraint it wanted to apply.

In the real-life case I'm dealing with

Apart from waiting for a fix, which may take me a little while to get to, I think your best bet is likely to be picking out the revision of dep-test-a that you know dep-test-b will eventually want, and hand-populating it into Gopkg.lock. Having a revision in Gopkg.lock already causes that revision to be the first item in the version queue, and obviate the need for this entire code path. That may mean you can't use dep init to set up your projects at first, unfortunately.

Alternatively, ask the maintainers of dep-test-a if they might be willing to push a dummy branch or non-semver tag (it could just point to the initial commit of the repo).

Also, even though I'm digging into this revision issue, I really do appreciate that dep discourages using specific hashes, and I think that's a great approach. It sets a good tone for dependency hygiene in the community.

i'm glad you agree! 🎉 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment