[WIP] dep: Enable importing other tools and add integration tests #1277

ChinmayR · 2017-10-17T00:27:43Z

Turned off skip tools flag by default
Added glide integration tests for the init case
Move removing transitive dependencies from the imported manifest to
finalize manifest stage to avoid ignoring some constraints when
importing metadata from other tools

What does this do / why do we need it?

Enables dep init to pull data from other tools, not only from direct but also transitive dependencies.

What should your reviewer look out for in this PR?

Moving a.removeTransitiveDependencies(...) from import manifest to finalize manifest. The trans-trans-conflict test fails without this change.

Do you need help or clarification on anything?

Is this a valid approach or should we be adding imported transitive constraints as overrides in the root manifest?

Which issue(s) does this PR fix?

Fixes the dep init part, still need to tackle ensure.
fixes #821
fixes #920

googlebot · 2017-10-17T00:27:47Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If your company signed a CLA, they designated a Point of Contact who decides which employees are authorized to participate. You may need to contact the Point of Contact for your company and ask to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the project maintainer to go/cla#troubleshoot.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again.

ChinmayR · 2017-10-17T18:29:20Z

I signed it!

googlebot · 2017-10-17T18:29:23Z

CLAs look good, thanks!

uberbits · 2017-10-18T19:00:41Z

@jmank88 it seems that TestGitSourceListVersionsNoDupes test is a flaky test due to git/github reliability issues. What is your recommendation to go about fixing it?

57bcc65#diff-e5840bdf34ab76e533df859b43e25eb3

Here is the log: https://ci.appveyor.com/project/golang/dep/build/2399

jmank88 · 2017-10-18T20:59:23Z

We generally just re-run the CI when a failure like that occurs, but this appears to have resolved itself for now.

ChinmayR · 2017-10-18T21:09:54Z

Hi @carolynvs , I wanted your input on whether or not this is the best way to solve this. I am afraid this might be going against how preferred versions work but I do not understand them enough to say for sure.

I was running into the same issue as this: #920 (review)

And solved it by not removing transitive dependencies when we import the manifest, but instead removing them during the finalize manifest step.

darkowlzz · 2017-10-20T12:04:39Z

CHANGELOG.md

@@ -40,7 +40,7 @@ WIP:

 * gps: Process canonical import paths. (#1017)
 * gps: Persistent cache. (#1127, #1215)
-
+* dep: Enable importing other tools and add integration tests


We just released v0.3.2. Please move this under v0.3.3.
And also add the PR #. 🙂

Updated the PR with the changes.

- Turned off skip tools flag by default - Added glide integration tests for the init case - Move removing transitive dependencies from the imported manifest to finalize manifest stage to avoid ignoring some constraints when importing metadata from other tools

uberbits · 2017-10-31T03:07:46Z

@carolynvs is there any chance you can review this diff? Thanks!

carolynvs · 2017-10-31T20:16:27Z

@ChinmayR Apologies for my slow response! I will have time to do a review tomorrow.

Also I talked with the other maintainers yesterday and created an epic (#1335) to cover all the changes that need to be made before we can "flip the switch" on importing during solve (as opposed to only during init).

Once this is ready, what I will probably end up doing is merging but not turning on the feature immediately, not until the other "safety features" are in as well.

carolynvs · 2017-11-03T23:00:31Z

@ChinmayR I won't have a chance to review until next week sorry! Life came up. 😞

carolynvs

Sorry this took so long!!! You've done a ton of great work that is setting us up well for the rest of this epic (#1335). So even though I may shuffle your test data around, it's really useful. Thank you for the extensive tests! 😀

carolynvs · 2017-11-08T13:51:17Z

cmd/dep/root_analyzer.go

@@ -165,6 +164,11 @@ func (a *rootAnalyzer) DeriveManifestAndLock(dir string, pr gps.ProjectRoot) (gp
 }

 func (a *rootAnalyzer) FinalizeRootManifestAndLock(m *dep.Manifest, l *dep.Lock, ol dep.Lock) {
+	// Transitive dependencies could sneak into the manifest when other importers are used


There is another open issue to appropriately handle imported constraints during solve: #1316. Would you please back out this change?

Since #1316 is going to impact many of your test cases, and you have a great set of data, if backing out this change breaks something, we can review and disable temporarily until #1316 is in.

We are looking to migrate folks over from glide to dep internally asap. I will remove it from this PR and we will use this code internally for now until #1316 is done. Please let me know if I can help in any way to expedite that issue. We can discuss how we want to tackle it and perhaps I can help with the changes?

I will add a comment of our need in #1316 , which is basically also needing to import transient dependencies of the root repo and add them to constraints to enable correct resolving.

carolynvs · 2017-11-08T15:23:17Z

cmd/dep/testdata/harness_tests/init/glide/corrupt-glide/initial/glide.lock

+hash: 16053c82a71f9bd509b05a4523df6bc418aed2083e4b8bd97a870bbc003256f8
+updated: 2017-03-07T17:02:32.214383898-06:00
+imports:
+- name: github.com/ChinmayR/deptestglideA


FYI, after this is merged into the feature branch, I am going to fork your test repo(s) and switch dep to point to that instead. dep has a policy of only relying on external repos that are managed by one of the maintainers, so that once this is in, you won't be on the hook to keep around the repos.

Thank you for building up a good set of test data! ❤️

Sounds good 👍

carolynvs · 2017-11-08T15:27:35Z

cmd/dep/testdata/harness_tests/init/glide/corrupt-glide/testcase.json

+  "commands": [
+    ["init", "-no-examples", "-v"]
+  ],
+  "error-expected": "v0.3.0: unable to parse"


FYI, I am going to keep this test case for now but once #1315 (which is also part of the epic) is implemented, it will probably be removed because the behavior is changing to not hard fail during solve. The testdata in your repo will be kept and used though, thank you for building that out! ❤️

carolynvs · 2017-11-08T15:32:06Z

cmd/dep/testdata/harness_tests/init/glide/direct-dep/README.md

@@ -0,0 +1 @@
+Take a direct dependency where the version is defined by glide.


Let's remove this since the scenario is already covered by glide/case1.

Missed that, will remove it.

carolynvs · 2017-11-08T16:03:43Z

cmd/dep/testdata/harness_tests/init/glide/direct-trans-conflict/README.md

@@ -0,0 +1 @@
+Take a direct dependency on a transient dependency where the versions are conflicted. Resolving should fail.


If this test starts failing after the other changes I have requested are made, let's ignore it test temporarily ( add a .ignore file extension to the testcase.json file). The scenario may require additional handling and will be addressed in another issue: #1316.

This test does not fail since the existing code only removes transient dependencies read from importers. Since there is also a direct dependency here, the test passes.

carolynvs · 2017-11-08T16:19:12Z

CHANGELOG.md

@@ -4,6 +4,10 @@ BUG FIXES:

 * Releases targeting Windows now have a `.exe` suffix (#1291).

+WIP:
+
+* dep: Enable importing other tools and add integration tests (#1277)


Let's change this to Enable importing external configuration from dependencies during init, so that the scope of the change is more clear.

That is a better comment, updated it.

carolynvs · 2017-11-08T18:41:05Z

cmd/dep/testdata/harness_tests/init/glide/direct-trans-no-conflict/README.md

@@ -0,0 +1 @@
+Take a direct dependency on a transient dependency where the versions are not conflicted. Resolving should pass.


If this test starts failing after the other changes I have requested are made, let's ignore it test temporarily ( add a .ignore file extension to the testcase.json file). The scenario may require additional handling and will be addressed in another issue: #1316.

This test does not fail since the existing code only removes transient dependencies read from importers. Since there is also a direct dependency here, the test passes.

carolynvs · 2017-11-08T18:50:12Z

cmd/dep/testdata/harness_tests/init/glide/trans-trans-conflict/README.md

@@ -0,0 +1 @@
+Have two transient dependencies have different versions of the same repo. Resolving should fail.


This test is asserting an unwanted behavior that will be addressed in #1316. But it's a useful test case for that other story, so let's not remove it. Would you please add a .ignore to the testcase.json, and when we implement the other issue, we'll turn this test on, with the correct expected results (i.e. it should not fail).

This test starts to fail since we do not import transient dependencies as constraints of the root repo. This results in incorrect resolving since only constraints that are imported are of direct dependencies of the root. I will add a comment to #1316 to also handle this case explicitly. Shouldn't the resolving for this test fail when the test case is enabled again as part of #1316 ?

I believe preferred versions handle this case but currently we have no idea of dealing with conflicting preferred versions.

Yeah, I'm a bit fuzzy off the top of my head on the topic of conflicting preferred versions. 😊 But that is a dep-wide question, not just for import, so I'd rather deal with it in that other issue and rope in Sam then.

carolynvs · 2017-11-08T18:51:55Z

cmd/dep/testdata/harness_tests/init/glide/case3/final/Gopkg.lock

@@ -14,8 +14,8 @@
  version = "v0.8.1"

 [solve-meta]
-  analyzer-name = "dep+import"
+  analyzer-name = "dep"


Doh! I wonder why this didn't get picked up sooner? Thanks for fixing!

ChinmayR · 2017-11-09T21:45:39Z

Thank you taking a look at this @carolynvs , glad to have it moving forward. I addressed the PR comments with some open questions.

We are looking to migrate folks over from glide to dep internally asap and #1316 is a hard blocker for us. I removed the fix from this PR but we will use this code internally for now until #1316 is done. Please let me know if I can help in any way to expedite that issue. We can discuss how we want to tackle it and perhaps I can help with the changes?

carolynvs · 2017-11-10T17:15:56Z

tip is failing but that is not related to this PR, and we are seeing that elsewhere. I would love help with getting this feature pushed through, most especially on the test and scenario identification end. I am starting on #1316 this weekend and we can build from there! ❤️

uberbits · 2017-11-10T17:43:21Z

@ChinmayR congrats! on your first accepted commit.

ChinmayR requested review from carolynvs and sdboyer as code owners October 17, 2017 00:27

googlebot added the cla: no label Oct 17, 2017

ChinmayR force-pushed the import-glide branch from 7f8ee8c to dd23d04 Compare October 17, 2017 00:53

googlebot added cla: yes and removed cla: no labels Oct 17, 2017

ChinmayR force-pushed the import-glide branch 2 times, most recently from b35247a to 23565ae Compare October 18, 2017 18:59

darkowlzz reviewed Oct 20, 2017

View reviewed changes

darkowlzz added this to the v0.3.3 milestone Oct 20, 2017

ChinmayR force-pushed the import-glide branch from 23565ae to 20c92c8 Compare October 20, 2017 22:20

ChinmayR force-pushed the import-glide branch from 20c92c8 to 97cf253 Compare October 20, 2017 22:23

ChinmayR and others added 2 commits October 24, 2017 17:54

Merge branch 'master' into import-glide

ebcbca4

Merge branch 'master' into import-glide

9379c0e

This was referenced Oct 31, 2017

[WIP] Import transitive constraints from glide #920

Closed

Import during solve #1335

Closed

carolynvs changed the base branch from master to import-during-solve November 8, 2017 15:01

carolynvs suggested changes Nov 8, 2017

View reviewed changes

carolynvs removed the request for review from sdboyer November 8, 2017 18:57

Merge branch 'master' into import-glide

4f64d31

Addressed PR comments

95be023

carolynvs approved these changes Nov 10, 2017

View reviewed changes

carolynvs merged commit e1f5c5f into golang:import-during-solve Nov 10, 2017

		@@ -0,0 +1 @@
		Take a direct dependency where the version is defined by glide.

		@@ -0,0 +1 @@
		Take a direct dependency on a transient dependency where the versions are conflicted. Resolving should fail.

		@@ -0,0 +1 @@
		Have two transient dependencies have different versions of the same repo. Resolving should fail.

[WIP] dep: Enable importing other tools and add integration tests #1277

[WIP] dep: Enable importing other tools and add integration tests #1277

Conversation

ChinmayR commented Oct 17, 2017

What does this do / why do we need it?

What should your reviewer look out for in this PR?

Do you need help or clarification on anything?

Which issue(s) does this PR fix?

googlebot commented Oct 17, 2017

ChinmayR commented Oct 17, 2017

googlebot commented Oct 17, 2017

uberbits commented Oct 18, 2017 • edited Loading

jmank88 commented Oct 18, 2017

ChinmayR commented Oct 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uberbits commented Oct 31, 2017

carolynvs commented Oct 31, 2017

carolynvs commented Nov 3, 2017

carolynvs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChinmayR commented Nov 9, 2017

carolynvs commented Nov 10, 2017

uberbits commented Nov 10, 2017

uberbits commented Oct 18, 2017 •

edited

Loading