Import glide configuration during init #500

carolynvs · 2017-05-02T14:36:33Z

During dep init if glide.yaml and glide.lock are present, then use the glide configuration when initializing dep's manifest and lock. Importing config from other dependency manager's will be the default, specify -skip-tools to ignore existing configuration.

Read glide.yaml and populate initial manifest constraints.
Hit the dependency repo to determine if a glide project version is a branch or tag.
Drop a dependency from the manifest if it's in glide.yaml but isn't imported.
Read glide.lock and populate initial lock revisions.
Unmarshal unsupported glide fields, detect if present and warn that they are being ignored.

sdboyer · 2017-05-02T15:40:52Z

First, yay!! Glad you're moving forward on this - I was actually just about to post something on one of the other issues 😄

I can't remember if I said this anywhere, but repeating just in case - the ideal place to attach this behavior would be as a method on the Analyzer - one that's compatible with the arguments to DeriveManifestAndLock().

sdboyer · 2017-05-03T02:20:00Z

the ideal place to attach this behavior would be as a method on the Analyzer - one that's compatible with the arguments to DeriveManifestAndLock().

Now, looking over the PR as-is - it's fine to have the bulk of the logic in internal, but the entry points should still be attached to the analyzer.

carolynvs · 2017-05-03T21:36:05Z

I've rebased to take in the gps merge. Tomorrow I'll move my hook out of init and into the Analyzer.

sdboyer · 2017-05-12T22:18:43Z

yay!

carolynvs · 2017-05-13T13:28:19Z

@sdboyer Can you point me in the right direction for looking up a version string like "master" or "v2" against a dependency's repo to determine if it's a branch/tag/version? I mostly am not sure if this lookup exists in some form or another already.

UPDATE: I found what I was looking for in the ensure command. I'll see if I can get the branch/tag/version deduction working tonight.

carolynvs · 2017-05-14T15:08:20Z

Hurray for progress! It now

checks the dependency repo to tell if a glide "version" is a branch/tag/version
uses the default branch when no version information is given in glide.yaml for a package.

Both of that is rolled into the same function used by ensure so that everyone (ensure, importers, etc) all benefit.

I think all that's left is:

housekeeping on the importer interface, going to bring it down to a single method
fix merge conflict with master
document how this works better so people know it exists. Right now just the helptext mentions -skip-tools but doesn't say what's supported or how it works.

darkowlzz · 2017-05-15T14:36:32Z

cmd/dep/init.go

+			Analyzers: []rootProjectAnalyzer{
+				newGopathAnalyzer(ctx.Loggers, ctx, pkgT, cpr, sm),
+				newImportAnalyzer(ctx.Loggers, sm),
+			}}


Just trying to understand, the compositeAnalyzer would also be added to analyzer.go's DeriveManifestAndLock() so that the dependency projects are also analyzed for dep, glide, godep, etc config in them, right?

The way I have things organized as of right now:

dep/analyzer.go handles identifying if a project uses dep and importing its config.

dep/cmd/dep/importAnalyzer.go handles organizing all the importers for other tools together and importing config for them recursively for all dependencies. The first tool which identities that its config is present is used, and the remaining tools are not used.

dep/cmd/dep/gopathAnalyzer.go contains all the existing logic for generating an initial manifest and lock based on what is on disk under GOPATH.

dep/cmd/dep/compositeAnalyzer.go lets us combine multiple analyzers, where the results of each is overlaid on top of the results of the previous, combining the results of an importer with whatever is found in the GOPATH.

The init command uses either just the gopathAnalyzer when -skip-tools is present (which is the behavior in master) for the root project and dependencies, or makes a compositeAnalyzer to determine the root manifest (first inspecting the GOPATH, then overlaying the import results) and uses the importAnalyzer for any dependencies.

Things are still in-flux as I've been developing in small steps. 😀 I haven't decided yet if it makes sense to combine the gopathAnalazyer with compositeAnalyzer, because they are only used together, but I kind of prefer to not have a hodge-podge struct that does all the things.

Let me know if that makes sense or not.

Yep, totally makes sense. I have started writing godep cofig over this PR. 🙂

But I was wondering if we should help gps analyze the dependency projects too with the new importers. From my understanding, gps calls analyzer's DeriveManifestAndLock() for every (direct & transitive) project, and tries to analyze their dependency constraints. Right now it receives nil for manifest and lock of those projects. But as projects adopt dep, and for those projects that already use some other tool, gps should be able to analyze them, right?

Please correct me if I misunderstood 😅

I think we are both trying to say the same thing. 👍 That's what I was trying to say with "using the importers for both the root project and the dependencies".

The solver takes an an initial manifest and lock for the root project, which I am generating using either just the gopathAnalyzer (for -skip-tools) or the compositeAnalayzer. But it also takes an analyzer to be applied to any dependencies it finds. When not skipping importing from other tools, I give the solver the importAnalyzer so that the same import logic we applied at the root is also applied to dependencies.

Thank you for asking about this! I forget to add an integration test for this scenario.

sdboyer

(just some ongoing comments that i had built up)

sdboyer · 2017-05-14T13:17:40Z

cmd/dep/ensure.go

 		}
 	}

 	// If not a plain SHA1 or bzr custom GUID, assume a plain version.
-	// TODO: if there is amgibuity here, then prompt the user?
-	return gps.NewVersion(s)
+	// return gps.NewVersion(s)


this is now cruft, let's remove

sdboyer · 2017-05-14T13:21:18Z

cmd/dep/glideConfig.go

@@ -0,0 +1,148 @@
+// Copyright 2016 The Go Authors. All rights reserved.


snake_case for filenames, please

carolynvs · 2017-05-15T17:34:51Z

@sdboyer The analyzer name returned by analyzer.Info() impacts the lock memo. I am using multiple analyzers, depending on if -skip-tools was specified. Do we want them to have different names (and therefore hashes)?

For example: dep/1 and dep.importer/1. If we do that, I'm not sure if I should try to distinguish between the external tool used to import as well, like dep.importer.glide or dep.importer.govendor?

sdboyer · 2017-05-15T18:12:23Z

@carolynvs I haven't reviewed your changes yet, so I don't know how much this lines up with what you've already done - so, speaking in generalities 😄

My instinctive preference is to have one actual analyzer struct that, perhaps based on some flag property, decides whether it will look for all existing tool metadata files, or only dep's. The name should similarly vary based on the state of that flag. When passing the analyzer to gps.Prepare(), just make sure it has the flag set for the narrower behavior.

For reference, the purpose of the name and version fields is really just to provide a simple way of categorizing the compatibility of solutions. More formally, their meaning would be:

If a solution differs from inputs by analyzer name, then the relationship is undefined, and no guarantee is made that the solution is valid for those inputs.
If a solution and inputs have the same analyzer name, but vary by version, then there is still no guarantee that the solution is valid for those inputs, but the user can be safely advised that their analyzer is either "newer" or "older" than the version used to create that lock.
If a solution and inputs have the same analyzer name and number, then it is guaranteed that the solution is valid for those inputs (with certain caveats)

My plan is to expose these fields directly in Gopkg.lock in #508, so that saner errors messages can be provided.

carolynvs · 2017-05-15T21:22:02Z

Thanks! That helps guide my next steps then. I was tempted to merge the two analyzers (skipping tools vs the current behavior) and that sounds like the way to go.

sdboyer

incremental review - holy crap, a lot of progress here 😄

sdboyer · 2017-05-17T13:46:15Z

cmd/dep/ensure_test.go

 	if err != nil {
 		t.Fatal(err)
 	}

 	constraints := map[string]gps.Constraint{
-		"v1.2.3": sv,
+		"v0.8.1": sv,
+		"master": gps.NewDefaultBranch("master"),


Not necessarily opposed to this, but...

Is making it a default branch actually material to the test here? Branch default-ness is something that's supposed to only really matter in internal sorting; that's why the func for it isn't currently exported from gps.

Derp! I don't know why I didn't try using c.String() != expected.String() to workaround default causing the equality check to fail. That fixes the test without having to expose NewDefaultBranch, I'll revert that change.

sdboyer · 2017-05-17T14:09:17Z

cmd/dep/glide_config.go

+
+	// Unsupported fields that we will warn if used
+	Subpackages []string `yaml:"subpackages"`
+	OS          string   `yaml:"os"`


Just noting here that we may be able to deal with these eventually - #291 - and if we warn on seeing these, we should probably include a link to that issue.

sdboyer · 2017-05-17T14:13:08Z

cmd/dep/glide_config.go

+func (g *glideFiles) load(projectDir string) error {
+	y := filepath.Join(projectDir, glideYamlName)
+	if g.loggers.Verbose {
+		g.loggers.Err.Printf("glide: Loading %s", y)


We want the output here to follow the pattern generally described in the spec, initial implementation in #512. Maybe this becomes the "section header" message? (If so, though, that header is printed regardless of verbose flag)

sdboyer · 2017-05-17T14:26:59Z

cmd/dep/glide_config.go

+
+func (g *glideFiles) buildProjectConstraint(pkg glidePackage, sm gps.SourceManager) (pc gps.ProjectConstraint, err error) {
+	if pkg.Name == "" {
+		err = errors.New("glide: Invalid glide configuration, package name is required")


ehhh, i'm on the fence about this. does glide care that much about having this value present? i can't actually recall.

This is to detect thje invalid configuration below, where an import section failed to set the package name. I don't require the root package name to be set. Granted it's unlikely to ever happen... Let me know if you still want the check removed.

import: - package: "" version: 1.0.0

ohhhhh totally misread, ok 👍

sdboyer · 2017-05-17T14:33:09Z

cmd/dep/glide_config.go

+			g.loggers.Err.Printf("glide: The %s package specified an arch, but that isn't supported by dep, and will be ignored.\n", pkg.Name)
+		}
+		if len(pkg.Subpackages) > 0 {
+			g.loggers.Err.Printf("glide: The %s package specified subpackages, but that is calcuated by dep, and will be ignored.\n", pkg.Name)


subpackages are basically dead information in glide right now, anyway. I think it's fine to just ignore them entirely; no need to warn.

sdboyer · 2017-05-17T14:37:47Z

cmd/dep/gopath_analyzer.go

+		if newProject {
+			// Check if it's in notondisk project map. These are direct deps, should
+			// be added to manifest.
+			if _, ok := a.pd.notondisk[pr]; ok {


The notondisk map isn't what we want to check here - rather, we need to be looking at the set of external roots. For that, we need something like what I've added over in my WIP ensure PR - https://github.com/sdboyer/dep/blob/1a546666f038281855255a52a3ff8532ff758b1b/cmd/dep/ensure.go#L356-L375

I think a couple of your comments may not be applicable due to the confusion over what gopath_analyzer does? It's just me extracting functions out of the init command into it's own struct and isn't for import at all.

Yep, this probably doesn't apply anymore

sdboyer · 2017-05-17T14:41:32Z

cmd/dep/gopath_analyzer.go

+func (a *gopathAnalyzer) DeriveRootManifestAndLock(path string, n gps.ProjectRoot) (*dep.Manifest, *dep.Lock, error) {
+	var err error
+
+	a.pd, err = getProjectData(a.ctx, a.pkgT, a.cpr, a.sm)


getProjectData is really only for the GOPATH detection route, not when we're importing from another tool. (we should probably rename it...)

This is for the gopath detection route (see file name: gopath_analyzer.go). I had extracted the logic for init into multiple analyzers (because as I was working on it, made it easier for me to think about). Based on your stated preference for one analyzer, I am now working to combine them back into a single struct.

ah! ok, i clearly didn't grok that reorganization from context here 😄

sdboyer · 2017-05-17T14:43:11Z

cmd/dep/gopath_analyzer.go

+			// be added to manifest.
+			if _, ok := a.pd.notondisk[pr]; ok {
+				m.Dependencies[pr] = getProjectPropertiesFromVersion(x.Version())
+				feedback(x.Version(), pr, fb.DepTypeDirect, a.ctx)


Also, though we had some confusion in #512 (I wasn't clear about this initially), the goal of the feedback is to report what we learned by importing from the other tool before we go into the solve, not what the ultimate results are. We want to tell the user what we extracted from glide - what we ultimately end up with is a different question.

Same as above.

I'll keep an eye out for the follow-up PR for #512 to make sure that any code I moved is merged appropriately.

sdboyer · 2017-05-17T14:44:25Z

cmd/dep/import_analyzer.go

+		return depAnalyzer.DeriveManifestAndLock(dir, pr)
+	}
+
+	// The assignment back to an interface prevents interface-based nil checks from failing later


👍
ugh typed nils

Yeah I struggled to figure out how to reuse that logic without causing typed nil troubles. Let me know if it's preferable to copy/paste code vs. calling the function and coercing the nils.

I think we may end up wanting to change the return pattern, anyway, per the discussion over in #594, which may help with this.

carolynvs · 2017-05-17T17:59:59Z

@sdboyer Okay I just finished consolidating the little analyzer structs I had made into a single analyzer in cmd/dep/root_analyzer.go. Hopefully that makes things less confusing to review. If I need to doc more, just poke me with a sharp stick. 😀

I stopped short of combining that with analyzer.go because I wanted to get feedback and ... because moving the analyzer out of cmd/dep into dep forces me to make more decisions about exported/unexported/internal which is a can of worms.

Working on your other feedback right now.

darkowlzz · 2017-05-17T18:04:28Z

cmd/dep/glide_config_test.go

+
+	rev := lpv.Underlying()
+	if rev != "6a741be0cc55ecbe4f45690ebfd606a956d5f14a" {
+		t.Fatalf("Expected locked revision to be '6a741be0cc55ecbe4f45690ebfd606a956d5f14a', got %s", lv)


Change lv to rev? lv results in the version string, in this case v1.0.0.

Thank you for the second set of 👀 !

darkowlzz · 2017-05-17T18:05:57Z

cmd/dep/glide_config_test.go

+
+	ver := lpv.String()
+	if ver != "v1.0.0" {
+		t.Fatalf("Expected locked version to be 'v1.0.0', got %s", lv)


Maybe change lv to ver? This would still be fine because lv and ver have same string values.

darkowlzz · 2017-05-17T18:46:19Z

cmd/dep/testdata/harness_tests/init/glide/case2/testcase.json

+  ],
+  "error-expected": "",
+  "gopath-initial": {
+    "github.com/sdboyer/deptestdos": "3f4c3bea144e112a69bbe5d8d01c1b09a544253f",


"github.com/sdboyer/deptest" 👀

carolynvs · 2017-05-19T20:41:57Z

@sdboyer When glide.yaml has a constraint for a transitive dependency, we want to create an override for that in the manifest, right? I wasn't sure if we wanted to force people to notice a warning in the output, and then make the override manually so that it's such a PITA that no one bothers. 😉

carolynvs · 2017-05-19T21:19:43Z

@sdboyer Another question. 😊 Now that we have the nifty feedback in place, the text displayed is misleading because the importer doesn't currently have all the facts up-front.

I am relying on the solver doing what it does best and then making sure the manifest is cleaned up to match before writing to disk. I populate the initial root manifest with everything I find in glide.yaml. If after solving it turns out that the package was transitive or not imported, then I remove it from the manifest.

Now with instant-feedback, I either need to pass more info to the importer to correctly display feedback, or tweak the feedback text to make it more clear that it's not final.

Here's an example:

deptest is a transitive dependency
deptestdos is a direct dependency
lint is not imported

Current (Misleading) Output

Converting from glide.yaml and glide.lock...
  Using ^1.0.0 as constraint for direct dep github.com/sdboyer/deptest
  Locking in v1.0.0 (ff2948a) for direct dep github.com/sdboyer/deptest
  Using ^2.0.0 as constraint for direct dep github.com/sdboyer/deptestdos
  Locking in v2.0.0 (5c60720) for direct dep github.com/sdboyer/deptestdos
  Using master as constraint for direct dep github.com/golang/lint
  Locking in master (cb00e56) for direct dep github.com/golang/lint

Option A: More vague output which is refined later during solve

Converting from glide.yaml and glide.lock...
  Using ^1.0.0 as constraint for imported dep github.com/sdboyer/deptest
  Locking in v1.0.0 (ff2948a) for imported dep github.com/sdboyer/deptest
  Using ^2.0.0 as constraint for imported dep github.com/sdboyer/deptestdos
  Locking in v2.0.0 (5c60720) for imported dep github.com/sdboyer/deptestdos
  Using master as constraint for imported dep github.com/golang/lint
  Locking in master (cb00e56) for imported dep github.com/golang/lint

Option B: Importers use the packageTree(?) to get it right the first time

Converting from glide.yaml and glide.lock...
  Ignoring transitive dep github.com/sdboyer/deptest
  Using ^2.0.0 as constraint for direct dep github.com/sdboyer/deptestdos
  Locking in v2.0.0 (5c60720) for direct dep github.com/sdboyer/deptestdos
  Ignoring unimported dep github.com/golang/lint

I'm leaning towards option b. What do you think?

carolynvs · 2017-05-28T16:25:20Z

@sdboyer This is ready for review.

This only covers init. The ensure and status commands are still using dep.Analyzer when solving which doesn't look at external tools. We'll need another pass to address that after this PR.
Since I wasn't sure what to do about imported constraints for transitive deps, I decided to ignore them for now.
I'm not sure how to deal with code climate. Some stuff, like doc on exported functions seems to be ignored for existing code but enforced for new code. Should I add an ignore in CC, doc the new function, doc all the exported functions?
I'm not sure why appveyor is failing. I assume I borked up test cleanup(?) but nothing is jumping out at me.

Overview of changes

cmd/dep/root_analyzer handles calling dep.Analyzer and the importers correctly based on -skip-tools. The analyzer name changes depending on this flag (dep vs dep+import).
cmd/dep/gopath_scanner is just the gopath scanning logic extracted out of the init command. It overlays whatever it finds on the results from the rootAnalyzer, filling in any gaps, but not changing values found by the importers.
cmd/dep/glide_importer implements the importer interface for glide.
Updated test harness tests to use -skip-import unless they are actually testing import.

sdboyer · 2017-05-30T23:25:58Z

Reviewing now 😄

As a starter, looking at the appveyor failures, it looks like a couple tests might've forgotten a call to SourceManager.Release().

carolynvs · 2017-05-30T23:52:16Z

Doh! Thanks that was it.

sdboyer · 2017-05-31T01:55:12Z

I'm not sure how to deal with code climate. Some stuff, like doc on exported functions seems to be ignored for existing code but enforced for new code. Should I add an ignore in CC, doc the new function, doc all the exported functions?

Yeah, I just flipped the switch on the CC analyzer for FIXMEs, TODOs, etc. It only applies to changed lines, not existing ones.

We need to buckle down and be diligent about following these rules, at least with the new stuff we introduce. So yes, let's make sure we're correctly documenting new exported identifiers.

sdboyer

OK! i think we're really pretty close at this point - i'm happy with the fundamental architecture. one more pass and i think we can merge 🎉 🎉

sdboyer · 2017-05-30T23:28:03Z

analyzer.go

-func (a Analyzer) DeriveManifestAndLock(path string, n gps.ProjectRoot) (gps.Manifest, gps.Lock, error) {
-	// TODO: If we decide to support other tools manifest, this is where we would need
-	// to add that support.
+func (a Analyzer) HasConfig(path string) bool {


Let's call this HasDepMetadata() instead; HasConfig() is a bit generic.

sdboyer · 2017-05-31T02:03:25Z

cmd/dep/ensure.go

-			if !found {
-				return constraint, errors.Errorf("%s is not a valid version for the package %s", versionStr, arg)
+		for _, v := range versions {
+			if v.Type() == gps.IsBranch {


We're not guaranteed sorted output from ListVersions() - the first branch we encounter might not be the default. Need to gps.SortPairedForUpgrade() on the slice to get the outcome we want.

sdboyer · 2017-05-31T02:07:47Z

cmd/dep/glide_importer.go

+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package main


fyi, i'm not gonna review the actual glide conversion logic in detail. if there are problems, i'd likely miss them; we'd end up needing to rely on user testing and incremental improvement anyway. so 😄

sdboyer · 2017-05-31T02:10:18Z

cmd/dep/gopath_scanner.go

+	}
+}
+
+// Perform analysis of the filesystem tree rooted at path, with the


Doc text should start with "InitializeRootManifestAndLock"

sdboyer · 2017-05-31T02:28:26Z

cmd/dep/init.go

+If configuration files for other dependency management tools are found, they
+are used to pre-populate the manifest and lock. Specify -skip-tools to disable
+this behavior.
+
 A Gopkg.toml file will be written with inferred version constraints for all


This text needs to be updated, as well. (Can be a little sloppy about it, given that it's going to change again once we add the -gopath flag.)

sdboyer · 2017-05-31T02:35:19Z

cmd/dep/root_analyzer.go

+	"github.com/golang/dep/internal/gps"
+)
+
+// importer handles importing configuration from other dependency managers into our format


Just a note - I think I'd eventually like to see this exported.

sdboyer · 2017-05-31T02:39:00Z

cmd/dep/root_analyzer.go

+// importer handles importing configuration from other dependency managers into our format
+type importer interface {
+	Name() string
+	Import(path string, pr gps.ProjectRoot) (*dep.Manifest, *dep.Lock, error)


Seems like importer should just compose gps.ProjectAnalyzer, rather than having a new method called Import with the same (well, an equivalent) signature, and the same basic purpose, as DeriveManifestAndLock. Is there a reason not to do that?

I did it this way for a couple reasons:

I needed the importers to return a dep manifest and lock, not the gps interfaces. Otherwise I would have to cast it immediately back.

I didn't want the importers to implement functions that would go unused, Analyzer.Info.

So while they are similar, but I didn't see anything to be gained by reusing that interface.

sdboyer · 2017-05-31T02:39:40Z

cmd/dep/root_analyzer.go

+	HasConfig(dir string) bool
+}
+
+// rootAnalyzer supplies manifest/lock data both from dep and external tool's config files


add periods; complete sentences is generally 👍

sdboyer · 2017-05-31T02:41:58Z

cmd/dep/testdata/harness_tests/init/glide/case1/final/Gopkg.toml

@@ -0,0 +1,10 @@
+ignored = ["github.com/sdboyer/dep-test","github.com/golang/notexist/samples"]


you do not, sorry, shouldn't have mentioned it 😄

sdboyer · 2017-05-31T02:43:37Z

cmd/dep/testdata/harness_tests/init/glide/case1/initial/samples/samples.go

+
+package samples
+
+import dt "github.com/carolynvs/go-dep-test"


ok, as long as you solemnly swear on your maintainer badge that this repo is frozen and will never change or go away once we merge this ❄️ 😺

Haha, I can swear to that. Or you can clone/fork it and put it with sdboyer/deptest for consistency?

sdboyer · 2017-05-31T03:25:34Z

hmm - looking at dep init -v output run against Masterminds/glide:

Importing configuration from glide. These are only initial constraints, and are further refined during the solve process.
Detected glide configuration files...
  Loading /Users/sdboyer/ws/go/src/github.com/Masterminds/glide/glide.yaml
  Loading /Users/sdboyer/ws/go/src/github.com/Masterminds/glide/glide.lock
Converting from glide.yaml and glide.lock...
  Using 0bdedde as initial hint for imported dep github.com/codegangsta/cli
  Using abff190 as initial hint for imported dep github.com/Masterminds/semver
  Using 3084677 as initial hint for imported dep github.com/Masterminds/vcs
  Using b8bc1bf as initial hint for imported dep github.com/mitchellh/go-homedir
  Using a3f3340 as initial hint for imported dep gopkg.in/yaml.v2
Searching GOPATH for projects...
Root project is "github.com/Masterminds/glide"
 17 transitively valid internal packages
 5 external packages imported from 5 projects
(0)   ✓ select (root)
(1)	? attempt github.com/Masterminds/semver with 1 pkgs; at least 1 versions to try
(1)	    try github.com/Masterminds/semver@v1.3.0
(1)	✓ select github.com/Masterminds/semver@v1.3.0 w/1 pkgs
(2)	? attempt github.com/Masterminds/vcs with 1 pkgs; at least 1 versions to try
(2)	    try github.com/Masterminds/vcs@v1.11.1
Importing configuration from glide. These are only initial constraints, and are further refined during the solve process.
Detected glide configuration files...
  Loading /Users/sdboyer/ws/go/pkg/dep/sources/https---github.com-Masterminds-vcs/glide.yaml
Converting from glide.yaml...
(2)	✓ select github.com/Masterminds/vcs@v1.11.1 w/1 pkgs
(3)	? attempt github.com/mitchellh/go-homedir with 1 pkgs; at least 1 versions to try
(3)	    try github.com/mitchellh/go-homedir@master
(3)	✓ select github.com/mitchellh/go-homedir@master w/1 pkgs
(4)	? attempt gopkg.in/yaml.v2 with 1 pkgs; 1 versions to try
(4)	    try gopkg.in/yaml.v2@v2
(4)	✓ select gopkg.in/yaml.v2@v2 w/1 pkgs
(5)	? attempt github.com/codegangsta/cli with 1 pkgs; at least 1 versions to try
(5)	    try github.com/codegangsta/cli@v1.19.1
(5)	✓ select github.com/codegangsta/cli@v1.19.1 w/1 pkgs
  ✓ found solution with 5 packages from 5 projects

Solver wall times by segment:
           b-gmal: 1.686396366s
      b-list-pkgs: 1.181170341s
      select-root:   1.577029ms
      select-atom:    355.459µs
          satisfy:    266.412µs
         new-atom:    129.091µs
  b-list-versions:    102.977µs
            other:      25.49µs
       b-pair-rev:     15.052µs
  b-source-exists:      8.269µs

  TOTAL: 2.870046486s

  Using master as constraint for direct dep github.com/mitchellh/go-homedir
  Locking in master (b8bc1bf) for direct dep github.com/mitchellh/go-homedir
  Using v2 as constraint for direct dep gopkg.in/yaml.v2
  Locking in v2 (cd8b52f) for direct dep gopkg.in/yaml.v2
  Using ^1.19.1 as constraint for direct dep github.com/codegangsta/cli
  Locking in v1.19.1 (0bdedde) for direct dep github.com/codegangsta/cli

we still have this thing where we produce feedback after the solve. either i'm forgetting something about the spec i wrote, or there's something fishy there. why are we not even applying constraints at all until after solving?

This makes it easier for the importers to reuse its logic

I tweaked ensure so that the existing behavior of ensure X applies the any (*) constraint.

carolynvs · 2017-06-01T19:55:24Z

@sdboyer There are a couple problems with the output. 😊

It was a bug that it wasn't properly printing the imported constraints. After fixing the truncated output from the importer looks like this:

Converting from glide.yaml and glide.lock...
Using ^1.16.0 as initial constraint for imported dep github.com/codegangsta/cli
Trying ^1.16.0 (0bdedde) as initial lock for imported dep github.com/codegangsta/cli

The last line should print the version that goes with the revision. @darkowlzz wrote that logic and was asking about where it should live, or if it exists already in gps. I'm going to steal his code and put it in a function next to deduceConstraint for now.
The feedback shown after the solve is from a bad merge. I think we used to log feedback after solving, and must have dropped it since I started on this feature. I missed that when I copied that code from init to the gopath_scanner. Fixing!

* init imports by default unless -skip-tools is specified * glide config in dependencies is taken into account during solve

carolynvs · 2017-06-01T21:28:12Z

I've addressed all your feedback, rebased and squashed. Here's the latest output of running dep init -v on glide.

$ dep init -v
Importing configuration from glide. These are only initial constraints, and are further refined during the solve process.
Detected glide configuration files...
  Loading /Users/carolynvs/go/src/github.com/Masterminds/glide/glide.yaml
  Loading /Users/carolynvs/go/src/github.com/Masterminds/glide/glide.lock
Converting from glide.yaml and glide.lock...
  Using ^1.19.1 as initial constraint for imported dep github.com/codegangsta/cli
  Trying v1.19.1 (0bdedde) as initial lock for imported dep github.com/codegangsta/cli
  Using ^1.3.0 as initial constraint for imported dep github.com/Masterminds/semver
  Trying v1.3.0 (abff190) as initial lock for imported dep github.com/Masterminds/semver
  Using ^1.11.1 as initial constraint for imported dep github.com/Masterminds/vcs
  Trying v1.11.1 (3084677) as initial lock for imported dep github.com/Masterminds/vcs
  Using master as initial constraint for imported dep github.com/mitchellh/go-homedir
  Trying master (b8bc1bf) as initial lock for imported dep github.com/mitchellh/go-homedir
  Using a3f3340 as initial hint for imported dep gopkg.in/yaml.v2
Searching GOPATH for projects...
Root project is "github.com/Masterminds/glide"
 53 transitively valid internal packages
 5 external packages imported from 5 projects
(0)   ✓ select (root)
(1)	? attempt github.com/Masterminds/semver with 1 pkgs; at least 1 versions to try
(1)	    try github.com/Masterminds/semver@v1.3.0
(1)	✓ select github.com/Masterminds/semver@v1.3.0 w/1 pkgs
(2)	? attempt github.com/Masterminds/vcs with 1 pkgs; at least 1 versions to try
(2)	    try github.com/Masterminds/vcs@v1.11.1
Importing configuration from glide. These are only initial constraints, and are further refined during the solve process.
Detected glide configuration files...
  Loading /Users/carolynvs/go/pkg/dep/sources/https---github.com-Masterminds-vcs/glide.yaml
Converting from glide.yaml...
(2)	✓ select github.com/Masterminds/vcs@v1.11.1 w/1 pkgs
(3)	? attempt github.com/mitchellh/go-homedir with 1 pkgs; at least 1 versions to try
(3)	    try github.com/mitchellh/go-homedir@master
(3)	✓ select github.com/mitchellh/go-homedir@master w/1 pkgs
(4)	? attempt gopkg.in/yaml.v2 with 1 pkgs; 1 versions to try
(4)	    try gopkg.in/yaml.v2@v2
(4)	✓ select gopkg.in/yaml.v2@v2 w/1 pkgs
(5)	? attempt github.com/codegangsta/cli with 1 pkgs; at least 1 versions to try
(5)	    try github.com/codegangsta/cli@v1.19.1
(5)	✓ select github.com/codegangsta/cli@v1.19.1 w/1 pkgs
  ✓ found solution with 5 packages from 5 projects

Solver wall times by segment:
      b-list-pkgs: 1.730042899s
           b-gmal: 1.573923904s
      select-root:   3.099974ms
      select-atom:    349.794µs
          satisfy:    340.791µs
         new-atom:    189.522µs
            other:     20.486µs
  b-source-exists:     16.998µs
  b-list-versions:      9.218µs
       b-pair-rev:      1.732µs

  TOTAL: 3.307995318s

sdboyer · 2017-06-02T00:03:44Z

(2)	    try github.com/Masterminds/vcs@v1.11.1
Importing configuration from glide. These are only initial constraints, and are further refined during the solve process.
Detected glide configuration files...
  Loading /Users/carolynvs/go/pkg/dep/sources/https---github.com-Masterminds-vcs/glide.yaml
Converting from glide.yaml...
(2)	✓ select github.com/Masterminds/vcs@v1.11.1 w/1 pkgs

The question of whether we read e.g. glide files in deps is a separate one from whether we read them from the current project during dep init. Can we make it controllable via a single boolean flag - e.g. if true, it'll try all the types of files we support; if false, it'll only look for Gopkg.toml and Gopkg.lock.

Also, either way, I don't think we want to print there, as it interrupts the trace output.

* Ignore glide's test case for importing during solve

carolynvs · 2017-06-02T01:31:51Z

I've disabled importing during solve. This means that if a dependency has glide config, it is not used at all. Previously -skip-tools controlled import both for the root project and dependencies, now it only controls at the root, and we never import for dependencies.

I've also tweaked the importer so that nothing is logged during solve. So if we eventually use the importer during solve, it won't interrupt the output. Here's the new output:

$ dep init -v
Importing configuration from glide. These are only initial constraints, and are further refined during the solve process.
Detected glide configuration files...
  Loading /Users/carolynvs/go/src/github.com/Masterminds/glide/glide.yaml
  Loading /Users/carolynvs/go/src/github.com/Masterminds/glide/glide.lock
Converting from glide.yaml and glide.lock...
  Using ^1.19.1 as initial constraint for imported dep github.com/codegangsta/cli
  Trying v1.19.1 (0bdedde) as initial lock for imported dep github.com/codegangsta/cli
  Using ^1.3.0 as initial constraint for imported dep github.com/Masterminds/semver
  Trying v1.3.0 (abff190) as initial lock for imported dep github.com/Masterminds/semver
  Using ^1.11.1 as initial constraint for imported dep github.com/Masterminds/vcs
  Trying v1.11.1 (3084677) as initial lock for imported dep github.com/Masterminds/vcs
  Using master as initial constraint for imported dep github.com/mitchellh/go-homedir
  Trying master (b8bc1bf) as initial lock for imported dep github.com/mitchellh/go-homedir
  Using a3f3340 as initial hint for imported dep gopkg.in/yaml.v2
Searching GOPATH for projects...
Root project is "github.com/Masterminds/glide"
 65 transitively valid internal packages
 5 external packages imported from 5 projects
(0)   ✓ select (root)
(1)	? attempt github.com/Masterminds/semver with 1 pkgs; at least 1 versions to try
(1)	    try github.com/Masterminds/semver@v1.3.0
(1)	✓ select github.com/Masterminds/semver@v1.3.0 w/1 pkgs
(2)	? attempt github.com/Masterminds/vcs with 1 pkgs; at least 1 versions to try
(2)	    try github.com/Masterminds/vcs@v1.11.1
(2)	✓ select github.com/Masterminds/vcs@v1.11.1 w/1 pkgs
(3)	? attempt github.com/mitchellh/go-homedir with 1 pkgs; at least 1 versions to try
(3)	    try github.com/mitchellh/go-homedir@master
(3)	✓ select github.com/mitchellh/go-homedir@master w/1 pkgs
(4)	? attempt gopkg.in/yaml.v2 with 1 pkgs; 1 versions to try
(4)	    try gopkg.in/yaml.v2@v2
(4)	✓ select gopkg.in/yaml.v2@v2 w/1 pkgs
(5)	? attempt github.com/codegangsta/cli with 1 pkgs; at least 1 versions to try
(5)	    try github.com/codegangsta/cli@v1.19.1
(5)	✓ select github.com/codegangsta/cli@v1.19.1 w/1 pkgs
  ✓ found solution with 5 packages from 5 projects

Solver wall times by segment:
           b-gmal: 1.832750336s
      b-list-pkgs: 1.383528239s
      select-root:   3.428211ms
          satisfy:    330.136µs
      select-atom:    318.279µs
         new-atom:    140.255µs
            other:     44.367µs
  b-source-exists:     11.541µs
  b-list-versions:      6.211µs
       b-pair-rev:        794ns

  TOTAL: 3.220558369s

sdboyer · 2017-06-02T15:25:02Z

OK so there is just NO apparent reason for that test windows test failure. Github is up...I don't know.

Either way, this is close enough, and has gone on way too long. It's time to merge. We can work out the rest incrementally 😄

googlebot added the cla: yes label May 2, 2017

carolynvs force-pushed the import-glide branch from 9e08a58 to bdd0625 Compare May 3, 2017 21:25

kamilchm mentioned this pull request May 5, 2017

glide support? kamilchm/go2nix#19

Open

carolynvs mentioned this pull request May 8, 2017

Hide dep guts under internal/ #527

Closed

carolynvs force-pushed the import-glide branch 2 times, most recently from fd6d7ff to 4a51515 Compare May 12, 2017 20:38

carolynvs mentioned this pull request May 14, 2017

Support importing godep config during dep init #577

Closed

darkowlzz reviewed May 15, 2017

View reviewed changes

carolynvs force-pushed the import-glide branch from 2e95a71 to b1c99a8 Compare May 15, 2017 15:55

sdboyer reviewed May 15, 2017

View reviewed changes

darkowlzz mentioned this pull request May 16, 2017

WIP - Import godep #589

Closed

sdboyer reviewed May 17, 2017

View reviewed changes

carolynvs changed the title ~~WIP - Import glide configuration during init~~ Import glide configuration during init May 17, 2017

darkowlzz reviewed May 17, 2017

View reviewed changes

carolynvs force-pushed the import-glide branch from 05c8a62 to 41ed164 Compare May 19, 2017 21:49

carolynvs mentioned this pull request May 23, 2017

Extract command implementions into sub-packages #609

Closed

carolynvs force-pushed the import-glide branch from 93ad0c0 to 364a10d Compare May 28, 2017 16:20

carolynvs mentioned this pull request May 30, 2017

workflow: moving from gvt to dep #679

Closed

sdboyer suggested changes May 31, 2017

View reviewed changes

This was referenced May 31, 2017

Feedback: Existing project using gvt vs. dep #138

Closed

Break up Ctx.VersionInWorkspace #688

Merged

carolynvs added 3 commits June 1, 2017 13:11

Add dep ensure package@version tests

d8e32d8

Consolidate version/branch validation into deduceConstraint

53fa2b1

This makes it easier for the importers to reuse its logic

Detect the default branch in deduceConstraint when constraint is empty

66b1df0

I tweaked ensure so that the existing behavior of ensure X applies the any (*) constraint.

carolynvs added 2 commits June 1, 2017 16:22

Support importing glide configuration

cbc263f

* init imports by default unless -skip-tools is specified * glide config in dependencies is taken into account during solve

Skip importing from external tools in non-import tests

6f7a9d3

carolynvs force-pushed the import-glide branch from 364a10d to 6f7a9d3 Compare June 1, 2017 21:25

Add missing Release calls to glide tests

892a157

sdboyer mentioned this pull request Jun 2, 2017

Document Analyzer and Project #690

Merged

carolynvs added 2 commits June 1, 2017 20:12

Don’t run importers during solve yet

373556d

* Ignore glide's test case for importing during solve

Suppress logging from importers during solve

1ef8181

sdboyer approved these changes Jun 2, 2017

View reviewed changes

sdboyer merged commit 01294a8 into golang:master Jun 2, 2017

carolynvs deleted the import-glide branch June 11, 2017 15:27

carolynvs mentioned this pull request Jun 19, 2017

Support importing glide config during dep init #380

Closed

carolynvs mentioned this pull request Oct 31, 2017

Import during solve #1335

Closed

		@@ -0,0 +1,148 @@
		// Copyright 2016 The Go Authors. All rights reserved.

		@@ -0,0 +1,10 @@
		ignored = ["github.com/sdboyer/dep-test","github.com/golang/notexist/samples"]

Import glide configuration during init #500

Import glide configuration during init #500

Conversation

carolynvs commented May 2, 2017 • edited Loading

sdboyer commented May 2, 2017

sdboyer commented May 3, 2017

carolynvs commented May 3, 2017

sdboyer commented May 12, 2017

carolynvs commented May 13, 2017 • edited Loading

carolynvs commented May 14, 2017

Choose a reason for hiding this comment

carolynvs May 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolynvs commented May 15, 2017

sdboyer commented May 15, 2017

carolynvs commented May 15, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolynvs May 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolynvs commented May 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolynvs commented May 19, 2017

carolynvs commented May 19, 2017

carolynvs commented May 28, 2017

sdboyer commented May 30, 2017

carolynvs commented May 30, 2017

sdboyer commented May 31, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdboyer commented May 31, 2017

carolynvs commented Jun 1, 2017

carolynvs commented Jun 1, 2017

sdboyer commented Jun 2, 2017

carolynvs commented Jun 2, 2017 • edited Loading

sdboyer commented Jun 2, 2017

carolynvs commented May 2, 2017 •

edited

Loading

carolynvs commented May 13, 2017 •

edited

Loading

carolynvs May 15, 2017 •

edited

Loading

carolynvs May 17, 2017 •

edited

Loading

carolynvs commented Jun 2, 2017 •

edited

Loading