Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Initial integration of gps into glide #384
This pull request swaps in gps for much of the core glide engine.
This checklist is sorta starting in media res of a long-running pull request...but hey, better late than never.
This checklist represents what I think should be necessary to at least merge this PR, though not necessarily be a complete, finished conversion. It might be preferable to merge this into a separate branch, so we can then iterate on other issues in smaller PRs.
@sdboyer first, wow... that's a huge changeset. It's going to take a little bit to look at that.
Second, in your quick review request of c708bf7, I'll need to look further at why you're asking for the manifest / lock pairing to understand what's going on.
I would prefer that the mechanism that looks for external imports be pluggable rather than hard coded. So, it would iterate over a list of the things rather than go one by one. Though, this is an area the current importer could really improve on.
@mattfarina the large changeset is almost entirely under vendor; as I noted above, it's mostly
If you look at diffs restricted to real, local code additions, there's very little as of now:
Right, this is the crux of what there is to grok/review. As currently designed,
The sole method analyzers currently have,
vsolver needs both the manifest and lock as both are relevant to the solving process:
This last idea is one that, as far as I know, will be unique to glide, as I first suggested in my article. @technosophos and I have discussed it several times as possibly being very helpful in dealing with the go project ecosystem "as-is" - full of locked deps from godep, etc.
That's seems doable. I don't think it's really either/or, though - we can have the hardcoded analysis for godep, gom, etc., and if those don't work, let a plugin have a go? Or let the plugin have a go first, and then fall back on those. Either way, my concern is that the use case for this does not immediately spring to mind (though glide obviously has a history with this, so I suspect you have something concrete), but the complications for caching the returned data (which
I'm confused, "iterating over a list" and "go one by one" seem like synonyms to me. Unless you're suggesting that they all run unconditionally, even if one indicates a successful analysis?
I guess that's possible. Seems to me you'd have a mess, though, and I'm not clear on what case that's trying to cover. Some project simultaneously using, say, godep and glide, maybe? If anyone is doing that, then erroring out seems like the saner thing to do than trying to make sense of a fundamentally nonsensical situation. (The user could always be encouraged to fork that bizarre project and fix it - which really seems saner to me than trying to have tooling figure out what manner of custom crazy happens from the combination of two systems).
@sdboyer Two pieces of feedback so far...
I'm going to keep poking at this but I wanted to record these before I switched off reviewing for the day.
i wrote that logging layer while i was still getting the core solver algorithm right. initially, i imagined that it might be helpful for users, but immediately found (once i started using it to debug the solver) that it was difficult even for ME to use. so, i opened sdboyer/gps#5, the outcome of which will specifically be a trace logger, with output looking like the linked file. it will be, dare i say,
that said, i don't think there's actually a problem here. the reason i'm using
(we discussed a bit in gitter after you posted this, so reiterating a bit)
yes, this is crucial. we already do half of it now: reading the dev/test deps out of the manifest. the other half is ensuring we incorporate test files in static analysis. the (untested) code is already (mostly) written for that; i ultimately put it down because the requirement was still a bit murky for me, and i felt that the work required to integrate with glide would clarify it.
so, yeah, the static analysis part is high on the priority list. my plan was to incorporate it if not in this PR, then in the next one.
referenced this pull request
Apr 22, 2016
Small progress update:
The latest set of changes takes one approach at solving the issue raised in #391. It's provisional, hacky code, but it does the job well enough for now, and it can give some basis for discussion, at least.
In any case, having at least a temporary solution in place for that, I'm now up against the next barrier, which has turned out to be package-vs-project-level analysis. That is, since I'm working against glide itself here, and that annoying
Tackling this properly means refactoring to include at least some of the static analysis I mentioned in my last comment. There won't be much visible on this side, but there'll be a fair few changes on the
@sdboyer can you use the normal logger interface for your application and pass in logrus for your test runs? Then anything implementing the logger interface can be passed into vsolver. I can even make a logger interface for the
referenced this pull request
Sep 2, 2016
@sdboyer Thanks for the updates on this. I just pulled the latest and ran into another problem.
First, I ran
I next tried to run
$ go run glide.go up # github.com/Masterminds/glide/vendor/gopkg.in/yaml.v2 vendor/gopkg.in/yaml.v2/error.go:4: "ERROR: the correct import path is gopkg.in/yaml.v2 ... " evaluated but not used # github.com/Masterminds/glide/vendor/github.com/sdboyer/gps vendor/github.com/sdboyer/gps/constraints.go:53: undefined: semver.Constraint
hash: 3975bda2fdb43b0cccab2b038f89cbe099f5b6f25b77084b4769d7e15a1f62a8 updated: 2016-09-15T09:25:57.859680773-04:00 imports: - name: github.com/armon/go-radix branch: master revision: 4239b77079c7b5d1243b7b4736304ce8ddb6f0f2 - name: github.com/codegangsta/cli version: v1.14.0 revision: 71f57d300dd6a780ac1856c005c4b518cfd498ec - name: github.com/kr/pretty version: go.r60 revision: a6f3cf47d962d7c6eae13b363e2b860eb561e528 - name: github.com/Masterminds/semver version: v1.1.1 revision: 8d0431362b544d1a3536cca26684828866a7de09 - name: github.com/Masterminds/vcs version: v1.8.0 revision: fbe9fb6ad5b5f35b3e82a7c21123cfc526cbf895 - name: github.com/sdboyer/gps version: v0.10.0 revision: 278a227dfc3d595a33a77ff3f841fd8ca1bc8cd0 - name: github.com/termie/go-shutil revision: bcacb06fecaeec8dc42af03c87c6949f4a05c74c - name: gopkg.in/yaml.v2 branch: master revision: bec87e4332aede01fb63a4ab299d8af28480cd96 testImports: 
Notice the version for semver here isn't right. You can see the update run at https://gist.github.com/mattfarina/4171e5f18db51a3ee8110a27e779854c. You'll see the message about the 2.x conflict.
What's going on?
The pushed commits should fix those issues. I've run
The single biggest issue was that the
which was an indication that it was using the legacy logic to interpret glide.yaml, causing all
Now, initially I was more concerned by something else: even if the root manifest was halfway converted (and thus it didn't pick up the
Turns out, the answer's in the trace:
The halfway converted root
And that's where things got fun, because
package: github.com/sdboyer/gps owners: - name: Sam Boyer email: email@example.com import: # legacy name - package: github.com/Masterminds/semver branch: 2.x # new key vcs: git - package: github.com/Masterminds/vcs vcs: git - package: github.com/termie/go-shutil version: bcacb06fecaeec8dc42af03c87c6949f4a05c74c vcs: git
So, we had the same problem repeating itself a layer down - glide's analyzer autoconverted that glide.yaml, silently dropped the branch constraint, which made it look like v0.10.0 was cool with any version of semver, which of course meant that there was no longer a conflict with the already-selected firstname.lastname@example.org - so the algo picked email@example.com, and off we went.
Again, I think this is mostly a problem that's unique to the bootstrapping situation. However, this particular incident does change some of my feelings on how glide's analyzer is currently written. Right now, it tries really hard not to error on
Also, just to explain - the reason these half-conversions exist at all is because I had to deal with the ambiguity of semver-or-branch discussed in #391 early on, because of the
It's a nice reminder of how profoundly important "backwards" compatibility is in this domain - we kinda have to deal with all projects, across all time. Old forms don't vanish into the background in the way they often do with other software.