Skip to content
This repository has been archived by the owner on Sep 9, 2020. It is now read-only.

[WIP] minimal module awareness #1963

Closed

Conversation

joshuarubin
Copy link

@joshuarubin joshuarubin commented Jul 27, 2018

  • Update pkgtree.ListPackages to set the correct import path based on the go.mod file
  • Ensure that the vendor/ directory is populated correctly (import x/y/v2/foo should not be stored as vendor/x/y/foo)
  • Add tests
  • Update CHANGELOG.md

What does this do / why do we need it?

Adds minimal module awareness, as Go 1.10.3 did.

Which issue(s) does this PR fix?

Fixes #1962

Signed-off-by: Joshua Rubin <joshua@rubixconsulting.com>
@joshuarubin
Copy link
Author

Right now, dep init and dep ensure succeed, but the vendor/ directory isn't populated correctly.
The version, e.g. v2 does not exist in the vendor file tree. What I mean is import x/y/v2/foo is copied into the vendor/ directory as vendor/x/y/foo. Similarly, x/y/v2 exists as vendor/x/y.

Perhaps as an artifact of this, pruning unused packages causes the entire package, e.g. vendor/x/y, to be removed from the vendor tree.

@joshuarubin joshuarubin changed the title [WIP] start work to support minimal module awareness [WIP] minimal module awareness Jul 27, 2018
Signed-off-by: Joshua Rubin <joshua@rubixconsulting.com>
Signed-off-by: Joshua Rubin <joshua@rubixconsulting.com>
@joshuarubin
Copy link
Author

Hi @sdboyer, I'm having a hard time tracing how things go from pkgtree.ListPackages to being written by the DeltaWriter. Maybe I'm barking up the wrong tree. You mentioned changing the API a bit for figuring out if a given path exists. Are you still referring to pkgtree? If you could give me a little nudge in the direction you'd like, I'm sure I could finish this up.

Signed-off-by: Joshua Rubin <joshua@rubixconsulting.com>
@joshuarubin
Copy link
Author

I've gotten the vendor directory to be constructed properly. However, it isn't particularly elegant. After exporting the revision from the source, I simply move the directories as required based on the results of pkgtree.ListPackages.

@sdboyer
Copy link
Member

sdboyer commented Aug 4, 2018

@joshuarubin sorry, busy week.

re: this:

changing the API a bit for figuring out if a given path exists

i meant that in reference to how the solver itself works. That'd be a check made in this vicinity. Probably the easiest thing to do is duplicate the Package at both the mod-aware and non-mod-aware path, but i've honestly not thought through the implications yet.

Re: exporting, i'd say that all the logic in there should just be updated to be module-sensitive - but, because we only operate at the whole-repo level, yes, it will probably entail some hacky moving-around-of-things after the full tree write is complete. That's fine, though - it looks like you're on the right path with where you've put those moveModules() calls.

Other questions?

olivere added a commit to olivere/elastic-with-go-modules that referenced this pull request Aug 4, 2018
This commit updates the README to include Go 1.11 beta 3.

We also add a few notes about "minimal module awareness" that has been
included in Go 1.10.3 and 1.9.7. Unfortunately, the `dep` tool is still
on its way to implement that.

* [Minimal module awareness in Go 1.10.3 and Go
1.9.7](https://go.googlesource.com/go/+/d4e21288e444d3ffd30d1a0737f15ea3fc3b8ad9)
* [Tracking implementation of "minimal module awareness" in dep](golang/dep#1963)
@joshuarubin
Copy link
Author

@sdboyer the gps solver works with modules already in this PR. Is there something you think should be done differently?

The exporter also seems to work well in most cases already, so I'd really appreciate any sort of review on that as well.

The one case where things don't seem to work, and I haven't tracked down, is when changing a constraint to a new major version (e.g. package foo/bar (which is imported at foo/bar/v1), version 1.0.0 to version 2.0.0, obviously imported at foo/bar/v2. I forget exactly what happens in this case, but either dep itself fails or the exported vendor tree doesn't contain foo/bar at all, or has the wrong version. I'll look into this issue more once I figure out if this PR is on the right track.

@joshuarubin
Copy link
Author

Any chance we could keep this moving forward?

@joshuarubin
Copy link
Author

@sdboyer? @jmank88?

@sdboyer
Copy link
Member

sdboyer commented Sep 4, 2018 via email

@sdboyer
Copy link
Member

sdboyer commented Sep 4, 2018 via email

@joshuarubin
Copy link
Author

Bueller?

@joshuarubin
Copy link
Author

It would be awesome to get some movement on this

@sdboyer
Copy link
Member

sdboyer commented Oct 3, 2018 via email

@joshuarubin
Copy link
Author

Just another reminder to see how things are going with this

brandur-stripe pushed a commit to stripe/stripe-go that referenced this pull request Jan 16, 2019
Reverts:
#712

(And an earlier PR that added a `go.mod` file in the first place.)

Unfortunately, there are still some fairly bad incompatibilities between
Go modules and Dep for users still on the latter. For now, we're going
to revert Go module support because there doesn't seem to be any easy
way (i.e., anything that's not maintaining a fork) of doing a good job
of both. Projects can still use the Go module system, but will have to
pull this package in as a pre-module incompatible one.

We are hoping to eventually see basic module awareness merged into Dep:

golang/dep#1963

If/when that gets done, we'll revert this revert after a small grace
period, and hopefully be back to a place where both packaging systems
are well supported.

I'll also modify the README to clarify the current situation in another
follow up PR.
brandur-stripe pushed a commit to stripe/stripe-go that referenced this pull request Jan 16, 2019
Reverts:
#712

(And an earlier PR that added a `go.mod` file in the first place.)

Unfortunately, there are still some fairly bad incompatibilities between
Go modules and Dep for users still on the latter. For now, we're going
to revert Go module support because there doesn't seem to be any easy
way (i.e., anything that's not maintaining a fork) of doing a good job
of both. Projects can still use the Go module system, but will have to
pull this package in as a pre-module incompatible one.

We are hoping to eventually see basic module awareness merged into Dep:

golang/dep#1963

If/when that gets done, we'll revert this revert after a small grace
period, and hopefully be back to a place where both packaging systems
are well supported.

I'll also modify the README to clarify the current situation in another
follow up PR.
@kevinburke
Copy link
Collaborator

kevinburke commented Jun 27, 2019 via email

@bytheway
Copy link

I have pulled down this patch and given it try. It seems to do the trick for me, enabling support especially of go module libraries with v2 import paths.

I've written some limited gps tests and an init integration test, which extends the commits in this repo.

https://github.com/bytheway/dep/tree/module-tests.

I'm happy to make another PR based on that fork-of-a-fork if that would make sense.

@adubkov
Copy link

adubkov commented Jun 29, 2019

Any change to move it forward anytime soon? It keep getting more annoying because more and more packages not using go modules.

@dobegor
Copy link

dobegor commented Jul 2, 2019

@sdboyer @theckman are there any objections to merge this?

@kollektiv
Copy link

Could this change be moved forward?

@virtuald
Copy link

Building a custom version of dep based on this PR allowed me to successfully use a go module based project with dep. 👍

@raushan2016
Copy link

raushan2016 commented Sep 24, 2019

Can we get this PR in. @sdboyer @jmank88

@harsimranmaan
Copy link

@sdboyer @jmank88 Is this waiting on something?

@90wukai
Copy link

90wukai commented Oct 30, 2019

We use this, running perfect!

Copy link

@sseago sseago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was getting the same failure when one of our dependencies started using go modules. With this PR dep is working fine.

sseago added a commit to sseago/openshift-migration-plugin that referenced this pull request Dec 6, 2019
containers/image now uses go modules, so for 'dep ensure' to work
with the updated code, this upstream dep PR is needed:
golang/dep#1963

Without this PR, dep does not work with dependencies which
use go modules.
@amitech74
Copy link

@sdboyer @jmank88 looks like this PR is waiting on a stamp from you guys. would you be able to move this forward?

@jmank88
Copy link
Collaborator

jmank88 commented Mar 20, 2020

Unfortunately, @sdboyer and I have not been active on this project for some time.
@theckman or @kevinburke may be able to assist.

brandur-stripe pushed a commit to stripe/stripe-go that referenced this pull request Apr 10, 2020
Similar to the original implementation for Go Modules in #712, here we
add a `go.mod` and `go.sum`, then proceed to use Go Modules style
import paths everywhere that include the current major revision.

Unfortunately, this may still cause trouble for Dep users who are trying
to upgrade stripe-go, but the project's now had two years to help its
users with basic Go Module awareness, and has chosen not to do so. It's
received no commits of any kind since August 2019, and therefore would
be considered unmaintained by most definitions. Elsewhere, Go Modules
now seem to be the only and obvious way forward, so we're likely to see
more and more users on them.

`scripts/check_api_clients/main.go` is also updated to be smarter about
breaking down package paths which may now include the major.

[1] golang/dep#1963
remi-stripe pushed a commit to stripe/stripe-go that referenced this pull request Apr 16, 2020
Similar to the original implementation for Go Modules in #712, here we
add a `go.mod` and `go.sum`, then proceed to use Go Modules style
import paths everywhere that include the current major revision.

Unfortunately, this may still cause trouble for Dep users who are trying
to upgrade stripe-go, but the project's now had two years to help its
users with basic Go Module awareness, and has chosen not to do so. It's
received no commits of any kind since August 2019, and therefore would
be considered unmaintained by most definitions. Elsewhere, Go Modules
now seem to be the only and obvious way forward, so we're likely to see
more and more users on them.

`scripts/check_api_clients/main.go` is also updated to be smarter about
breaking down package paths which may now include the major.

[1] golang/dep#1963
remi-stripe pushed a commit to stripe/stripe-go that referenced this pull request Apr 16, 2020
Similar to the original implementation for Go Modules in #712, here we
add a `go.mod` and `go.sum`, then proceed to use Go Modules style
import paths everywhere that include the current major revision.

Unfortunately, this may still cause trouble for Dep users who are trying
to upgrade stripe-go, but the project's now had two years to help its
users with basic Go Module awareness, and has chosen not to do so. It's
received no commits of any kind since August 2019, and therefore would
be considered unmaintained by most definitions. Elsewhere, Go Modules
now seem to be the only and obvious way forward, so we're likely to see
more and more users on them.

`scripts/check_api_clients/main.go` is also updated to be smarter about
breaking down package paths which may now include the major.

[1] golang/dep#1963
brandur-stripe added a commit to stripe/stripe-go that referenced this pull request Apr 17, 2020
* Make API response accessible on returned API structs (#1054)

* Make API response accessible on returned API structs

Makes an API response struct containing niceties like the raw response
body, status, and request ID accessible via API resource structs
returned from client functions. For example:

    customer, err := customer.New(params)
    fmt.Printf("request ID = %s\n", customer.LastResponse.RequestID)

This is a feature that already exists in other language API libraries
and which is requested occasionally here, usually for various situations
involving more complex usage or desire for better observability.

-- Implementation

We introduce a few new types to make this work:

* `APIResponse`: Represents a response from the Stripe API and includes
  things like request ID, status, and headers. I elected to create my
  own object instead of reusing `http.Response` because it gives us a
  little more flexibility, and hides many of myriad of fields exposed by
  the `http` version, which will hopefully give us a little more API
  stability/forward compatibility.

* `APIResource`: A struct that contains `LastResponse` and is meant to
  represent any type that can we returned from a Stripe API endpoint. A
  coupon is an `APIResource` and so is a list object. This struct is
  embedded in response structs where appropriate across the whole API
  surface area (e.g. `Coupon`, `ListMeta`, etc.).

* `LastResponseGetter`: A very basic interface to an object that looks
  like an `APIResource`. This isn't strictly necessary, but gives us
  slightly more flexibility around the API and makes backward
  compatibility a little bit better for non-standard use cases (see the
  section on that below).

`stripe.Do` and other backend calls all start taking objects which are
`LastResponseGetter` instead of `interface{}`. This provides us with some
type safety around forgetting to include an embedded `APIResource` on
structs that should have it by making the compiler balk.

As `stripe.Do` finishes running a request, it generates an `APIResponse`
object and sets it onto the API resource type it's deserializing and
returning (e.g. a `Coupon`).

Errors also embed `APIResource` and similarly get access to the same set
of fields as response resources, although in their case some of the
fields provided in `APIResponse` are duplicates of what they had
already (see "Caveats" below).

-- Backwards compatibility

This is a minor breaking change in that backend implementations methods
like `Do` now take `LastResponseGetter` instead of `interface{}`, which
is more strict.

The good news though is that:

* Very few users should be using any of these even if they're
  technically public. The resource-specific clients packages tend to do
  all the work.

* Users who are broken should have a very easy time updating code.
  Mostly this will just involve adding `APIResource` to structs that were
  being passed in.

-- Naming

* `APIResponse`: Went with this instead of `StripeResponse` as we see in
  some other libraries because the linter will complain that it
  "stutters" when used outside of the package (meaning, uses the same
  word twice in a row), i.e. `stripe.StripeResponse`. `APIResponse`
  sorts nicely with `APIResource` though, so I think it's okay.

* `LastResponse`: Copied the "last" convention from other API libraries
  like stripe-python.

* `LastResponseGetter`: Given an "-er" name per Go convention around
  small interfaces that are basically one liners -- e.g. `Reader`,
  `Writer, `Formatter`, `CloseNotifier`, etc. I can see the argument
  that this maybe should just be `APIResourceInterface` or something
  like that in case we start adding new things, but I figure at that
  point we can either rename it, or create a parent interface that
  encapsulates it:

    ``` go
    type APIResourceInterface interface {
        LastResponseGetter
    }
    ```

-- Caveats

* We only set the last response for top-level returned objects. For
  example, an `InvoiceItem` is an API resource, but if it's returned
  under an `Invoice`, only `Invoice` has a non-nil `LastResponse`. The
  same applies for all resources under list objects. I figure that doing
  it this way is more performant and makes a little bit more intuitive
  sense. Users should be able to work around it if they need to.

* There is some duplication between `LastResponse` and some other fields
  that already existed on `stripe.Error` because the latter was already
  exposing some of this information, e.g. `RequestID`. I figure this is
  okay: it's nice that `stripe.Error` is a `LastResponseGetter` for
  consistency with other API resources. The duplication is a little
  unfortunate, but not that big of a deal.

* Rename `LastResponseGetter` to `LastResponseSetter` and remove a function

* Update stripe.go

Co-Authored-By: Olivier Bellone <ob@stripe.com>

* Move `APIResource` onto individual list structs instead of having it in `ListMeta`

Co-authored-by: Brandur <brandur@stripe.com>
Co-authored-by: Olivier Bellone <ob@stripe.com>

* Remove all beta features from Issuing APIs

* Multiple breaking API changes

* `PaymentIntent` is now expandable on `Charge`
* `Percentage` was removed as a filter when listing `TaxRate`
* Removed `RenewalInterval` on `SubscriptionSchedule`
* Removed `Country` and `RoutingNumber` from `ChargePaymentMethodDetailsAcssDebit`

* Start using Go Modules

Similar to the original implementation for Go Modules in #712, here we
add a `go.mod` and `go.sum`, then proceed to use Go Modules style
import paths everywhere that include the current major revision.

Unfortunately, this may still cause trouble for Dep users who are trying
to upgrade stripe-go, but the project's now had two years to help its
users with basic Go Module awareness, and has chosen not to do so. It's
received no commits of any kind since August 2019, and therefore would
be considered unmaintained by most definitions. Elsewhere, Go Modules
now seem to be the only and obvious way forward, so we're likely to see
more and more users on them.

`scripts/check_api_clients/main.go` is also updated to be smarter about
breaking down package paths which may now include the major.

[1] golang/dep#1963

* Change v71 back to v70

Move back down to current major version so that we can test that our
release script will bump it to v71 properly when the time comes.

Co-authored-by: Brandur <brandur@stripe.com>
Co-authored-by: Olivier Bellone <ob@stripe.com>
Co-authored-by: Remi Jannel <remi@stripe.com>
@mvdan
Copy link
Member

mvdan commented Sep 4, 2020

Dep was officially deprecated earlier this year, and the proposal to archive this repository was accepted. As such, I'm closing outstanding issues before archiving the repository. For any further comments, please use the proposal thread on the Go issue tracker. Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Missing support for "minimal module-awareness"