Cant 'go get' anymore

[pangteypiyush]

Bug Report

• What happened?
  Trying to go get subscription module

go get -u -v github.com/Azure/azure-sdk-for-go/services/subscription/mgmt/2020-09-01/subscription
go: downloading github.com/Azure/azure-sdk-for-go v64.2.0+incompatible
go: github.com/Azure/azure-sdk-for-go/services/subscription/mgmt/2020-09-01/subscription: create zip: module source tree too large (max size is 524288000 bytes)

• What did you expect or want to happen?
  Finish without an error.
  The last version which I didnt had this issue:

go get github.com/Azure/azure-sdk-for-go@v50.2.0
go: added github.com/Azure/azure-sdk-for-go v50.2.0+incompatible

[OscarVanL]

I get this problem too when doing go list -u -m -json all

go: loading module retractions for github.com/Azure/azure-sdk-for-go@v16.2.1+incompatible: verifying go.mod: github.com/Azure/azure-sdk-for-go@v64.2.0+incompatible/go.mod: reading https://sum.golang.org/lookup/github.com/!azure/azure-sdk-for-go@v64.2.0+incompatible: 410 Gone
        server response: not found: create zip: module source tree too large (max size is 524288000 bytes)

[jhendrixMSFT]

I believe we will need to prune old API versions to reduce the repo size.

[jhendrixMSFT]

While we're working on removing old content, you should be able to use v64.1.0 instead (unless you need something specific in v64.2.0).

[ceejatec]

@jhendrixMSFT That's unfortunately not true - we have a module which references v64.1.0 in its go.mod, but when running go list -mod=readonly -m -u -json all in that directory, we get the above error message anyway. I don't know why go is still trying to download that newer version.

[ceejatec]

Only workaround I've found so far is to set the GONOSUMDB environment variable to github.com/Azure/azure-sdk-for-go.

[jhendrixMSFT]

Thanks for the data-point. It works for me, so there must be some other bit we don't yet understand. At any rate, we should have an update out soon that shrinks the size under the 500MB limit.

[lootek]

While we're working on removing old content, you should be able to use v64.1.0 instead (unless you need something specific in v64.2.0).

No, we're still not able to freeze the version in go.mod due to an error:

go.mod:75:2: replace github.com/Azure/azure-sdk-for-go: version "v64.1.0" invalid: should be v0 or v1, not v64

Unless you add +incompatible because this repo does not follow Go directory naming rules for modules-based versioning.

The workaround that worked was to replace to freeze to the latest commit hash or to the prerelease instead (it's an ugly and hopefully a temporary one though):

github.com/Azure/azure-sdk-for-go => github.com/Azure/azure-sdk-for-go v64.1.0+incompatible

Tryiing to use the newest prerelease v64.2.0+incompatible fails as well:

verifying go.mod: github.com/Azure/azure-sdk-for-go@v64.2.0+incompatible/go.mod: reading https://sum.golang.org/lookup/github.com/!azure/azure-sdk-for-go@v64.2.0+incompatible: 410 Gone
	server response: not found: create zip: module source tree too large (max size is 524288000 bytes)

And, v65 does not work as well:

go.mod:74:2: replace github.com/Azure/azure-sdk-for-go: version "v65.0.0" invalid: should be v0 or v1, not v65

[jhendrixMSFT]

The v65.0.0 release didn't get tagged for some reason. I just tagged it now so that part should be resolved.

[prochac]

May I ask, why is the repo so heavy?
After running go clean --modcache and redownloading all modules I use,
Azure SDK makes 34% of my $GOPATH/go/mod size.
Only competitor is github.com/aws/aws-sdk-go with 10%.

[jhendrixMSFT]

Azure uses date-based API versions in the query string to partition updates to a service that aren't backward-compatible (I don't believe AWS has such a mechanism). Our legacy SDK, which predates Go modules, releases all of these API versions side-by-side to reduce/remove breaking changes across service updates. You can see an example of this here. Unfortunately, it does mean that the repo grows over time. We have a work item to eliminate old/unused API versions to reduce the size, it just hasn't happened yet.

Going forward, our new SDKs use Go modules to manage breaking changes, so we won't be releasing all API versions side-by-side.