Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools: write release tool to run your package's callers' tests #25483

Open
bradfitz opened this Issue May 21, 2018 · 13 comments

Comments

Projects
None yet
10 participants
@bradfitz
Copy link
Member

bradfitz commented May 21, 2018

In #24301 (comment) I proposed in a comment:

We've been discussing some sort of go release command that both makes releases/tagging easy, but also checks API compatibility (like the Go-internal go tool api checker I wrote for Go releases). It might also be able to query godoc.org and find callers of your package and run their tests against your new version too at pre-release time, before any tag is pushed. etc.

And later in that issue, I said:

With all the cloud providers starting to offer pay-by-the-second containers-as-a-service, I see no reason we couldn't provide this as an open source tool that anybody can run and pay the $0.57 or $1.34 they need to to run a bazillion tests over a bunch of hosts for a few minutes.

@SamWhited and @mattfarina had objections (#24301 (comment), #24301 (comment)) about the inclusivity of such a tool, so I'm opening this bug so we don't cause too much noise in the other issue.

Such a tool would involve:

  • querying a service (such as godoc.org or anything implementing the "Go workspace abstraction") to find your callers
  • running the tests (or a fraction thereof) either
    • locally, possibly overnight
    • on a cloud provider of your choice (AWS Fargate, Azure Container Service, Digital Ocean, GCP, etc)

... and telling you if they pass before your change, but fail after your change, so you can release a new version of your package with confidence.

In local mode, it'd use your local GOOS/GOARCH. On cloud, Linux containers are cheapest, but it could also spin up Windows VMs like we do with Go. (Each Go commit gets a fresh new Windows VM that boots in under a minute and runs a few tests and then the VM is destroyed).

None of this costs much, and the assumption is that this would be used by people (optionally) who are willing to pay for the extra assurance, and/or those whose time needed to fix regressions later is worth more than the cloud costs.

And maybe some cloud CI/CD company(s) could sponsor such builders.

@mattfarina

This comment has been minimized.

Copy link
Member

mattfarina commented May 21, 2018

Here are a few problems to be solved/questions...

  1. Can you share how my bold question here is solved by this?
  2. How is this simpler or easier for people than something like dep that has configurable maximums on ranges?
  3. What about all the people who don't use a release tool (because it's complicated and extra work to setup) and break compatibility?
  4. How does costing work for this? If you're popular you pay more. Doesn't this enable those with a lot of money while being worse for those who don't?
@SamWhited

This comment has been minimized.

Copy link
Member

SamWhited commented May 21, 2018

you're taking issue over an optional feature of a hypothetical design

I had missed the "optional feature" part, it sounds to me when reading it that the point was that the tool spun up machines on some cloud service and used them to download lots of code and run tests. I'm not sure that it would be practical to run on a laptop if you have a popular project, but I'm less concerned and it seems worth exploring if that's the case.

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

AlexRouSg commented May 21, 2018

In addition to pulling all sources and running in one VM, since the users of your package may have their own test environment (like cgo usage) and their own os/arch combo, also to allow closed source users of your package to have this benefit...

How about the tool somehow notifies users of your package (uses a list file for closed source users) to pull your changes and test on their builders.

Maybe even have a service/daemon tool to automate this somehow and report back to you through smtp/email.

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

AlexRouSg commented May 21, 2018

btw would Go itself use the tool to check if some change to the compiler or std broke some popular package?

@randall77

This comment has been minimized.

Copy link
Contributor

randall77 commented May 22, 2018

This sounds like a security minefield. A nefarious actor just has to import your project in order to get you to run their code for them. Any such tests would have to be very well sandboxed.

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

AlexRouSg commented May 22, 2018

@randall77 i.e. bitcoin miners

One of the reasons I suggested they run the code.

@bradfitz

This comment has been minimized.

Copy link
Member Author

bradfitz commented May 22, 2018

@randall77 @AlexRouSg, all of this assumes sandboxes. I assume we'd limit CPU execution time & run things under gVisor where possible, and for Windows at least use firewalled VMs (limiting network in/out, so they can't e.g. send email). What I don't know how to do is nice sandboxing for GOOS=windows on GOHOSTOS=windows without new VMs. Maybe we assume Hyper-V or something.

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

AlexRouSg commented May 22, 2018

@bradfitz do you think having a mode to inform/let users of your package to run the test on their builders is something to look at?

I feel like people with non trivial build processes or uses of C/C++ libs would be left out.

@dolmen

This comment has been minimized.

Copy link

dolmen commented May 22, 2018

Prior art in the Perl community: CPAN Testers. Community testing of Perl modules releases since 1998. Example report.

@mbyio

This comment has been minimized.

Copy link

mbyio commented May 22, 2018

This sounds like a great idea. It's like a generalization of what Rust does with https://github.com/rust-lang-nursery/crater, except instead of testing the compiler for breaking changes, it's testing libraries. Many many times I see people accidentally introduce breaking changes not because they are being irresponsible but because they didn't realize what the full scope of the change was. This tool could really help with that.

I do suspect that only the best supported libraries will end up using it, unless some company makes it easy to do test runs.

I guess the main downside that I see is some people might think it's okay to make a breaking change if they find that no tests failed. For some packages I bet there are fewer tests using them directly (eg. logging libraries). But I don't think that's really a reason to not make this tool.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Jun 11, 2018

Not entirely sure why this is in the proposal process. Is there a proposal for the Go team to evaluate?

@bradfitz bradfitz changed the title proposal: write release tool to run your package's callers' tests x/tools: write release tool to run your package's callers' tests Jun 11, 2018

@bradfitz bradfitz added NeedsFix and removed Proposal labels Jun 11, 2018

@bradfitz bradfitz modified the milestones: Proposal, Unreleased Jun 11, 2018

@bradfitz

This comment has been minimized.

Copy link
Member Author

bradfitz commented Jun 11, 2018

@rsc, yeah, no need for the proposal process. Removed.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Sep 25, 2018

See also #26420. As far as I can tell, these issues are basically dual: the tool proposed here is “test my dependencies with the current version of my module”, and #26420 is “test the current version of my module with the versions as seen by dependencies”.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.