Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/coordinator: tracking bug for running benchmarks in the coordinator. #19871

Open
bradfitz opened this issue Apr 6, 2017 · 28 comments

Comments

@bradfitz
Copy link
Member

commented Apr 6, 2017

(Split off #8930, "repair perf builders", the https://perf.golang.org/ work)

This is a tracking bug for running benchmarks in the coordinator, both pre-submit (trybots) and post-submit.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

I anticipate that pre- and post-submit benchmark runs will be somewhat different (e.g. expected latency, # of iterations, etc.), and post-submit benchmark runs require trend analysis, so I am focusing on pre-submit trybot benchmark runs at the moment.

We've already had several design discussions about this; I'm going to attempt to replay them into this issue for posterity.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2017

For trybots:

Currently we:

  • run make.bash on one buildlet
  • snapshot the make.bash-built world to GCS
  • and then shard the tests (go tool dist test --list work units) across N buildlets. The creation of those helper buildlets is timed considering the expected make.bash duration and the expected buildlet creation time, the idea being that we don't hold on to buildlets when we won't need them, but ask for them earlier enough that they're available by the time make.bash is done and we want to run tests.

If machines die, they can resume later just from snapshots, bypassing make.bash. The untarring of the snapshot to the tmpfs is fast.

I think benchmarks should work similarly. They should be another work unit(s), like unit tests, sharded over machines.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

Perf Trybots Sketch

(reviewed Mar 3 by @bradfitz)

We want to collect benchmark data as part of the normal Gerrit CL process. To effect this, I propose the following modifications to the build coordinator:

  • Change buildStatus.build() to add
    if remoteErr == nil && err == nil && st.shouldBench() {st.runBench()}
    after the tests finish
  • Implement buildStatus.shouldBench to check:
    st.isTry() &&
    !st.isSubrepo() &&
    benchBuilders[st.name]  // map listing builders to run benchmarks on
    
  • Implement buildStatus.runBench() to:
    • Download snapshots for the try commit (if not already present) and its base commit
    • Build bench binaries for all affected packages
    • Run bench binaries and capture the results
    • Upload the results to perfdata.golang.org
  • Initially, have this only happen on the staging coordinator and not on the prod coordinator. I do not want to introduce a separate "Run-Perfbots" Gerrit label, because I want to quickly get to the point where it's not a burden to run for every CL. (Back of the envelope calculations suggest we're already there: running benchmarks require less time than running tests.)
  • Once we're happy with performance, turn this on in prod and include the result links in the "trybots complete" message that is posted on the commits

Builder choice

Some builders are probably too slow to run benchmarks on. I suggest we initially run benchmarks on:

  • freebsd-amd64-gce101
  • linux-386
  • linux-amd64
  • linux-arm
  • windows-amd64-gce
    This gives us a good swath of goos and goarch, but avoids the slowest of our builders.

@bradfitz's comments:
Q: why download a snapshot if you just completed a build & tests? it's already there in $WORKDIR, no?
A: I wasn't sure if I was going to be reusing the buildlet that had done the tests. You're right it might be there already.

Q: there's no guarantee [the base commit's snapshot] exists. is that a problem?
A: I think we should just not try to run benchmarks if the base commit didn't pass tests. Is there another way snapshots can not exist?

Q: how many iterations are you going to run to get statistically interesting numbers?
A: TBD? I'm going to start with 5.

Q: [freebsd-amd64-gce101] is just another amd64 so I'm not sure it helps so much. Seems overkill for trybots. I even think Windows is a bit redundant but justifiable due to its drastic difference. Linux and FreeBSD are relatively similar. Mostly seeing three different architectures is the interesting bit. I think running builders are three different Intel CPU families (in different GCE zones would be even more interesting)
A: I think we have enough different in the runtime on BSDs that it's worth running benchmarks, no? Do we as a user get to control which CPU family we get? Also, keep in mind this is for trybots, so a quick sanity check is more important than exhaustive coverage of goos/goarch/CPU type.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

Email thread with @bradfitz about how to fit benchmarks into the existing work units for make and test.

@quentinmit:
I'd like your ideas on how to best handle sequencing for the trySet.

I know you have your goal of posting trybot results in <5m. Obviously, that's simply not enough time to post trybot results, and we can't (I think?) parallelize the benchmarks because we'd like them to come from the same machine. Currently with 5 iterations of the benchmarks it's taking 10-20m to complete.

So I think that leads me to "we should post make + test results first, and then follow up with bench results in a separate Gerrit comment". Do you think that's a reasonable UI? Alternatively, we could include in the existing Gerrit post "benchmarks are being run, ", and have the benchmark results just stream in at that page without making another post when they're done.

The question becomes then how to fit that into trySet. There's an ominous comment in buildStatus.build:

	// bc (aka st.bc) may be invalid past this point, so let's
	// close it to make sure we we don't accidentally use it.

Ideally, we would like to log the eventDone with the make+test results, and then continue on to run benchmarks on the same bc. I think what I'd like to do is move the call to ts.noteBuildComplete from its current location in awaitTryBuild to call it directly from bs.build() right after eventDone is logged. AFAICT the ominous comment above has no teeth, and the bc doesn't actually become invalid until build returns and the context is canceled.

Sorry for the long-winded e-mail. Does this all sound okay?

@bradfitz:
There's an open bug (#13076) about making permanent URLs that work either during or after a build, whether it's successful or failed.

There's been some progress towards that in the past year (unique build IDs and try IDs used mostly for BigQuery), but if we finished that, then we'd have a great place to reference perf data too.

So we'd record the Trybot-Result + comment on Gerrit and then keep running benchmarks. People interested in benchmarks can get to it from the original already-spammed comment on Gerrit. Perhaps it should only do a follow-up comment if the benchmark found notable change.

@quentinmit:
What do you think about my proposal for how to implement this, then?

@bradfitz:
I would prefer not to modify not complicate the design by putting in reuse at that layer. It would also complicate scheduling later. (#19178)

I'd prefer it to be a logically separate work unit, but do the buildlet sharing as an optimization via the buildlet pool layer. There's an open bug about that somewhere--- if a buildlet is no longer being used (especially ones that are slow to create) and it's never seen a failure and has no new processes running from when it started, then add it back to the free pool, at least for everything but building a release where we really want things to be hermetic.

@quentinmit:
It's not just an optimization to reuse the buildlet, though - we need a buildlet that already has the built go unpacked on disk. I agree that my suggestion is complicated, though.

I could certainly build a mechanism to pass off the buildlet client to a logically separate work unit, but then I also run into the question of how to handle logging. Right now it seems like there's an assumption that there is only one buildStatus for each commit and buildlet type. I think I would have to break that assumption if we want to have the "bench" buildStatus separate from the "make and test" buildStatus.

(Code-wise, the benchmark stuff is already logically separate, it has a single entry point of bs.runBench().)

@bradfitz:
Let's not make reusing the files on disk a requirement. The cost of untarring them to the tmpfs from GCS is super tiny.

I'd rather have a simple design (benchmarks always start with a fresh buildlet, as a new work unit) and pay the cost of untarring the work that was just done a moment ago in the uncontended case.

But in the contended case (like this morning), we'd free up the buildlets for important work (trybots, building the release-branch.go1.8 most recent commit, etc), and catch up on benchmarking later when idle.

@quentinmit:
I'm talking only about trybot runs - benchmarks are just as important for as test results for trybots. I completely agree for non-trybots that they should be separate and lower-priority than builds.

Okay, with that, I believe that brings this issue up to the current state of discussion.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

Okay, so, if I understand your proposal correctly, you want us to treat benchmarks as another type of helper buildlet, so we untar the snapshot built by make.bash and then run the benchmarks in a fresh buildlet.

Understood, and that makes sense (bearing in mind the overhead of starting another VM, which can be mitigated with your reuse proposal). In the meantime until we do that, I'd like to just put that logic here into buildStatus, so that if the parent commit is ready, we pass the buildlet client off to runBench to reuse instead of getting a new one. This is much simpler than the generic reuse you proposed, because we already know the buildlet is in an okay state because we know the make and test just finished.

Now, given that, I'm still unsure of exactly how to tie that into the existing buildStatus struct and logging infrastructure. Do you have any suggestions that differ from what I proposed above in e-mail?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2017

Okay, so, if I understand your proposal correctly, you want us to treat benchmarks as another type of helper buildlet, so we untar the snapshot built by make.bash and then run the benchmarks in a fresh buildlet.

I want to treat each benchmark as a unit of work that needs to be done, just like a "go tool dist test --list" work unit. I don't care whether it runs on the initial buildlet or on one of the helper buildlets. Ideally they'd run on all N of them, including the original one, in parallel. (I assume we're running more than 1 benchmark, and running it more than 1 time.)

Understood, and that makes sense (bearing in mind the overhead of starting another VM, which can be mitigated with your reuse proposal).

We already have the buildlets available. We just created ~N=5 to run unit tests on, and now they're idle. (Also, the Kubernetes buildlets create in seconds. There's not much overhead there.)

You'd need to modify runAllSharded to not destroy the helper buildlets when they're done with unit tests.

In the meantime until we do that,

Let's not cut corners. I have to maintain this. If there's a dependency to do, do that first.

I'd like to just put that logic here into buildStatus,

The logic for what?

so that if the parent commit is ready, we pass the buildlet client off to runBench to reuse instead of getting a new one.

Reusing the idle buildlets is fine.

Other question: are you only doing benchmarks for the main repo ("go") or also for subrepos?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2017

What work does the benchmarking need to do, and what environment does it need?

It needs the output from make.bash. How many benchmarks does it run, and which, and how many iterations? Can the iterations run on different VMs if they're the same processors/sizes?

Does the parent commit need to be submitted? Can we use benchmark output computed previously for the parent, if it's from the same VM processors/sizes?

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

Okay, so, if I understand your proposal correctly, you want us to treat benchmarks as another type of helper buildlet, so we untar the snapshot built by make.bash and then run the benchmarks in a fresh buildlet.

I want to treat each benchmark as a unit of work that needs to be done, just like a "go tool dist test --list" work unit. I don't care whether it runs on the initial buildlet or on one of the helper buildlets. Ideally they'd run on all N of them, including the original one, in parallel. (I assume we're running more than 1 benchmark, and running it more than 1 time.)

Okay, yes. I am starting with the assumption that we need to run all benchmarks on a single buildlet, so that the results are comparable. With the CL as currently implemented, we are running 4 separate benchmark binaries, and we are running each of them 5 times. At a minimum, I think we need to make sure we run the same set of benchmarks on each buildlet (so we could shard it 5 ways, but not 20 ways). I don't know yet how much variance that introduces.

Understood, and that makes sense (bearing in mind the overhead of starting another VM, which can be mitigated with your reuse proposal).

We already have the buildlets available. We just created ~N=5 to run unit tests on, and now they're idle. (Also, the Kubernetes buildlets create in seconds. There's not much overhead there.)

You'd need to modify runAllSharded to not destroy the helper buildlets when they're done with unit tests.

I think this conflicts with the fact that we may need to wait for the parent commit to make + test, and we don't want to tie up buildlets while waiting.

I'd like to just put that logic here into buildStatus,

The logic for what?

The logic for deciding whether a buildlet can be reused for benchmarks.

Other question: are you only doing benchmarks for the main repo ("go") or also for subrepos?

At the moment, only the go repo is in scope. Running benchmarks in subrepos is also interesting, but I'm not yet tackling that because I haven't designed a way to decide which benchmarks to run.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

What work does the benchmarking need to do, and what environment does it need?

We need to have built benchmark binaries (which could have been cross-compiled, I suppose), and then we just need to run them on a buildlet. We do require a GOROOT, because some of the benchmarks try to (re)compile parts of the go distribution.

It needs the output from make.bash. How many benchmarks does it run, and which, and how many iterations? Can the iterations run on different VMs if they're the same processors/sizes?

I have the assumption that the iterations can't run on different VMs, but see above.

Does the parent commit need to be submitted? Can we use benchmark output computed previously for the parent, if it's from the same VM processors/sizes?

The parent commit does not need to be submitted. It does need to have been built and tested. We can't use previously computed benchmark output because there is far too much variation on GCE. (We could of course use a pre-compiled benchmark binary for the parent, however.)

@josharian

This comment has been minimized.

Copy link
Contributor

commented Apr 7, 2017

Related: #19327

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2017

How long do each of the ~5 benchmark binaries take to run?

For a given binary, do you want to run the old-vs-new binaries as:

./old -benchtime=3s
./new -benchtime=3s

Or interleaved old/new like:

./old -benchtime=1s
./new -benchtime=1s
./old -benchtime=1s
./new -benchtime=1s
./old -benchtime=1s
./new -benchtime=1s

? I can see some advantage of interleaving them if the run times are long and something unrelated on the VM (different customer) changes performance somehow.

Implementation wise: if you want this part of trybots, we really need to work to minimize overlap what runs when.

The benchmarks can be sharded, and since we already have sharded testing buildlets, that won't add much.

I'm more concerned about the extra make.bash time (~90 seconds) needed to build the parent commit if it's not built already. I'd like to overlap that make.bash on a separate buildlet alongside the main make.bash. But because Windows VMs take ~45 seconds to boot and become usable, you really need to ask for the second buildlet about the same time you ask for the first one. You can't delay your check of whether the parent git commit's make.bash is cached on GCS until later. It would need to be done in (*buildStatus).build in parallel with the existing check. If you find that the cached build isn't available, then you'd need to ask for a second buildlet (from the same pool as (*buildStatus).getHelpersReadySoon later creates).

Refactoring the helper buildlets code from being just about test sharding to being earlier and more general (and letting you ask for just 1 immediately vs the timer creating all N helpers at once) will be the trickiest part.

As for sharding the benchmarks out over buildlets, I would treat them as normal test units, modifying (*buildStatus).distTestList somehow to either return a better "work unit" type instead of the current []string of dist test names, or keep it as is and make a higher level "(*buildStatus).getShardedTestWork() []tbItem" sort of test-or-bench work unit, and have getShardedTestWorkcalldistTestList` and then append its ~5 benchmark work units.

Then you'd modify (*buildStatus).runTests (the thing that coordinates the execution over sharded buildlets and shutting down buildlets at the end) to instead call getShardedTestWork. Some work units will be tests, and some will be benchmarks. The first benchmark work unit would wait for the make.bash to be done, and the first benchmark work unit per buildlet would untar the snapshotted parent-commit.tar.gz into a different top-level $WORKDIR directory, like $WORKDIR/parent-go instead of $WORKDIR/go.

I drew a picture of how the sharding works for tests earlier today and @adams-sarah took a picture of it but I don't think it was a great drawing on its own. I can probably make a Gantt chart of each buildlet and what it's doing if that'd help.

@adams-sarah

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2017

img_3425

@josharian

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2017

As they say, a caption is worth a thousand words.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2017

I'll recreate it with words. Thanks, @adams-sarah.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2017

How long do each of the ~5 benchmark binaries take to run?

I've been logging that with spans and we have a fair amount of data now. Do you have any reporting tools or should I just dig in the datastore myself?

From a quick glance, it looks like the go1 binary is the worst, at ~60s per run. The other benchmark binaries are around 10-20s each.

(go1 contains many benchmarks, of course, so we can split it in half to get 30s, or in three to get 20s.)

For a given binary, do you want to run the old-vs-new binaries or interleaved old/new like:
? I can see some advantage of interleaving them if the run times are long and something unrelated on the VM (different customer) changes performance somehow.

I agree, we definitely want to interleave them.

Implementation wise: if you want this part of trybots, we really need to work to minimize overlap what runs when.

I don't know what you mean by "minimize overlap". Do you mean that we should try to schedule Kube pods on different hosts?

I'm more concerned about the extra make.bash time (~90 seconds) needed to build the parent commit if it's not built already. I'd like to overlap that make.bash on a separate buildlet alongside the main make.bash. But because Windows VMs take ~45 seconds to boot and become usable, you really need to ask for the second buildlet about the same time you ask for the first one. You can't delay your check of whether the parent git commit's make.bash is cached on GCS until later. It would need to be done in (*buildStatus).build in parallel with the existing check. If you find that the cached build isn't available, then you'd need to ask for a second buildlet (from the same pool as (*buildStatus).getHelpersReadySoon later creates).

I don't think it makes sense to rebuild the parent commit if a snapshot is not available. We don't even necessarily know where to fetch it from (since it could be in a separate Gerrit CL, or on master). I think we should just wait until the parent commit has been built normally. This means we might take longer than 5m if a whole train of CLs is pushed at once, but I think that's a reasonable tradeoff (and we might wait a while in that case anyway before we can get a buildlet to run tests, too). When buildlet scheduling is implemented, we can teach the scheduling to run CLs in order from oldest to newest.

Refactoring the helper buildlets code from being just about test sharding to being earlier and more general (and letting you ask for just 1 immediately vs the timer creating all N helpers at once) will be the trickiest part.

Yeah, that makes sense. Initially, I'd like to punt on this change and instead just use a completely separate buildlet for benchmarks in parallel with the sharded test buildlets. Then reusing the same buildlets just becomes an optimization.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2017

I've been logging that with spans and we have a fair amount of data now. Do you have any reporting tools or should I just dig in the datastore myself?

x/build/cmd/buildstats -sync will sync from datastore to bigquery, and then you can write SQL.

(go1 contains many benchmarks, of course, so we can split it in half to get 30s, or in three to get 20s.)

Yeah, we'll want to do that.

I don't know what you mean by "minimize overlap". Do you mean that we should try to schedule Kube pods on different hosts?

Er, sorry, that sentence made little sense. I meant we want to maximize overlap, to not have idle buildlets while we do many steps in serial. We want to take advantage of potential parallelism, as it currently does.

I don't think it makes sense to rebuild the parent commit if a snapshot is not available.
We don't even necessarily know where to fetch it from (since it could be in a separate Gerrit CL, or on master).

You just fetch it from the gitmirror, which will return it regardless of where it is.

I think we should just wait until the parent commit has been built normally.

Idling while waiting for a buildlet isn't acceptable. We're not going to do that.

When buildlet scheduling is implemented, we can teach the scheduling to run CLs in order from oldest to newest.

That's actually not the priority that was proposed in #19178. Recent things are more important than things 5 commits back.

Yeah, that makes sense. Initially, I'd like to punt on this change and instead just use a completely separate buildlet for benchmarks in parallel with the sharded test buildlets.

Sure, how about your use one buildlet (maybe total for now, until #19178 can prioritize buildlet users) and do your work totally async with the trybots. The trybots can write "This will eventually be available at $URL", and users can just poll that URL, or your process can leave a comment saying the results are available, if it detected anything interesting.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 19, 2017

CL https://golang.org/cl/38306 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 21, 2017
cmd/coordinator: run benchmarks on try work
Benchmarks are treated as unit tests and distributed to the test
helpers, which allows them to fit in our 5m trybot budget.

Currently we only run the go1 and x/benchmarks. Running package
benchmarks is a TODO.

This feature is disabled by default, and is enabled by the
"farmer-run-bench" project attribute.

Updates golang/go#19178
Updates golang/go#19871

Change-Id: I9c3a14da60c3662e7e2cb4e71953060915cc4364
Reviewed-on: https://go-review.googlesource.com/38306
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented May 9, 2017

CL https://golang.org/cl/43052 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 10, 2017
cmd/coordinator: populate commit label on benchmarks
This was lost in one of the many refactorings of CL 38306.

Updates golang/go#19871

Change-Id: I4514d4350feb1420861eae9a9dd021fee5b5ba6f
Reviewed-on: https://go-review.googlesource.com/43052
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented May 15, 2017

CL https://golang.org/cl/43532 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 17, 2017
cmd/coordinator, dashboard: trim benchmark builders for initial launch
- Enable benchmarks on only linux-amd64, with 3 additional test
  helpers, to ensure we fit in the 5m trybot budget
  (disabling freebsd-amd64-gce101, linux-386, linux-arm,
  windows-amd64-2008, and windows-386-2008).
- Run benchmarks with GOMAXPROCS=2 to reduce variation in
  benchmarks. See
  https://perf.golang.org/search?q=upload%3A20170512.4+cores%3Aall+parallel%3A4+%7C+gomaxprocs%3A2+vs+gomaxprocs%3A16+vs+gomaxprocs%3A32

Updates golang/go#19871

Change-Id: I8a28127be86c0c8691605cae4c9d603e3c8122cc
Reviewed-on: https://go-review.googlesource.com/43532
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Avelino <t@avelino.xxx>
@gopherbot

This comment has been minimized.

Copy link

commented May 25, 2017

CL https://golang.org/cl/44175 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented May 25, 2017

CL https://golang.org/cl/44176 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented May 25, 2017

CL https://golang.org/cl/44173 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented May 25, 2017

CL https://golang.org/cl/44174 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented May 25, 2017

CL https://golang.org/cl/44211 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 25, 2017
cmd/coordinator: remove buildStatus from most of benchmarks.go
This refactoring is in preparation for moving benchmarks.go to a
separate package.

Updates golang/go#19871

Change-Id: I2b30bf5416937e52b603aec8102131fdccceee42
Reviewed-on: https://go-review.googlesource.com/44173
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 25, 2017
cmd/coordinator: create separate spanlog package
This moves main.spanLogger and main.eventSpan to spanlog.Logger and
spanlog.Span. This change is necessary to start pulling build and
benchmark code out of coordinator. The interfaces cannot simply be
copied because Logger contains a function whose return type is
Span, so the interface must be defined in exactly one place.

Updates golang/go#19871

Change-Id: I0a48192e6a5c8f5d0445f4f3d3cce8d91c90f8d3
Reviewed-on: https://go-review.googlesource.com/44174
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 26, 2017
cmd/coordinator, internal/buildgo: create buildgo package
This package contains the BuilderRev type moved from cmd/coordinator.

The rest of the CL is simply updating coordinator with the new
exported names of the type and its fields.

This refactoring is in preparation for moving the benchmark building and running
code into a separate package.

(Most of the diff could have been avoided with a type alias, but I
assume we'd rather not do that.)

Updates golang/go#19871

Change-Id: Ib6ce49431c8529d6b4e72725d3cd652b9d0160db
Reviewed-on: https://go-review.googlesource.com/44175
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 26, 2017
cmd/coordinator, internal/buildgo: add RunMake
This moves RunMake from coordinator.go, allowing other packages to
build Go on a buildlet.

Updates golang/go#19871

Change-Id: Ic0e7c056020ef434f0620ae816f52b9112ea1c8d
Reviewed-on: https://go-review.googlesource.com/44176
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 26, 2017
cmd/coordinator, internal: move benchmark building to separate package
This moves getSourceTgz* to a new package called sourcecache. The
original functions couldn't be left in coordinator.go because that
would cause a single process to have multiple caches.

The code for building and running benchmarks is moved into the buildgo
package, and the public API becomes GoBuilder.EnumerateBenchmarks and
Run on the resulting BenchmarkItem objects.

Updates golang/go#19871

Change-Id: I28b660e1cdaa6d1c6b0378c08de30f5e58316cc6
Reviewed-on: https://go-review.googlesource.com/44211
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Jun 13, 2017

CL https://golang.org/cl/45671 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Jun 13, 2017

CL https://golang.org/cl/45672 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Jun 13, 2017

CL https://golang.org/cl/45670 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Jul 28, 2017

Change https://golang.org/cl/51590 mentions this issue: maintner/maintnerd, cmd/coordinator: use maintnerd to find TryBot work

gopherbot pushed a commit to golang/build that referenced this issue Jul 28, 2017
maintner/maintnerd, cmd/coordinator: use maintnerd to find TryBot work
Instead of polling Gerrit every 60 seconds, poll maintnerd every
second. This should improve TryBot-requested to TryBot-running latency
by 30 seconds on average.

(Ideally it'd long poll for changes and not even have ~500ms delay,
but this is an improvement.)

Also, in the process this does most of the work for golang/go#17626.
And this cleans up the repo head tracking complication in the coordinator
and just asks maintner for it instead.

Running benchmarks in the coordinator has been disabled since
2017-06-23 but this CL disables it further, removing some code it'd
need to work. But considering that benchmarking would need code
repairs to get working again anyway, I consider this acceptable in the
short term. I left TODO notes.

Updates golang/go#17626
Updates golang/go#19871

Change-Id: Idab43f65a9cca861bffb37748b036a5c9baee864
Reviewed-on: https://go-review.googlesource.com/51590
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.