Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: productionize maintner #19866

Closed
bradfitz opened this issue Apr 6, 2017 · 20 comments

Comments

Projects
None yet
4 participants
@bradfitz
Copy link
Member

commented Apr 6, 2017

Move maintner to GKE & give it a hostname.

@bradfitz bradfitz self-assigned this Apr 6, 2017

@gopherbot gopherbot added this to the Unreleased milestone Apr 6, 2017

@gopherbot gopherbot added the Builders label Apr 6, 2017

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2017

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 12, 2017

Yup!

It'll likely look similar to https://github.com/golang/build/blob/master/cmd/gitmirror/Dockerfile which you'll probably gag looking at.

But so far that's the best mechanism I've found to pin versions of Go deps in container images.

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2017

I saw it cringed and then realized it was actually an inventive way to do a versioning system lol

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 12, 2017

Really there should be one Dockerfile to build the binary, and then a Makefile (cringe?) or other driver tool to copy the binary out of Dockerfile 1 into super-minimal Dockerfile 2 that lacks the go compiler or any go source code.

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2017

eventually we can use this new feature for nested builds: moby/moby#32063

BUT I have a makefile technique for other repos with the one dockerfile to build, one to actually use song and dance

@gopherbot

This comment has been minimized.

Copy link

commented Apr 12, 2017

CL https://golang.org/cl/40478 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 12, 2017

maintner/maintnerd: Add Dockerfile and Makefile
Adds a Dockerfile and Makefile so maintnerd can be deployed to GKE.

Updates golang/go#19866

Change-Id: I83d8d409c2acab3022c2b74516157d32515fe28b
Reviewed-on: https://go-review.googlesource.com/40478
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2017

can you assign this one to me so i remember to close it when i make them all minimal :)

@gopherbot

This comment has been minimized.

Copy link

commented Apr 13, 2017

CL https://golang.org/cl/40572 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 17, 2017

maintner: move record reading and formatting code to its own package
It's starting to pollute the package, and seems separable. I also plan
to use it more from elsewhere in upcoming CLs.

Updates golang/go#19866

Change-Id: I7b2add37f74ed42c2f78939924f19d8322179823
Reviewed-on: https://go-review.googlesource.com/40868
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Apr 17, 2017

CL https://golang.org/cl/40868 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2017

maintner/maintnerd: change Dockerfile to FROM scratch
Before:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/maintnerd     786MB

After:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/maintnerd     8.85MB

Updates golang/go#18817
Updates golang/go#19866

Change-Id: Ibfdadbaf043e44b26d5807c1da45746e934a7dce
Reviewed-on: https://go-review.googlesource.com/40572
Reviewed-by: Kevin Burke <kev@inburke.com>
@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Apr 21, 2017

Thanks for the Dockerfile. I'll try to productize this soon. Still fighting a Github sync issue that's easier to debug locally for now.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 27, 2017

CL https://golang.org/cl/41824 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 29, 2017

CL https://golang.org/cl/42146 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 29, 2017

autocertcache: move coordinator's GCS autocert.Cache impl to its own …
…package

This will also be used by maintner.golang.org shortly.

Updates golang/go#19866

Change-Id: Id952065831920a206e3cb97bd1f451ceaea34927
Reviewed-on: https://go-review.googlesource.com/42146
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

gopherbot pushed a commit to golang/build that referenced this issue Apr 29, 2017

maintner: add support for storing mutations on Google Cloud Storage
And Kubernetes config.

Updates golang/go#19866

Change-Id: I67b9c93c9c931c19d5c4275a55753ee146e13f21
Reviewed-on: https://go-review.googlesource.com/41824
Reviewed-by: Kevin Burke <kev@inburke.com>
@gopherbot

This comment has been minimized.

Copy link

commented Apr 29, 2017

CL https://golang.org/cl/42148 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 29, 2017

maintner: flesh out godata package, add caching network mutation source
Also, the maintner.Corpus constructor is now gone. An API is added for
maintnerd to become the leader and specify the cache dir.

Updates golang/go#19866

Change-Id: Ia726aa00ca1337b6c130cfee040ff9a1f935d0c2
Reviewed-on: https://go-review.googlesource.com/42148
Reviewed-by: Kevin Burke <kev@inburke.com>
@gopherbot

This comment has been minimized.

Copy link

commented May 1, 2017

CL https://golang.org/cl/42184 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 1, 2017

maintner: fix netsource Update offset errors.
When resuming from a point, the reclog reading code would double check
the record headers offsets but because netsource MutationSource would
seek on the *os.File, what reclog saw and expected didn't match what
it read.

Also move temporary 5 second sleep to 2 seconds and adjust some
logging.

Updates golang/go#19866

Change-Id: I66d1f9df8bb36cf028b715ddd284cb10bc74b45b
Reviewed-on: https://go-review.googlesource.com/42184
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

kevinburke added a commit to kevinburke/build that referenced this issue May 2, 2017

maintner: fix netsource Update offset errors.
When resuming from a point, the reclog reading code would double check
the record headers offsets but because netsource MutationSource would
seek on the *os.File, what reclog saw and expected didn't match what
it read.

Also move temporary 5 second sleep to 2 seconds and adjust some
logging.

Updates golang/go#19866

Change-Id: I66d1f9df8bb36cf028b715ddd284cb10bc74b45b
Reviewed-on: https://go-review.googlesource.com/42184
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@AlekSi

This comment has been minimized.

Copy link
Contributor

commented May 4, 2017

There are data races that should be fixed. Noticed with

go get -u -v -race golang.org/x/build/maintner/...
maintnerd -config=go
@bradfitz

This comment has been minimized.

Copy link
Member Author

commented May 4, 2017

gopherbot pushed a commit to golang/build that referenced this issue May 4, 2017

maintner: fix data race in github sync
The lastUpdated field can only be used by the sync goroutine, but we
did labels and milestones concurrently in their own goroutines.

Remove the update because it was redundant with the caller's update
anyway.

Noticed in:
golang/go#19866 (comment)

Change-Id: I4b0270a00eaf2994ed096631e24a7bd75d0210ab
Reviewed-on: https://go-review.googlesource.com/42615
Reviewed-by: Kevin Burke <kev@inburke.com>
@AlekSi

This comment has been minimized.

Copy link
Contributor

commented May 4, 2017

Another thing: there is no handling of termination signals in maintnerd. Ctrl+C instantly kills the processes without any graceful context cancelations. If I understand correctly, this greatly increases the chances of data corruption (cache becoming out-of-sync), especially during initial sync.

I'm interested in helping with this if dust is settled enough in this place.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented May 4, 2017

Lack of graceful shutdown is not a problem because all robust programs must be able to also survive un-graceful shutdowns, which maintnerd should. It has a pretty paranoid on-disk record format.

@gopherbot

This comment has been minimized.

Copy link

commented May 17, 2017

CL https://golang.org/cl/43557 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 17, 2017

maintnerd/maintnerd: add --config=godata mode, to run using using godata
For upcoming API development, permit running maintnerd in non-syncing
(no new mutations logged) mode, using only the data from the godata
package.

Updates golang/go#19866
Updates golang/go#20222

Change-Id: I8c66c7f5e29b5702b481da0fec8f270f93f484a9
Reviewed-on: https://go-review.googlesource.com/43557
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented May 17, 2017

CL https://golang.org/cl/43560 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 18, 2017

maintnerd: start of gRPC service for maintner
Also, update docs on how to re-generate.
Switches to using go4.org/grpc and go4.org/grpc-codegen/protoc-gen-go4grpc
for now.

Updates golang/go#19866
Updates golang/go#20222

Change-Id: Ifa8a123fca2a30f17270c3c558b7395a02064eae
Reviewed-on: https://go-review.googlesource.com/43560
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

@bradfitz bradfitz closed this May 18, 2017

@golang golang locked and limited conversation to collaborators May 18, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.