-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ref(*): consolidate builder into fewer Go binaries #72
Conversation
removing |
logs from release start until the hang: 10.168.0.1 "POST /v2/hooks/build HTTP/1.1" 200 56 "deis-builder"
10.168.0.1 "POST /v2/hooks/push HTTP/1.1" 201 423 "deis-builder"
10.168.0.1 "POST /v2/hooks/config HTTP/1.1" 200 194 "deis-builder"
INFO gotest: build gotest-7e00396 created
INFO gotest: release gotest-v4 created
INFO [gotest]: arschles deployed 816b0de
DEBUG deploy gotest_v4.web, img gotest:git-816b0de22a15a89e773a19ffe20830eb683d41f9, params {u'num': 0, u'tags': {}, u'envs': {}, u'cpu': {}, u'version': u'v4', u'memory': {}, u'aname': u'gotest'}, cmd "bash -c 'example-go'"
DEBUG scaling release gotest-v4-web to 1 out of final 1 |
Testing InstructionsAssuming you have no deis cluster installed:
|
Is quay.io/arschles/builder:go-rewrite up to date with all the new changes? happy to build my own image if this isn't automated :) |
@bacongobbler that image is not up to date, but it's pushing now. If you want to build yourself, here's the command: IMAGE_PREFIX=$MY_REPO_NAME VERSION=go-rewrite make build docker-build docker-push |
@bacongobbler ok, Also, to answer your question, building images for this PR isn't automated. |
return fmt.Errorf("copying %s to %s (%s)", appTgz, tarURL, err) | ||
} | ||
|
||
log.Info("Starting build") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a long pause right here while we're waiting to boot the pod. Why not something like Starting build... but first, coffee!
so people know that it's intentionally going to take a while
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could do that, but I'd rather do it in a future PR, so that the logs don't change from what was previously here. can you create an issue?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay I'll file another issue
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks!
My build was caught in a loop forever with the following:
My guess is that it we should use the short sha here? Also seems like the ol' |
@bacongobbler I'll look at removing the remote prefix. Also, are you up to date on eveything, and running on, the That endless loop is likely not related to the short sha issue - instead, it's more likely that the slugbuilder didn't upload the slug to the expected location because it failed. Do you see any logs above that indicate the failure? |
Looks like I am using 27fd67f so I'm out of date. I'll rebase and try again later.
I dunno. My screen just started dumping that message like no tomorrow, which was enough to hit my scrollback limit. I'll try another build and see if I get anything useful. It's likely that me being outdated could have caused that issue. |
thanks @bacongobbler |
@arschles for reference, this is what stripped the remote prefix in the old gitreceive: https://github.com/progrium/gitreceive/blob/0cc00c01aa6acb8a1db72d4b745675ed23ce591f/gitreceive#L119-L124 |
@bacongobbler thanks, that saves me a ton of searching |
also needs to send info logs to STDOUT so that the strip prefix logic works properly
…itreceive package
@bacongobbler ok, I added the strip prefix logic back in arschles@3ac4e48. Sample output from a ENG000656:example-go aaronschlesinger$ git commit -m "asd" --allow-empty && git push deis master
[master 27a04f1] asd
Counting objects: 85, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (80/80), done.
Writing objects: 100% (85/85), 18.28 KiB | 0 bytes/s, done.
Total 85 (delta 40), reused 0 (delta 0)
remote: mc: <ERROR> Unable to make bucket ‘http://10.171.250.102:9000/git’. The requested bucket name is not available.
-----> Starting build
-----> Go app detected
-----> Checking Godeps/Godeps.json file.
-----> Installing go1.4.2... done
-----> Running: godep go install -tags heroku ./...
-----> Discovering process types
Procfile declares types -> web
-----> Compiled slug size is 1.7M
-----> Build complete.
-----> Launching app.
-----> Launching...
-----> Done, gotest:v1 deployed to Deis
-----> Use 'deis open' to view this application in your browser
-----> To learn more, use 'deis help' or visit http://deis.io
To ssh://git@deis.130.211.154.75.xip.io:2222/gotest.git
* [new branch] master -> master |
I'm on the latest version (
...Which then spews that error to infinity. Logs from minio:
|
@bacongobbler the first error from the Finally, can you run the builder with |
Yep, here's a gist: https://gist.github.com/bacongobbler/3ea3cd8362e542d649aa It loops forever on the |
Ok, let's see what the output is when |
Unrelated networking issues with my cluster. LGTM! |
ref(*): consolidate builder into fewer Go binaries
This is a big patch to refactor part of this component to Go from shell scripts. See testing instructions below.
Guide to Changes
pkg/gitreceive
replaces therootfs/etc/confd/templates/builder
androotfs/etc/confd/templates/receive
scripts, which were invoked by the script generated by thePreReceiveHookTpl
template inpkg/git/git.go
. This is by far the biggest change in this PRMakefile
now only builds aboot
binaryboot.go
now accepts the following commands:server
runs the SSH server (to acceptgit push
es) and the fetcher, which is an old filesystem based object storegit-receive
is run in the pre-receive hook by the SSH server in response to everygit push
pkg/src
was previously used by therootfs/etc/confd/templates/builder
androotfs/etc/confd/templates/receiver
scripts, and was merged into code inside ofpkg/gitreceive
pkg/git/git.go
was refactored to split the repo creation step and the pre-receive hook rendering steppkg/gitreceive
)TODOs
io.Writer
. This functionality is currently used to collect stderr logs in abytes.Buffer
and then print out those logs on errorpkg/src
into a singlegitreceive
binaryAs a result of the final bullet, the entire builder will be a single binary. The code changes involved to do that are below:
boot.go
to accept 2 commands (uses cli to build the commands) -server
andgit-receive
.server
starts the ssh server (and fetcher for now, but not after Remove the Fetcher #6), andgit-receive
is started by the git post-receive hookpkg/gitreceive
directory and packagerootfs/etc/confd/templates/{builder,receiver}
scripts to Go, and put them inpkg/gitreceive
Makefile
to just compileboot.go
into a singleboot
executable, which still lives inrootfs/bin/boot
Dockerfile
's entrypoint to include theserver
command onboot
log
package for pretty-printing logspreReceiveHookTplStr
to set the appropriate environment vars and call/bin/boot git-receive
getBuilderKey
)remote:
prefixes. See relevant gitreceive code for how it was doneNote that this PR will not be removing confd due to the
authorized_keys
file. See #83 (now replaced by #81) for thatFixes #29
Fixes #37
Fixes #51
Depends on #71