Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref(*): consolidate builder into fewer Go binaries #72

Merged
merged 150 commits into from
Jan 14, 2016

Conversation

arschles
Copy link
Member

This is a big patch to refactor part of this component to Go from shell scripts. See testing instructions below.

Guide to Changes

  • Everything inside if pkg/gitreceive replaces the rootfs/etc/confd/templates/builder and rootfs/etc/confd/templates/receive scripts, which were invoked by the script generated by the PreReceiveHookTpl template in pkg/git/git.go. This is by far the biggest change in this PR
  • The Makefile now only builds a boot binary
  • boot.go now accepts the following commands:
    • server runs the SSH server (to accept git pushes) and the fetcher, which is an old filesystem based object store
    • git-receive is run in the pre-receive hook by the SSH server in response to every git push
  • Related to the previous, everything inside of pkg/src was previously used by the rootfs/etc/confd/templates/builder and rootfs/etc/confd/templates/receiver scripts, and was merged into code inside of pkg/gitreceive
  • pkg/git/git.go was refactored to split the repo creation step and the pre-receive hook rendering step
  • A log package was added, and is used inside of the git-receive hook (everything under pkg/gitreceive)
  • Builder manifests were added to support public object storage systems

TODOs

  • Run the SSH server and fetcher HTTP server in the same process (no more separate binary build for fetcher)
  • Also output stderr for the git-receive hook to auxiliary io.Writer. This functionality is currently used to collect stderr logs in a bytes.Buffer and then print out those logs on error
  • Rewrite the entire git-receive hook in Go. This work will consolidate all of the binaries that are built from the code in pkg/src into a single gitreceive binary
  • Add instructions in this PR on how to test (see ref(*): consolidate builder into fewer Go binaries #72 (comment))

As a result of the final bullet, the entire builder will be a single binary. The code changes involved to do that are below:

  • Refactor boot.go to accept 2 commands (uses cli to build the commands) - server and git-receive. server starts the ssh server (and fetcher for now, but not after Remove the Fetcher #6), and git-receive is started by the git post-receive hook
  • Add a pkg/gitreceive directory and package
  • Port the rootfs/etc/confd/templates/{builder,receiver} scripts to Go, and put them in pkg/gitreceive
  • Change the Makefile to just compile boot.go into a single boot executable, which still lives in rootfs/bin/boot
  • Change the Dockerfile's entrypoint to include the server command on boot
  • Add a log package for pretty-printing logs
  • Add all necessary dependencies to the glide files
  • Change preReceiveHookTplStr to set the appropriate environment vars and call /bin/boot git-receive
  • Implement the etcd key fetching logic in the git receive hook (a function called getBuilderKey)
  • Port the changes in feat(builder): add support for custom buildpack url #78 to here
  • Port the applicable changes in feat(builder): adjust dockerbuilder and slugrunner templates #74 to here
  • Port the changes in feat(builder): remove extra git archive tar #88 to here
  • Remove the remote: prefixes. See relevant gitreceive code for how it was done

Note that this PR will not be removing confd due to the authorized_keys file. See #83 (now replaced by #81) for that

Fixes #29
Fixes #37
Fixes #51
Depends on #71

@arschles arschles self-assigned this Dec 31, 2015
@arschles arschles added this to the v2.0-beta1 milestone Dec 31, 2015
@arschles arschles changed the title ref(*): beginning rewrite for all of builder to Go (WIP) ref(*): consolidate all of builder to 1 Go binary (WIP) Dec 31, 2015
@arschles arschles changed the title ref(*): consolidate all of builder to 1 Go binary (WIP) ref(*): consolidate all of builder to a single Go binary (WIP) Dec 31, 2015
@arschles arschles changed the title ref(*): consolidate all of builder to a single Go binary (WIP) ref(*): consolidate builder into fewer Go binaries Jan 4, 2016
@arschles
Copy link
Member Author

arschles commented Jan 8, 2016

removing awaiting review because launching the app takes an unusually long time after this change. investigating

@arschles
Copy link
Member Author

arschles commented Jan 8, 2016

logs from release start until the hang:

10.168.0.1 "POST /v2/hooks/build HTTP/1.1" 200 56 "deis-builder"
10.168.0.1 "POST /v2/hooks/push HTTP/1.1" 201 423 "deis-builder"
10.168.0.1 "POST /v2/hooks/config HTTP/1.1" 200 194 "deis-builder"
INFO gotest: build gotest-7e00396 created
INFO gotest: release gotest-v4 created
INFO [gotest]: arschles deployed 816b0de
DEBUG deploy gotest_v4.web, img gotest:git-816b0de22a15a89e773a19ffe20830eb683d41f9, params {u'num': 0, u'tags': {}, u'envs': {}, u'cpu': {}, u'version': u'v4', u'memory': {}, u'aname': u'gotest'}, cmd "bash -c 'example-go'"
DEBUG scaling release gotest-v4-web to 1 out of final 1

@arschles
Copy link
Member Author

Testing Instructions

Assuming you have no deis cluster installed:

  • helm repo add https://github.com/deis/charts
  • helm fetch deis/deis-dev
  • Edit the ~/.helm/workflow/charts/deis-dev/manifests/deis-builder-rc.yaml to use the quay.io/arschles/builder:go-rewrite image
    • You can also add DEBUG=true to the env to see debug output on the builder
  • helm install deis-dev
  • git clone https://github.com/deis/example-go.git && cd example-go
  • deis register deis.$INGRESS_IP.xip.io (use the email address in your ~/.ssh/id_rsa.pub)
  • deis keys:add ~/.ssh/id_rsa.pub
  • git remote remove deis
  • deis create gotest
  • Optional: deis config:set BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-go#6eeb09f to set a custom buildpack URL
  • git push deis master
  • curl gotest.$INGRESS_IP.xip.io to see the newly deployed app in action

@bacongobbler
Copy link
Member

Is quay.io/arschles/builder:go-rewrite up to date with all the new changes? happy to build my own image if this isn't automated :)

@arschles
Copy link
Member Author

@bacongobbler that image is not up to date, but it's pushing now. If you want to build yourself, here's the command:

IMAGE_PREFIX=$MY_REPO_NAME VERSION=go-rewrite make build docker-build docker-push

@arschles
Copy link
Member Author

@bacongobbler ok, quay.io/arschles/builder:go-devel is up to date now.

Also, to answer your question, building images for this PR isn't automated.

return fmt.Errorf("copying %s to %s (%s)", appTgz, tarURL, err)
}

log.Info("Starting build")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a long pause right here while we're waiting to boot the pod. Why not something like Starting build... but first, coffee! so people know that it's intentionally going to take a while

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could do that, but I'd rather do it in a future PR, so that the logs don't change from what was previously here. can you create an issue?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay I'll file another issue

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#94

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@bacongobbler
Copy link
Member

My build was caught in a loop forever with the following:

remote: mc: <ERROR> Unable to stat ‘http://10.247.115.24:9000/git/home/go:git-5450cbcdaaf9afe6fadd219c94ac9c449bd62413/push’. The specified key does not exist.

My guess is that it we should use the short sha here?

Also seems like the ol' remote: prefix is back. History on this bug here: deis/deis#1216

@arschles
Copy link
Member Author

@bacongobbler I'll look at removing the remote prefix. Also, are you up to date on eveything, and running on, the deis-dev chart?

That endless loop is likely not related to the short sha issue - instead, it's more likely that the slugbuilder didn't upload the slug to the expected location because it failed. Do you see any logs above that indicate the failure?

@bacongobbler
Copy link
Member

Looks like I am using 27fd67f so I'm out of date. I'll rebase and try again later.

Do you see any logs above that indicate the failure?

I dunno. My screen just started dumping that message like no tomorrow, which was enough to hit my scrollback limit. I'll try another build and see if I get anything useful. It's likely that me being outdated could have caused that issue.

@arschles
Copy link
Member Author

thanks @bacongobbler

@bacongobbler
Copy link
Member

@arschles for reference, this is what stripped the remote prefix in the old gitreceive: https://github.com/progrium/gitreceive/blob/0cc00c01aa6acb8a1db72d4b745675ed23ce591f/gitreceive#L119-L124

@arschles
Copy link
Member Author

@bacongobbler thanks, that saves me a ton of searching

Aaron Schlesinger added 3 commits January 14, 2016 08:57
@arschles
Copy link
Member Author

@bacongobbler ok, I added the strip prefix logic back in arschles@3ac4e48. Sample output from a git push is below. Notice that warnings & errors still have the remote: prefix, which I believe is ok. Let me know if you disagree.

ENG000656:example-go aaronschlesinger$ git commit -m "asd" --allow-empty && git push deis master
[master 27a04f1] asd
Counting objects: 85, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (80/80), done.
Writing objects: 100% (85/85), 18.28 KiB | 0 bytes/s, done.
Total 85 (delta 40), reused 0 (delta 0)
remote: mc: <ERROR> Unable to make bucket ‘http://10.171.250.102:9000/git’. The requested bucket name is not available.
-----> Starting build
-----> Go app detected
-----> Checking Godeps/Godeps.json file.
-----> Installing go1.4.2... done
-----> Running: godep go install -tags heroku ./...
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size is 1.7M
-----> Build complete.
-----> Launching app.
-----> Launching...
-----> Done, gotest:v1 deployed to Deis

-----> Use 'deis open' to view this application in your browser

-----> To learn more, use 'deis help' or visit http://deis.io

To ssh://git@deis.130.211.154.75.xip.io:2222/gotest.git
 * [new branch]      master -> master

@bacongobbler
Copy link
Member

I'm on the latest version (192.168.0.18:5000/deis/builder:git-74ded2e) and I'm still seeing minio errors:

><> git push deis master
Counting objects: 75, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (40/40), done.
Writing objects: 100% (75/75), 18.20 KiB | 0 bytes/s, done.
Total 75 (delta 30), reused 75 (delta 30)
remote: mc: <ERROR> Unable to make bucket ‘http://10.247.110.165:9000/git’. The requested bucket name is not available.
-----> Starting build
-----> Go app detected
remote: mc: <ERROR> Unable to stat ‘http://10.247.110.165:9000/git/home/foo:git-5450cbcdaaf9afe6fadd219c94ac9c449bd62413/push’. The specified key does not exist.

...Which then spews that error to infinity. Logs from minio:

><> kd logs deis-minio-al13u
minio server /home/minio/

AccessKey: 8TZRY2JRWMPT6UMXR6I5  SecretKey: gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk

To configure Minio Client.

        $ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
        $ chmod 755 mc
        $ ./mc config host add http://localhost:9000 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
        $ ./mc mb localhost/photobucket
        $ ./mc cp ~/Photos... localhost/photobucket

Starting minio server:
Listening on http://127.0.0.1:9000
Listening on http://10.246.28.79:9000
time="2016-01-14T19:03:55Z" level=error msg="MakeBucket failed." Error={Bucket exists: git fs.BucketExists [{125 pkg/fs/fs-bucket.go fs.Filesystem.MakeBucket map[]} {255 bucket-handlers.go main.CloudStorageAPI.PutBucketHandler map[]}] map[host.lang:go1.5.2 mem.total:9.3MB mem.heap.used:1.9MB host.arch:amd64 host.name:deis-minio-al13u mem.used:1.9MB mem.heap.total:5.7MB host.os:linux host.cpus:4]} 
time="2016-01-14T19:04:37Z" level=error msg="MakeBucket failed." Error={Bucket exists: git fs.BucketExists [{125 pkg/fs/fs-bucket.go fs.Filesystem.MakeBucket map[]} {255 bucket-handlers.go main.CloudStorageAPI.PutBucketHandler map[]}] map[host.os:linux host.cpus:4 mem.total:9.3MB mem.heap.used:1.9MB mem.heap.total:5.7MB host.arch:amd64 host.name:deis-minio-al13u host.lang:go1.5.2 mem.used:1.9MB]}

@arschles
Copy link
Member Author

@bacongobbler the first error from the git push logs is expected (builder tries to create the bucket it needs every time it runs), but the second of course does not. Do logs continue after the second error? If so, I'm assuming they loop forever on the mc ls command?

Finally, can you run the builder with DEBUG="true" and run again?

@bacongobbler
Copy link
Member

Yep, here's a gist: https://gist.github.com/bacongobbler/3ea3cd8362e542d649aa

It loops forever on the mc ls command, yeah

@arschles
Copy link
Member Author

Ok, let's see what the output is when DEBUG="true" is on. I fixed an issue a few days ago that was causing the endless mc ls loop, and I haven't been able to repro since.

@bacongobbler
Copy link
Member

Unrelated networking issues with my cluster. LGTM! :shipit:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants