Updates to the TPS pipeline to render tiles end-to-end #9
Conversation
Eliminate the need for GOPATH. The go binaries are expected to already be installed as pre-requisites.
* add docker (needed for building images) * add user to docker group * add tz-missing-meta-tiles-write command * GOPATH env var is needed to find location of binaries
batch-setup/make_tiles.py
Outdated
| @@ -102,7 +102,7 @@ def vpc_of_sg(sg_id): | |||
| 'missing-meta-tiles-write'): | |||
| config = { | |||
| 'logging': { | |||
| 'config': 'logging.conf.sample' | |||
| 'config': 'batch-setup/logging.conf.sample' | |||
There was a problem hiding this comment.
Should we have the full path to the logging config (i.e: os.path.join(os.path.dirname(os.path.realpath(__file__)), 'logging.conf.sample') or similar), so that it doesn't matter what the working directory is?
There was a problem hiding this comment.
Yup, I'm always running all commands from the tileops root anyway because I'm not sure if there would be other problems, but we certainly don't lose anything by using the full path and it's easy to do. db0cecb
batch-setup/provision.sh
Outdated
| @@ -24,6 +26,7 @@ for repo in raw_tiles tilequeue vector-datasource tileops; do | |||
| done | |||
| cat > /usr/local/etc/planet-env.sh << eof | |||
| #!/bin/bash | |||
| export GOPATH=/usr/local | |||
There was a problem hiding this comment.
This seems like it would stomp on other stuff in /usr/local/. Should we put it in /usr/local/gopath and add /usr/local/gopath/bin to $PATH in /etc/profile?
There was a problem hiding this comment.
On second thoughts, it looks like the changes to docker/missing-meta-tiles-write/Makefile mean that we don't need $GOPATH at all?
There was a problem hiding this comment.
This seems like it would stomp on other stuff in /usr/local/
Is there a problem with that though?
I've been trying to move away from having any go build requirements on the instance itself. I put the binaries themselves onto s3, and we're just pulling those down on startup. I wasn't sure if there was anything else that needed it, so just setting GOPATH seemed like a faster way to sidestep the problem.
There's code in the python that expects GOPATH to be set, and from what I can tell it's just to find the commands in the first place. Should we just explicitly set that instead? ie take in the path to where those commands are via a command line arg, and not set GOPATH at all?
There was a problem hiding this comment.
Perhaps we can make it optional? When I've been running this locally, $GOPATH/bin was never in my $PATH (and I get kinda paranoid about not having random stuff in my $PATH), so $GOPATH seemed like a sensible way to do it. Do you think we could change the script so that it looks for the binary in $GOPATH if it's set, in $PATH if not, and errors out if it doesn't find the binary in either?
There was a problem hiding this comment.
With my pedantic hat on, searching for commands in $GOPATH at runtime isn't ideal. It's just meant to have meaning at compile time. $PATH really does seem like the right place for it. If we don't want to have these commands in $PATH, we could just pass in the full paths to them via config or command line, or just put all of these in a separate base path these we send in via an arg.
I took the hat off, and made the change in 18315b1 :)
… TPS/automation code.
…sed for finding missing RAWR tiles.
…ead missing tile outputs.
…ket arguments are non-empty.
…submitted. This can be really useful for shortening the compile-run-debug cycle.
…rmat-type for consistency.
…ge, and whether the list printed is the missing tiles or the present tiles.
…edly bouncing an environment. Added extra wait for job queue deletion, as this would occasionally take long enough to cause an error with compute environment deletion.
…on) and hash-prefix keys.
… amount of memory needed for the TPS instance. Add compressed output for the tile list and compressed input for the low/high split, reducing the amount of disk space needed.
zerebubuth
changed the title
|
There's still a bunch of stuff that needs adding to this, but it's become a monster PR and it would be more readable to make smaller PRs for the extra changes. |
These are the updates that I've been tracking locally up until the step to make rawr tiles.