deis pull as alternative to git push workflow
#1190
Conversation
| def _build_dockerfile(image, config): | ||
| dockerfile = ["FROM "+image] | ||
| for k, v in config.items(): | ||
| dockerfile.append("ENV {} {}".format(k.upper(), v)) |
|
From an architectural standpoint, there seems to be a lot of confusion on what each component is supposed to accomplish. Why do we even need a docker engine in the controller at all? Maybe we can do the last-mile layer stuff in the builder instead. This would mean a lot of refactoring... though it makes sense that the builder should do the last-mile layer stuff from an architectural point of view. From my point of view (in a perfect world):
Right now we have the builder doing controller stuff (builder is POST'ing a request to the controller to launch an app, relies on CURL'ing the controller to inject envvars at compile-time) and the controller doing builder stuff (last-mile layer configuration should be handled by the builder). Maybe this isn't the right PR to discuss this, but rather we should really take a good look at these two components and try to separate the two from each other if we can. @gabrtv let me know if this should belong in a separate issue, but I believe there is valid discussion to talk about in this PR to prevent code sprawl (post-1.0?) |
|
To add on from internal discussions, I don't like the "controller pulls, controller pushes to registry, fleet job pulls from registry" workflow for this (lotta pushing for just one image to launch), though I don't see any other way since we have to do last-mile configuration on the image, then push it to the registry so the cluster can retrieve the image. Maybe there's a way we can inject those environment variables into the fleet job itself, and just make the fleet job pull from their source registry (whether that'd be DockerHub or an in-house registry)? The app image is already stored somewhere else and I'd assume that users would want to put their in-house registry near their Deis cluster anyways, so maybe we don't even have to use those components for this workflow. |
|
After a ton of discussion with @bacongobbler and others, we've decided to try a different approach to adding last-mile layers -- one that hopefully results in a slight performance gain on deploys, versus the ~2x performance hit from the current approach. Rather than bundling a Docker-in-Docker on the controller, we will try to forking the Docker registry and implementing registry-to-registry image transfers via the Registry API. This feature, combined with last-mile layers implemented on the registry (as we do today) should make for a much more powerful solution. |
|
Note this is being assigned to @bacongobbler to explore registry transfers as an alternative to Docker-in-Docker inside the controller. Thankfully, most of the PR will remain usable. |
|
I don't think this PR is mergeable as is, given the performance implications of the extra layer transfer. If we have to cut 0.10 without 'deis build', so be it.
|
|
@gabrtv it possible to add a third option uploading a tgz file? |
|
@aledbf do you mean like a ROOTFS tarball of your image or a tarball of your application such as https://github.com/deis/example-go/archive/master.tar.gz? |
|
@bacongobbler a tarball of an application (the result of |
|
Jenkins, test this please. |
|
PR's ready for review (barring documentation). I've manually tested pulling both private and public images using
The hacks done to implement registry-to-registry transfers can be seen here: https://github.com/deis/docker-registry/compare/repository-import Rollback still works:
registry logs (looks like the tag was accidentally sent in the API call): |
|
fixed: |
|
PR ready for review once more. Let's chat about it on monday when @gabrtv is back and alive :) |
|
I'm back and partially alive. Let's start readying this for a merge! @bacongobbler let's add docs and get to testing. |
|
I've added documentation which is syntactically correct, but some of the links are broken at the moment. I'll resurrect #1314 and break it down such that we can get some of these nice fixes in so the links work properly. |
The logic has been moved over to the registry, so it's not needed any more.
When target_image was POST'd to the controller through /api/hooks/build, the tag was not originally sent. Reverting the behaviour so `git push deis master` works again.
'sha' is just metadata. it is not used as a tag on the registry. The release ledger is append-only, which means that the latest image should be used by default, with an optional source tag for rollbacks.
This reverts commit 7756f8a.
the registry module will take care of that
some packages in dev_requirements.txt are required for the test suite (such as mock).
Changing the name to something that seems more accurate, as some people associate "build" with `docker build`. It also complements `docker pull` quite nicely, since we are effectively "pulling" images into Deis.
Previously, `deis create` would store session data in a ~/.deis/apps.yml
file for applications with no git root to facilitate `deis pull`. This
file looked similar to this:
{'/tmp/deis-test': 'united-mainsail'}
The issue with this means if you move your application's root directory
somewhere else, the reference to the instance is lost. With this change,
if no git root is present in the application directory, the base name
of the current working directory (in the example above, that would be
'deis-test') is synonymous with the app name. With this in mind,
`deis create` in a git repo retains the old behaviour (via the
auto-generator from the server), but in a non-git directory, it takes
the name of the current working directory. As always, --app works just
like it used to, or you can still manually specify a name with
`deis create <id>`.
If you're in a git repo and you did not supply an app name to `deis create`, the client will fail because app_name was never initialized.
This way, we can bust the cache when we rebase the branch.
this reference exists in #1327 which as of this writing is not merged yet.
|
Jenkins, test this please. |
|
Jenkins is having problems with this due to some of the infrastructural changes. However, we've done a great deal of manual QA and are ready to merge. Hold on to your hats. |
`deis pull` as alternative to `git push` workflow
A regression from #1190 was introduced where the config file used was the sample that came from dotcloud/docker-registry instead of using the confd templated config we use today. Updating the file to upstream's changes with the common layer as well as changing the reference to the file in the Dockerfile fixes this regression.
A regression from #1190 was introduced where the config file used was the sample that came from dotcloud/docker-registry instead of using the confd templated config we use today. Updating the file to upstream's changes with the common layer as well as changing the reference to the file in the Dockerfile fixes this regression.
The new
deis pullcommand is an alternative to thegit pushworkflow that allows promotion of existing Docker images, bit-for-bit, into a Deis application. It pulls the image from your repository and merges it with existing config to create a new release.Overview of Changes
deis pullprocess)deis pullby 33%, which can be seen and reviewed at https://github.com/deis/docker-registry/compare/repository-importHow to Test
Test backwards-compat for existing applications
Install a Fresh Cluster from Master
deis registerdeis clusters:createdeis keys:addCreate an Application
git pushgit commit --allow-empty) andgit pusha newv3curlordeis opento test that the app is upPerform an in-place Upgrade
make uninstall(should retain database data and application units)deis-buildmake build rundeis loginto re-establish a session with the controllerCheck Backwards Compatibility
deis infoanddeis opento check the health of the applicationgit commit --allow-empty) and usegit push deis masterto deploy itdeis opento confirm it worksdeis rollback v2to roll back to an image created on the old codedeis opento confirm it worksgit push deis masterto ensure the initial build worksTest New Functionality
mkdir /tmp/example-go && cd /tmp/example-godeis createto create "example-go"deis pull gabrtv/example-goto deploy from the public Docker Indexdeis opento confirm the app is workingdeis config:set POWERED_BY=blahto confirm config layers are set correctlydeis rollback v2to confirm rollback worksdeis pull registry.bacongobbler.com/helloworldto deploy from an in-house Docker indexdeis opento confirm the app is workingdeis run envto verify that the changes to the run command works