Make way for host-containers: Task-iterations and command-template functionality

[Blizzke]

PR to put out some feelers about handling deployments for projects running in containers on the remote host. This is by no means meant to be "the final solution" (although it can be), but simply an initial attempt to get some discussion going. It does work and does exactly what I wrote it for, for your information :)

Q	A
Bug fix?	No
New feature?	Yes
BC breaks?	No
Deprecations?	No
Fixed tickets	N/A

I did not want to make too big of a change to the base functionality. I also wanted to avoid adding container specific functionality in this project, as that should probably be a recipe.

Managed to come up with the following 2-fold approach to allow the functionality I wanted:

• A Task now has iterate-functionality. This allows you to specify a set of configuration "delta" items and the task will be ran on the host for each of those deltas. In between runs (and after) the replaced values will be restored (or set to null if they were added because deleting is not supported). Idea here is that we want to be able to run a task on one or more containers on the host. It's also the most compact way to add the needed functionality to a Task
• The run()-function can now wrap the command to execute in a so called command_template. As long as this configuration value is not empty (and apply_command_template is not set to false), the template will be used as the command instead. The command to execute is stored in the command-configuration value.

Using this we can achieve running commands within docker on the host fairly easy (assuming the docker/similar recipe included at the bottom is there):

Wrap existing task (move into container)

runTaskInDocker('deploy:vendors', 'my_container_name', ['release_path' => '{{container_source_path}}']);

Everything the "deploy:vendors" task does will now be ran in the my_container_name-container on the host. The release_path variable will - for the duration of the task - be replace with whatever container_source_path is set to, so that we don't have to rewrite any of the original task but it will run fine within the docker container.

Within a container task: run statement on the host

runOnHost('{{bin/docker-compose}} up -d');

Using the runWithoutTemplate functionality, this disables using the command_template for a single callback and thus runs the regular way.

Some explanation up front

Why iterations?
I get that, most of the time, it will probably be a single container, but adding it within a possible iteration just seemed cleaner to me. Like that it also works for both a single and multiple runs of that task. I can see someone setting up a bunch of containers in 1 go with the same task, so why not allow it. The functionality is also not container specific, so who knows what else can be done.

{{?command}}?

The command template has support for tokens that should only be parsed when we're actually about to run the task. This is needed because we want to get the value of the template and "install" it before any parsing is done on the original command.

For example imagine you want to run "{{bin/php}} -i" in a container: If the command_template would simply contain "{{command}}", attempting to get the template value would result in a call to locateBinaryPath('php') since everything is parsed recursively (assuming that set('command', ...) was called already).

locateBinaryPatth would start running commands to try to locate the php executable. Unfortunately these will run on the host instead of in the container.

By delaying substitution for some tokens (like the command), we can fetch the template value, change "{{?command}}" into "{{command}}" and then run the template with normal parsing.
Instead of command -v 'php' this will now result in /usr/bin/docker exec -i my_container_name bash -c 'command -v '\''php'\'''

docker_command_template?
I have been in some situations where I want to be able to specify a different user to run a command, so this allows you to override the template with one of your own, for example:

set('docker_command_template', '{{bin/docker}} exec -i --user not_root {{container}} bash -c {{?command}}');

Curious about your thoughts.

Docker recipe

namespace Deployer;

function runTaskInDocker($task, $containers, $additionalConfiguration = [])
{
    $task = Deployer::get()->tasks->get($task);

    if (!$task) {
        throw new \InvalidArgumentException('Invalid task specified');
    }

    $containers = (array) $containers;
    $iterations = [];
    foreach ($containers as $name) {
        $iteration = [
            'command_template' => has('docker_command_template') ?
                get('docker_command_template') : '{{bin/docker}} exec -i {{container}} bash -c {{?command}}',
            'container' => $name
        ];

        foreach ($additionalConfiguration as $key => $value) {
            $iteration[$key] = $value;
        }

        $iterations[] = $iteration;
    }

    $task->iterate($iterations);
}

function runOnHost($command, $options = []) {
    return runWithoutTemplate(function() use ($command, $options) {
        return run($command, $options);
    });
}

set('bin/docker', function () {
    return runWithoutTemplate(function() {
        return locateBinaryPath('docker');
    });
});

set('bin/docker-compose', function () {
    return runWithoutTemplate(function() {
        return locateBinaryPath('docker-compose');
    });
});

[antonmedv]

Hi,

Thanks for your contribution, very interesting approach to manipulate containers.

But I think it's too early to merge functionality like this into main deployer. What about creating separate repo under deployer/*** and name it experimental? It should be extension for deployer which add this PR functionality. In this repo we can play and look for better API without worrying of breaking something (as it called experimental).

Also this allows of less major version releases of deployer. After API stabilization we can merge such functionality into deployer.

What do you think?

[Blizzke]

Like I said, it's just something to get the discussion going, which it did :)

Impact on the main repository isn't extremely big imo, you have the possible template and the looping in a task (which could be left out of in favor of adding configuration).
Most of the rest of the functionality would be externally in a recipe or something like that.

I don't really see what you mean with "until the API changes stabilize" as the entire base functionality as is right now hasn't changed.

(sorry for closing the PR, bit to eager with keyboard navigation)

[antonmedv]

Please tell me more about how you use containers and deploy them? Why don't oyu use something like kubernetes?

[Blizzke]

Currently the system in place is using docker-compose to setup one or more container networks on a server. For example having a php container for the fpm part of a deploy means having to run all artisan commands in that container as the host system doesn't have PHP installed

[Putr]

Has there been any decisions? Will this be merged?

[antonmedv]

Nope, sorry. Still didn't found time to review all of this.

If someone can provide example project which is using this functionality it will be much better.

[jasperf]

We would like to use it with Laradock (Docker for Laravel) . Not just for local development, but for production as well. We were discussing issues here #1636 as well.

[jasperf]

Nope, sorry. Still didn't found time to review all of this.

If someone can provide example project which is using this functionality it will be much better.

Did add an example project and details at #1770 . Here I show the Laradock setup I use with docker-compose to launch several containers including one loading php-fpm and the way you can run commands inside that container or workspace to make PHP work from within a container. I also show how that repo works with the Laravel repo and how I set up the server with Ansible. Only now I am kind of forced to do git updates only using tagging, but I really would like Deployer to work here. So think the explanation there with repo links could help. And I must say that this recipe suggested here in 2017 could possibly be an awesome way to run php commands inside the container post ssh-ing into the host server.

[vesper8]

thank you for all the information you've shared @jasperf, it's very handy for someone that is just taking his first steps at implementing CI and using Docker. As someone that's used Deployer for over 2 years, I really hope (and figure it would make a lot of sense) that a future update makes it compatible.. so we don't have to learn yet another tool and keep using Deployer for many more years =D