how to use mco applications

[logicminds]

I have a bunch of applications that bring together multiple agents to perform complex orchestration tasks. How do you plane to support applications in addition to agents?

[iNecas]

Hi,

For now, the mcollective integration if Foreman is focusing more on single remote tasks (such as install/update/remove package etc.) Said that, there is no reason more complex orchestration couldn't be doable with mcoflow approach: it's kind of prepared to do so. It's also probably much more useful than the single commands style of remote execution.

Could you describe your usage of such an orchestration task (ideally with some real examples) that we could take as a use-case we could try to solve?

[logicminds]

I have a rolling update application that calls a few agents and performs the following. So all of this is done with one command since the application class can talk to multiple agents where as the agent class cannot (afaik).

1. sets system in MAINT mode in proxy pool
2. runs puppet on system via puppet agent
3. run health agent to verify things are good
4. sets system to enable mode in proxy pool

[iNecas]

What about being able to specify this in some kind of data structure (represented for example in json), building the workflow from agents calls. In this example something like:

[
  {"agent":"proxy_disable", "args":{"hostname":"updated.host.example.com"}},
  {"agent":"puppetrun", "filter":{"identity":"updated.host.example.com"}},
  {"agent":"healthcheck", "filter":{"identity":"updated.host.example.com"}},
  {"agent":"proxy_disable", "args":{"hostname":"updated.host.example.com"}}
]

This would be doable even with this implementation (approx 20 lines of new code).

I can also imagine this put in the context of Foreman in general, not just mcollective. Let's consider this example:

[ {"concurrence":[
    {"action":"Foreman::ProvisionHost","args":{"hostname":"foreman.example.com","hostgroup":"foreman" /** ...**/}},
    {"action":"Foreman::ProvisionHost","args":{"hostname":"foreman-proxy.example.com","hostgroup":"foreman" /** ...**/}}
  ]
  {"action":"Mcoflow::ForemanProxyRegister","filter":{"identity":"foreman-proxy.example.com"}, "args":{"master":"foreman.example.com"}}
]

Now imagine you could construct the template with the foreman templating system, that would output the json file.

This would effectively give us "orchestration templates" (probably something like HEAT openstack does, but would be usable in everything Foreman supports). We started the Dynflow project to be able to support all kinds of orchestration (mcollective is just one example of using it). It's designed to support any orchestration that happens inside Foreman (including network services orchestration etc.). In short, if we used dynflow for orchestration in general, we could be able to do any of this crazy things. Right now, we're working primarily on getting Katello on Dynflow to stabilize the orchestration of content management. The main goals we try to address is: 1) reporting the status of the tasks that are happening in background and 2) being able to resume the complex process if something goes wrong. Dynflow is quite far with being able to do that + we work on improving that every day.

I'm also working on a foreman plugin that add the notion of task into foreman, and it will be connected to the Dynflow library, so that the orchestration tasks will be visible (something like I show with Katello in the second part of the screencast

Anyway, this is kind of mid-term goal. I think however, that the mcollective part might be useful already. What do you think? Would something like this work in your case? So instead of writing the application you would be able to combine the agents this way.

As a side-effect, you would get the possibility to be able to resume the update if something goes wrong (for example if health agent reprots issue, the process pauses, you could fix that manually and resume the process), track the progress and other things that we need to solve for other parts of Foreman and Katello.