Add support for requestedHostId to rancher-compose

[deitch]

The scheduling rules have default host_label for docker version and linux kernel version, see http://docs.rancher.com/rancher/rancher-compose/scheduling/ "Automatically Applied Host Labels." These let me easily schedule things with:

labels:
    io.rancher.scheduler.affinity:host_label: io.rancher.host.docker_version=1.9.1

Can we add the hostname? If I want to schedule a service on a specific host (common use case is performance testing where very specific hardware is required), I should be able to do:

labels:
    io.rancher.scheduler.affinity:host_label: io.rancher.host.hostname=baremetalserver01

I can do this now, but only if I add the label manually every time I add a host. It is redundant work, easy to forget and make mistakes... yet should be really easy to automate from the Rancher side, since we already add 2 host labels on host creation.

[roynasser]

I assume it could work to add a label through the Add options with something like CATTLE_HOST_LABELS='io.rancher.scheduler.affinity%3Ahost_label%3A%20io.rancher.host.hostname=$(echo $HOSTNAME)' being sent to the agent exec command... This opens it up to lots of possibilities... depending on host OS/packages installed you can script "dynamic" values for host memory, cpu, etc,etc... And it would be a process you would do once per host provisioning, independent of host count...

At the moment, however, the UI doesnt let you add a $ sign, its urlencoding it or something in order to send the string value to the docker agent... this would need to be changed/supported in the UI which will result in a label with the command, instead of the output of the command as the label...

it may be overkill, but perhaps one option is to create "default labels" (per env?) that get added to all agents when you go into the Add screen (or to just remember settings for previous host add?), be it for a custom box or a box provisioned by rancher itself...

[deitch]

It could, and I could manually add it, but Rancher already adds some default labels in every environment as of 0.46.0. Hostname seems like an obvious one.

[roynasser]

I dont disagree, I was merely trying to get you a quick workaround to add host label to as many machines as you provision quickly...

I tried searching in some of the repos to see where its defining the docker_version and other labels... I couldnt easily find it to propose a PR, I found some reference to it in one of the repos, but it seemed like constant declarations, but I'm not certain... I'm still having a look in the repos see if I find where its adding it, should be quite simple to add the hostname...

[deitch]

Oh. In that case, allow me to amend my previous comment to: "thank you."

If there is a simple workaround, that would be really helpful. Right now, I have to tell my admins to add the label with each new host. That can get out of hand pretty quickly....

[roynasser]

No worries... I found some reference to digitalocean deployments but its not that... weird, I tried searching in cattle, ui, agent, and a bunch of other repos and cant see where its setting that.... very strange, it must be right under my nose...

What type of deployment are you using? In case its a private deployment, you can have your admins just add that custom label to the docker agent init string, I'm trying to find where to add that tho...

[deitch]

Private, yeah. If there is some way to automate it, really would help.

[vincent99]

sudo docker run -e CATTLE_HOST_LABELS="hostname=$(hostname)&other=labels" ...the rest of the registration command...?

[deitch]

As part of new host registration? Is there any way to configure it to happen automatically?

[roynasser]

I found the host creation/agent deployment code... Unlucky for me its in Go and I have never written a single line :p

Nonetheless, if your sysadmins deploy the agent manually, In that case, have them insert the label into the command that is generated in the rancher add host ui, like @vincent99 's example... so you would basically run the same command everywhere (as I understand they do now), and it just adds the label... you can substitute hostname for whatever variable name you choose... (I'm not sure I'd add a label with the io.rancher.xxx convention... even if it may be allowed now it may not be possible in the future...but thats your call)

They basically insert -e CATTLE_HOST_LABELS="hostname=$(hostname)&other=labels" right after the "docker run" part...

[vincent99]

Well most of the time I think you'd be better off having labels that describe whatever specific capabilities the host has that you may require (e.g. network=lab, chassis=dell-r610, gpu=true, etc) and scheduling on those criteria instead of requiring an exact hostname..

But for the specific hostname case I think the right solution is to add support for requestedHostId to rancher-compose, as there is in the API/UI, rather than using labels at all, because you're basically bypassing scheduling and picking a specific host:

But that doesn't help you today. For new hosts, running the registration command like that the shell will evaluate and expand $(hostname) at the time you run it so the label will be set. (I just saw @RVN-BR said the same earlier.. if you type that into the UI the special chars are escaped so it won't work, but there's no reason you can't construct your own command and use shell expansion).

For existing hosts we don't have a field specifically for hostname, but the name defaults to it when hosts are registered. So you could backfill a label on each host with the host's name by running this in a browser console:

s.findAllUnremoved('host').then(function(hosts) {
  hosts.forEach(function(host) {
    host.get('labels')['hostname'] = host.get('name');
    host.save();
  })
});

[vincent99]

Good point also @RVN-BR.. io.rancher is our namespace and we reserve the right to trample all over it. You should definitely not create your own labels underneath it.

[roynasser]

Yes... I would have thought to add some labels such as CPUs/Cores, RAM, Diskspace, etc... In our case we have a few variations so I'm not sure if we will do by "feature" or "flavor" (feature being multiple labels describing the machine & its capabilities, and "flavor" being something like small, medium, large, etc similar to what amazon does...) I personally think describing the machine's aspects and then scheduling based on those (+ someday considering availability of resources as well) is best, but it does introduce some complexity depending on your requirements...

[deitch]

@Vincent98 you are right. I'm really trying to just schedule it onto a single host. Scheduling works great because it uses labels. If I want to bypass scheduling to give it a single host, I can do that in the UI, but is there any way to do it via a compose yml? I couldn't find it. Even "global" (run on all hosts) has a label.

You could argue that there is s better way to do what I want than host specific scheduling; that is a different question entirely. Suffice to say that I can only change so much corporate behaviour at once.

For now, though, no label-based support for specific host request?

[vincent99]

There exists a field in the API to do that (requestedHostId) that the UI uses, but there is no corresponding rancher-compose.yml field to set it today.

The workaround is setting a label, and you now have a way both to automatically apply it to new hosts and back-fill it on existing hosts based on their name.

[roynasser]

@deitch None of the two methods above work for you?

@vincent99 Could I make a suggestion to allow a form of escaping characters in the labels in host creation? That way we can deploy some machines with specific labels coming from the system... something as simple of considering $$ or $ as $ and allowing that to be evaluated by the OS when it starts the agent?

[vincent99]

@RVN-BR The UI escapes the string it displays, and I feel like that's a sane thing to do for a command we're telling people to run to prevent it from ending up with syntax errors or the like. Are you really generating it from the UI multiple times? Typically if you're adding more than occasional hosts you'd have a script that contains the command and you could just set them there. The registration token (URL at the end) is reusable (per-environment) and good "forever" (10 years) by default.

If you know what you're doing there's no reason you can't construct your own with unescaped variables/etc that will be evaluated by your shell. The resulting string just has to be urlencoded query-param-style key=value&key2=value2.

[roynasser]

Hi @vincent99 thats fine for when I register hosts "manually" (in fact I store the command once and just run it whenever needed, so I wouldnt be doing it many times for "Other" hosts), but my thought is how to deploy 10 or 20 servers on AWS and have it include it there... I didnt test it, but would it evaluate correctly if I add a label with an external provider? In production we will probably be using rancher to deploy once things are working...

For now in dev I'm unable to use the rancheros machines (because they quit whenever a container is started on a host #2196, #3202 & possibly #2335 if proven to be related) which is why I'm manually deploying each time for my tests...

[deitch]

@RVN-BR

None of the two methods above work for you?

They work, but they still require an extra step for each new host added. I have to do one of:

1. Add the host and then add the label in the UI manually
2. Change the docker run .... command for each new host we add
3. Add some new scripting to automate adding it using the API

I am trying to avoid manual extra steps, if I can, because they are likely to be forgotten or have errors, and writing new scripting for a temporary solution.

Right now, we are adding hosts one at a time manually. We are not yet at the fully automated stage. Working to get this particular client to PXE boot + Chef, then everything will be automated, and I can change the label add just once.

Am I correct that these are my only solutions at this point?

[vincent99]

I feel like it's been made abundantly clear that exactly what you want does not exist. So yes, just the three workable alternatives, of which I gave you a way to automate (1) and reiterate that (2) is the same command for every host and does not involve changing anything for each host. $(hostname) literally goes into the string and is interpreted by the shell when run.

[vincent99]

@RVN-BR For all the other providers variable substitution wouldn't work because the command is interpreted on rancher/server when calling docker-machine, well before the machine actually exists.

[deitch]

Yes, precisely. As long as I click "add new host" once, copy the string, and make it part of some automated host build, it will work. If I add hosts manually, then I need to prepend it manually each time or add the label in the UI manually each time.

So we are leaving this as an open request that Rancher either adds a default host label for hostname or that rancher-compose adds support (prob via label as well) for requestedHostId?

[vincent99]

That would be the title of the issue...

[deitch]

Just making sure we are all on the same page. :-)

[roynasser]

Got it @vincent99 I thought it would send the docker command to run on the server itself... But this makes sense... Is there any plans to do scheduling based on host or container specs? (like hd, cpu, ram?) I think that could be useful as a general metric, either as a system label, or as a scheduler rule...

For now I'll add a label per provisioning host addition, which will work for me but it wouldnt be as flexible for scheduling

[roynasser]

@vincent99 Just 1 more question... your comment about the key validity on host add has just come up to my mind... Are these keys somehow able to be changed in case they get exposed/exploited somehow? (I realize this is a real edge case as the hosts need connection to each other in order to join, etc, but just for sakes of security, is there a way to revoke/re-issue these?) Would there be a way of maybe removing/readding similar to the API keys?

On a semi-related note, can you add the same physical host to 2 environments? Are there any problems/things to be aware?

[deitch]

I had thought the key was a one-time thing to set up the initial connection, but after that, they exchange newly generated private keys. But curious to hear what @vincent99 has to say.

[cjellick]

UX/functionality:

Introduce new scheduling label:
io.rancher.scheduler.affinity:host_name: "hostname1,hostname2,...hostNameN

Will consider both host.name and host.hostname. Any host with a name or hostname that matches any of the comma delimited values is an elligible candidate

We will NOT implement _soft or _ne at this point.

[deniseschannon]

With the release of Rancher 2.0, development on v1.6 is only limited to critical bug fixes and security patches.