Change Machine Metadata in runtime

[MissingRoberto]

Hello, for my project I would like to change the metadata of the running machines in order to reschedule units to a determinate group of machines. For example, machines with low resource usage.

I have tried to do it by the etcd, but the change is remove by the machine, and I can´t do it from systemd either because I seems like the variable FLEET_METADATA is loaded at the very beginning.

Would be any way to change this metadata during execution?

[bcwaldon]

You can do this today by changing the metadata in your local config, then sending SIGHUP to the running process (systemctl kill --signal SIGHUP fleet). Hopefully this is enough for you until we tackle #555.

[MissingRoberto]

Excuse me, but I don´t know which local file.

I have changed: /etc/fleet/fleet.conf to content:

metadata="region=us-west,az=us-west-1"

And additionally the file /run/systemd/system/fleet.service.d/20-cloudinit.conf :

[Service]
Environment="FLEET_METADATA=health=loaded"

and used $ sudo systemctl kill --signal SIGHUP fleet

But it seems, it doesn't received any signal or I changed the wrong files. Any idea?

[bcwaldon]

@totemteleko what did you observe after you called systemctl kill --signal SIGHUP fleet?

[MissingRoberto]

I found the problem.

If you set up the "metadata" in the cloud-init file (in the fleet part), it doesn't use the fleet.conf.

I set up the cloud-init to give a default value, but I will with the fleet.conf instead.

Thanks for the help and answering so quickly.

[MissingRoberto]

Sorry again, but something that I didn´t expect happened.

If I change the machine metadata, the running units are stopped (or moved).

I would say that there may be 2 alternatives:
Update the metadata of the units or force the scheduler to omit this change and in that way, the units will stay where they are.

Is there any solution to do one of them or another one?

This is similar to the idea of the beginning of this issue Redeploy unit in place #760

[bcwaldon]

@totemteleko Did you change the metadata to something that causes the MachineMetadata options of locally-scheduled units to fail?

[MissingRoberto]

The units have MachineMetadata=health=idle

And I change the Metadata to "health=busy". So in that moment, there is a conflict between the unit and machine metadata and they unit is stopped.

I just need to skip this change in the running units.

I am trying to schedule to machines which have enough resources but still supporting the failure recovery feature.

[bcwaldon]

Filed #959

[MissingRoberto]

So, anyway to change the running units metadata? or update the entire unit file?

[bcwaldon]

@totemteleko Once a unit enters the system, it is immutable. Destroy it and create a new unit (possibly with the same name).