🐛 node labels inconsistencies: s4l-lite services are not guaranteed anymore to run alone on autoscaled nodes

[sanderegg]

This is a blocker for any production release of the machine labels!!!

With the current node labels on staging the s4l-lite services are not guaranteed anymore to run alone on autoscaled nodes

Staging now has the following node labels defined:

• gpu workers: [dasksidecar==true, dynamicsidecar==true, gpu==true]
• autoscaled nodes for s4l-lite: [dynamicsidecar==true, gpu=true, io.simcore.autoscaled-node==true]

this means that s4l-lite goes as usual to the correct nodes.

BUT: any dynamic service created by director-v0/director-v2 might go there as well. That is not the intention and must be fixed before these labels move to production.

[sanderegg]

Possible fixes:

• define a negative constraint by default on all services dispatched by dv0/dv2 such as node.labels.io.simcore.autoscaled-node!=true (to test if that works with dv0, dv2 should work)
• define again some kind of standardworker label so that there is at least one difference between the autoscaled nodes and *non-*autoscaled nodes

[YuryHrytsuk]

Possible fixes:

* define a negative constraint by default on all services dispatched by dv0/dv2 such as `node.labels.io.simcore.autoscaled-node!=true` (to test if that works with dv0, dv2 should work)

* define again some kind of `standardworker` label so that there is at least one difference between the autoscaled nodes and *non-*autoscaled nodes

I have updated the configuration to go with the first approach you advise. Please, see all the PRs in here.

I will need to consult @sanderegg, @GitHK and, perhaps, @pcrespov to make sure my changes properly address the issue

[YuryHrytsuk]

These pull requests were merged today. From now on, this issue shall be fixed with node.labels.io.simcore.autoscaled-node!=true constraints