Support ordering constraint set attribute

[krig]

From @beekhof:

FYI, I am making the following changes to constraint sets

diff --git a/xml/constraints-2.1.rng b/xml/constraints-2.1.rng
index 85fcf48..6f40d9b 100644
--- a/xml/constraints-2.1.rng
+++ b/xml/constraints-2.1.rng
@@ -62,6 +62,14 @@
            <attribute name="require-all"><data type="boolean"/></attribute>
          </optional>
          <optional>
+           <attribute name="ordering">
+             <choice>
+               <value>group</value>
+               <value>listed</value>
+             </choice>
+           </attribute>
+         </optional>
+         <optional>
            <attribute name="action">
              <ref name="attribute-actions"/>
            </attribute>

The current behaviour 'group' will remain the default.

'group' will remain consistent with the ordering and colocation logic used by groups.
'listed' will be what most people expect when they create a colocated set.

- 'group'
  crm colocate A B C => B with A, C with B
  crm order A B C => A then B, B then C

- 'listed'
  crm colocate A B C => A with B, B with C
  crm order A B C => A then B, B then C

It is suggested that CLI's begin explicitly setting one of these values and, because
so many people seem to get it wrong, work out a way to use 'listed' by default.

[krig]

To clarify this issue for myself and others interested in making constraints easier to use in crmsh and pacemaker, here is the core problem:

With two resources, crmsh uses the attribute form of defining colocation constraints using the rsc and with-rsc attributes. Here we encode "colocate A with B" intuitively as colocation a-with-b A B.

With more than two resources, crmsh needs to use resource sets to express the colocation constraint. Between resource sets, the semantics are the same as above: "colocate set A with set B with set C" becomes colocation a-with-b-with-c ( A ) ( B ) ( C ). I think.

The problem is that within a resource set, the semantics are reversed. By default, crmsh creates a single resource set unless given parentheses. This creates a confusing situation. Quoting an email to the mailing list on the topic:

> If you ever consider do something about it, here is another thing that
> can be lived with, but is non-intuitive.
> 
> 1) colocation c1 inf: A B
> 
> the most significant is B (if B is stopped nothing else will be running)
> 
> 2) colocation c2 inf: A B C
> 
> most significant - A
> 
> 3) colocation c3 inf: ( A B ) C
> 
> most significant - C
> 
> 4) colocation c4 inf: ( A B ) C D
> 
> most significant - C again
> 
> I am trying to find a logic to remember this, but fails so far :)

[krig]

I am considering a new syntax, which would supersede the previous one.

Here, the top level always expresses the "with" relationship. In fact, I'd introduce an explicit with keyword:

• A with B: colocate a-with-b A with B
• A with B, B with C: colocate a-with-b-with-c A with B with C

As an added benefit thanks to the with keyword, we can elide the explicit ID if the ID is exactly of the form <rsc>-with-<rsc>..., so we get

• A with B: colocate A with B
• A with B, B with C: colocate A with B with C

Now, to express anything else we need parens. There is only one parenthesis form, ( ... [attributes] ) which encodes require-all=true, sequential=false unless explicitly set in the attributes section:

• colocate A and B with C: colocate ( A B ) with C
• colocate A and B with C and D, start C and D as soon as either A or B is started: colocate ( A B require-all=false ) with ( C D )

[krig]

More:

The confusion around the syntax implying that roles are per-resource when they are actually per resource-set can be fixed:

• colocate A:Master with B - rsc-role is set to Master
• colocate A:Master with B:Started - with-rsc-role is set to Started
• colocate ( A B ):Master with C - role is assigned to the set, not the members of the set

[krig]

Another aspect of the current syntax which is confusing and which would be solved by the above suggestion:

Guess what this does:

colocation c inf: ( ms_A:Master ms_B:Master ms_C:Master D )

It's one non-sequential resource set, right? No, that's not possible, 
because the role is set per-set, so can't be both master and null. So 
it's an error, right? Also no. It's actually two resource sets, which 
you can see after you have CRM tell you what it did:

crm(live)configure# show c
colocation c inf: ( ms_A:Master ms_B:Master ms_C:Master ) ( D )

The problem is that CRM syntax suggests that the role is a per-resource 
property, but it's actually a per-resource-set property. To accommodate 
this disparity in syntax, it mangles what you said into a valid XML 
configuration in a complex and surprising way, and when that's 
impossible, it just does something else.

[dmuhamedagic]

On Thu, Jan 07, 2016 at 07:40:42AM -0800, Kristoffer Grönlund wrote:

crm(live)configure# show c
colocation c inf: ( ms_A:Master ms_B:Master ms_C:Master ) ( D )
[...]
The problem is that CRM syntax suggests that the role is a per-resource
property, but it's actually a per-resource-set property. To accommodate
this disparity in syntax,

Yes, that's true, it should've somehow exposed that the
role/action is always on a per-set basis. But I was wary of
introducing yet another syntax form. rsc:role/action was already
present in the standard 2-resource constraints.

The issue, however, is also that the sets CRM syntax was created
long before we realized that there's a semantics problem with
collocations (yes, Andrew too realized that later). At the time
there was a short discussion between Lars, him, and me on how the
sets were designed, but they both dismissed my complaints as
nitpicking.

it mangles what you said into a valid XML
configuration in a complex and surprising way, and when that's
impossible, it just does something else.

Quite possible, as keeping sanity is not easy. I wonder though if
you know what does it exactly do?

[krig]

Ah sorry, the above was quoting from another old mailing list mail. I do know what it does now. :) But yes, I think the new syntax can fix this as mentioned above:

Since resource sets are always either top-level resource references or pairs of ( ), we can allow the :role on those two and not anywhere else. That way, the syntax always matches the generated sets:

colocate ( A B C ):Started with D:Master

[kgaillot]

FYI, I am proposing to deprecate the ordering attribute in #3141. The default behavior (ordering="group") would be retained.

While one could argue that either behavior is preferable, offering both just makes it even more confusing, and higher-level tools can easily map whichever order they prefer to the CIB order.

[liangxin1300]

Close this issue as crmsh has never supported the ordering attribute, and it is now scheduled for deprecation