Alternatives to substitution mapping

[tliron]

I am here resurrecting an issue discussed on the mailing list in Apr 2021 (this thread). Pasting the email here with minor fixes:

I think what we need here is a way for the providing service to "expose" features and also for the consuming template to "inject" them. These
features thus become "shared" between both services. And I think Adam's proposal is the right starting point, something that looks more like a node template than a node type, something that can be constructed rather than "mapped".

One major problem with the current grammar is that mappings are one-to-one, but real world use cases are ... more complicated. But also more simple in terms of defining what users want. Here's a simple example: imagine a providing service that has two nodes, an application and a database, and that both of these can be installed on the same server. The consuming service doesn't care that internally the providing service comprises several nodes, all it wants to do is specify "install this (whole) service on this server". There is no way to do this with substitution mapping, because you can only map the installation requirement of either node template, not both of them at once. There's this mismatch between "mapping" and the control of the actual features we need.

Allow me to try to solve this with some back-of-the-napkin brainstorming by building on Adam's proposal and my idea of "exposing":

# Providing service
service_template:
  inputs:
    cores:
      type: integer
      default: { $get_property: [ app, cores ] }

  node_templates:
    app:
      directives:
      - expose # shared with the consuming service
      type: WebApplication # this type derives from "Application"
      properties:
        name: { $get_property: [ database,  capabilityname, name ] }
        cores: 1
      attributes:
        id: { $get_attribute: [ database, capabilityname, nested1, nested2, id ] }
     requirements:
     - host: my-host

    database:
      type: DB
     requirements:
     - host: myhost

   my-host:
      directives:
      - expose # *also* shared with the consuming service
      type: Machine

# Consuming service
service_template:
  node_templates:
    myservice:
      type: Application
      properties:
        cores: 5 # this will and override the value (our values here are higher priority), and thus also the property, which will then affect the input
      requirements:
      - host: my-vm

   my-vm:
      type: VirtualMachine # this type derives from "Machine" and the node will be "injected" into the providing template

If you follow this example you'll see that rather than substituting a single node, we are instead "exposing" two nodes from the providing
service. Once exposed they are essentially "shared" between the two services. The node template names are different but as long as the matching can be done correctly (by policy?) they should end up referring to the same exact node representations. Also note that the node types have to be the same or derive from each other. Of course references to the base type won't be able to access features of the derived type, but that is intended by design. So the consuming service only knows that this is an (abstract) "Application", while the providing template uses "WebApplication" and refers to additional properties and attributes. On the flip side, the providing service knows that it connects to an (abstract) "Machine", while the consuming service provides a concrete sub-type, "VirtualMachine".

The magic of sharing means that we can indeed install both the app and db nodes on the same server. And we're no longer talking about "substitution nodes" but rather a more open sharing of one or more nodes between the two services.

[lauwers]

I believe that this is exactly the use case for which substitution mapping is the perfect solution, assuming we fix the issue with the requirement mapping syntax as identified in #92 (comment)

Using current substitution mapping syntax, here is TOSCA for the top-level file from Tal's example (the consuming template using Tal's terminology):

tosca_definitions_version: tosca_2_0

# Consuming service
service_template:
  node_templates:
    myservice:
      type: Application
      directives: [ substitute ]
      properties:
        cores: 5 
      requirements:
      - host: my-vm

   my-vm:
      type: VirtualMachine

The myservice node is then substituted by the orchestrator (as instructed using the substitute directive) using the following service template, which is adapted from Tal's providing template (and using the proposed new requirement mapping syntax):

tosca_definitions_version: tosca_2_0

service_template:

  substitution_mappings:
    node_type: Application
    properties:
      cores: [ cores ]
    requirements:
      - host: [ app, host ]
      - host: [ database, host ]

  inputs:
    cores:
      type: integer
      default: 1
      
  node_templates:
    app:
      type: WebApplication # This type does not have to derive from "Application"
      properties:
        name: { $get_property: [ database,  capabilityname, name ] }
        cores: { $get_input: cores }
      attributes:
        id: { $get_attribute: [ database, capabilityname, nested1, nested2, id ] }

    database:
      type: DB

In the substituting template, both the app and the database nodes have a host requirement that uses automatic assignment. Using the updated requirement mapping syntax, the target for the host requirement in the substituted node (the myservice node) will be mapped twice, once each for the host requirement of the app node and the host requirement of the database node. The result is that both these nodes will land on the same my-vm server.

I believe this syntax is very elegant and very clear and a perfect use of substitution mapping.

[tliron]

It is not clear from the proposed new syntax that you would necessarily want the two requirements to be satisfied by the same target, or in other words that the two requirements should be somewhat joined together into one. Actually that may very well not be the case: perhaps you have more than one available target and you would rather not constrain the requirement satisfaction in such a way. In other words, you might want to allow for the "app" and "db" to be installed on different hosts. There is no way to choose how requirements are "joined" or if they should be. Does the substituted node end up having one relationship or two?

Moreover, it's just one very simple example to try to zero in on the issue. The allowed interactions can be much more complex. Imagine a substituting topology that has many node templates that are "endpoints", a very common design. Each endpoint can have requirements as well as capabilities of several different kinds. To connect endpoints, one endpoint would "require" a capability in another. To integrate this kind of topology using substitution mapping would mean mapping each supported requirement and capability from each endpoint node template into a node type, meaning that we would have to clumsily design a node type for all of it. Here's a node type for three endpoints:

node_types:
  NodeWithThreeEndpoints:
    capabilities:
      endpoint1-data: DataPlaneCapability
      endpoint1-control: ControlPlaneCapability
      endpoint1-management: ManagementPlaneCapability
      endpoint2-data: DataPlaneCapability
      endpoint2-control: ControlPlaneCapability
      endpoint2-management: ManagementPlaneCapability
      endpoint3-data: DataPlaneCapability
      endpoint3-control: ControlPlaneCapability
      endpoint3-management: ManagementPlaneCapability
    requirements:
    - endpoint1-data: { capability: DataPlaneCapability }
    - endpoint1-control: { capability: ControlPlaneCapability }
    - endpoint1-management: { capability: ManagementPlaneCapability }
    - endpoint2-data: { capability: DataPlaneCapability }
    - endpoint2-control: { capability: ControlPlaneCapability }
    - endpoint2-management: { capability: ManagementPlaneCapability }
    - endpoint3-data: { capability: DataPlaneCapability }
    - endpoint3-control: { capability: ControlPlaneCapability }
    - endpoint3-management: { capability: ManagementPlaneCapability }

And then you would have to map each of these in substitution mapping. Do you have a topology with 6 endpoints? You would have to create a custom node type for it. This is all, I hope quite obviously, very undesirable. And also not what we ever wanted from the original design -- we don't want to create special node types just to support mappings, right? We wanted the substituting node types to be "abstract" and general in some way. But it ends up that this elegant basic idea just doesn't work in so many real-world use cases, really in most of what I need. We're not really "mapping", despite the stated philosophy and keynames: in reality we are importing a bunch of node templates (node representation) into another bunch of node templates (=composition) and we want and need much more choice over how they interact.

My point is that substituting a node type, with mappings, addresses only very specific use cases, and not actually that common. Moreover the grammar (and syntax) has become increasingly convoluted with each release of TOSCA as we discover more and more real-world awkwardness. In particular I strongly object to mapping properties to topology inputs, a very weird cross-usage of two very different kinds of entities. I think we have to admit that this concept isn't as obviously useful as was that way back when it was first introduced and that it's time to rethink the goals and approaches.

[lauwers]

A couple of comments:

• Substitution is not about composition, it is about decomposition. It enables top-down designs that start with abstract nodes that are decomposed into more concrete realizations, possibly recursively. So yes, this implies that we will have different node types for the abstract entities than for the nodes in the substituting templates. That is a good thing, not a bad thing.
• If there are multiple ways in which an abstract node can get decomposed, then we will need multiple substituting templates. That is sort of the whole point. There is no way around this, even with your proposal, and again that is a good thing, not a bad thing. Our goal here is to make sure the syntax lets the service designer specify exactly what is desired in an unambiguous way.
• In my opinion, the substitution syntax is not convoluted at all, and it is much easier to visualize than the concept of injecting nodes from one template into another template.
• Would you mind providing other use cases that are not readily supported by substitution mapping? They can only help the discussion.

[pmbruun]

I am still being a bit quiet in the discussions, because as mentioned, we do these things in a different, and far simpler way, our fundamental concepts are quite different, and could not easily be ported into TOSCA.

I am looking forward to the expanded example from @tliron, so when I better understand what you are trying to do, I will try to show what the same thing looks like with parameterization instead of substitution.

[tliron]

Thanks @pmbruun ! No need to wait for me to go first, I already take up too much space. :)

[pmbruun]

I want to make sure I understand your scenario first.

[tliron]

Here's a very very long full version of my previous example:

tosca_definitions_version: tosca_2_0

capability_types:

  ControlPlane:
    properties:
      protocol-version: { type: string, default: '1.0' }
    attributes:
      current-status: { type: string }

  DataPlane:
    properties:
      protocol-version: { type: string, default: '1.0' }
    attributes:
      current-status: { type: string }

  ManagementPlane:
    properties:
      protocol-version: { type: string, default: '1.0' }
    attributes:
      current-status: { type: string }

node_types:

  HighCapacityEndpoint:
    attributes:
      mode: { type: string }
      linked: { type: boolean }
    capabilities:
      control-plane1: ControlPlane
      control-plane2: ControlPlane
      control-plane3: ControlPlane
      data-plane1: DataPlane
      data-plane2: DataPlane
      data-plane3: DataPlane
      management-plane1: ManagementPlane
      management-plane2: ManagementPlane
      management-plane3: ManagementPlane
    requirements:
    - control-plane-up1: { capability: ControlPlane, count_range: [ 1, 1 ] }
    - control-plane-up2: { capability: ControlPlane, count_range: [ 1, 1 ] }
    - control-plane-down1: { capability: ControlPlane, count_range: [ 1, 1 ] }
    - control-plane-down2: { capability: ControlPlane, count_range: [ 1, 1 ] }
    - data-plane-up1: { capability: DataPlane, count_range: [ 1, 1 ] }
    - data-plane-up2: { capability: DataPlane, count_range: [ 1, 1 ] }
    - data-plane-down1: { capability: DataPlane, count_range: [ 1, 1 ] }
    - data-plane-down2: { capability: DataPlane, count_range: [ 1, 1 ] }
    - management-plane-up1: { capability: ManagementPlane, count_range: [ 1, 1 ] }
    - management-plane-up2: { capability: ManagementPlane, count_range: [ 1, 1 ] }
    - management-plane-down1: { capability: ManagementPlane, count_range: [ 1, 1 ] }
    - management-plane-down2: { capability: ManagementPlane, count_range: [ 1, 1 ] }

  LowCapacityEndpoint:
    attributes:
      mode: { type: string }
      linked: { type: boolean }
    capabilities:
      control-plane1: ControlPlane
      control-plane2: ControlPlane
      data-plane1: DataPlane
      data-plane2: DataPlane
      management-plane1: ManagementPlane
      management-plane2: ManagementPlane
    requirements:
    - control-plane-up: { capability: ControlPlane, count_range: [ 1, 1 ] }
    - control-plane-down: { capability: ControlPlane, count_range: [ 1, 1 ] }
    - data-plane-up: { capability: DataPlane, count_range: [ 1, 1 ] }
    - data-plane-down: { capability: DataPlane, count_range: [ 1, 1 ] }
    - management-plane-up: { capability: ManagementPlane, count_range: [ 1, 1 ] }
    - management-plane-down: { capability: ManagementPlane, count_range: [ 1, 1 ] }

  Converter:
    attributes:
      backhaul-mode: { type: string }
      backhaul-linked: { type: boolean }
      midhaul-mode: { type: string }
      midhaul-linked: { type: boolean }
      fronthaul-mode: { type: string }
      fronthaul-linked: { type: boolean }
    capabilities:
      backhaul-control-plane1: ControlPlane
      backhaul-control-plane2: ControlPlane
      backhaul-control-plane3: ControlPlane
      backhaul-data-plane1: DataPlane
      backhaul-data-plane2: DataPlane
      backhaul-data-plane3: DataPlane
      backhaul-management-plane1: ManagementPlane
      backhaul-management-plane2: ManagementPlane
      backhaul-management-plane3: ManagementPlane
      midhaul-control-plane1: ControlPlane
      midhaul-control-plane2: ControlPlane
      midhaul-data-plane1: DataPlane
      midhaul-data-plane2: DataPlane
      midhaul-management-plane1: ManagementPlane
      midhaul-management-plane2: ManagementPlane
      fronthaul-control-plane1: ControlPlane
      fronthaul-control-plane2: ControlPlane
      fronthaul-data-plane1: DataPlane
      fronthaul-data-plane2: DataPlane
      fronthaul-management-plane1: ManagementPlane
      fronthaul-management-plane2: ManagementPlane
    requirements:
    - backhaul-control-plane-up1: ControlPlane
    - backhaul-control-plane-up2: ControlPlane
    - backhaul-control-plane-down1: ControlPlane
    - backhaul-control-plane-down2: ControlPlane
    - backhaul-data-plane-up1: ControlPlane
    - backhaul-data-plane-up2: ControlPlane
    - backhaul-data-plane-down1: ControlPlane
    - backhaul-data-plane-down2: ControlPlane
    - backhaul-management-plane-up1: ControlPlane
    - backhaul-management-plane-up2: ControlPlane
    - backhaul-management-plane-down1: ControlPlane
    - backhaul-management-plane-down2: ControlPlane
    - midhaul-control-plane-up: ControlPlane
    - midhaul-control-plane-down: ControlPlane
    - midhaul-data-plane-up: ControlPlane
    - midhaul-data-plane-down: ControlPlane
    - midhaul-management-plane-up: ControlPlane
    - midhaul-management-plane-down: ControlPlane
    - fronthaul-control-plane-up: ControlPlane
    - fronthaul-control-plane-down: ControlPlane
    - fronthaul-data-plane-up: ControlPlane
    - fronthaul-data-plane-down: ControlPlane
    - fronthaul-management-plane-up: ControlPlane
    - fronthaul-management-plane-down: ControlPlane

service_template:

  node_templates:

    processor:
      type: VM
      requirements:
      - internal-endpoint: backhaul-endpoint
      - internal-endpoint: midhaul-endpoint
      - internal-endpoint: fronthaul-endpoint

    backhaul-endpoint:
      type: HighCapacityEndpoint

    midhaul-endpoint:
      type: LowCapacityEndpoint

    fronthaul-endpoint:
      type: LowCapacityEndpoint

  substitution_mappings:
    node_type: Converter
    attributes:
      backhaul-mode: [ backhaul-endpoint, mode ]
      backhaul-linked: [ backhaul-endpoint, linked ]
      midhaul-mode: [ midhaul-endpoint, mode ]
      midhaul-linked: [ midhaul-endpoint, linked ]
      fronthaul-mode: [ fronthaul-endpoint, mode ]
      fronthaul-linked: [ fronthaul-endpoint, linked ]
    capabilities:
      backhaul-control-plane1: [ backhaul-endpoint, control-plane1 ]
      backhaul-control-plane2: [ backhaul-endpoint, control-plane2 ]
      backhaul-control-plane3: [ backhaul-endpoint, control-plane3 ]
      backhaul-data-plane1: [ backhaul-endpoint, data-plane1 ]
      backhaul-data-plane2: [ backhaul-endpoint, data-plane2 ]
      backhaul-data-plane3: [ backhaul-endpoint, data-plane3 ]
      backhaul-management-plane1: [ backhaul-endpoint, management-plane1 ]
      backhaul-management-plane2: [ backhaul-endpoint, management-plane2 ]
      backhaul-management-plane3: [ backhaul-endpoint, management-plane3 ]
      midhaul-control-plane1: [ midhaul-endpoint, control-plane1 ]
      midhaul-control-plane2: [ midhaul-endpoint, control-plane2 ]
      midhaul-data-plane1: [ midhaul-endpoint, data-plane1 ]
      midhaul-data-plane2: [ midhaul-endpoint, data-plane2 ]
      midhaul-management-plane1: [ midhaul-endpoint, management-plane1 ]
      midhaul-management-plane2: [ midhaul-endpoint, management-plane2 ]
      fronthaul-control-plane1: [ fronthaul-endpoint, control-plane1 ]
      fronthaul-control-plane2: [ fronthaul-endpoint, control-plane2 ]
      fronthaul-data-plane1: [ fronthaul-endpoint, data-plane1 ]
      fronthaul-data-plane2: [ fronthaul-endpoint, data-plane2 ]
      fronthaul-management-plane1: [ fronthaul-endpoint, management-plane1 ]
      fronthaul-management-plane2: [ fronthaul-endpoint, management-plane2 ]
    requirements:
      backhaul-control-plane-up1: [ backhaul-endpoint, control-plane-up1 ]
      backhaul-control-plane-up2: [ backhaul-endpoint, control-plane-up2 ]
      backhaul-control-plane-down1: [ backhaul-endpoint, control-plane-down1 ]
      backhaul-control-plane-down2: [ backhaul-endpoint, control-plane-down2 ]
      backhaul-data-plane-up1: [ backhaul-endpoint, data-plane-up1 ]
      backhaul-data-plane-up2: [ backhaul-endpoint, data-plane-up2 ]
      backhaul-data-plane-down1: [ backhaul-endpoint, data-plane-down1 ]
      backhaul-data-plane-down2: [ backhaul-endpoint, data-plane-down2 ]
      backhaul-management-plane-up1: [ backhaul-endpoint, management-plane-up1 ]
      backhaul-management-plane-up2: [ backhaul-endpoint, management-plane-up2 ]
      backhaul-management-plane-down1: [ backhaul-endpoint, management-plane-down1 ]
      backhaul-management-plane-down2: [ backhaul-endpoint, management-plane-down2 ]
      midhaul-control-plane-up: [ midhaul-endpoint, control-plane-up ]
      midhaul-control-plane-down: [ midhaul-endpoint, control-plane-down ]
      midhaul-data-plane-up: [ midhaul-endpoint, data-plane-up ]
      midhaul-data-plane-down: [ midhaul-endpoint, data-plane-down ]
      midhaul-management-plane-up: [ midhaul-endpoint, management-plane-up ]
      midhaul-management-plane-down: [ midhaul-endpoint, management-plane-down ]
      fronthaul-control-plane-up: [ fronthaul-endpoint, control-plane-up ]
      fronthaul-control-plane-down: [ fronthaul-endpoint, control-plane-down ]
      fronthaul-data-plane-up: [ fronthaul-endpoint, data-plane-up ]
      fronthaul-data-plane-down: [ fronthaul-endpoint, data-plane-down ]
      fronthaul-management-plane-up: [ fronthaul-endpoint, management-plane-up ]
      fronthaul-management-plane-down: [ fronthaul-endpoint, management-plane-down ]

[pmbruun]

I think the fundamental problem with substitution is that the TOSCA language doesn't allow node types as types of properties and inputs.

What we want to express is that the abstract topology wants to pass the same network to two different abstract nodes, and inside the concrete substitution we may have multiple nodes that need to have relations to the same external node.

With TOSCA today, the only way to express that is to define sufficiently distinct capabilities that the node can be uniquely identified by a corresponding set of requirements. That tends to explode into a huge set of individual capabilities and associated requirements.

Today an input or property can have a simple type like string or integer, but if I have a node type NT, then I want to be able to define an input n of type: NT to my concrete template and be able to pass a relevant node, x, that is a NT to the abstract node.

So at the abstract level, x gets bound to n and then inside the substituting template, you need to be able to create relations to n, and if multiple nodes create relations to n then they all get connected to x after the substitution has happened.

This is how I do it in our language, and it works really well. It is easy to understand and easy to use because it aligns with well known parameter passing principles of other languages.

The way we do it with requirements inside the substituting templates that match the outside capabilities can work nicely in some cases (and our language has a function for that, of course), but in a lot of cases, like Tal's above, it creates a strong coupling between the context of the abstract node and the definition of the substituting template.

This reminds me of the GOSUB call in old BASIC, where there was no parameter passing. So to pass parameters to a subroutine, you needed to set up global variables and make sure that both the caller and the subroutine used those variable names consistently. This severely limited the ability to re-use a subroutine in multiple contexts.

I'm afraid that Tal's bespoke capabilities (backhaul-control-plane3, ...) in the example above are like the bespoke global variables for a GOSUB routine in a BASIC program, and they have the same effect on abstractness and reusability of substituting templates.

This doesn't mean that I have a full solution to how to have nodes-as-values and node-types-as-value-types in TOSCA - that would require a fairly major change of the language - so definitely not something we can achieve for TOSCA 2.0.

[lauwers]

I think that the requirement mapping section in the substituting template does exactly what @pmbruun wants to accomplish:

• Let's assume that the abstract node in the top-level template has a requirement called target.
• In the top-level template, that requirement is fulfilled using a node called target_node
• Let's assume the substituting template maps the target requirement of the abstract node to a needs requirement of some node in the substituting template.
• When performing the substitution, the target_node that fulfills the target requirement of the abstract node will now also be used as the node that fulfills the needs requirement in the substituting template.

Effectively, the substitution will pass the target node to the substituting template albeit not as an input, but as a node that fulfills a requirement. Different syntax, but same effect.

[pmbruun]

I agree that we can effectively do this using substitution mappings, but we are not discussing the ability to express this in TOSCA, only how much the syntax helps you getting it right without a lot of boiler-plate definitions to get it right. I think that is what @tliron's example shows - a lot of boiler-plate to achieve something that should be a simple pattern.

Nodes, N1, ..., Nn, in the top-level template are basically in a flat namespace within that template. In the parameter metaphor, the N1, ..., Nn are like n variables in a namespace, at run-time each one "containing" a node representation object of the node-type. So the variable-value metaphor is not far off.

Now, if I want to pass target node Nx (1<=x<=n) of node-type, T, to node needs in a substituting template, S, I would like to logically define the abstract node A as A(needs: T) and then define S implements A or something like that, and then refer to the node needs inside the definition of template S. Now in the top-level template, you could write something amounting to A(Nx) and then every implementation of A would map Nx appropriately inside the respective substituting templates. Notice, that this would also make the concept of node-types more semantically meaningful in TOSCA than today.

Yes, there is a lot of syntax missing before this would work - I am not talking TOSCA 2.0, but rather 3.0 or later. But I think it is important for our language design to have a longer term vision.

The point is that typically, the designer knows which node, Nx, should be mapped to needs in every possible substituting template, but it is hard in TOSCA today to express that knowledge. Today, it is just hard and cumbersome for the designer to express that knowledge.

This is what happens today, according to my understanding, paraphrasing the description above:

1. The abstract node, A in the top-level template has a requirement called target.
2. In the top-level template, that requirement is fulfilled using a node Nx
3. The substituting template maps the target requirement of the abstract node to a needs requirement of some node in the substituting template.
4. When performing the substitution, the Nx in the top-template, that fulfills the target requirement of A will now also be used as the node that fulfills the needs requirement in the substituting template.

In some cases, this is the right metaphor - let the orchestrator pick an arbitrary Nx in the top template with inherent capabilities that satisfy some requirements.

But in other cases, this is the wrong metaphor. If Nx and Ny in the top template are semantically equally good, why not just let the implementation pick one?

In some cases one or more substituting templates need Nx or Ny in multiple places, and true, you don't care which one is picked, but you do care that they pick the same one. So if one place picks Nx, then the other must pick Nx, and if one picks Ny then the other one must pick Ny.

This is exactly the case that is causing trouble in substitution mappings - imposing a "sameness" constraint on multiple mappings, that to the orchestrator appears unrelated unless there is a convenient language feature that can be used for expressing this "sameness".

Today, as I understand it, the language feature that can impose sameness is to add a new, synthetic capability Cx to Nx - not because Cx is in any way an inherent capability of Nx that makes it semantically different from Ny, but only so that the two substituting templates (or two mappings in the same template) that need the same node can be guaranteed to pick the same one, either Nx or Ny, and not accidentally having one pick Nx and the other pick Ny.

This, I believe, is what is happening in @tliron 's example above - a lot of boilerplate consisting of synthetic capabilities and requirements.

Now what this mechanism does is that all the relevant substituting templates need to be aware of this synthetic capability Cx, but now Cx is suddenly a name that exists in a totally global namespace to represent Nx in the top template, and Cy is a global name representing the capability to select Ny.

This effectively turns Cx and Cy into a kind of global variable that are referenced inside the substituting templates - and this makes the analogy to GOSUB complete, and explains why I totally agree with @tliron that there is a language feature missing. Not because the language is unable to express this, but because, as his example shows, it is a horribly inconvenient way of expressing something that should have been simple.

Now in programming languages, using formal-actual reference/object-type parameters is the standard solution to this kind of problem. So that is how I deal with it in our language, and it is simple, clean and easy to understand. If that is also the right solution for TOSCA - in the short and long term - is obviously open for our discussion.

[tliron]

Thanks Peter, very clear contribution. I'd like to think it may be possible to do this in TOSCA 2.0. :) Hopefully my idea of "facades" as first-class citizens could make sense in this context. Instead of having "global variables", which I assume we would all agree are extremely hard to manage, the "facade" can be a namespace where these variables live. All users of this "facade" thus share these "variables" and are essentially bound by its contract. Another way to think of this is compare interface-oriented programming to object-oriented programming. A "facade" is an "interface" rather than a "class".

The challenging part of this abstract discussion is that TOSCA already exists as a programming language, and already has its own entities (variable types). Substitution mapping can only work at the level of the lowest-level entities -- capabilities, requirements, interfaces, properties/inputs, and attributes. But the most important entity type, namely node templates, cannot participate.

[lauwers]

There is nothing in the substitution mapping syntax that assumes global variables. There is a clear membrane between the substituted node and the substituting template, and that membrane is the node type of the substituted node. All information is passed through this membrane.

This is exactly the interface/implementation paradigm that Tal is referring to. I have been using the word façade to describe this membrane, since interface already means something else in TOSCA. I would prefer to not hijack the word façade to mean an entirely new concept.

[lauwers]

By the way, this facade concept is documented here: 3.5 Decomposition of Service Templates

[lauwers]

I think we're confusing two concepts: requirement fulfillment and substitution mapping.

• In the top level template, the orchestrator will fulfill all (dangling) requirements of the abstract node. After these requirements have been fulfilled, there will be one unique target node for each (mandatory) requirement of the abstract node. Designers can (and should) leave it up to the orchestrator to select the appropriate target nodes.
• Substitution mapping then passes on the target node for each mapped requirement to the substituting template. How things are mapped are unambiguously specified by the designer. While the orchestrator may have to choose between different substituting templates, once such a template has been selected, it does not have flexibility to decide how things are mapped. Those mappings are specified by the designer of the substituting template.

I continue to believe that this is very elegant. All we need is (simple) enhancements to the requirement mapping syntax to support multiple requirements with the same name. The reason @tliron 's use case does not look elegant is that it is not a good use case for substitution mapping. Substitution mapping is intended to support top-down designs that start with abstract concepts that are decomposed into concrete realizations. Tal's use case is not about abstraction, it is about hiding details of the concrete realizations. Tal's example shows a bottom-up design where the top-level template and the substituting template operate at the same level of abstraction. If such functionality is required, then we should investigate mechanisms other than substitution mapping.

@pmbruun, would it be possible to create a picture for the scenario you described above? That would help to guide the discussion during our Wednesday meetings.

[tliron]

I've given this a lot of thought (and work!) and think the key is to make "facades" first-class citizens in TOSCA, just like we did for profiles. I think it works well with our "mental model".

Feature Proposal: Facades and Implementations

Top-level keyname: facade

A profile is a library of types. A facade is a collection of implementable node templates. We know that node templates manifest as node representations (either new ones or preexisting ones) during orchestration, and that doesn't change with this proposal.

Like profiles, facades cannot have a service_template keyname, because that keyname assumes that we provide a full topology. Instead they use the facade keyname. We can assign values (properties, interfaces, etc.) but these can be overridden by implementations (more on that below).

(It's open to discussion whether facades can also have some topogological information, i.e. relationships between node templates. For now let's keep things simple and avoid that. Also, could a single file possibly have multiple facades? In these examples I allow one facade per file.)

Importing Facades

We can import facades just like we import profiles, but instead of bringing in new types we are bringing in new implementable node templates. To facilitate referring to them unambiguously they are namespaced just as for imported types.

Thus for the following imported facade:

imports:
- namespace: converter
  url: converter-facade.yaml

we would be able to refer to a node template in the facade with the colon syntax, e.g. converter:backhaul-endpoint.

Importantly, all the node templates imported from a single facade are associated with each other. Thus, if you refer in any way to converter:backhaul-endpoint, the orchestrator will bring in all the other node representations that are part of that facade. That means that if you want to refer to two separate implementations of the converter facade, you need to import the facade twice into two different namespaces:

imports:
- namespace: converter1
  url: converter-facade.yaml
- namespace: converter2
  url: converter-facade.yaml

This will be illustrated in the example where we indeed bring in two separate converters and connect them to each other.

Now we can do one of two things with the imported facades:

Node template keyname: implements

This means that our node template will provide the node representation for the facade's node template. Our node template's type must be the same as that of the facade node template or derived from it.

A single node template can implement more than one facade node template, and indeed those can belong to mode than one imported facade, thus implements can accept a single string or a list of strings. In other words, a single node representation can end up appearing in more than one topology (under different names) -- as long as it conforms to the facade.

Allowing for multiple implements references addresses a very common use case: we can have a single "server" (or "endpoint") that is to be used as part of the implementation for several "components".

Node template keyname: implemented_by

This means that our node template will get its node representation from elsewhere, namely a complete implementation of the referenced facade (our node template's type must be the same as that of the facade node template and cannot be a derived type). In other words, when providing the implementation, the orchestrator does not just bring in that single node template, but indeed the entire service template associated with it.

Note that it possible to assign properties, interfaces, etc., locally. These would override any values provided by the implementation (and in turn by the facade).

(An interesting issue here is whether the TOSCA parser should consider it an error if you do not provide a value for a required property here. That value may be part of the implementation. Thus until the implementation is selected we cannot know if that required property is will have a value.)

Components

A single service template can use both the implements and the implemented_by keynames on different node templates. As long as all requirements are satisfied in the final topology (between node representations) it should not matter which service templates provide the node representations and which make use of implementations provided by other service templates. This two-way street solves the problem of wanting to bring in a "component" from elsewhere while also having to provide it with certain dependencies, e.g. for the very common use of case of needing to provide a host for an application (the "component") to be installed into. (Primitive service template inputs are absolutely not a good enough solution, unless we could pass entire topology fragments as inputs!)

This implies that what qualifies as a "component" is up to our semantics and how we designed our profiles and facades. The TOSCA grammar does not know. A "component" can be entirely self-contained (a use case I don't believe is very common in the real world), or it might expect certain dependencies (e.g. hosts) to be provided by other "components". Indeed multiple "components" might need to work together for the topology to be complete. This is actually a common use case: one "component" can be the application (implementable by various vendors), one "component" can be the cloud platform (implementable by various cloud providers), and we can tie them all together in our service design, with our service being the third "component". In my example I'll keep things simple and have only two "components": an application "component" and the service "component".

Full Example

converter.zip

---
title: Converter Example
---
flowchart TD
    P{{Networking Profile}} -->|imports| I[Acme Converter Implementation]
    P -->|imports| F
    P -->|imports| S[Network Service]
    I -.->|implements| backhaul-endpoint
    I -.->|implements| midhaul-endpoint
    I -.->|implements| fronthaul-endpoint
    storage -.->|implemented_by| S
    subgraph F[Converter Facade]
        backhaul-endpoint
        midhaul-endpoint
        fronthaul-endpoint
        storage
    end

Mappings

What should we do with substitution_mapping? At its core it's just a special case of a facade with a single node template. Otherwise, I would suggest that the "mapping" syntax be separate and optionally complementary to the facade concept.

Specifically, I propose a map keyname for node templates that allows them to map properties, attributes, interface, requirements, and capabilities from other node templates. This "virtual" node template can then also include the implements keyname, in effect fully duplicating the substitution_mapping feature. Here's a full example:

web-application.zip

---
title: Web Application Example
---
flowchart TD
    P{{Web Profile}} -->|imports| C[Acme Web Application Component]
    P -->|imports| F(Web Application Facade)
    P -->|imports| S[Service]
    C -.->|implements| F -.->|implemented_by| S

[calincurescu]

I see the facade proposal from Tal as either a "collaboration" or a merge between two service templates.

• In the collaboration case: If both the service templates are deployed, today we can do this with the "selection" directive, i.e. the 3 nodes are selected by the second service from the frist service
• In the merge case: If only the second template is supposed to be deployed, and using the first template, then I think we should use a textual merge to create the final template. Maybe using something like merge_filter.