About actual portability of the core type profile

[Roberto-Mameli-Eng]

I want to share an issue I'm currently facing concerning the core profile currently defined in the community repository.

As known, this profile defines general-purpose TOSCA types that are intended to be shared by all other profiles, at all levels of abstractions. The current version provides several type definitions that are inherently portable among different orchestrators (e.g. data types). On the other hand, some other definitions depends on the characteristics of the orchestrator. This is especially true for custom functions (and more generally for artifact types, which are tightly related to artifact processors implemented by orchestrators).

Concerning specifically functions, here is an excerpt from the current core profile:

functions:
  to_string:
    description: >-
      Convert an integer to string
    signatures:
      - arguments:
          - type: integer
            description: >-
              The integer to convert to string
        result: string
        implementation:
          primary:
            type: Python
            file: functions/to_string.py

This definition works for sure on Ubicity orchestrator, which includes a python artifact processor able to serve the purpose. On puccini, the profile is properly compiled (because it is syntactically correct), but the function cannot be used at run time, because the suite on which puccini is based natively supports a different mechanism for implementing custom functions (i.e. wasm plugins). This makes this profile not completely portable among different platforms.

One possible approach to cope with this problem could be to split the profile definition into a set of "portable" definitions and "orchestrator-dependent" definitions.

tosca_definitions_version: tosca_2_0
profile: community.tosca.core:0.1

# In the imports below, uncomment the orchestrator specific file
imports:
- portable-core-types.yaml
#- ubicity-specific-types.yaml
#- khutulun-specific-types.yaml
#- acme-specific-types.yaml

The portable-core-types.yaml would be something like:

data_types:
  JSON:
    description: >-
      The JSON type defines a string that contains data in the JSON
      format
    ...

capability_types:
   ...

while the rest of the definitions, which depend strictly on the orchestrator and on the type of artifacts it is able to support, should be defined in dedicated files. For example, for Ubicity ubicity-specific-types.yaml would appear as follows:

tosca_definitions_version: tosca_2_0
...

# Custom functions implementations for Ubicity
functions:
  to_string:
    description: >-
      Convert an integer to string
    signatures:
      - arguments:
          - type: integer
            description: >-
              The integer to convert to string
        result: string
        implementation:
          primary:
            type: Python
            file: functions/to_string.py
  ...

On the other hand, khutulun-specific-types.yaml may be characterized by different implementations:

tosca_definitions_version: tosca_2_0
imports:
- profile: puccini
  namespace: puccini

...

# Custom functions implementations using the Puccini TOSCA profile
functions:
  to_string:
    description: >-
      Convert an integer to string
    signatures:
      - arguments:
          - type: integer
            description: >-
              The integer to convert to string
        result: string
        implementation:
          primary:
            type: puccini:WasmPlugin
            file: ../artifacts/wasm/string_functions-0.0.1.wasm

Another option, slightly different, consists in removing at all implementations from community profiles. The core profile should be as it appears today, but without implementations in the function signatures. Since the standard allows function definition refinement in the service templates (sec. 10.4 of the TOSCA 2.0 specification), a set of specific profiles detailing all orchestrator dependent implementations should be provided by each orchestrator platform at each release of community profiles.

For example, when the community releases TOSCA community.tosca.*:x.y.z profiles, each orchestrator platform in turn releases one or more associated profiles that refine community types by adding proper implementations (e.g. com.ubicity.implementation:x.y.z or cloud.puccini.implementation:x.y.z, arbitrary names just to explain the concept). A service template would appear as follows:

tosca_definitions_version: tosca_2_0

imports:
# Community profiles (orchestrator independent)
  - profile: community.tosca.core:0.1
  - profile: community.tosca.abstract.platform:0.1
  - profile: community.tosca.abstract.application:0.1
# Uncomment next line if using Ubicity Orchestrator
  - profile: com.ubicity.implementation:0.1
# Uncomment next line if using Puccini/Floria/Khutulun orchestrator
# - profile: cloud.puccini.implementation:0.1
  
service_template:
  node_templates:
    ...

The 2 alternatives above are feasible with the current standard. Another option, that I write here just as a suggestion, may consist in adding support for "conditional" compilation, as in several programming languages. For example in C/C++ you can write:

#if defined(_WIN32) || defined(_WIN64)
    #define PLATFORM "Windows"
    void show_message(void) {
        printf("Windows-specific code\n");
    }

#elif defined(__linux__)
    #define PLATFORM "Linux"
    void show_message(void) {
        printf("Linux-specific code\n");
    }

#else
    #define PLATFORM "Unknown"
    void show_message(void) {
        printf("Unknown platform\n");
    }

#endif

and the actual code that is compiled depends on the macros. Following this concept, we may imagine to extend the language so that a TOSCA parser/compiler includes/excludes part of the TOSCA file depending on some specific comments in it. According to this approach, the profile may appear as follows:

functions:
  to_string:
    description: >-
      Convert an integer to string
    signatures:
      - arguments:
          - type: integer
            description: >-
              The integer to convert to string
        result: string
        #! __UBICITY__ 
        implementation:
          primary:
            type: Python
            file: functions/to_string.py
        #! __PUCCINI__ 
        implementation:
          primary:
            type: puccini:WasmPlugin
            file: ../artifacts/wasm/string_functions-0.0.1.wasm
  ...

I don't know if other approaches are possible, so far I do not have further ideas. Any comment/opinion on that?

[Roberto-Mameli-Eng]

I add another bit. I did not try any of the options above, so I'm not completely sure that they actually work

[lauwers]

Thanks Roberto, this is the right problem to raise before the 0.1. I went through what §10.4 permits, and three sentences in it decide most of the question.

An implementation is optional, and its absence has a defined meaning.

If no implementation is specified, then it's assumed that the TOSCA processor is preconfigured to handle the function call.

So a profile that declares signatures and no implementations is not an incomplete profile. It says what the community means: here is the contract, every processor provides the function.

Two profiles may not define the same function name.

Namespacing works as for types. Overlapping definitions under the same <function_name> are not allowed.

An import under a namespace makes the call $prefix:to_string, so a template written for one orchestrator no longer runs on another.

A function definition may carry several signatures, tested in order, where "the first matching implementation is used" — but matching is on the arguments, not on what the processor can execute.

What that means for the three options

• Splitting core and selecting by commented-out imports asks the consumer to edit the published profile, and a release is one signed CSAR per profile, so there is no variant to ship.
• An implementation profile per orchestrator is blocked by the overlapping-definitions rule. The specification's own version of the idea is refinement inside a service template, which puts the override in every template rather than in one profile.
• Conditional inclusion puts a preprocessor in YAML comments. Every parser, linter, editor and CSAR builder would have to implement it, and a file would stop meaning what it says.

What I would propose for the 0.1

Declare the signatures in core and ship no implementations there. It is what the specification already provides for, it makes one CSAR usable on every processor, and it is the decision the community took in a different context: community implementations are reference implementations, kept apart from the definitions (integrations/). The Python files move there and stay available to anyone who wants to run them.

It has a tidy consequence. Python is in core only because the function implementations name it, and Bash is named by no profile at all, so core ends with no artifact types: data types and function signatures, which is what A8 said core is for.

The cost is real and worth stating: a processor that does not already implement these functions has to, rather than executing what the CSAR carries.

And a language extension worth asking the TC for

The mechanism for choosing an implementation nearly exists. Signatures are already an ordered list with first-match selection, so the smallest useful change is to let a processor skip a signature whose implementation it cannot execute and try the next match. Then one profile can carry a Python implementation and a WebAssembly one for the same function, and each processor takes the one it can run. The same question applies to operations, where an implementation may be a shell script, a Python module, an Ansible playbook or a Terraform configuration and only one of them can be declared.

One thing to know about refinement

The service-template refinement rules do not hold up well enough to build on. Signatures are identified by position with no way to name one, so inserting a signature upstream rebinds every refinement; the "empty element" placeholder for an unrefined signature is indistinguishable from a signature that takes no arguments, which is legal; and no rule says that a refining element inherits the arguments and result of the signature it refines. I have filed that as oasis-tcs/tosca-specs#376.

This is item 3.3 on tomorrow's agenda.

[Roberto-Mameli-Eng]

[tliron]

I think you posted this to the wrong issue, I have not participated in this one.

[Roberto-Mameli-Eng]

Yes, you are right... I repost it on the correct one... thank you

[Roberto-Mameli-Eng]

Deleted and posted on the right discussion.... sorry for the confusion

[Roberto-Mameli-Eng]

I would like to revisit this discussion in light of the related exchange in Discussion #384, because I think the problem can now be stated more precisely and, perhaps more importantly, addressed at two different levels: what we can practically do today with TOSCA 2.0, and what could eventually be improved in a future version of the language.

I believe there is useful common ground around the following principle:

• a reusable profile should maximize what is processor-independent;
• the contract should be kept separate from processor-specific execution details wherever possible;
• different processors should remain free to provide different implementations of the same functionality, using whatever execution model is appropriate for them.

@tliron used an analogy that I find particularly useful: TOSCA can play the role of a C .h file, defining the contract, while processor-specific integrations provide the corresponding .c implementations.

This is essentially the problem I originally wanted to address in this discussion.

Consider the current community core profile, which defines generally useful functions such as $now, $in_range, $validate_json, $random_number, etc. Ideally, I would like to distinguish between:

• a portable core profile, containing the common function definitions/signatures and making no assumption about their execution;
• processor-specific core profiles, providing equivalent functionality together with implementations suitable for a particular execution environment.

Conceptually:

common portable core
        |
        +-- function definitions/signatures
        +-- no artifact calling convention
        +-- no processor-specific implementation

processor-specific core variants
        |
        +-- Ubicity-specific implementation
        |
        +-- Puccini/Floria-specific implementation
        |
        +-- other processor-specific implementations

For example, the Puccini/Floria variant could provide the relevant functionality through WASM plugins, while another processor could use Python artifacts, native/built-in functionality, or a completely different mechanism.

With the current TOSCA 2.0 grammar, I don't see a clean way for a second profile to take the definitions from the portable core profile and simply add or replace their implementations at profile level.

Therefore, leaving possible future language improvements aside for a moment, a pragmatic solution available today might simply be to split what is currently represented by a single core profile into separate profile files: one explicitly portable profile containing only the common definitions, and separate processor-specific variants containing the corresponding implementations.

This is admittedly less elegant than having a language mechanism to associate alternative implementations with a shared definition. It may also introduce some duplication between the portable and processor-specific variants. However, it has the advantage of requiring no change to TOSCA 2.0 while making the portability boundary explicit.

It would also fit naturally with a possible reorganization of the community repository, where users could clearly distinguish portable profiles from profiles or implementations targeting a specific processor.

So, for the current standard, the practical question I would like to discuss is:

Would splitting the current core profile into a portable definition-only profile and separate processor-specific variants be a reasonable way to maximize portability without imposing any artifact execution model on TOSCA processors?

Separately, and as a potential future improvement to TOSCA, I still think it would be useful to explore whether the language should eventually provide a cleaner mechanism for defining a contract in one profile and associating alternative implementations with it in other profiles.

But I would keep that as a separate, longer-term language-design question. For now, I am primarily interested in finding the cleanest solution that works with TOSCA 2.0 as it exists today.