Benchmark specification

The CPS benchmark compares the performance of different tools when executing a given scenario consisting of a sequence of phases with different cases.

• Tools can be complex configurations (e.g. a given model-to-model transformation and code generator pair).
• Scenarios describe what phases should be performed by the tools and in which order.
• Each phase defines a separate part of the scenario that can be measured with metrics (e.g. runtime or memory usage).
• A case specifies the input model and other aspects that depend on it (e.g. model modifications).

Tools

Transformation types

See the TransformationTypes enum for an up-to-date list and this wiki page for additional details.

Generator types

See the GeneratorTypes enum for an up-to-date list.

Scenarios

Toolchain

This scenario measures the performance of a model-driven application toolchain where an input CPS model is first transformed into a deployment model (model-to-model transformation), then a code generator is executed to output the final source code of the CPS system. After an initial execution, the input model is modified and the toolchain is executed again. This modification and execution loop is executed multiple times.

The scenario has two versions:

• Batch: if the M2M transformation always re-creates the complete output model, then the code generator will also have to be executed on the whole deployment model. In this version, a change monitor that watches the state of the output model would decrease the performance of the transformation.
• Incremental: if the M2M transformation is change aware and only modifies the output model as needed, the code generator can also be executed on the changed parts of the output model. In this version, a change monitor is initialized after the first execution and is used in subsequent code generation phases to update the source code.

Toolchain Batch scenario

Sequence [
  EMFResourceInitialization
  Generation
  Initialization
  M2MTransformation
  M2TTransformation
  Iteration (5 times) [
    Modification
    M2MTransformation (distinguished in results from the first run by adding a postfix)
    M2TTransformation (distinguished in results from the first run by adding a postfix)
  ]
]

Toolchain Incremental scenario

Sequence [
  EMFResourceInitialization
  Generation
  Initialization
  M2MTransformation
  M2TTransformation
  ChangeMonitorInitialization
  Iteration (5 times) [
    Modification
    M2MTransformation (distinguished in results from the first run by adding a postfix)
    M2TDeltaTransformation
  ]
]

M2M only

This scenario is used for comparing the performance of only the model-to-model transformation variants. As before, model modifications are performed to measure the incremental performance as well.

Sequence [
  EMFResourceInitialization
  Generation
  Initialization
  M2MTransformation
  Iteration (5 times) [
    Modification
    M2MTransformation (distinguished in results from the first run by adding a postfix)
  ]
]

Phases

Phase	Description
EMFResourceInitialization	Create a resource set with the CPS, deployment and tracability models each in a different resource and create root elements of each model.
Generation	Invokes the model generator to build the CPS model according to the specifications of the case, see case specifics below.
Initialization	Initializes the M2M transformation wrapper, for variants using VIATRA Query this step will perform the preparation of the query engine.
M2MTransformation	Invokes the wrapper to execute the CPS-to-Deployment transformation.
M2TTransformation	Invokes the wrapper to execute the M2T transformation and generate the files.
Modification	Modifies the CPS model according to the specifications of the case, see case specifics below.
ChangeMonitorInitialization	Initializes a VIATRA Query based change monitor that watches the Deployment model to identify where the M2T transformation should be re-executed after modifications. This is only relevant for incremental M2M transformations.
M2TDeltaTransformation	Invokes the M2T transformation on the parts of the Deplyoment model that were modified by the M2M transformation after modifications. This step uses the delta from the change monitor and is only relevant for incremental M2M transformations.

Cases

Case	Description
AdvancedClientServer	This is a modified Client-server case where the scale has wider affect for the number of various model elements and model structure. The forest of stars (many clients connected and sending messages to a single server) structure is the same as for the Client-Server.
ClientServer	This case creates an increasing number of "many client - one server" groups where both the host connections and the transition triggers point from clients to servers. This results in a forest of stars when visualizing the host instance communication structure.
LowSynch	This case creates a CPS model where state machines have only a few transitions with actions so triggering is limited.
PublishSubscribe	This case is similar to the Client-Server case, but both the communication and triggering directions are inverted to have a single publisher send messages to subscribers.
SimpleScaling	This simple case does not restrict the communications for independent sets of hosts, instead tries to uniformly allocate applications to the host instances.
StatisticBased	This case uses model statistics from industrial UML models mapped to the CPS domain.

Case-specific phase implementations

Case	Generate	Modify
AdvancedClientServer	Use generic generation workflow with case specific constraints	Find the first client application type and host instance then create a new application instance for this type allocated to the given host, see implementation
ClientServer	Use generic generation workflow with case specific constraints	Find the first client application type and host instance then create a new application instance for this type allocated to the given host, see implementation
LowSynch	Use generic generation workflow with case specific constraints	No modifications done
PublishSubscribe	Use generic generation workflow with case specific constraints	No modifications done
SimpleScaling	Use generic generation workflow with case specific constraints	No modifications done
StatisticBased	Use specific generation workflow with case specific constraints	Find the first application type with state machine and host instance that has applications allocated and create a new application instance for this type allocated to the given host, see implementation

Benchmark configuration file

The following is an example of the JSON format for defining benchmark configurations:

• Each element of the root benchmark is considered a separate configuration.
• Each configuration is executed by taking the Cartesian product of each array element.
• All elements are currently mandatory!
• For the possible String values, see the corresponding Enums in the code.

The available configurations are found in data.json files in the scripts/configs/config folders.

{
  "benchmark": [
    {
      "scenario" : "TOOLCHAIN_BATCH",
      "cases" : [
        "STATISTICS_BASED"
      ],
      "transformation_types": [
        "BATCH_SIMPLE",
        "BATCH_OPTIMIZED",
        "BATCH_VIATRA_QUERY_RETE",
        "BATCH_VIATRA_QUERY_LOCAL_SEARCH",
        "BATCH_VIATRA_QUERY_LOCAL_SEARCH_NO_FLAT",
        "BATCH_VIATRA_QUERY_LOCAL_SEARCH_DUMB_PLANNER",
        "BATCH_VIATRA_QUERY_LOCAL_SEARCH_WO_INDEXER",
        "BATCH_VIATRA_TRANSFORMATION"
      ],
      "generator_types": [
        "TEMPLATE"
      ],
      "scales": [1,2,4,8,16,32,64],
      "timeout": 1000,
      "runs": 5
    },
    {
      "scenario" : "TOOLCHAIN_INCREMENTAL",
      "cases" : [
        "STATISTICS_BASED"
      ],
      "transformation_types": [
        "INCR_VIATRA_QUERY_RESULT_TRACEABILITY",
        "INCR_VIATRA_EXPLICIT_TRACEABILITY",
        "INCR_VIATRA_AGGREGATED",
        "INCR_VIATRA_TRANSFORMATION"
      ],
      "generator_types": [
        "TEMPLATE"
      ],
      "scales": [1,2,4,8,16,32,64],
      "timeout": 1000,
      "runs": 5
    },
    {
      "scenario" : "TOOLCHAIN_BATCH",
      "cases" : [
        "STATISTICS_BASED"
      ],
      "transformation_types": [
        "BATCH_VIATRA_TRANSFORMATION"
      ],
      "generator_types": [
        "JDT"
      ],
      "scales": [1,2,4,8,16,32,64],
      "timeout": 1000,
      "runs": 5
    },
    {
      "scenario" : "TOOLCHAIN_INCREMENTAL",
      "cases" : [
        "STATISTICS_BASED"
      ],
      "transformation_types": [
        "INCR_VIATRA_AGGREGATED",
        "INCR_VIATRA_TRANSFORMATION"
      ],
      "generator_types": [
        "JDT"
      ],
      "scales": [1,2,4,8,16,32,64],
      "timeout": 1000,
      "runs": 5
    }
  ]
}