DSL evaluation for capella - please take a look

[dzw1999]

Hello, greetings from the software engineering group from Beihang University, China. We have been working on the evaluation of Domain Specific Modeling Languages (DSMLs ), the main purpose is to find out the flaws of the design of the DSML. To test our evaluation framework, we evaluated your software design, mainly based on your ecore and odesign files, and found out that even though the overall quality of your software design is very good, there still exists some minor problems. The details are as follows:

rule-based evaluation

Firstly, we did some rule-based evaluation, mainly focusing on the graphic part.

Evaluation of Semiotic Clarity

We've checked your odesign file according to Goodman's theory of symbols[Goodman N. Languages of Art: An Approach to a Theory of Symbols[M]. Hackett publishing, Indianapolis, 1976.], and there are some minor problems that can decrease the semiotic clarity of your design as follows:

1. Redundancy

   There are some cases where various graphic symbols are used to represent the same element of the language.

   • interaction.InstanceRole : capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\eol.svg; capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\handlelifeline.svg;

2. overloading

   There are some graphic symbols in your design that is assigned to more than one element.

   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\OutFlowPort.svg : Port; ComponentExchangeCategory; ComponentPort;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\FlowPort.svg : Port; ComponentExchangeCategory; ComponentPort;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\FunctionSplitDiagram.svg : FunctionalChainInvolvementFunction; LogicalFunction; PhysicalFunction; SystemFunction;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\Pseudostate_entryPoint.svg : IState; EntryPointPseudoState;
   • capella\core\plugins\org.polarsys.capella.core.ui.resources\icons\full\svg\LogicalActor.svg : cs::Component; Component;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\Pseudostate_exitPoint.svg : IState; ExitPointPseudoState;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\Actor.svg : cs::Component; Component; ctx::SystemComponent;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\FunctionGatherDiagram.svg : FunctionalChainInvolvementFunction; LogicalFunction; PhysicalFunction; SystemFunction;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\FunctionRouteDiagram.svg : FunctionalChainInvolvementFunction; LogicalFunction; PhysicalFunction; SystemFunction;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\FunctionDuplicateDiagram.svg : FunctionalChainInvolvementFunction; LogicalFunction; PhysicalFunction; SystemFunction;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\FunctionSelectDiagram.svg : FunctionalChainInvolvementFunction; LogicalFunction; PhysicalFunction; SystemFunction;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\StandardPortSmall.svg : Port; ComponentExchangeCategory; ComponentPort;
   • capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\InFlowPort.svg : Port; ComponentExchangeCategory; ComponentPort;

3. Deficit Elements

   There are some elements in your design that are not represented by any graphic symbols.

   Certainly, we know that some elements do not need to be assigned symbols to, but we believe it is best to check whether these elements really do not need symbols or they were just overlooked.

   • GenericTrace
   • TransfoLink
   • JustificationLink
   • StateEvent
   • ChangeEvent
   • TimeEvent
   • ModellingBlock
   • ModellingArchitecture
   • ModellingArchitecturePkg
   • ReuseLink
   • Classifier
   • Feature
   • PropertyValuePkg
   • Project
   • Folder
   • Library
   • DeploymentTarget
   • FunctionPkg
   • FunctionKind
   • FunctionPort
   • DomainElement
   • KeyPart
   • Signal
   • Gate
   • Location
   • and so on...(in all 197 elements)

4. Deficit Symbols

   There are some referenced symbols in your design that do not actually exist in the project.

   • capella\core\plugins\org.eclipse.sirius.ui\icons\full\obj16\DAnalysis.gif

Evaluation of Symbol Similarity

We believe that overly similar symbols are not conducive to users' understanding and use of language. Therefore, we use image features such as SSIM to detect the similarity between the symbols you use. Below are some pairs of symbols that may be too similar:

1. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\FunctionInputPort.svg & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\description\images\CategoryInput.svg

   

2. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PrimeItemCI.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\HWCI.gif

3. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PrimeItemCI.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\SystemCI.gif

4. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PrimeItemCI.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\COTSCI.gif

5. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PrimeItemCI.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\NDICI.gif

6. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PrimeItemCI.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\CSCI.gif

7. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PrimeItemCI.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\InterfaceCI.gif

   

8. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\OperationalAnalysis.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PhysicalArchitecture.gif

9. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\OperationalAnalysis.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\EPBSArchitecture.gif

10. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\OperationalAnalysis.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\LogicalArchitecture.gif

11. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\OperationalAnalysis.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\SystemAnalysis.gif

12. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PhysicalArchitecture.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\EPBSArchitecture.gif

13. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PhysicalArchitecture.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\LogicalArchitecture.gif

14. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\PhysicalArchitecture.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\SystemAnalysis.gif

15. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\EPBSArchitecture.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\LogicalArchitecture.gif

16. capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\EPBSArchitecture.gif & capella\core\plugins\org.polarsys.capella.core.sirius.analysis\icons\full\obj16\SystemAnalysis.gif
    

LLMS-based evaluation

Secondly, we did some LLMS-based evaluation, mainly focusing on the model part. Generally speaking, we try to make LLM act as a domain expert to provide multidimensional evaluations of your language design.

Model Completeness

We give your ecore design to LLM and ask the LLM to guess which domain your language is designed for. And then we ask LLM to add possible missing elements to your language (perhaps not taken into account in the first version of the design, but can be considered for inclusion in future versions). The results are as follows:

Based on the provided meta model and relationships, this model is highly likely to belong to the field of system modeling, especially in applications such as system engineering and architecture design. More specifically, it may be related to MBSE (Model Based Systems Engineering), especially in scenarios where Arcadia methodology and Capella modeling tools are used. Arcadia is a systems engineering methodology that focuses on architecture design, while Capella is an open-source modeling tool that supports Arcadia.

Some key meta models that need to be supplemented and their relationships may include descriptions of system requirements, evolution, validation, and validation activities, as follows:

Element:

SystemRequirement: isFunctional; Priority;
Operational Scenario Requirement: scenarioRef;
VerificationActivity: description; Method; Outcome;
Validation Activity: description; Objective; Outcome;
SystemTransition: transitionJustification;
SystemEvolution: description; Rational;

Relationship:

SystemRequirement (1)<- drivedFrom ->Requirement (1)
SystemRequirement (1)<- example ->SystemComponent (n)
Operational Scenario Requirement (1)<- descriptedBy ->Operational Scenario (n)
VerificationActivity (1)<- verifies ->SystemFunction (n)
VerificationActivity (1)<- basedOn ->SystemRequirement (n)
ValidationActivity (1)<- validates ->SystemComponent (n)
Validation Activity (1)<- rational ->Requirement (n)
SystemTransition (1)<- romState ->BlockArchitecture (1)
SystemTransition (1)<- state ->BlockArchitecture (1)
SystemEvolution (1)<- volumes ->BlockArchitecture (n)
SystemEvolution (1)<- basedOn ->ChangeRequest (n)

These supplementary meta models and relationships contribute to a more comprehensive understanding of requirements management, system evolution, and verification and validation (V&V) activities in systems engineering activities. All these elements and relationships are crucial aspects in the field of system modeling, especially in the process of system design and analysis using the MBSE method.

We are not sure if these issues actually constitute a problem, the decision to fix them or not is still up to your team, looing forward to your respond, thanks!