Introduction

Welcome to yet another busy Dogen sprint! Originally, we had intended to focus on the fabled "generation refactor" but, alas, it was not to be (yet again). Our preparatory analysis revealed some fundamental deficiencies on the variability implementation and, before you knew it, we were stuck wading in the guts of the variability subsystem for the entirety of the sprint. On the plus side, the end product was a much better designed subsystem, free of unwanted dependencies, and a newly found clarity in the conceptual model with regards to both logical and physical dimensions. On the down side, the refactor produced a lot of churn with regards to stereotypes and feature names, resulting on a fair bit of breakage to user diagrams. In other words, it was quite the eventful sprint. Let's see how we fared in more detail.

User visible changes

This section covers stories that affect end users, with the video providing a quick demonstration of the new features, and the sections below describing them in more detail. There have been a number of breaking changes, which have been highlighted with the symbol ⚠️.

Video 1: Sprint 22 Demo.

Split templatised from non-templatised variability meta-model elements

A pet peeve of ours, pretty much since profiles were introduced to the meta-model many moons ago, was the name chosen for the stereotype: masd::variability::profile_template. The postfix _template was a glaring leak from the implementation; a result of trying to be "too clever by half" in generalising all profiles to be "profile templates", when, in reality, there were only 2 or 3 cases of actual profile template instantiation across the code base. As it was, with this story we finally tackled this annoyance. However, before we proceed, a word is probably needed on what is meant by "templates" and "instantiation" in this context. The explanation will also prove helpful in understanding much of the remaining work carried out in the release.

Setting the Scene: Quick Primer on Variability Templates

As with many other modeling approaches, MASD divides the modeling of software products into two distinct dimensions: the logical dimension and the physical dimension. The logical dimension is pretty much what you are used to when creating UML class diagrams: the structural world of classes and their relationships (though, of course, in MASD there is a twist to it, but we need to leave that for another time). The physical dimension is, predictably, the world of files and directories. So far, so similar to UML and the like. What MASD does differently, however, is to impose a well-defined shape into the entities that live in the physical dimension, as well as a process by which these instances are derived. That shape is governed by the physical model's meta-model, which has existed since the early days of Dogen, albeit in an implicit manner. It is composed of vocabulary such as kernel (e.g., "masd"), backend (e.g., C++, C#), facet (e.g., "types", "hash", "serialisation" and so forth) and archetype (e.g., "class header", "class implementation", etc.).

Figure 1: Examples of Dogen feature bundles prior to the refactor.

The shape of the physical dimension is a function of the implementation; that is, as we add formatters (model-to-text transforms) to generate new kinds of output, these inject archetypes and facets and so on, augmenting the physical dimension. It became clear early on that adding features needed by all formatters manually was too painful. For example, we need to know if a kernel, backend, facet or archetype is enabled or disabled by the users. Thus a feature called enabled must exist for every element of the physical meta-model. We started by doing this manually, but it soon became obvious that what we were after was a generic way of saying that a feature with a given name n applies to every registered x - with x being an element of a set X, composed of kernels, backends, facets or archetypes. And so it was that variability templates were born. These were subsequently modeled within the logical model as both "feature bundles" (i.e., providing feature definitions, as per Figure 1) and "profile templates" (i.e., groups of configurations created by users for reuse purposes, performing feature selection; see Figure 2). In both cases we had the notion of an "instance template":

#DOGEN masd.variability.template_kind=instance

This was a "pseudo" or "identity" template, which does not really get instantiated but is instead copied across. We also had "real templates", associated with one of the "levels" in physical space (e.g., all, backend, facet, archetype):

#DOGEN masd.variability.template_kind=archetype

An additional modeling error was that, whilst profile templates only allowed a template kind at the profile level (that is, all attributes in the profile are of the same template_kind), we did not take the same approach for feature bundles, opening the gates for all sorts of weird and wonderful permutations: one attribute could be a template of kind instance whereas another could be a template of kind archetype. In practice, we were disciplined enough to avoid any such crazy stunts but, as old saying goes, "a good domain model should make invalid states unrepresentable".

Figure 2: Dogen's Profiles model before the refactor.

One final word on the dependency between the variability model and the physical model. Though its clear that there is a connection between the two models - at the end of the day, templates can only be initialised when we know the lay of the physical land - it is not necessarily the case that the coupling needs to be made in terms of "direct dependencies" (i.e. using a type from the physical model), because it comes at a cost: the graph of dependencies is made more complex because variability is used by many models, and these are then coupled to the physical model by way of this small connection. In truth, these models were joined more due to expediency than thought, for, as we mentioned, most features do not actually need template instantiation. Therefore, our core objective was to decouple the physical model from the variability model.

The tidy-up

One of the side-effects of the decoupling was to make us focus on creating a clear separation between the templatised and non-templatised elements of the logical model modeling variability. This was mainly to avoid increasing the end users cognitive load for no good reason ("why is this a 'template'? what's an 'instance template'?", etc.). As a result, the stereotypes are now as follows:

⚠️ Breaking change: the names and meaning of these stereotypes have changed. User diagrams must be updated.

Stereotype	Description
`masd::variability::profile_template`	Meta-model element defining a profile template. The template is instantiated over a domain, as we shall explain in the next section.
`masd::variability::profile`	Meta-model element defining a non-templatised profile. This is equivalent to the deprecated template kind of `instance`.
`masd::variability::feature_template_bundle`	Meta-model element defining a feature bundle template. As with profile templates, the template is instantiated over a domain. Note that all features belong to the same domain and all are templates, cleaning up the previous modeling mistake.
`masd::variability::feature_bundle`	Meta-model element defining a non-templatised feature bundle. This is equivalent to the deprecated template kind of `instance`.
`masd::variability::initializer`	Replaces the previous `masd::variability::feature_template_initializer`, providing initialisation for both feature templates and features.

Table 1: Stereotypes related to feature bundles and profiles.

While we were at it, we took the opportunity to update the colour theme, making the distinction between these elements more obvious:

Figure 3: Colour theme for all variability meta-model elements.

In addition to the stereotype changes, we also modified the approach to template instantiation, as explained on the next story.

Introduce "Domains" for Template Instantiation

The concept of domains was introduced as a way to achieve the before mentioned decoupling of the variability model from the physical model. Domains are simple sets of strings that can be used as the basis for template instantiation. When users declare templates (e.g., profile templates or feature bundle templates), they must now also provide the domain under which instantiation will take place:

#DOGEN masd.variability.instantiation_domain_name=masd

This is, of course, a breaking change:

⚠️ Breaking change: masd.variability.template_kind is no longer supported and must be replaced with masd.variability.instantiation_domain_name. This feature can only be used at the top level with masd::variability::profile_template and masd::variability::feature_template_bundle.

The following domains are available (with ${X} representing a "pseudo-code" variable):

Domain name	Sample `X` Value	Description
`${kernel}`	`masd`	The only supported kernel at present. All backends, facets, and archetypes are part of it.
`${kernel}.backend`	`masd.backend`	All backends in the MASD kernel. At present, C++ and C#.
`${kernel}.facet`	`masd.facet`	All facets in the MASD kernel, across all backends.
`${kernel}.archetype`	`masd.archetype`	All archetypes in the MASD kernel, across all backends and facets.
`${backend}.facet`	`masd.generation.cpp.facet`	All facets in the C++ backend of the MASD kernel.
`${backend}.archetype`	`masd.generation.cpp.archetype`	All archetypes in the C++ backend of the MASD kernel.
`${facet}.archetype`	`masd.generation.cpp.types.archetype`	All archetypes in the `types` facet, in the C++ backend of the MASD kernel.

Table 2: List of domains available out of the box.

Mind you, not all of these domains are being used at present, but, for completeness sake, we created a simple combinatorial function over the existing physical type to generate all sensible permutations. With this very simple approach we get all of the functionality we had previously, without any direct dependencies between the variability and physical models.

Remove name duplication from feature bundles and profiles

As you can clearly see from both Figure 1 and Figure 2, defining a profile or a feature bundle often resulted in a great deal of duplication of feature name prefixes, e.g., masd.generation.decoration in the case of the decoration profile:

masd.generation.decoration.enabled
masd.generation.decoration.licence_name
masd.generation.decoration.modeline_group_name
...

This release introduces a new feature that allows setting a prefix for all features in the bundle or profile:

#DOGEN masd.variability.key_prefix=masd.generation.decoration

Given a sensible profile or feature bundle name, the individual attributes should be meaningful enough to determine what they are about, with minimal repetition. For cases where mixing and matching is required, the old behaviour is still available.

Mapped default values for feature templates

In the past we found certain weird cases of feature templates where we needed the feature to expand over a domain, but we required different defaults for certain elements of the domain. For example, take the postfix feature. Ideally, each facet should have the postfix set to a string that correlates with a facet name (say hash) but sometimes to a smaller string (say lc for lexical_cast) or sometimes to the empty string (say for types). This setup was so complicated we just decided to create these features manually.

With this release we found a solution for the problem in the form of mapped default values. These are KVPs as follows:

#DOGEN masd.variability.default_value_override.cpp.tests="tests"
#DOGEN masd.variability.default_value_override.cpp.hash="hash"
#DOGEN masd.variability.default_value_override.cpp.lexical_cast="lc"
#DOGEN masd.variability.default_value_override.cpp.io="io"
...

Dogen performs a match on the input key for the key supplied, e.g. cpp.tests. If the input key ends with the supplied key, the match succeeds, meaning it will the value as its default value. In this case masd.text.cpp.tests would succeed but masd.text.cpp.not_a_facet would fail. With this new feature, we managed to model with one single feature template features that previously required tens of instances.

Add command line option to dump all specs

Dogen relies heavily on dynamic registration for a lot of its functionality, be it for injectors, features, backends and so forth. To top it all off, we keep changing names of things in our quest for tidying up the conceptual model. As a result, we find ourselves often grepping the code base to figure out what is available - an option that is not exactly practical for end users. With this release we've added a new activity to the command line client: dumpspecs. It works like so:

$ ./dogen.cli dumpspecs
Group: Injection
Purpose: Read external formats into Dogen.
    injection.dia: Decodes Dia diagrams. Extension: '.dia'
    injection.json: Decodes diagrams in JSON format. Extension: '.json'

Group: Conversion
Purpose: Output to an external format from a Dogen model.
    injection.dia: Encodes diagrams as JSON documents. Extension: '.json'

Group: Generators
Purpose: Available backends for code generation.
    masd.generation.cpp: Generates C++ code according to the MASD generative model.
    masd.generation.csharp: Generates C# code according to the MASD generative model.

Group: Features
Purpose: Available features for configuration.
    masd.decoration.modeline.editor: Editor to use in this modeline. Binding point: 'any'. Value type: 'masd::variability::text'.
    masd.decoration.modeline.location: Where to place the modeline. Binding point: 'any'. Value type: 'masd::variability::text'.
    masd.decoration.modeline.technical_space: Technical space targeted by the modeline. Binding point: 'any'. Value type: 'masd::variability::text'.
    masd.enumeration.add_invalid_enumerator: If true, adds an enumerator to represent an invalid choice. Binding point: 'element'. Default value: ''. Value type: 'masd::variability::boolean'.
    masd.enumeration.underlying_element: Name of the underlying element to use for the enumeration. Binding point: 'element'. Value type: 'masd::variability::text'.
...

Though the documentation may not be the best, we did go through all features and provided some kind of description. Note also that for feature templates, all instances share the same comment.

Renaming of Extraction Features

With the merging of the extraction model into the physical model (see internal stories below), we found ourselves having to rename a number of features. These names are not final, but at least they avoid referring to a model that no longer exists.

⚠️ Breaking change: Users that are making use of any of these features must update their diagrams as per Table 3.

Old Feature Name	New Feature Name
`masd.extraction.delete_extra_files`	`masd.physical.delete_extra_files`
`masd.extraction.output_technical_space`	`masd.physical.output_technical_space`
`masd.extraction.force_write`	`masd.physical.force_write`
`masd.extraction.delete_empty_directories`	`masd.physical.delete_empty_directories`
`masd.extraction.enable_backend_directories`	`masd.physical.enable_backend_directories`

Table 3: List of renamed features.

Development Matters

In this section we cover topics that are mainly of interest if you follow Dogen development, such as details on internal stories that consumed significant resources, important events, etc. As usual, for all the gory details of the work carried out this sprint, see the sprint log.

Significant Internal Stories

The sprint was mostly dominated by a large number of small refactors that changed the internals of Dogen dramatically - though in many cases, mainly with regards to naming and location of classes. We've aggregated all of these stories under two themes.

The Variability Model Refactor

The majority of the work in refactoring the variability model had user facing consequences, and so is described in great detail above. The main internal consequence was a dramatic reduction on the number of features required, due to an increased use of feature templates now that we can default them correctly; but there were also other smaller tasks related to this work:

dramatic simplification of the template instantiation code, which now merely loops through the list of elements in the domain when instantiating feature templates and profile templates.
changes related to ensuring lists and key value pairs within variability are stable sorted. In the past we had used unordered maps in the processing of variability data, resulting on tests breaking across operative systems due to re-ordering. We ended up having to make a fairly difficult surgical intervention, which resulted in a fair amount of breakage.

⚠️ Breaking change: Order of header files may change with this release. Other values dependent of order of lists and KVPs may also change such as order of database systems in ORM, and so forth.

The Physical Model Refactor

The second largest refactor this sprint was related to the physical model. This was comprised of a number of tasks:

rename the assets model to logical. In truth, assets has always been the model housing all of the meta-modeling elements for the logical model, so it makes sense to name it after its function.
rename the archetypes model physical model, and merge it with the extraction model. It took us a long time to understand that the extraction model was really the physical model in disguise. Originally, we had only used it to write files into the filesystem, but now it has taken on additional responsibilities such as defining the types in the physical meta-model.
move features related to physical aspects to physical model. This task was started but has not yet been completed.
rename the namespace meta-model used in a number of models to entities. The name was more or less meaningless the way it was being used. In addition, now that we need a meta-model for the physical model, it was becoming confusing. The "blander" name entities should avoid this confusion.
deletion of unused types in the generation model, as well as the removal of the partially implemented support for RapidJSON in the C++ model.

Resourcing

All and all, it was a very successful sprint from a resourcing perspective. At 51%, our utilisation rate was high but not quite the highest it's ever been (the previous sprint wins on that front at 56%). The high utilisation rate was a reflection of the fact that we worked full time for a big portion of the sprint. Sadly, this indicator is scheduled for a massive drop next sprint as we resume part-time work on Dogen proper, but hey-ho, we should celebrate the wins and this sprint was surely one on this front. Additionally, due to the undivided focus we managed to allocate over 82% of the commitment to stories directly related to the sprint's mission, including a couple of spikes (6.8% on unexpected tests breakage). We spent 17.5% on process, with a solid 10% on backlog grooming. Over half of the product backlog was reviewed this sprint, which we consider to be a task of vital importance. In addition, the cost of the demo has gone down dramatically since we started doing "one take demos", and we achieved a new low this sprint of 0.5%. The quality may not be quite what it used to be, but given the worse is better approach we favour so much, we deem it to be "good enough". A final note on Emacs, which had some minor blips but was overall fairly well behaved, costing us around 1.3%.

Figure 4: Cost of stories for sprint 22.

Roadmap

The road map continues to work rather like a Delphic oracle, and we keep trying to divine some kind of prediction that makes sense in terms of the current work. Thus far, it has failed to provide any such information but the visualisation of the gantt chart seems to be reassuring us that there is an end in sight - even though, like the proverbial carrot, it keeps moving forwards.

Next Sprint

We finally started the generation refactor this sprint, though, to be fair, we just about scratched the surface. Next sprint we will hopefully proceed in anger onto the generation breach and finally make a dent on it.

Binaries

You can download binaries from Bintray for OSX and Linux (all 64-bit):

Note 1: Due to a bug on the build scripts, Windows binaries were not generated for this release. If you do not want to build Windows from source, you can grab the unstable binaries for the next sprint: dogen-1.0.23-Windows-AMD64.msi.
Note 2: The OSX and Linux binaries are not stripped at present and so are larger than they should be. We have an outstanding story to address this issue, but sadly CMake does not make this a trivial undertaking.

For all other architectures and/or operative systems, you will need to build Dogen from source. Source downloads are available below.

Happy Modeling!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dogen v1.0.22, "Cine Teatro Namibe"