- Finish restoring support for CI/CD.
- Get PlantUML diagrams into a usable state.
| <75> | ||||
| Headline | Time | % | ||
|---|---|---|---|---|
| Total time | 82:09 | 100.0 | ||
| Stories | 82:09 | 100.0 | ||
| Active | 82:09 | 100.0 | ||
| Edit release notes for previous sprint | 10:13 | 12.4 | ||
| Create a demo and presentation for previous sprint | 2:06 | 2.6 | ||
| Update boost to latest in vcpg | 0:20 | 0.4 | ||
| Improve diffing output in tests | 0:19 | 0.4 | ||
| Sprint and product backlog grooming | 6:29 | 7.9 | ||
| Nightly builds are failing due to missing environment var | 1:07 | 1.4 | ||
| Remove deprecated uses of boost bind | 0:09 | 0.2 | ||
| Full generation support in tests is incorrect | 3:18 | 4.0 | ||
| Tests failing with filesystem errors | 2:10 | 2.6 | ||
| Remove uses of mock configuration factory | 0:54 | 1.1 | ||
| Add continuous builds to C++ reference product | 2:13 | 2.7 | ||
| Add continuous builds to C# reference product | 1:50 | 2.2 | ||
| Remove ODB support from Dogen | 0:45 | 0.9 | ||
| Model significant relations at a higher level | 1:05 | 1.3 | ||
| Create a mock configuration builder | 1:58 | 2.4 | ||
| Enable CodeQL | 0:21 | 0.4 | ||
| Change namespaces note implementation in PlantUML | 1:48 | 2.2 | ||
| Consider using a different layout engine in PlantUML | 1:31 | 1.8 | ||
| Upgrade PlantUML to latest | 0:44 | 0.9 | ||
| Code coverage in CDash has disappeared | 0:23 | 0.5 | ||
| Investigate nightly issues | 0:15 | 0.3 | ||
| Gitter notifications for builds are not showing up | 2:07 | 2.6 | ||
| Create a nightly github workflow | 5:39 | 6.9 | ||
| Run nightlies only when there are changes | 0:09 | 0.2 | ||
| Nightly builds are taking too long | 5:04 | 6.2 | ||
| Add support for UML profiles and XMI | 0:11 | 0.2 | ||
| Remove Dia and JSON support from Dogen | 3:23 | 4.1 | ||
| Make reference products git sub-modules | 0:41 | 0.8 | ||
| Ignore vcpkg path length warning | 0:12 | 0.2 | ||
| Windows package is broken | 0:20 | 0.4 | ||
| Warning on OSX build | 2:23 | 2.9 | ||
| Use clang format to format the code base | 0:58 | 1.2 | ||
| Add PlantUML relationships to diagrams | 17:19 | 21.1 | ||
| Update CMakeLists to match latest | 0:45 | 0.9 | ||
| Configuration as stereotype causes noise | 2:09 | 2.6 | ||
| Consider using meta-data within codec model | 0:29 | 0.6 | ||
| Thoughts on refactoring variability | 0:22 | 0.4 |
Agenda:
(org-agenda-file-to-front)Add github release notes for previous sprint.
Release announcements:
_Graduation day for the PhD programme of Computer Science at the University of Hertfordshire, UK. (C) 2022 Shahinara Craveiro._
# Introduction
After an hiatus of almost ten months, we've finally managed to get another Dogen release out. When looked at purely from a software engineering perspective, this wasn't exactly the most compelling of releases since almost all our stories are infrastructural. More specifically, the majority of resourcing was shifted towards getting Continuous Integration (CI) to work again, in the wake of the carnage left by Travis CI's decommission. However, the _true_ focus of the last few months lays outside the bounds of software engineering; our time was spent mainly on completing the PhD thesis, getting it past a myriad of red-tape processes and, perhaps most significantly of all, on passing the final exam called _the viva_. And so we did. Given it has taken some eight years to complete the PhD programme, you'll forgive us for the break with the tradition in naming releases after Angolan places or events; regular service will resume on the next release, for this as well as in the engineering front ```<knocks on wood, nervously>```. So grab a cupper, sit back, relax, and get ready for the release notes that mark the end of academic life in the Dogen project.
# User visible changes
This section covers stories that affect end users, with the video providing a quick demonstration of the new features, and the sections below describing them in more detail. However, as we've only had a couple of those - and even then, as these are fairly minor - the demo spends some time reflecting on the PhD programme overall.
[](https://youtu.be/ei8B1Pine34)
_Video 1_: Sprint 31 Demo.
## Deprecate support for dumping tracing to a relational database
It wasn't _that_ long ago Dogen was extended to dump tracing information into relational databases such as [PostgreSQL](https://www.postgresql.org/) and their ilk. In fact, [v1.0.20](https://github.com/MASD-Project/dogen/releases/tag/v1.0.20)'s release notes announced this new feature with great fanfare, and we genuinely had high hopes for its future. You are of course forgiven if you fail to recall what the fuss was all about, so it is perhaps worthwhile doing a quick recap. Tracing - or _probing_ as it was known then - was introduced in the long forgotten days of [Dogen v1.0.05](https://github.com/MASD-Project/dogen/releases/tag/v1.0.05), the idea being that it would be useful to inspect model state as the transform graph went through its motions. Together with log files, this treasure trove of information enabled us to understand where things went wrong quickly, more often than not without necessitating a debugger. And it was indeed incredibly useful to begin with, but we soon got bored of manually inspecting trace files. You see, the trouble with these crazy critters is that they are rather plump blobs of JSON, thus making it difficult to understand "before" and "after" diffs for the state of a given model transform - even when allowing for [json-diff](https://github.com/andreyvit/json-diff) and the like. To address the problem we doubled-down on our usage of [JQ](https://stedolan.github.io/jq/), but the more we did so, the clearer it became that JQ queries competed in the readability space with computer science classics like regular expressions and perl. A few choice data points should give a flavour of our troubles:
```bash
# JQ query to obtain file paths:
$ jq .models[0].physical.regions_by_logical_id[0][1].data.artefacts_by_archetype[][1].data.data.file_path
# JQ query to sort models by elements:
$ jq '.elements|=sort_by(.name.qualified)'
# JQ query for element names in generated model:
$ jq ."elements"[]."data"."__parent_0__"."name"."qualified"."dot"
```
It is of course deeply unfair to blame JQ for all our problems, since "meaningful" names such as ```__parent_0__``` fall squarely within Dogen's sphere of influence. Moreover, as a tool JQ is extremely useful for what it is _meant_ to do, as well as being incredibly fast at it. Nonetheless, we begun to accumulate more and more of these query fragments, glued them up with complex UNIX shell pipelines that dumped information from trace files into text files, and then dumped diffs of said information to other text files which where then... - well, you get the drift. These scripts were extremely brittle and mostly "one-off" solutions, but at least the direction of travel was obvious: what was needed was a way to build up a number of queries targeting the "before" and "after" state of any given transform, such that we could ask a series of canned questions like "has object X gone missing in transform T0?" or "did we update field Y incorrectly in transform T1?", and so on. One can easily conceive that a large library of these queries would accumulate over time, allowing us to see at a glance what changed between transforms and, in so doing, make routine investigations several orders of magnitude faster. Thus far, thus logical. We then investigated PostgreSQL's JSON support and, at first blush, found it to be [very comprehensive](https://www.postgresql.org/docs/current/functions-json.html). Furthermore, given that Dogen always had basic support for [ODB](https://www.codesynthesis.com/products/odb/), it was "easy enough" to teach it to dump trace information into a relational database - which we did in the [aforementioned release](https://github.com/MASD-Project/dogen/releases/tag/v1.0.20).
Alas, after the initial enthusiasm, we soon realised that expressing our desired questions as database queries was _far_ more difficult than anticipated. Part of it is related to the complex graph that we have on our JSON documents, which could be helped by creating a more relational-database-friendly model; and part of it is the inexperience with PostgreSQL's JSON query extensions. Sadly, we do not have sufficient time address either question properly, given the required engineering effort. To make matters worse, even though it was not being used in anger, the maintenance of this code was become increasingly expensive due to two factors:
- its reliance on a beta version of ODB ([v2.5](https://www.codesynthesis.com/pipermail/odb-users/2021-October/004696.html)), for which there are no DEBs readily available; instead, one is expected to build it from source using [Build2](https://build2.org/), an extremely interesting but rather _suis generis_ build tool; and
- its reliance on either manual install of the ODB C++ libraries or a patched version of [vcpkg](https://vcpkg.io/en/getting-started.html) with support for v2.5. As vcpkg undergoes constant change, this means that every time we update it, we then need to spend ages porting our code to the new world.
Now, one of the rules we've had for the longest time in Dogen is that, if something is not adding value (or worse, _subtracting_ value) then it should be deprecated and removed until such time it can be proven to add value. As with any spare time project, time is extremely scarce, so we barely have enough of it to be confused with the real issues at hand - let alone speculative features that may provide a pay-off one day. So it was that, with great sadness, we removed all support for the relational backend on this release. Not all is lost though. We use [MongoDB](https://www.mongodb.com/) a fair bit at work, and got the hang of its query language. A much simpler alternative is to dump the JSON documents into MongoDB - a shell script would do, at least initially - and then write Mongo queries to process the data. This is an approach we shall explore next time we get stuck investigating an issue using trace dumps.
## Add "verbatim" PlantUML extension
Since we moved away from [Dia](https://wiki.gnome.org/Apps/Dia), the quality of our diagrams degraded considerably. This is to be expected; when we originally added PlantUML support in the [previous release](https://github.com/MASD-Project/dogen/releases/tag/v1.0.30), it was as much a feasibility study as it was the implementation of a new feature. So the understanding was that we'd spend a number of sprints adding improvements to this new codec, until it got to the point where the diagrams where of comparable quality to the Dia ones. However, this sprint it dawned on us just how much machinery would be required to properly model relations in the rich way we had in Dia. Worse: it is not necessarily possible to merely record relations between entities in the input codec and then map those to a UML diagram, the reason being that, in Dia, we cleverly choose which relations are of significance and ignore those we deemed to be less interesting when conveying meaning on a diagram. To make matters more concrete, imagine a [vocabulary type](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2125r0.pdf) such as ```entities::name``` in model ```dogen::identification```. It is used throughout the whole of Dogen, and any entity with a representation in the LPS (Logical-Physical Space) will use it. A blind approach of modeling each and every relation to a core type such as this would result in a mess of inter-crossing lines, removing any meaning from the resulting diagram.
After a great deal of pondering, we decided that the PlantUML output needs two kinds of data sources: _automated_, where the relationship is obvious and uncontroversial, such as say the attributes that make up a class; and _manual_, where the relationship requires hand-holding by a human. This is useful for example in the above case, where one would like to suppress the relationships with a basic vocabulary type. This feature was implemented by means of adding a PlantUML _verbatim_ attribute to models. It is called "verbatim" because we merely add **exactly** what you put in there into the final PlantUML output. By convention, these statements are placed straight after the entity they were added to. It is perhaps easier to understand this feature by means of an example. Say in the ```dogen.codec``` model one wishes to add a relationship between ```model``` and ```element```. One could go about it as follows:
_Figure 1_: Use of the verbatim PlantUML property in the ```dogen.codec``` model.
As you can see, the property ```masd.codec.plantuml``` is extremely simple: it merely allows one to enter valid PlantUML statements, which are subsequently transported into the generated source code without modification, _e.g._:
_Figure 2_: PlantUML source code for ```dogen.codec``` model.
For good measure, we can observe the final (graphical) output produced by PlantUML, with the two relations. Its worth highlighting a couple of things here. The first is that we added a relationship with the object template ```Element```. Now, it is not entirely clear this is the correct way in UML to model relationships with object templates - the last expert I consulted was not entirely pleased with this approach - but no matter. The salient point is not whether this specific representation is correct or incorrect, but that one can choose to use this or any other representation quite easily, as desired. Secondly and similarly, the aggregation between ```model_set```, ```model``` and ```element``` is something that one would like to highlight in this model, and it is possible to do so trivially by means of this feature. Each of these classes is composed of a number of attributes which are not particularly interesting from a relationship perspective, and adding relations for all of those would greatly increase the amount of noise in the diagram.
_Figure 3_: Graphical output produced by PlantUML from Dogen-generated sources.
This feature is a great example of how often one needs to think of a problem from many different perspectives before arriving at a solution; and that, even though the problem may appear extremely complex at the start, sometimes all it takes is to view it from a completely different angle. All and all, the feature was implemented in just over two hours; we had originally envisioned lots of invasive changes at the lowers levels of Dogen just to propagate this information, and likely an entire sprint dedicated to it. To be fair, the jury is not out yet on whether this is really the correct approach. Firstly, because we now need to go through each and every model and compare the relations we had in Dia to those we see in PlantUML, and implement them if required. Secondly, we have no way of knowing if the PlantUML input is correct or not, short of writing a parser for their syntax - which we won't consider. This means the user will only find out about syntax errors after running PlantUML - and given it will be within generated code, it is entirely likely the error messages will be less than obvious as to what is causing the problem. Finally and somewhat related: the _verbatim_ nature of this attribute entails bypassing the Dogen type system entirely, by design. This means that if this information is useful for purposes other than PlantUML generation - say for example for regular source code generation - we would have no access to it.
A possibly better way of modeling this property is to add a non-verbatim attribute such as "significant relationship" or "user important relationship" or some such. Whatever its name, said attribute would model the notion of there being an important relationship between some types within the Dogen type system, and it could then be used by the PlantUML codec to output it in its syntax. However, before we get too carried away, its important to remember that we always take the simplest possible approach first and wait until use cases arrive, so all of this analysis has been farmed off to the backlog for some future use.
## Video series on MDE and MASD
In general, we tend to place our YouTube videos under the Development Matters section of the release notes because these tend to be about coding within the narrow confines of Dogen. As with so many items within this release, an exception was made for one of the series because it is likely to be of interest to Dogen developers and users alike. The series in question is called "MASD: An introduction to Model Assisted Software Development", and it is composed of 10 parts as of this writing. Its main objective was to prepare us for the _viva_, so the long arc of the series builds up to why one would want to create a new methodology and ends with an explanation of what that methodology might be. However, as we were unsure as to whether we could use material directly from [the thesis](https://uhra.herts.ac.uk/handle/2299/25708), and given our shortness of time to create new material specifically for the series, we opted for a high-level description of the methodology which is somewhat unsatisfactory due to a lack of visuals. We are therefore considering an additional 11th part which reviews a couple of key chapters from the thesis, namely Chapters 5 and 6.
At any rate, the individual videos are listed on Table 1, with a short description. They are also available as a playlist, as per link below.
_Video 2_: Playlist "MASD: An introduction to Model Assisted Software Development".
|Video | Description |
|---------|-----------------|
| [Part 1](https://www.youtube.com/watch?v=yRFjSegsC_s) | This lecture is the start of an overview of Model Driven Engineering (MDE), the approach that underlies MASD.|
| [Part 2](https://www.youtube.com/watch?v=Q-5Ic_gOd0Y)|In this lecture we conclude our overview of MDE by discussing Platforms and Technical Spaces, and we start to look at the field in more detail, critiquing its foundations.|
| [Part 3](https://www.youtube.com/watch?v=P20uEmc0wtc)|In this lecture we discuss the two fundamental concepts of MDE: Models and Transformations.|
| [Part 4](https://www.youtube.com/watch?v=_x5Wnab8Ipk)|In this lecture we take a large detour to think about the philosophical implications of modeling. In the detour we discuss Russell, Whitehead, Wittgenstein and Meyers amongst others.|
| [Part 5](https://www.youtube.com/watch?v=w1ZH4v8UiJU)|In this lecture we finish our excursion into the philosophy of modeling and discuss two core topics: Technical Spaces (TS) and Platforms.|
|[Part 6](https://www.youtube.com/watch?v=TcCNNpH4EfM)|In this video we take a detour and talk about research, and how our programme in particular was carried out - including all the bumps and bruises we faced along the way.|
|[Part 7](https://www.youtube.com/watch?v=r33MbmOv2ag)|In this lecture we discuss Variability and Variability Management in the context of Model Driven Engineering (MDE).|
|[Part 8](https://www.youtube.com/watch?v=AAvopzFQm9Q)|In this lecture we start a presentation of the material of the thesis itself, covering state of the art in code generation, and the requirements for a new approach.|
|[Part 9](https://www.youtube.com/watch?v=EFPMWq5SNGQ)|In this lecture we outline the MASD methodology: its philosophy, processes, actors and modeling language. We also discuss the domain architecture in more detail.|
|[Part 10](https://www.youtube.com/watch?v=EFPMWq5SNGQ)|In this final lecture we discuss Dogen, introducing its architecture.|
_Table 1_: Video series for "MASD: An introduction to Model Assisted Software Development".
# Development Matters
In this section we cover topics that are mainly of interest if you follow Dogen development, such as details on internal stories that consumed significant resources, important events, etc. As usual, for all the gory details of the work carried out this sprint, see the sprint log. As usual, for all the gory details of the work carried out this sprint, see [the sprint log](https://github.com/MASD-Project/dogen/blob/master/doc/agile/v1/sprint_backlog_31.org).
## Milestones and Éphémérides
This sprint marks the end of the PhD programme that started in 2014.
_Figure 3_: PhD thesis within the University of Hertfordshire archives.
## Significant Internal Stories
From an engineering perspective, this sprint had one goal which was to restore our CI environment. Other smaller stories were also carried out.
### Move CI to GitHub actions
A great number of stories this sprint (listed below) were connected with returning to a sane world of continuous integration, which we had lost with the demise of the open source support for [Travis CI](https://www.travis-ci.org). First and foremost, I'd like to give a huge shout out to Travis CI for all the years of supporting open source projects, even when perhaps it did not make huge financial sense. Prior to this decision, we had relied on Travis CI quite a lot, and in general it just worked. To my knowledge, they were the first ones to introduce the simple YAML markup for their IaC language, and it still supports features that we could not map to in our new approach (_e.g._ the infamous issue [#399](https://github.com/actions/toolkit/issues/399)). So it was not without sadness that we lost Travis CI support and found ourselves needing to move on to a new, hopefully stable, home. As we have support for [GitHub](https://github.com/MASD-Project/dogen), [BitBucket](https://bitbucket.org/MASD-Project/dogen/src/master/) and [GitLab](https://gitlab.com/DomainDrivenConsulting/dogen) as Git clones, we considered these three providers. In the end, we settled on GitHub actions, mainly because of the wealth of example projects using C++. All things considered, the move was remarkably easy, though not without its challenges. At present we seem to have all Dogen builds across Linux, Windows and OSX working reliably - though, as always, much work still remains such as porting all of our reference products.
_Figure 4_: GitHub actions for the Dogen project.
**Related Stories**: "Move build to GitHub", "Can't see build info in github builds", "Update the test package scripts for the GitHub CI", "Remove deprecated travis and appveyor config files", "Create clang build using libc++", "Add packaging step to github actions", "Setup MSVC Windows build for debug and release", "Update build instructions in readme", "Update the test package scripts for the GitHub CI", "Comment out clang-cl windows build", "Setup the laptop for development", "Windows package is broken", "Rewrite CTest script to use github actions".
### Improvements to vcpkg setup
As part of the move to GitHub actions, we decided to greatly simplify our builds. In the past we had relied on a hack: we built all our third party dependencies and placed them, as a zip, on DropBox. This worked, but it meant that updating these dependencies was a major pain. In particular, we often forgot how exactly those builds had been done and where we had sourced all of the libraries. As part of the research on GitHub actions, it became apparent that all the cool kids had moved on to using [vcpkg](https://vcpkg.io/en/getting-started.html) within the CI itself, with a set of supporting actions that made this use case much easier than before. This is highly advantageous because it means that updating the third party dependencies means merely having to update a git submodule. We took this opportunity and simplified all of our dependencies, which meant that sadly we had to remove our support for ODB since v2.5 is not available on vcpkg (see above). Nonetheless, the new setup is an improvement of several orders of magnitude, especially because in the past we had to have our own OSX and Windows Physicals/VM's to build the dependencies whereas now we rely only on vcpkg.
**Related Stories**: "Update vcpkg to latest", "Remove third-party dependencies outside vcpkg", "Update nightly builds to use new vcpkg setup".
### Improvements to CTest and CMake scripts
Closely related to the work on vcpkg and GitHub actions was a number of fundamental changes to our CMake and CTest setup. First and foremost, we like to point out the move to use CMake Presets. This is a great little feature in CMake that enables one to pack all of the CMake configuration into a preset file, removing the need for the old ```build.*``` scripts that had littered our build directory. It also means that building from Emacs - as well as other editors and IDEs which support presets, of course - is now really easy. In the past we had to supply a number o environment variables and other swuch incantations to the build script in order to setup the required environment. With presets all of that is encapsulated into a self comntained ```CMakePresets.json``` file, making the build much simpler:
```
cmake --preset linux-clang-release
cmake --build --preset linux-clang-release
```
You can also list the available presets very easily:
```
$ cmake --list-presets
Available configure presets:
"linux-clang-debug" - Linux clang debug
"linux-clang-release" - Linux clang release
"linux-gcc-debug" - Linux gcc debug
"linux-gcc-release" - Linux gcc release
"windows-msvc-debug" - Windows x64 Debug
"windows-msvc-release" - Windows x64 Release
"windows-msvc-clang-cl-debug" - Windows x64 Debug
"windows-msvc-clang-cl-release" - Windows x64 Release
"macos-clang-debug" - Mac OSX Debug
"macos-clang-release" - Mac OSX Release
```
This ensures a high degree of regularity of Dogen builds if you wish to stick to the defaults, which is the case for almost all our use cases. The exception had been nightlies, but as we explain elsewhere, with this release we also managed to make those builds conform to the same overall approach.
The release also saw a general clean up of the CTest script, now called ```CTest.cmake```, which supports both continuous as well as nighly builds with minimal complexity. Sadly, the integration of presets with CTest is not exactly perfect, so it took us a fair amount of time to work out how to best get these two to talk to each other.
**Related Stories**: "Rewrite CTest script to use github actions", "Assorted improvements to CMake files"
### Smaller stories
In addition to the big ticket items, a number of smaller stories was also worked om.
- **Fix broken org-mode tests**: due to the _ad-hoc_ nature of our org-mode parser, we keep finding weird and wonderful problems with code generation, mainly related to the introduction of spurious whitelines. This sprint we fixed yet another group of these issues. Going forward, the right solution is to remove org-mode support from within Dogen, since we can't find a third party library that is rock solid, and add instead an XMI-based codec. We can then extend Emacs to generate this XMI output. There are downsides to this approach - for example, the loss of support to non-Emacs based editors such as VI and VS Code.
- **Generate doxygen docs and add to site**: Every so often we update manually the Doxygen docs available [on our site](https://mcraveiro.github.io/dogen/doxygen/index.html). This time we also added a badge linking back to the documentation. Once the main bulk of work is finished with GitHub actions, we need to consider adding an action to regenerate documentation.
- **Update build instructions in README***: This sprint saw a raft of updates to our [REAMDE file](https://github.com/MASD-Project/dogen/blob/master/README.md), mostly connected with the end of the tesis as well as all the build changes related to GitHub actions.
- **Replace Dia IDs with UUIDs**: Now that we have removed Dia models from within Dogen, it seemed appropriate to get rid of some of its vestiges such as Object IDs based on Dia object names. This is yet another small step towards making the org-mode models closer to their native representation. We also begun work on supporting proper capitalisation of org-mode headings ("Capitalise titles in models correctly"), but sadly this proved to be much more complex than expected and has since been returned to the product backlog for further analysis.
- **Tests should take full generation into account**: Since time immemorial, our nightly builds have been, welll, _different_, from regular CI builds. This is because we make use of a feature called "full generation". Full generation forces the instantiation of model elements across all facets of physical space regardless of the requested configuration within the user model. This is done so that we exercise generated code to the fullest, and also has the great benefit of valgrinding the generated tests, hopefully pointing out any leaks we may have missed. One major down side of this approach was the need to somehow "fake" the contents of the Dogen directory, to esnure the generated tests did not break. We did this via the "pristine" hack: we kept two checkouts of Dogen, and pointed the tests of the main build towards this printine directory, so that the code geneation tests did not fail. It was ugly but just about worked. That is, until we introduced CMake Presets. Then, it caused all sorts of very annoying issues. In this sprint, after the longest time of trying to extend the hack, we finally saw the obvious: the easiest way to address this issue is to extend the tests to also use full generation. This was very easy to implement and made the nightlies regular with respect to the continuous builds.
### Video series of Dogen coding
This sprint we recorded a series of videos titled "MASD - Dogen Coding: Move to GitHub CI". It is somewhat more generic than the name implies, because it includes a lot of the side-tasks needed to make GitHub actions work such as removing third party dependencies, fixing CTest scripts, _etc._ The video series is available as a playlist, in the link below.
[](https://youtu.be/ei8B1Pine34)
_Video 3_: Playlist for "MASD - Dogen Coding: Move to GitHub CI".
The next table shows the individual parts of the video series.
|Video | Description |
|---------|-----------------|
| [Part 1](https://www.youtube.com/watch?v=l13FwDpvcA8)|In this part we start by getting all unit tests to pass.|
| [Part 2](https://www.youtube.com/watch?v=v7ebzs6XIf4)|In this video we update our vcpkg fork with the required libraries, including ODB. However, we bump into problems getting Dogen to build with the new version of ODB.|
| [Part 3](https://www.youtube.com/watch?v=JOQPzueENB0)|In this video we decide to remove the relational model altogether as a way to simplify the building of Dogen. It is a bittersweet decision as it took us a long time to code the relational model, but in truth it never lived up to its original promise.|
| [Part 4](https://www.youtube.com/watch?v=zu-YeZ6akcM)|In this short video we remove all uses of Boost DI. Originally, we saw Boost DI as a solution for our dependency injection needs, which are mainly rooted in the registration of M2T (Model to Text) transforms.|
| [Part 5](https://www.youtube.com/watch?v=OdDDQlV72BA)|In this video we update vcpkg to use latest and greatest and start to make use of the new standard machinery for CMake and vcpkg integration such as CMake presets. However, ninja misbehaves at the end.|
| [Part 6](https://www.youtube.com/watch?v=aY_OLBtkEHY)|In this part we get the core of the workflow to work, and iron out a lot of the kinks across all platforms.|
| [Part 7](https://www.youtube.com/watch?v=gtV9frKFZTw)|In this video we review the work done so far, and continue adding support for nightly builds using the new CMake infrastructure.|
| [Part 8](https://www.youtube.com/watch?v=Pf-nD5UpLT8)|This video concludes the series. In it, we sort out the few remaining problems with nightly builds, by making them behave more like the regular CI builds.|
_Table 2_: Video series for "MASD - Dogen Coding: Move to GitHub CI".
## Resourcing
At over ten months duration, this sprint was characterised mainly by its irregularity, rendering metrics such as utilisation rate rather meaningless. It would of course be an unfair comment if we stopped at that - given how much was achieved on the PhD front - but alas these are not resourcing concerns, given its sole focus on engineering effort. Looking at the sprint as a whole, it must be classified was very productive, weighing in at just over 85 hours and haing largely achieved our sprint goals. It is of course very disappointing to spend this much effort just to get back to where we were in terms of CI/CD in the Travis CI golden days, but it is what it is, and if anything our new setup is certainly a step up in terms of functionality when compared to the Travis/AppVeyor approach.
The most expensive story, by far, was the rewrite of the CTest scripts, at almost 16% of total effort, and it was closely followed by our series of lectures on MDE and MASD (11%). We also spent an uncharacteristic large amount of time refining our sprint and product backlogs: 10% versus the 7% of sprint 30 and the 3.5% of sprint 29. Of course, in the context of ten months with very little coding, it does make sense that we spent a lot of time having ideas about coding. All told, just under 60% of the sprint's total resourcing was directly related to its missing
_Figure 5_: Cost of stories for sprint 31.
## Roadmap
# Binaries
You can download binaries from either [Bintray](https://bintray.com/masd-project/main/dogen/1.0.30) or [GitHub](https://github.com/MASD-Project/dogen/releases/tag/v1.0.30), as per Table 3. All binaries are 64-bit. For all other architectures and/or operative systems, you will need to build Dogen from source. Source downloads are available in [zip](https://github.com/MASD-Project/dogen/archive/v1.0.30.zip) or [tar.gz](https://github.com/MASD-Project/dogen/archive/v1.0.30.tar.gz) format.
| Operative System | Debug | Release |
|--------------------------|------------|-----------|
| Linux Debian/Ubuntu (Deb) | [linux-clang-debug](https://github.com/MASD-Project/dogen/suites/8228081571/artifacts/359021758) | [linux-clang-release](https://github.com/MASD-Project/dogen/suites/8228081571/artifacts/359021759) |
| Linux Debian/Ubuntu (Deb) | [linux-gcc-debug](https://github.com/MASD-Project/dogen/suites/8228081571/artifacts/359021760) | [linux-gcc-release](https://github.com/MASD-Project/dogen/suites/8228081571/artifacts/359021761) |
| Windows (MSI) | [windows-msvc-debug](https://github.com/MASD-Project/dogen/suites/8228081572/artifacts/359031416) | [windows-msvc-release](https://github.com/MASD-Project/dogen/suites/8228081572/artifacts/359031417) |
| Mac OSX (DMG) | [macos-clang-debug](https://github.com/MASD-Project/dogen/suites/8228081569/artifacts/359027762) | [macos-clang-release](https://github.com/MASD-Project/dogen/suites/8228081569/artifacts/359027763) |
_Table 3_: Binary packages for Dogen.
**Note 1:** The Linux binaries are not stripped at present and so are larger than they should be. We have [an outstanding story](https://github.com/MASD-Project/dogen/blob/master/doc/agile/product_backlog.org#linux-and-osx-binaries-are-not-stripped) to address this issue, but sadly CMake does not make this a trivial undertaking.
**Note 2:** Due to issues with Travis CI, we did not manage to get OSX to build, so and we could not produce a final build for this sprint. The situation with Travis CI is rather uncertain at present so we may remove support for OSX builds altogether next sprint.
# Next Sprint
That's all for this release. Happy Modeling!Time spent creating the demo and presentation.
Marco Craveiro Domain Driven Development Released on 4th September 2022
Boost 1.80 is now available.
When a test fails with differences, we get the following output:
Differences found. Outputting head of first 5 diffs. diff -u include/dogen.identification/io/entities/name_io.hpp include/dogen.identification/io/entities/name_io.hpp Reason: Changed generated file. --- include/dogen.identification/io/entities/name_io.hpp +++ include/dogen.identification/io/entities/name_io.hpp @@ -1,11 +1,5 @@ /* -*- mode: c++; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- * - * These files are code-generated via overrides to test dogen. Do not commit them. - * - * Generation timestamp: 2022-09-19T00:04:25 - * WARNING: do not edit this file manually. - * Generated by MASD Dogen v1.0.32 - * * Copyright (C) 2012-2015 Marco Craveiro <marco.craveiro@gmail.com> * * This program is free software; you can redistribute it and/or modify
There are problems with this:
- it appears as if the generated files are missing these lines. However, when we look at the filesystem, they are absent from the original files. So it may be the generated files are generating this and shouldn’t. We should always check from the perspective of the files in the filesystem.
- the
---and+++should say what they mean. - actually upon investigation, the test files did contain the output:
* These files are code-generated via overrides to test dogen. Do not commit them. * * Generation timestamp: 2022-09-19T00:04:25 * WARNING: do not edit this file manually. * Generated by MASD Dogen v1.0.32 * * Copyright (C) 2012-2015 Marco Craveiro <marco.craveiro@gmail.com>
Something went wrong with full generation. The problem appears to be that full generation overrides the decoration settings.
Updates to sprint and product backlog.
We have a few tests failing with the following error:
/home/marco/nightly/dogen/master/projects/dogen.utility/src/types/environment/variable_reader.cpp(96): Throw in function strict_read_environment_variable Dynamic exception type: boost::wrapexcept<dogen::utility::environment::environment_exception> std::exception::what: Environment variable is empty or not defined: DOGEN_PROJECTS_DIRECTORY unknown location(0): fatal error: in "Test setup": std::runtime_error: Error during test /home/marco/nightly/dogen/master/projects/dogen.codec/tests/main.cpp(35): last checkpoint: initializer Running 1 test case... *** No errors detected Test setup error:
We do not seem to be using presets in the nightly for some reason.
Notes:
- this is due to a bug on the CTest script which is resetting the CMake arguments for nightlies.
- it appears we are not using parallel builds during nightly, we are taking over 8h for a single build. This has now been fixed.
- one of the tests is now timing out:
dogen.logical.generated_tests/entities_input_model_set_tests/xml_roundtrip_produces_the_same_entity Failed 10m 10ms Completed (Timeout)We need to find out how to increase the timeout.
- clang builds have the wrong DWARF2 format:
unhandled dwarf2 abbrev form code 0x25
Links:
- ctest_build
- FireFox: Valgrind run fails when building with clang 14
- kde: Valgrind does not read properly DWARF5 as generated by Clang14
- SO: How to overwrite Ctest default timeout 1500 in CMakeLists.txt
Merged stories:
Nightly builds are failing due to missing variable
At present we are getting the following error:
/home/marco/nightly/dogen/master/projects/dogen.utility/src/types/environment/variable_reader.cpp(96): Throw in function static std::string dogen::utility::environment::variable_reader::strict_read_environment_variable(const std::string&) Dynamic exception type: boost::wrapexcept<dogen::utility::environment::environment_exception> std::exception::what: Environment variable is empty or not defined: DOGEN_PROJECTS_DIRECTORY unknown location(0): fatal error: in "Test setup": std::runtime_error: Error during test /home/marco/nightly/dogen/master/projects/dogen.codec/tests/main.cpp(35): last checkpoint: initializer Running 1 test case...
Fix errors in nightly builds
[56/2312] Building CXX object projects/dogen/generated_tests/CMakeFiles/dogen.generated_tests.dir/spec_category_tests.cpp.o
In file included from /usr/include/boost/smart_ptr/detail/sp_thread_sleep.hpp:22,
from /usr/include/boost/smart_ptr/detail/yield_k.hpp:23,
from /usr/include/boost/smart_ptr/detail/spinlock_gcc_atomic.hpp:14,
from /usr/include/boost/smart_ptr/detail/spinlock.hpp:42,
from /usr/include/boost/smart_ptr/detail/spinlock_pool.hpp:25,
from /usr/include/boost/smart_ptr/shared_ptr.hpp:29,
from /usr/include/boost/shared_ptr.hpp:17,
from /usr/include/boost/test/tools/assertion_result.hpp:21,
from /usr/include/boost/test/tools/old/impl.hpp:20,
from /usr/include/boost/test/test_tools.hpp:46,
from /usr/include/boost/test/unit_test.hpp:18,
from /home/marco/nightly/dogen/master/projects/dogen/generated_tests/spec_category_tests.cpp:29:
/usr/include/boost/bind.hpp:36:1: note: ‘#pragma message: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS
Links:
Nightly build now uses full generation for tests. The problem is that full generation expresses decoration as well:
* These files are code-generated via overrides to test dogen. Do not commit them. * * Generation timestamp: 2022-09-19T00:04:25 * WARNING: do not edit this file manually. * Generated by MASD Dogen v1.0.32 * * Copyright (C) 2012-2015 Marco Craveiro <marco.craveiro@gmail.com>
We need a way to set decoration to false in the model and respect that somehow. Actually, it seems the problem is we are not honouring the variability overrides in the tests.
The issue was we were supplying the command line incorrectly:
--variability-override masd.variability.profile,masd.variability.profile,"
The command line argument --variability-override is not necessary. However,
when we fixed this we then created a whole raft of problems:
- we are now fully generating everything, including all reference products.
- for some reason the profile cannot be found for the c++ reference product:
std::exception::what: Configuration references a profile that could not be found: dogen.profiles.base.test_all_facets - not clear why we do not throw on an invalid variability override. One for the backlog.
The quick hack is to only use the overrides on Dogen tests somehow.
With the builder changes we now get the following error:
Running 1 test case... Differences found. Outputting head of first 5 diffs. /home/marco/nightly/dogen/master/projects/dogen.orchestration/tests/dogen_org_product_tests.cpp(83): error: in "dogen_product_org_tests/dogen_cli_org_produces_expected_model": check mg::check_for_differences(od, m) has failed *** 1 failure is detected in the test module "dogen.orchestration.tests"
This appears to reveal some bug in the diffing logic given that we do not see any differences.
Notes:
- its not obvious what is causing this difference but it seems there is some logic error in the check for differences method. We must be falling through the cracks on some unforeseen case.
The problem is we had disabled diffing. Enabling diffing we now see:
Unexpected write: dogen.identification/include/dogen.identification/types/entities/name_fwd.hpp
Reason: { "__type__": "operation_reason", "value": "changed_generated" }
diff -u include/dogen.identification/types/entities/name_fwd.hpp include/dogen.identification/types/entities/name_fwd.hpp
Reason: Changed generated file.
--- include/dogen.identification/types/entities/name_fwd.hpp
+++ include/dogen.identification/types/entities/name_fwd.hpp
@@ -2,7 +2,7 @@
*
* These files are code-generated via overrides to test dogen. Do not commit them.
*
- * Generation timestamp: 2022-09-21T00:04:26
+ * Generation timestamp: not-a-date-time
* WARNING: do not edit this file manually.
* Generated by MASD Dogen v1.0.32
*
/home/marco/nightly/dogen/master/projects/dogen.orchestration/tests/dogen_org_product_tests.cpp(204): error: in "dogen_product_org_tests/dogen_identification_org_produces_expected_model": check mg::check_for_differences(od, m) has failedThere are now two problems:
- why are we not generating a timestamp?
- if we did, we would still have a diff. We need a way to force the timestamp to a known value.
Links:
- v1.0.19: “Add support for variability overrides in Dogen”
The next batch of test failures is related to filesystem errors:
Running 1 test case... /home/marco/nightly/dogen/master/projects/dogen.orchestration/tests/code_generation_chain_tests.cpp(222): error: in "code_generation_chain_tests/empty_folders_are_not_deleted_when_delete_empty_folders_flag_is_off": check exists(first_empty_folders) has failed /home/marco/nightly/dogen/master/projects/dogen.orchestration/tests/code_generation_chain_tests.cpp(223): error: in "code_generation_chain_tests/empty_folders_are_not_deleted_when_delete_empty_folders_flag_is_off": check exists(second_empty_folders) has failed
/home/marco/nightly/dogen/master/projects/dogen.utility/src/types/test_data/dogen_product.cpp(125): Throw in function initialize Dynamic exception type: boost::wrapexcept<dogen::utility::test_data::test_data_exception> std::exception::what: Failed to delete output directory. unknown location(0): fatal error: in "Test setup": std::runtime_error: Error during test /home/marco/nightly/dogen/master/projects/dogen.orchestration/tests/main.cpp(39): last checkpoint: initializer Running 1 test case...
D:\a\dogen\dogen\projects\dogen.utility\src\types\test_data\dogen_product.cpp(125): Throw in function initialize Dynamic exception type: struct boost::wrapexcept<class dogen::utility::test_data::test_data_exception> std::exception::what: Failed to delete output directory. unknown location(0): fatal error: in "Test setup": class std::runtime_error: Error during test D:\a\dogen\dogen\projects\dogen.codec\tests\main.cpp(35): last checkpoint: initializer Running 1 test case...
The problem is a race condition on how we are using the filesystem. The product initialisers are recreating the top-level product directories, and this causes a race condition between the tests generating code and the initialiser. We need to have a way to setup / clean each test so that they do not affect each other.
We only seem to have three tests that actually write to the filesystem. So to fix this:
- remove the recreation of directories from the product classes. Add it to utilities.
- add a unique prefix to each test’s output directory and recreate that directory.
- add comments on the tests where we do not write to the filesystem to make it more obvious.
We don’t really need a builder and a factory. Also remove the various flags we left scattered to handle diffing, reporting etc.
Since we list travis we lost support for nightlies.
Since we list travis we lost support for CI.
Since we list travis we lost support for CI.
Merged stories:
Add github actions build for C#
We need to build on .Net 6.
Rationale: the changes to test structure resolved this issue.
We are experiencing a strange CI error:
D:\a\dogen\dogen\projects\dogen.utility\src\types\test_data\dogen_product.cpp(125): Throw in function initialize Dynamic exception type: struct boost::wrapexcept<class dogen::utility::test_data::test_data_exception> std::exception::what: Failed to delete output directory. unknown location(0): fatal error: in "Test setup": class std::runtime_error: Error during test D:\a\dogen\dogen\projects\dogen.orchestration\tests\main.cpp(39): last checkpoint: initializer Running 1 test case... *** No errors detected Test setup error:
We also have this related error:
Running 1 test case... /home/runner/work/dogen/dogen/projects/dogen.orchestration/tests/code_generation_chain_tests.cpp(169): fatal error: in "code_generation_chain_tests/empty_folders_are_deleted_when_delete_empty_folders_flag_is_on": critical check are_generated_files_healthy(od, t, 60 ) has failed *** 1 failure is detected in the test module "dogen.orchestration.tests"
Rationale: Actually it seems we are not compiling this code as it stands so for now its OK to leave it as is.
Last sprint we removed the relational model from Dogen. This sprint we need to g one step further and remove ODB support. Now, we may not need to remove it entirely: the headers Dogen generates are simple C++ headers that do not require ODB libraries to compile, e.g.:
#ifdef ODB_COMPILER
#pragma db object(categories) schema("NORTHWIND")
#pragma db member(categories::category_id_) id
#pragma db member(categories::description_) null
#pragma db member(categories::picture_) null
#endifWe could conceivably continue to generate these, but we must not add the associated ODB files (generated by ODB) because then we pull in the ODB C++ libraries and these are not supported by vcpkg. If we leave the pragmas we at least know we are not making ODB support any worse. This is still useful as we may return to it in the future. It also ensure some variation in the logical model (in particular in the cartridges domain).
Merged stories:
Reference implementation build is borked
We need to upgrade the ODB version of the reference implementation. Annoyingly this will mean hitting the usual issues with vcpkg. We should probably consider deprecating ODB from the reference implementation as well, or at least disabling the building of the generated ODB code.
At present the URLs for the binaries are 404ing. We need to upload binaries manually to the release.
Release notes have been updated:
Rationale: this story and associated tasks have all been implemented.
One very simple way to improve diagrams is to allow users to associate a fragment of PlantUML code with a class, for example:
masd.codec.plantuml: myclass <>-- other_class : test
This fragments are added after the class, verbatim. Its up to the users to annotate diagrams as they see fit, we merely copy and paste these annotations.
In the future, we may spot patterns of usage that can be derived from meta-data, but for now we just need the diagrams to be usable like they were in Dia.
Notes:
- notes are not indented at present.
- we are not leaving a space after inheritance.
- empty classes still have brackets.
- no top-level namespace for model. We didn’t have this in Dia either.
Tasks:
- add new feature in codec model.
- add properties in model and element to store the data.
- when converting into PlantUML, output the new properties after dumping the class.
- move codec to codec tests from orchestration to codec component.
- codec needs to have a way to bootstrap its context without requiring orchestration.
Rationale: this has been done in Dogen.
Instead of a product level models directory, we should have separate component
level directories. We can’t do the PMM implementation just yet but we can use
regexes to get the directory in the correct shape and then use it to target the
changes in the PMM. The directory should be called modeling to reflect the
fact that it will contain more than models.
Notes:
- when we do this we will break the dogen product unit tests.
- we need to add the targets to each component (generation, conversion).
Last sprint we added the PlantUML verbatim property, i.e.:
:masd.codec.plantuml: model o-- element : composed of :masd.codec.plantuml: Element <|.. model
This was meant to allow us to add the missing relations in the PlantUML diagrams. However, there are issues with this approach:
- we may enter invalid PlantUML syntax, and will only find out at diagram generation time. The error will probably be very hard to figure out as well.
- we need to know the exact element name. Given the “spaces for underscores” approach, this is not very nice (e.g. we replace “a model type” with “a_model_type”).
- if you rename a type, this will fail.
Seems like a better approach is to name the relations and add them as codec attributes:
:masd.codec.composition: 294DC761-8784-3D74-824B-48E7BCC2CFB2, description :masd.codec.aggregation: 294DC761-8784-3D74-824B-48E7BCC2CFB2, another description :masd.codec.association: 294DC761-8784-3D74-824B-48E7BCC2CFB2, yet another description
These relations then give rise to a mapping to the element name during resolution. This copes with renames.
Notes:
- actually, this story is related to the modeling of relationships in general. We need to look through the backlog to find out what analysis had been done on this and see how much of it is needed in order to implement this functionality.
- we need to split out two different activities. The current activity is just to get the PlantUML diagrams into a usable state. If we get side-tracked into solving relations in general, this will take too long. Also, by manually updating diagrams with Verbatim we will get a much better handle on the use cases, and we can then replace those over time. For now, unwind any changes we did for this and put this story in the backlog.
Links:
Merged stories:
Consider modeling relations at a higher level of abstration
Note: this story captures the high-level analysis for implementing relations across dogen. We then need to create specific stories for its implementation.
At present we model relations in logical model as two object templates:
Generalisablefor inheritance (implements and extends).Associatablefor composition.
In reality, we should have created the UML relationships as a top-level construct:
- association: composition, aggregation
- dependency
- generalisation
- realisation
Relationships should have an associated comment or description.
This story implements the functionality described in this story but only as far as the codec model is concerned.
Notes:
- relationships should already exist in the codec model. These exist for “local” relationships only (that is, elements in the same model). They can be used for generalisation. This does mean generalisation could be “remote” though as we some times inherit from other diagrams. We need a way to distinguish between local and remote relations, which could be by “resolving” the GUID into an element.
- relationships can be user-annotated, and used for UML diagram generation.
- generalisation and realisation remove the need for the parent meta-data.
- relationships can be derived from attributes. This is what the “resolver” does. It is in fact not a resolver but a transform that converts properties in the element into relationships.
- relationships should use the GUID as well as the qualified name.
- relationships should really be modeled as org-mode headings. However, one downside of this approach is that we will create a lot of noise when generating documentation. However, given we will only use them for local relationships (generalisation, UML purposes), maybe the noise is not that bad.
- transparent and opaque associations as well associative container keys need to be mapped to the appropriate UML stereotypes. Leaves and root parents as well. If none is appropriate we should create them.
- add a new type of relationship to codec model. We probably also need an enum to capture the type of relationship. This can be supplied in org-mode as meta-data. Relationships belong to elements.
- object templates are incorrectly modeled as stereotypes. These are realisations.
- profiles are also incorrectly modeled as stereotypes. These are also relations. However, the problem will be that once we remove them from stereotypes we cannot see them in UML. We need to have a section in the documentation which shows these properties for an element.
- The name of the relation is its description, e.g. “throws”. We can have duplicate relation names.
- for now, do a hack in the logical model that takes relations of certain types (say realisation) and adds them to stereotypes in the logical model. However, we must be able to ignore other types (say attributes annotated by the user).
- best mapping for org-mode is:
- title is the type we point to.
- description is the name of the relationship.
- attribute
relationshipto denote codec type. - meta-data to denote relationship type.
- add GUID if you want the relationship to show up in PlantUML.
- make object templates interfaces. Modeling a concept is a realisation.
Links:
Rationale: a better approach was implemented by adding state to the model producer.
At present we are using a factory for creating mock configurations. This was fine because we only had one or two variations, so it was easy enough to construct the configuration in one call. However, with variability overrides we now have several different scenarios. It would be easier to have a builder, with sensible defaults, that returns a full configuration which is then supplied to the model generator.
Notes:
- consider adding all variables to the result of the builder, to make the code a bit less repetitive.
GitHub seems to have new security tooling. Enabled but not quite sure what it does.
Links:
At present we are adding notes to namespaces like so:
note top of variability
Houses all of the meta-modeling elements related to variability.
end noteThe problem with this approach is that the notes end up floating above the namespace with an arrow, making it hard to read. A better approach is a floating note:
note A1
Houses all of the meta-modeling elements related to variability.
end noteThe note is declared inside the namespace. We probably need to ensure the note has a unique name. We probably need to use a GUID for the note. Actually maybe we can use the ID of the namespace in the note.
At present model level comments look dodgy:
os << "note as N1" << std::endl
<< m.comment().documentation() << std::endl
<< "end note" << std::endl << std::endl;We should also use GUIDs here.
Links:
At present PlantUML is rendering using the basic dot engine. This results in very horizontal diagrams. It also seems to crash in some cases (not sure if this is dot or not). It would be great to experiment with other layout engines, if they exist.
COMMAND PLANTUML_LIMIT_SIZE=65536 ${PLANTUML_PROGRAM} -Playout=smetana
Smetana fails to generate the diagram, even without namespace to namespace relations:
java.lang.ArrayIndexOutOfBoundsException: 3
at gen.lib.dotgen.mincross__c.left2right(mincross__c.java:369)
at gen.lib.dotgen.mincross__c.transpose_step(mincross__c.java:522)
at gen.lib.dotgen.mincross__c.transpose(mincross__c.java:571)
at gen.lib.dotgen.mincross__c.mincross_step(mincross__c.java:1645)
at gen.lib.dotgen.mincross__c.mincross_(mincross__c.java:627)
at gen.lib.dotgen.mincross__c.dot_mincross(mincross__c.java:194)
at gen.lib.dotgen.dotinit__c.dotLayout(dotinit__c.java:370)
at gen.lib.dotgen.dotinit__c.doDot(dotinit__c.java:492)
at gen.lib.dotgen.dotinit__c.dot_layout(dotinit__c.java:547)
at gen.lib.dotgen.dotinit__c$2.exe(dotinit__c.java:539)
at gen.lib.gvc.gvlayout__c.gvLayoutJobs(gvlayout__c.java:153)
at net.sourceforge.plantuml.sdot.CucaDiagramFileMakerSmetana.createFileLocked(CucaDiagramFileMakerSmetana.java:381)
at net.sourceforge.plantuml.sdot.CucaDiagramFileMakerSmetana.createFile(CucaDiagramFileMakerSmetana.java:336)
at net.sourceforge.plantuml.cucadiagram.CucaDiagram.exportDiagramInternal(CucaDiagram.java:620)
at net.sourceforge.plantuml.classdiagram.ClassDiagram.exportDiagramInternal(ClassDiagram.java:188)
at net.sourceforge.plantuml.UmlDiagram.exportDiagramNow(UmlDiagram.java:135)
at net.sourceforge.plantuml.AbstractPSystem.exportDiagram(AbstractPSystem.java:179)
at net.sourceforge.plantuml.PSystemUtils.exportDiagramsDefault(PSystemUtils.java:209)
at net.sourceforge.plantuml.PSystemUtils.exportDiagrams(PSystemUtils.java:93)
at net.sourceforge.plantuml.SourceFileReaderAbstract.getGeneratedImages(SourceFileReaderAbstract.java:186)
at net.sourceforge.plantuml.Run.manageFileInternal(Run.java:518)
at net.sourceforge.plantuml.Run.processArgs(Run.java:401)
at net.sourceforge.plantuml.Run.manageAllFiles(Run
It was also suggested we try ELK:
COMMAND PLANTUML_LIMIT_SIZE=65536 ${PLANTUML_PROGRAM} -Playout=elk
This layout does appear to be superior to the regular PlantUML dot layout.
Links:
- GH: Alternative layout engines from graphviz #1110
- GH: Allow Arrows in any direction #1078
- Context of the Smetana project
- graphviz: Layout Engines
- Eclipse Layout Kernel
At present, when we add a relation between classes in inner namespaces, it crashes PlantUML. Before we submit a ticket we should update to latest and try to add the relation.
It seems we did some kind of hack to get latest on Debian:
$ cd /usr/share/plantuml/ $ ls -l total 18251 -rw-r--r-- 1 root root 8618641 2020-03-10 2020 plantuml-1.2020.02.jar -rw-r--r-- 1 marco marco 10070645 2022-04-09 09:47 plantuml-1.2022.3.jar lrwxrwxrwx 1 root root 21 2022-04-09 09:49 plantuml.jar -> plantuml-1.2022.3.jar
We should follow the same pattern. We also need to update it in the laptop to avoid oscillation between the two versions and rewriting diagrams each time.
Notes:
- updated to v1.2022.10
We do not seem to have code coverage any longer.
First we started to have very strange errors in the nightly: 3816 failed tests, with errors such as:
valgrind: Fatal error at startup: a function redirection valgrind: which is mandatory for this platform-tool combination valgrind: cannot be set up. Details of the redirection are: valgrind: valgrind: A must-be-redirected function valgrind: whose name matches the pattern: strlen valgrind: in an object with soname matching: ld-linux-x86-64.so.2 valgrind: was not found whilst processing valgrind: symbols from the object with soname: ld-linux-x86-64.so.2 valgrind: valgrind: Possible fixes: (1, short term): install glibc's debuginfo valgrind: package on this machine. (2, longer term): ask the packagers valgrind: for your Linux distribution to please in future ship a non- valgrind: stripped ld.so (or whatever the dynamic linker .so is called) valgrind: that exports the above-named function using the standard valgrind: calling conventions for this platform. The package you need valgrind: to install for fix (1) is called valgrind: valgrind: On Debian, Ubuntu: libc6-dbg valgrind: On SuSE, openSuSE, Fedora, RHEL: glibc-debuginfo valgrind: valgrind: Note that if you are debugging a 32 bit process on a valgrind: 64 bit system, you will need a corresponding 32 bit debuginfo valgrind: package (e.g. libc6-dbg:i386). valgrind: valgrind: Cannot continue -- exiting now. Sorry.
This is likely related to some dist-upgrade. However, the biggest problem is
that the nightly has now gone missing altogether. Nothing in the logs, no email
in cron. Checked job, looks fine. Will try again next day.
Builds are missing because since the last dist-upgrade our PC switches off into sleep mode. However, the right fix is to move nightlies to github (separate story).
It may be useful to commit and push all of the generated code on a nightly. That way we can look at the code when there are problems. The downsides are:
- we need to delete these branches.
- CI is both building and pushing into git, which is not ideal.
Notes:
Perhaps a better way would be to have 2 separate workflows: one just for full generation and one for building that branch.
We used to see travis and appveyor build notifications. We stopped seeing them after moving to github actions. This is useful because we can see them from Emacs in IRC.
Notes:
- it seems the settings have an option for this in webhooks. Redo the hook to see if it helps.
- its now working, but the problem is we are posting as ourselves rather than a BOT. This is very confusing. Also, the details are not very useful:
Marco Craveiro @mcraveiro 22:59 Ref: refs/heads/master Event: push Action: https://github.com/MASD-Project/dogen/commit/17e2909dcdbec19753b72c1669dd7dd02497396f/checks Message: - success Ref: refs/heads/master Event: push Action: https://github.com/MASD-Project/dogen/commit/17e2909dcdbec19753b72c1669dd7dd02497396f/checks Message: - success
- created a BOT account.
Links:
- Gitter: github integrations
- Gitter notify - Github action
- GitHub Developer - Authentication
- REST API
- GH: Using an environment
- How to run a github-actions step, even if the previous step fails, while still failing the job
It seems we have all that is required to run nightlies in github. Its better to do it there because it avoids issues with local PC such as PC is off, cron issues etc.
Notes:
- For some reason we are not setting up vcpkg path in the includes, Try merging steps to see if that makes any difference.
- new warning: “Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: lukka/get-cmake, lukka/run-vcpkg, lukka/run-vcpkg”. Bump versions to latest.
- build is now failing because we need to reconfigure to find the code
generator:
2022-10-14T02:12:45.8641079Z ninja: error: unknown target 'gao', did you mean 'rat'? - we still have the problem of linking:
projects/dogen/generated_tests/configuration_tests.cpp:29:10: fatal error: boost/test/unit_test.hpp: No such file or directoryThe best thing may be to split the nightly into two workflow steps.
- deleted the existing nightly/YYYYMMDD branch which was causing issues:
Switched to a new branch 'nightly' To https://github.com/MASD-Project/dogen ! [remote rejected] nightly -> nightly (cannot lock ref 'refs/heads/nightly': 'refs/heads/nightly/20221112' exists; cannot create 'refs/heads/nightly') error: failed to push some refs to 'https://github.com/MASD-Project/dogen' - nightly builds are failing to compile because the generated tests are using a
CMakeLists with globals:
target_link_libraries(${tests_target_name} ${lib_target_name} ${CMAKE_REQUIRED_LIBRARIES} ${CMAKE_THREAD_LIBS_INIT} ${Boost_LIBRARIES})
This means we can only fix the nightlies by updating CMake templates to take into account dependencies. However, since we managed to compile the nightly locally, this means the main story is complete. We need a separate story to get the nightly to go green.
Links:
Rationale: already implemented.
At present we expect the reference models to be either on the data directory (for system models) or on the same directory as the target. Presumably, users may also want to have models on other directories. For example, if one were to extend Dogen with a different project, it would be required to load models from the dogen directory.
We could simply add a command line argument for reference directories; if the reference is not found in the target model directory, we would then try all available reference directories.
We should implement this as a -I parameter akin to compilers. We should also
have a command to output the current include path - check GCC to see what
command they use for this.
This will be a requirement in order to support PDMs because we shall have many directories with models. We will need the concept of “system include directories” for this. We need to look into how compilers do this. We must also dump these in dumpspecs.
Merged Stories:
Add additional data files directories
Story: As a dogen user, I want dogen to use my own private data libraries so that I don’t have to supply them as diagrams.
Users should be able to provide directories for their own JSON models. We just need to add a new parameter to the knitter and transport it all the way to OM’s workflow.
In the future, when everything is a model, data file directories and reference directories will become one and the same.
Rationale: implemented.
We should not need to run nightlies every night if there are no commits.
Links:
- GH: ci.yml Example action with check.
- SO: How to schedule a GitHub Actions nightly build, but run it only when there where recent code changes?
Rationale: implemented.
At present we are sending gitter out notifications as ourselves, which is very confusing. We need a separate BOT account for this. Unfortunately, this entails creating a new email account, a new GitHub account etc.
Links:
It seems we are bursting the 6h limit for nightlies now. We are running around 900 tests in 6h. We need to reduce the total number of tests run under valgrind.
The solution is to have 2 nightlies:
- a full generation build which runs all tests (e.g. 3.6k).
- a normal build from master which only runs the ~600 tests.
In order to support this use case we just need a flag to disable memcheck.
Notes:
- all builds pass but the fullgen ones did not make it into CTest. Rename builds. This will also help with coverage.
- issues with badges.
- valgrind fails due to CMake version update.
Links:
- SO: How to schedule a GitHub Actions nightly build, but run it only when there where recent code changes?
- Example workflow with check date
Submit issue to CDash project: https://github.com/Kitware/CDash/issues
Actually it was not needed, the problem was with the build name: we can’t have two nightlies with the same name as they overwite each other.
Hi CDash developers,
Thanks very much for a great tool, which I have relied on daily for more than a decade. I have a strange problem, and not quite sure how to debug it further. My open source project [1] makes use of the community CDash instance [2]. In general, all my builds have worked just fine but of late I am having a strange problem with a new set of builds. To the layperson, the CTest submission logs look normal, e.g. [3]:
```
SetCTestConfiguration:BuildDirectory:/home/runner/work/dogen/dogen/build/output/linux-clang-debug
SetCTestConfiguration:SourceDirectory:/home/runner/work/dogen/dogen
SetCTestConfigurationFromCMakeVariable:DropMethod:CTEST_DROP_METHOD
SetCTestConfiguration:DropMethod:http
SetCTestConfigurationFromCMakeVariable:DropSite:CTEST_DROP_SITE
SetCTestConfiguration:DropSite:my.cdash.org
SetCTestConfigurationFromCMakeVariable:DropLocation:CTEST_DROP_LOCATION
SetCTestConfiguration:DropLocation:/submit.php?project=MASD+Project+-+Dogen
Submit files
SubmitURL: http://my.cdash.org/submit.php?project=MASD+Project+-+Dogen
Upload file: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Update.xml to http://my.cdash.org/submit.php?project=MASD+Project+-+Dogen&FileName=github___linux-clang-debug___20221126-0000-Nightly___XML___Update.xml&build=linux-clang-debug&site=github&stamp=20221126-0000-Nightly&MD5=56f692fc3ddcd82bf709978b14f82f8d Size: 553
Uploaded: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Update.xml
Upload file: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Configure.xml to http://my.cdash.org/submit.php?project=MASD+Project+-+Dogen&FileName=github___linux-clang-debug___20221126-0000-Nightly___XML___Configure.xml&build=linux-clang-debug&site=github&stamp=20221126-0000-Nightly&MD5=6ccfef78f7a4fa53aff8bcb633109666 Size: 20947
Uploaded: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Configure.xml
Upload file: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Build.xml to http://my.cdash.org/submit.php?project=MASD+Project+-+Dogen&FileName=github___linux-clang-debug___20221126-0000-Nightly___XML___Build.xml&build=linux-clang-debug&site=github&stamp=20221126-0000-Nightly&MD5=bbf8a77fc3c9de8f93be2f2ce769714d Size: 1120
Uploaded: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Build.xml
Upload file: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Test.xml to http://my.cdash.org/submit.php?project=MASD+Project+-+Dogen&FileName=github___linux-clang-debug___20221126-0000-Nightly___XML___Test.xml&build=linux-clang-debug&site=github&stamp=20221126-0000-Nightly&MD5=10192675e22f3562805fa0e8e4eb2981 Size: 8212762
Uploaded: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Test.xml
Upload file: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Done.xml to http://my.cdash.org/submit.php?project=MASD+Project+-+Dogen&FileName=github___linux-clang-debug___20221126-0000-Nightly___XML___Done.xml&build=linux-clang-debug&site=github&stamp=20221126-0000-Nightly&MD5=c8da7710474223551534d3969a8c4d98 Size: 107
Uploaded: /home/runner/work/dogen/dogen/build/output/linux-clang-debug/Testing/20221126-0000/Done.xml
Submission successful
```
However, nothing comes up in the dashboard. Other builds (both nightly and continuous) display just fine.
[1] https://github.com/MASD-Project/dogen
[2] https://my.cdash.org/index.php?project=MASD+Project+-+Dogen
[3] https://github.com/MASD-Project/dogen/actions/runs/3551520273/jobs/5965773135Rationale: complex, added to the main story.
Original story: In order to communicate with external tools, we should try to support XMI as an injector. For this we could use the XSD for XMI as the basis for the injector. We also need to create the MASD UML profile with full support for all of the logical model elements.
Reason why this is not a good idea:
We have spent a lot of time analysing what should be Dogen’s “native” format. That is to say, given the different ways of representing input models, finding the one with the smallest impedance mismatch between its concepts and Dogen’s conceptual model. XMI and related formats, paradoxically, have a very high impedance mismatch. This is because:
- creating documents in XML/XSD is not easy to do manually; they have a lot of infrastructure which is just noise for humans. We could easily write an exporter from emacs that outputs into XMI, but the question then is: why spend time on that engineering effort when we do not have a use case for it? We can always tackle this in the future if such use case arises, but what is clear is that XMI is not the “natural” representation.
- we do not care about the serialisation of instances at the modeling level. This is to be handled at the generated code level. We do not expect any interaction between meta-models and instances. We never authored instance documents nor is this expected to ever be a requirement.
- we do not care about open ended models, or meta-meta-modeling.
- parsing XML/XSD is full of corner cases, and the mapping will be by necessity incomplete. That is to say, even though we could add support for XMI, it is likely we’d have to test each tool’s output to make sure we are mapping entities correctly.
- org-mode is by far the best input format in which to write models in, including support for a lot of important features such as code snippets, org-roam, org-brain etc. That is because our approach is based on literate modeling.
Thus, in conclusion, org-mode is the native format for Dogen. It makes more sense to create external adapters outside of Dogen which generate org-mode from whatever application format than to carry these converters within Dogen. It would be very difficult for Dogen to prove the conversion is done correctly, and we would have to carry N different adapters with formats which we do not know and have no easy way to validate or update.
Previous understanding:
We now understand most of the picture of how UML is related to Dogen. eCore is
an implementation of UML’s meta-model - it diverges from UML slightly, but for
our purposes we can think of it as equivalent to UML’s meta-model. At the
meta-model level, it is possible to extend UML: this is done via profiles and
stereotypes. Profiles are basically a collection of types which make use of UML
stereotypes to extend the UML meta-model. So it is possible to inherit from say
Class and create a new meta-class that extends Class in some way (in eCore
terms, eClass).
For instance, we could create a meta-class called visitor. In fact, all
meta-types in masd can be redone as extensions to the UML meta-model (e.g.
MASD profile). It just so happens that we did not understand the UML meta-model
properly so we created a completely separate meta-model and then we mapped the
Dia representation of the UML meta-model into yarn. But this muddles things
because we are using Dia’s diagram meta-model, which contains an incomplete
representation of UML’s meta-model (since it exists purely for drawing
diagrams). A better way to go about this is:
- create a c++ implementation of the ecore model. Use exactly the same
terminology - e.g.
eclassetc. - create a dia to XMI exporter. This would have obtained the essence of the UML diagram required for code-generation, removing all of the “diagramatic” aspects of the model. XMI example:
<?xml version="1.0"?>
<XMI xmi.version="1.2" xmlns:UML="org.omg/UML/1.4">
<XMI.header>
<XMI.documentation>
<XMI.exporter>ananas.org stylesheet</XMI.exporter>
</XMI.documentation>
<XMI.metamodel xmi.name="UML" xmi.version="1.4"/>
</XMI.header>
<XMI.content>
<UML:Model xmi.id="M.1" name="address" visibility="public"
isSpecification="false" isRoot="false"
isLeaf="false" isAbstract="false">
<UML:Namespace.ownedElement>
<UML:Class xmi.id="C.1" name="address" visibility="public"
isSpecification="false" namespace="M.1" isRoot="true"
isLeaf="true" isAbstract="false" isActive="false">
<UML:Classifier.feature>
<UML:Attribute xmi.id="A.1" name="name" visibility="private"
isSpecification="false" ownerScope="instance"/>
<UML:Attribute xmi.id="A.2" name="street" visibility="private"
isSpecification="false" ownerScope="instance"/>
<UML:Attribute xmi.id="A.3" name="zip" visibility="private"
isSpecification="false" ownerScope="instance"/>
<UML:Attribute xmi.id="A.4" name="region" visibility="private"
isSpecification="false" ownerScope="instance"/>
<UML:Attribute xmi.id="A.5" name="city" visibility="private"
isSpecification="false" ownerScope="instance"/>
<UML:Attribute xmi.id="A.6" name="country" visibility="private"
isSpecification="false" ownerScope="instance"/>
</UML:Classifier.feature>
</UML:Class>
</UML:Namespace.ownedElement>
</UML:Model>
</XMI.content>
</XMI>- note that the dia to XMI exporter would have to make use of some “extensions”
to dia such as the
yarn.dia.comment. But the key thing is we would end up with a UML meta-model description of the diagram (XMI is roughly speaking an XML representation of the UML meta-model). - create a UML profile for
yarndescribing all of the meta-model elements in terms of the UML meta-model. This could be the exact same diagram we have foryarnright now, except all the meta-model entities would be marked asstereotypes(i.e. with a UML stereotype ofstereotype) and would have to inherit fromecoreclasses such aseclass. - create a
ecoreXMI reader. - create a
yarn.ecoretransformer. The transformer outputs ayarnmodel, just like the currentyarnfrontends would. The remaining of the processing is done as it is for the currentyarnmodels. - in the main
yarnworkflows: whenever it encounters a class with a stereotype ofstereotypeit knows it must inherit from the appropriate meta-model. It effectively expands these meta-stereotypes into inheritance relationships from the meta-model. - create a UML profile for
quilt.cpp(andquilt.csharp). Again, this could be the exact samequilt.cppmodel we have, except all of the meta-model entities are described as meta-stereotypes, and inherit from theyarnmeta-model entities. This means that instead of extending theyarntypes via composition, we would have to use inheritance. This is somewhat non-trivial because we need to extend the base classelementsince allquilt.cppelements have common properties. - note: it is important to note that we’re not interested in eCore models per
se. To generate an eCore model requires having the eCore model at run-time as
well. This is a completely separate kernel from
quilt. The kind of models we’d be generating would be very different from thequiltmodels (generated types would inherit fromeObject, they would have reflection support and so forth - all features that we are not interested in at present). However, eCore as a plain meta-model is useful because it is directly related to XMI and to UML profiles (since its just UML meta-model in disguise). - proxy models now become just regular XMI models (either JSON or XML) with suitably annotated models indicating its a proxy model.
- the total set of expanders are our implementation of meta-model to meta-model transformation. Some of these expanders are replaceable by generated code: all that are simply reading meta-data. Others must remain. This task will be tricky because we have mixed the meta-data expansion with additional logic in the expanders. We now need to decouple the two.
- we will lose scoping on the tagged values. We can still have some kind of
hierarchical structure, perhaps
MODEL.CLASS.ATTRIBUTEbut this is now limited to the meta-model structure because we must be able to mechanically map from a tag value to a class and attribute. We should also read the UML spec on valid tagged values in case they forbid some characters such as dots. - up to now we have extended the dia format to perform yarn-like operations; for
example, we read stereotypes in the
yarn.diafrontend and convert them intoyarntypes. In an XMI world this will no longer happen. We must pass-through the dia stereotypes into a XMI class. All changes to dia must be seen as XMI extensions.
Advantages of this approach:
- we’d be relying on standards: any tool that can output XMI can be used as a front-end, and all the extensions used for Dogen are part of XMI/UML. Nothing is proprietary.
- since all extensions to meta-model elements are done via profiles, we are able
to figure out where the meta-data is (all tagged values in the profile are
annotations). It is still not totally clear how one would map the
annotationtypes (e.g.text,numberetc) to programming language types, but the gist is we should be able to generate the factories that read meta-data and instantiate objects. Not all mappings are trivial of course. - we can create arbitrary extensions to the meta-model just by using the meta-stereotype. Users can create their own on their diagrams, and because of the stitch integration, they can already add their own templates. This makes the system arbitrarily extensible. Of course, we still need to plug in the new meta-model items to the pipeline (how would they get populated?).
High-level tasks to get us there:
- create a
ecoreyarnmodel. - add a
ecore.xmimodel that reads XMI and instantiatesecore. Actually we should probably create anxmimodel that reads XML and JSON and then use that model as input for theecore.xmitransformer. - change the
yarnmodel to inherit from the appropriateecoreentities. This will be a bit problematic if there is use of reflection (e.g. eObject etc). - transform
quilt.cppandquilt.csharpintoyarnextensions. Instead of composition we need to use inheritance somehow. Composition is possible of course, but basically we need to be able to express it as a UML profile. - add a
yarn.ecoremodel that convertsecoreintoyarn. - update
yarnto understands meta-stereotypes. When it sees a stereotype ofstereotypeit looks for a tagged value for inheritance and instantiates the inheritance relationship of meta-model items. In practice we can support this right now in yarn, there is no dependency on anything else for this. Users just need to add the meta-model to their list of referenced models, add a stereotype ofstereotypeand the appropriate generalisation relationship. Thestereotypeis just for cosmetic purposes,yarnworkflows would already inherit correctly. - create a
dia.xmimodel that outputs XMI documents, possibly in JSON as well. Delete the currentyarn.diafrontend and rewrite it in terms ofdia.xmiandyarn.xmi.
Note:
- it is interesting to notice that in practice we don’t actually need to link
ecoretoyarn(inheritance, etc), provided that we support interoperability between the two: stereotypes, tagged values, etc. As long as the mapping results in the correct instantiation of theyarnmodel, this is semantically equivalent to makingyarna proper specialisation of the UML meta-model. Problems start when users try to make use of UML meta-model types for their own templates though. - it seems there is no publicly available XMI XSD for UML. Actually it seems
there is no proper UML support in XMI and each vendor has created incompatible
extensions. See:
- https://bugs.kde.org/show_bug.cgi?id=56184
- http://plantuml.com/xmi
- https://github.com/plantuml/plantuml
The only solution for this problem is to create a “libxmi” that has many frontends and supports a single object model that covers all of UML as per OMG UML XMI. Then manually create XMI XSD schemas for each variant of UML, use the XSD tool to create C++ representations of these and manually create mappings between these dialects and the standard UML object model. One should try to make the mappings bidirectional so that the library could be used to create a converter between XMI formats. However, this is a very large task and its not clear its worth it. Actually we probably want a “libuml” product which contains the XMI adapters. It should also contain other adapters such as Dot/GraphViz, Dia, etc. It should also define its own libuml XMI with XSD. It should also include OCL parsing. It could share code with umbrello.
Links:
- EMF4CPP (see also https://github.com/mdoerfel/emf4cpp/tree/v2)
- MDE4CPP
- Standard stereotypes for UML models
- XMI XSD v2.5.1
- XMI specification v2.5.1
- slides: Introduction to XML Metadata Interchange
Previous Understanding
It would be nice to be able to read an eCore model and generate code from it. This would be particularly useful if we could generate java code. We would require a mapping layer of eCore types into LAM.
We have maintained the old codecs because we thought they could still be useful in the future. However, we now have started to experience limitations due to them: some features are very easy to implement in org-mode but have no direct correspondence in other codecs. And we need to go back and implement tests etc for these codecs to make sure the feature is working. Given we do not see any future in these codecs, we should just remove all functionality related to them.
Notes:
- in this case, the “codec” model should be renamed because its not a generic
model that supports a number of codecs. Its just an intermediate
representation between org-mode and the logical model. A better name would be
“intermediate representation” or
ir. Actually this is not correct: we still need to support two formats: org-mode and PlantUML. Its just that we do not envision adding large numbers of injectors and extractors in the future. - remove
:masd.codec.dia.comment: true
We can keep the existing structure in terms of coding for the reference products, but as far as testing goes we should make these git sub-modules just like we do for VCPKG. This has a lot of advantages:
- code will work for all users when they do the clone.
- no need for special cases in the CMake presets.
- no need to force users to follow a Dogen specific layout just to build and run Dogen’s tests.
Notes:
- always enable tests, do not make it optional.
Building boost-system[core]:x64-windows... CMake Warning at scripts/cmake/vcpkg_buildpath_length_warning.cmake:4 (message): boost-system's buildsystem uses very long paths and may fail on your system. We recommend moving vcpkg to a short path such as 'C:\src\vcpkg' or using the subst command. Call Stack (most recent call first): ports/boost-system/portfile.cmake:3 (vcpkg_buildpath_length_warning) scripts/ports.cmake:147 (include)
Clues about path length:
-- Downloading https://github.com/boostorg/system/archive/boost-1.80.0.tar.gz -> boostorg-system-boost-1.80.0.tar.gz... -- Extracting source D:/a/dogen/dogen/vcpkg/downloads/boostorg-system-boost-1.80.0.tar.gz
Links:
- [[microsoft/vcpkg#11119 Please add advice to enable long paths on Windows 10 #11119]]
- [[microsoft/vcpkg#19141 Please add advice to enable long paths on Windows 10 #19141]]
- Maximum Path Length Limitation
- MAX_PATH lengths on Windows environment #1052
When we install the windows package under wine, it fails with:
E0fc:err:module:import_dll Library boost_log-vc143-mt-x64-1_78.dll (which is needed by L"C:\\Program Files\\DOGEN\\bin\\dogen.cli.exe") not found 00fc:err:module:import_dll Library boost_filesystem-vc143-mt-x64-1_78.dll (which is needed by L"C:\\Program Files\\DOGEN\\bin\\dogen.cli.exe") not found 00fc:err:module:import_dll Library boost_program_options-vc143-mt-x64-1_78.dll (which is needed by L"C:\\Program Files\\DOGEN\\bin\\dogen.cli.exe") not found 00fc:err:module:import_dll Library libxml2.dll (which is needed by L"C:\\Program Files\\DOGEN\\bin\\dogen.cli.exe") not found 00fc:err:module:import_dll Library boost_thread-vc143-mt-x64-1_78.dll (which is needed by L"C:\\Program Files\\DOGEN\\bin\\dogen.cli.exe") not found 00fc:err:module:LdrInitializeThunk Importing dlls for L"C:\\Program Files\\DOGEN\\bin\\dogen.cli.exe" failed, status c0000135
This will probably be fixed when we move over to the new way of specifying dependencies in CMake. Do that first and revisit this problem.
Actually, this did not help. We then used the new VCPKG macro (see links) which now includes all of boost. We are failing on:
00fc:err:module:import_dll Library MSVCP140_CODECVT_IDS.dll (which is needed by L"C:\\Program Files\\DOGEN\\bin\\boost_log-vc143-mt-x64-1_78.dll") not found 00fc:err:module:import_dll Library boost_log-vc143-mt-x64-1_78.dll (which is needed by L"C:\\Program Files\\DOGEN\\bin\\dogen.cli.exe") not found
Notes:
- Check if we are on latest MSVC.
Links:
- CMake: provide option to deploy DLLs on install() like VCPKG_APPLOCAL_DEPS #1653
- InstallRequiredSystemLibraries MSVCP140.dll is missing
- InstallRequiredSystemLibraries purpose
- IRSL should install MSVCP140_CODECVT_IDS.dll: CMake versions after 3.16 should install this DLL.
We seem to have a single warning on OSX:
ld: warning: direct access in function
'boost::archive::basic_text_oprimitive<
std::__1::basic_ostream<char,
std::__1::char_traits<char>
>
>
::~basic_text_oprimitive()'
from file
'vcpkg_installed/x64-osx/debug/lib/libboost_serialization.a(basic_text_oprimitive.o)'
to global weak symbol
'std::__1::basic_ostream<
char, std::__1::char_traits<char>
>&
std::__1::endl<char, std::__1::char_traits<char> >(
std::__1::basic_ostream<char, std::__1::char_traits<char> >&
)'
from file 'projects/dogen.utility/tests/CMakeFiles/dogen.utility.tests.dir/indenter_filter_tests.cpp.o'
means the weak symbol cannot be overridden at runtime. This was likely caused by
different translation units being compiled with different visibility settings.
The flags that control this behaviour are:
cxxflags=-fvisibility=hidden cxxflags=-fvisibility-inlines-hidden
Compare our settings with Boost.
By removing the current settings for OSX we get over 50 warnings:
ld: warning: direct access in function 'boost::test_tools::tt_detail::print_log_value<char [48]>::operator()(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const (&) [48])' from file 'projects/dogen.identification/tests/CMakeFiles/dogen.identification.tests.dir/legacy_logical_name_tree_parser_tests.cpp.o' to global weak symbol 'boost::test_tools::tt_detail::static_const<boost::test_tools::tt_detail::impl::boost_test_print_type_impl>::value' from file 'vcpkg_installed/x64-osx/debug/lib/libboost_unit_test_framework.a(framework.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
In addition it also causes failures in tests:
dogen.utility.tests/resolver_tests/resolver_returns_test_data_directory_for_empty_path dogen.utility.tests/resolver_tests/validating_resolver_returns_test_data_directory_for_empty_paths
Notes:
- try removing special handling for boost.
Since every single warning on my debug builds is related to ```~basic_text_oprimitive```, I decided to investigate how this symbol is exported in boost. We start with macro ```BOOST_SYMBOL_VISIBLE``` which is defined as follows [1]:
> Defines the syntax of a C++ language extension that indicates a symbol is to be globally visible. If the compiler has no such extension, the macro is defined with no replacement text. Needed for classes that are not otherwise exported, but are used by RTTI. Examples include class for objects that will be thrown as exceptions or used in dynamic_casts, across shared library boundaries.
This appears sensible enough. We can see ```basic_text_oprimitive``` making use of it [2]:
```c++
// class basic_text_oprimitive - output of prmitives to stream
template<class OStream>
class BOOST_SYMBOL_VISIBLE basic_text_oprimitive
{
```
In GCC [3] this macro is defined as follows:
```
#define BOOST_SYMBOL_VISIBLE __attribute__((__visibility__("default")))
```
In Clang too [4]:
```
define BOOST_SYMBOL_VISIBLE __attribute__((__visibility__("default")))
```
The general conclusion is that by setting visibility to default we should match the symbols definition. We now turn our attention to the destructor [2]:
```c++
BOOST_ARCHIVE_OR_WARCHIVE_DECL
basic_text_oprimitive(OStream & os, bool no_codecvt);
BOOST_ARCHIVE_OR_WARCHIVE_DECL
~basic_text_oprimitive();
```
The macro ```BOOST_ARCHIVE_OR_WARCHIVE_DECL``` is defined as follows:
```c++
#if defined(BOOST_WARCHIVE_SOURCE) || defined(BOOST_ARCHIVE_SOURCE)
#define BOOST_ARCHIVE_OR_WARCHIVE_DECL BOOST_SYMBOL_EXPORT
#else
#define BOOST_ARCHIVE_OR_WARCHIVE_DECL BOOST_SYMBOL_IMPORT
#endif
```
The macros ```BOOST_SYMBOL_EXPORT``` and ```BOOST_SYMBOL_IMPORT``` are cousins of BOOST_SYMBOL_VISIBLE. Once more, clang and GCC are identical. GCC [3]:
```c++
# define BOOST_SYMBOL_EXPORT __attribute__((__visibility__("default")))
# define BOOST_SYMBOL_IMPORT
```
Whereas Clang says [4]:
```c++
# define BOOST_SYMBOL_EXPORT __attribute__((__visibility__("default")))
...
# define BOOST_SYMBOL_IMPORT
```
This means when we are importing, visibility is not defined. We now need to find out if that is a good thing or bad.
[1] https://www.boost.org/doc/libs/master/libs/config/doc/html/boost_config/boost_macro_reference.html
[2] https://www.boost.org/doc/libs/1_80_0/boost/archive/basic_text_oprimitive.hpp
[3] https://www.boost.org/doc/libs/1_80_0/boost/config/compiler/gcc.hpp
[4] https://www.boost.org/doc/libs/1_80_0/boost/config/compiler/clang.hppSent email to boost users.
Actually a really easy way to test this is to hack a script that overwrites this file in OSX with the fixes and see what happens to the warnings. We can even leave it in for now until the PR is merged.
We were patching the wrong file it seems, the problem is not with oarchive,
its with oprimitive.
Links:
- [[https://stackoverflow.com/questions/36567072/why-do-i-get-ld-warning-direct-access-in-main-to-global-weak-symbol-in-this][Why do I get “ld: warning: direct access in _main to global weak symbol” in this simple code? [duplicate]]]
- xcode with boost : linker(Id) Warning about visibility settings
- Boost linker warnings on OSX #4497
- Strange “direct access” warning on OSX for basic_text_oprimitive #265
It seems clang-format is being used by quite a lot of people to save time with the formatting of the code. More info:
Emacs support:
Links:
- clang-format-check: GitHub Action for clang-format checks. Note that this Action does NOT format your code for you - it only verifies that your repository’s code follows your project’s formatting conventions. Example repos.
- HPX clang format
- Succeeding With ClangFormat, Part 1: Pitfalls And Planning
- example: clang format in rotor
- clang-format-infer GH
- clang-format-configurator
- http://clangformat.com/
- Unformat: Python3 utility to generate a .clang-format file from example code-base.
- C/C++ pre-commit hooks for static analyzers and linters
- quant lib clang format.
- ORE clang format
We need to go through each and every model and add the relations we add in Dia to make diagrams more readable. Models:
- dogen: done
- dogen.cli: done
- dogen.codec: done
- dogen.identification: done
- dogen.logical: done
- dogen.modeling: no changes
- dogen.orchestration: done
- dogen.org: done
- dogen.physical: done
- dogen.text: started
Links:
- Class diagrams: attaining a more “square-like” use of space in large diagrams #1187
- Section “Help on layout” in manual
- Using ELK layout engine
- “The Hitchhiker’s Guide to PlantUML”, section 6. “Layout”
- “PlantUML layout and styles tutorial”
- “PlantUML GraphViz Layout”
We have modified locally the CMakeLists to match the modern approach, but we never updated the templates. As part of doing this, we should remove ODB support. This is because:
- we don’t use ODB at present;
- when we do look into ODB again, it will be done as part of a cartridge framework rather than via build files.
Actually this is a big ask. We have a lot of missing requirements in order to do this:
- component type: library or executable. However, if its executable, we are still building a library and we need to supply dependencies for both.
- missing parts: we need a part for modeling and another for generated tests.
- features: we need a templatised feature which expands across the parts. The feature will carry dependencies.
- the dependency needs to have the following information:
- include: public or private
- standard dogen model or exogenous?
- link
- official location for generated files:
PRIVATE ${stage_inc_dir}/ # generated header files
One possible approach is to create a model element for references which contains all of the required information. Example:
* some reference :reference: :PROPERTIES: :masd.codec.reference: dogen.tracing :masd.logical.reference.type: public :masd.logical.reference.link: boost::boost :END:
:
[[../../dogen.tracing/modeling/dogen.tracing.org]]
Notes:
- we do not want to handle transitive references in this way; from a dogen perspective we want to load these models, but from a code generation perspective we do not want to add references recursively. We want instead to rely on transitivity.
At present we have very large classes (in terms of width) because they have configuration associated with them as stereotypes. This is a particular problem in the text model. Nothing stops us from having a separate way of handling configuration - for example a different property which is not a stereotype. It could be expressed differently in PlantUML - perhaps a separate section as per “Advanced class body”. We could name the section “Configuration” or “Profiles”.
Notes:
- at present we have several different “kinds” of information in the stereotypes
field:
- the meta-type (e.g. enumeration, object, etc). This is probably the most in keeping with UML’s notion of stereotypes.
- the associated object templates used by the class.
- the associated configurations.
We could have two fields for each of these (e.g. templates, configurations) and then combine them all as stereotypes in logical model. This allows us to express them as different groups within PlantUML.
- we should express
masd::objectin the UML diagrams even though its the default. This would make diagrams clearer. - we could create a named section for enumerators, fields, etc.
- we could express the type of an enumeration, if supplied.
- we could express the type of a primitive, if supplied.
- meta information could appear in a group called “meta-information”.
- consider using
structorentityformasd::objectandannotationformasd::object_template. - if class is abstract, use
abstract. - check why feature model is not available on codec to codec transform and see how hard it is to get it.
At present we are hooking directly into the tags within the codec model in order to access meta-data. This is because we only read in profiles etc later on in the transform graph. In fact the problem is somewhat recursive: the fundamental problem is that we did not expect to bootstrap a full context at the codec level; instead we relied on a “minimal context” bootstrapped within the codec model itself, allowing us to run the conversions without needing orchestration. However, this has now proven to be incorrect: we need meta-data in the codec model therefore we should bootstrap a full context before we perform conversion. This requires a fair bit of surgery.
Notes:
- we need to move the conversion tests back to orchestration.
Originally, we introduced tagged values in Dia because we needed to add meta-data to types which was not directly supported by the tool. We soon extended it to all sorts of annotations. But now that we are no longer constrained to Dia, we need to revisit this decision. Fundamentally, there are two kinds of datum modeled as features:
- data which has a functional dependency on the geometry of physical space; and
- data which does not.
The first case involves the use of templates which expand over physical space,
and this cannot be avoided (e.g. whether a facet is enabled or not). The
second case however is quite trivial. In fact, org-mode does not suffer from the
same limitations as Dia; one can add all necessary properties as tags, and these
can be deserialised (manually?) into what we call the codec model at present. In
this particular case, variability is a bit of an overkill: we know precisely
what needs to be read, and where to put it. We could simply add logic around
codec object creation to read these properties in.
Having said that, we would still end up with something looking like the features. This is because we still need code to loop through the list of KVPs, convert them to a well-defined type, etc. So we need variability as-is; its just that we have 2 use cases (regular/static and template-expansion/dynamic). In reality, the key problem we have is that we do not want to pull in the physical model into the codec model. There are two cases:
- in “the real world”: this is a full blown instantiation of Dogen. We do not need to worry about this since the context is created in orchestration. Therefore we do not need to add a dependency because of this, just refactor how context bootstrapping works so that we can have a feature model in the codec context.
- for testing purposes: since in the intermediate model we do not rely on features that depend on template instantiation, we can just use any old set of template instantiation domains. This should be sufficient for the tests.
Notes:
- update the existing workflow for conversion to bootstrap a complete context.
Since we list travis we lost support for nightlies.
Actually we never did have nightlies for C++.
Rationale: this story is too ambitious. We don’t really have a use case for this and it would be non-trivial.
It would be helpful to know what a yarn type means in terms of UML/MOF, and perhaps even explain why we have chosen certain names instead of the UML ones. We should also cover the modeling of relationships and the relation between yarn concepts and UML/MOF classes. This will form a chapter in the manual.
The UML specification is available here and MOF specification is available here.
We need a way to uniquely identify a property. This could be done by appending the containing type’s qualified name to the property name.
See also The Unified Modeling Language for a more accessible treatment.
See Stereotypes of the UML-to-C++ transformation profile for ideas.
Rationale: with the end of academic work these stories no longer apply.
Links: