Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add extra directory for meta-data transport #586

Merged
merged 5 commits into from Nov 10, 2019

Conversation

@pmai
Copy link
Collaborator

pmai commented Jun 26, 2019

This proposal mirrors the way SSP defines extra directory for additional
meta-data that is supposed to travel with the FMU/SSP, and can be added,
modified, or deleted by tools other than the FMU creator (in addition to
the FMU creator). This mechanism allows for layered standards
specifying the exact nature of added meta-data, and other extensions.

This is the proposed implementation for FCP_018 (issue #585).

This proposal mirrors the way SSP defines extra directory for additional
meta-data that is supposed to travel with the FMU/SSP, and can be added,
modified, or deleted by tools other than the FMU creator (in addition to
the FMU creator).  This mechanism allows for layered standards
specifying the exact nature of added meta-data, and other extensions.
@pmai pmai added the tbd label Jun 26, 2019
@pmai pmai added this to the v3.0 milestone Jun 26, 2019
@pmai pmai self-assigned this Jun 26, 2019
@@ -222,3 +225,10 @@ Furthermore, in order that the C compiler can check for consistent function argu
It would therefore be counter-productive (unsafe) if this header file was present. +
These header files are not included in the `binaries` directory, since they are already utilized to build the target simulator executable.
The version number of the header file used to construct the FMU can be deduced via attribute `fmiVersion` in file `modelDescription.xml` or via function call `fmi3GetVersion`.]_

This comment has been minimized.

Copy link
@t-sommer

t-sommer Jul 4, 2019

Collaborator

We don't currently use bold and UPPERCASE in the documentation. IMHO this excessive formatting makes the text less readable.

This comment has been minimized.

Copy link
@pmai

pmai Jul 5, 2019

Author Collaborator

It is the RFC 2119 keyword formatting that we introduced at the design meeting, which is the usual way the keywords are highlighted (in document formats that offer bold). But these can be changed to just UPPERCASE (or whatever other format that makes them stick out) once we do the document regularization. Only in the specified formatting do they get their normative meaning.

This comment has been minimized.

Copy link
@t-sommer

t-sommer Jul 5, 2019

Collaborator

What would be the problem if we always assume the normative meaning of the keywords (regardless of formatting)?

This comment has been minimized.

Copy link
@pmai

pmai Jul 5, 2019

Author Collaborator

The idea behind RFC 2119 is to be explicit, i.e. to prevent accidental use of words like can, may, etc., becoming normative, and forcing the authors to think carefully about each required behavior.

@@ -43,6 +43,9 @@ resources // resources needed by the FMU (optional)
// In order for the FMU to access these resource files, the resource directory
// must be available in unzipped form and the absolute path to this directory
// must be reported via argument "fmuResourceLocation" via fmi3Instantiate.
extra // Additional meta-data of the FMU (optional)

This comment has been minimized.

Copy link
@t-sommer

t-sommer Jul 4, 2019

Collaborator

If the folder only contains metadata, shouldn't it be meta or metadata?

This comment has been minimized.

Copy link
@chrbertsch

chrbertsch Jul 4, 2019

Collaborator

But then for eFMI from the EMPHYSIS project we would then need on the same level an eFMU folder.

This comment has been minimized.

Copy link
@t-sommer

t-sommer Jul 4, 2019

Collaborator

I was just wondering why "extra".

This comment has been minimized.

Copy link
@pmai

pmai Jul 5, 2019

Author Collaborator

The actual reason: Because SSP already uses extra. And SSP didn't use meta-data because it is not really always clear what is data and what is meta-data. So I think I'll try to reword the comment to not (only) talk about meta-data.

There are probably better names, but it would be nice to use the same as SSP.

t-sommer added 3 commits Jul 10, 2019
@KarlWernersson KarlWernersson self-requested a review Jul 10, 2019
Copy link
Collaborator

KarlWernersson left a comment

ok

@t-sommer

This comment has been minimized.

Copy link
Collaborator

t-sommer commented Jul 11, 2019

In yesterday's steering committee meeting, @chrbertsch proposed to wait until Sept 2 for feedback by @hubertus65 regarding a possible synchronization with an upcoming ISO standard for the storage of meta information.

@t-sommer t-sommer requested a review from hubertus65 Jul 11, 2019
@hubertus65

This comment has been minimized.

Copy link
Collaborator

hubertus65 commented Jul 29, 2019

I forwarded the information about this to Adrian Murton from Airbus, who leads the ISO standardization of what was developed in Mossec. I got a quick thanks back, but I think he is on vacation from now for a few weeks.

@pmai

This comment has been minimized.

Copy link
Collaborator Author

pmai commented Jul 29, 2019

It should be noted that this is not intended to standardize meta-data or other additional data in itself, it is intended as a facility to let additional data travel with the FMU, so that e.g. layered standards (cf. eFMI) and processes can leverage the FMU archive to ensure data travels as one coherent set across process/company boundaries.

As such I do not see much connection with approaches like Mossec, which are more concerned with standardizing meta-data and processes themselves, and rely more on OSLC and other mechanisms to ensure coherence with seperate data travel.

@hubertus65

This comment has been minimized.

Copy link
Collaborator

hubertus65 commented Aug 2, 2019

Pierre, I got that. I just wanted to make sure that our approach is "good enough" for them to add their meta-data in that place.

@hubertus65

This comment has been minimized.

Copy link
Collaborator

hubertus65 commented Aug 2, 2019

I just realized that it would be very easy for all Modelica-based FMU generators to add lot's of useful meta-data under org.modelica by including the html-documentation of all models that are used in the FMU, hierarchically through the components. Currently none of the standards says anything about the generation of FMU's from Modelica, but this would be an easy way to improve the coordination between the standards, and reusability of FMUs.

Copy link
Contributor

andreas-junghanns left a comment

Can you please provide concrete examples for what this could be used for?
How would a reader know what is inside and how to interprete it?
Should we not specify which layered standard to use inside the /extra or /meta directory?

@pmai

This comment has been minimized.

Copy link
Collaborator Author

pmai commented Sep 2, 2019

@andreas-junghanns The idea behind this is that layered standards (i.e. separate documents) specify the rules for the content of specific reverse dns namespaces.

E.g. if ProStep standardised a meta-data Format for models, they’d specify in a document that this meta data file should reside under extra/org.prostep.smmd/modelMetaData.smmd, with referenced resources available using relative URIs from that base URI.

Or if someone defined a digital signature standard for FMUs (based e.g. on the various XML digital signature standards), this could initially reside under extra/com.example.foo/, and if it is later adopted by e.g. MAP FMI, it might move to extra/org.fmi-standard.layered.digsig/ or whatever.

In this sense it is like annotations but for files, and with clearer processing expectations and name spacing rules (if people want to, we could even mandate the reverse dns part; in SSP we thought the recommendation enough, since if you don’t follow it, you are likely to suffer the consequences).

In SSP, this mechanism is currently being used e.g. for traceability extensions, among other things.

In SSP this mechanism is also intended to provide for backward compatible enhancements (minor releases), though we’ll have to see how successful we’ll be in this regard.

@andreas-junghanns

This comment has been minimized.

Copy link
Contributor

andreas-junghanns commented Sep 2, 2019

OK, so we reduce the chance for file/directory mess by requiring all extra files to be in /extra.
Are we now stating somewhere that people cannot use any other folder name in the zip archive?
If not, I think this should be a consequence for creating a place where "extraneous stuff" resides.

@pmai

This comment has been minimized.

Copy link
Collaborator Author

pmai commented Sep 2, 2019

I think this is an orthogonal issue to some extent:

We could still allow FMU generators to put stuff anywhere else as they please, since other tools in the tool chain now have the reserved extra namespace without danger of clashing with FMU generator specific stuff (the generator can put specific stuff into extra, but using its own reverse domain, thus not clashing).

Or we can restrict the generator to only use the known resource/source/binary/documentation/extra/... directories, thus reserving all other top-level directories for future use, or something.

I have no strong opinion on this, and probably keeping the current state (which implicitly allows all kind of content anywhere, it seems to me) is the easiest on implementations.

@andreas-junghanns

This comment has been minimized.

Copy link
Contributor

andreas-junghanns commented Sep 3, 2019

I think we should make this clear: You shall not polute the zip file! There is a corner where you can leave your mess, keep the rest clean. We are in the business of declaring how to behave properly so everyone can produce predictable artefacts and why not be clear about what not to do - especially now that we mandate where and how to create your own little nitche where you will be left alone to do whatever?

@hubertus65

This comment has been minimized.

Copy link
Collaborator

hubertus65 commented Sep 27, 2019

I just got some feedback from the ISO 10303-AP243 group. They like the reverse namespace solution and see that as a viable option to put their meta-data. But they ask for a standard way to find out what's there through an API call. I agree with that. Without an API to figure out what's there, the extra directory is just a dump for information without a way to ask what's there, or add stuff without clashes.

@andreas-junghanns

This comment has been minimized.

Copy link
Contributor

andreas-junghanns commented Sep 28, 2019

I just got some feedback from the ISO 10303-AP243 group. They like the reverse namespace solution and see that as a viable option to put their meta-data. But they ask for a standard way to find out what's there through an API call. I agree with that. Without an API to figure out what's there, the extra directory is just a dump for information without a way to ask what's there, or add stuff without clashes.

I understood the proposal so far that the FMU binary is not effected by what is dumped into the enclosing FMU ZIP file. This made sense to me as the binary should/can not know what someone else later places into the ZIP. An API call (They likely mean a C-API call, right?) would mean the binary needed to know what is in the extra/ directory and that is precicely the role of the resources/ (containing stuff for the binary to work) directory, not extra/ (containing stuff for importers to work). So either I am missing a point here, or I suggest people use a directory crawler to pickup what is there (but see my comment below).

@andreas-junghanns

This comment has been minimized.

Copy link
Contributor

andreas-junghanns commented Sep 28, 2019

Hubertus´ request above raises two questions that implicitly support that the binary itself can check what someone else packed (later) into the ZIP container. If yes, then:

  1. Should we specifically require extra/ to be extracted?
  2. Should we specifically require extra/ to be extracted parallel to resources so it can be found from within the binary in any importing environment?
@pmai

This comment has been minimized.

Copy link
Collaborator Author

pmai commented Sep 28, 2019

@hubertus65 could you elaborate on what kind of API they’d like to see?

As @andreas-junghanns replies, the extra data is intended to be read and manipulated prior to runtime, so the “API” is zip file manipulation.

Conflicts between independent stuff are avoided through the namespacing mechanism; conflicts for the same namespace are to be handled in the way that is specified for that namespace (e.g. whether there can be more than one entry for some meta-data is up to the meta data format spec to spell out).

Currently the extra stuff is not intended as a communication mechanism from the outside into the FMU code itself, but rather as communication between generators, management tools and importers. If we want to enable the FMU code itself to read stuff from extra we should discuss @andreas-junghanns suggested approaches. I don’t think we want to allow FMU code to write to extra, cause that would turn FMUs into containers with offline state...

@chrbertsch

This comment has been minimized.

Copy link
Collaborator

chrbertsch commented Sep 28, 2019

But they ask for a standard way to find out what's there through an API call. I agree with that. Without an API to figure out what's there, the extra directory is just a dump for information without a way to ask what's there, or add stuff without clashes.

@hubertus65 : addtionally to @pmai 's question: could you please also provide a rationale why the ISO working group would need that?

In my understanding the "extra repository" is not "functional" in the sense that its content should not incluence the result of simulating the FMU. This is in contrast to the content of the resources folder, where, e.g. data can be stored that is needed for the execution of the FMU.

For eFMI from the EMPHYSIS project we wouldn't need this API functionality. It is there intended to have in the eFMI subfolder of the extra directory a manifest.xml file describing the content of this subfolder. This manifest will be changed when the content changes, e.g. during different transformation steps.

What I could imagine to have on the top level subfolders of the extra directory a standardized description file, possibly linking to a layered standard that is to be applied for the content.

@hubertus65

This comment has been minimized.

Copy link
Collaborator

hubertus65 commented Sep 29, 2019

Can you please provide concrete examples for what this could be used for?
How would a reader know what is inside and how to interpret it?
Should we not specify which layered standard to use inside the /extra or /meta directory?

Modelica models have the possibility to have embedded html in a document annotation section. Today, that documentation is lost when an FMU is generated. If the documentation features is well used -- it is pretty well supported by most tools -- then this documentation can be very useful for users of the FMU later on.

I am aware that this is moving into new territory: the standards are not completely independent, but we'd specify that if you have model documentation in the model, please move it into that specific location in /extra so it does not get lost.

For example we would have extra/org.modelica/modelDocumentation with .html and e.g. .png files in that directory that can be loaded with any browser. I find this pretty self-explanatory when you have a directory crawler that returns the contents of that directory.

@andreas-junghanns

This comment has been minimized.

Copy link
Contributor

andreas-junghanns commented Sep 29, 2019

Can you please provide concrete examples for what this could be used for?
How would a reader know what is inside and how to interpret it?
Should we not specify which layered standard to use inside the /extra or /meta directory?

Modelica models have the possibility to have embedded html in a document annotation section. Today, that documentation is lost when an FMU is generated. If the documentation features is well used -- it is pretty well supported by most tools -- then this documentation can be very useful for users of the FMU later on.

I am aware that this is moving into new territory: the standards are not completely independent, but we'd specify that if you have model documentation in the model, please move it into that specific location in /extra so it does not get lost.

For example we would have extra/org.modelica/modelDocumentation with .html and e.g. .png files in that directory that can be loaded with any browser. I find this pretty self-explanatory when you have a directory crawler that returns the contents of that directory.

For documentation we have documention/. Can you give us another example?

@chrbertsch chrbertsch self-requested a review Oct 21, 2019
@chrbertsch

This comment has been minimized.

Copy link
Collaborator

chrbertsch commented Oct 21, 2019

It was decided to include this feature in the FM 3.0 feature list in the FMI Steering Committee on Oct 2nd, see minutes https://github.com/modelica/fmi-design/tree/master/Meetings/2019-10-02-FMI-Steering

@t-sommer t-sommer merged commit 2022020 into modelica:master Nov 10, 2019
1 check passed
1 check passed
ci/circleci: build Your tests passed on CircleCI!
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.