Allocate MCP number for "Generalized Modelica URIs for external resources"

[henrikt-ma]

The email thread Problems with section 13.2.3 initiated by @mtiller is becoming too long to remain outside the tracking system. Opening this draft PR so that we can continue the discussion whether to initiate a new MCP here.

[henrikt-ma]

Given that it is impossible for Dymola to deal with a resolution of Modelica URIs based on how a library is stored in the package directory hierarchy, there seems to be two main areas for improving the specification of Modelica URIs for external resources:

1. Using the host part was a mistake since it isn't case sensitive.
2. The lack of support for anything but fully qualified class names make Modelica URIs as well as the organization of external resources hard to maintain.

Both of these should be addressed by this MCP.

Other potential MCP topics that were also discussed in the email thread, but that is outside the scope of this MCP include:

1. Modelica URIs with #icon and #diagram in documentation img tags.
2. Standardized inclusion of minimalistic reference results in libraries.

[henrikt-ma]

For those who didn't follow the lengthy discussion in the email thread, one attempt at improving the Modelica URIs for external resources is as follows.

1. Introduce an operator with function syntax called resolveURI. Unlike a normal function, it knows about its call site, so that it can apply normal lookup rules. It would make sense to standardize on either of the following two variants:

   1. It only operates on constant strings. This way, application of lookup rules doesn't require a runtime representation of the class tree.
   2. It only evaluates at runtime, allowing a built simulation to be transferred from one tool installation to another.

2. Extend the modelica URI format to allow the following forms (examples and more details were included in the email thread, and could be written up again if we decide to proceed with this MCP:

   1. "modelica://host/relpath" (non-empty host) — This is the form defined today, and should probably be deprecated.
   2. "modelica:///~/relpath" — This is a reference to the current top level package. This is useful for organizing all resources in a hierarchy which is separate from the package hierarchy, like the current MSL.
   3. "modelica:///{../}class/relpath" (that is, zero or more occurrences of "../" followed by a classname that may be fully qualified) — Starting from the current scope, each "../" moves one level up the class tree. From that location in the class tree, the class is resolved using normal lookup. A special and important case is to leave class empty, meaning a reference to the class where the lookup starts. The lookup-based rule is useful for organizing external resources with a close coupling to the classes that use them.

3. When mapping the forms with empty host to a resource location, each identifier of the fully qualified class name is mapped to a subdirectory level of the external resource hierarchy (in the same way as the old form with host), where the relpath is resolved within the directory package-resources (alternatively resources.d).

   1. Unlike the old form with host, there is no restriction on the first part of relpath.
   2. However, in the same spirit, the relpath is not allowed to reference anything outside the package-resources directory, so that the only way a Modelica URI can reference a resource is by referencing the class to which the package-resources directory belongs.

[mtiller]

I do not like option 1 at all because of the need to wrap it in a function. This is a declarative approach, we should just use the URI. Browsers don't require us to wrap function call syntax around a URI, JSON APIs that return URIs don't have function syntax. Tacking on such evaluation semantics is, to me, a very bad idea.

2.i - Definitely needs to be deprecated (sooner the better, as far as I'm concerned).

2.ii - Doesn't this encourage a "kitchen sink" approach to resources? Won't everything end up in "Modelica 3.2.1/Resources". That doesn't seem like a good idea at all. The MSL should, at some point, get broken up and this just makes more work for untangling all this stuff.

2.iii - I don't really like relative paths. As far as I am concerned, all Modelica packages should (ultimately) have an implied encapsulated around them. This makes their location completely context independent. This would just work against that.

Option 3 is interesting to me because it means that you can, a priori, determine all class names that can appear in modelica:///<class name>/<relpath> by the presence of the package-resources directory. It tightly associates the directory with the package and it allows you to refer to any package in any package hierarchy (so you can reference resources from another package). This a priori knowledge means that you can actually quickly scan the file system, find all package-resources directories and then look in the corresponding package.mo file and using the package name and the within clause immediately build a map of directories associated with modelica:///<class name> with only the semantics I've described here. That I like. You don't need to worry about name lookup semantics, relative packages or even figuring out exactly where the directory should live from which you resolve the <relpath>. That seems way simpler to me.

Keep in mind that there are use cases that are completely external to a Modelica tool where we might want to reference resources. So don't assume it is a Modelica tool that is resolving these things. That is my biggest beef with option 1. Who says the modelica:///... URI even has to appear in Modelica source code?

[HansOlsson]

This a priori knowledge means that you can actually quickly scan the file system, find all package-resources directories and then look in the corresponding package.mo file and using the package name and the within clause immediately build a map of directories associated with modelica:/// with only the semantics I've described here.

The way you describe it does not work, because in a non-file based storage there is no corresponding package.mo. In addition the discussion in modelica/ModelicaStandardLibrary#2975 would be complicated if changing storage of a package would impact the resources.

[mtiller]

@HansOlsson every storage mechanism will have to define its own mapping. I am speaking here only about file based storage. I don't see anything in what I'm saying that prevents a non-file based storage system from doing whatever it wants...do you? If so, it then seems irrelevant to me what non-file based systems do. Can you explain why you think it is relevant?

With respect to file based approaches, I'm not sure how much the issue you point out impacts things. I would be perfectly happy if the specification said that modelica://<classname>/<relpath> requires that <classname> is stored as a directory (vs. a single file). This is completely aligned with the ticket you point to (at least the initial request in that ticket). But whether we break every package into individual files is not important in this discussions. I'm not saying we need to go that far (which is what Christian was advocating in that ticket). I'm just saying if you want to store resources, you need that package to be stored as a directory. That seems like an extremely reasonable position and it avoids a lot of confusion about the storage scheme.

[henrikt-ma]

2.ii - Doesn't this encourage a "kitchen sink" approach to resources? Won't everything end up in "Modelica 3.2.1/Resources". That doesn't seem like a good idea at all. The MSL should, at some point, get broken up and this just makes more work for untangling all this stuff.

Yes, this is for the kitchen sink approach currently embraced by the MSL. It is only included in the proposal to get the MSL developers on-board.

However, I hear you also mentioning the relation to encapsulated classes, and it makes me think (and maybe this is what you were proposing) that having a dedicated way to refer to the enclosing encapsulation barrier is actually more important than having a dedicated way to refer to the top level package. In fact, having a convenient way to bypass the encapsulation border seems to defy the purpose of encapsulation. Hence, I'd like to change 2.ii as follow for this discussion (if we decide to allocate an MCP number, this sort of change could have been made in-place in a markdown file describing what we want out of the MCP):

• 2.ii "modelica:///~/relpath" — This is a reference to the nearest enclosing encapsulated class, or the current top level package in case no enclosing class is encapsulated. This is useful for organizing all resources in a hierarchy which is separate from the package hierarchy of each encapsulated package, like the current MSL (a Git grep for encapsulated package gives only two matches, and I would guess that these encapsulations don't matter much for the sake of argument here).

[beutlich]

2.ii - Doesn't this encourage a "kitchen sink" approach to resources? Won't everything end up in "Modelica 3.2.1/Resources". That doesn't seem like a good idea at all. The MSL should, at some point, get broken up and this just makes more work for untangling all this stuff.

Really more work? Does not sound too hard for me to collect Modelica/Resources/Data/Electrical, Modelica/Resources/Documentation/Electrical and Modelica/Resources/Images/Electrical for a future Electrical library with separated resources.

[henrikt-ma]

I do not like option 1 at all because of the need to wrap it in a function. This is a declarative approach, we should just use the URI. Browsers don't require us to wrap function call syntax around a URI, JSON APIs that return URIs don't have function syntax. Tacking on such evaluation semantics is, to me, a very bad idea.
…
2.iii - I don't really like relative paths. As far as I am concerned, all Modelica packages should (ultimately) have an implied encapsulated around them. This makes their location completely context independent. This would just work against that.
…
Keep in mind that there are use cases that are completely external to a Modelica tool where we might want to reference resources. So don't assume it is a Modelica tool that is resolving these things. That is my biggest beef with option 1. Who says the modelica:///... URI even has to appear in Modelica source code?

The resolveURI syntax would be a Modelica thing, and would only be used in Modelica code where string constants are just values that don't carry the class tree context needed for lookup-based resolution. Anywhere a Modelica URI has a given class tree context (this includes all annotations for icons, diagrams, documentation, etc), the URI stands for itself. Outside the context of a Modelica class tree, it is obviously meaningless to use Modelica URIs with anything but fully qualified classnames. To make this distinction more clear, I should have made them separate items:

• 2.iii "modelica:///.class/_relpath" (using a fully qualified classname) — This is the form of Modelica URIs to use outside the context of a Modelica class tree.

• 2.iv "modelica:///{../}class/relpath" (that is, zero or more occurrences of "../" followed by a classname that isn't fully qualified) — This form of Modelica URIs is only for use in the context of a Modelica class tree.

Note that the use of encapsulation barriers is making the same sort of assumption about a class tree context as the lookup-based rule. Let's call these together the class tree based Modelica URIs.

I should perhaps stress some of the reasons for using class tree based Modelica URIs when in the context of a Modelica class tree:

• The relation to existing classes in the class tree is the guarantee that an external resource is always associated with an existing class. Without class tree based resolution, it gets much harder to argue that the file Modelica 3.2.3/Modelica/Foo/package-resources/img.jpg shouldn't be referenced by "modelica:///.Modelica.Foo/img.jpg" even though Modelica.Foo isn't a class. With class tree based resolution, a package-resources directory can always be considered dangling as soon as the class it was associated with has been removed.
• Using URIs with referencing the current class or a near enclosing class provides for modular organization of external resources when a library is stored in a (filesystem) directory hierarchy. The resource referenced by "modelica:////images/img.jpg" is stored as close as possible to the class referencing it, only limited by how finely the library is broken down into a directory hierarchy. This makes changes to the external resources local to the part of the library where they are used, makes it easy to reorganize a library's class hierarchy while maintaining a corresponding structure of external resources, etc.
• Since it is not really possible to tell that a string literal should be maintained as a Modelica URI unless it is the direct argument to resolveURI, automatic updating of Modelica URIs when classes are renamed or moved is hard in general, making Modelica URI maintenance a concern in its own. Using URIs with minimal amount of actual identifiers minimizes maintenance work for the URIs when classes are renamed or moved.

[sjoelund]

Using URIs with minimal amount of actual identifiers minimizes maintenance work for the URIs when classes are renamed or moved.

I'm not sure if I like the idea of having relative URIs. They are skipped for file:// URIs for good reason. And in Modelica it would be quite annoying to duplicate a class and having it not work because the corresponding resources were not copied as well. So I think your statement is wrong; the current structure makes maintenance quite easy (unless you want to move files in the resources directory as that is not backwards compatible).

[HansOlsson]

So I think your statement is wrong; the current structure makes maintenance quite easy (unless you want to move files in the resources directory as that is not backwards compatible).

I agree, and I'm also worried that we are discussing large changes here, when we have many opened issues and MCPs that need work.

Obviously we should deprecate "modelica://A/B/d" due to the host-name being case-insensitive in later RFC - but we could just introduce "modelica:///A.B/d" and "modelica:///A/B/d" as alternatives and say that they all map to the same resources if "A" and "A.B" are classes, and then rewrite the text to be more readable.

[henrikt-ma]

Using URIs with minimal amount of actual identifiers minimizes maintenance work for the URIs when classes are renamed or moved.

I'm not sure if I like the idea of having relative URIs. They are skipped for file:// URIs for good reason. And in Modelica it would be quite annoying to duplicate a class and having it not work because the corresponding resources were not copied as well. So I think your statement is wrong; the current structure makes maintenance quite easy (unless you want to move files in the resources directory as that is not backwards compatible).

If you didn't want the resource to be copied, I'd say you placed it too far down in the package hierarchy. If placed at the right level of the hierarchy, I think it would actually make sense to also make a copy of the external resource when making a copy of the class.

[henrikt-ma]

I agree, and I'm also worried that we are discussing large changes here, when we have many opened issues and MCPs that need work.

It is very true that there are many other opened issues and MCPs that also need work. Perhaps we could settle with some things to be specified now, while still making sure that we don't close the door for further improving the handling of external resources in the future. For example, if we omit the relative classnames now, I would like the new URIs to only allow fully qualified names (that is, "modelica:///.Modelica/…", not "modelica:///Modelica/…").

Also note that the proposal, as presented above, doesn't really change anything (except, perhaps, for deprecating the current form). The important part of the proposal is the things that it adds by giving meaning to URIs that didn't have a meaning before.

[mtiller]

I'm generally in favor of simply cleaning up what we have. However, I think that is easier said than done with the current semantics. If I have a URL modelica:///A.B.C/foo.txt, what does that resolve to when I have C stored as C.mo? What directory do we start the ./foo.txt search from? The same directory that C.mo is in?

It is for this reason that I really like the idea that any modelica:/// URIs have to map to an explicitly declared package-resource directory. As I pointed out before, the explicit nature of this makes it so easy to figure out where the files should be found and you need to know almost nothing about Modelica semantics or storage schemes to the point where external tools could easily resolve this.

Bottom line...if we want to just cleanup what we have by deprecating modelica://A.B syntax in favor of modelica:///A.B and cleaning up the specification text, that is fine by me. But I think the "cleaning up the specification text" is going to be much harder than just imposing the package-resource scheme because we'll end up with an only slightly less confusing explanation given the complicated semantics.

[mtiller]

P.S. - I don't really care for the relative URIs either.

[beutlich]

See also #1623.

[HansOlsson]

I would like the new URIs to only allow fully qualified names (that is, "modelica:///.Modelica/…", not "modelica:///Modelica/…").

That seems odd (and a bit excessive) as URIs with a specified scheme normally contain fully qualified names (for file:/// and http://). Thus I would say that the natural interpretation is that they are absolute - similarly as for "import Modelica.Electrical;" (which is also a fully qualified name).

[mtiller]

I agree with Hans, I don't see the need for the leading ..

[henrikt-ma]

The purpose of the leading . is that it is the obvious way of keeping the door open for allowing relative classnames in the future.

It's not like a Modelica URI such as "modelica:///Modelica.Mechanics/c.jpg" would be universal just because the classname is always required to be absolute; you still need a Modelica environment with loaded libraries of selected versions in order to resolve the URI. Given this, I don't see why it would be such a big deal to let this resolution of classnames be a little bit more clever and take the current context into consideration.

Then, I can understand that different people like to organize their projects in different ways, but just because @GallLeo (see email thread) and I are the only ones who can currently relate to the benefit of keeping external resources close to the Modelica sources, it shouldn't lead to standardization of something that works against it. If it was standardized that the classname of "modelica:///Modelica.Mechanics/c.jpg" is absolute (without being a fully qualified name), can someone show what would be a natural way to handle relative classnames if we would change our mind about embracing local storage of external resources in the future?

[mtiller]

@henrikt-ma I think you have it a bit backward here. Modelica is unusual in that it uses path separators that are dots, not (back)slashes. As such, I think leaving off the leading . actually help you here. Adding a leading slash looks like it is a relative path and it takes some work to determine that it isn't. But if you leave it off, it is much simpler. Consider:

• Absolute - modelica:///Modelica.Electrical/A.jpg
• Relative - modelica:///./Analog/C.jpg
• Relative - modelica:///../Electrical/B.jpg

i.e., dropping the leading slash not only still allows for relative paths, it actually makes them easier to identify and it is more intuitive. Compare this to:

• Absolute - modelica:///.Modelica.Electrical/B/C.jpg
• Relative - modelica:///Analog/C.jpg

I would argue that, intuitively, these look backwards. But if I understand your proposal, they are exactly as you intend. To my eye, the former is much more intuitive and easier to see very clearly what is relative and what is absolute (and it doesn't prevent relative paths).

[henrikt-ma]

I think you may have misunderstood what I mean by relative. I don't mean relative in the sense of a relative file path, which is resolved by concatenating it to the current working directory (or the directory of the current file, or some similar reference directory). I mean relative in the sense that the meaning of a classname depends on where in the class tree (abstract syntax tree) is it resolved.

For example, take your URI "modelica:///./Analog/C.jpg", and resolve it from Modelica.Electrical.Analog.Examples.IdealTriacCircuit. To me, it looks as if the intention of this URI is that it should resolve to something like Modelica 3.2.3/Modelica/Electrical/Analog/Examples/IdealTriacCircuit/Analog/package-resources/C.jpg. What I want is a URI with relative classname "Analog" that resolves to Modelica 3.2.3/Modelica/Electrical/Analog/package-resources/C.jpg, by looking up Analog from the current point in the class tree, similar to when declaring a component of type Analog.Basic.Ground.

What I mean by "modelica:///../Electrical/B.jpg" is then to first move one level up in the class tree, and then do the lookup of "Electrical". This form will rarely be of importance; the important form is when you omit the classname, as in "modelica:///..//B.jpg", which resolves to Modelica 3.2.3/Modelica/Electrical/Analog/Examples/package-resources/C.jpg (one level up from IdealTriacCircuit). The same resource could have been specified as "modelica:///Examples/B.jpg", but this URI breaks if the name of the package Examples is changed, where as the "../" URI would remain valid. I don't think we should discuss the use of "../" further until the idea of plain relative classnames in Modelica URIs has been accepted — it's an extension that can be added later in a backwards compatible way.

[henrikt-ma]

Compare this to:

• Absolute - modelica:///.Modelica.Electrical/B/C.jpg
• Relative - modelica:///Analog/C.jpg

I would argue that, intuitively, these look backwards. But if I understand your proposal, they are exactly as you intend. To my eye, the former is much more intuitive and easier to see very clearly what is relative and what is absolute (and it doesn't prevent relative paths).

They only look backwards if you disregard the Modelica syntax. In Modelica, .Modelica.Electrical, is the absolute way of referring to Modelica.Electrical from any point in the class tree. Conversely, Modelica may or may not refer to the top level package Modelica, depending on shadowing that may happen in some parts of the class tree.

To me, the most important aspect is the ability to use relative classnames, not the URI syntax for doing it. However, I would really be in favor of a syntax that isn't totally backwards compared to Modelica syntax, such as "modelica:///.Analog/C.jpg". While the "modelica:///./Analog/C.jpg" that you proposed isn't my favorite, I find it acceptable, in particular in view of also allowing the "../" form.

[mtiller]

Ineed, they only look backwards if you disregard Modelica syntax. But these are URIs which often leverage "file path" like semantics (even when references resources served by an HTTP server). Furthermore, there is no getting around the fact that the file path (not the Modelica class) follows "file path" semantics. So while I completely understand your point about . being the path separator for Modelica classes and even the shadowing concern, I still think using Modelica path separators will be confusing to people.

[henrikt-ma]

So while I completely understand your point about . being the path separator for Modelica classes and even the shadowing concern, I still think using Modelica path separators will be confusing to people.

I guess you're referring to the leading fully qualified marker specifically, and not the classname separators in general? What I don't quite follow then is that your proposal doesn't use a leading '/' to indicate a fully qualified classname. Do you mean that we should sacrifice consistency for the sake of readability? As long as we make this tradeoff clear, I agree that it's an interesting alternative.

[mtiller]

My proposal is that there is no host name. The first element in the path should be the fully qualified Modelica class name using . as the separator but without the leading .. The remaining elements in the path should be the path (e.g., on the file system) relative to the classes storage location. Does that help.

[henrikt-ma]

Yes, I don't think there is anyone in favor of using the host name, except for some form of backwards compatibility variant.

Our proposals only differ in the interpretation of the first element in the path:

• Your way, this must be a fully qualified classname, and if one doesn't consider extension to other ways to reference classes in the future, it makes good sense to exclude the leading fully qualified marker.
• My way, I want to hold the door open for lookup-based resolution in the future, and then it would be more natural to follow Modelica's convention to use a leading fully qualified marker to indicate a fully qualified classname. Still, as I said above, I would consider it an interesting alternative not to do so, in particular in view of also supporting several other forms that would put something else than a classname in the first element of the path, including "modelica:///./Analog/a.jpg", "modelica:///.//a.jpg", "modelica:///..//a.jpg", "modelica:///~/a.jpg", etc.

To me, it seems that the only thing we could be able to agree on regarding the form of the new Modelica URIs boils down to what you are proposing.

It leaves us with the other two of the main items:

1. The operator with function syntax called resolveURI
2. Modelica URI form: "modelica:///Modelica.Electrical.Analog/a.jpg"
3. The package-resources directory

Would it be easier to agree on the need for resolveURI in the Modelica language as the modern replacement for the MSL function loadResource, now that the support for relative Modelica URIs is being withdrawn from the proposal? Note that this function is only intended for use in Modelica code. Outside Modelica code, there will be other ways to know what to interpret as a Modelica URI, and the concept of relative Modelica URIs wouldn't even apply and therefore not be a concern.

Regarding package-resources (that I'd rather call resources.d for brevity) for the mapping to a file system, I don't recall seeing any real objections. Should we simply collect more candidate directory names and poll?

[dietmarw]

@henrikt-ma I'm wondering since there is still this personal branch (should really been done on your personal fork) present if the MCP 0034 should be allocated (and hence the branch merged and deleted)

[henrikt-ma]

@dietmarw, the whole purpose of this PR is to decide whether an MCP number should be allocated or not. As soon as we have decided that we want an MCP for Generalized Modelica URIs for external resources, there will be a branch for it in the central repository that will be used for developing the MCP further. Until there is such a decision, we don't even know that the number will be 0034.

[dietmarw]

@henrikt-ma I understand that, but hence this branch should be on your fork and not he central repository. Otherwise the Modelica Specification repository has soon loads for different WIP branches from different contributors.

[henrikt-ma]

Yes, that's a valid point. On the other hand, I'm not sure we have the solution at the moment, since we've agreed that MCP work should be done in the central repository, not personal forks.

Say we had allocated the MCP number without thinking so much. Then we would have had the MCP branch set up more or less from the start of this discussion, and an MCP working group should have had this discussion around the first iterations of formulating the more detailed goals for the MCP. However, with so much disagreement about what we want, it would have made sense to actually come up with proposals for this somewhere else than on the main MCP branch. If we would have followed the rule that MCP work should be carried out in the central repository, this would be very similar to what we have now, with the only difference that the PR would be targeted towards the MCP branch (which could be named according to which MCP it belongs to, like MCP/0034+scope) rather than master. It seems a little bit better, but not a whole lot better, and certainly not solving your problem with too much going on in the central repository.

[henrikt-ma]

I've now merged with master (resolving the conflict, of course). According to today's web meeting, when should then merge this so that we can get rid of the branch for this PR, and then continue the discussion under the actual MCP.

[henrikt-ma]

I've now merged with master (resolving the conflict, of course). According to today's web meeting, when should then merge this so that we can get rid of the branch for this PR, and then continue the discussion under the actual MCP.

When making that comment, I forgot to re remove the Draft status to reflect the intention of getting it merged. It's been removed now.

[HansOlsson]

Looks ok to allocate the MCP-number.

[henrikt-ma]

Too bad I forgot to add this to the milestone for yesterday's phone meeting.

[HansOlsson]

Hmm.. seemed I messed up the merging. will redo.

[HansOlsson]

Ok, restored.
I will not do as git messages say the next time.

[henrikt-ma]

Please not that there still hasn't been any language group decision on allocating this MCP number. This is why it was scheduled for today's phone meeting.

[HansOlsson]

Phone meeting retroactively allocated this.