Inconsistent path resolution with symbolic links between display editor and display runtime

[minijackson]

My realization of the week: at least on Linux, when symbolic links are at play, a/b/../c doesn't necessarily means a/c (b can be a symbolic link, so a/b/.. means the parent of the target directory of b).

When using symbolic linked directories, it seems the display editor and the display runtime doesn't resolve relative paths the same way.

How to reproduce:

• In a given directory, run:

  mkdir -p test/a/b
  ln -sfn ./a/b test/c

• Open Phoebus, and:

  • create a display test-target.bob in the newly created test folder,
  • create a display test-source.bob in the test/c/ folder (through the symbolic link),
  • in test-source.bob, create an Embedded Display, and using the Phoebus interface, make it point to test-target.bob,
  • run the display, and observe Phoebus saying that it can't find the file

Created structure:

.
└── test
    ├── a
    │   └── b
    │       └── test-source.bob
    ├── c -> ./a/b
    └── test-target.bob

The created Embedded Display points to ../../test-target.bob which correct: relative to the display this is test/c/../../test-target.bob which resolves to test/a/b/../../test-target.bob -> test/test-target.bob (and not test-target.bob).

For us, this happened because we develop some of our displays in different Git repositories, and have a different structure to present to the operators in production. For development, we created that structure using symbolic links so that modifying displays in that structure modifies the displays in our Git repositories:

dev-bundle
├── sources
│   └── our-internal-gitlab.example.com
│       ├── git-repo1
│       ├── git-repo2
│       └── etc.
└── views
    ├── LCS
    │   └── LLRF -> ../../sources/our-internal-gitlab.example.com/LLRF-views
    └── MCS -> ../sources/our-internal-gitlab.example.com/MCS-views

[kasemir]

That's a somewhat unfair example because there are at least two paths to test-source.

One is test/a/b/test-source.bob, from which the target can be reached as ../../test-target.bob because test/a/b/../../test-target.bob evaluates to test/test-target.bob.

The other is test/c/test-source.bob, from which the target can be reached as ../test-target.bob because test/c/../test-target.bob evaluates to test/test-target.bob.

When editing test-source.bob and creating the relative path for the embedded content, you need to know if that display will later be opened as test/a/b/test-source.bob or test/c/test-source.bob. You may open it as one in the editor but later call it via the other path in the runtime. There's no solution that works for both.

When you create symbolic links, you might also have to somehow document/explain how these are meant to be used because files now end up with multiple paths in the file system and only one is "correct" for the various use cases like making a 'development' and a 'production' file layout look the same.

create an Embedded Display, and using the Phoebus interface, make it point to test-target.bob

There are two ways to do that in the editor:

1. Manually enter the path
2. Click the ... button next to the path to open a file browser

The file browser tends to then hand us a path were symbolic links have been expanded, which is often NOT what you want.
So you might have to manually enter the "correct" path that meets the ideas behind site-specific symlink gymnastics.

[kasemir]

My realization of the week

It's only Wednesday, who knows what else we might find this week!

seems the display editor and the display runtime doesn't resolve relative paths the same way

There can certainly be differences. A related display path might rely on macros that are available in the runtime but not the editor. You may edit a display in a development file layout that's different from the production runtime layout.
Resolving the path to a related display is inherently a runtime operation. For that reason, the first version of the editor would actually only show related displays as a box with the path name, NOT trying to load & represent the embedded content. That is of course very limiting, you do want to have a decent preview of what it would look like, so you do want to load & show the embedded content in the editor. Still, it might not always work out because the runtime environment will be different.

So basic question here: Is what you see a bug in the editor, or is it an unsolvable problem because you need to manually enter the "correct" relative path, the one that will work in the runtime, and the editor can't predict that when there are multiple possible paths, or the runtime paths are simply not applicable while editing?

[minijackson]

To me, this is a bug in the runtime viewer, the relative path automatically entered by the editor is correct. In the example I showed, if cd test/c and I cat ../../test-target.bob, I get the content of test-target.bob.

Some of my colleagues have had this bug for some time now, and the experience is:

• Use Phoebus' file picker to choose a related Phoebus file (for an Embedded View, action button, etc.)
• See in the editor that it works
• Open the display in the runtime viewer, see that it doesn't work
• Try to change the path to something that pleases the runtime viewer, by manually figuring out the relative path of the target display without symbolic link resolution
• See that it breaks the runtime editor, but it's better than breaking the viewer

If you have that issue, there are no relative path that satisfies both the display editor and the display runtime viewer. Only one ever works.

Note: I've personally stumbled upon a very similar problem during a Python development for checking .bob file relative links, and I've seen that Python has removed some lexical normalization in their "new" pathlib library, exactly because a/b/../c might not mean a/c.

[minijackson]

Sorry, I didn't see your first message

One is test/a/b/test-source.bob, from which the target can be reached as ../../test-target.bob because test/a/b/../../test-target.bob evaluates to test/test-target.bob.

The other is test/c/test-source.bob, from which the target can be reached as ../test-target.bob because test/c/../test-target.bob evaluates to test/test-target.bob.

According to Linux, that's not what's happening. In my test folder:

• running ls test/c/../test-target.bob leads to an error
• running ls test/c/../../test-target.bob works

On Linux, the .. folder in both test/c and test/a/b is the same inode, and points to test/a.

When editing test-source.bob and creating the relative path for the embedded content, you need to know if that display will later be opened as test/a/b/test-source.bob or test/c/test-source.bob. You may open it as one in the editor but later call it via the other path in the runtime. There's no solution that works for both.

It depends on how you handle paths:

• If you use the path: test/c/../../test-target.bob, everything should work, no matter from where you load the display
• If you try to normalize it lexically, and replacing it with just test-target.bob, it doesn't work
• If you resolve the path using Linux' realpath() function, symbolic links gets resolved, and you get test/test-target.bob, and everything should also work

In any case, I think making the display editor and the display runtime viewer consistent between each other should be a good thing™.

[kasemir]

In the example I showed, if cd test/c and I cat ../../test-target.bob, I get the content of test-target.bob.

I don't think we want that to work.

After cd xxxx/test/c, ../../test-target.bob should refer to xxxx/test/c/../../test-target.bob which evaluates to xxxx/test-target.bob, a nonexistent file.

I get it that it might work on a Linux file system. But what if we make the same displays from xxxx/test available for remote access via http://our_display_file_server.site.org/xxxx.
You opened http://our_display_file_server.site.org/xxxx/test/c/main.bob. If an embedded display in there refers to ../../test-target.bob, that's now http://our_display_file_server.site.org/xxxx/test/c/../../test-target.bob which evaluates to http://our_display_file_server.site.org/xxxx/test-target.bob.

The runtime must use a literal evaluation that turns xxx/../yyy into xxx/../yyy, NOT peeking into symlinks and somehow expanding them.

[minijackson]

Ah, I didn't think about HTTP servers… It's true that it's annoying, but if we want to be pedantic, whether HTTP servers resolves .. lexically or by really resolving the path is entirely up to the server's implementation.

After cd xxxx/test/c, ../../test-target.bob should refer to xxxx/test/c/../../test-target.bob which evaluates to xxxx/test-target.bob, a nonexistent file.

I don't get the should in this sentence. Whether the path corresponds to xxx/test-target.bob or xxx/test/test-target.bob is dictated by the filesystem. Trying to handle path differently is going to make Phoebus inconsistent with every other Linux program.

I get it that it might work on a Linux file system. But what if we make the same displays from xxxx/test available for remote access via http://our_display_file_server.site.org/xxxx. You opened http://our_display_file_server.site.org/xxxx/test/c/main.bob. If an embedded display in there refers to ../../test-target.bob, that's now http://our_display_file_server.site.org/xxxx/test/c/../../test-target.bob which evaluates to http://our_display_file_server.site.org/xxxx/test-target.bob.

The runtime must use a literal evaluation that turns xxx/../yyy into xxx/../yyy, NOT peeking into symlinks and somehow expanding them.

Is there something that is inherently bad in having .. in URLs? If so, how about resolving paths lexically for HTTP, but not for the filesystem? After all, the filesystem dictates where the path leads, not Phoebus.

[kasemir]

Don't have the authoritative definition at hand, but https://developer.mozilla.org/en-US/docs/Web/API/URL_API/Resolving_relative_references says

Each instance of ../ strips a folder from the current directory, and then any text after ../ is appended to the remaining base path

That's what we should do for any path, no matter if file system, http, ftp, ... A file system may be constructed with symlinks, a web server may be configured to map paths to resources on the server, ... we can't peek into that, we just treat ".." as "one up".

That's what we do in https://github.com/ControlSystemStudio/phoebus/blob/master/app/display/model/src/test/java/org/csstudio/display/builder/PathTest.java:

        path = ModelResourceUtil.normalize("/old/../new/path/../dir/file.opi");
        assertThat(path, equalTo("/new/dir/file.opi"));

        path = ModelResourceUtil.normalize("https://old/../new/path/../dir/file.opi");
        assertThat(path, equalTo("https://new/dir/file.opi"));

[minijackson]

Each instance of ../ strips a folder from the current directory, and then any text after ../ is appended to the remaining base path

That's what we should do for any path, no matter if file system, http, ftp, ... A file system may be constructed with symlinks, a web server may be configured to map paths to resources on the server, ... we can't peek into that, we just treat ".." as "one up".

That's what we do in https://github.com/ControlSystemStudio/phoebus/blob/master/app/display/model/src/test/java/org/csstudio/display/builder/PathTest.java:

It's true that Phoebus does that for the runtime viewer, but it appears not for the display editor.

I'm not a fan of Phoebus doing that when loading .bob files from the filesystem, because it means that depending on the directory where you load the .bob file, links may or may not be broken.

By using the OS' mechanism, links always work.

[kasemir]

The design was based on getting the same behavior for the file system and web links, as shown in PathTest.
Changing that now in the runtime would be disruptive to existing installations.

I'm OK with updating the editor, with the acknowledgment that runtime and editor can never be the same.
For starters, http resources can't be edited, you can only write to a local file.