Modify composefile loading behavior towards relative paths

[simonferquel]

Currently, relative sources in bind mounts in a compose file, are expanded to absolute paths from the client perspective. The only case where it is correct is when the clients points to a single-node swarm/kubernetes cluster running on the same machine.

• If we connect to a remote engine, or to a swarm/kubernetes cluster with multiple worker nodes it makes no sense
• If on Docker for Desktop we run the deploy command from a directory which is not mapped with the same absolute path in the Linux VM, it is broken
• If on Docker for Windows we run the deploy command in Kubernetes mode, it is broken

So now, by default, paths are not expanded locally, and we let the underlying stack backend fail if it requires absolute paths. For local dev purposes, docker stack deploy now supports an --expand-bind-sources flag restoring the old behavior.

[thaJeztah]

This sounds like a huge breaking change, as paths have always been relative to the compose file 😅. I think in the Docker for Desktop situation, the idea was to make the "remote" machine "feel" as if it's running locally (which is why the whole osxfs filesystem was added).

For bind-mounts (volume: /local/path:/remote-path/) this is known to be problematic, because it will create a path inside the VM if it doesn't exist, but for the extended format (--mount on docker run and docker service create, and their equivalent in docker-compose.yml), the service should fail if the path doesn't exist.

[simonferquel]

@thaJeztah, On Kubernetes, it is a bit more problematic: from a windows client, relative paths generate complete garbage (not the valid /c/local/path, as it does not go trough our moby api proxy).

[simonferquel]

(and I am not sure if there is any kind of validation when a pod starts with a mount from a non-existing path, which is far less than ideal)

[simonferquel]

One thing that we could do, is to warn the user when a source path seems to be relative (and advise that old behavior can be reproduced with the additional flag)

[dnephin]

-1 on a breaking change. If you want absolute paths, use absolute paths. The real problem here seems to be:

On Kubernetes, from a windows client, relative paths generate complete garbage

Let's fix that problem instead of breaking everyone.

[dnephin]

Could you open an issue with more detail about where the path is turned into garbage, so that we can discuss solutions?

[thaJeztah]

I think moby/moby#33375 was originally intended for this as well

[simonferquel]

@dnephin @thaJeztah @vdemeester here are 2 scenarios currently completely broken:

• from a windows client, deploy a stack with linux services with bind mounts: on Windows, paths such as /var/data are considered relative and will be translated into some ugly things such as c:\current\working\dir\/var/data.
• from a linux/mac client, deploy a stack with windows services with bind mounts: on Unix-like systems, paths such as c:\data are considered relative and will be translated into /working/dir/c:\data.

One thing I imagined was to get OS information from the node we are connected to and change the path expansion to take this into account, but it would not work: on hybrid clusters you can have Linux masters with Windows workers (or the contrary). I also thought about examining the service image to detect target OS, but with multi-arch images there is no determinist way to detect target OS.

If we want to avoid the breaking change, we can also keep the broken behavior by default and add a --skip-bind-source-expansion flag that would fix it on user demand. In that case, I would suggest that we add a warning on the output when we expand a path telling the user that if the expansion is not expected, they should use the --skip-bind-source-expansion flag.

[simonferquel]

I just added a commit that does just that (default behavior does not change, fix is opt-in).

[dnephin]

Thanks for adding more detail about the problem. I'm still not a big fan of the flag. I think we can handle this better by examining the path. The problem seems to be just the path.IsAbs check.

from a windows client, deploy a stack with linux services with bind mounts: on Windows, paths such as /var/data are considered relative

Do you know where this is happening? In loader.go we're using path.IsAbs() which always checks for path[0] == '/', so that should return the same value (true) on every client, and no paths should be converted to relative paths.

from a linux/mac client, deploy a stack with windows services with bind mounts: on Unix-like systems, paths such as c:\data are considered relative

I think we can fix this. Instead of using just IsAbs we can expand the check to use isWindowsDrive() (edit: actually that signature doesn't work, so we'll need a new similar function). So it would be:

if !path.IsAbs(filePath) && !isWindowsDrivePath(filePath)) {
    filePAth = absPath(...)
}

func isWindowsDrivePath(p string) bool {
    return len(p) >= 3 && unicode.IsLetter(p[0]) && p[1:#] == ":\"
}

Shouldn't that handle both cases?

[simonferquel]

About the windows case, actually this one is not broken. but ./some/relative/path is, and translates to garbage.

In any case, I am not a big fan of finding ways to understand both filesystem path semantics at the same time... There are so many corner cases that we might not anticipate (for example, UNC paths on Windows where you can express c:\my_dir with \\?\C:\my_dir).

[dnephin]

I am not a big fan of finding ways to understand both filesystem path semantics at the same time

Ya, it's not great, but unfortunately this is something we already need to do for parsing volume specs, so we have this problem already. I think we should solve the problem the same way instead of adding confusing flags.

There are definitely a couple edge cases, but I don't think there are so many that it can't be solved. My example code from above can easily handle the UNC paths as well with another condition.

but ./some/relative/path is, and translates to garbage.

Where does that translation happen?

[simonferquel]

./some/relative/path is relative both in filepath and path packages on Windows. This is obviously a linux style path, but the result of joining it seems not quite predictable (not to mention that it makes absolutely no sense)

[dnephin]

The windows client should be able to understand unix paths. I think the problem is that when we added the file.IsAbs check we forgot to fix loader.absPath(). It is still using filepath, so it generates bad paths. We've been handling a lot of this platform differences in Docker for Windows, so now that it isn't part of the chain we need to handle more in the client.

We should use filepath.FromSlash() / filepath.ToSlash() , or path.Join() so that it handles both types of paths.

[simonferquel]

The problem with filepath.FromSlash / ToSlash is that there implementation is different on Windows and MacOS/Linux (on unix-like, FromSlash does nothing).

I suggest that we create our own cross-platform (without platform-specific code) filepath package with the following features:

• heuristic to detect a given path characteristics: target os (windows or linux), kind (relative/absolute/home-rooted)
• primitives to join paths, resolve relative paths, expand home dir etc. in a way that works x-plat (make it possible to join windows paths from a unix machine, or unix paths from a windows machine)
• convert relative/home-rooted paths (but not absolute paths) from one os to the other
• tools to normalize paths

This way we would have a comprehensive that we could trust to have the same consistent behavior on all platforms we support.

[simonferquel]

@dnephin @thaJeztah I've put together a cross platform paths library that is capable to handle both windows and unix style file paths, convert them, combine them, and do extra stuff like detecting best possible os-match for a given path. It supports the various weird Windows paths (like the win32 filesystem namespace, and device namespaces used for using non-shell-supported chars in file paths, or referencing named pipes), and this fix the original issue for this PR

[thaJeztah]

Can you make sure that repository has a license before we merge?

[simonferquel]

@thaJeztah done!

[vdemeester]

LGTM 🐯
But needs to be 2 commits (one for the vendor, one for the change)

cc @dnephin

[dnephin]

These tests look great, but we're missing test coverage for the crosspath.HomeRooted case, arent' we?

There is gotestyourself/env.Patch for patching HOME which should make it easier to test

[mat007]

For what is worth with this PR two tests now fail when running the unit tests on Windows in Powershell:

PS C:\dev\src\github.com\docker\cli> .\scripts\make.ps1 -TestUnit
…
--- FAIL: TestGetConfigDetails (0.01s)
        loader_test.go:29: assertion failed: expected map[CommonProgramFiles:C:\Program…
…
ack\loader] (length 50) to have length 49
--- FAIL: TestGetConfigDetailsStdin (0.00s)
        loader_test.go:46: assertion failed: expected map[windir:C:\Windows CommonProgramW6432:C:\Program…
…
ISION:8e0a] (length 50) to have length 49

(ellipses added for clarity)

[mat007]

I'm coming a bit late to the party but my understanding is that what is needed is a library for path handling which behaves as path/filepath would if os.GOOS were windows but on non-windows platform, right? Sort of the reverse of what path does for linux-style paths on all platforms (including windows)?
That way we could use path to target linux paths and winpath (or whatever) to manipulate windows paths.
So I'm wondering:

• is something missing from path as it is for linux style paths?
• couldn't the new library be a (almost) drop-in replacement for path and filepath (i.e. same functions, etc…)?
• am I missing something? ;)

[simonferquel]

@mat007 there are some corner cases where we have to combine windows absolute paths with linux relative paths or the contrary (like, handling ~/data or ./data from a windows machine). So we need a lib that can:

• have good heuristics to detect for which target OS a path string is likely written for
• have some rules to allow appending relative paths to existing paths and do automatic conversions from one os to the over
• have the exact same behavior on all platforms

[thaJeztah]

I'm coming a bit late to the party but my understanding is that what is needed is a library for path handling which behaves as path/filepath would if os.GOOS were windows but on non-windows platform, right?

I think that's definitely something that's missing in the go library (perhaps also the reason why I was looking at this library - see my review comments). Basically filepath.SetOS("windows") / filepath.SetOS("linux") to use Windows semantics on Linux and vice-versa.

have good heuristics to detect for which target OS a path string is likely written for

Combined with the above, that would become;

fmt.Println(pathutil.DetectFromPath(`C:\Users\foo`))
# windows

fmt.Println(pathutil.DetectFromPath(`\\hostname\path`))
# windows

fmt.Println(pathutil.DetectFromPath(`\\?\C:\Windows\foo`))
# windows

fmt.Println(pathutil.DetectFromPath(`/usr/share`))
# unix

fmt.Println(pathutil.DetectFromPath(`~/.docker`))
# unix

fmt.Println(pathutil.DetectFromPath(`/usr/some\ path\ with\ spaces`))
# unix

And feed the result to filepath.SetOS()®

have some rules to allow appending relative paths to existing paths and do automatic conversions from one os to the over

Append relative to absolute paths is what filepath.Join() does; https://play.golang.org/p/Ejeng2FkGtz, perhaps in some cases assisted with filepath.IsAbs(), filepath.Abs()

automatic conversions from one os to the over

Which would be a combination of pathutil.DetectFromPath() (detect path style) and filepath.FromSlash() (and a counterpart to that doing ToSlash() perhaps)

[thaJeztah]

Stumbled upon this issue; we should check if this PR resolves that as well; moby/moby#33746

[simonferquel]

@thaJeztah That is exactly the issue it fixes, yes :)

[simonferquel]

@thaJeztah Rebased and added Raw() function and having String() normalize the paths

[simonferquel]

@thaJeztah are you satisfied with the state of the code, or should I change something ?

[vdemeester]

One last question on the new HomeDir field of ConfigDetails.

[vdemeester]

Looking at that function, we should probably create a LoadServicesFromConfigDetails (which would take configDetails as argument instead of those 3) or something.
That said LoadServices is exported so… if we break it once, let's try to break it in a future-proof way

[simonferquel]

Should I add this function alongside the existing one or rename LoadServices to LoadServicesFromConfigDetails (and modify its parameters) ?

[vdemeester]

Hum actually, looking back, I'm not sure anymore 😅

[vdemeester]

Why do we need to pop-up HomeDir in the ConfigDetails ? (also, it's a bit unrelated to the change as we could do the homedir.Get at the same point of the code where we did the lookupEnv("HOME"), right ?

[simonferquel]

The home path resolution on Windows was broken (it did only do a lookupEnv("HOME")), and as we already have WorkingDir in config details, it made sense to also put HomeDir there (or remove both of them for good) for consistency towards ambient path.
Having them both in ConfigDetails makes it easier to test combinations of windows and linux base environments though.

[vdemeester]

Yeah, I'm not a huge fan of that but… ok fair enough 😛

[vdemeester]

LGTM 🐫

[Stijn98s]

Would be nice that this can be fixed

[thaJeztah]

edit:wrong issue

[thaJeztah]

ignore my comment; commented on the wrong issue

[Stijn98s]

@thaJeztah is it possible that this can be implemented?

[simonferquel]

@thaJeztah, @vdemeester feels like I should revive this PR, WDYT?

[Stijn98s]

Is there any workaround for this?

[lk77]

hello,

any news on this ?

[olljanat]

@simonferquel @thaJeztah @vdemeester assuming that this was already approved and only waiting for rebase I actually created rebase one to #1871 PTAL

[karthickj25]

hi,
Is there any workaround available for this?

[thaJeztah]

Closing this one, as there was no consensus. There's a ticket in moby/moby that's somewhat related to this, on which I left some comments; moby/moby#43483 (comment)