[Bug]: bwrap: Can't make symlink at /var/run: File exists, involving /var/run/media

[aaronw2]

Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for a bug that matches the one I want to file, without success.
• If this is an issue with a particular app, I have tried filing it in the appropriate issue tracker for the app (e.g. under https://github.com/flathub/) and determined that it is an issue with Flatpak itself.
• This issue is not a report of a security vulnerability (see here if you need to report a security issue).

Flatpak version

1.12.5

What Linux distribution are you using?

openSUSE

Linux distribution version

15.4

What architecture are you using?

x86_64

How to reproduce

1. flatpak install org.freecadweb.FreeCAD
2. flatpak run org.freecadweb.FreeCAD
3. bwrap: Can't make symlink at /var/run: File exists

Expected Behavior

I expect it to work.

Actual Behavior

bwrap: Can't make symlink at /var/run: File exists

I see this with numerous other flatpaks as well, such as SteamVR.

Additional Information

This was closed in issue #2000, but it clearly is not fixed and is still an issue.

echo $XDG_RUNTIME_DIR
/run/user/500
export XDG_RUNTIME_DIR=/var/run/user/500
Same error.

Note that /var/run is a symlink to /run on OpenSUSE.

[smcv]

As I tried to indicate on [edit: #2710], we are not going to be able to magically solve this without more information, so please provide the output of flatpak run -v -v org.freecadweb.FreeCAD (or any other app that shows the same symptom, the simpler the better). For what it's worth, I can't reproduce this with FreeCAD on Ubuntu 22.04.

There are a lot of possible root causes for this error message, because it's coming from a low-level component that just doesn't have the necessary information to provide more context. #2200 was fixed by #2710; #3477 is the same error message for a different reason; and this one might be the same as #3477, or something different. We can't know without more details.

This was closed in issue #2000

I think you mean #2200. #2000 looks like something unrelated.

[aaronw2]

If I change line 1466 in bubblewrap.c to fprintf(stderr, ...) it works.

[smcv]

You're replacing die_with_error() with fprintf()? That's a workaround, not a solution, and might equally well break something else. bubblewrap is intentionally failing with an error message in situations where what it has been asked to do is impossible, and it's up to a larger component (in this case Flatpak) to ask it to do something that is possible instead.

Please attach the Flatpak verbose log so that we can see what is breaking this.

[aaronw2]

fp.log

I agree it's a workaround, but I've been hitting this bug for quite some time and need a quick way to continue.

[smcv]

F: bwrap --args 38 = ...
...
F:     --dir
F:     /run/user/500
F:     --setenv
F:     XDG_RUNTIME_DIR
F:     /run/user/500
F:     --symlink
F:     ../run
F:     /var/run
...
F:     --symlink
F:     var/run/media
F:     /media
...

[aaronw2]

This also fixed SteamVR for me as well.

[smcv]

I think this is another report of the same root cause as #3477. Key factors:

• you have a symlink in the root directory (in your case it's /media) pointing to a path below /var/run
• /var/run is itself a symlink to /run
• your app has the --filesystem=host or --filesystem=/media permission

Flatpak tries to preserve the symlink structure of the host system to the best of its ability, which includes creating a symlink /var/run -> /run; but that conflicts with the setup that Flatpak already does in /var/run and /run.

Workaround: remove the /var/run/media symlink, and replace it with a symlink to /run/media, which should have equivalent behaviour while avoiding this bug.

I'm closing this as a duplicate of #3477 since it seems to be a good match for what happened there.

[smcv]

Workaround: remove the /var/run/media symlink, and replace it with a symlink to /run/media, which should have equivalent behaviour while avoiding this bug.

Please try reverting your workaround in bubblewrap, and doing this instead.

[aaronw2]

I hit the same problem. /media now points to /run/media instead of /var/run/media.

[smcv]

I hit the same problem

Do you mean you still get "Can't make symlink at /var/run: File exists", even with /media -> /run/media? Or some similar message? Either way, please collect a new log.

It would also be useful to know whether the app has any overrides, similar to #3477 (comment).

Flatpak 1.12.5 is an old version. If you're able to run 1.14.x or even 1.15.x (either at OS level or compiled from source), that has considerably better diagnostic messages for exports/filesystem handling - older versions only tell us what they are doing, but from 1.14.0 onwards it also tells us why.

[aaronw2]

fp.log
Here is a log with the latest version.
F: /var is not a symlink
F: /var/run is a symlink, resolved to /var/../run
F: Trying to export the target instead: /var/../run/media
F: Trying to export read/write: /var/../run/media
F: /run is not a symlink
F: /run/media is not a symlink
F: Will export read/write: /run/media
F: Will create symbolic link: /var/run <===
F: Will create symbolic link: /media

This is the wrong behavior since it has already detected that /var/run is a symbolic link. If the link exists, it should not create a new one. It also does not matter if /var/run is a link to /run or to ../run. I suspect that it will also fail when it tries to create the /media symbolic link since it also already exists.

It might also make sense to use the realpath function to resolve these to absolute paths.

[smcv]

Thanks, that log clarifies what is happening. However, it still looks like #3477. You said "/media now points to /run/media instead of /var/run/media" which would be the workaround for #3477; but in your latest log, Flatpak still says F: /media is a symlink, resolved to /var/run/media which implies that you either didn't use that workaround, or used that workaround but then reverted it.

If you replace /media -> /var/run/media with /media -> /run/media, does that make launching the app succeed?

If yes, then this issue is the same as #3477 (which brings us one step closer to solving it, because we already know how to reproduce #3477 and why it's a problem).

If no, then please share a log using a recent Flatpak, but with the workaround in place, so that you are not experiencing #3477, and we can investigate the new problem that is (so far) unique to you.

This is the wrong behavior since it has already detected that /var/run is a symbolic link. If the link exists, it should not create a new one.

If we were creating symlinks on the host system, you'd be right, but that's not what Flatpak is doing: it's creating symlinks inside the Flatpak sandbox, which mirror the structure of the relevant directories on the host system. The Flatpak sandbox starts as an empty tmpfs, then we mount the Flatpak runtime onto /usr, mount the app onto /app, populate /etc, /var and /run in more complicated ways (see the source code for details), and create symlinks and bind-mounts for all the --filesystem paths that are shared between the host system and the app.

The problem in #3477 is that /var/run, specifically, is something that we can't create in a generic way like this, because a different part of the Flatpak code already creates /var/run as part of our setup for /var and /run.

It might also make sense to use the realpath function to resolve these to absolute paths

Flatpak does use realpath in many places, but we are not just trying to replicate the real directories inside the sandbox, we also need to replicate the structure of symlink -> symlink -> symlink -> real directory. We can't do that with realpath: we have to use readlink and reconstruct the structure from there.

If we didn't do that, then we would get weird bugs when a configuration file or other state refers to the same directory by multiple paths that are equivalent on the host system but not equivalent inside the container. For instance, if you have an external disk mounted on /run/media/usbdisk, and a configuration file that variously points to /run/media/usbdisk/file.1, /media/usbdisk/file.2 and /var/run/media/usbdisk/file.3, then that configuration won't work unless we make usbdisk appear at all of those paths - and we prefer to use symlinks rather than bind mounts for this, because the closer we are to the structure seen on the host system, the better it will work.

I suspect that it will also fail when it tries to create the /media symbolic link since it also already exists.

Similarly, this is not trying to overwrite /media on your host system: that would fail, not just because it already exists but also because you're not root. It's trying to create /media inside the Flatpak sandbox, which is in a tmpfs where /media does not exist, until/unless we ask for it.

[aaronw2]

I'm out of town until next week. I recall testing both cases of /media with the same result. I will double check next week.

…

-Aaron

On Thu, Jan 12, 2023, 6:03 AM Simon McVittie ***@***.***> wrote: As I tried to indicate on #2200 <#2200>, we are not going to be able to magically solve this without more information, so please provide the output of flatpak run -v -v org.freecadweb.FreeCAD. There are a lot of possible root causes for this error message, because it's coming from a low-level component that just doesn't have the necessary information to provide more context. #2200 <#2200> was fixed by #2710 <#2710>; #3477 <#3477> is the same error message for a different reason; and this one might be the same as #3477 <#3477>, or something different. We can't know without more details. This was closed in issue #2000 <#2000> I think you mean #2200 <#2200>. #2000 <#2000> looks like something unrelated. — Reply to this email directly, view it on GitHub <#5255 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADJUAEYX3G43XI7OBV3PXDDWSAFK3ANCNFSM6AAAAAATZHZS3I> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[aaronw2]

It looks like changing the /media symlink fixes the issue. It is now working.

[aaronw2]

I am still seeing this error with Steam flatpak games.

[smcv]

It looks like changing the /media symlink fixes the issue. It is now working.

That implies that at least this part of your issue report was a duplicate of #3477, which is already tracked.

I am still seeing this error with Steam flatpak games.

Even with /media -> /run/media? If yes, please describe how to reproduce the issue, and show me a log, preferably from 1.14.x or 1.15.x. If you have /media -> /run/media and you still get this, then this part will not be a duplicate of #3477.

Some of the details that will be relevant:

• is Steam installed at system-level (from a RPM or similar), or via the unofficial Flatpak app?
• is the issue reproduced by launching Steam itself, or by launching a "non-Steam game" that is itself a Flatpak app, or by launching a Steam game that uses Proton or Steam Linux Runtime, or what?
• what is the exact error message?

[jkaivo]

Not OP here, but I can provide a sample of Steam failing to run when symlinks are involved.

This is with Flapak 1.14.4. The user was created specifically to demonstrate this fault. Beginning with an essentially empty home directory and no user-specific environment variables, I created ~/.var/real-app as a directory with ~/.var/app as a symlink to that. Attempting to run Steam (via the unofficial Flatpak app) fails with:

bwrap: Can't make symlink at /home/steam/.var/app: File exists

Full Logs

It seems like bwrap dies when creating the symlink fails. A possible solution might be for bwrap to not treat symlink() returning non-zero as an error if errno is EEXIST and the existing path is already a symlink to the desired target. I will admit that I don't know how tricky this becomes with the containers involved, or I'd submit a patch myself.

[smcv]

A possible solution might be for bwrap to not treat symlink() returning non-zero as an error if errno is EEXIST and the existing path is already a symlink to the desired target.

That's containers/bubblewrap#549, pull requests welcome (I'm unlikely to have time to implement it any time soon). It's in quite low-level code, but shouldn't be particularly difficult to implement.

[jkaivo]

Pull request submitted. I was able to run Steam in exact same scenario as before without issues.

[AleVul]

I think this is another report of the same root cause as #3477. Key factors:

• you have a symlink in the root directory (in your case it's /media) pointing to a path below /var/run
• /var/run is itself a symlink to /run
• your app has the --filesystem=host or --filesystem=/media permission

Flatpak tries to preserve the symlink structure of the host system to the best of its ability, which includes creating a symlink /var/run -> /run; but that conflicts with the setup that Flatpak already does in /var/run and /run.

Workaround: remove the /var/run/media symlink, and replace it with a symlink to /run/media, which should have equivalent behaviour while avoiding this bug.

I'm closing this as a duplicate of #3477 since it seems to be a good match for what happened there.

This helped me for Steam installed via flatpak on Fedora Silverblue. My use case was that I also allowed additional filesystem paths (that are mounted on separate drive under /run/media/) for games and mesa cache