[Feature Request] Support Fuse overlayfs for layered WINE prefixes

[pchome]

I'm currently experimenting with such prefixes, but it's hard to manually control all mount/unmount commands. It would be great if q4wine can add experimental support for prefixes mounted with fuse-overlayfs.

More details:

• [Feature request] Reuse common WINE components ValveSoftware/Proton#1011
• q4wine-next: call for ideas suggestions or comments  #69 (comment) (partially applied to 4./5.)
• https://github.com/containers/fuse-overlayfs
• https://en.wikipedia.org/wiki/OverlayFS

  The main mechanics of OverlayFS relate to the merging of directory access when both filesystems present a directory for the same name. Otherwise, OverlayFS presents the object, if any, yielded by one or the other, with the "upper" filesystem taking precedence. Unlike some other overlay filesystems, the directory subtrees being merged by OverlayFS do not necessarily have to be from distinct filesystems.

---

For first step:

• create such prefixes

  • check current WINE version ($ wine --version) and create corresponding base layer e.g. opfx-win64-base-wine-3.15
  • split common components from base to e.g. opfx-win64-common-wine-3.15
  • create "workspace" e.g. opfx-myprefix-user, opfx-myprefix-work and opfx-myprefix

• mount/unmount on game/tool run/stop

  $ fuse-overlayfs -o lowerdir=opfx-win64-common-wine-3.15:opfx-win64-base-wine-3.15,upperdir=opfx-myprefix-user,workdir=opfx-myprefix-work opfx-myprefix
  

  or for opfx-myprefix2

  $ fuse-overlayfs -o lowerdir=opfx-win64-common-wine-3.15:opfx-win64-base-wine-3.15,upperdir=opfx-myprefix2-user,workdir=opfx-myprefix2-work opfx-myprefix2
  

  or add preconfigured template (opfx-myprefix-user layer can be used as template after an "initial setup stage")

  $ fuse-overlayfs -o lowerdir=opfx-win64-template-mypreconf:opfx-win64-common-wine-3.15:opfx-win64-base-wine-3.15,upperdir=opfx-myprefix3-user,workdir=opfx-myprefix3-work opfx-myprefix3
  

  unmount

  $ fusermount3 -u opfx-myprefix
  

Next step:

• convert from/to regular WINE prefix
• advanced manage

[brezerk]

Thank you for the feature request. Sounds interesting. I am on vocation atm, will review it on monday. ср, 5 сент. 2018 г., 7:15 ПП pchome <notifications@github.com>:

…

I'm currently experimenting with such prefixes, but it's hard to manually control all mount/unmount commands. It would be great if q4wine can add experimental support for prefixes mounted with fuse-overlayfs. *More details*: - ValveSoftware/Proton#1011 <ValveSoftware/Proton#1011> - #69 (comment) <#69 (comment)> (partially applied to 4./5.) - https://github.com/containers/fuse-overlayfs - https://en.wikipedia.org/wiki/OverlayFS The main mechanics of OverlayFS relate to the merging of directory access when both filesystems present a directory for the same name. Otherwise, OverlayFS presents the object, if any, yielded by one or the other, with the "upper" filesystem taking precedence. Unlike some other overlay filesystems, the directory subtrees being merged by OverlayFS do not necessarily have to be from distinct filesystems. ------------------------------ *For first step*: - create such prefixes - check current WINE version ($ wine --version) and create corresponding base layer e.g. opfx-win64-base-wine-3.15 - split common components from base to e.g. opfx-win64-common-wine-3.15 - create "workspace" e.g. opfx-myprefix-user, opfx-myprefix-work and opfx-myprefix - mount/unmount on game/tool run/stop $ fuse-overlayfs -o lowerdir=opfx-win64-common-wine-3.15:opfx-win64-base-wine-3.15,upperdir=opfx-myprefix-user,workdir=opfx-myprefix-work opfx-myprefix or for opfx-myprefix2 $ fuse-overlayfs -o lowerdir=opfx-win64-common-wine-3.15:opfx-win64-base-wine-3.15,upperdir=opfx-myprefix2-user,workdir=opfx-myprefix2-work opfx-myprefix2 or add preconfigured template (opfx-myprefix-user layer can be used as template after an "initial setup stage") $ fuse-overlayfs -o lowerdir=opfx-win64-template-mypreconf:opfx-win64-common-wine-3.15:opfx-win64-base-wine-3.15,upperdir=opfx-myprefix3-user,workdir=opfx-myprefix3-work opfx-myprefix3 unmount $ fusermount3 -u opfx-myprefix *Next step*: - convert from/to regular WINE prefix - advanced manage — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#136>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABdDNauPSHvO4cH1HVmIuUJYP_9s7xfks5uX_iGgaJpZM4WbPsy> .

[pchome]

Here is some example scripts (to log required steps).
I used those to create new opfx base for fresh wine version, then using it with new and an old prefixes.

Upgrade from wine-3.15 to wine-3.16 was smooth as expected, except part with .update-timestamp, this needed for wine to not recreate/update prefix.

x_opfx_conf

opfx_dir="/media/Games/wineprefix"
wine_ver="$(wine --version)"

# compatible commons
common_ver="wine-3.15"

x_opfx_init

#!/bin/bash

# config data
. x_opfx_conf

cd "${opfx_dir}"

export WINEPREFIX="${opfx_dir}/opfx-win64-base-${wine_ver}"

# init prefix
[[ ! -d ${WINEPREFIX} ]] && wineboot --init || exit


if [[ ${wine_ver} != ${common_ver} ]]; then
  # reuse commons
  ln -s "${opfx_dir}/opfx-win64-common-${common_ver}" "${opfx_dir}/opfx-win64-common-${wine_ver}"

  # clean prefix
  rm -rf "${WINEPREFIX}/drive_c/windows/Installer"
  rm -rf "${WINEPREFIX}/drive_c/windows/Microsoft.NET"
  rm -rf "${WINEPREFIX}/drive_c/windows/mono"

  rm -rf "${WINEPREFIX}/drive_c/windows/system32/gecko"
  rm -rf "${WINEPREFIX}/drive_c/windows/syswow64/gecko"
else
  # Move common components
  common_target="${opfx_dir}/opfx-win64-common-${wine_ver}/drive_c/windows"

  mkdir -p "${common_target}"
  mkdir -p "${common_target}/system32"
  mkdir -p "${common_target}/syswow64"

  mv "${WINEPREFIX}/drive_c/windows/Installer" "${common_target}/"
  mv "${WINEPREFIX}/drive_c/windows/Microsoft.NET" "${common_target}/"
  mv "${WINEPREFIX}/drive_c/windows/mono" "${common_target}/"

  mv "${WINEPREFIX}/drive_c/windows/system32/gecko" "${common_target}/system32/"
  mv "${WINEPREFIX}/drive_c/windows/syswow64/gecko" "${common_target}/syswow64/"
fi

x_opfx_mount

#!/bin/bash

# config data
. x_opfx_conf

if [[ -z $1 ]]; then
  echo "Usage: x_opfx_mount <prefix name>"
  exit
else
  opfx_name="${1}"
  mkdir -p "${opfx_name}-user" "${opfx_name}-work" "${opfx_name}"
fi

cd "${opfx_dir}"

# unmount
fusermount3 -qu ${opfx_name}

# wine updated
cp "${opfx_dir}/opfx-win64-base-${wine_ver}/.update-timestamp" "${opfx_dir}/${opfx_name}-user/"

fuse-overlayfs -o lowerdir=opfx-win64-common-${wine_ver}:opfx-win64-base-${wine_ver},upperdir=${opfx_name}-user,workdir=${opfx_name}-work ${opfx_name}

x_opfx_umount

#!/bin/bash

# config data
. x_opfx_conf

if [[ -z $1 ]]; then
  echo "Usage: x_opfx_umount <prefix name>"
  exit
fi

opfx_name="${1}"

cd "${opfx_dir}"

fusermount3 -u ${opfx_name} && echo "${opfx_name}: unmounted"

[brezerk]

hi @pchome

jfyi: there is an ability to specify pre/post execution scripts per "Icon":

it does exposing all required wine variables like WINEPREFIX and so on, it could help a bit in testing.

getting back to the idea.

let's clarify the scope:

• you would like to have some hooks to execute x_opfx_init && x_opfx_mount on the application startup;
• you would like to have some hook to execute x_opfx_umount on application tear down. this one probably should not be fatal, as you can have multiple applications running in the same sandbox overlay.

this is pretty match to the pre/post scripts idea implemented on pre icon basis. but you would like to have this for prefix level as well?

[pchome]

I would like an option for prefix settings, which will point a prefix is an overlayed prefix and should be mounted before any wine app/util launched.

Then it should be mounted/unmounted on literally every app/util start/stop.
E.g. winecfg/winetricks/icon/... , then similar actions in other prefix.

In other words : I want internal "links" to work too. And I don't want to keep many prefixes mounted all the time. And all this without manually running anything. Literally built-in overlayfs support.

If this feature don't suit q4wine's goals -- you can close this issue.
Because simply mounting prefix before any actions with q4wine does the same.

[brezerk]

right

so current pre/post scripts are working as you described.

• pre script is executed before any wine process starts up, so it is good idea to put (x_opfx_init && x_opfx_mount) scripts here. actually I think you need to merge this scripts into one script (at least I see no reason not to do so).

• post script is executed after wine process dies. (there can be a race condition tho, b/c some child wine processes like wineserver, winedevice.exe, services.exe still will be running for couple seconds)

• both pre/post scripts are getting WINE* environment variables exposed. so you can change x_opfx_conf to something like this:

opfx_dir="${WINEPREFIX}"
wine_ver="$(wine --version)"

# compatible commons
common_ver="wine-3.15"

to get the opfx_dir dir dynamically.

you can tests this statements to see if it works as expected on Icon level. If it suits fine -- I could add the same settings/mechanic for the Prefix level too.

[pchome]

Ok, I'll do it later.
BTW, I've noticed Proton using wineserver -w, so I'll try this too (in post script).

[pchome]

Let's forget about common stuff, it's too complex for 1st stage and no big difference at the moment.

So using only base as solid newly created WINE prefix, skipping init part, and modified mount/unmount scripts to use $WINEPREFIX env var:

• mount

#!/bin/bash

opfx_dir="$(sed 's/[^\/]*$//' <<< $WINEPREFIX)"
wine_ver="$(wine --version)"

mkdir -p "${WINEPREFIX}-user" "${WINEPREFIX}-work" "${WINEPREFIX}"

# unmount
fusermount3 -qu ${WINEPREFIX}

# wine updated
cp "${opfx_dir}/opfx-win64-base-${wine_ver}/.update-timestamp" "${WINEPREFIX}-user/"

fuse-overlayfs -o lowerdir=${opfx_dir}/opfx-win64-base-${wine_ver},upperdir=${WINEPREFIX}-user,workdir=${WINEPREFIX}-work ${WINEPREFIX}

• umount

#!/bin/bash

opfx_name="$(sed 's/^.*\///' <<< $WINEPREFIX)"

wineserver -w

fusermount3 -u $WINEPREFIX && echo "${opfx_name}: unmounted"

Those two scripts works as pre/post run scripts for icon.

Note:

• opfx_dir is not ${WINEPREFIX} for me, it's q4wine's default directory for new prefixes
• you may want to keep all those *-base,*-work,*-user directories somewhere in q4wine's cache(?) directory
• just in case, if you using gentoo: fuse-overlayfs ebuild

[brezerk]

Hi @pchome

I can merge the scripts, but maybe it will be better to implement this as a Q4Wine "Plugin" and just WGET'ing it from your git repo's releases (just like it is currently done for winetricks). So you will not depend on q4wine's release cycle (which is currently a bit weird I believe :D)

So pre/post run scripts for icon works fine, right?

default directory for new prefixes

I can export this into Q4WINE_DEFAULT_PREFIXDIR env variable or something if you wish.

As for the next moves: how you would like this to be integrated?

[pchome]

I'm not sure I correctly understand how this "Plugin" supposed to work.
Moreover, those scripts meant to be a visual guide, an example how to use fuse-overlayfs. Not for merging as-is into q4wine.

For q4wine I prefer to see something like icotool integration:

1. check if fuse-overlayfs available
   • enable [ ] Use OverlayFS prefixes (experimental) option in preferences for selection
2. if selected: check prefix
   • if overlayed: mount
3. after overlayed prefix mounted, there is no difference (i hope) for WINE compared to regular prefix, so no more work required
4. for now you may keep all of them mounted (no need to unmount every time)

Note: consider to use opfx-$ARCH-$(wine --version | md5sum) for "base".
For e.g. "wine-3.16 (Staging)" versions. A command is an guide, use method preferred for Qt.

[pchome]

Partially resolved by recent WINE/Proton updates.
https://github.com/madewokherd/wine-mono/releases/tag/wine-mono-4.8.1

While fuse-overlayfs remains useful for e.g. moding/DLC mounting/..., there is no needs to manually create shared wine components anymore.

[brezerk]

thank you for the info