Build runtime package with required libs

[plata]

Some applications require additional libraries. As discussed, these shall be provided as part of the Wine package.

We need (each 32 and 64 bit):

• openal (LoL)
• ldap (ElsterFormular)

List to be continued.

[ImperatorS79]

Here is a list computed from running ldd on every .dll.so:
libs.txt

But it is missing the deps of the deps ^^.

But I do not think it should be added inside every .tar.gz, but be downloaded once and installed once (and then use LD_LIBRARY_PATH).

If the file is really big, one could imagine a system that only downloads the libs that are not installed on the user system (using ldconfig -p | grep lib.so.x for example).

We should also try to force the user to install basic i386 on his system in the .deb (using control file ?).

[ImperatorS79]

Ok, here is a little bash script that do the job for the libs.txt I provided:

#!/bin/bash

if [[ -z "$1" ]]; then
	echo "Please pass the lib file as argument"
	exit 1
fi

LIBFILE=$1

while read FILELINE; do
	if [[ $FILELINE != libwine.so.* ]]; then
	    echo "Checking for $FILELINE."
	    EXISTS=$(ldconfig -p | grep $FILELINE)
	    if [[ "$EXISTS" != *"$FILELINE"* ]]; then
	    	echo "$FILELINE is not installed on the system!"
	    	FINDINGS=$(apt-file find $FILELINE)
	    	if [[ -z "$FINDINGS" ]]; then
	    		echo "Seems no .deb provides $FILELINE."
	    	else
	    		FINDINGSNAME=${FINDINGS%:*}
	    		echo ".deb $FINDINGSNAME provides $FILELINE."
	    	fi
	    elif [[ "$EXISTS" != *"i386"* ]]; then
	    	echo "$FILELINE is installed in 64 bits, but not installed in 32 bits!"
	    elif [[ "$EXISTS" != *"x86_64"* ]]; then
	        echo "$FILELINE is in 32 bits, but not installed in 64 bits!"
	    else
	        echo "$FILELINE is installed!"
	    fi
	fi
done < $LIBFILE

Meaningful output:

Checking for libavcodec.so.57.
libavcodec.so.57 is not installed on the system!
Seems no .deb provides libavcodec.so.57.
Checking for libavutil.so.55.
libavutil.so.55 is not installed on the system!
Seems no .deb provides libavutil.so.55.
Checking for libm.so.6.
libm.so.6 is already installed!
Checking for libc.so.6.
libc.so.6 is already installed!
Checking for libz.so.1.
libz.so.1 is already installed!
Checking for libvkd3d.so.1.
libvkd3d.so.1 is already installed!
...
Checking for libpulsecommon-12.2.so.
libpulsecommon-12.2.so is not installed on the system!
.deb libpulse0 provides libpulsecommon-12.2.so.
Checking for libdbus-1.so.3.
libdbus-1.so.3 is already installed!
...

I am aware apt-file does not exist everywhere. Equivalent -> https://www.ostechnix.com/find-package-provides-specific-file-linux/.

[plata]

But I do not think it should be added inside every .tar.gz

This really depends on the size. If it's just a few MB, it's probably not worth the trouble to separate it.

[ImperatorS79]

But is it so much difficult to do ? I do not think so.

[plata]

I'm not sure. For example: when would they be deleted/installed?

[ImperatorS79]

Three case:

• Flatpak user does not seem to have missing dependencies, as far as I understand.
• (potentialy) distribution provided .deb user should have already a lot of dependencies installed (control file) (for example I am only missing two libs).
• .deb provided by phoenicis atm, which does not force you to install any deps (even i386).

So running the "script" only once should be enough. You could ask the user: those libs can be installed through packaging system, those libs cannot. And maybe add a button somewhere if the user wants to run again the "script" (after updating for example).

[plata]

The idea was to have a general set of libs for everybody (also flatpak and OSX).

[ImperatorS79]

OSX users does not ave this problem since the wine package already embed the libs if I understand correctly (maybe it could also be a good idea to put them in a dedicated folder too on OSX to avoid duplication). For flatplak I do not know, you said before that there was no problem.

For me, the idea is to give the user the required libs when there is a version mismatch with system one, without downloading libs that are already installed (the whole set of wine deps is 700 Mo minimum on my system, it is completely nightmare to put them inside each wine package, and to download all of them when most of them are already installed on the system).

[Gcenx]

OSX users does not ave this problem since the wine package already embed the libs if I understand correctly (maybe it could also be a good idea to put them in a dedicated folder too on OSX to avoid duplication). For flatplak I do not know, you said before that there was no problem.

For macOS the libs are indeed embedded for 32bit builds it only includes 32bit versions to save space, for 64bit it keep the full universal file and now symlinks into /lib64 to ensure wine64 works.

The problem for macOS trying to not include the libs within the wine engine means needing to work around SIP, SIP strips DYLD_FALLBACK_LIBRARY_PATH meaning then needing to workaround that in fun ways. If the intention is really to shrink the size then yes removing them could help and the better workaround for SIP would be making symlinks for all libs into /lib and /lib64.

But doing that can also cause dependency hell like I'm currently facing

[plata]

@Gcenx the idea really was to provide the required libs and avoid dependency hell. So basically something like we have for OSX just more general also for Linux.

@qparis can you please join the discussion as well? I think we need your input.

[qparis]

@Gcenx SIP does not strip DYLD_FALLBACK_LIBRARY_PATH. In fact, only system process strips them (including bash), so it is easy to workaround.
700Mo is indeed too large. Are the lib striped?

[Gcenx]

@plata that sounds good since the major grip for PoL seems to be dependancies been an issue.

@qparis yes sorry I forgot this project does not use bash scripts
Wineskin does for custom wine executable names plus workarounds and winebottler also use bash scripts for similar reasons.
But yeah 700mb seems a lot....

[plata]

I would really be interested where these 700mb are coming from.

[qparis]

@Gcenx, POLv4 use bash script (a lot in fact).
SIP does not prevent you from setting DYLD_FALLBACK_LIBRARY_PATH when you run wine, it prevents you from setting SET_DYLD_FALLBACK_LIBRARY_PATH when you run bash.
It means that a bash script can export DYLD_FALLBACK_LIBRARY_PATH, but the variable will be strip if bash runs another bash script

As a workaround, you can chose an arbitrary variable name, like SET_DYLD_FALLBACK_LIBRARY_PATH, and run DYLD_FALLBACK_LIBRARY_PATH="$SET_DYLD_FALLBACK_LIBRARY_PATH" before running wine.

[Gcenx]

@qparis I don’t use POL/POM so I wasn’t sure.

But yes we do something similar also;
DYLD_FALLBACK_LIBRARY_PATH=“${WINESKIN_LIB_PATH_FOR_FALLBACK}”
that’s done for;

• wineserver
• wine/wine-preloader
• wine64/wine64-preloader
• Command Line Wine Test (used to launch wine almost like winehq packages)
  All of the above are bash scripts that launch the original renames binaries.

Bash scripts are setup as required, also setting freetype to use "35" rendering so we only need to provide the latest freetype version but render fonts correctly on older wine versions.

So in total Wineskin now has at max 4 bash scripts

• wineserver
• wine/wine-preloader
• wine64/wine64-preloader
• Command Line Wine Test (used to launch wine almost like winehq packages)

Edited for clarity

The above is off topic but may be useful.

All I can think for that 700mb is everything is being packaged even the system libs not just what wine needs to run.

[plata]

I would suggest to add the libraries we know are required and then see where we end up with it. If it becomes to big, we have to think again.

[Kreyren]

• openal (LoL)

Why do we need openal for lol? -> If the issue is only with openal we can exclude it since it's not needed for leagueoflegends afaik. (uses pulseaudio for 3D sound afaik)

[ImperatorS79]

@Kreyren This is a general issue, if openal is a dependency of wine, and cannot be installed from the system with the right version, then it has to be downloaded in the "runtime".

Concerning League of Legends, it is a FACT: when you launch the installer and look at the log, you clearly see that the hang is due to missing libavcodec (at the version wine is asking).

[Kreyren]

@ImperatorS79 League Of Legends works without openal afaik my GPU AMD7870 doesn't support OpenAL anyway.. It may require it because of something else missing never notice it on my system.

[ImperatorS79]

OpenAL have nothing to do with the GPU. It is OpenAudioLibrary.

I do not know for the game itself, but the installer needs libavcodec.

[Kreyren]

@ImperatorS79 Noted, still doubt that it's needed afaik i'm running leagueoflegends atm without it.. Not even lutris mensiones it on https://lutris.net/games/install/3552/view

About msvcp140 works without this on my system afaik. Instructions to confirm? mby disable it's dll?

EDIT: Lied about msvcp140

[plata]

@qparis you mentioned that you wanted to introduce a runtime. Can you elaborate how this should look like?

[qparis]

I think it will just ba a python file that extract libraries from Docker image. Then a unique name will be given and we host it.
Does it sound ok?

[ImperatorS79]

How will phoenicis download that runtime? Will phoenicis be able to only download the libs that are not installed ?

[qparis]

In a first time, we could download the whole archive, right?

[ImperatorS79]

Of course, but it just a friendly reminder that it should not be the perfect solution ;-)

[qparis]

Do you mean that we do not need a perfect solution for the moment?

[ImperatorS79]

I mean if you want to create a first implementation that just download to whole runtime to show that it solves the problem, you can of course do it.

But do not close this issue (or create a new one) for the part where the whole runtime is not downloaded.

[qparis]

I suggest we try to remove freetype from the runtime on linux, and see if it is ok for everyone

[Kreyren]

We could also make some method that allows scripts to download required dependencies on demand from predefined source like it was discussed before to permanently resolve issues alike since runtime libs are always required.

Or depending on package manager present on the system we can parse the instructions to it so that those deps would be fetched using distro repo.

[ImperatorS79]

@Keyren this can be second phase. First let's implement the runtime since it is working now.

[Kreyren]

i personally hate runtimes since they are usually bloated af making system slower and pure hell to maintain.. (based on results on my gentoo where i have libs compiled specifically on the hardware)

[qparis]

@Kreyren Please stop contesting each decision..

[Kreyren]

@qparis You need to realize that this is not exclusive MacOS community.. We in linux blame each other about provided code, decisions and info for Quality assurance which you should welcome..

Probably if you would respect GNU ethics from time to time we would have working phoenicis now 🤔

[qparis]

@qparis You need to realize that this is not exclusive MacOS community.. We in linux blame each other about provided code, decisions and info for Quality assurance which you should welcome..

Please put into practice what you are saying, especially for Gentoo. While you are always pushing for specific distribution solutions, we are trying to imagine agnostic solutions that will not require a different implementation for each configuration and without reinventing the wheel.

Probably if you would respect GNU ethics from time to time we would have working phoenicis now 🤔

Please don't make me wrong, code and solution challenging is very welcome. What you are generally proposing are in general soilless concepts, without any working proof of concept behind.
We spend a lot time (probably hours!) to explain to you why the solution is not working, and most of the time, we loop and rexplain the same things. ==> You cannot argue the fact that we are not open to discussion.
As you said, just before we are lacking time, so I suggest we are efficient on the decision making process.

I have no vanity for this project, so if you manage to build a new community from scratch with better GNU ethics, please do so and I will sincerely salute it.

Because we do not (collectively) agree with one of you solution does not mean that we are not open to the discussion. Thoughts for food.

Can we now focus back on the discussion? @ImperatorS79 what do you think about making freetype a disitribution dependency and managing the rest within the runtime?

[ImperatorS79]

@qparis As I said, this is a good first solution. You also know that I would prefer having the less things possible in the runtime.

Do you think it would be possible to have a file listing what libs needs to be downloaded from the runtime per distribution? So that we can say to the user : "please installs the following package to ensure wine will work fine". If the distribution is not found in it, you just download the full runtime.

[qparis]

The most tricky part will be to detect what librairie you need. How you would handle it?

[ImperatorS79]

See the beginning of this issue. I showed to .sh file. One simply run ldd on each wine .so file to determine the libs, the other one use apt-file (this changes depending on the distribution) to determine if the lib exists in the good version in a .deb.

[qparis]

apt-file is not agnostic. You'll probably end up with the same problems. The version is not neceserally the only compatibility criteria.
Note that the runtime will generally be downloaded once

[ImperatorS79]

What do you mean by "not agnostic" ?

[qparis]

You need to define a different implementation for each package manager

[ImperatorS79]

Yes, that is what I said in "this changes depending on the distribution". Anyway, I was thinking of a file stored server side. So that, for, a distribution that has been tested and where you are sure that system libs works, prefer system libs from runtime ones.

[qparis]

I see. So what you are suggesting is some kind of "lazy downloading" system. I propose we try to handle it in a second step

[ImperatorS79]

Yes, sure.

[qparis]

@Kreyren Please keep this thread dedicated for the discussion about the runtime issue

[ImperatorS79]

@qparis Now you should be able to upload amd64 and x86 runtime somewhere, then someone could implement it's usage in phoenicis.

[qparis]

I'm working on it right now

[qparis]

https://phoenicis.playonlinux.com/index.php/runtime?os=linux

x64 is coming

[plata]

When should we download the runtime? Best would probably be with every engine installation, right? Shall we add a property to (not) use the runtime?

[qparis]

The runtime should be indeed downloaded with every engine installation but only if it has not been installed before

[plata]

Is it enough to check if it has not been installed before? What I'm thinking about it: The runtime could change. So either we have to compare a hash or redownload it unconditionally.

[qparis]

In a first version it will be sufficient. Later, we may assign a runtime per version

[ImperatorS79]

Don't do a runtime per version, runtime is about 500 MB! And since all wine versions are built on the same system, it can be common to all wine versions.

Checking a hash does seem to be a good idea to see if the runtime has changed.

[qparis]

A runtime per version of course do not mean a different runtime per version

[ImperatorS79]

I guess this is solved.