wget doesnt work in MSYS2 anymore

[lovetox]

i get

C:/msys64/usr/bin/wget.exe: error while loading shared libraries: ?: cannot open shared object file: No such file or directory

to reproduce just open a msys shell and type wget

i tried to reinstall wget but it didnt help

i didnt change anything in my build script, it just stopped working 1 or 2 days ago.

[Ede123]

Likely you're updating wget somewhere in your code (e.g. using pacman -Syuu base-devel). However this will not update its dependencies (libicu in this case). As wget was recently re-compiled and is now linked against icu 59 it can't use the pre-installed version anymore and will fail.

I faced the same issue in my builds and determined the missing library using ntldd -R wget.

The safest fix is to make sure to do a full system upgrade using pacman -Syuu (i.e. run this at least twice in your code - once to update the core system, once to do the actual system upgrade).

Due to the significantly outdated version of MSYS2 on AppVeyor (see e.g. #1799) this will however take ~5 minutes (i.e. an update of the pre-installed MSYS2 might be helpful).

[lovetox]

thanks @Ede123

i switched to curl for the moment

[FeodorFitsner]

@Ede123 Is there an "official" way (set of commands) to update MSYS2 or it's better to re-install it completely to get the latest?

[Ede123]

Re-installing MSYS2 will not give you the latest version but whatever is bundled in the installer (which is not updated often).

The correct way to update MSYS2 is to run pacman -Syuu* from an MSYS2 shell.

Only gotcha is that sometimes a "core system upgrade" is necessary (e.g. parts of the cygwin emulation layer or pacman itself), so the update can not be performed in a single step. It's then necessary to re-sart the shell (pacman will inform about this) and afterwards execute the same command again to complete the update. In principle you can just execute the above command until pacman prints "there is nothing to do" for all upgrade steps if unsure.

**-S is the general command to synchronizes packages on the local machine with versions available from a remote repository, y tells pacman to update the local package database from the server, uu starts an upgrade of installed packages (actually u would suffice but in case packages are downgraded on the server uu allows for downgrades, something that can also happen for a few packages that have a git commit hash as version number)
For further details on Pacman the available documentation from Arch Linux can be used (MSYS2's pacman works exactly the same)*.

[MSP-Greg]

@FeodorFitsner I did an upgrade script with:

  # To avoid duplicated executables in PATH, see https://github.com/ruby/spec/pull/468
  - set PATH=C:\msys64\usr\bin;C:\ruby%RUBY_VERSION%\bin;C:\Program Files\7-Zip;C:\Program Files\AppVeyor\BuildAgent;C:\Program Files\Git\cmd;C:\Windows\system32;C:\Program Files;C:\Windows
  - pacman -Syuu --noconfirm
  - echo %ERRORLEVEL%
  - pacman -Suu --noconfirm

Console output here. I added the echo just to see if there was any return info. It was 0...

[FeodorFitsner]

Thank you!

[lazka]

wget should work again msys2/MSYS2-packages#1020

[vszakats]

@FeodorFitsner: I second the desire to refresh MSYS2 packages that come with the default image. Doing this 2-4 times a year may be even better.

At the moment it takes about 10 minutes to get everything in line before each build. I'm thinking of the mingw-w64 toolchain as the most crucial here.

It'd also be super nice to include the LLVM/Clang compiler (now at version 5.0.0) — the package is called mingw-w64-{i686,x86_64}-clang (and it depends on the mingw-w64 toolchain). It's using less resources (memory/CPU-time) than gcc when compiling.

[Ede123]

@vszakats - Given that packages in the AppVeyor images are likely to be outdated from time to time I'd vote against including even more packages into the images (especially a very big one like LLVM/Clang!).

As this issue highlights partial updates are problematic so a full system upgrade is often the safest solution. However the more packages are included, the more time a full system upgrade will need...

I'd go as far as excluding all toolchains and only install the base-devel group, so users can decide for themselves which packages to install (with pacman that should be easy enough). Every pre-selection of packages will include a lot of packages that would not be required for a CI job while at the same time missing packages that are needed.

[vszakats]

Having great tools installed by default is a good thing IMO. The practice of doing an update on everything (as part of the build process) is a necessity because the default image is rarely updated at the moment — IOW, it wouldn't be necessary for most users if the packages would get regular updates like MSVC and other tools do.

As for LLVM/Clang, it's a plug-in replacement for gcc and overall a better experience — faster to compile, similar quality generated code (and improving faster), better warnings. The created libraries are compatible with gcc generated ones (so much so that currently they both use the same ld from binutils), same mingw-w64 CRTL and headers.

In fact LLVM/Clang is already available on AppVeyor images, but that is only usable inside the MSVC environment (aka clang-cl). It'd be quite nice to offer this compiler for the open-source toolchain as well. This is trivially easy to do via MSYS2.

[Ede123]

I agree that it would be nice to have everything pre-installed but as mentioned above it's impossible to achieve and as also mentioned above partial updates (which pacman -S package is) are likely to break things which will result in very hard to debug CI fails.

While I agree with all of your points in principle I don't think there's a viable option to generate images that have an always up-to-date (or even sufficiently up-to-date) MSYS2 distribution...

[vszakats]

I understand what you mean and of course issues may happen, so it's definitely important to get the timing right to offer a package tree that is in good shape — or make the updates as frequent as possible, because so far such these issues were fixed within a few days.

All in all though I think the usefulness of LLVM/Clang weights more strongly than the extra update strain it causes and this will be more and more so with time as clang continues catching up.

The package size for llvm+clang is ~100MB (for x86 an x64 each) — not something I'd consider large, and certainly not something as large as making it the "last straw". (Much less rarely used tools like gcc-ada, gcc-fortran and gcc-objc are installed/updated since the beginning by default — and cause little/no harm overall.)

[Ede123]

I understand what you mean and of course issues may happen, so it's definitely important to get the timing right to offer a package tree that is in good shape

I think you're missing my point. The problem is not to have "a tree that is in good shape", the problem is that in general it is not safely possible to install any package without doing a full system upgrade.

If a package is pre-installed and depends on a library, this library will be pre-installed, too. If you now try to install a package that depends on the same library it will not be installed or upgraded unless a specific version is requested by the package (which most MSYS2 packages do not do - that's the idea of "rolling release" even if its impractical at times). If the library was updated in the meantime the newly installed package will not work properly.
This is exactly the issue we had here with wget (the solution was to update the version number for the dependency, but we can not rely on that and it is certainly nothing that will be done for all packages).

Short of syncing AppVeyor image updates with MSYS2 repo updates (which will likely not happen) this will consequently require users to figure out how to update all dependencies whenever installing a new package. While there are some possibilities they are fragile at best, so at the end of the day a full system upgrade is always the safest bet and therefore reducing the pre-installed MSYS2 distribution to the bare minimum is certainly not an ideal option, but it's the best option we have to reduce build times without risking breakage.

The package size for llvm+clang is ~100MB (for x86 an x64 each) — not something I'd consider large, and certainly not something as large as making it the "last straw". (Much less rarely used tools like gcc-ada, gcc-fortran and gcc-objc are installed/updated since the beginning by default — and cause little/no harm overall.)

It's not so much the package size as the install time which is significant for some components of llvm/clang.

With respect to less used gcc compilers: You were initially complaining about updating taking ~10 minutes. A good part of that is in fact those often unused compilers. That's why I suggested to even think about dropping the mingw64-{i686,x86_64}-toolchains.

[MSP-Greg]

FYI, current list of AV installed packages here

[vszakats]

I wasn't complaining about 10 minutes, I'm happy that the AppVeyor service exists at all — and made two suggestions. I'm not into entering a war here and sorry for posting these in the wrong room.

[MSP-Greg]

@lovetox @vszakats

From what I'm seeing, MSYS2 / MinGW was updated today.

[FeodorFitsner]

Yep, published update images but not closed the issues yet.

[lovetox]

thanks !