Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve delivery of MSYS2 #1572

Closed
alepauly opened this issue Sep 10, 2020 · 10 comments
Closed

Improve delivery of MSYS2 #1572

alepauly opened this issue Sep 10, 2020 · 10 comments
Labels
enhancement New feature or request investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Windows

Comments

@alepauly
Copy link
Contributor

alepauly commented Sep 10, 2020

As discussed in #1525, there is probably room to improve on how and what we install with MSYS2.

I believe what we want is for the install of MSYS2 that's available to be reliable and flexible enough to allow others to add the packages they need without unexpected results. Ideally, it would behave as a cached install so that setup-msys2 type Actions can interoperate and take advantage of whatever is already laid out on the image to avoid having to do unecessary and repetitive work across workflow runs.

@MSP-Greg
Copy link
Contributor

MSP-Greg commented Sep 10, 2020

As mentioned elsewhere, most (but not all) of my experience with MSYS2 is related to compiling Ruby extension gems. As such, the main concern is the availability of MSYS2 gcc and associated tools. Outside of Ruby, more tools are needed. You may recall that the PR's I submitted to install MSYS2 started with only gcc, but as others asked for more tools, they were added to the install script.

Like a lot of software, people are happy to use whatever is installed, especially given the frequency of image updates. Most of the repos I'm involved with I update the gcc tools.

What seems to be the central issue is whether an MSYS2 installation can be partially updated, or whether it always requires a full update.

With a full update, every package is updated, and can take quite a while given the number of packages currently installed.

Conversely, partial updates can fail, although it's not common. But, given the frequency of runner updates, partial updates can happen in a few seconds, as often the needed packages are up to date, and the only thing updated is the package database.

Re the issues with Git bash tools vs MSYS2 bash tools, they're the same, so I don't think anyone will be bothered if the bash shell changes.

But since MSYS2 has the tools installed, there will be conflicts if the C:/msys64/mingw64/bin folder is included in Path.

See:
https://github.com/MSP-Greg/actions-image-testing/runs/1082867165?check_suite_focus=true#step:13:5

Note that make & gcc exist in at least two of MSYS2, C:\ProgramData\Chocolatey\bin, and C:\Strawberry\c\bin folders.

An option might be to only add C:/msys64/mingw64/bin to Path if the MSYS2 bash shell is used.

@eine
Copy link

eine commented Sep 10, 2020

Hi @alepauly! First off, there is a very relevant issue to bear in mind: MSYS2 is a rolling environment for development. Until very few months ago, it was maintained by a single person, who built and tested every single package on his own computer. Fortunately, other maintainers stepped in, and a partially automated workflow has been set up, which uses Actions and Releases: https://github.com/msys2/msys2-autobuild.

As a result, there are several constraints that you need to be aware of:

  • MSYS2 is not considered to be production ready. Hundreds/thousands of users do use MSYS2 every day, the ecosystem is very healthy, and as a user I can tell that it is as stable as any other product/tool I use. Yet, it is not officially for production.
  • Because it is a rolling release (tightly related to Arch Linux release infrastructure), it is a priori not possible to keep an specific "old" version of a package/tool. Users need to always use the latest of all the environment.
  • For long time, the installation tarball was released once a year or so. Since May, that is more frequent (when internal/foundation packages need so).

As a result, you might want to be cautious about making the default bash in windows-latest point to MSYS2.


From a technical point of view, the problem with the current implementation in this repo is that it is shaped after an unsupported workflow:

  • When the base image/environment is generated/updated:
    • Install the latest MSYS2 release.
    • Update all the packages.
    • Install some GB of additional packges (gcc, clang, llvm...).
  • When a workflow is executed:
    • Install required additional packages only.

The reason it is unsupported is the last step: updating some packages only is expected to fail frequently. The supported use case is to always update the system first, and then install additional packages.

Hence, since this environment is updated once every 1-2 weeks, any change in gcc, clang, llvm... forces all the users which are following the recommended procedure to uninstall and reinstall GBs of tools. That introduces a severe time penalty for users that don't need those tools, and which need to wait several minutes before the additional tools that they actually need can be installed.

Conversely, using setup-msys2, the workflow is the following:

  • When a workflow is executed:
    • Check if a cache hit exists.
    • Otherwise:
      • Install the latest MSYS2 release or use the built-in installation.
      • Update all the packages.
    • Install the additional packages requested by the user.
    • Save cache.

As a result, the most obvious enhancement would be NOT to install any additional packages in this environment. Just extract the latest stable release, and optionally update the environment once. Then, setup-msys2 would take care of caching (ad-hoc for each user).

For a better integration, this environment should be updated as soon as MSYS2 releases a new tarball. Indeed, that can be automated through cross-repo triggers.


Apart from that, MSYS2 currently caches package tarballs only. Not the locations where the tools are actually installed. We tried to save both, the tarballs and the installation. However, it seems that some files are lost in the cache: msys2/setup-msys2#61.

/cc @lazka

@eine
Copy link

eine commented Sep 10, 2020

What seems to be the central issue is whether an MSYS2 installation can be partially updated, or whether it always requires a full update.

@MSP-Greg, being unsupported means that it cannot be done, because more or less frequently it won't work. A user might need to have the system updated for using a package which you are not even aware of, and which is unrelated to any of the tools that make other people happy. That user will be paying money for others happiness. Moreover, they might be affected by broken updates of packages which they don't care about.

Overall, supporting partial updates should be discussed with MSYS2 maintainers, before making it the default for thousands of users.

@MSP-Greg
Copy link
Contributor

MSP-Greg commented Sep 11, 2020

@eine

because more or less frequently it won't work.

Not sure about the time frame, but I've been building with MSYS2 for five years +. When exactly I started helping with MSYS2 & Ruby Windows in CI, I'm not sure.

Regardless, re partial updates, problems have happened very rarely. A few times on AppVeyor (with infrequent image updates), and I don't recall any with Actions. Maybe possibly one involving gpg keys. Not sure.

As mentioned in #1525, if people have problems with the installed MSYS2 tools, they can always use your action.

before making it the default for thousands of users.

I haven't seen many complaints. Maybe we should all wait and see if that continues?

@eine
Copy link

eine commented Sep 11, 2020

Not sure about the time frame
I'm not sure
very rarely
A few times
infrequent
don't recall
Maybe possibly one
Not sure

@MSP-Greg, doesn't sound as something to base decisions on, does it?

Maybe we should all wait and see if that continues?

When things fail (because they will in an undetermined period), will you take care of fixing the problems that will affect thousands of users? Unless you can and want to do so, you should not introduce the problems in the first place. If you are willing to do so, please see the 5-6X time penalty that is affecting any user with a supported use case.

if people have problems with the installed MSYS2 tools, they can always use your action

Everyone can and should use the official action. "My" action does not exist. setup-msys2 existed before I contributed, and it will keep existing after I stop contributing. That's an official solution for all users of MSYS2, including you. It allows installing ad-hoc packages, and having them cached for reducing startup time. I honestly cannot understand why you did not try it and give any feedback yet.

If this is about authorship and/or ego, all yours. I don't want or need to spend a single second on that.


@alepauly I did a quick test for illustrating the problem: https://github.com/eine/virtual-environments/blob/test-msys2/.github/workflows/msys2.yml. As you can see in https://github.com/eine/virtual-environments/actions/runs/249443402:

  • Using the built-in install needs 8min 9s.
  • Ignoring the built-in install needs 1min 29s.

That's 5.5X slower due to all the unnecessarily installed dependencies. It is ironic that Greg considers 90s "too much".

In the second execution, GCC was added explicitly. At the same time, caches are used, because previous runs were successful. Still, the difference is very relevant, as shown in https://github.com/eine/virtual-environments/actions/runs/249464693:

  • Using the built-in install needs 10min 47s.
  • Ignoring the built-in install needs 1min 45s.

So, 6.2X slower. As a result, unless Greg can provide an alternative solution, it is very difficult to recommend anyone to use the built-in install.

@al-cheb al-cheb added enhancement New feature or request investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Windows labels Sep 14, 2020
@maxim-lobanov
Copy link
Contributor

maxim-lobanov commented Jul 29, 2021

Hello @eine !
I would like to resurrect this discussion since we are planning to add new Windows image to GitHub Actions in near future. It will be Windows Server 2022 and since it is the new platform, it is allowed to do breaking changes in new image.
We think it is good time to revisit any bad decisions done in past and rework installation of MSYS to be more convenient for customers and setup-msys task.

As I can see, currently, installation script perform the following operations:

  1. Download latest release from https://github.com/msys2/msys2-installer in tar.xz format
  2. Extract it to C:\msys64
  3. Run pacman-key --init (Link)
  4. Run pacman-key --populate msys2 (Link)
  5. Run pacman.exe -Syyuu --noconfirm and pacman.exe -Syuu --noconfirm (Link)
  6. Install msys2 packages
  7. Install mingw packages
  8. clean all packages to decrease image size

As I understand, your proposal was getting rid of installation any additional packages (point 6 and 7). Am I right?
What about 3, 4, 5? Does it make sense to keep them?

TheNorthMemory added a commit to TheNorthMemory/easyalipay that referenced this issue Aug 19, 2021
ref actions/runner-images#1572 discussion, we need issue a `default installation` proposal onto the new WinSvr2022 env
@TheNorthMemory
Copy link

TheNorthMemory commented Aug 19, 2021

Hi there,
Should we think about the bc package installed onto the Windows VM platform as default?
Sometimes, we need caculate the BigInteger. Thebc command maybe the greate choice as default.

@maxim-lobanov
Copy link
Contributor

maxim-lobanov commented Aug 31, 2021

Windows Server 2022 is released and it doesn't contain MSYS2 packages installed by default. Only pure MSYS2.
The recommended approach is using https://github.com/msys2/setup-msys2 to install dependencies in runtime and cache them.
I am closing the issue for now but please let me know if you have any concerns.

siko1056 pushed a commit to gnu-octave/octave that referenced this issue Oct 6, 2021
* .github/workflows/make.yaml (windows): GitHub recommends using the
  msys2/setup-msys2 action for installing MSYS2 on their runners. See e.g.
  actions/runner-images#1572 (comment)
  Replace the manual installation steps by that action.
mtmiller pushed a commit to mtmiller/octave that referenced this issue Oct 6, 2021
* .github/workflows/make.yaml (windows): GitHub recommends using the
  msys2/setup-msys2 action for installing MSYS2 on their runners. See e.g.
  actions/runner-images#1572 (comment)
  Replace the manual installation steps by that action.
@eine
Copy link

eine commented Oct 20, 2021

  1. Download latest release from https://github.com/msys2/msys2-installer in tar.xz format

  2. Extract it to C:\msys64

  3. Run pacman-key --init (Link)

  4. Run pacman-key --populate msys2 (Link)

  5. Run pacman.exe -Syyuu --noconfirm and pacman.exe -Syuu --noconfirm (Link)

  6. Install msys2 packages

  7. Install mingw packages

  8. clean all packages to decrease image size

As I understand, your proposal was getting rid of installation any additional packages (point 6 and 7). Am I right? What about 3, 4, 5? Does it make sense to keep them?

@maxim-lobanov, I'm sorry I skipped this message from you. Overall, I think that the current solution on Windows Server 2022 is adequate.

As you said, step 6 and 7 are not necessary. With regard to 3, 4, 5 and 8, this is the procedure used by setup-msys2:

pacman --noconfirm -Syuu --overwrite '*'
taskkill.exe /F /FI "MODULES eq msys-2.0.dll"
pacman --noconfirm -Syuu --overwrite '*'

Hence, the one used in https://github.com/actions/virtual-environments/blob/main/images/win/scripts/Installers/Install-Msys2.ps1#L78-L83 seems valid.

@actions actions deleted a comment Oct 20, 2021
@miketimofeev
Copy link
Contributor

miketimofeev commented Oct 20, 2021

@eine thanks for the confirmation!

fdintino added a commit to fdintino/pillow-avif-plugin that referenced this issue Oct 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Windows
Projects
None yet
Development

No branches or pull requests

7 participants