Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

git-extra: publish both with and without MINGW prefix #455

Merged
merged 2 commits into from
Dec 21, 2022

Conversation

dennisameling
Copy link
Contributor

@dennisameling dennisameling commented Dec 20, 2022

Following up on this discussion.

Until now, the git-extra package was published under that exact name. However, it's actually a MINGW package which installs things both in e.g. /mingw64/bin and /etc/profile.d. This is problematic for GfW's upcoming arm64 support, which uses the x64 MSYS shell, so it'll need its own package. At the same time, packages for multiple architectures can't be installed side-by-side like other MINGW-packages, as this one writes to shared folders like /etc/profile.d as well. Therefore we mark them as conflicting each other.

With this change, we'll publish the following packages side by side, so the repos will look like this:

  • git-for-windows/git-extra (old)
  • git-for-windows-mingw32/git-extra (old)
  • git-for-windows/mingw-w64-x86_64-git-extra (new)
  • git-for-windows-mingw32/mingw-w64-i686-git-extra (new)
  • git-for-windows-clangarm64/mingw-w64-aarch64-git-extra (new)

Publishing side by side allows us to gradually update other parts of GfW to start using the new package names moving forward.

Test results

After building with MINGW_ARCH=mingw64 makepkg-mingw, two files were created:

  • git-extra-1.1.614.d551cf0cc-1-any.pkg.tar.zst (legacy package: git-extra)
  • mingw-w64-x86_64-git-extra-1.1.614.d551cf0cc-1-any.pkg.tar.zst (new package: mingw-w64-git-extra)

Assuming that git-extra is already installed (e.g. when working from git-sdk-64), let's test these packages.

  1. Updating the existing git-extra package still works as normal after the changes in this PR:
$ pacman -U git-extra-1.1.614.d551cf0cc-1-any.pkg.tar.zst
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) git-extra-1.1.614.d551cf0cc-1

Total Installed Size:  0.47 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring                                                      [###############################################] 100%
(1/1) checking package integrity                                                    [###############################################] 100%
(1/1) loading package files                                                         [###############################################] 100%
(1/1) checking for file conflicts                                                   [###############################################] 100%
(1/1) checking available disk space                                                 [###############################################] 100%
:: Processing package changes...
(1/1) upgrading git-extra                                                           [###############################################] 100%
  1. Trying to install mingw-w64-git-extra correctly indicates that it conflicts with git-extra and Pacman suggests to remove the old package first:
$ pacman -U mingw-w64-x86_64-git-extra-1.1.614.d551cf0cc-1-any.pkg.tar.zst
loading packages...
resolving dependencies...
looking for conflicting packages...
:: mingw-w64-x86_64-git-extra and git-extra are in conflict. Remove git-extra? [Y/n] Y

Packages (2) git-extra-1.1.614.d551cf0cc-1 [removal]  mingw-w64-x86_64-git-extra-1.1.614.d551cf0cc-1

Total Installed Size:  0.47 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring                                                      [###############################################] 100%
(1/1) checking package integrity                                                    [###############################################] 100%
(1/1) loading package files                                                         [###############################################] 100%
(1/1) checking for file conflicts                                                   [###############################################] 100%
(2/2) checking available disk space                                                 [###############################################] 100%
:: Processing package changes...
(1/1) removing git-extra                                                            [###############################################] 100%
(1/1) installing mingw-w64-x86_64-git-extra                                         [###############################################] 100%
Optional dependencies for mingw-w64-x86_64-git-extra
    vim [installed]
    filesystem [installed]
  1. Let's check whether the old package name, git-extra, is indeed provided by the new package:
$ pacman -Q git-extra
mingw-w64-x86_64-git-extra 1.1.614.d551cf0cc-1
  1. Can we actually go back to git-extra if we want? Yes, we can:
$ pacman -U git-extra-1.1.614.d551cf0cc-1-any.pkg.tar.zst
loading packages...
resolving dependencies...
looking for conflicting packages...
:: git-extra and mingw-w64-x86_64-git-extra are in conflict. Remove mingw-w64-x86_64-git-extra? [Y/n] Y

Packages (2) mingw-w64-x86_64-git-extra-1.1.614.d551cf0cc-1 [removal]  git-extra-1.1.614.d551cf0cc-1

Total Installed Size:  0.47 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring                                                      [###############################################] 100%
(1/1) checking package integrity                                                    [###############################################] 100%
(1/1) loading package files                                                         [###############################################] 100%
(1/1) checking for file conflicts                                                   [###############################################] 100%
(2/2) checking available disk space                                                 [###############################################] 100%
:: Processing package changes...
(1/1) removing mingw-w64-x86_64-git-extra                                           [###############################################] 100%
(1/1) installing git-extra                                                          [###############################################] 100%
Optional dependencies for git-extra
    vim [installed]
    filesystem [installed]
  1. Installing for another architecture also correctly triggers the conflict logic:
$ pacman -U mingw-w64-clang-aarch64-git-extra-1.1.616.e442fe2d7-1-any.pkg.tar.zst
loading packages...
resolving dependencies...
looking for conflicting packages...
:: mingw-w64-clang-aarch64-git-extra and mingw-w64-x86_64-git-extra are in conflict (git-extra). Remove mingw-w64-x86_64-git-extra? [Y/n] Y

Packages (2) mingw-w64-x86_64-git-extra-1.1.616.e442fe2d7-1 [removal]  mingw-w64-clang-aarch64-git-extra-1.1.616.e442fe2d7-1

Total Installed Size:   0.38 MiB
Net Upgrade Size:      -0.09 MiB

:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring                                                                                       [#####################################################################] 100%
(1/1) checking package integrity                                                                                     [#####################################################################] 100%
(1/1) loading package files                                                                                          [#####################################################################] 100%
(1/1) checking for file conflicts                                                                                    [#####################################################################] 100%
(2/2) checking available disk space                                                                                  [#####################################################################] 100%
:: Processing package changes...
(1/1) removing mingw-w64-x86_64-git-extra                                                                            [#####################################################################] 100%
(1/1) installing mingw-w64-clang-aarch64-git-extra                                                                   [#####################################################################] 100%
Optional dependencies for mingw-w64-clang-aarch64-git-extra
    vim [installed]
    filesystem [installed]

Suggested strategy

  1. Start publishing the old and new package versions side by side (this PR)
  2. Start updating all parts of GfW that install git-extra instead of mingw-w64-git-extra
  3. Stop publishing (and remove?) the old git-extra package

Comment on lines +11 to +12
arch=('any')
mingw_arch=('mingw32' 'mingw64' 'clangarm64')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pkgrel=1
pkgdesc="Git for Windows extra files"
arch=('i686' 'x86_64')
arch=('any')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of publishing the packages concurrently.

However, we will need to publish the i686 and x86_64 variants of git-extra, not the any variant, because Git for Windows' tooling is unable to handle any package without mingw-w64- prefix and without the CPU architecture in its suffix.

I also wondered whether there is a better way wherein the git-extra package would be automatically uninstalled and the mingw-w64-git-extra package would be installed. On a first read, https://wiki.archlinux.org/title/PKGBUILD#replaces would be the correct solution there. But I fear that won't work for us because we will provide mingw-w64-git-extra both in the git-for-windows and in the git-for-windows-mingw32 repository. And if there are two packages that replace the same package, and those two packages conflict with each other, I wonder whether Pacman does the right thing.

But then, we could exclude that package specifically from git-for-windows-mingw32 (and from git-for-windows-mingw64) so that only one or the other is available, depending whether you run Pacman in git-sdk-32 or git-sdk-64.

Having said all that, I think you're right and we can only do that once we have taken care of all of our tooling, for which we need a transitional period where we build both git-extra and mingw-w64-git-extra.

Back to the arch problem: we cannot specify arch conditional on the pkgname, sadly, as it is a global.

So I fear that we will have to either put in a hack, or copy the PKGBUILD file whole-sale. The hack would look somewhat like this:

if test -z "$BUILD_MINGW_W64_GIT_EXTRA"
then
  pkgname=$_realname
  arch=('i686' 'x86_64')
else
  pkgname=$MINGW_PACKAGE_PREFIX-$_realname
  arch=('any')
fi

Copy link
Contributor Author

@dennisameling dennisameling Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Fixed by simply overriding the arch in package_git-extra() 🎉 I'm really impressed by the flexibility we have in those PKGBUILD files.

The resulting package names look good:

  • git-extra-1.1.616.3f26b1b0c-1-x86_64.pkg.tar.zst
  • mingw-w64-x86_64-git-extra-1.1.616.3f26b1b0c-1-any.pkg.tar.zst

Until now, the git-extra package was published under that exact name.
However, it's _actually_ a MINGW package which installs things both in
e.g. /mingw64/bin and /etc/profile.d. This is problematic for GfW's
upcoming arm64 support, which uses the x64 MSYS shell, so it'll need
its own package. At the same time, packages for multiple architectures
can't be installed side-by-side like other MINGW-packages, as this one
writes to shared folders like /etc/profile.d as well. Therefore we mark
them as conflicting each other.

With this change, we'll publish the following packages side by side, so
the repos will look like this:
- git-for-windows/git-extra (old)
- git-for-windows-mingw32/git-extra (old)
- git-for-windows/mingw-w64-x86_64-git-extra (new)
- git-for-windows-mingw32/mingw-w64-i686-git-extra (new)
- git-for-windows-clangarm64/mingw-w64-aarch64-git-extra (new)

Publishing side by side allows us to gradually update other parts of GfW
to start using the new package names moving forward.

Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Copy link
Member

@dscho dscho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! I confirmed (by downloading the PR build artifacts and inspecting them locally) that the packages are built correctly, i.e. their file name and their .PKGINFO files contain the expected architecture. That's much cleaner than the hack I feared we need!

@dscho
Copy link
Member

dscho commented Dec 21, 2022

/deploy git-extra

EDIT: I guess I should not have tried to avoid a regular expression in git-for-windows/gfw-helper-github-app@1cf6d04...

dscho added a commit to git-for-windows/gfw-helper-github-app that referenced this pull request Dec 21, 2022
…ring

Apparently it is not, as I got this response for
git-for-windows/build-extra#455 (comment):

	TypeError: Cannot read properties of undefined (reading 'startsWith')

Let's just do the same as for the `/mention` command: use regular
expressions. If we use only greedy expressions, we should not run into
the same issues as caused the scary StackOverflow outage described in
https://stackstatus.tumblr.com/post/147710624694/outage-postmortem-july-20-2016.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@dscho
Copy link
Member

dscho commented Dec 21, 2022

/deploy git-extra

Okay, I think I finally got it. Let's try this again, it was so much fun.

dscho added a commit to git-for-windows/gfw-helper-github-app that referenced this pull request Dec 21, 2022
Currently, we won't see any stack trace in the error output, e.g. while
diagnosing the problem we encountered in
git-for-windows/build-extra#455 (comment).

Let's make that easier to pinpoint.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho added a commit to git-for-windows/gfw-helper-github-app that referenced this pull request Dec 21, 2022
Still trying to figure out why
git-for-windows/build-extra#455 (comment)
failed...

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@dscho
Copy link
Member

dscho commented Dec 21, 2022

/deploy git-extra

The workflow run was started

@dscho
Copy link
Member

dscho commented Dec 21, 2022

The workflow run was started

That one worked! (After kicking of a re-run after syncing some new pacman package version into the SDK, we'll probably need to turn this bash step into a PowerShell step that replicates the logic to re-run pacman -Su upon a system upgrade).

@dscho dscho merged commit 53403f7 into git-for-windows:main Dec 21, 2022
@dennisameling dennisameling deleted the git-extra-side-by-side branch December 21, 2022 21:37
@dennisameling
Copy link
Contributor Author

There we go! Thank you so much!!

denni@DESKTOP-8HTP3NV CLANGARM64 / (main)
$ pacman -Ss git-extra
git-for-windows/git-extra 1.1.616.3f26b1b0c-1 (VCS) [installed]
    Git for Windows extra files
git-for-windows/mingw-w64-x86_64-git-extra 1.1.616.3f26b1b0c-1 (VCS)
    Git for Windows extra files
git-for-windows-mingw32/git-extra 1.1.616.3f26b1b0c-1 (VCS) [installed]
    Git for Windows extra files
git-for-windows-mingw32/mingw-w64-i686-git-extra 1.1.616.3f26b1b0c-1 (VCS)
    Git for Windows extra files
msys/git-extras 6.5.0-1
    GIT utilities -- repo summary, commit counting, repl, changelog population and more

denni@DESKTOP-8HTP3NV CLANGARM64 / (main)
$ pacman -S mingw-w64-x86_64-git-extra
resolving dependencies...
looking for conflicting packages...
:: mingw-w64-x86_64-git-extra and git-extra are in conflict. Remove git-extra? [Y/n] Y

Packages (2) git-extra-1.1.616.3f26b1b0c-1 [removal]  mingw-w64-x86_64-git-extra-1.1.616.3f26b1b0c-1

Total Download Size:   0.12 MiB
Total Installed Size:  0.47 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] Y
:: Retrieving packages...
 mingw-w64-x86_64-git-extra-1.1.616.3f26b1b0c-1-any                                           123.9 KiB   113 KiB/s 00:01 [#########################################################################] 100%
(1/1) checking keys in keyring                                                                                            [#########################################################################] 100%
(1/1) checking package integrity                                                                                          [#########################################################################] 100%
(1/1) loading package files                                                                                               [#########################################################################] 100%
(1/1) checking for file conflicts                                                                                         [#########################################################################] 100%
(2/2) checking available disk space                                                                                       [#########################################################################] 100%
:: Processing package changes...
(1/1) removing git-extra                                                                                                  [#########################################################################] 100%
(1/1) installing mingw-w64-x86_64-git-extra                                                                               [#########################################################################] 100%
Optional dependencies for mingw-w64-x86_64-git-extra
    vim [installed]
    filesystem [installed]

denni@DESKTOP-8HTP3NV CLANGARM64 / (main)
$ pacman -Ss git-extra
git-for-windows/git-extra 1.1.616.3f26b1b0c-1 (VCS)
    Git for Windows extra files
git-for-windows/mingw-w64-x86_64-git-extra 1.1.616.3f26b1b0c-1 (VCS) [installed]
    Git for Windows extra files
git-for-windows-mingw32/git-extra 1.1.616.3f26b1b0c-1 (VCS)
    Git for Windows extra files
git-for-windows-mingw32/mingw-w64-i686-git-extra 1.1.616.3f26b1b0c-1 (VCS)
    Git for Windows extra files
msys/git-extras 6.5.0-1
    GIT utilities -- repo summary, commit counting, repl, changelog population and more

@dscho
Copy link
Member

dscho commented Dec 21, 2022

@dennisameling should we use the replaces construct and teach quick_add to avoid adding the mingw-w64-git-extra to the Pacman databases that only include the "other" MINGW packages (probably here)? I.e. in a git-sdk-32, we do not want the git-for-windows-mingw64 database to offer the x86_64 variant of mingw-w64-git-extra, nor do we want in a git-sdk-64 to let the git-for-windows-mingw32 database offer the i686 variant.

On a totally unrelated topic: how about moving https://github.com/dennisameling/git-sdk-arm64/ to the git-for-windows org?

@dennisameling
Copy link
Contributor Author

dennisameling commented Dec 21, 2022

@dennisameling should we use the replaces construct and teach quick_add to avoid adding the mingw-w64-git-extra to the Pacman databases that only include the "other" MINGW packages (probably here)? I.e. in a git-sdk-32, we do not want the git-for-windows-mingw64 database to offer the x86_64 variant of mingw-w64-git-extra, nor do we want in a git-sdk-64 to let the git-for-windows-mingw32 database offer the i686 variant.

That makes sense. I'll have a look at that tomorrow or on Friday.

On a totally unrelated topic: how about moving https://github.com/dennisameling/git-sdk-arm64/ to the git-for-windows org?

Sounds good to me! I tried sending it over to the git-for-windows org but ran into an error. You should have full access to my repo, so would you mind trying to transfer it over?

image

Update: I tried transferring it to your personal user but that also didn't work, as my repo is a fork of https://github.com/git-for-windows/git-sdk-64:

image

Please feel free to take whatever action you deem necessary to port it over to the git-for-windows org 👍🏼

@dscho
Copy link
Member

dscho commented Dec 22, 2022

I tried sending it over to the git-for-windows org but ran into an error. You should have full access to my repo, so would you mind trying to transfer it over?

Unfortunately, I do not have full access, but I am rather convinced that I would have run into the same troubles.

Instead, I re-forked git-sdk-64 into https://github.com/git-for-windows/git-sdk-arm64. You should have full admin rights there, so please feel free to use them!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants