Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[vcpkg_from_git] fetching explicit ref using old gits fails #14424

Closed
abique opened this issue Nov 6, 2020 · 35 comments
Closed

[vcpkg_from_git] fetching explicit ref using old gits fails #14424

abique opened this issue Nov 6, 2020 · 35 comments
Assignees
@strega-nil @NancyLi1013 @JackBoosY
Labels
requires:repro The issue is not currently repro-able

Comments

@abique
Copy link
Contributor

abique commented Nov 6, 2020

Host Environment

  • OS: Linux
  • Compiler: gcc 9.2.0

To Reproduce
Steps to reproduce the behavior:
./vcpkg install bzip2

Failure logs

Building package bzip2[core]:x64-linux...
Could not locate cached archive: /home/buildbot/.cache/vcpkg/archives/3d/3d9803b8e70234e4e0840b2f1f7029d4213869fa.zip
-- Fetching https://sourceware.org/git/bzip2.git...
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:106 (message):
    Command failed: /usr/bin/git fetch https://sourceware.org/git/bzip2.git 75a94bea3918e612b879d6a11ca64b8689526147 --depth 1 -n
    Working Directory: /home/buildbot/vcpkg/downloads/git-tmp
    Error code: 128
    See logs for more information:
      /home/buildbot/vcpkg/buildtrees/bzip2/git-fetch-x64-linux-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_from_git.cmake:78 (vcpkg_execute_required_process)
  ports/bzip2/portfile.cmake:1 (vcpkg_from_git)
  scripts/ports.cmake:135 (include)


Error: Building package bzip2:x64-linux failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with `./vcpkg update`, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: bzip2:x64-linux
  Vcpkg version: 2020.06.15-unknownhash

Additionally, attach any relevant sections from the log files above.

error: no such remote ref 75a94bea3918e612b879d6a11ca64b8689526147
@abique
Copy link
Contributor Author

abique commented Nov 6, 2020

The correct sha1 should be: 6a8690fc8d26c815e798c588f796eabe9d684cf0

vcpkg/ports/bzip2/portfile.cmake

Line 4 in 7fbbc59

REF 75a94bea3918e612b879d6a11ca64b8689526147 # REFERENCE BZIP2 VERSION 1.0.8

@abique
Copy link
Contributor Author

abique commented Nov 6, 2020

@StarGate-One

@NancyLi1013 NancyLi1013 self-assigned this Nov 6, 2020
@NancyLi1013
Copy link
Contributor

Hi @abique

Thanks for posting this issue.

I can build bzip2 successfully on my machine.

vcpkg$ ./vcpkg install bzip2
Computing installation plan...
The following packages will be built and installed:
    bzip2[core]:x64-linux
Detecting compiler hash for triplet x64-linux...
Starting package 1/1: bzip2:x64-linux
Building package bzip2[core]:x64-linux...
Could not locate cached archive: /home/.cache/vcpkg/archives/fa/fa192ac911ba1575d557c832b5ac79d5060ec3a5.zip
-- Fetching https://sourceware.org/git/bzip2.git...
-- Cleaning sources at /home/vcpkg/buildtrees/bzip2/src/8689526147-d8c5a1c4b4.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/vcpkg/downloads/bzip2-75a94bea3918e612b879d6a11ca64b8689526147.tar.gz
-- Applying patch fix-import-export-macros.patch
-- Using source at /home/vcpkg/buildtrees/bzip2/src/8689526147-d8c5a1c4b4.clean
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Fixing pkgconfig file: /home/vcpkg/packages/bzip2_x64-linux/lib/pkgconfig/bzip2.pc
-- Fixing pkgconfig file: /home/vcpkg/packages/bzip2_x64-linux/debug/lib/pkgconfig/bzip2.pc
-- Installing: /home/vcpkg/packages/bzip2_x64-linux/share/bzip2/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/.cache/vcpkg/archives/fa/fa192ac911ba1575d557c832b5ac79d5060ec3a5.zip
Building package bzip2[core]:x64-linux... done
Installing package bzip2[core]:x64-linux...
Installing package bzip2[core]:x64-linux... done
Elapsed time for package bzip2:x64-linux: 15.72 s

The package bzip2 is compatible with built-in CMake targets:

    find_package(BZip2 REQUIRED)
    target_link_libraries(main PRIVATE BZip2::BZip2)

So it should be not related with REF.

@NancyLi1013 NancyLi1013 added the requires:repro The issue is not currently repro-able label Nov 6, 2020
@abique
Copy link
Contributor Author

abique commented Nov 6, 2020

Why would I can't find the ref?
Are we checking out the same repository or is there a man in the middle when checking out from France?

@StarGate-One
Copy link
Contributor

@abique cc: @NancyLi1013
FYI...

  1. PR [bzip2] Update to 1.0.8. #13178 [bzip2] Update to 1.0.8 was where bzip2 source URL was changed and the REF line was added, which is needed when pulling from an active git repository, either a reference or tag is required to obtain a copy of a stable code base.
  2. PR [bzip2] Add SHA512 hash check #13242 [bzip2] Add SHA512 hash check I added a comment to the REF line, NOT the REF line, but a comment, added the SHA512 Hash and deleted the CONTROL file and replaced it with file vcpkg.json.
  3. Have you tried creating a new clone of vcpkg, maybe the instance being used is corrupt in some way?
  4. I did not have any issues installing bzip2 on OpenSUSE Leap 15.2 in WSL2 or in a Hyper-V machine, and I just created a new clone, compiled/linked vcpkg binary and installed bzip2 on 20201106 ~20:00 GMT or ~16:00 eastern.
    Note the download bzip2 filename contains the commit hash where bzip2 was upgraded to version 1.0.8 75a94bea3918e612b879d6a11ca64b8689526147:

$ vcpkg install bzip2
Computing installation plan...
The following packages will be built and installed:
bzip2[core]:x64-linux
Detecting compiler hash for triplet x64-linux...
Starting package 1/1: bzip2:x64-linux
Building package bzip2[core]:x64-linux...
-- Fetching https://sourceware.org/git/bzip2.git...
-- Extracting source /home/openSUSE/git/vcpkg/downloads/bzip2-75a94bea3918e612b879d6a11ca64b8689526147.tar.gz
-- Applying patch fix-import-export-macros.patch
-- Using source at /home/openSUSE/git/vcpkg/buildtrees/bzip2/src/8689526147-d8c5a1c4b4.clean
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Fixing pkgconfig file: /home/openSUSE/git/vcpkg/packages/bzip2_x64-linux/lib/pkgconfig/bzip2.pc
-- Fixing pkgconfig file: /home/openSUSE/git/vcpkg/packages/bzip2_x64-linux/debug/lib/pkgconfig/bzip2.pc
-- Installing: /home/openSUSE/git/vcpkg/packages/bzip2_x64-linux/share/bzip2/copyright
-- Performing post-build validation
-- Performing post-build validation done
Building package bzip2[core]:x64-linux... done
Installing package bzip2[core]:x64-linux...
Installing package bzip2[core]:x64-linux... done
Elapsed time for package bzip2:x64-linux: 4.274 s

Total elapsed time: 5.606 s

The package bzip2 is compatible with built-in CMake targets:

find_package(BZip2 REQUIRED)
target_link_libraries(main PRIVATE BZip2::BZip2)

@abique
Copy link
Contributor Author

abique commented Nov 7, 2020

I did clone bzip2 and had a different sha1 for this commit.

@abique
Copy link
Contributor Author

abique commented Nov 7, 2020

image

@abique
Copy link
Contributor Author

abique commented Nov 7, 2020

Here on my computer it works (I was on a build machine yesterday):

20-11-07 10:47:16 abique@knot:~/d/bzip2 git:master> git checkout 75a94bea3918e612b879d6a11ca64b8689526147
Note: switching to '75a94bea3918e612b879d6a11ca64b8689526147'.

HEAD is now at 6a8690f Prepare for 1.0.8 release.
20-11-07 10:48:23 1 abique@knot:~/d/bzip2 git:(detached)>  git checkout master
Previous HEAD position was 6a8690f Prepare for 1.0.8 release.
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
20-11-07 10:48:35 abique@knot:~/d/bzip2 git:master> git checkout 6a8690fc8d26c815e798c588f796eabe9d684cf0
Note: switching to '6a8690fc8d26c815e798c588f796eabe9d684cf0'.

HEAD is now at 6a8690f Prepare for 1.0.8 release.
20-11-07 10:48:39 abique@knot:~/d/bzip2 git:(detached)>

I don't understand why both sha1 works, yet I think 6a8690fc8d26c815e798c588f796eabe9d684cf0 is the correct one to use and older git client might not be able to checkout 75a94bea3918e612b879d6a11ca64b8689526147.

@abique
Copy link
Contributor Author

abique commented Nov 7, 2020

I have git version 2.29.2 on my local machine.
The build server has git version 2.7.4.

@abique
Copy link
Contributor Author

abique commented Nov 7, 2020 •
edited
Loading

Also git log -p -n1 gives:

commit 6a8690fc8d26c815e798c588f796eabe9d684cf0 (HEAD, tag: bzip2-1.0.8)
Author: Mark Wielaard <mark@klomp.org>
Date:   Sat Jul 13 17:17:58 2019 +0200

    Prepare for 1.0.8 release.

So I'd say that the sha1 used in vcpkg is definitely not the one given by git.

@NancyLi1013
Copy link
Contributor

@abique
Thanks for your further investigation about this issue.

I also checked this here https://sourceware.org/git/?p=bzip2.git;a=commit;h=6a8690fc8d26c815e798c588f796eabe9d684cf0
and found that 6a8690fc8d26c815e798c588f796eabe9d684cf0 seems to be the correct one.

@StarGate-One
Thanks for your clarifications. Seems that this REF is not updated by you.

@Sibras
Could you please help confirm where you get the REF 75a94bea3918e612b879d6a11ca64b8689526147 ?

I tried to search this commit in https://sourceware.org/git/bzip2.git. But I found nothing about this.

@abique
Copy link
Contributor Author

abique commented Nov 9, 2020

You're welcome.
I think it would be good to have a tool that periodically check that the package sources are fine.

@StarGate-One
Copy link
Contributor

It looks like the tag for bzip2 v1.0.8 https://sourceware.org/git/?p=bzip2.git;a=tag;h=75a94bea3918e612b879d6a11ca64b8689526147 pointing to commit https://sourceware.org/git/?p=bzip2.git;a=commit;h=6a8690fc8d26c815e798c588f796eabe9d684cf0

@abique
Copy link
Contributor Author

abique commented Nov 9, 2020

So that would be the hash of the tag, and older git client would not be able to resolve it?

@StarGate-One
Copy link
Contributor

  • I honestly do not know why an older version can not resolve the tag?
  • I went to https://git-scm.com/docs/git-tag git documentation on tags and it shows the tag command has been available since git v2.1.4 although there are various changes in some versions since then, it would be conceivable that older version may not be able to read tags created in new versions, but that is just a guess on my part.
    image

@NancyLi1013
Copy link
Contributor

@StarGate-One
Thanks for your further information about this issue.

Now we can confirm that the REF 75a94bea3918e612b879d6a11ca64b8689526147 used in vcpkg is a tag for bzip2 v1.0.8 instead of the commit 6a8690fc8d26c815e798c588f796eabe9d684cf0.

It looks like the tag for bzip2 v1.0.8 https://sourceware.org/git/?p=bzip2.git;a=tag;h=75a94bea3918e612b879d6a11ca64b8689526147 pointing to commit https://sourceware.org/git/?p=bzip2.git;a=commit;h=6a8690fc8d26c815e798c588f796eabe9d684cf0

It seems that this is related with git version.

@abique
Could you please help confirm which version of git you used that fetch bzip2 source failed? Would you like to have a try based on another git version?

@abique
Copy link
Contributor Author

abique commented Nov 10, 2020

The buildbot used: git version 2.7.4

@abique
Copy link
Contributor Author

abique commented Nov 10, 2020

The build server is using Ubuntu 16.04.3 LTS, and there is no newer git available from the package manager.

@abique
Copy link
Contributor Author

abique commented Nov 10, 2020

This is strange I did the clone of the git repository manually and I could checkout the tag reference.
I'll try again from vcpkg.

@abique
Copy link
Contributor Author

abique commented Nov 10, 2020

This command fails and is repoduceable: /usr/bin/git fetch https://sourceware.org/git/bzip2.git 75a94bea3918e612b879d6a11ca64b8689526147 --depth 1 -n

@abique
Copy link
Contributor Author

abique commented Nov 10, 2020

This one also fails:
/usr/bin/git fetch https://sourceware.org/git/bzip2.git 6a8690fc8d26c815e798c588f796eabe9d684cf0 --depth 1 -n error: no such remote ref 6a8690fc8d26c815e798c588f796eabe9d684cf0

@StarGate-One
Copy link
Contributor

@abique cc: @NancyLi1013
Did some more research... Found a version of git 2.7.4 for windows portable...

  1. So it does seem version git 2.7.4 does not work for either reference
  2. But running the same commands from git 2.29.2(2) windows

@abique
Copy link
Contributor Author

abique commented Nov 11, 2020

I can see a few solutions to this issue:

  • Raise the minimum version requirement of git, and perform the necessary checks
  • Use different checkout strategy depending on the git version
    • with recent git client, lightweight checkout
    • with older git client, full checkout then revision checkout
  • Maybe the web interface to the git repository provides a way to download a tarball of a given revision instead of using the git client

What do you think?

@StarGate-One
Copy link
Contributor

@abique cc: @NancyLi1013 @JackBoosY @BillyONeal @strega-nil @Neumann-A @voskrese @vicroms @ras0219-msft

  • I think this would be something the Project Sponsors at Microsoft would need to discuss and decide on a direction to go.
  • All your suggestions are viable solutions...
  • Except mandating a new version of git only supporting lightweight checkout, although the easiest solution to implement, may leave many older versions of operating systems unable to use vcpkg as they may not have the supporting libraries available for a new git version? This would need research and testing on the most common, older long term support operating systems.

@BillyONeal BillyONeal changed the title [bzip2] build failure [vcpkg_from_git] fetching explicit ref using old gits fails Nov 11, 2020
@strega-nil
Copy link
Contributor

I think that the solution to this is to stop using the ref of the tag, and just use the commit ref (which should work...)

@abique
Copy link
Contributor Author

abique commented Nov 11, 2020

I think that the solution to this is to stop using the ref of the tag, and just use the commit ref (which should work...)

See #14424 (comment) using the commit ref does not always work.

@Neumann-A
Copy link
Contributor

Hmm does vcpkg not automatically get portable git from somewhere? I thought it would do that. Maybe just insert a version check and see if the system version is higher than the one within vcpkg? If not just use the version within vcpkg. (Just like Ninja/Meson etc.)
Also using an outdated git version is a security risk so somebody should probably update it.

@BillyONeal
Copy link
Member

At this time git is one of our prerequisites; we assume the user's git works because that's how they got our repo in the first place, and I don't think we've had issues of this kind before.

@strega-nil
Copy link
Contributor

strega-nil commented Nov 11, 2020 •
edited
Loading

@abique we should not be fetching one commit (do we?); git servers hate that.

@StarGate-One
Copy link
Contributor

To summarize:

  • The issue is with a git fetch
  • Where a repository is using a lightweight framework commit in tags/releases
  • Where a repository is only allowing fetches on advertised objects (in this case only tags/releases)
  • Which is not supported by older versions of git
  • and fail for all versions of git when attempting to fetch unadvertised objects

@JackBoosY JackBoosY self-assigned this Nov 12, 2020
@strega-nil strega-nil self-assigned this Nov 12, 2020
@StarGate-One StarGate-One mentioned this issue Dec 7, 2020
Closed
@yangninghua
Copy link

I succeeded!!!

sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

ubuntu16.04:
gcc version 6.5.0 20181026 (Ubuntu 6.5.0-2ubuntu1~16.04)

vcpkg-2020.11

@NancyLi1013 NancyLi1013 mentioned this issue Jan 29, 2021
Closed
@unusual-aspect unusual-aspect mentioned this issue Feb 2, 2021
Closed
@JaosonMa
Copy link

JaosonMa commented Feb 4, 2021

6.5.0-2ubuntu

what's your git version?

@Lnstree
Copy link

Lnstree commented Feb 25, 2021

change vcpkg_from_git =>vcpkg_from_github May useful

  1. modify ports/bzip2/portfile.cmake
#vcpkg_from_git(
#    OUT_SOURCE_PATH SOURCE_PATH
#    URL https://sourceware.org/git/bzip2.git
#    REF 75a94bea3918e612b879d6a11ca64b8689526147 # REFERENCE BZIP2 VERSION 1.0.8
#    SHA512 4611105f9090477b5f6f6dbd303a282099df71644e04d8a998ef81de487f6c8cac4c0ec1283ad737f6767c51f1e3b4e24e2ee021c6dd085925617d9ed145b2ba
#    PATCHES
#        fix-import-export-macros.patch
#)

vcpkg_from_github(
    OUT_SOURCE_PATH SOURCE_PATH
    REPO opencor/bzip2
    REF opencor-1.0.8
    SHA512 33c53bf71378e9879732779dea6a52a89dcc1f57025a2d5f331217481fbd4f82006cc4c9ad03fa789983b9421a6c988fdf9ac54980a4e6a7b98d37186c2b3fe3
    HEAD_REF master
    PATCHES
        fix-import-export-macros.patch
)
  1. rm -rf buildtrees/bzip2/
  2. ./vcpkg install bzip2

@StarGate-One StarGate-One mentioned this issue Mar 23, 2021
Closed
@deayalar
Copy link

I'm using Ubuntu 16.04
I had git 2.7. I updated to 2.31 and it worked without modifying the cmake file

@NancyLi1013 NancyLi1013 mentioned this issue Jun 29, 2021
Merged
@ras0219-msft ras0219-msft mentioned this issue Jul 1, 2021
Closed
@JackBoosY
Copy link
Contributor

We haven't been able to repro this; if more information comes up, or this issue appears again, please reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@strega-nil strega-nil

@NancyLi1013 NancyLi1013

@JackBoosY JackBoosY

Labels
requires:repro The issue is not currently repro-able
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests