Properly handle OpenSSL semver #3065

SirLynix · 2022-11-15T15:31:27Z

Is your feature request related to a problem? Please describe.

OpenSSL uses a weird semver n1.n2.n3x (https://wiki.openssl.org/index.php/Versioning), where n1 and n2 are major, n3 is minor and x is a one or more letter patch. (example: 1.1.1a)

xmake-repo has an openssl package using those versions but you can't really use it.

For example, I'm trying to patch the python package which requires OpenSSL 1.1.1, but I can't tell that to xmake. I tried:

package("python")
    add_deps("openssl >=1.1.1")

and it fails:

error: @programdir/core/main.lua:280: @programdir/modules/private/action/require/impl/package.lua:379: package(openssl): version(>=1.1.1) not found!

same for add_deps("openssl >=1.1.1a"), add_deps("openssl >=1.1.1h") (first 1.1.1 version in xmake-repo), even add_deps("openssl 1.1.1h") failed with error: unable to parse semver range '1.1.1h'.

Describe the solution you'd like

Either a patch in xmake semver to handle this weird case or a change in openssl package versioning to allow to select the versions we want.

Describe alternatives you've considered

No response

Additional context

Tested on latest version of xmake dev (6a29345)

The text was updated successfully, but these errors were encountered:

waruqi · 2022-11-15T15:59:13Z

1.1.1h is not a standard semantic format and we should not make semver to support it.

maybe we can modify version format, like this?

xmake> semver.satisfies("1.1.1-a", ">=1.1.1-h")
< false
xmake> semver.satisfies("1.1.1-d", ">=1.1.1-h")
< false
xmake> semver.satisfies("1.1.1-z", ">=1.1.1-h")
< true

SirLynix · 2022-11-15T16:08:10Z

seems good, I will make a pull request to change the package

SirLynix · 2022-11-16T09:23:55Z

While this issue is open, I'm using it for another issue.

I'm trying to install the python package on macOS, which should install OpenSSL, however this is not the case:
https://github.com/NazaraEngine/NazaraEngine/actions/runs/3477254257/jobs/5813232492

openssl is not found:

finding openssl from xmake ..
checking for xmake::openssl ... no
finding openssl from find_openssl ..
finding openssl from brew ..
finding openssl from vcpkg ..
finding openssl from conan ..
finding openssl from pkgconfig ..
finding openssl from system ..
checking for openssl ... no

but is not installed before python is built:

checking for openssl/ssl.h in /usr/local/ssl... no
checking for openssl/ssl.h in /usr/lib/ssl... no
checking for openssl/ssl.h in /usr/ssl... no
checking for openssl/ssl.h in /usr/pkg... no
checking for openssl/ssl.h in /usr/local... no
checking for openssl/ssl.h in /usr... no
checking whether compiling and linking against OpenSSL works... no
checking for --with-openssl-rpath... 
checking whether OpenSSL provides required APIs... no

even though openssl is a package dep: https://github.com/NazaraEngine/NazaraEngine/blob/macos-fix/xmake-repo/packages/p/python/xmake.lua#L46

am I doing something wrong?

waruqi · 2022-11-16T09:33:02Z

It works for me.

$ xmake l scripts/test.lua -vD --shallow python
/usr/bin/curl -SL -A "Xmake/2.7.3+dev.736765d (Linux;5.10.25-linuxkit) curl/7.83.1" https://github.com/openssl/openssl/archive/refs/tags/OpenSSL_1_1_1q.zip -o OpenSSL_1_1_1q.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100  7724    0  7724    0     0   3372      0 --:--:--  0:00:02 --:--:--  7670^C

SirLynix · 2022-11-16T09:35:27Z

it seems to work on my Linux pipeline too, but not on macOS?

I don't have a macOS to test it

waruqi · 2022-11-16T09:38:09Z

It works for me too on macOS.

$ xmake l scripts/test.lua -vD --shallow python
{
  "python"
}

create test ...
  [+]: xmake.lua
  [+]: src/main.cpp
  [+]: .gitignore
create ok!
/private/var/folders/32/w9cz0y_14hs19lkbs6v6_fm80000gn/T/.xmake501/221116/xmake-repo/test
add local repository(local-repo): /Users/ruki/projects/personal/xmake-repo ok!
local repositories:
    local-repo /Users/ruki/projects/personal/xmake-repo

global repositories:
    build-artifacts https://gitee.com/xmake-mirror/build-artifacts.git main
    xmake-repo https://gitee.com/tboox/xmake-repo.git master
    builtin-repo /Users/ruki/.local/share/xmake/repository

4 repositories were found!
checking for platform ... macosx
checking for architecture ... x86_64
checking for Xcode directory ... /Applications/Xcode.app
checking for Codesign Identity of Xcode ... Apple Development: waruqi@gmail.com (T3NA4MRVPU)
checking for SDK version of Xcode for macosx (x86_64) ... 12.3
checking for Minimal target version of Xcode for macosx (x86_64) ... 12.3
checking for dmd ... /usr/local/bin/dmd
checking for zig ... /usr/local/bin/zig
configure
{
    proxy_pac = pac.lua
    arch = x86_64
    network = public
    buildir = build
    ccache = true
    theme = default
    xcode = /Applications/Xcode.app
    xcode_codesign_identity = Apple Development: waruqi@gmail.com (T3NA4MRVPU)
    kind = static
    ndk_stdcxx = true
    mode = release
    host = macosx
    plat = macosx
    clean = true
}
checking for unzip ... /usr/bin/unzip
checking for git ... /usr/bin/git
checking for gzip ... /usr/bin/gzip
checking for tar ... /usr/bin/tar
/usr/bin/git rev-parse HEAD
finding openssl from xmake ..
checking for xmake::openssl ... no
finding openssl from find_openssl ..
finding openssl from brew ..
checking for brew ... /usr/local/bin/brew
finding openssl from vcpkg ..
finding openssl from conan ..
finding openssl from pkgconfig ..
checking for pkg-config ... /usr/local/bin/pkg-config
finding openssl from system ..
checking for openssl ... no
finding ca-certificates from xmake ..
checking for xmake::ca-certificates ... no
finding ca-certificates from brew ..
finding ca-certificates from vcpkg ..
finding ca-certificates from conan ..
finding ca-certificates from pkgconfig ..
finding ca-certificates from system ..
checking for ca-certificates ... no
checking for ping ... /sbin/ping
pinging for the host(www.python.org) ... 65 ms
pinging for the host(github.com) ... 65535 ms
/usr/bin/unzip -o OpenSSL_1_1_1q.zip -d source.tmp -x */fuzz/*
Archive:  OpenSSL_1_1_1q.zip
29708a562a1887a91de0fa6ca668c71871accde9
   creating: source.tmp/openssl-OpenSSL_1_1_1q/
  inflating: source.tmp/openssl-OpenSSL_1_1_1q/ACKNOWLEDGEMENTS
  inflating: source.tmp/openssl-OpenSSL_1_1_1q/AUTHORS
  inflating: source.tmp/openssl-OpenSSL_1_1_1q/CHANGES
  inflating: source.tmp/openssl-OpenSSL_1_1_1q/CONTRIBUTING
   creating: source.tmp/openssl-OpenSSL_1_1_1q/Configurations/
  inflating: source.tmp/openssl-OpenSSL_1_1_1q/Configurations/00-base-templates.conf
  inflating: source.tmp/openssl-OpenSSL_1_1_1q/Configurations/10-main.conf
  inflating: source.tmp/openssl-OpenSSL_1_1_1q/Configurations/15-android.conf

SirLynix · 2022-11-16T09:40:38Z

this is weird, the only references I get for "openssl" on my macOS pipeline are xmake which fails to find it and python which doesn't find it, no installation is made.

any clue on how to debug this?

waruqi · 2022-11-16T09:56:00Z

fork xmake source and create a debug branch to debug it.

then switch to your xmake branch.

uses: xmake-io/github-action-setup-xmake@v1
with:
  xmake-version: your_repo#branch@master

https://github.com/xmake-io/github-action-setup-xmake/blob/ef151cf2d232cc7ee46035f9b225282f4eeea4a9/src/versions.ts#L117

Or clone and install xmake with given debug branch from source on your ci directly.

SirLynix · 2022-11-16T13:27:49Z

https://github.com/NazaraEngine/NazaraEngine/actions/runs/3479090238/jobs/5817999753

I added quite a lot of logs (here's the full list: dev...SirLynix:xmake:openssl_debug)

should_install(openssl)
should_install(openssl) checking parent python
should_install(python)
should_install(python) checking parent xcb-proto
should_install(xcb-proto)
should_install(xcb-proto) checking parent libxcb
should_install(libxcb)
 _compatible_with_previous_librarydeps(libxcb)
   - yes (no any dependencies)
should_install(libxcb) - no (_compatible_with_previous_librarydeps returned true)
should_install(xcb-proto) parent:exists = true
should_install(xcb-proto) option.get("force") = nil
should_install(xcb-proto) _must_depend_on(libxcb, xcb-proto) = false
should_install(xcb-proto) - nil (end of function)
should_install(python) parent:exists = false
should_install(python) option.get("force") = nil
should_install(python) _must_depend_on(xcb-proto, python) = false
should_install(python) - nil (end of function)
should_install(openssl) parent:exists = false
should_install(openssl) option.get("force") = nil
should_install(openssl) _must_depend_on(python, openssl) = false
should_install(openssl) - nil (end of function)
_should_install_package(openssl) = false

so openssl installation is skipped because it finds libxcb, which required xcb-proto, which required, which required openssl (so no need to install any of those).

However python is installed because later it's required for aqt:

_should_install_package(python)
should_install(python)
should_install(python) checking parent aqt
should_install(aqt)
should_install(aqt) checking parent qt5base
should_install(qt5base)
should_install(qt5base) checking parent qt5core
should_install(qt5core)
should_install(qt5core) checking parent qt5widgets
should_install(qt5widgets)
should_install(qt5widgets) checking parent nodeeditor
should_install(nodeeditor)
should_install(nodeeditor) - yes
should_install(qt5widgets) - yes (should_install nodeeditor returned true)
should_install(qt5core) - yes (should_install qt5widgets returned true)
should_install(qt5base) - yes (should_install qt5core returned true)
should_install(aqt) - yes (should_install qt5base returned true)
should_install(python) - yes (should_install aqt returned true)
_should_install_package(python) = true

but openssl is not rechecked.

I'm wondering why aqt isn't a parent of python, but this looks like a bug of xmake, you will probably understand it better than me (or maybe if you know in which file I should add logs if you're busy).

waruqi · 2022-11-16T14:18:16Z

but openssl is not rechecked.

maybe cache cause it?

xmake/xmake/modules/private/action/require/impl/install_packages.lua

Line 363 in 6a29345

local result = _g.package_status_cache[tostring(instance)]

SirLynix · 2022-11-16T15:08:41Z

I'm not sure but maybe.

What I don't understand is that python is dismissed, it should have more than one parents as:

package(python):parents_add(aqt)

but aqt is never checked as a package using python.

Here's the full log with parents_add logged:
https://github.com/NazaraEngine/NazaraEngine/actions/runs/3479090238/jobs/5820293329

waruqi · 2022-11-16T15:19:00Z

you can check here

xmake/xmake/modules/private/action/require/impl/package.lua

Line 885 in 6db72d7

dep:parents_add(package)

SirLynix · 2022-11-16T15:23:43Z

Yeah I logged it, and package(python):parents_add(aqt) is called.

But

_should_install_package(python)
should_install(python)
should_install(python) checking parent xcb-proto
should_install(xcb-proto)
should_install(xcb-proto) checking parent libxcb
should_install(libxcb)
 _compatible_with_previous_librarydeps(libxcb)
   - yes (no any dependencies)
should_install(libxcb) - no (_compatible_with_previous_librarydeps returned true)
should_install(xcb-proto) parent:exists = true
should_install(xcb-proto) option.get("force") = nil
should_install(xcb-proto) _must_depend_on(libxcb, xcb-proto) = false
should_install(xcb-proto) - nil (end of function)
should_install(python) parent:exists = false
should_install(python) option.get("force") = nil
should_install(python) _must_depend_on(xcb-proto, python) = false
should_install(python) - nil (end of function)
_should_install_package(python) = false

only check one parent (xcb-proto), I don't understand hwy

waruqi · 2022-11-16T15:34:23Z

you can print python package instance address, tostring(package)

maybe another python instance?

SirLynix added the feature request label Nov 15, 2022

SirLynix mentioned this issue Nov 16, 2022

Fix package:parents_add overwriting if another instance of the package… #3072

Merged

waruqi mentioned this issue Nov 17, 2022

Improve package parents #3073

Merged

waruqi added this to the v2.7.4 milestone Nov 17, 2022

SirLynix closed this as completed Nov 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Properly handle OpenSSL semver #3065

Properly handle OpenSSL semver #3065

SirLynix commented Nov 15, 2022 •

edited

Loading

waruqi commented Nov 15, 2022 •

edited

Loading

SirLynix commented Nov 15, 2022

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022 •

edited

Loading

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022 •

edited

Loading

SirLynix commented Nov 16, 2022 •

edited

Loading

waruqi commented Nov 16, 2022

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022

Properly handle OpenSSL semver #3065

Properly handle OpenSSL semver #3065

Comments

SirLynix commented Nov 15, 2022 • edited Loading

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

waruqi commented Nov 15, 2022 • edited Loading

SirLynix commented Nov 15, 2022

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022 • edited Loading

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022 • edited Loading

SirLynix commented Nov 16, 2022 • edited Loading

waruqi commented Nov 16, 2022

SirLynix commented Nov 16, 2022

waruqi commented Nov 16, 2022

SirLynix commented Nov 15, 2022 •

edited

Loading

waruqi commented Nov 15, 2022 •

edited

Loading

waruqi commented Nov 16, 2022 •

edited

Loading

waruqi commented Nov 16, 2022 •

edited

Loading

SirLynix commented Nov 16, 2022 •

edited

Loading