[openssl,vcpkg-cmake-get-vars] Fix android and emscripten, add feature `tools` #29966

dg0yt · 2023-03-02T09:32:30Z

Fixes #29947. (Android regression from #27261.)
Fixes #30037. Fixes #29575. (wasm32-emscripten, broken before #27261.)
Fixes #29674. (arm64-linux, broken before #27261.)
The openssl tool is now an opt-in feature. Resolves #29948. (Not a regression, but changed by #27261.)

Community test success reported for android, ios, wasm-emscripten and mingw triplets.

While fixing Android, I noticed that vcpkg-cmake-get-vars was probably not working correctly with regard to osx deployment targets - we cannot rely on APPLE or ANDROID being set before calling project.

Testing also revealed insufficent handling of quotes in vcpkg-cmake-get-vars with regard to the generated cmake files. This PR adds uniform escaping, and it ports relevant changes from vcpkg-cmake-get-vars to the global script.
Finally it turned out that additional quoting fixes were needed with regard to passing options to tools in vcpkg_configure_make.

Last not least, it generally fixes libpq's openssl version detection in a way which doesn't need to run the (target:zap:) openssl tool.

Changes comply with the maintainer guide
SHA512s are updated for each updated download
The "supports" clause reflects platforms that may be fixed by this new version
Any fixed CI baseline entries are removed from that file.
Any patches that are no longer applied are deleted from the port's directory.
The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
Only one version is added to each modified port's versions file.

System variables like 'APPLE' or 'ANDROID' aren't set before 'project()'.

ports/vcpkg-cmake-get-vars/cmake_get_vars/CMakeLists.txt

talregev · 2023-03-02T10:44:48Z

@dg0yt
Thank you for the fix!
Openssl is open source. Can you make a fix suggestion to them?

AenBleidd · 2023-03-02T15:26:20Z

@dg0yt, thank you. I'll try to test this during next two days.

P.S. Sorry I haven't seen you was working on that PR since October. Unfortunately, I don't track all changes that are happening on vcpkg. I only see smth that breaks my CI.
Don't hesitate to ping me next time you'll be working on openssl stuff or smth related to it, I'll try to help you testing it on some community triplets.

talregev · 2023-03-05T06:25:10Z

This fix doesn't work for android:

Building openssl[core]:armv6-android...
-- [OVERLAY] Loading triplet configuration from: /home/tal/boinc/3rdParty/vcpkg_ports/triplets/ci/armv6-android.cmake
-- Downloading https://github.com/openssl/openssl/archive/openssl-3.0.8.tar.gz -> openssl-openssl-openssl-3.0.8.tar.gz...
-- Extracting source /home/tal/boinc/3rdParty/android/vcpkg/downloads/openssl-openssl-openssl-3.0.8.tar.gz
-- Applying patch disable-apps.patch
-- Applying patch disable-install-docs.patch
-- Applying patch script-prefix.patch
-- Applying patch windows/install-layout.patch
-- Applying patch windows/install-pdbs.patch
-- Applying patch windows/umul128-arm64.patch
-- Applying patch unix/android-cc.patch
-- Applying patch unix/move-openssldir.patch
-- Applying patch unix/no-empty-dirs.patch
-- Applying patch unix/no-static-libs-for-shared.patch
-- Using source at /home/tal/boinc/3rdParty/android/vcpkg/buildtrees/openssl/src/nssl-3.0.8-1c0004bf25.clean
-- Found external ninja('1.11.1.git.kitware.jobserver-1').
-- Getting CMake variables for armv6-android
CMake Warning (dev) at buildtrees/openssl/cmake-get-vars-armv6-android-rel.cmake.log:45:
  Syntax Warning in cmake code at column 55

  Argument not separated from preceding token by whitespace.
Call Stack (most recent call first):
  buildtrees/openssl/cmake-get-vars-armv6-android.cmake.log:1 (include)
  ports/openssl/unix/portfile.cmake:34 (include)
  ports/openssl/portfile.cmake:66 (include)
  scripts/ports.cmake:147 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Getting CMake variables for armv6-android-rel
CMake Warning (dev) at buildtrees/openssl/cmake-vars-armv6-android-rel.cmake.log:31:
  Syntax Warning in cmake code at column 55

  Argument not separated from preceding token by whitespace.
Call Stack (most recent call first):
  buildtrees/openssl/cmake-vars-armv6-android-release.cmake.log:1 (include)
  scripts/cmake/vcpkg_configure_make.cmake:137 (include)
  ports/openssl/unix/portfile.cmake:103 (vcpkg_configure_make)
  ports/openssl/portfile.cmake:66 (include)
  scripts/ports.cmake:147 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring armv6-android-rel
CMake Warning (dev) at buildtrees/openssl/cmake-vars-armv6-android-rel.cmake.log:31:
  Syntax Warning in cmake code at column 55

  Argument not separated from preceding token by whitespace.
Call Stack (most recent call first):
  buildtrees/openssl/cmake-vars-armv6-android-release.cmake.log:1 (include)
  scripts/cmake/vcpkg_build_make.cmake:3 (include)
  scripts/cmake/vcpkg_install_make.cmake:2 (vcpkg_build_make)
  ports/openssl/unix/portfile.cmake:117 (vcpkg_install_make)
  ports/openssl/portfile.cmake:66 (include)
  scripts/ports.cmake:147 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Building armv6-android-rel
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:134 (message):
    Command failed: /usr/bin/make V=1 -j 13 -f Makefile build_sw
    Working Directory: /home/tal/boinc/3rdParty/android/vcpkg/buildtrees/openssl/armv6-android-rel/
    See logs for more information:
      /home/tal/boinc/3rdParty/android/vcpkg/buildtrees/openssl/build-armv6-android-rel-out.log
      /home/tal/boinc/3rdParty/android/vcpkg/buildtrees/openssl/build-armv6-android-rel-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_build_make.cmake:149 (vcpkg_execute_build_process)
  scripts/cmake/vcpkg_install_make.cmake:2 (vcpkg_build_make)
  ports/openssl/unix/portfile.cmake:117 (vcpkg_install_make)
  ports/openssl/portfile.cmake:66 (include)
  scripts/ports.cmake:147 (include)

buildtrees.zip

one of the errors in the log:

In file included from ../src/nssl-3.0.8-1c0004bf25.clean/crypto/aes/aes_ecb.c:10:
In file included from /usr/include/assert.h:35:
In file included from /usr/include/features.h:392:
/usr/include/features-time64.h:20:10: fatal error: 'bits/wordsize.h' file not found
#include <bits/wordsize.h>
         ^~~~~~~~~~~~~~~~~

dg0yt · 2023-03-05T06:49:20Z

Show buildtrees/openssl/cmake-vars-armv6-android-rel.cmake.log.
armv6-android: not even a community triplet.

talregev · 2023-03-05T06:53:18Z

Show buildtrees/openssl/cmake-vars-armv6-android-rel.cmake.log. armv6-android: not even a community triplet.

armv6 not added as a community triplet and support added here:
https://github.com/microsoft/vcpkg/blob/master/scripts/toolchains/android.cmake#L14

dg0yt · 2023-03-05T07:38:06Z

Again, show buildtrees/openssl/cmake-vars-armv6-android-rel.cmake.log.
That triplet doesn't work with NDK 23c or NDK r25:

CMake Error at /<NDK_ROOT>/build/cmake/android-legacy.toolchain.cmake:164 (message):
  armeabi is no longer supported.  Use armeabi-v7a.

dg0yt · 2023-03-05T07:43:13Z

Got it from your zip:

set(VCPKG_DETECTED_CMAKE_CXX_STANDARD_LIBRARIES "-lm "/home/tal/boinc/3rdParty/android/android-ndk-r15c/sources/cxx-stl/llvm-libc++/libs/armeabi/libc++.a" "-latomic"")

Obviously broken. But I'm not convinced that this is related to this PR.

talregev · 2023-03-05T07:56:20Z

it working with ndk r15c

talregev · 2023-03-05T08:09:28Z

Got it from your zip:
set(VCPKG_DETECTED_CMAKE_CXX_STANDARD_LIBRARIES "-lm "/home/tal/boinc/3rdParty/android/android-ndk-r15c/sources/cxx-stl/llvm-libc++/libs/armeabi/libc++.a" "-latomic"")
Obviously broken. But I'm not convinced that this is related to this PR.

This is a suggestion for fix?

talregev · 2023-03-05T08:14:02Z

Got it from your zip:
set(VCPKG_DETECTED_CMAKE_CXX_STANDARD_LIBRARIES "-lm "/home/tal/boinc/3rdParty/android/android-ndk-r15c/sources/cxx-stl/llvm-libc++/libs/armeabi/libc++.a" "-latomic"")
Obviously broken. But I'm not convinced that this is related to this PR.

I added this workaround for armv6-android triplet, and I get the same error.

talregev · 2023-03-05T08:27:21Z

armv6 was working before your change here:
#27261

dg0yt · 2023-03-05T08:34:10Z

This is a suggestion for fix?

No. I was only pointing at the line which was reported as syntax error.

I added this workaround for armv6-android triplet, and I get the same error.

This is not how things work. And if we know that the syntax is broken, it is probably not helpful to add it anywhere.

The error can be reproduced in master with openssl:arm-android and NDK r18b. (Even this NDK doesn't support armv6-android.) It is not a regression.

dg0yt · 2023-03-05T08:50:28Z

armv6 was working before your change here: #27261

So it is a regression from this PR. But the issue is not in the scripts modified by that PR but used since that PR.

Neumann-A · 2023-03-06T21:38:19Z

Maybe helpful note: vcpkg might to pass compiler flags via CC etc instead of CFLAGS etc. in the future. So you can just deactivate whatever the internal buildsystem of a port would add.

dg0yt · 2023-03-07T05:14:55Z

Maybe helpful note: vcpkg might to pass compiler flags via CC etc instead of CFLAGS etc. in the future. So you can just deactivate whatever the internal buildsystem of a port would add.

For openssl, it doesn't help too much because it ignores CC for some triplets.
It also doesn't help with another general problem: You cannot pass quoted arguments ("-I/with space") via variables ( CFLAGS, CC) which war substituted when make is executed.

talregev · 2023-03-07T07:23:05Z

@dg0yt Thank you for these fixes. I can confirm that it working for all android triplets and wasm32-emscripten.

dg0yt · 2023-03-07T18:24:19Z

@m-kuhn Do you openssl on android, ios or emscripten? (In particular ios wasn't tested yet AFAICT.)

m-kuhn · 2023-03-07T18:47:34Z

@m-kuhn Do you openssl on android, ios or emscripten? (In particular ios wasn't tested yet AFAICT.)

Ios and android (r25), yes. Should I give this PR a testrun?

dg0yt · 2023-03-07T19:45:55Z

@m-kuhn That would be great.. There is some room for unintented side effects.

m-kuhn · 2023-03-08T08:53:27Z

It builds and links fine, thank you for asking.

talregev · 2023-03-13T19:46:21Z

May I ask why we put this pr on hold?

AenBleidd · 2023-03-14T12:35:45Z

@Neumann-A, any chance to get this merged this week?

Neumann-A · 2023-03-14T13:27:54Z

@AenBleidd i don't have any power here so i am the wrong person to mention for that.

dg0yt added 6 commits March 2, 2023 09:43

[vcpkg-cmake-get-vars] Call 'project' early

02ef3cf

System variables like 'APPLE' or 'ANDROID' aren't set before 'project()'.

Get Android variables

9986572

[openssl] Ensure that compiler is reached via 'PATH'

6c949b3

Fix Android configuration

c122427

Add 'tools' feature

db4628d

Update versions

ce61c5e

dg0yt commented Mar 2, 2023

View reviewed changes

ports/vcpkg-cmake-get-vars/cmake_get_vars/CMakeLists.txt Show resolved Hide resolved

ports/vcpkg-cmake-get-vars/cmake_get_vars/CMakeLists.txt Show resolved Hide resolved

Adela0814 assigned LilyWangLL Mar 2, 2023

LilyWangLL added the category:community-triplet A PR or issue related to community triplets not officially validated by the vcpkg team. label Mar 2, 2023

flichtenheld mentioned this pull request Mar 2, 2023

[openssl] do not try to install openssl as tool if cross-compiling #29949

Closed

7 tasks

dg0yt force-pushed the openssl branch from c01c761 to f22d871 Compare March 4, 2023 13:36

dg0yt added 4 commits March 4, 2023 23:28

[openssl] Initialize android with vcpkg settings

db099dd

[openssl] Really don't build apps

504bbb4

[vcpkg-cmake-get-vars] Update android vars

fc9f70c

[libpq] Pass openssl version from pkgconfig

3c17992

dg0yt force-pushed the openssl branch 2 times, most recently from af8102b to 3c17992 Compare March 4, 2023 22:42

talregev mentioned this pull request Mar 7, 2023

[openssl] Revert. support for android and wasm32-emscripten #30053

Closed

[openssl] Fix emscripten

c36815b

dg0yt changed the title ~~[openssl,vcpkg-cmake-get-vars] Fix Android, add feature tools~~ [openssl,vcpkg-cmake-get-vars] Fix android and emscripten, add feature tools Mar 7, 2023

dg0yt marked this pull request as ready for review March 7, 2023 06:42

This was referenced Mar 7, 2023

[openssl] Build error Include '[openssl] Build error #30065

Closed

[openssl] build failure by emscripten #29575

Closed

dg0yt mentioned this pull request Mar 7, 2023

[openssl] build failure on Android #29947

Closed

dg0yt mentioned this pull request Mar 8, 2023

[openssl] Build error with arm64-linux triplet #29674

Closed

LilyWangLL approved these changes Mar 9, 2023

View reviewed changes

LilyWangLL assigned Cheney-W Mar 9, 2023

Cheney-W approved these changes Mar 10, 2023

View reviewed changes

LilyWangLL added the info:reviewed Pull Request changes follow basic guidelines label Mar 10, 2023

This was referenced Mar 14, 2023

[qscintilla] update version to 2.13.4 #30158

Merged

Use msys2 bash regardless of target and of Windows Apps #30172

Merged

dg0yt mentioned this pull request Mar 14, 2023

[vcpkg] Android CI support #29406

Merged

dan-shaw merged commit 73b9a2a into microsoft:master Mar 14, 2023

dg0yt deleted the openssl branch March 15, 2023 04:58

dg0yt mentioned this pull request Mar 16, 2023

[libpng] Fix arm-neon-android #30226

Merged

7 tasks

sjperkins mentioned this pull request Apr 2, 2023

GH-33804: [Python] Add support for manylinux_2_28 wheel apache/arrow#34818

Merged

sjperkins mentioned this pull request Apr 13, 2023

[CI][Packaging] Pin vcpkg to release that includes OpenSSL linux-headers fix apache/arrow#35099

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[openssl,vcpkg-cmake-get-vars] Fix android and emscripten, add feature `tools` #29966

[openssl,vcpkg-cmake-get-vars] Fix android and emscripten, add feature `tools` #29966

dg0yt commented Mar 2, 2023 •

edited

Loading

talregev commented Mar 2, 2023 •

edited

Loading

AenBleidd commented Mar 2, 2023

talregev commented Mar 5, 2023 •

edited

Loading

dg0yt commented Mar 5, 2023 •

edited

Loading

talregev commented Mar 5, 2023

dg0yt commented Mar 5, 2023

dg0yt commented Mar 5, 2023 •

edited

Loading

talregev commented Mar 5, 2023

talregev commented Mar 5, 2023

talregev commented Mar 5, 2023

talregev commented Mar 5, 2023

dg0yt commented Mar 5, 2023 •

edited

Loading

dg0yt commented Mar 5, 2023

Neumann-A commented Mar 6, 2023

dg0yt commented Mar 7, 2023

talregev commented Mar 7, 2023 •

edited

Loading

dg0yt commented Mar 7, 2023

m-kuhn commented Mar 7, 2023

dg0yt commented Mar 7, 2023

m-kuhn commented Mar 8, 2023

talregev commented Mar 13, 2023 •

edited

Loading

AenBleidd commented Mar 14, 2023

Neumann-A commented Mar 14, 2023

[openssl,vcpkg-cmake-get-vars] Fix android and emscripten, add feature tools #29966

[openssl,vcpkg-cmake-get-vars] Fix android and emscripten, add feature tools #29966

Conversation

dg0yt commented Mar 2, 2023 • edited Loading

talregev commented Mar 2, 2023 • edited Loading

AenBleidd commented Mar 2, 2023

talregev commented Mar 5, 2023 • edited Loading

dg0yt commented Mar 5, 2023 • edited Loading

talregev commented Mar 5, 2023

dg0yt commented Mar 5, 2023

dg0yt commented Mar 5, 2023 • edited Loading

talregev commented Mar 5, 2023

talregev commented Mar 5, 2023

talregev commented Mar 5, 2023

talregev commented Mar 5, 2023

dg0yt commented Mar 5, 2023 • edited Loading

dg0yt commented Mar 5, 2023

Neumann-A commented Mar 6, 2023

dg0yt commented Mar 7, 2023

talregev commented Mar 7, 2023 • edited Loading

dg0yt commented Mar 7, 2023

m-kuhn commented Mar 7, 2023

dg0yt commented Mar 7, 2023

m-kuhn commented Mar 8, 2023

talregev commented Mar 13, 2023 • edited Loading

AenBleidd commented Mar 14, 2023

Neumann-A commented Mar 14, 2023

[openssl,vcpkg-cmake-get-vars] Fix android and emscripten, add feature `tools` #29966

[openssl,vcpkg-cmake-get-vars] Fix android and emscripten, add feature `tools` #29966

dg0yt commented Mar 2, 2023 •

edited

Loading

talregev commented Mar 2, 2023 •

edited

Loading

talregev commented Mar 5, 2023 •

edited

Loading

dg0yt commented Mar 5, 2023 •

edited

Loading

dg0yt commented Mar 5, 2023 •

edited

Loading

dg0yt commented Mar 5, 2023 •

edited

Loading

talregev commented Mar 7, 2023 •

edited

Loading

talregev commented Mar 13, 2023 •

edited

Loading