GHA: use vcpkg to install packages for MSVC jobs by talregev · Pull Request #13979 · curl/curl

talregev · 2024-06-21T01:04:12Z

You already have vcpkg in your msvc jobs in your ci. You didn't aware of it.
I refactor the vcpkg and add more libraries that can compile with curl.
I added cache that can be restore, and not building every library from scratch every PR, it will use the cache. (save ~7min on openssl and schannel build).

talregev · 2024-06-21T08:21:33Z

I am closing this PR because I want to debug the cache, and I don't want to always activate curl ci. I will do it on my own ci.

talregev · 2024-06-21T09:10:31Z

on this line I must use double quotes. The ci will fail on single quotes
https://github.com/curl/curl/pull/13979/files#diff-5ce5c9ff86f58b1f87b35b5227b2e84cb69f022d6741e1854f3e1e181091773cR468

vszakats · 2024-06-21T10:01:06Z

Aside question: Few years ago I remember vcpkg was installing packages from source. Is this still true? On a cursory look the install command finished quickly that hinted this might no longer be the case. If so, do we need caching?

On environment variables I must use double quotes. when I use single quotes it leave the $ symbol, and not give the value of the variable:
export 'tal=$tal'
echo $tal
output: $tal

That's correct. (I wasn't suggesting single quotes in bash string with variables in them)

On the other hand ${{ }} is replaced by GHA before execution, so those can be single-quoted to exclude the chance of an unwanted evaluation.

talregev · 2024-06-21T10:03:15Z

Aside question: Few years ago I remember vcpkg was installing packages from source. Is this still true? On a cursory look the install command finished quickly that hinted this might no longer be the case. If so, do we need caching?
On environment variables I must use double quotes. when I use single quotes it leave the $ symbol, and not give the value of the variable:
export 'tal=$tal'
echo $tal
output: $tal
That's correct. (I wasn't suggesting single quotes in bash string with variables in them)

On the other hand ${{ }} is replaced by GHA before execution, so those can be single-quoted to exclude the chance of an unwanted evaluation.

vcpkg still installing packages from source
Because of my cache implementation on the ci, it use the cache and install the binaries faster.
We need the cache for save ci time.

- change to VCPKG_TRIPLET_DIR

talregev · 2024-06-28T04:20:44Z

@vszakats Can you approve the PR?

talregev · 2024-06-30T16:08:38Z

@vszakats Thank you!

talregev · 2024-06-30T16:09:05Z

@bagder Can you take a look on this PR?

bagder

Sorry, I cannot add anything here. Windows and cmake are two weak points of mine. @vszakats is an appropriate authority here!

vszakats · 2024-06-30T21:11:42Z

Thank you @talregev, merged now!

- cache on a per-package basis. Replace manual caching with a built-in solution. It shares cached package builds between jobs, e.g. libssh2 only builds once per platform (instead of once per job). Individual packages are built as needed (not the whole per-job tree). It also fixes the duplicate cache entry issues. Ref: https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-actions-cache Follow-up to e26cbe2 #13979 Follow-up to cb22cfc #14077 - add BoringSSL job with ECH enabled. The first such job in the curl CI. - add LibreSSL job. - use vcpkg pre-installed on the runner image, instead of rolling our own. This is quicker, simpler and more robust. Follow-up to e26cbe2 #13979 - show pre-installed vcpkg and ports version. - drop `gsasl` dependency till it reaches the pre-installed vcpkg ports. - re-add `find .` to see the binaries generated. - simplify setting up `PATH`. - exclude failing tests for any job enabling WinIDN. - drop collecting and uploading log archives. We already dump CMake logs, and our build doesn't use Ninja. Rest of files weren't generated by the curl build. We don't aim to debug vcpkg package builds. Closes #14090

- move `curl --version` into separate step. - move configure log to separate step. Run on success, too. - add step with `curl_config.h` dump (full and brief/sorted). - make `autoreconf` a separate step. - add each job configuration a short name. - shorten job names. Dedupe/drop redundant info, introduce abbreviations: AM = autotools, CM = CMake, U = Unicode, R = Release, not -> `!`, etc. Instead of mentioning `debug`, mentioned when it's not. - simplify `PATH` forming for MSVC jobs. It's sufficient to add the release binary directory of vcpkg, the debug one is redundant. Follow-up to e26cbe2 #13979 - other minor tidy-ups. Closes #14116

Follow-up to f99c08d curl#14090 Follow-up to e26cbe2 curl#13979 Closes #xxxxx

Now that the package reached the CI runner image. Follow-up to f99c08d #14090 Follow-up to e26cbe2 #13979 Closes #14170

SFTP/SCP tests were failing in CI with WinCNG libssh2 since we first added such job. With `curl: (67) Authentication failure`. The reason is that the default `ssh-keygen` RSA private key format changed to OpenSSH (RFC4716) in 2018. libssh2 does not support this format with some of its crypto backends. Fix it by generating keys explicitly in PEM format as necessary via the `-m` option. This format is universally recognized for RSA keys. 2018-08-24: https://www.openssh.com/txt/release-7.8: OpenSSH format becomes default 2010-08-23: https://www.openssh.com/txt/release-5.6: `-m` option first supported This fixed the auth issue, just to reveal a known flakiness issue in libssh2 + WinCNG, causing: ``` curl: (2) Failure establishing ssh session: -8, Unable to exchange encryption keys ``` Ref: https://github.com/curl/curl/actions/runs/14000494428/job/39205633258?pr=16781#step:15:1796 Tracked here: libssh2/libssh2#804 Mitigated in libssh2 tests by retrying them. Due to this, keep ignoring these test results. Also: - add an env to customize key format: `CURL_TEST_SSH_KEY_FORMAT` - display the generated format in the log. - GHA/linux: document the wolfSSH error code causing it to fail tests: ``` curl: (79) wolfssh SFTP connect error -1051 / WS_MATCH_KEY_ALGO_E / cannot match key algo with peer ``` Follow-up to 4911e7a #16735 Follow-up to 0ec72c1 #16672 Follow-up to e53523f #14859 Follow-up to e26cbe2 #13979 Closes #16781

SFTP/SCP tests were failing in CI with WinCNG libssh2 since we first added such job. With `curl: (67) Authentication failure`. The reason is that the default `ssh-keygen` RSA private key format changed to OpenSSH (RFC4716) in 2018. libssh2 does not support this format with some of its crypto backends. Fix it by generating keys explicitly in PEM format as necessary via the `-m` option. This format is universally recognized for RSA keys. 2018-08-24: https://www.openssh.com/txt/release-7.8: OpenSSH format becomes default 2010-08-23: https://www.openssh.com/txt/release-5.6: `-m` option first supported This fixed the auth issue, just to reveal a known flakiness issue in libssh2 + WinCNG, causing: ``` curl: (2) Failure establishing ssh session: -8, Unable to exchange encryption keys ``` Ref: https://github.com/curl/curl/actions/runs/14000494428/job/39205633258?pr=16781#step:15:1796 Tracked here: libssh2/libssh2#804 Mitigated in libssh2 tests by retrying them. Due to this, keep ignoring these test results. Also: - add an env to customize key format: `CURL_TEST_SSH_KEY_FORMAT` - display the generated format in the log. - GHA/linux: document the wolfSSH error code causing it to fail tests: ``` curl: (79) wolfssh SFTP connect error -1051 / WS_MATCH_KEY_ALGO_E / cannot match key algo with peer ``` Follow-up to 4911e7a curl#16735 Follow-up to 0ec72c1 curl#16672 Follow-up to e53523f curl#14859 Follow-up to e26cbe2 curl#13979 Closes curl#16781

github-actions bot added Windows Windows-specific CI Continuous Integration labels Jun 21, 2024

talregev force-pushed the TalR/vcpkg_windows_ci branch 5 times, most recently from 1d67ad3 to 75bc7ce Compare June 21, 2024 08:18

talregev closed this Jun 21, 2024

vszakats requested changes Jun 21, 2024

View reviewed changes

vszakats mentioned this pull request Jun 21, 2024

Add vcpkg ci #13963

Closed

talregev reopened this Jun 21, 2024

talregev force-pushed the TalR/vcpkg_windows_ci branch 3 times, most recently from a521870 to 8a0017d Compare June 21, 2024 09:07

talregev force-pushed the TalR/vcpkg_windows_ci branch 3 times, most recently from acbd46e to 8817151 Compare June 21, 2024 09:20

This comment was marked as resolved.

Sign in to view

talregev force-pushed the TalR/vcpkg_windows_ci branch from 8817151 to 745a5e8 Compare June 21, 2024 09:34

vszakats changed the title ~~msvc vcpkg ci~~ GHA: use vcpkg to install packages for msvc Jun 21, 2024

vszakats changed the title ~~GHA: use vcpkg to install packages for msvc~~ GHA: use vcpkg to install packages for MSVC jobs Jun 21, 2024

talregev force-pushed the TalR/vcpkg_windows_ci branch from 745a5e8 to 0b60eee Compare June 21, 2024 09:39

vszakats reviewed Jun 21, 2024

View reviewed changes

Comment thread .github/workflows/windows.yml Outdated

talregev force-pushed the TalR/vcpkg_windows_ci branch from 0b60eee to 80fd29d Compare June 21, 2024 09:59

vszakats reviewed Jun 21, 2024

View reviewed changes

Comment thread .github/workflows/windows.yml Outdated

talregev force-pushed the TalR/vcpkg_windows_ci branch from 80fd29d to 39cab21 Compare June 21, 2024 10:12

- Add quotes to cmake config command and prepare log on failure

b85e94a

- change to VCPKG_TRIPLET_DIR

vszakats reviewed Jun 27, 2024

View reviewed changes

Comment thread .github/workflows/windows.yml Outdated

remove '.' from cmake config command

7072d1f

talregev force-pushed the TalR/vcpkg_windows_ci branch from e650035 to 7072d1f Compare June 27, 2024 17:27

vszakats requested changes Jun 27, 2024

View reviewed changes

Comment thread .github/workflows/windows.yml Outdated

vszakats reviewed Jun 27, 2024

View reviewed changes

Comment thread .github/workflows/windows.yml

vszakats reviewed Jun 27, 2024

View reviewed changes

Comment thread .github/workflows/windows.yml

refactor according review comments

415f547

talregev requested a review from vszakats June 27, 2024 18:00

vszakats approved these changes Jun 30, 2024

View reviewed changes

talregev requested a review from bagder June 30, 2024 16:08

bagder reviewed Jun 30, 2024

View reviewed changes

vszakats closed this in e26cbe2 Jun 30, 2024

vszakats added the cmake label Jun 30, 2024

talregev deleted the TalR/vcpkg_windows_ci branch June 30, 2024 21:17

vszakats mentioned this pull request Jul 7, 2024

GHA: improve vcpkg cache, add BoringSSL ECH and LibreSSL MSVC jobs #14090

Closed

3 tasks

vszakats mentioned this pull request Jul 7, 2024

GHA/windows: usability improvements #14116

Closed

vszakats added a commit to vszakats/curl that referenced this pull request Jul 12, 2024

GHA/windows: re-add gsasl

e336aec

Follow-up to f99c08d curl#14090 Follow-up to e26cbe2 curl#13979 Closes #xxxxx

vszakats mentioned this pull request Jul 12, 2024

GHA/windows: re-add gsasl to MSVC jobs #14170

Closed

vszakats added a commit that referenced this pull request Jul 12, 2024

GHA/windows: re-add gsasl to MSVC jobs

1448218

Now that the package reached the CI runner image. Follow-up to f99c08d #14090 Follow-up to e26cbe2 #13979 Closes #14170

vszakats mentioned this pull request Mar 21, 2025

runtests: fix test key format for libssh2 WinCNG (and others) #16781

Closed

Uh oh!

Conversation

talregev commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

talregev commented Jun 21, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

talregev commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

vszakats commented Jun 21, 2024

Uh oh!

talregev commented Jun 21, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

talregev commented Jun 28, 2024

Uh oh!

talregev commented Jun 30, 2024

Uh oh!

talregev commented Jun 30, 2024

Uh oh!

bagder left a comment

Choose a reason for hiding this comment

Uh oh!

vszakats commented Jun 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

talregev commented Jun 21, 2024 •

edited

Loading

talregev commented Jun 21, 2024 •

edited

Loading