[vcpkg] Fix more cases of semicolon mishandling in "scripts" - follow-up to PR #12926 #13968

jgehw · 2020-10-10T15:57:55Z

also fix semicolons in vcpgk_execute_required_process (follow-up of [execute_process] Don't strip embedded semicolons #12926)
add optional output and error variable to vcpgk_execute_required_process
apply fix to gdal port (currently quotes lists to prevent expansion)
this PR is a spin-off of PR More or less completely rewritten tensorflow-cc port #13028 to separate changes in vcpgk_execute_required_process

incorporate changes from microsoft:master

Revert "incorporate changes from microsoft:master"

…te_required_process_repeat

JackBoosY

Please bump the port version. See documentation.

…s from vcpkg_configure_cmake bump gdal port version remove semicolon-workaround from aws-sdk-cpp port

jgehw · 2020-10-15T15:08:03Z

I don't see any relation between remaining CI failures and code changes of this PR. Over the last 3 CI runs, errors appeared and disappeared without any code changes in this PR, just by re-running and/or merging master. In particular:

last but 2nd run:
x86-windows: ois osgearth
x64-windows: plib osgearth
x64-windows-static: openxr-loader qt5-tools
x64-uwp: vcpkg-gfortran protobuf
arm-windows: protobuf boost*
x64-linux: openxr-loader qt5-tools

last but 1st run:
x86-windows: ogre sqlcipher
x64-windows: ogre parmetis qt5-scxml qt5-sensors qt5-serialport sqlcipher
x64-windows-static: ogre openxr-loader
x64-linux: openxr-loader

last run:
x86-windows: sqlcipher
x64-windows: apr sqlcipher
x64-windows-static: ogre
x64-linux: apr

sqlcipher is currently broken also in master (Error: There are invalid field types in the CONTROL or manifest file of C:\work\tools\vcpkg\ports\sqlcipher\vcpkg.json). apr has transient error (SSL connection while downloading).

JackBoosY · 2020-10-16T06:45:02Z

sqlcipher:

couldn't read file ";TCLVERSION=90;ORIGINAL_SRC=D:\buildtrees\sqlcipher\src\v4.4.0-36c69edad7.clean;EXT_FEATURE_FLAGS=-DSQLITE_TEMP_STORE=2": no such file or directory

I think that's related to your changes.

JackBoosY · 2020-10-16T06:50:39Z

/azp run

azure-pipelines · 2020-10-16T06:50:51Z

Azure Pipelines successfully started running 1 pipeline(s).

…ecute_required_process has been fixed

jgehw · 2020-10-16T16:21:01Z

sqlcipher:

couldn't read file ";TCLVERSION=90;ORIGINAL_SRC=D:\buildtrees\sqlcipher\src\v4.4.0-36c69edad7.clean;EXT_FEATURE_FLAGS=-DSQLITE_TEMP_STORE=2": no such file or directory

I think that's related to your changes.

Sorry, you're right. It turned out that the error I was seeing on my machine (Error: There are invalid field types in the CONTROL or manifest file) was hiding the error you quoted. After I rebuilt vcpkg.exe, I was able to reproduce the issue. I'll push a fix.

jgehw · 2020-10-16T20:47:52Z

just re-running CI lead to these changes in CI failures:

previous run:
x86-windows: sqlcipher
x64-windows: apr sqlcipher
x64-windows-static: ogre
x64-linux: apr

after re-running:
x86-windows: sqlcipher osgearth
x64-windows: sqlcipher osgearth
x64-windows-static: plib ogre
arm-uwp: anyrpc

I then fixed sqlcipher to work wit my changes of this PR. Now all tests succeed. Thus, apr, ogre, osgearth, anyrpc were unrelated and transient issue.

Anyway, the main thing is: is working now :-) So, this PR is ready for final review and merge.

jgehw · 2020-10-20T16:14:58Z

Merging master caused these new CI failures:
x86-windows: akali, libpqxx, moos-core, osgearth
x64-windows-static: ogre, qt5-tools
arm64-windows: libpqxx

Looking through the logs, it's mostly some file or lib not found. I observed similar transient issues on my local machine, where the culprit was the anti-virus scanner locking a file while scanning and thus making it unavailable to the build process. As

ogre, osgearth, qt5-tools already had transient issues appearing and disappearing without code changes during the history of this PR (see above),
libpqxx was updated in master but I don't see any changes in the diff being affected by this PR,
akali, moos-core had no relevant code changes since previous merge of master,

can you please re-run all failed CI to see which of them are not transient errors?

ianier · 2020-10-20T17:10:18Z

Looking through the logs, it's mostly some file or lib not found. I observed similar transient issues on my local machine, where the culprit was the anti-virus scanner locking a file while scanning and thus making it unavailable to the build process.

As someone interested in getting this PR through to ultimately get tensorflow-cc back to life, I can see that this is much more painful than it should be. Please don’t give up, @jgehw, the world is counting on you :-)

BillyONeal · 2020-10-20T21:55:56Z

Unfortunately the nature of this PR causes the entire binary cache to be invalidated, so that PR build tries to build everything, so you're subject to the union of build flakiness of all ~1350 ports and their associated servers. We are always working to make things more reliable and things are better now than they were a few months ago, but still not "rock solid". @ras0219 / @ras0219-msft is working on caching package sources rather than just outputs which will help a lot since it will mean we don't break because some server is down.

jgehw · 2020-10-21T17:01:14Z

Today's CI failures:

libpqxx (x86-windows, x64-windows, arm64-windows) "dl.lib" not found - seems to be an issue with the build environment, it's compiling fine on my machine
qt5-multimedia (x86-windows, x64-windows, x64-windows-static) network issues while download -> transient error
freetds (x64-windows) permission denied on temp file -> transient error
ogre (x64-windows-static) compile error "definition of dllimport static data member not allowed" - not reproducable on my machine - either transient or specific to build environment

@BillyONeal @JackBoosY can you please re-run failed CIs? (I can't, insufficient permissions)

BillyONeal · 2020-10-22T21:58:10Z

We talked about this today and resolved that for "rebuild the world from contributors" changes like this we will merge them upon manual review that all the failures match a previous CI nightly build (currently https://dev.azure.com/vcpkg/public/_build?definitionId=29 )

BillyONeal · 2020-10-27T02:30:09Z

Well I was going to merge this anyway with potential failures but it passed now, yay! :D :D :D :D

BillyONeal · 2020-10-27T02:30:48Z

Thanks so much for your contribution and dealing with our PR / CI system issues :D

jgehw and others added 20 commits August 17, 2020 15:58

Merge pull request #1 from microsoft/master

d3121ce

incorporate changes from microsoft:master

Revert "incorporate changes from microsoft:master"

c743e49

Merge pull request #2 from jgehw/revert-1-master

2571bd1

Revert "incorporate changes from microsoft:master"

Revert "Revert "incorporate changes from microsoft:master""

98ecc0c

Merge branch 'master' of https://github.com/microsoft/vcpkg

ab8fdf7

Merge branch 'master' of https://github.com/microsoft/vcpkg

36143ac

Merge branch 'master' of https://github.com/microsoft/vcpkg

1866a82

Merge branch 'master' of https://github.com/microsoft/vcpkg

9451d33

Merge branch 'master' of https://github.com/microsoft/vcpkg

fbbde32

Merge branch 'master' of https://github.com/microsoft/vcpkg

cc3823d

Merge branch 'master' of https://github.com/microsoft/vcpkg

f7ee6c6

Merge branch 'master' of https://github.com/microsoft/vcpkg

37f79c6

Merge branch 'master' of https://github.com/microsoft/vcpkg into master

f0d1c89

Merge branch 'master' of https://github.com/microsoft/vcpkg into master

3df3b9b

Merge branch 'master' of https://github.com/microsoft/vcpkg into master

067323c

Merge branch 'master' of https://github.com/microsoft/vcpkg into master

b8e9523

Merge branch 'master' of https://github.com/microsoft/vcpkg

39312c7

Merge branch 'master' of https://github.com/microsoft/vcpkg into master

6384898

Merge branch 'master' of https://github.com/microsoft/vcpkg

d3e3c2a

follow-up to PR microsoft#12926

63c0a0f

jgehw mentioned this pull request Oct 10, 2020

More or less completely rewritten tensorflow-cc port #13028

Merged

Joachim Gehweiler added 2 commits October 11, 2020 14:29

remove list quoting for un-fixed execute_required_process

b4fd975

for sake of completeness apply PR microsoft#12926 also to vcpkg_execu…

63c2a24

…te_required_process_repeat

PhoebeHui assigned JackBoosY Oct 12, 2020

JackBoosY suggested changes Oct 12, 2020

View reviewed changes

JackBoosY added category:port-bug The issue is with a library, which is something the port should already support category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly requires:discussion labels Oct 12, 2020

remove semicolon-workaround for unfixed vcpkg_execute required_proces…

d50d45a

…s from vcpkg_configure_cmake bump gdal port version remove semicolon-workaround from aws-sdk-cpp port

jgehw added 2 commits October 15, 2020 16:33

Merge branch 'master' of https://github.com/microsoft/vcpkg into master

2ad839b

Merge branch 'master' into execute-process-pr-spinoff

da220ba

remove work-around (prevent list-expansion by quoting) after vcpkg_ex…

c9541e8

…ecute_required_process has been fixed

JackBoosY approved these changes Oct 19, 2020

View reviewed changes

Merge remote-tracking branch 'origin/master' into HEAD

b04939c

jgehw added 2 commits October 21, 2020 14:30

Merge branch 'master' of https://github.com/microsoft/vcpkg into master

f4e64da

Merge branch 'master' into execute-process-pr-spinoff

a09f30e

BillyONeal self-assigned this Oct 22, 2020

BillyONeal added 2 commits October 23, 2020 21:49

Merge remote-tracking branch 'origin/master' into HEAD

057fb9e

Merge remote-tracking branch 'origin/master' into HEAD

e25d3ad

BillyONeal changed the title ~~follow-up to PR #12926~~ [vcpkg] Fix more cases of semicolon mishandling in "scripts" - follow-up to PR #12926 Oct 26, 2020

BillyONeal merged commit 0ff75ac into microsoft:master Oct 27, 2020

jgehw deleted the execute-process-pr-spinoff branch October 27, 2020 12:04

PhoebeHui mentioned this pull request Nov 3, 2020

[magnum] build failure - actual issue is vcpkg_copy_tools.cmake #14329

Closed

jgehw mentioned this pull request Nov 4, 2020

[magnum] Removing quotes around tools list where vcpkg_copy_tools is involked #14383

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vcpkg] Fix more cases of semicolon mishandling in "scripts" - follow-up to PR #12926 #13968

[vcpkg] Fix more cases of semicolon mishandling in "scripts" - follow-up to PR #12926 #13968

jgehw commented Oct 10, 2020 •

edited

Loading

JackBoosY left a comment

jgehw commented Oct 15, 2020

JackBoosY commented Oct 16, 2020

JackBoosY commented Oct 16, 2020

azure-pipelines bot commented Oct 16, 2020

jgehw commented Oct 16, 2020

jgehw commented Oct 16, 2020

jgehw commented Oct 20, 2020

ianier commented Oct 20, 2020

BillyONeal commented Oct 20, 2020

jgehw commented Oct 21, 2020

BillyONeal commented Oct 22, 2020

BillyONeal commented Oct 27, 2020

BillyONeal commented Oct 27, 2020

[vcpkg] Fix more cases of semicolon mishandling in "scripts" - follow-up to PR #12926 #13968

[vcpkg] Fix more cases of semicolon mishandling in "scripts" - follow-up to PR #12926 #13968

Conversation

jgehw commented Oct 10, 2020 • edited Loading

JackBoosY left a comment

Choose a reason for hiding this comment

jgehw commented Oct 15, 2020

JackBoosY commented Oct 16, 2020

JackBoosY commented Oct 16, 2020

azure-pipelines bot commented Oct 16, 2020

jgehw commented Oct 16, 2020

jgehw commented Oct 16, 2020

jgehw commented Oct 20, 2020

ianier commented Oct 20, 2020

BillyONeal commented Oct 20, 2020

jgehw commented Oct 21, 2020

BillyONeal commented Oct 22, 2020

BillyONeal commented Oct 27, 2020

BillyONeal commented Oct 27, 2020

jgehw commented Oct 10, 2020 •

edited

Loading