[ignition-msgs] Fix flaky CI tests for ignition-msgs ports #14464

traversaro · 2020-11-08T15:15:22Z

We fix the flaky CI tests by not adding anymore the bin and debug/bin directories to the PATH, but rather by making sure that VCPKG_APPLOCAL_DEPS option is correctly propagated.

Describe the pull request

What does your PR fix? Fixes the problem mentioned in [ignition] Updates on various pkgs 🤖 #14324 (comment), not sure if there is a corresponding issue.
Which triplets are supported/not supported? Have you updated the CI baseline? The PR should not change the supported triplets.
Does your PR follow the maintainer guide? Yes

cc @BillyONeal

We fix the flaky CI tests by not adding anymore the bin and debug/bin directories to the PATH, but rather by making sure that VCPKG_APPLOCAL_DEPS option is correctly propagated.

traversaro · 2020-11-08T16:59:41Z

As I needed to modify the vcpkg_configure_cmake script that is used by basically every CMake port, I imagine we will have the usual endless CI and spurious CI failures of those cases. :D

JackBoosY · 2020-11-09T02:40:03Z

Nicole's PR #14399 also tried to fix this issue.

traversaro · 2020-11-09T06:55:46Z

Nicole's PR #14399 also tried to fix this issue.

To clarify, this is a cleanup that should avoid problems of this kind also in the future.

traversaro · 2020-11-09T06:59:05Z

The osgearth failure is due to this error:

D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\Tracker(23): fatal error C1083: Cannot open include file: 'osgEarth/ThreadingUtils': No such file or directory
[377/606] "C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DCURL_STATICLIB -DOSGEARTH_HAVE_GEOS -DOSGEARTH_HAVE_MBTILES -DOSGEARTH_HAVE_MVT -DOSGEARTH_HAVE_SQLITE3 -DOSGEARTH_LIBRARY -DTIXML_USE_STL -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D__STDC_FORMAT_MACROS -DosgEarth_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src -Isrc\osgEarth /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarth\CMakeFiles\osgEarth.dir\PlaceNode.cpp.obj /Fdsrc\osgEarth\CMakeFiles\osgEarth.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarth\PlaceNode.cpp
[378/606] "C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DCURL_STATICLIB -DOSGEARTH_HAVE_GEOS -DOSGEARTH_HAVE_MBTILES -DOSGEARTH_HAVE_MVT -DOSGEARTH_HAVE_SQLITE3 -DOSGEARTH_LIBRARY -DTIXML_USE_STL -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D__STDC_FORMAT_MACROS -DosgEarth_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src -Isrc\osgEarth /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarth\CMakeFiles\osgEarth.dir\LocalGeometryNode.cpp.obj /Fdsrc\osgEarth\CMakeFiles\osgEarth.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarth\LocalGeometryNode.cpp
[379/606] "C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DOSGEARTH_HAVE_MBTILES -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -Dosgdb_osgearth_cache_leveldb_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\LevelDBCacheBin.cpp.obj /Fdsrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\LevelDBCacheBin.cpp
FAILED: src/osgEarthDrivers/cache_leveldb/CMakeFiles/osgdb_osgearth_cache_leveldb.dir/LevelDBCacheBin.cpp.obj 
"C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DOSGEARTH_HAVE_MBTILES -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -Dosgdb_osgearth_cache_leveldb_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\LevelDBCacheBin.cpp.obj /Fdsrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\LevelDBCacheBin.cpp
D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\Tracker(23): fatal error C1083: Cannot open include file: 'osgEarth/ThreadingUtils': No such file or directory
[380/606] "C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DCURL_STATICLIB -DOSGEARTH_HAVE_GEOS -DOSGEARTH_HAVE_MBTILES -DOSGEARTH_HAVE_MVT -DOSGEARTH_HAVE_SQLITE3 -DOSGEARTH_LIBRARY -DTIXML_USE_STL -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D__STDC_FORMAT_MACROS -DosgEarth_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src -Isrc\osgEarth /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarth\CMakeFiles\osgEarth.dir\ModelNode.cpp.obj /Fdsrc\osgEarth\CMakeFiles\osgEarth.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarth\ModelNode.cpp
[381/606] "C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DCURL_STATICLIB -DOSGEARTH_HAVE_GEOS -DOSGEARTH_HAVE_MBTILES -DOSGEARTH_HAVE_MVT -DOSGEARTH_HAVE_SQLITE3 -DOSGEARTH_LIBRARY -DTIXML_USE_STL -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D__STDC_FORMAT_MACROS -DosgEarth_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src -Isrc\osgEarth /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarth\CMakeFiles\osgEarth.dir\FeatureNode.cpp.obj /Fdsrc\osgEarth\CMakeFiles\osgEarth.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarth\FeatureNode.cpp
[382/606] "C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DOSGEARTH_HAVE_MBTILES -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -Dosgdb_osgearth_cache_leveldb_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\LevelDBCacheDriver.cpp.obj /Fdsrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\LevelDBCacheDriver.cpp
FAILED: src/osgEarthDrivers/cache_leveldb/CMakeFiles/osgdb_osgearth_cache_leveldb.dir/LevelDBCacheDriver.cpp.obj 
"C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DOSGEARTH_HAVE_MBTILES -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -Dosgdb_osgearth_cache_leveldb_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\LevelDBCacheDriver.cpp.obj /Fdsrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\LevelDBCacheDriver.cpp
D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\Tracker(23): fatal error C1083: Cannot open include file: 'osgEarth/ThreadingUtils': No such file or directory

It does not seem to be directly related to this modification. I started #14474 to check if the ports fails also in the current master.

traversaro · 2020-11-09T07:25:43Z

I started #14474 to check if the ports fails also in the current master.

Unfortunately #14474 worked fine, so either the osqearth failure is non-deterministic, or it is actually related to the change of the PR. Another possibility is that the failure is due to the rebuild of one of osgeart dependencies.

BillyONeal · 2020-11-10T00:23:39Z

osgearth has also been sporadically failing; I suspect they have some form of races in their build. As such I don't think we should block merging over that.

However, I have a concern with forcing VCPKG_APPLOCAL_DEPS on for the tree -- that's a huge waste of copying time for most ports since it triggers a copy into the buildtrees directory which is then immediately deleted.

JackBoosY · 2020-11-10T02:07:21Z

D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\Tracker(23): fatal error C1083: Cannot open include file: 'osgEarth/ThreadingUtils': No such file or directory

command:

"C:\PROGRA~2\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe"   /TP -DOSGEARTH_HAVE_MBTILES -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -Dosgdb_osgearth_cache_leveldb_EXPORTS -Ibuild_include -ID:\installed\x64-windows\include -ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP  /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /showIncludes /Fosrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\LevelDBCache.cpp.obj /Fdsrc\osgEarthDrivers\cache_leveldb\CMakeFiles\osgdb_osgearth_cache_leveldb.dir\ /FS -c D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarthDrivers\cache_leveldb\LevelDBCache.cpp

Include dirs:
-ID:\installed\x64-windows\include
-ID:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src

And some warnings:

D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarth\GeoMath.cpp(1): warning C4828: The file contains a character starting at offset 0x1892 that is illegal in the current source character set (codepage 65001).
D:\buildtrees\osgearth\src\18c0ed88a1-76bf0cae95.clean\src\osgEarth\GeoMath.cpp(1): warning C4828: The file contains a character starting at offset 0x18b1 that is illegal in the current source character set (codepage 65001).

traversaro · 2020-11-10T06:13:55Z

osgearth has also been sporadically failing; I suspect they have some form of races in their build. As such I don't think we should block merging over that.

However, I have a concern with forcing VCPKG_APPLOCAL_DEPS on for the tree -- that's a huge waste of copying time for most ports since it triggers a copy into the buildtrees directory which is then immediately deleted.

That is the reason why it is enabled only for ignition-msgs* ports and not for all the ports.

BillyONeal

That is the reason why it is enabled only for ignition-msgs* ports and not for all the ports.

Ah, derp, I somehow read the default you set as backwards to what you actually set 🤣

traversaro · 2020-11-10T07:48:48Z

That is the reason why it is enabled only for ignition-msgs* ports and not for all the ports.

Ah, derp, I somehow read the default you set as backwards to what you actually set 🤣

No problem. I think that in general having an opt-in VCPKG_APPLOCAL_DEPS set to ON is a good idea for all the projects that need to run a code generator as part of their build.

strega-nil · 2020-11-12T00:38:25Z

I'll merge after CI runs through.

strega-nil · 2020-11-12T20:39:35Z

/azp run

azure-pipelines · 2020-11-12T20:39:43Z

Azure Pipelines successfully started running 1 pipeline(s).

strega-nil · 2020-11-13T21:36:24Z

/azp run

azure-pipelines · 2020-11-13T21:37:19Z

Azure Pipelines successfully started running 1 pipeline(s).

strega-nil · 2020-11-16T20:56:56Z

/azp run

azure-pipelines · 2020-11-16T20:57:13Z

Azure Pipelines successfully started running 1 pipeline(s).

Neumann-A · 2020-11-17T21:40:08Z

No problem. I think that in general having an opt-in VCPKG_APPLOCAL_DEPS set to ON is a good idea for all the projects that need to run a code generator as part of their build.

isn't that why ADD_BIN_TO_PATH exists?

traversaro · 2020-11-17T22:02:25Z

No problem. I think that in general having an opt-in VCPKG_APPLOCAL_DEPS set to ON is a good idea for all the projects that need to run a code generator as part of their build.

isn't that why ADD_BIN_TO_PATH exists?

Good catch! Yes, I think that would also be sufficient, and will not require the crazy luck of being able to do a full re-build of all the cmake-based ports without any spurious failures, that is apparently really unlikely. I think I can use that one. Given that the path is modified only during the build phase, this should also avoid all the problems of finding the clang as the compiler during the configure phase.

strega-nil · 2020-11-18T21:05:50Z

Waiting for the ADD_BIN_TO_PATH change.

traversaro · 2020-11-23T18:57:30Z

Waiting for the ADD_BIN_TO_PATH change.

Done in #14730 .

traversaro added 2 commits November 8, 2020 16:08

Fix flaky CI tests for ignition-msgs* ports

47c74a6

We fix the flaky CI tests by not adding anymore the bin and debug/bin directories to the PATH, but rather by making sure that VCPKG_APPLOCAL_DEPS option is correctly propagated.

Also fix ignition-msgs1

e3a3b63

NancyLi1013 assigned JackBoosY Nov 9, 2020

traversaro mentioned this pull request Nov 9, 2020

[DO NOT MERGE] Check if osgearth builds fine in current CI #14474

Closed

JackBoosY added the category:port-bug The issue is with a library, which is something the port should already support label Nov 9, 2020

JackBoosY approved these changes Nov 9, 2020

View reviewed changes

BillyONeal approved these changes Nov 10, 2020

View reviewed changes

BillyONeal added the info:reviewed Pull Request changes follow basic guidelines label Nov 10, 2020

traversaro mentioned this pull request Nov 10, 2020

Add robotology software ports to this repository robotology/robotology-vcpkg-ports#15

Closed

Merge remote-tracking branch 'origin/master' into HEAD

37bdc8c

traversaro mentioned this pull request Nov 13, 2020

[ignition-msgs1] cmake dev warning in portfile #14565

Closed

JackBoosY linked an issue Nov 16, 2020 that may be closed by this pull request

[ignition-msgs1] cmake dev warning in portfile #14565

Closed

strega-nil removed the info:reviewed Pull Request changes follow basic guidelines label Nov 18, 2020

JackBoosY added the requires:author-response label Nov 19, 2020

traversaro mentioned this pull request Nov 23, 2020

[ignition-msgs*] Fix flaky CI tests for ignition-msgs* ports (take 2) #14730

Merged

traversaro closed this Nov 23, 2020

traversaro deleted the fix-flaky-tests-ignition-msgs branch November 23, 2020 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ignition-msgs] Fix flaky CI tests for ignition-msgs ports #14464

[ignition-msgs] Fix flaky CI tests for ignition-msgs ports #14464

traversaro commented Nov 8, 2020

traversaro commented Nov 8, 2020

JackBoosY commented Nov 9, 2020

traversaro commented Nov 9, 2020

traversaro commented Nov 9, 2020 •

edited

Loading

traversaro commented Nov 9, 2020

BillyONeal commented Nov 10, 2020

JackBoosY commented Nov 10, 2020 •

edited

Loading

traversaro commented Nov 10, 2020

BillyONeal left a comment

traversaro commented Nov 10, 2020

strega-nil commented Nov 12, 2020

strega-nil commented Nov 12, 2020

azure-pipelines bot commented Nov 12, 2020

strega-nil commented Nov 13, 2020

azure-pipelines bot commented Nov 13, 2020

strega-nil commented Nov 16, 2020

azure-pipelines bot commented Nov 16, 2020

Neumann-A commented Nov 17, 2020

traversaro commented Nov 17, 2020 •

edited

Loading

strega-nil commented Nov 18, 2020

traversaro commented Nov 23, 2020

[ignition-msgs*] Fix flaky CI tests for ignition-msgs* ports #14464

[ignition-msgs*] Fix flaky CI tests for ignition-msgs* ports #14464

Conversation

traversaro commented Nov 8, 2020

traversaro commented Nov 8, 2020

JackBoosY commented Nov 9, 2020

traversaro commented Nov 9, 2020

traversaro commented Nov 9, 2020 • edited Loading

traversaro commented Nov 9, 2020

BillyONeal commented Nov 10, 2020

JackBoosY commented Nov 10, 2020 • edited Loading

traversaro commented Nov 10, 2020

BillyONeal left a comment

Choose a reason for hiding this comment

traversaro commented Nov 10, 2020

strega-nil commented Nov 12, 2020

strega-nil commented Nov 12, 2020

azure-pipelines bot commented Nov 12, 2020

strega-nil commented Nov 13, 2020

azure-pipelines bot commented Nov 13, 2020

strega-nil commented Nov 16, 2020

azure-pipelines bot commented Nov 16, 2020

Neumann-A commented Nov 17, 2020

traversaro commented Nov 17, 2020 • edited Loading

strega-nil commented Nov 18, 2020

traversaro commented Nov 23, 2020

[ignition-msgs] Fix flaky CI tests for ignition-msgs ports #14464

[ignition-msgs] Fix flaky CI tests for ignition-msgs ports #14464

traversaro commented Nov 9, 2020 •

edited

Loading

JackBoosY commented Nov 10, 2020 •

edited

Loading

traversaro commented Nov 17, 2020 •

edited

Loading