-
Notifications
You must be signed in to change notification settings - Fork 184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
faketime tests stall when S3 support is enabled #4956
Comments
@teo-tsirpanis can you take a look? |
Will take a look tomorrow. |
@jdblischak do you have credentials for S3 configured? It is likely that because S3 is enabled, <opinion> |
No, and I'd prefer not to
That makes sense. Thanks for diagnosing the problem!
Can I pass that flag directly to CMake? Here is how I am currently invoking the TileDB tests: cmake --build build-libtiledb --target check --config Release
# Is this right?
cmake --build build-libtiledb --target check --config Release --vfs native |
It's not currently possible within CMake; you will have to manually run |
How can I manually build the tests in order to manually run them? docker run --rm -it ubuntu:22.04
# Setup
apt-get update
apt-get install --yes cmake curl g++ gcc git pkg-config tar unzip zip
git clone https://github.com/TileDB-Inc/TileDB.git
cd TileDB
git log -n 1 --oneline
## 474fc1ef5 (HEAD -> dev, origin/dev, origin/HEAD) Migrate APIs out of StorageManager: array_get_encryption. (#4950)
# with S3 support
cmake -B build-libtiledb-s3 \
-D TILEDB_WERROR=ON \
-D TILEDB_SERIALIZATION=ON \
-D CMAKE_BUILD_TYPE=Release \
-D VCPKG_TARGET_TRIPLET=x64-linux-release \
-D TILEDB_S3=ON
cmake --build build-libtiledb-s3 -j $(nproc) --config Release
./build-libtiledb/tiledb/test/tiledb_unit --vfs native
## bash: ./build-libtiledb/tiledb/test/tiledb_unit: No such file or directory I tried searching the CMake files, but didn't have much luck. I found where the custom target TileDB/cmake/TileDB-Superbuild.cmake Lines 176 to 182 in 474fc1e
I also tried the targets Lines 550 to 558 in 474fc1e
cmake --build build-libtiledb-s3 --target tests --config Release
## gmake: *** No rule to make target 'tests'. Stop.
cmake --build build-libtiledb-s3 --target ordinary_unit_tests --config Release
## gmake: *** No rule to make target 'ordinary_unit_tests'. Stop. |
CMake is being called in the TileDB/.github/workflows/unit-test-runs.yml Lines 56 to 64 in 7387605
|
@teo-tsirpanis Thanks for the explanation! I was able to build and execute the tests locally. However, there were 3 failed tests. Are these known failures? docker run --rm -it ubuntu:22.04
# Setup
apt-get update
apt-get install --yes cmake curl g++ gcc git pkg-config tar unzip zip
git clone https://github.com/TileDB-Inc/TileDB.git
cd TileDB
git log -n 1 --oneline
## 8de7d1ca4 (HEAD -> dev, origin/dev, origin/HEAD) Add v2_23_0 arrays to backward compatibility matrix. (#4965)
# with S3 support
cmake -B build-libtiledb-s3 \
-D TILEDB_WERROR=ON \
-D TILEDB_SERIALIZATION=ON \
-D CMAKE_BUILD_TYPE=Release \
-D VCPKG_TARGET_TRIPLET=x64-linux-release \
-D TILEDB_S3=ON
cmake --build build-libtiledb-s3 -j $(nproc) --config Release
# Build unit tests
make -C build-libtiledb-s3/tiledb tests -j $(nproc)
## [100%] Building CXX object test/CMakeFiles/tiledb_unit.dir/src/unit.cc.o
## [100%] Built target ordinary_unit_tests
## [100%] Built target all_unit_tests
## [100%] Linking CXX executable tiledb_unit
## [100%] Built target tiledb_unit
## [100%] Built target tests
# Run unit tests
./build-libtiledb-s3/tiledb/test/tiledb_unit --vfs native
## Randomness seeded to: 158965112
##
## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## tiledb_unit is a Catch2 v3.4.0 host application.
## Run with -? for options
##
## -------------------------------------------------------------------------------
## C API: Test vacuuming leaves array dir in a consistent state
## -------------------------------------------------------------------------------
## /TileDB/test/src/unit-capi-consolidation.cc:7378
## ...............................................................................
##
## /TileDB/test/src/unit-capi-consolidation.cc:4656: FAILED:
## REQUIRE( rc == (expect_fail ? (-1) : (0)) )
## with expansion:
## 0 == -1
##
## terminate called after throwing an instance of 'std::filesystem::__cxx11::filesystem_error'
## what(): filesystem error: cannot set permissions: No such file or directory [/TileDB/tiledb_test/test_consolidate_sparse_array/__fragments/__1715616656780_1715616656780_71deef232b74224ba6dfd2553ec29ece_21]
## /TileDB/test/src/unit-capi-consolidation.cc:4656: FAILED:
## {Unknown expression after the reported line}
## due to a fatal error condition:
## SIGABRT - Abort (abnormal termination) signal
##
## ===============================================================================
## test cases: 341 | 340 passed | 1 failed
## assertions: 3084887 | 3084885 passed | 2 failed
##
## Aborted |
Next I tried running the tests this way in my nightly setup. Without S3 support, the tests passed All tests passed (13941361 assertions in 1427 test cases) However, when I enabled S3 support, there were even more test failures than when I ran it locally above: test cases: 1434 | 1409 passed | 25 failed
assertions: 13863516 | 13863491 passed | 25 failed |
Apparently |
Ok, so it seems that currently it is not possible to run the test suite with |
Ok, for now I enabled S3 support in order to run the tiledbvcf-py tests and stopped running the libtiledb tests (jdblischak/centralized-tiledb-nightlies@3febc7b). Ideally in the future I can update this to run a subset of the tests that don't require S3 authentication |
x-ref https://app.shortcut.com/tiledb-inc/story/47373
In my nightly builds, if I build with S3 support (
TILEDB_S3=ON
), the faketime tests added in #4883 stall. They pass when S3 support is disabled.xref: jdblischak/centralized-tiledb-nightlies#5 (comment)
Below is a reproducible example I created in an Ubuntu Docker image. Note that I had to install more dependencies than those listed in the Prerequisites of
BUILD.md
.The text was updated successfully, but these errors were encountered: