Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++][CI] arrow-s3fs-test fails on some nightly jobs #36447

Closed
raulcd opened this issue Jul 4, 2023 · 4 comments
Closed

[C++][CI] arrow-s3fs-test fails on some nightly jobs #36447

raulcd opened this issue Jul 4, 2023 · 4 comments
Labels
Component: C++ Component: Continuous Integration Priority: Blocker Marks a blocker for the release Type: bug
Milestone
13.0.0

Comments

@raulcd
Copy link
Member

raulcd commented Jul 4, 2023

Describe the bug, including details regarding any error messages, version, and platform.

It seems that the arrow-s3fs-test fails with file exists for some of the tests:

Error: mkdir /tmp/s3fs-test-9kk8hwry/bucket/somefile/: file exists (*fs.PathError)

This is happening on:

The full error log:

75/108 Test  #69: arrow-s3fs-test ...........................***Failed   21.30 sec
Running arrow-s3fs-test, redirecting output into /build/cpp/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
/arrow/cpp/build-support/run-test.sh: line 88: 30269 Aborted                 (core dumped) $TEST_EXECUTABLE "$@" > $LOGFILE.raw 2>&1
Running main() from /home/buildozer/aports/main/gtest/src/googletest-14aa11db02d9851d957f93ef9fddb110c1aafdc6/googletest/src/gtest_main.cc
[==========] Running 62 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from S3OptionsTest
[ RUN      ] S3OptionsTest.FromUri
[       OK ] S3OptionsTest.FromUri (579 ms)
[ RUN      ] S3OptionsTest.FromAccessKey
[       OK ] S3OptionsTest.FromAccessKey (132 ms)
[ RUN      ] S3OptionsTest.FromAssumeRole
[       OK ] S3OptionsTest.FromAssumeRole (135 ms)
[----------] 3 tests from S3OptionsTest (847 ms total)

[----------] 4 tests from S3RegionResolutionTest
[ RUN      ] S3RegionResolutionTest.PublicBucket
[       OK ] S3RegionResolutionTest.PublicBucket (372 ms)
[ RUN      ] S3RegionResolutionTest.RestrictedBucket
[       OK ] S3RegionResolutionTest.RestrictedBucket (341 ms)
[ RUN      ] S3RegionResolutionTest.NonExistentBucket
[       OK ] S3RegionResolutionTest.NonExistentBucket (1490 ms)
[ RUN      ] S3RegionResolutionTest.InvalidBucketName
[       OK ] S3RegionResolutionTest.InvalidBucketName (135 ms)
[----------] 4 tests from S3RegionResolutionTest (2340 ms total)

[----------] 2 tests from S3FileSystemRegionTest
[ RUN      ] S3FileSystemRegionTest.Default
[       OK ] S3FileSystemRegionTest.Default (135 ms)
[ RUN      ] S3FileSystemRegionTest.EnvironmentVariable
[       OK ] S3FileSystemRegionTest.EnvironmentVariable (136 ms)
[----------] 2 tests from S3FileSystemRegionTest (271 ms total)

[----------] 1 test from TestMinioServer
[ RUN      ] TestMinioServer.Connect
[       OK ] TestMinioServer.Connect (706 ms)
[----------] 1 test from TestMinioServer (706 ms total)

[----------] 26 tests from TestS3FS
[ RUN      ] TestS3FS.GetFileInfoRoot
[       OK ] TestS3FS.GetFileInfoRoot (197 ms)
[ RUN      ] TestS3FS.GetFileInfoBucket
[       OK ] TestS3FS.GetFileInfoBucket (244 ms)
[ RUN      ] TestS3FS.GetFileInfoObject
[       OK ] TestS3FS.GetFileInfoObject (300 ms)
[ RUN      ] TestS3FS.GetFileInfoSelector
[       OK ] TestS3FS.GetFileInfoSelector (257 ms)
[ RUN      ] TestS3FS.GetFileInfoSelectorRecursive
[       OK ] TestS3FS.GetFileInfoSelectorRecursive (277 ms)
[ RUN      ] TestS3FS.GetFileInfoGenerator
[       OK ] TestS3FS.GetFileInfoGenerator (240 ms)
[ RUN      ] TestS3FS.CreateDir

API: PutObject(bucket=bucket, object=somefile/)
Time: 01:32:48 UTC 07/04/2023
DeploymentID: 010901b4-1aed-45ad-a0ee-e0af17423c04
RequestID: 176E861CC5143A4A
RemoteHost: 127.0.0.1
Host: 127.0.0.8:54373
UserAgent: aws-sdk-cpp/1.10.55 Linux/5.15.0-1040-azure x86_64 GCC/11.2.1
Error: mkdir /tmp/s3fs-test-9kk8hwry/bucket/somefile/: file exists (*fs.PathError)
       5: internal/logger/logger.go:278:logger.LogIf()
       4: cmd/fs-v1.go:1036:cmd.(*FSObjects).putObject()
       3: cmd/fs-v1.go:1012:cmd.(*FSObjects).PutObject()
       2: cmd/object-handlers.go:1830:cmd.objectAPIHandlers.PutObjectHandler()
       1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()

API: PutObject(bucket=bucket, object=somefile/)
Time: 01:32:48 UTC 07/04/2023
DeploymentID: 010901b4-1aed-45ad-a0ee-e0af17423c04
RequestID: 176E861CC5143A4A
RemoteHost: 127.0.0.1
Host: 127.0.0.8:54373
UserAgent: aws-sdk-cpp/1.10.55 Linux/5.15.0-1040-azure x86_64 GCC/11.2.1
Error: mkdir /tmp/s3fs-test-9kk8hwry/bucket/somefile/: file exists (*fs.PathError)
       5: internal/logger/logger.go:278:logger.LogIf()
       4: cmd/api-errors.go:2123:cmd.toAPIErrorCode()
       3: cmd/api-errors.go:2148:cmd.toAPIError()
       2: cmd/object-handlers.go:1832:cmd.objectAPIHandlers.PutObjectHandler()
       1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()
[       OK ] TestS3FS.CreateDir (335 ms)
[ RUN      ] TestS3FS.DeleteFile
[       OK ] TestS3FS.DeleteFile (243 ms)
[ RUN      ] TestS3FS.DeleteDir
[       OK ] TestS3FS.DeleteDir (294 ms)
[ RUN      ] TestS3FS.DeleteDirContents
[       OK ] TestS3FS.DeleteDirContents (278 ms)
[ RUN      ] TestS3FS.DeleteDirContentsAsync

API: PutObject(bucket=bucket, object=somefile/)
Time: 01:32:49 UTC 07/04/2023
DeploymentID: a9a8af82-9903-4a4b-9d24-8647ccf47604
RequestID: 176E861D0468B79A

API: CopyObject(bucket=s3fs-test-bucket, object=xxx/mno)
Time: 01:32:58 UTC 07/04/2023
DeploymentID: b07a70c3-9139-46de-abf3-6bdfc538bf1c
RequestID: 176E861F1AAC1E91
RemoteHost: 127.0.0.1
Host: 127.0.0.36:33197
UserAgent: aws-sdk-cpp/1.10.55 Linux/5.15.0-1040-azure x86_64 GCC/11.2.1
Error: file access denied (cmd.StorageErr)
       6: internal/logger/logger.go:278:logger.LogIf()
       5: cmd/fs-v1-helpers.go:360:cmd.fsRenameFile()
       4: cmd/fs-v1.go:1107:cmd.(*FSObjects).putObject()
       3: cmd/fs-v1.go:685:cmd.(*FSObjects).CopyObject()
       2: cmd/object-handlers.go:1501:cmd.objectAPIHandlers.CopyObjectHandler()
       1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()
[       OK ] TestS3FSGeneric.MoveFile (380 ms)
[ RUN      ] TestS3FSGeneric.MoveDir
/arrow/cpp/src/arrow/filesystem/test_util.cc:453: Skipped
Filesystem doesn't allow moving directories
[  SKIPPED ] TestS3FSGeneric.MoveDir (160 ms)
[ RUN      ] TestS3FSGeneric.CopyFile

API: CopyObject(bucket=s3fs-test-bucket, object=def/mno)
Time: 01:32:58 UTC 07/04/2023
DeploymentID: d53f2a07-9c5c-4a8e-aa2a-e6ba8713c057
RequestID: 176E861F397DC956
RemoteHost: 127.0.0.1
Host: 127.0.0.38:52109
UserAgent: aws-sdk-cpp/1.10.55 Linux/5.15.0-1040-azure x86_64 GCC/11.2.1
Error: file access denied (cmd.StorageErr)
       6: internal/logger/logger.go:278:logger.LogIf()
       5: cmd/fs-v1-helpers.go:360:cmd.fsRenameFile()
       4: cmd/fs-v1.go:1107:cmd.(*FSObjects).putObject()
       3: cmd/fs-v1.go:685:cmd.(*FSObjects).CopyObject()
       2: cmd/object-handlers.go:1501:cmd.objectAPIHandlers.CopyObjectHandler()
       1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()
[       OK ] TestS3FSGeneric.CopyFile (368 ms)
[ RUN      ] TestS3FSGeneric.GetFileInfo
[       OK ] TestS3FSGeneric.GetFileInfo (242 ms)
[ RUN      ] TestS3FSGeneric.GetFileInfoVector
[       OK ] TestS3FSGeneric.GetFileInfoVector (295 ms)
[ RUN      ] TestS3FSGeneric.GetFileInfoSelector
[       OK ] TestS3FSGeneric.GetFileInfoSelector (318 ms)
[ RUN      ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
[       OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (398 ms)
[ RUN      ] TestS3FSGeneric.GetFileInfoAsync
[       OK ] TestS3FSGeneric.GetFileInfoAsync (231 ms)
[ RUN      ] TestS3FSGeneric.GetFileInfoGenerator
[       OK ] TestS3FSGeneric.GetFileInfoGenerator (298 ms)
[ RUN      ] TestS3FSGeneric.OpenOutputStream
[       OK ] TestS3FSGeneric.OpenOutputStream (385 ms)
[ RUN      ] TestS3FSGeneric.OpenAppendStream
/arrow/cpp/src/arrow/filesystem/test_util.cc:932: Skipped
Filesystem doesn't allow file appends
[  SKIPPED ] TestS3FSGeneric.OpenAppendStream (215 ms)
[ RUN      ] TestS3FSGeneric.OpenInputStream
[       OK ] TestS3FSGeneric.OpenInputStream (288 ms)
[ RUN      ] TestS3FSGeneric.OpenInputStreamWithFileInfo
[       OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (280 ms)
[ RUN      ] TestS3FSGeneric.OpenInputStreamAsync
[       OK ] TestS3FSGeneric.OpenInputStreamAsync (253 ms)
[ RUN      ] TestS3FSGeneric.OpenInputFile
[       OK ] TestS3FSGeneric.OpenInputFile (265 ms)
[ RUN      ] TestS3FSGeneric.OpenInputFileWithFileInfo
[       OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (281 ms)
[ RUN      ] TestS3FSGeneric.OpenInputFileAsync
[       OK ] TestS3FSGeneric.OpenInputFileAsync (260 ms)
[ RUN      ] TestS3FSGeneric.SpecialChars
[       OK ] TestS3FSGeneric.SpecialChars (301 ms)
[----------] 26 tests from TestS3FSGeneric (8043 ms total)

[----------] Global test environment tear-down
[==========] 62 tests from 6 test suites ran. (21022 ms total)
[  PASSED  ] 60 tests.
[  SKIPPED ] 2 tests, listed below:
[  SKIPPED ] TestS3FSGeneric.MoveDir
[  SKIPPED ] TestS3FSGeneric.OpenAppendStream
pure virtual method called
terminate called without an active exception
/build/cpp/src/arrow/filesystem

Component(s)

C++, Continuous Integration
@raulcd
Copy link
Member Author

raulcd commented Jul 4, 2023

CC @zeroshade @pitrou

@raulcd raulcd added the Priority: Blocker Marks a blocker for the release label Jul 4, 2023
@raulcd raulcd added this to the 13.0.0 milestone Jul 4, 2023
@pitrou
Copy link
Member

pitrou commented Jul 4, 2023

The tests complete successfully. The actual error is this:

pure virtual method called
terminate called without an active exception

It will hopefully be fixed by #36442

@raulcd
Copy link
Member Author

raulcd commented Jul 4, 2023

I see, I saw all the error file exists and got confused. Thanks @pitrou !

@pitrou
Copy link
Member

pitrou commented Jul 4, 2023

Closing as duplicate of #36346.

@pitrou pitrou closed this as not planned Won't fix, can't repro, duplicate, stale Jul 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: C++ Component: Continuous Integration Priority: Blocker Marks a blocker for the release Type: bug
Projects
None yet
Milestone
13.0.0
Development

No branches or pull requests
2 participants
@raulcd @pitrou