Skip to content

[C++] S3 Finalization functions need to exist even when S3 isn't enabled. #36974

@robertmaynard

Description

@robertmaynard

Describe the bug, including details regarding any error messages, version, and platform.

Before Arrow 12 you could have c++ consumers of Arrow conditional use S3 without knowing if the Arrow they are linking to has S3 support enabled.

 std::string s3_uri = "s3://.....?region=us-east-1";
 auto const no_s3_support = arrow::fs::FileSystemFromUri(s3_uri).status().IsNotImplemented();
 ....

But with Arrow 12 due to AWS SDK issues ( #33858 ) creation of an S3 filesystem
now requires an explicit call to fs::FinalizeS3() during shutdown of the application. Since the FinalizeS3 function is in the optionally compiled cpp/src/arrow/filesystem/s3fs.cc there is no easy way for a C++ user to determine if they can call FinalizeS3.

I believe that the correct solution is that Arrow needs some generalized shutdown API that is always provided. This API would be aware if S3 support is enabled and call the function when needed. Currently this can't be done by consumers since S3 support can't be safely determined by consumers:

The current solution to work around this, is that every C++ user/consumer needs to use dlopen ( or equivalant on windows ) on either libArrow or the root executable for static builds ( dlopen(NULL) ) and check if the FinalizeS3 functions exists.

Component(s)

C++

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions