Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add runtime detection for CA Certs on Linux #1393

Merged
merged 3 commits into from Oct 4, 2019

Conversation

@Shelnutt2
Copy link
Member

Shelnutt2 commented Oct 2, 2019

This solves a major issue when we build libcurl as part of a super build on Linux. Curl at configuration/compile time detects the systems CA bundle path, and then hard codes this into the compiled lib. This causes issues when libtiledb has a statically linked libcurl and is copied between
linux systems (i.e pip or maven). Often then the hard coded CA bundle path is no longer valid and curl fails to handle SSL.

The solution is to at runtime try to detect the location of the CA cert bundle for the host os.

Currently the CA cert issues only exists on linux, so we only do runtime detection for linux.

I have left adding an config override for the REST server to another PR as #1392 is reworking on the config is handled internally.

[ch814]

@Shelnutt2 Shelnutt2 requested a review from tdenniston Oct 2, 2019
@Shelnutt2 Shelnutt2 force-pushed the ss/add-runtime-curl-ca-cert-handling branch from ecabd36 to 0f925f9 Oct 2, 2019
tiledb/sm/global_state/libcurl_state.cc Outdated Show resolved Hide resolved
namespace https {
std::string find_ca_certs_linux(const tiledb::sm::VFS& vfs) {
// Check ever cert file location to see if the certificate exists
for (std::string cert : constants::cert_files_linux) {

This comment has been minimized.

Copy link
@tdenniston

tdenniston Oct 2, 2019

Contributor

Do for (const auto& cert : ...) instead

tiledb/sm/misc/utils.cc Outdated Show resolved Hide resolved
tiledb/sm/misc/utils.h Outdated Show resolved Hide resolved
@tdenniston tdenniston added this to the 1.7.0 milestone Oct 2, 2019
@tdenniston tdenniston added the s3 label Oct 2, 2019
tiledb/sm/misc/constants.cc Outdated Show resolved Hide resolved
@Shelnutt2 Shelnutt2 force-pushed the ss/add-runtime-curl-ca-cert-handling branch from 0f925f9 to b278909 Oct 2, 2019
@Shelnutt2 Shelnutt2 requested review from tdenniston and ihnorton Oct 2, 2019
@Shelnutt2 Shelnutt2 force-pushed the ss/add-runtime-curl-ca-cert-handling branch 2 times, most recently from 7ab3be5 to c60d268 Oct 2, 2019
Copy link
Contributor

tdenniston left a comment

LGTM

// for each s3/rest call as appropriate
VFS vfs;
vfs.init(config_.vfs_params());
cert_file_ = utils::https::find_ca_certs_linux(vfs);

This comment has been minimized.

Copy link
@ihnorton

ihnorton Oct 3, 2019

Member

don't let it slow the PR down, but this seems like something to log if/when that is feasible.

This comment has been minimized.

Copy link
@Shelnutt2

Shelnutt2 Oct 3, 2019

Author Member

@tdenniston do we have a way to log debug messages? I checked and only saw LOG_ERROR and LOG_STATUS.

This comment has been minimized.

Copy link
@tdenniston

tdenniston Oct 3, 2019

Contributor

No we don't -- in general our error reporting and logging needs an overhaul.

@Shelnutt2 Shelnutt2 force-pushed the ss/add-runtime-curl-ca-cert-handling branch 3 times, most recently from 907a781 to a5dcecc Oct 3, 2019
@ihnorton ihnorton force-pushed the ss/add-runtime-curl-ca-cert-handling branch from bd78af9 to 6492456 Oct 3, 2019
Shelnutt2 and others added 3 commits Oct 2, 2019
This solves a major issue when we build libcurl as part of a super
build. Curl at configuration/compile time detects the systems CA bundle
path, and then hard codes this into the compiled lib. This causes issues
when libtiledb has a statically linked libcurl and is copied between
linux systems (i.e pip or maven). Often then the hard coded CA bundle
path is no longer valid and curl fails to handle SSL.

The solution is to at runtime try to detect the location of the CA cert
bundle for the host os.

Currently the CA cert issues only exists on linux, so we only do runtime
detection for linux.

[ch814]
In the S3  case, vfs->init calls GetGlobalState, so it must be initialized first.
- After the previous patch to change initialization order, if GlobalState::init fails
  then the StorageManager may be left in a partially initialized state. If the
  StorageManager destructor is then called to clean up, we must avoid calling
  VFS operations.

  Bug description:
  In the TBB tests, we try initializing TBB with thread_count = -3. This
  properly fails, in the GlobalState::init call, leading to destruction of the
  Ctx object, here:
  - https://github.com/TileDB-Inc/TileDB/blob/1ff92bda90921df0ab6e1a44dec88dd5d26bf998/tiledb/sm/c_api/tiledb.cc#L1113-L1114
  https://github.com/TileDB-Inc/TileDB/blob/dev/tiledb/sm/c_api/tiledb.cc#L1114

  which then calls the StorageManager destructor, which makes several
  calls on the `vfs_` object:

  - https://github.com/TileDB-Inc/TileDB/blob/b175d59bcea9ca243e060445bbd14d308681a155/tiledb/sm/storage_manager/storage_manager.cc#L77-L98

  These calls fail, because with the patched initialization order in the
  previous commit, VFS has not been assigned or initialized and is
  nullptr.
@ihnorton ihnorton force-pushed the ss/add-runtime-curl-ca-cert-handling branch from 6492456 to bf9aa0c Oct 4, 2019
@Shelnutt2 Shelnutt2 merged commit 4e4146d into dev Oct 4, 2019
11 checks passed
11 checks passed
TileDB-Inc.TileDB Build #20191004.5 succeeded
Details
TileDB-Inc.TileDB (Windows VS2015) Windows VS2015 succeeded
Details
TileDB-Inc.TileDB (Windows VS2017) Windows VS2017 succeeded
Details
TileDB-Inc.TileDB (linux) linux succeeded
Details
TileDB-Inc.TileDB (linux_asan) linux_asan succeeded
Details
TileDB-Inc.TileDB (linux_hdfs) linux_hdfs succeeded
Details
TileDB-Inc.TileDB (linux_s3) linux_s3 succeeded
Details
TileDB-Inc.TileDB (linux_serialization) linux_serialization succeeded
Details
TileDB-Inc.TileDB (macOS) macOS succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.