Skip to content

Commit

Permalink
Use the new Azure SDK for C++. (#3910)
Browse files Browse the repository at this point in the history
To communicate with Azure Storage we have been using the [Azure C++ Storage Client Lite](https://github.com/Azure/azure-storage-cpplite) which is no longer in active development. This PR switches the Azure VFS to use the new [Azure SDK for C++](https://github.com/Azure/azure-sdk-for-cpp/). The dependency to the old SDK is removed.

The migration is mostly a straightforward change from the old SDK calls to the new ones. The new SDK does not use futures, simplifying the code. As a bonus, we now pass more detailed error messages. Some stuff around configuration [will now work differently](https://github.com/Azure/azure-sdk-for-cpp/blob/azure-storage-blobs_12.6.2/sdk/storage/MigrationGuide.md), and I added relevant warnings.

[SC-21996](https://app.shortcut.com/tiledb-inc/story/21996/)
[SC-25039](https://app.shortcut.com/tiledb-inc/story/25039/)
[SC-25045](https://app.shortcut.com/tiledb-inc/story/25045/)
[SC-25046](https://app.shortcut.com/tiledb-inc/story/25046/)

---
TYPE: IMPROVEMENT
DESC: Use the new Azure SDK for C++.
  • Loading branch information
teo-tsirpanis committed Apr 23, 2023
1 parent 9910141 commit 969dd1c
Show file tree
Hide file tree
Showing 36 changed files with 444 additions and 1,473 deletions.
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,3 @@ build
builds
scripts
!scripts/capnpupdate.sh
.git
3 changes: 3 additions & 0 deletions .github/workflows/build-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,9 @@ jobs:
- name: Prepare git
run: git config --global core.autocrlf false
- uses: actions/checkout@v3
with:
submodules: true
fetch-depth: 0
- name: core tiledb windows build
run: |
$ErrorView = "NormalView"
Expand Down
196 changes: 0 additions & 196 deletions cmake/Modules/FindAzureSDK_EP.cmake

This file was deleted.

8 changes: 4 additions & 4 deletions cmake/Options/TileDBToolchain.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
# TileDB Toolchain Setup
############################################################

# Only enable vcpkg on GCS builds for now
if (NOT TILEDB_VCPKG AND NOT TILEDB_GCS)
# Only enable vcpkg on Azure and GCS builds for now
if (NOT TILEDB_VCPKG AND NOT (TILEDB_AZURE OR TILEDB_GCS))
return()
endif()

# For testing we're using --enable-gcs
if(TILEDB_GCS AND NOT TILEDB_VCPKG)
# For testing we're using --enable-gcs or --enable-azure
if((TILEDB_AZURE OR TILEDB_GCS) AND NOT TILEDB_VCPKG)
set(TILEDB_VCPKG ON)
endif()

Expand Down
6 changes: 1 addition & 5 deletions cmake/TileDB-Superbuild.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ if (NOT WIN32)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindOpenSSL_EP.cmake)
endif()

if (TILEDB_S3 OR TILEDB_AZURE OR TILEDB_GCS OR TILEDB_SERIALIZATION)
if (TILEDB_S3 OR TILEDB_GCS OR TILEDB_SERIALIZATION)
# Need libcurl either with S3 or serialization support.
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindCurl_EP.cmake)
endif()
Expand All @@ -130,10 +130,6 @@ if (TILEDB_S3)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindAWSSDK_EP.cmake)
endif()

if (TILEDB_AZURE)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindAzureSDK_EP.cmake)
endif()

if (TILEDB_GCS)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindGCSSDK_EP.cmake)
endif()
Expand Down
14 changes: 7 additions & 7 deletions cmake/inputs/patches/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,23 @@ TileDB dependencies must be built against source packages of released software v
To simplify patchset maintenance, we may also fork a specific repository version in
the TileDB-Inc organization.

Several illustrative examples, using the Azure CPP Lite dependency:
Several illustrative examples, using the AWS SDK dependency:
- applying a patchset to a repo fork:

```
git clone https://github.com/Azure/azure-storage-cpplite
cd azure-storage-cpplite
git am /path/to/TileDB/cmake/inputs/patches/ep_azure/v0.3.0-patchset.patch
git clone https://github.com/aws/aws-sdk-cpp.git
cd aws-sdk-cpp
git am /path/to/TileDB/cmake/inputs/patches/ep_awssdk/awsccommon.patch
```

- re-creating the patchset from updated fork:

```
cd azure-storage-cpplite
cd aws-sdk-cpp
# commit or cherry-pick changes
git format-patch v0.3.0 --stdout > /path/to/TileDB/cmake/inputs/patches/ep_azure/v0.3.0-patchset.patch
git format-patch v0.3.0 --stdout > /path/to/TileDB/cmake/inputs/patches/ep_awssdk/awsccommon.patch
# git add, commit, push to TileDB-Inc fork
```

Note that the target branch above is for illustration only. The target version should match the version
of the dependency used in the `cmake/Modules/Find<DEP>.cmake` file.
of the dependency used in the `cmake/Modules/Find<DEP>.cmake` file.
43 changes: 0 additions & 43 deletions cmake/inputs/patches/ep_azuresdk/base64.cpp.patch

This file was deleted.

Loading

0 comments on commit 969dd1c

Please sign in to comment.