Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the new Azure SDK for C++. #3910

Merged
merged 40 commits into from
Apr 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
dc7b102
Add dependency to the new Azure SDK via vcpkg and remove the old one.
teo-tsirpanis Mar 8, 2023
c2ac7f8
Port the Azure Blob Storage client construction to the new SDK.
teo-tsirpanis Feb 15, 2023
03279f6
Start porting some operations to the new SDK.
teo-tsirpanis Feb 16, 2023
abaa16b
Port `read` and `touch`.
teo-tsirpanis Feb 16, 2023
e4c895d
Port five more operations.
teo-tsirpanis Feb 17, 2023
6903d11
Port `copy_blob`.
teo-tsirpanis Feb 17, 2023
4dc60f7
Finish the porting with the write code.
teo-tsirpanis Feb 17, 2023
fc0c4de
Fix compile errors.
teo-tsirpanis Feb 17, 2023
009bc38
Avoid explicitly unlocking locks.
teo-tsirpanis Feb 17, 2023
6706d63
Remove unsupported config options and add relevant warnings.
teo-tsirpanis Feb 17, 2023
14640a6
Use TileDB unique pointers.
teo-tsirpanis Feb 20, 2023
1f2b208
Remove deprecated config options.
teo-tsirpanis Feb 22, 2023
801d903
Log events from the Azure SDK.
teo-tsirpanis Feb 20, 2023
2726b32
Use `make_unique` in one occasion.
teo-tsirpanis Feb 20, 2023
efea433
Remove `DenseArrayFx::set_small_memory_budget`.
teo-tsirpanis Feb 21, 2023
b6e8cbb
Fix formatting.
teo-tsirpanis Feb 22, 2023
d670870
Remove polling for blob/container propagation/deletion.
teo-tsirpanis Mar 3, 2023
12e5767
Bring back polling for blob propagation, for copying blobs.
teo-tsirpanis Mar 3, 2023
bfafa62
Update Azurite version on Linux.
teo-tsirpanis Mar 8, 2023
a90b100
Clone submodules on Windows CI.
teo-tsirpanis Mar 8, 2023
b688d7c
Do not depend on libcurl if we use Azure.
teo-tsirpanis Mar 10, 2023
f0c1421
Fix indentation in vcpkg.json.
teo-tsirpanis Mar 28, 2023
f5e9209
Fix documentation of the `vfs.azure.blob_endpoint` option.
teo-tsirpanis Mar 28, 2023
806d959
Avoid calling `getenv` many times.
teo-tsirpanis Mar 28, 2023
1c84903
Do not warn about the Azure blob endpoint's scheme when it is not spe…
teo-tsirpanis Mar 28, 2023
e81d9c6
Simplify checking for the SAS token config option and use nullptr.
teo-tsirpanis Mar 28, 2023
0b6f43e
Use the default Azure SDK retry policy.
teo-tsirpanis Mar 28, 2023
f8e5e79
Remove vcpkg port overlays for the old Azure SDK.
teo-tsirpanis Apr 4, 2023
f431d9b
Include the Azure SDK headers first.
teo-tsirpanis Apr 5, 2023
502ebbc
Use `PollUntilDone` to wait until the blob copying completes.
teo-tsirpanis Apr 10, 2023
a9b9bfd
Fix formatting and compile errors.
teo-tsirpanis Apr 10, 2023
26143c7
Build spdlog as a static library.
teo-tsirpanis Apr 11, 2023
699aaa4
Merge remote-tracking branch 'origin/dev' into teo-azure
ihnorton Apr 15, 2023
79261a2
Install missing autoconf/make required for build
ihnorton Apr 15, 2023
e2549e1
Fixes for Dockerfile build w/ vcpkg
ihnorton Apr 16, 2023
6d83532
Fix typo.
teo-tsirpanis Apr 18, 2023
b234248
Fix formatting.
teo-tsirpanis Apr 18, 2023
df260f3
Clean-up leftovers from removed config options.
teo-tsirpanis Apr 19, 2023
ff79e3e
Add config options for the Azure retry policies.
teo-tsirpanis Mar 28, 2023
7fbc6c6
Set the default value for `vfs.azure.max_retries` to 5; the same with…
teo-tsirpanis Apr 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,3 @@ build
builds
scripts
!scripts/capnpupdate.sh
.git
3 changes: 3 additions & 0 deletions .github/workflows/build-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,9 @@ jobs:
- name: Prepare git
run: git config --global core.autocrlf false
- uses: actions/checkout@v3
with:
submodules: true
fetch-depth: 0
- name: core tiledb windows build
run: |
$ErrorView = "NormalView"
Expand Down
196 changes: 0 additions & 196 deletions cmake/Modules/FindAzureSDK_EP.cmake

This file was deleted.

8 changes: 4 additions & 4 deletions cmake/Options/TileDBToolchain.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
# TileDB Toolchain Setup
############################################################

# Only enable vcpkg on GCS builds for now
if (NOT TILEDB_VCPKG AND NOT TILEDB_GCS)
# Only enable vcpkg on Azure and GCS builds for now
if (NOT TILEDB_VCPKG AND NOT (TILEDB_AZURE OR TILEDB_GCS))
return()
endif()

# For testing we're using --enable-gcs
if(TILEDB_GCS AND NOT TILEDB_VCPKG)
# For testing we're using --enable-gcs or --enable-azure
if((TILEDB_AZURE OR TILEDB_GCS) AND NOT TILEDB_VCPKG)
set(TILEDB_VCPKG ON)
endif()

Expand Down
6 changes: 1 addition & 5 deletions cmake/TileDB-Superbuild.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ if (NOT WIN32)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindOpenSSL_EP.cmake)
endif()

if (TILEDB_S3 OR TILEDB_AZURE OR TILEDB_GCS OR TILEDB_SERIALIZATION)
if (TILEDB_S3 OR TILEDB_GCS OR TILEDB_SERIALIZATION)
shaunrd0 marked this conversation as resolved.
Show resolved Hide resolved
# Need libcurl either with S3 or serialization support.
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindCurl_EP.cmake)
endif()
Expand All @@ -130,10 +130,6 @@ if (TILEDB_S3)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindAWSSDK_EP.cmake)
endif()

if (TILEDB_AZURE)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindAzureSDK_EP.cmake)
endif()

if (TILEDB_GCS)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/FindGCSSDK_EP.cmake)
endif()
Expand Down
14 changes: 7 additions & 7 deletions cmake/inputs/patches/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,23 @@ TileDB dependencies must be built against source packages of released software v
To simplify patchset maintenance, we may also fork a specific repository version in
the TileDB-Inc organization.

Several illustrative examples, using the Azure CPP Lite dependency:
Several illustrative examples, using the AWS SDK dependency:
- applying a patchset to a repo fork:

```
git clone https://github.com/Azure/azure-storage-cpplite
cd azure-storage-cpplite
git am /path/to/TileDB/cmake/inputs/patches/ep_azure/v0.3.0-patchset.patch
git clone https://github.com/aws/aws-sdk-cpp.git
cd aws-sdk-cpp
git am /path/to/TileDB/cmake/inputs/patches/ep_awssdk/awsccommon.patch
```

- re-creating the patchset from updated fork:

```
cd azure-storage-cpplite
cd aws-sdk-cpp
# commit or cherry-pick changes
git format-patch v0.3.0 --stdout > /path/to/TileDB/cmake/inputs/patches/ep_azure/v0.3.0-patchset.patch
git format-patch v0.3.0 --stdout > /path/to/TileDB/cmake/inputs/patches/ep_awssdk/awsccommon.patch
# git add, commit, push to TileDB-Inc fork
```

Note that the target branch above is for illustration only. The target version should match the version
of the dependency used in the `cmake/Modules/Find<DEP>.cmake` file.
of the dependency used in the `cmake/Modules/Find<DEP>.cmake` file.
43 changes: 0 additions & 43 deletions cmake/inputs/patches/ep_azuresdk/base64.cpp.patch

This file was deleted.

Loading