Update the GCS SDK to its latest version. #4031

teo-tsirpanis · 2023-04-10T07:59:37Z

A subset of #4008, let's see what CI thinks.

TYPE: IMPROVEMENT
DESC: Update the Google Cloud Storage SDK to its latest version.

Using the old version 1.22.0 will no longer compile.

teo-tsirpanis · 2023-04-10T12:57:09Z

SC-27301

shortcut-integration · 2023-04-10T12:57:12Z

This pull request has been linked to Shortcut Story #27301: Update to the latest version of the GCS SDK..

ihnorton

I think we'll need to make the use of the new zero-copy InsertObject API conditional on the version available at compile-time for now. Moving to GCS 2.9 for 2.16 is blocked by the conda-forge package update for 2.9: conda-forge/google-cloud-cpp-feedstock#129, which is in turn blocked by https://github.com/conda/conda-build/issues/4842.

teo-tsirpanis · 2023-04-11T11:07:27Z

I don't know how to do that with vcpkg. Better wait until the conda-forge issue is resolved. 2.6.0 is better than 1.11.0.

ihnorton · 2023-04-11T12:26:25Z

I don't know how to do that with vcpkg.

You can use >= version specifier.

https://learn.microsoft.com/en-us/vcpkg/users/examples/versioning.getting-started#version-1

teo-tsirpanis · 2023-04-18T22:10:43Z

How about we close this in favor of #4008? I managed to make CI green.

…loud-cpp.

teo-tsirpanis · 2023-04-20T12:14:23Z

Hmm, copying the ports to our own folder causes failures. 🤔

davisp

+1 The port update matches what I get locally. Everything else looks trivial.

ihnorton

Thanks. Fixes sc-34943.

teo-tsirpanis · 2023-10-04T11:49:09Z

CI fails because it runs out of space. 😭

teo-tsirpanis · 2023-10-04T16:33:39Z

Maybe we could run a git clean -xdf in the vcpkg directory after we configure? 🤔

ihnorton · 2023-10-04T18:06:55Z

We are building abseil in both release and debug mode (along with everything else):

2023-10-04T15:55:29.2972334Z -- Fixing pkgconfig file: /home/runner/work/TileDB/TileDB/external/vcpkg/packages/abseil_x64-linux/lib/pkgconfig/absl_variant.pc
2023-10-04T15:55:30.3111449Z -- Fixing pkgconfig file: /home/runner/work/TileDB/TileDB/external/vcpkg/packages/abseil_x64-linux/debug/lib/pkgconfig/absl_absl_check.pc
2023-10-04T15:55:30.3115761Z -- Fixing pkgconfig file: /home/runner/work/TileDB/TileDB/external/vcpkg/packages/abseil_x64-linux/debug/lib/pkgconfig/absl_absl_log.pc
2023-10-04T15:55:30.3121185Z -- Fixing pkgconfig file: /home/runner/work/TileDB/TileDB/external/vcpkg/packages/abseil_x64-linux/debug/lib/pkgconfig/absl_algorithm.pc
...

I think we need to use the approach here across all platforms:

TileDB/.github/workflows/build-windows.yml

Lines 124 to 144 in 70f5b4f

    
                 - name: Prevent vcpkg from building debug variants 
        
                   shell: python 
        
                   run: | 
        
                     import os 
        
                     workspace = os.environ["GITHUB_WORKSPACE"] 
        
                     triplets = os.path.join(workspace, "external", "vcpkg", "triplets") 
        
                     for path, dnames, fnames in os.walk(triplets): 
        
                       for fname in fnames: 
        
                         fname = os.path.join(path, fname) 
        
                         print(fname) 
        
                         with open(fname, "rb") as handle: 
        
                           contents = handle.read() 
        
                         # If a file is already patched we skip patching it again because 
        
                         # this affects the hashes calculated by vcpkg for caching 
        
                         if b"VCPKG_BUILD_TYPE release" in contents: 
        
                           continue 
        
                         contents += b"\nset(VCPKG_BUILD_TYPE release)\n" 
        
                         with open(fname, "wb") as handle: 
        
                           handle.write(contents)

teo-tsirpanis · 2023-10-04T19:20:48Z

I will work on it tomorrow.

teo-tsirpanis · 2023-10-05T12:10:32Z

Dependecies will be built in Release mode only during CI with #4162.

teo-tsirpanis · 2023-10-05T15:29:09Z

CI is green.

ihnorton · 2023-10-05T15:35:55Z

Thanks!

This PR: * Removes the `abseil` vcpkg feature, the abseil linkage tests and the relevant bootstrap script options. These are dead code since #4055. * Abseil is still acquired from vcpkg if GCS is enabled. * Deletes unused patches for the ExternalProject of the GCS SDK, which has been removed in #4031. * Removes the vcpkg direct dependency to liblzma. The Core does not directly use liblzma. * Removes the crc32c linkage tests and the related CMake and bootstrap script options. They are not used by CI and their value has declined with the advent of vcpkg. Moreover using exported static libraries with `TILEDB_CRC32` enabled from CMake will fail because there is no `find_dependency(Crc32c)` in the config file. Nor there is a `crc32` vcpkg feature. And this option offers nothing at runtime. * Removes `unit_link_webp` as this was used to prove WebP was properly included into the build. Now that we have a WebP filter, this code is not necessary anymore. --- TYPE: IMPROVEMENT DESC: Clean-up unused build system code. TYPE: BUILD DESC: Remove the CMake and bootstrap script options related to abseil and crc32c linkage tests.

teo-tsirpanis added 4 commits April 10, 2023 10:55

Update the vcpkg submodule.

cf26c54

Update the Google Cloud SDK to its latest version.

c5fbcb3

Use the zero-copy InsertObject overload.

c9ccd1f

Remove GCSSDK_EP.

fa1c10c

Using the old version 1.22.0 will no longer compile.

teo-tsirpanis requested a review from ihnorton April 10, 2023 10:49

ihnorton added the pending-2.16 label Apr 10, 2023

ihnorton requested a review from davisp April 10, 2023 20:09

ihnorton requested changes Apr 10, 2023

View reviewed changes

teo-tsirpanis added the blocked label Apr 11, 2023

ihnorton mentioned this pull request Apr 19, 2023

Various improvements to the GCS VFS. #4008

Closed

teo-tsirpanis added 2 commits April 19, 2023 21:27

Bring back the custom port for abseil and update the one for google-c…

555ac5d

…loud-cpp.

Use zero-copy writes in one more case.

e989840

davisp approved these changes Apr 20, 2023

View reviewed changes

ihnorton approved these changes May 2, 2023

View reviewed changes

teo-tsirpanis added 7 commits May 2, 2023 15:53

Merge branch 'dev' into gcs-update

fd3eea2

Do not import Curl_EP if GCS is enabled.

45a1375

Merge branch 'dev' into gcs-update

3a27daf

Remove the custom ports of google-cloud-cpp and abseil.

bab9a26

Update to version 2.11.0.

79cb6e6

Merge branch 'dev' into gcs-update

4cffd97

Merge branch 'dev' into gcs-update

63d60c8

ihnorton mentioned this pull request Jul 18, 2023

Dlh/sc 17498 upd gcs wip #3423

Closed

teo-tsirpanis added 2 commits September 28, 2023 21:31

Merge branch 'dev' into gcs-update

3f47d02

Update the GCS SDK to version 2.15.1.

1299b72

Merge branch 'dev' into gcs-update

b2538cb

ihnorton added pending-2.18 and removed blocked pending-2.16 labels Oct 3, 2023

Pin a new version of abseil and enable its C++17 features.

b906aba

ihnorton approved these changes Oct 4, 2023

View reviewed changes

Merge remote-tracking branch 'origin/dev' into teo-tsirpanis/gcs-update

a12cba4

ihnorton merged commit ea86f85 into TileDB-Inc:dev Oct 5, 2023
50 checks passed

teo-tsirpanis deleted the gcs-update branch October 5, 2023 15:39

teo-tsirpanis mentioned this pull request Nov 20, 2023

Clean-up unused build system code. #4527

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the GCS SDK to its latest version. #4031

Update the GCS SDK to its latest version. #4031

teo-tsirpanis commented Apr 10, 2023

teo-tsirpanis commented Apr 10, 2023

shortcut-integration bot commented Apr 10, 2023

ihnorton left a comment •

edited

Loading

teo-tsirpanis commented Apr 11, 2023 •

edited

Loading

ihnorton commented Apr 11, 2023

teo-tsirpanis commented Apr 18, 2023

teo-tsirpanis commented Apr 20, 2023

davisp left a comment

ihnorton left a comment

teo-tsirpanis commented Oct 4, 2023

teo-tsirpanis commented Oct 4, 2023

ihnorton commented Oct 4, 2023

teo-tsirpanis commented Oct 4, 2023

teo-tsirpanis commented Oct 5, 2023

teo-tsirpanis commented Oct 5, 2023

ihnorton commented Oct 5, 2023

Update the GCS SDK to its latest version. #4031

Update the GCS SDK to its latest version. #4031

Conversation

teo-tsirpanis commented Apr 10, 2023

teo-tsirpanis commented Apr 10, 2023

shortcut-integration bot commented Apr 10, 2023

ihnorton left a comment • edited Loading

Choose a reason for hiding this comment

teo-tsirpanis commented Apr 11, 2023 • edited Loading

ihnorton commented Apr 11, 2023

teo-tsirpanis commented Apr 18, 2023

teo-tsirpanis commented Apr 20, 2023

davisp left a comment

Choose a reason for hiding this comment

ihnorton left a comment

Choose a reason for hiding this comment

teo-tsirpanis commented Oct 4, 2023

teo-tsirpanis commented Oct 4, 2023

ihnorton commented Oct 4, 2023

teo-tsirpanis commented Oct 4, 2023

teo-tsirpanis commented Oct 5, 2023

teo-tsirpanis commented Oct 5, 2023

ihnorton commented Oct 5, 2023

ihnorton left a comment •

edited

Loading

teo-tsirpanis commented Apr 11, 2023 •

edited

Loading