Wheels: ADIOS2 -> v2.9.0, blosc2 (#1387)

* Update blosc -> blosc2, ADIOS2 -> v2.9.0-rc1

ADIOS2 should be replace by v2.9.0 finally

* Fix Blosc2 CMake params

* Guard against CMAKE_OSX_ARCHITECTURES

* For now, deactivate the pthread patch in ADIOS2

* Test small patch for blosc2 CMake

* Add -fPIC flag

* MacOS compatibility

* -DADIOS2_USE_Blosc2=ON

* Build atl and ffs manually to avoid CMake import error

* Revert "Build atl and ffs manually to avoid CMake import error"

This reverts commit c5469a3.

* RC1 -> full release

* Don't build ADIOS1

* -DADIOS2_USE_MHS=OFF

* Use -DADIOS2_INSTALL_GENERATE_CONFIG=OFF workaround

See
ornladios/ADIOS2#3348 (comment)

* Do not disable find calls

* Revert "Do not disable find calls"

This reverts commit 7065b89.

* Try setting ADIOS2_PATH

* Keep ADIOS1 + c-blosc1

* Undo `build.yml` Changes

* ADIOS2: no c-blosc1 pthread issue

* HDF5: 1.14.1-2

* macOS CI: Now has 3 cores

* C-Blosc2 Updates

- CMake options
- Windows
- build stamp

* Windows Updates

* HDF5 Patches

* C-Blosc 2.9.3 + CMake Patch

* ARM: H5Detect

* Windows: Avoid CXX20 in C-Blosc2

* macOS arm64 HDF5: H5detect cross-compile

* HDF5 arm64: cross-compiling H5detect

* [Patch] ADIOS2 c-blosc2 windows.h std::min

* [Patch] C-Blosc2: External Zlib

* [Hack] Windows: Ignore Ext. ZLIB for c-blosc2

Just build again and close eyes.

* ppc64le: Skip C-Blosc & ADIOS1

Due to time limits on Travis-CI.

* Unix: Keep HDF5 1.12.2 for Now

- issues with cross-compile cannot quickly be solved
  for macOS arm64/aarch64
- disable HDF5 support in ADIOS2 for now

* [Patch] C-Blosc2 PUBLIC Linkage Zlib

* Use Blosc2Config.cmake Package

* Cleanup and `cat` ADIOS2 Config File

* [Patch] [Hack] CMake --trace-expand

* Cleanup: cat adios2-config.cmake

* [Travis] PPC64le: `travis_wait 30`

Takes longer now w/o output to the terminal than the default
wait time of 10min.
https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received

* [ADIOS2] CMake --trace-expand

* Undo Trace Expand Hacks

* [Travis] PPC64le: `travis_wait 45`

* ADIOS2: w/ SST

* Blosc2: CMake Config PR Merged

* macOS (x86): 10.15 -> 11.0

The macOS 10.15 build images on GH Actions are gone, so we remove
this EOL platform for good now.

* Windows: Skip Building Blosc1

Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com>

* Python: 3.8+

Skip older Python releases, 3.7 is EOL since last month.

* [Patch] ADIOS2: Remove FindBlosc2.cmake

* C-Blosc 2.10.1

* Win: ADIOS 2.9.1

* ADIOS2_Blosc2_PREFER_SHARED=OFF

ornladios/ADIOS2#3715 (comment)

* cibuildwheel 2.12.1 -> 2.14.1

For `pp310-manylinux_aarch64`

---------

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

.github/workflows/build.yml

@@ -40,10 +40,10 @@ jobs:
               CMAKE_GENERATOR: "Visual Studio 16 2019"
               CMAKE_GENERATOR_PLATFORM: "Win32"
 
-          - os: macos-10.15
+          - os: macos-11
             arch: "x86_64"
             env:
-              MACOSX_DEPLOYMENT_TARGET: 10.15
+              MACOSX_DEPLOYMENT_TARGET: 11.0
 
           # Apple Silicon M1/arm64/aarch64 builds:
           #   https://cibuildwheel.readthedocs.io/en/stable/faq/#apple-silicon
@@ -79,55 +79,45 @@ jobs:
     - uses: actions/setup-python@v1
       name: Install Python
       with:
-        python-version: '3.7'
+        python-version: '3.8'
 
     - name: Install cibuildwheel
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip install cibuildwheel==2.12.1
-
-#    # 0.15.0: CMake: Fix Python Install Directory
-#    # https://github.com/openPMD/openPMD-api/pull/1393
-#    - name: Download Patch 1/2
-#      uses: suisei-cn/actions-download-file@v1
-#      id: patch1
-#      with:
-#        url: "https://github.com/ax3l/openPMD-api/commit/b622cc5ea770f866c1e373185a9e389c04bdb54c.patch"
-#        target: src/.patch/
-#
-#    # 0.15.0.post2 bump
-#    - name: Download Patch 2/2
-#      uses: suisei-cn/actions-download-file@v1
-#      id: setupversion
-#      with:
-#        url: "https://gist.githubusercontent.com/ax3l/5e83edefe4b05cf6aa2a971649285fe0/raw/023cbd69e06715e5191ece741bb1de56560a9f96/0001-Bump-setup.py.patch"
-#        target: src/.patch/
-
-#    - name: Apply Patches
-#      run: |
-#        python -m pip install "patch==1.*"
-#        cd src
-#        python -m patch .patch/b622cc5ea770f866c1e373185a9e389c04bdb54c.patch
-#        python -m patch .patch/0001-Bump-setup.py.patch
+        python -m pip install cibuildwheel==2.14.1
+
+    # Hack: cmake --trace-expand
+    #- name: Download Patch 2/2
+    #  uses: suisei-cn/actions-download-file@v1
+    #  id: setupversion
+    #  with:
+    #    url: "https://gist.githubusercontent.com/ax3l/9b15c5bc73d7f3c1c66784187a016474/raw/e31dd54498c9c8fc465a7cbb6f900f768b9d0c0b/0001-Hack-setup.py-CMake-trace-expand.patch"
+    #    target: src/.patch/
+
+    #- name: Apply Patches
+    #  run: |
+    #    python -m pip install "patch==1.*"
+    #    cd src
+    #    python -m patch .patch/0001-Hack-setup.py-CMake-trace-expand.patch
 
     - name: Build wheel
       env:
         # (1) Disable PyPy (manylinux image: yum repo issues)
         #     https://github.com/pypa/manylinux/issues/899
         CIBW_SKIP: "pp*-manylinux*"
         CIBW_ARCHS: "${{ matrix.arch }}"
-        CIBW_PROJECT_REQUIRES_PYTHON: ">=3.7"
+        CIBW_PROJECT_REQUIRES_PYTHON: ">=3.8"
         # Install dependencies
         CIBW_BEFORE_BUILD_LINUX: bash -x .github/library_builders.sh
         CIBW_BEFORE_BUILD_MACOS: bash -x .github/library_builders.sh
         CIBW_BEFORE_BUILD_WINDOWS: 'cmd /E:ON /V:ON /C .github\library_builders.bat'
         # for the openPMD-api build, CMake shall search for
         # static dependencies of HDF5 and ADIOS1 (see setup.py)
         CIBW_ENVIRONMENT: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' ADIOS_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON'
-        CIBW_ENVIRONMENT_WINDOWS: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON' CMAKE_PREFIX_PATH='C:/Program Files (x86)/ADIOS2;C:/Program Files (x86)/blosc;C:/Program Files (x86)/HDF5;C:/Program Files (x86)/ZFP;C:/Program Files (x86)/zlib'
+        CIBW_ENVIRONMENT_WINDOWS: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON' CMAKE_PREFIX_PATH='C:/Program Files (x86)/ADIOS2;C:/Program Files (x86)/blosc;C:/Program Files (x86)/blosc2;C:/Program Files (x86)/HDF5;C:/Program Files (x86)/ZFP;C:/Program Files (x86)/zlib'
         # C++17 support in macOS 10.13+ (partial) and 10.14+ (std::visit) and 10.15+ (std::filesystem::path)
         #   https://cibuildwheel.readthedocs.io/en/stable/cpp_standards/#macos-and-deployment-target-versions
-        # arm64 Python interpreters are built with 11.0
+        # arm64/aarch64 (M1/M2) requires 11.0+
         MACOSX_DEPLOYMENT_TARGET: "${{ matrix.env.MACOSX_DEPLOYMENT_TARGET }}"
         CMAKE_OSX_ARCHITECTURES: "${{ matrix.env.CMAKE_OSX_ARCHITECTURES }}"
         # Windows cross-compilation

.travis.yml

@@ -10,7 +10,7 @@ env:
   global:
     - OPENPMD_GIT_REF="0.15.1"
 
-    - CIBW_PROJECT_REQUIRES_PYTHON=">=3.7"
+    - CIBW_PROJECT_REQUIRES_PYTHON=">=3.8"
     # Install dependencies on Linux and OSX
     - CIBW_BEFORE_BUILD="bash -x .github/library_builders.sh"
     # for the openPMD-api build, CMake shall search for
@@ -28,17 +28,17 @@ jobs:
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp37-manylinux_aarch64 cp38-manylinux_aarch64"
+        - CIBW_BUILD="cp38-manylinux_aarch64 cp39-manylinux_aarch64"
     - services: docker
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp39-manylinux_aarch64 cp310-manylinux_aarch64"
+        - CIBW_BUILD="cp310-manylinux_aarch64 cp311-manylinux_aarch64"
     - services: docker
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp311-manylinux_aarch64 pp37-manylinux_aarch64"
+        - CIBW_BUILD="pp310-manylinux_aarch64"
     - services: docker
       arch: arm64
       dist: focal
@@ -49,24 +49,14 @@ jobs:
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp37-musllinux_aarch64 cp38-musllinux_aarch64"
+        - CIBW_BUILD="cp38-musllinux_aarch64 cp39-musllinux_aarch64"
     - services: docker
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp39-musllinux_aarch64 cp310-musllinux_aarch64"
-    - services: docker
-      arch: arm64
-      dist: focal
-      env:
-        - CIBW_BUILD="cp311-musllinux_aarch64"
+        - CIBW_BUILD="cp310-musllinux_aarch64 cp311-musllinux_aarch64"
 
     # perform a linux PPC64LE build
-    - services: docker
-      arch: ppc64le
-      dist: focal
-      env:
-        - CIBW_BUILD="cp37-manylinux_ppc64le"
     - services: docker
       arch: ppc64le
       dist: focal
@@ -87,11 +77,6 @@ jobs:
       dist: focal
       env:
         - CIBW_BUILD="cp311-manylinux_ppc64le"
-    - services: docker
-      arch: ppc64le
-      dist: focal
-      env:
-        - CIBW_BUILD="cp37-musllinux_ppc64le"
     - services: docker
       arch: ppc64le
       dist: focal
@@ -131,7 +116,7 @@ install:
   - git clone --branch ${OPENPMD_GIT_REF} --depth 1 https://github.com/openPMD/openPMD-api.git src
   - cp library_builders.sh src/.github/
   - python -m pip install --upgrade pip setuptools wheel
-  - python -m pip install cibuildwheel==2.12.1
+  - python -m pip install cibuildwheel==2.14.1
   # twine & cryptography: see
   #   https://github.com/scikit-build/cmake-python-distributions/blob/4730aeee240917303f293dffc89a8d8d5a4787c4/requirements-deploy.txt
   #   https://github.com/pyca/cryptography/issues/6086
@@ -155,7 +140,7 @@ install:
 
 script:
   - cd src
-  - python3 -m cibuildwheel --output-dir ../wheelhouse
+  - travis_wait 45 python3 -m cibuildwheel --output-dir ../wheelhouse
   - cd ..
 
 deploy:

library_builders.bat

@@ -16,25 +16,22 @@ exit /b 0
 
 :build_adios2
   if exist adios2-stamp exit /b 0
-  curl -sLo adios2-2.7.1.zip ^
-    https://github.com/ornladios/ADIOS2/archive/v2.7.1.zip
-  powershell Expand-Archive adios2-2.7.1.zip -DestinationPath dep-adios2
+  curl -sLo adios2-2.9.1.zip ^
+    https://github.com/ornladios/ADIOS2/archive/v2.9.1.zip
+  powershell Expand-Archive adios2-2.9.1.zip -DestinationPath dep-adios2
 
-  :: Patch Blosc Compression w/ ADIOS 2.7.1
-  curl -sLo adios2-blosc.patch ^
-    https://patch-diff.githubusercontent.com/raw/ornladios/ADIOS2/pull/2746.patch
-  python -m patch -p 1 -d dep-adios2/ADIOS2-2.7.1 adios2-blosc.patch
-
-  cmake -S dep-adios2/ADIOS2-2.7.1 -B build-adios2 ^
+  cmake -S dep-adios2/ADIOS2-2.9.1 -B build-adios2 ^
     -DCMAKE_BUILD_TYPE=Release  ^
     -DBUILD_SHARED_LIBS=OFF     ^
     -DBUILD_TESTING=OFF         ^
     -DADIOS2_USE_MPI=OFF        ^
     -DADIOS2_BUILD_EXAMPLES=OFF ^
-    -DADIOS2_USE_Blosc=ON       ^
+    -DADIOS2_Blosc2_PREFER_SHARED=OFF ^
+    -DADIOS2_USE_Blosc2=ON      ^
     -DADIOS2_USE_BZip2=OFF      ^
     -DADIOS2_USE_Fortran=OFF    ^
     -DADIOS2_USE_HDF5=OFF       ^
+    -DADIOS2_USE_MHS=OFF        ^
     -DADIOS2_USE_PNG=OFF        ^
     -DADIOS2_USE_Profiling=OFF  ^
     -DADIOS2_USE_Python=OFF     ^
@@ -88,14 +85,47 @@ exit /b 0
   if errorlevel 1 exit 1
 exit /b 0
 
+:build_blosc2
+  if exist blosc2-stamp exit /b 0
+
+  curl -sLo blosc2-2.10.1.zip ^
+    https://github.com/Blosc/c-blosc2/archive/refs/tags/v2.10.1.zip
+  powershell Expand-Archive blosc2-2.10.1.zip -DestinationPath dep-blosc2
+
+  cmake -S dep-blosc2/c-blosc2-2.10.1 -B build-blosc2 ^
+    -DCMAKE_BUILD_TYPE=Release  ^
+    -DBUILD_SHARED=OFF          ^
+    -DBUILD_STATIC=ON           ^
+    -DBUILD_BENCHMARKS=OFF      ^
+    -DBUILD_EXAMPLES=OFF        ^
+    -DBUILD_FUZZERS=OFF         ^
+    -DBUILD_PLUGINS=OFF         ^
+    -DBUILD_TESTS=OFF
+::    -DPREFER_EXTERNAL_ZLIB=ON
+::    -DZLIB_USE_STATIC_LIBS=ON
+  if errorlevel 1 exit 1
+
+  cmake --build build-blosc2 --parallel %CPU_COUNT%
+  if errorlevel 1 exit 1
+
+  cmake --build build-blosc2 --target install --config Release
+  if errorlevel 1 exit 1
+
+  rmdir /s /q build-blosc2
+  if errorlevel 1 exit 1
+
+  break > blosc2-stamp
+  if errorlevel 1 exit 1
+exit /b 0
+
 :build_hdf5
   if exist hdf5-stamp exit /b 0
 
-  curl -sLo hdf5-1.12.2.zip ^
-    https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2.zip
-  powershell Expand-Archive hdf5-1.12.2.zip -DestinationPath dep-hdf5
+  curl -sLo hdf5-1.14.1-2.zip ^
+    https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.14/hdf5-1.14.1/src/hdf5-1.14.1-2.zip
+  powershell Expand-Archive hdf5-1.14.1-2.zip -DestinationPath dep-hdf5
 
-  cmake -S dep-hdf5/hdf5-1.12.2 -B build-hdf5 ^
+  cmake -S dep-hdf5/hdf5-1.14.1-2 -B build-hdf5 ^
     -DCMAKE_BUILD_TYPE=Release  ^
     -DCMAKE_VERBOSE_MAKEFILE=ON ^
     -DBUILD_SHARED_LIBS=OFF     ^
@@ -193,7 +223,7 @@ call :install_buildessentials
 call :build_zlib
 :: build_bzip2
 :: build_szip
-call :build_blosc
 call :build_zfp
+call :build_blosc2
 call :build_hdf5
 call :build_adios2