Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Search for libblosc2 with pkg-config if not in the python blosc2 installation #1000

Merged
merged 8 commits into from Mar 8, 2023
Merged

Search for libblosc2 with pkg-config if not in the python blosc2 installation #1000

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cfbe1f5
search for libblosc2 with pkg-config if not in the python blosc2 inst…
bnavigator Feb 19, 2023
31c5ddf
remove blosc2 from runtime requirements
bnavigator Feb 25, 2023
4c4eb91
Update build and runtime requirement specs
bnavigator Feb 25, 2023
3579028
only build one sdist for CI
bnavigator Feb 26, 2023
bbb02dd
Enhance blosc2 library search
bnavigator Mar 4, 2023
02ea1ab
Only use pkg-config for Unix-like systems
bnavigator Mar 4, 2023
f16c76f
Make sure the blosc2 wheel or conda pkg is not installed
bnavigator Mar 6, 2023
ee726ff
Bump minimum c-blosc2 version
bnavigator Mar 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
52 changes: 43 additions & 9 deletions .github/workflows/ci.yml
View file
Open in desktop
Expand Up @@ -3,9 +3,36 @@ name: CI
on: [push, pull_request]

jobs:

sdist:
name: Create sdist
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
submodules: true
- uses: actions/setup-python@v4
name: Install Python
with:
python-version: 3.x
- name: Ubuntu library headers
run: sudo apt install libblosc-dev libbz2-dev libhdf5-dev liblz4-dev liblzo2-dev libsnappy-dev libzstd-dev zlib1g-dev
- name: Source distribution
run: |
pip install build
python -m build --sdist
- name: Save source distribution
uses: actions/upload-artifact@v3
with:
name: tables-sdist
path: dist/*.tar.gz


build:
name: ${{ matrix.os }} ${{ matrix.python }} ${{ matrix.name }}
runs-on: ${{ matrix.os }}
needs: sdist
defaults:
run:
shell: bash -l {0}
Expand All @@ -29,15 +56,17 @@ jobs:
channel-priority: strict
- name: Install dependencies
run: |
conda install setuptools pip wheel build packaging numpy cython bzip2 hdf5 lzo 'typing_extensions<4.2'
conda install -q setuptools pip wheel build packaging numpy cython bzip2 hdf5 lzo
python -m pip install blosc2
python -m pip install -r requirements.txt
# conda install sphinx sphinx_rtd_theme numpydoc ipython
- name: Source distribution
run: |
python -m build --sdist
- name: Get source distrubtion
uses: actions/download-artifact@v3
with:
name: tables-sdist
- name: Installation
run: |
python -m pip install -v dist/*.tar.gz
python -m pip install -v tables-*.tar.gz
- name: 'Run test'
run: |
cd .. && python -m tables.tests.test_all -v
Expand All @@ -50,6 +79,7 @@ jobs:
build_cblosc:
name: Sdist with cblosc2
runs-on: ubuntu-latest
needs: sdist
defaults:
run:
shell: bash -l {0}
Expand All @@ -68,12 +98,16 @@ jobs:
- name: Install dependencies
run: |
conda install setuptools pip wheel build packaging py-cpuinfo numpy cython numexpr bzip2 hdf5 lzo 'typing_extensions<4.2' c-blosc2
- name: Source distribution
run: |
python -m build --sdist
- name: Get source distrubtion
uses: actions/download-artifact@v3
with:
name: tables-sdist
- name: Installation
run: |
python -m pip install -v dist/*.tar.gz --no-deps
python -m pip install -v tables-*.tar.gz --no-deps
- name: Python blosc2 wheel or conda pkg is not installed
run: |
! conda list | grep -E '^(python-)?blosc2'
- name: 'Run test'
run: |
cd .. && python -m tables.tests.test_all -v
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ubuntu.yml
View file
Open in desktop
Expand Up @@ -20,7 +20,7 @@ jobs:
sudo apt install latexmk texlive-fonts-recommended texlive-latex-recommended texlive-latex-extra texlive-plain-generic
- name: Install dependencies
run: |
python3 -m pip install --user --upgrade setuptools pip wheel build
python3 -m pip install --user --upgrade setuptools pip wheel build blosc2
python3 -m pip install --user --upgrade -r requirements.txt
python3 -c "import numpy as np; print('numpy:', np.__version__)"
python3 -c "import blosc2; print('blosc2:', blosc2.__version__)"
Expand Down
20 changes: 11 additions & 9 deletions doc/source/usersguide/installation.rst
View file
Open in desktop
Expand Up @@ -47,23 +47,24 @@ packages.
Prerequisites
~~~~~~~~~~~~~

First, make sure that you have
First, make sure that you have the following dependencies installed.
If you don't, fetch and install them before proceeding.

* Python_ >= 3.6 (PyTables-3.5 was the last release with Python 2.7 support)
* Python_ >= 3.8
* HDF5_ >= 1.10.5
* NumPy_ >= 1.19.0
* Numexpr_ >= 2.6.2
* Cython_ >= 0.29.21
* c-blosc_ >= 1.11.1 (sources are bundled with PyTables sources but the user can
use an external version of sources using the :envvar:`BLOSC_DIR` environment
variable or the `--blosc` flag of the :file:`setup.py`)
* python-blosc2_ ~= 2.0.0 This is the Python wheel containing *both* the C-Blosc2 libs
and headers (>= 2.6.0), as well as the Python wrapper for Blosc2 (not currently
used, but it might be in the future).

installed (for testing purposes, we are using HDF5_ 1.10.7, NumPy_ 1.23.2
and Numexpr_ 2.8.1 currently). If you don't, fetch and install them before
proceeding.
* Either
* python-blosc2_ ~= 2.0.0, this is the Python wheel containing *both* the C-Blosc2
libs and headers (>= 2.6.0), as well as the Python wrapper for Blosc2 (not
currently used, but it might be in the future), or
* A standalone installation of the c-blosc2_ library including the headers.
The latter are usually provided by Linux distribtions in a package named
`blosc2-devel`, `libblosc2-dev``, or similar.

.. _Python: http://www.python.org
.. _HDF5: http://www.hdfgroup.org/HDF5
Expand All @@ -72,6 +73,7 @@ proceeding.
.. _Cython: http://www.cython.org
.. _c-blosc: https://github.com/Blosc/c-blosc
.. _python-blosc2: https://github.com/Blosc/python-blosc2
.. _c-blosc2: https://github.com/Blosc/c-blosc2

Compile and install these packages (but see :ref:`prerequisitesBinInst` for
instructions on how to install pre-compiled binaries if you are not willing
Expand Down
25 changes: 14 additions & 11 deletions pyproject.toml
View file
Open in desktop
@@ -1,11 +1,15 @@
[build-system]
# keep in sync with requirements.txt
requires = [
"setuptools >=61.0.0",
"wheel",
"oldest-supported-numpy",
"packaging",
"py-cpuinfo",
"Cython >=0.29.21",
# Included here for seamless wheel builds.
# Packagers can choose to replace it by externally provided
# c-blosc2 library and headers
"blosc2 ~=2.0.0",
]
build-backend = "setuptools.build_meta"
Expand All @@ -20,16 +24,15 @@ build-backend = "setuptools.build_meta"
# "packaging",
# "py-cpuinfo",
# "Cython >=0.29.21",
# # for seamless wheel builds. Packagers can choose to replace
# # it by externally provided c-blosc2 library and headers
# "blosc2 ~=2.0.0",
# ]


[project]
name = "tables"
dynamic = [
"version",
"dependencies",
]
dynamic = [ "version" ]
description = "Hierarchical datasets for Python"
requires-python = ">=3.8"
license = { text = "BSD 3-Clause License" }
Expand Down Expand Up @@ -63,12 +66,13 @@ classifiers = [
"Topic :: Database",
"Topic :: Software Development :: Libraries :: Python Modules",
]
# dependencies = [
# "numpy >= 1.19.0",
# "numexpr >= 2.6.2",
# "packaging",
# "py-cpuinfo",
# ]
# Keep in sync with tables/req_versions.py and user doc
dependencies = [
"numpy >= 1.19.0",
"numexpr >= 2.6.2",
"packaging",
"py-cpuinfo",
]


[project.optional-dependencies]
Expand Down Expand Up @@ -117,7 +121,6 @@ license-files = [ "LICENSE.txt" ]

[tool.setuptools.dynamic]
version = { attr = "tables._version.__version__" }
dependencies = { file = "requirements.txt" }


[tool.setuptools.packages.find]
Expand Down
9 changes: 3 additions & 6 deletions requirements.txt
View file
Open in desktop
@@ -1,11 +1,8 @@
# Keep in sync with tables/req_versions.py and
# doc/source/usersguide/installation.rst *and*
# Build requirements
# Keep in sync with doc/source/usersguide/installation.rst *and*
# pyproject.toml
cython>=0.29.21
numpy>=1.19.0
numexpr>=2.6.2
# blosc2 wheel is actually only needed when compiling on conda envs.
# Otherwise, lib comes bundled in PyTables wheels (but it doesn't hurt either).
blosc2~=2.0.0
packaging
py-cpuinfo
# provide blosc2 wheel or c-blosc2 headers externally!
25 changes: 23 additions & 2 deletions setup.py
View file
Open in desktop
Expand Up @@ -97,7 +97,28 @@ def get_blosc2_version(headername):


def get_blosc2_directories():
"""Get Blosc2 directories for the C library by using wheel metadata"""
"""Get Blosc2 directories for the C library"""
# If the system provides the library and headers,
# get them from environment variables or find by pkg-config
if platform.system() in ["Linux", "Darwin"]:
try:
include_path = Path(os.environ.get(
"BLOSC2_INCDIR",
subprocess.check_output(
[PKG_CONFIG, '--variable=includedir', 'blosc2'],
text=True).strip()))
library_path = Path(os.environ.get(
"BLOSC2_LIBDIR",
subprocess.check_output(
[PKG_CONFIG, '--variable=libdir', 'blosc2'],
text=True).strip()))
except subprocess.CalledProcessError:
pass
else:
return include_path, library_path

# Otherwise get them from the PyPI published wheels for blosc2
# They package the library and headers directly
try:
import blosc2
except ModuleNotFoundError:
Expand Down Expand Up @@ -841,7 +862,7 @@ def find_runtime_path(self, locations=default_runtime_dirs):
shared_libs = glob.glob(str(libdir) + '/libblosc2*.dylib')
for lib in shared_libs:
shutil.copy(lib, dll_dir)

else:
copy_libs += ['libblosc2.dll']
dll_dir = 'C:\\Miniconda\\envs\\build\\Library\\bin'
Expand Down
28 changes: 19 additions & 9 deletions tables/__init__.py
View file
Open in desktop
Expand Up @@ -8,25 +8,35 @@
"""
import os
from ctypes import cdll
from ctypes.util import find_library
import platform


# Load the blosc2 library, and if not found in standard locations,
# try this directory (it should be automatically copied in setup.py).
current_dir = os.path.dirname(__file__)
platform_system = platform.system()
blosc2_lib = "libblosc2"
blosc2_lib_hardcoded = "libblosc2"
if platform_system == "Linux":
blosc2_lib += ".so"
blosc2_lib_hardcoded += ".so"
elif platform_system == "Darwin":
blosc2_lib += ".dylib"
blosc2_lib_hardcoded += ".dylib"
else:
blosc2_lib += ".dll"
try:
cdll.LoadLibrary(blosc2_lib)
except OSError:
cdll.LoadLibrary(os.path.join(current_dir, blosc2_lib))

blosc2_lib_hardcoded += ".dll"
blosc2_found = False
blosc2_search_paths = [blosc2_lib_hardcoded,
os.path.join(current_dir, blosc2_lib_hardcoded),
find_library("blosc2")]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see a few problems:

  1. find_library() returns None if it doesn't find the library, and cdll.LoadLibrary(None) doesn't DTRT:
>>> ctypes.util.find_library('missing.so') is None
True
>>> ctypes.cdll.LoadLibrary(None)
<CDLL 'None', handle 7f317dd372c0 at 0x7f316f9e7f10>

That loop needs an additional if blosc2_lib check.

  1. Loading of library from $PWD creates the obvious security issue where an attacker can place a file in $PWD and cause the user to load it. Loading from $PWD can be done in a test environment, but MUST NOT be done in production code.

  2. find_library() doesn't seem to fit the bill. Quoting the docs: "The purpose of the find_library() function is to locate a library in a way similar to what the compiler or runtime loader does (on platforms with several versions of a shared library the most recent should be loaded)". It would find the newest version of the library, which might or might not be what is needed. The whole point of the SONAME field (i.e. the thing that gives the .2 suffix) is to refer to a specific version from code using the library, so that when a future incompatible version appears, existing code is not broken.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://src.fedoraproject.org/rpms/python-tables/blob/rawhide/f/0007-Drop-misguided-check.patch
This is the patch I pushed in Fedora. It ignores non-Linux systems, so it's not suitable for inclusion upstream.

for blosc2_lib in blosc2_search_paths:
try:
cdll.LoadLibrary(blosc2_lib)
blosc2_found = True
break
except OSError:
pass
if not blosc2_found:
raise RuntimeError("Blosc2 library not found. "
f"I looked for \"{', '.join(blosc2_search_paths)}\"")

# Necessary imports to get versions stored on the cython extension
from .utilsextension import get_hdf5_version as _get_hdf5_version
Expand Down
8 changes: 5 additions & 3 deletions tables/req_versions.py
View file
Open in desktop
Expand Up @@ -3,12 +3,14 @@
from packaging.version import Version

# **********************************************************************
# Keep these in sync with setup.cfg and user's guide
# Rimtime requirements, keep these in sync with pyrpoject.toml,
# and the user's guide doc/source/usersguide/installation.rst
# **********************************************************************

# Minimum recommended versions for mandatory packages
min_numpy_version = Version('1.9.3')
min_numpy_version = Version('1.19.0')
min_numexpr_version = Version('2.6.2')
# These are library versions, not the python modules
min_hdf5_version = Version('1.10.5')
min_blosc_version = Version('1.11.1')
min_blosc2_version = Version('2.5.0')
min_blosc2_version = Version('2.6.0')