Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3Packages.tables: fix usage, add tests #119258

Merged
merged 2 commits into from May 25, 2021

Conversation

drewrisinger
Copy link
Contributor

@drewrisinger drewrisinger commented Apr 12, 2021

Motivation for this change

While trying to open a HDF5 file via python3Packages.pandas, realized that pandas' HDF5 functionality was broken b/c python3Packages.tables wasn't importing.

The main failure I was having was due to setuptools not being installed at runtime (couldn't import pkg_resources), but also noticed that tests weren't being run.
Trying to run the tests found that tables wasn't compatible with hdf5 v1.10.7 (PyTables/PyTables#845). tables seems to be the only package using hdf5_1_10 (

tables = if isPy3k then
callPackage ../development/python-modules/tables {
hdf5 = pkgs.hdf5_1_10;
}
else
callPackage ../development/python-modules/tables/3.5.nix {
hdf5 = pkgs.hdf5_1_10;
};
), so maybe we just revert the version of hdf5_1_10 to v1.10.5?

I know this probably needs a little bit of work/cleanup, just wanted to get some opinions on this.

Things tried:

Improvements:

  • Update website address to HTTPS
  • Add pythonImportsCheck
  • Run tests (via pytestCheckHook)
  • Add myself as maintainer
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@drewrisinger drewrisinger changed the title python3Packages.tables: fix build, add tests python3Packages.tables: fix usage, add tests Apr 12, 2021
@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Apr 12, 2021

so maybe we just revert the version of hdf5_1_10 to v1.10.5?

We already pulled in patches for 1.10.5 so reverting it is also work at this point.

I miss understood that. Yeah, changing hdf5_1_10 would be an option.

@r-rmcgibbo
Copy link

Result of nixpkgs-review pr 119258 at d4684190 run on aarch64-linux 1

59 packages marked as broken and skipped:
  • mlflow-server
  • python38Packages.arviz
  • python38Packages.caffe
  • python38Packages.clifford
  • python38Packages.cnvkit
  • python38Packages.dask-gateway
  • python38Packages.dask-jobqueue
  • python38Packages.dask-ml
  • python38Packages.dask-mpi
  • python38Packages.dask-xgboost
  • ...
1 package failed to build:
307 packages skipped due to time constraints:
  • aiodnsbrute
  • apache-airflow
  • backblaze-b2
  • check_systemd
  • checkov
  • conan
  • cq-editor
  • csvs-to-sqlite (python38Packages.csvs-to-sqlite)
  • deja-dup
  • dpt-rp1-py
  • ...
32 packages built successfully:
  • graph-cli
  • python38Packages.Quandl (python38Packages.quandl)
  • python38Packages.apptools
  • python38Packages.bkcharts
  • python38Packages.drms
  • python38Packages.flammkuchen
  • python38Packages.huggingface-hub
  • python38Packages.influxdb
  • python38Packages.mplfinance
  • python38Packages.pandas
  • python38Packages.partd
  • python38Packages.phik
  • python38Packages.pyezviz
  • python38Packages.pytrends
  • python38Packages.rapidfuzz
  • python38Packages.seaborn
  • python38Packages.tables
  • python38Packages.tqdm
  • python38Packages.vega_datasets
  • python38Packages.vidstab
  • python38Packages.yfinance
  • python39Packages.bkcharts
  • python39Packages.huggingface-hub
  • python39Packages.mplfinance
  • python39Packages.pandas
  • python39Packages.partd
  • python39Packages.pyezviz
  • python39Packages.pytrends
  • python39Packages.seaborn
  • python39Packages.tables
  • python39Packages.tqdm
  • python39Packages.yfinance
1 suggestion:
  • warning: unused-argument

    Unused argument: python.
    Near pkgs/development/python-modules/tables/default.nix:7:3:

      |
    7 | , python
      |   ^
    

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.

Copy link
Member

@SuperSandro2000 SuperSandro2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please combine this PR into 2 or maybe 3 commits.

@drewrisinger drewrisinger force-pushed the dr-pr-fix-python-tables branch 2 times, most recently from b4232ca to c588ada Compare April 12, 2021 21:55
@FRidh
Copy link
Member

FRidh commented Apr 27, 2021

libminc also uses 1.10 #119572. More packages may show up.
Check with @bcdarwin for libminc if reverting version is OK. If it works for libminc then I'd say go ahead.

@drewrisinger drewrisinger mentioned this pull request Apr 27, 2021
8 tasks
* add myself as maintainer
* update homepage
* incorporate minor patches
* add working tests
This package only exists for compatibility with
``python3Packages.tables``, as far as I can tell based on usage in
nixpkgs.

``tables`` tests fail on hdf5 > v1.10.5, so we are reverting this to
the last "good" version of hdf5 that works with ``tables``.
``hdf5_1_10`` should hopefully be unpinned on the next release of
``python3Packages.tables``.
PyTables/PyTables#845
@drewrisinger
Copy link
Contributor Author

@bcdarwin gave consent for rolling back hdf5 by a patch-version in #119572 (comment). libminc builds fine locally.

@SuperSandro2000
Copy link
Member

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 119258 run on x86_64-linux 1

38 packages marked as broken and skipped:
  • mlflow-server
  • python38Packages.arviz
  • python38Packages.clifford
  • python38Packages.dask-jobqueue
  • python38Packages.dask-xgboost
  • python38Packages.dftfit
  • python38Packages.geopandas
  • python38Packages.lammps-cython
  • python38Packages.mlflow
  • python38Packages.mlxtend
  • python38Packages.optuna
  • python38Packages.osmnx
  • python38Packages.pymatgen-lammps
  • python38Packages.pymc3
  • python38Packages.rl-coach
  • python38Packages.shap
  • python38Packages.skorch
  • python38Packages.textacy
  • python38Packages.tiledb
  • python38Packages.trackpy
  • python39Packages.basemap
  • python39Packages.dask-jobqueue
  • python39Packages.dask-xgboost
  • python39Packages.geopandas
  • python39Packages.google-cloud-automl
  • python39Packages.google-cloud-bigquery
  • python39Packages.google-cloud-monitoring
  • python39Packages.lammps-cython
  • python39Packages.mlflow
  • python39Packages.mlxtend
  • python39Packages.optuna
  • python39Packages.osmnx
  • python39Packages.pymatgen-lammps
  • python39Packages.rl-coach
  • python39Packages.skorch
  • python39Packages.textacy
  • python39Packages.tiledb
  • python39Packages.trackpy
58 packages failed to build and already failed to build on hydra master:
  • apache-airflow: log was empty
  • backblaze-b2: log was empty
  • conan: log was empty
  • ocrmypdf: log was empty
  • python38Packages.aplpy: log was empty
  • python38Packages.apptools: log was empty
  • python38Packages.asdf: log was empty
  • python38Packages.cnvkit: log was empty
  • python38Packages.dask-image: log was empty
  • python38Packages.drms: log was empty
  • python38Packages.envisage: log was empty
  • python38Packages.graspologic: log was empty
  • python38Packages.imgaug: log was empty
  • python38Packages.ipydatawidgets: log was empty
  • python38Packages.labelbox: log was empty
  • python38Packages.mask-rcnn: log was empty
  • python38Packages.mayavi: log was empty
  • python38Packages.pygmt: log was empty
  • python38Packages.pymatgen: log was empty
  • python38Packages.pyregion: log was empty
  • python38Packages.pytest-astropy: log was empty
  • python38Packages.pytorch-metric-learning: log was empty
  • python38Packages.pywick: log was empty
  • python38Packages.radio_beam: log was empty
  • python38Packages.reproject: log was empty
  • python38Packages.spacy: log was empty
  • python38Packages.spectral-cube: log was empty
  • python38Packages.sumo: log was empty
  • python38Packages.sunpy: log was empty
  • python38Packages.tablib: log was empty
  • python38Packages.xgboost: log was empty
  • python39Packages.aplpy: log was empty
  • python39Packages.apptools: log was empty
  • python39Packages.asdf: log was empty
  • python39Packages.cnvkit: log was empty
  • python39Packages.dask-image: log was empty
  • python39Packages.drms: log was empty
  • python39Packages.envisage: log was empty
  • python39Packages.imgaug: log was empty
  • python39Packages.ipydatawidgets: log was empty
  • python39Packages.labelbox: log was empty
  • python39Packages.mask-rcnn: log was empty
  • python39Packages.mayavi: log was empty
  • python39Packages.nilearn: log was empty
  • python39Packages.pygmt: log was empty
  • python39Packages.pymatgen: log was empty
  • python39Packages.pyregion: log was empty
  • python39Packages.pytest-astropy: log was empty
  • python39Packages.pytorch-metric-learning: log was empty
  • python39Packages.pywick: log was empty
  • python39Packages.radio_beam: log was empty
  • python39Packages.reproject: log was empty
  • python39Packages.spacy: log was empty
  • python39Packages.spectral-cube: log was empty
  • python39Packages.sumo: log was empty
  • python39Packages.sunpy: log was empty
  • python39Packages.tablib: log was empty
  • python39Packages.xgboost: log was empty
16 packages failed to build and are new build failures:
  • deja-dup: log was empty
  • duplicity: log was empty
  • duply: log was empty
  • home-assistant: log was empty
  • noto-fonts-emoji: log was empty
  • nrfutil: log was empty
  • python38Packages.baselines: log was empty
  • python38Packages.ignite: log was empty
  • python38Packages.streamz: log was empty
  • python39Packages.baselines: log was empty
  • python39Packages.dask-glm: log was empty
  • python39Packages.qiskit: log was empty
  • python39Packages.qiskit-aqua: log was empty
  • sage: log was empty
  • sageWithDoc: log was empty
  • tts: log was empty
336 packages built:
  • EBTKS
  • aiodnsbrute
  • appdaemon
  • bicgl
  • bicpl
  • check_systemd
  • checkov
  • clevercsv (python38Packages.clevercsv)
  • conglomerate
  • cq-editor
  • csvs-to-sqlite (python38Packages.csvs-to-sqlite)
  • deepdiff (python38Packages.deepdiff)
  • dpt-rp1-py
  • evdevremapkeys
  • evillimiter
  • ezminc
  • ffmpeg-normalize
  • gdown (python38Packages.gdown)
  • gnome-keysign
  • graph-cli
  • hdf5_1_10
  • inormalize
  • jellyfin-mpv-shim
  • kaggle (python38Packages.kaggle)
  • keepkey_agent (python38Packages.keepkey_agent)
  • ldeep
  • ledger_agent (python38Packages.ledger_agent)
  • libminc
  • lieer
  • luigi
  • magic-wormhole (python38Packages.magic-wormhole)
  • manim
  • mapproxy
  • mathlibtools (python38Packages.mathlibtools)
  • minc_tools
  • minc_widgets
  • mnemosyne
  • mni_autoreg
  • n3
  • oobicpl
  • papis (python38Packages.papis)
  • past-time
  • plex-mpv-shim
  • python38Packages.Quandl (python38Packages.quandl)
  • python38Packages.acoustics
  • python38Packages.afdko
  • python38Packages.aioinflux
  • python38Packages.aiosmb
  • python38Packages.altair
  • python38Packages.ansible-kernel
  • python38Packages.ansible-runner
  • python38Packages.atomman
  • python38Packages.b2sdk
  • python38Packages.basemap
  • python38Packages.batchgenerators
  • python38Packages.bkcharts
  • python38Packages.caffe
  • python38Packages.cartopy
  • python38Packages.catboost
  • python38Packages.cirq
  • python38Packages.cufflinks
  • python38Packages.dask
  • python38Packages.dask-gateway
  • python38Packages.dask-glm
  • python38Packages.dask-ml
  • python38Packages.dask-mpi
  • python38Packages.datasets
  • python38Packages.datashader
  • python38Packages.dipy
  • python38Packages.distributed
  • python38Packages.dnachisel
  • python38Packages.duckdb
  • python38Packages.fastparquet
  • python38Packages.ffmpeg-progress-yield
  • python38Packages.flammkuchen
  • python38Packages.genome-collector
  • python38Packages.glymur
  • python38Packages.google-cloud-automl
  • python38Packages.google-cloud-bigquery
  • python38Packages.google-cloud-monitoring
  • python38Packages.gradient
  • python38Packages.gradient-utils
  • python38Packages.graphtage
  • python38Packages.gtts
  • python38Packages.habanero
  • python38Packages.holoviews
  • python38Packages.huggingface-hub
  • python38Packages.hvplot
  • python38Packages.hyperopt
  • python38Packages.image-match
  • python38Packages.imagecorruptions
  • python38Packages.imbalanced-learn
  • python38Packages.influxdb
  • python38Packages.influxgraph
  • python38Packages.intake
  • python38Packages.internetarchive
  • python38Packages.libagent
  • python38Packages.lsassy
  • python38Packages.mesa
  • python38Packages.moviepy
  • python38Packages.mplfinance
  • python38Packages.msldap
  • python38Packages.nagiosplugin
  • python38Packages.nbsmoke
  • python38Packages.ndjson
  • python38Packages.nilearn
  • python38Packages.nipype
  • python38Packages.nltk
  • python38Packages.nototools
  • python38Packages.osmpythontools
  • python38Packages.owslib
  • python38Packages.pandas
  • python38Packages.pandas-datareader
  • python38Packages.panel
  • python38Packages.papermill
  • python38Packages.parfive
  • python38Packages.partd
  • python38Packages.perfplot
  • python38Packages.phik
  • python38Packages.pims
  • python38Packages.plexapi
  • python38Packages.ppscore
  • python38Packages.proglog
  • python38Packages.pvlib
  • python38Packages.pyarrow
  • python38Packages.pyatv
  • python38Packages.pybids
  • python38Packages.pyezviz
  • python38Packages.pyfftw
  • python38Packages.pyppeteer
  • python38Packages.pyproj
  • python38Packages.pypykatz
  • python38Packages.pyro-ppl
  • python38Packages.pytest-astropy-header
  • python38Packages.python-daemon
  • python38Packages.python-miio
  • python38Packages.python-mpv-jsonipc
  • python38Packages.pytorch-lightning
  • python38Packages.pytrends
  • python38Packages.qiskit
  • python38Packages.qiskit-aer
  • python38Packages.qiskit-aqua
  • python38Packages.qiskit-ibmq-provider
  • python38Packages.qiskit-ignis
  • python38Packages.qiskit-terra
  • python38Packages.rapidfuzz
  • python38Packages.rbtools
  • python38Packages.rdkit
  • python38Packages.rising
  • python38Packages.rpy2
  • python38Packages.sacremoses
  • python38Packages.sagemaker
  • python38Packages.salmon-mail
  • python38Packages.scancode-toolkit
  • python38Packages.scikit-bio
  • python38Packages.scikitimage
  • python38Packages.seaborn
  • python38Packages.sfepy
  • python38Packages.slicedimage
  • python38Packages.slicer
  • python38Packages.sparse
  • python38Packages.spyder-kernels_0_5
  • python38Packages.spyder_3
  • python38Packages.statsmodels
  • python38Packages.stumpy
  • python38Packages.stytra
  • python38Packages.tables
  • python38Packages.tensorly
  • python38Packages.test-tube
  • python38Packages.thinc
  • python38Packages.toggl-cli
  • python38Packages.tqdm
  • python38Packages.traittypes
  • python38Packages.transformers
  • trezor_agent (python38Packages.trezor_agent)
  • python38Packages.tubeup
  • python38Packages.twine
  • python38Packages.uproot3
  • python38Packages.vega
  • python38Packages.vega_datasets
  • python38Packages.vidstab
  • python38Packages.wrf-python
  • python38Packages.xarray
  • python38Packages.yfinance
  • python39Packages.Quandl (python39Packages.quandl)
  • python39Packages.acoustics
  • python39Packages.afdko
  • python39Packages.aioinflux
  • python39Packages.aiosmb
  • python39Packages.altair
  • python39Packages.ansible-kernel
  • python39Packages.ansible-runner
  • python39Packages.atomman
  • python39Packages.b2sdk
  • python39Packages.batchgenerators
  • python39Packages.bkcharts
  • python39Packages.caffe
  • python39Packages.cartopy
  • python39Packages.catboost
  • python39Packages.cirq
  • python39Packages.clevercsv
  • python39Packages.csvs-to-sqlite
  • python39Packages.cufflinks
  • python39Packages.dask
  • python39Packages.dask-gateway
  • python39Packages.dask-mpi
  • python39Packages.datasets
  • python39Packages.deepdiff
  • python39Packages.dipy
  • python39Packages.distributed
  • python39Packages.dnachisel
  • python39Packages.duckdb
  • python39Packages.ffmpeg-progress-yield
  • python39Packages.flammkuchen
  • python39Packages.gdown
  • python39Packages.genome-collector
  • python39Packages.glymur
  • python39Packages.gradient
  • python39Packages.gradient-utils
  • python39Packages.graphtage
  • python39Packages.gtts
  • python39Packages.habanero
  • python39Packages.holoviews
  • python39Packages.huggingface-hub
  • python39Packages.hvplot
  • python39Packages.hyperopt
  • python39Packages.ignite
  • python39Packages.image-match
  • python39Packages.imagecorruptions
  • python39Packages.imbalanced-learn
  • python39Packages.influxdb
  • python39Packages.influxgraph
  • python39Packages.intake
  • python39Packages.internetarchive
  • python39Packages.kaggle
  • python39Packages.keepkey_agent
  • python39Packages.ledger_agent
  • python39Packages.libagent
  • python39Packages.lsassy
  • python39Packages.magic-wormhole
  • python39Packages.mathlibtools
  • python39Packages.mesa
  • python39Packages.moviepy
  • python39Packages.mplfinance
  • python39Packages.msldap
  • python39Packages.nagiosplugin
  • python39Packages.nbsmoke
  • python39Packages.ndjson
  • python39Packages.nipype
  • python39Packages.nltk
  • python39Packages.nototools
  • python39Packages.osmpythontools
  • python39Packages.owslib
  • python39Packages.pandas
  • python39Packages.pandas-datareader
  • python39Packages.panel
  • python39Packages.papermill
  • python39Packages.papis
  • python39Packages.parfive
  • python39Packages.partd
  • python39Packages.perfplot
  • python39Packages.pims
  • python39Packages.plexapi
  • python39Packages.ppscore
  • python39Packages.proglog
  • python39Packages.pvlib
  • python39Packages.pyarrow
  • python39Packages.pyatv
  • python39Packages.pybids
  • python39Packages.pyezviz
  • python39Packages.pyfftw
  • python39Packages.pyppeteer
  • python39Packages.pyproj
  • python39Packages.pypykatz
  • python39Packages.pyro-ppl
  • python39Packages.pytest-astropy-header
  • python39Packages.python-daemon
  • python39Packages.python-miio
  • python39Packages.python-mpv-jsonipc
  • python39Packages.pytorch-lightning
  • python39Packages.pytrends
  • python39Packages.qiskit-aer
  • python39Packages.qiskit-ibmq-provider
  • python39Packages.qiskit-ignis
  • python39Packages.qiskit-terra
  • python39Packages.rapidfuzz
  • python39Packages.rbtools
  • python39Packages.rdkit
  • python39Packages.rising
  • python39Packages.rpy2
  • python39Packages.sacremoses
  • python39Packages.sagemaker
  • python39Packages.salmon-mail
  • python39Packages.scancode-toolkit
  • python39Packages.scikit-bio
  • python39Packages.scikitimage
  • python39Packages.seaborn
  • python39Packages.sfepy
  • python39Packages.slicedimage
  • python39Packages.slicer
  • python39Packages.spyder-kernels_0_5
  • python39Packages.spyder_3
  • python39Packages.statsmodels
  • python39Packages.streamz
  • python39Packages.tables
  • python39Packages.test-tube
  • python39Packages.thinc
  • python39Packages.tqdm
  • python39Packages.traittypes
  • python39Packages.transformers
  • python39Packages.trezor_agent
  • python39Packages.tubeup
  • python39Packages.twine
  • python39Packages.uproot3
  • python39Packages.vega
  • python39Packages.vega_datasets
  • python39Packages.vidstab
  • python39Packages.wrf-python
  • python39Packages.xarray
  • python39Packages.yfinance
  • qgis
  • qgis-unwrapped
  • r128gain
  • sacad
  • spotdl
  • streamlit
  • sublime-music
  • tartube
  • tebreak
  • theharvester
  • tinyprog
  • tockloader
  • truvari
  • twitter-color-emoji
  • urlwatch
  • visidata

@SuperSandro2000 SuperSandro2000 merged commit ef9a1d4 into NixOS:master May 25, 2021
@drewrisinger drewrisinger deleted the dr-pr-fix-python-tables branch May 25, 2021 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants