Add page for related project/tools #1626

aragilar · 2020-08-07T14:06:19Z

This closes #1621 in that we would have a page for listing other projects users may be interested in. I've also moved the ipython extension docs onto this page, as they were quite hidden otherwise on the config page.

Definitely not complete, people should feel free to push links to projects.

codecov · 2020-08-07T14:06:23Z

Codecov Report

Base: 89.74% // Head: 90.01% // Increases project coverage by +0.26% 🎉

Coverage data is based on head (8f069a5) compared to base (2b04ad1).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1626      +/-   ##
==========================================
+ Coverage   89.74%   90.01%   +0.26%     
==========================================
  Files          17       17              
  Lines        2390     2394       +4     
==========================================
+ Hits         2145     2155      +10     
+ Misses        245      239       -6

Impacted Files	Coverage Δ
h5py/_hl/dataset.py	`93.43% <0.00%> (+0.19%)`	⬆️
h5py/_hl/files.py	`88.08% <0.00%> (+0.72%)`	⬆️
h5py/_hl/selections.py	`87.50% <0.00%> (+1.50%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

scopatz · 2020-08-07T15:17:54Z

CC @asmeurer - please add versioned-hdf5 to this page

takluyver

Other possible things to mention:

PyTables: Alternative Python interface, kind of higher level, more uses HDF5 as a storage format for its own data model (whereas h5py tries to expose the HDF5 model).
netCDF4-python: For working with netCDF4, a more specific format built on top of HDF5.

takluyver · 2020-08-08T09:16:46Z

docs/related_projects.rst

+   c = get_config()
+   c.InteractiveShellApp.extensions = ['h5py.ipy_completer']
+
+For <ipython-0.11, the completer can be enabled by adding the following lines


I think we can remove this chunk now - IPython 0.11 was released just over 9 years ago. Doesn't necessarily need to be in this PR, though.

takluyver · 2020-08-08T09:26:43Z

docs/related_projects.rst

+h5py does not contain a tool for exploring or visualising HDF5 files, but tools
+built on h5py that can display the structure of h5py include:
+
+ * `h5glance <https://github.com/European-XFEL/h5glance>`_


Suggested change

* `h5glance <https://github.com/European-XFEL/h5glance>`_

* `h5glance <https://github.com/European-XFEL/h5glance>`_ shows the structure of

HDF5 files in IPython & Jupyter, as well as at the command line.

takluyver · 2020-08-08T09:36:18Z

docs/related_projects.rst

+Field/Format Specific Libraries
+-------------------------------
+
+The are projects which build upon h5py to read and write to HDF5 files with a
+specific schema, users of these formats may be interested in using these
+higher-level wrappers of h5py:
+
+ *


Is this actually useful? I know of NeXus and CXI as specific schemas from photon & neutron science, but I don't know which specific libraries people use in Python to work with either. And this list is very likely to get outdated, because each contributor only knows about a small part of it.

I'd be inclined to leave it - especially as we don't have any examples at the moment.

Sure, let's drop it. I don't know of any astronomy standards for HDF5 (radio astronomy is apparently experimenting with HDF5, but most people are sticking to FITS). I figured putting this here would provoke lots of suggestions for libraries, but if there are no obvious ones, then there's not much point having it.

takluyver · 2020-08-08T09:58:10Z

docs/related_projects.rst

+------------------------------------
+
+h5py does not contain a tool for exploring or visualising HDF5 files, but tools
+built on h5py that can display the structure of h5py include:


Not built on h5py, but others it might be worth mentioning include HDFview, the official one from HDF group (a bit clunky IMO, but likely to be well supported), and ViTables, a GUI written in Python.

The project I'm part of also has a list of viewer tools. But there are quite a few, and I'm not very good at keeping up with what's still developed and supported.

I've added those links.

aragilar · 2020-08-09T02:18:25Z

I've added pytables to the intro, with links to the two faq entries.

Some other projects we may want to link to:
https://github.com/DaanVanVugt/h5pickle - I think I've seen this mentioned in issues
https://github.com/telegraphic/hickle - same (I'm unclear exactly how things are stored).
https://github.com/h5preserve/h5preserve - the wrapper I wrote (does versioning and uses as similar philosophy to http://eev.ee/blog/2015/10/15/dont-use-pickle-use-camel/)

Apparently you can't use the same text for different urls?

takluyver · 2020-08-09T10:22:02Z

I think all of those could be on the page, but I'm struggling to see how we categorise things and make it clear what they all are.

On a related note: seeing h5preserve for the first time, it took a surprising amount of reading to work out what it was. The README describe it as "a thin wrapper around h5py" - but what does it make more convenient? How do you use it? After reading the blog post about Camel I can kind of guess the rough shape of it. The docs landing page provides some explanation, but it's still not exactly clear what it's offering. The quickstart page finally explains it and demonstrates it, but it's not very amenable to scanning - you really need to read through it to make sense of it.

Anyway, feel free to take all that with a big grain of salt. Sometimes it can be useful to know what a project looks like to someone seeing it for the first time, but I'm just one person and it's totally possible that I missed something that's generally obvious.

aragilar · 2020-08-10T13:03:10Z

The page was initially for viewers and external filters (I thought there were other projects providing filters, as well as hdf5plugin), but I figured there were a bunch of projects building on h5py, it made sense to try to collect them together (rather than potentially having them scattered throughout the docs). As is, I think it's fine (and if someone wants to merge this, feel free to do it), but I figured there were widely used wrappers for specific formats, which may be better for users that using raw h5py (and potentially missing out key metadata).

With respect to h5preserve, that's definitely useful feedback, I still haven't really clarified how to think about it independent of camel, it does its job and I can think of it as a black box most of the time (it was initially a day or so of hacking, after repeatedly getting annoyed that I needed to worry about handling older versions of output in the middle of science code). I should probably sit down and refresh the docs at some point.

asmeurer · 2020-08-10T19:53:18Z

Here is a blurb for versioned-hdf5. It isn't clear to me where this would go on the current page:

Versioned-hdf5 (https://deshaw.github.io/versioned-hdf5/) is a library that builds a versioned abstraction on top of h5py. It provides a wrappers around the h5py API that allows storing different versions of groups and datasets within an HDF5 file.

Add page for related project/tools

5ad38d1

scopatz approved these changes Aug 7, 2020

View reviewed changes

takluyver reviewed Aug 8, 2020

View reviewed changes

Expand related projects and add info on PyTables

22dfcf6

Fix pytables link

89b8a23

Apparently you can't use the same text for different urls?

aragilar mentioned this pull request Jan 30, 2021

Strings not read correctly? #1769

Closed

takluyver and others added 5 commits December 21, 2022 12:12

Fix Sphinx extlinks config

272b5d2

Add some libraries which provide generic functionality on top of h5py

7a3bb41

Remove some unnecessary indentation

cb54d80

Some rearrangement of related projects page

3a00007

Merge branch 'master' into related_projects

8f069a5

takluyver closed this Jan 20, 2023

takluyver reopened this Jan 20, 2023

tacaswell approved these changes Jan 20, 2023

View reviewed changes

takluyver merged commit 2578d31 into h5py:master Jan 20, 2023

aragilar deleted the related_projects branch January 23, 2023 12:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add page for related project/tools #1626

Add page for related project/tools #1626

aragilar commented Aug 7, 2020

codecov bot commented Aug 7, 2020 •

edited

Loading

scopatz commented Aug 7, 2020

takluyver left a comment

takluyver Aug 8, 2020

aragilar Aug 9, 2020

takluyver Aug 8, 2020

takluyver Aug 8, 2020

aragilar Aug 9, 2020

takluyver Aug 8, 2020

takluyver Aug 8, 2020

aragilar Aug 9, 2020

aragilar commented Aug 9, 2020

takluyver commented Aug 9, 2020

aragilar commented Aug 10, 2020

asmeurer commented Aug 10, 2020

	* `h5glance <https://github.com/European-XFEL/h5glance>`_
	* `h5glance <https://github.com/European-XFEL/h5glance>`_ shows the structure of
	HDF5 files in IPython & Jupyter, as well as at the command line.

Add page for related project/tools #1626

Add page for related project/tools #1626

Conversation

aragilar commented Aug 7, 2020

codecov bot commented Aug 7, 2020 • edited Loading

Codecov Report

scopatz commented Aug 7, 2020

takluyver left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aragilar commented Aug 9, 2020

takluyver commented Aug 9, 2020

aragilar commented Aug 10, 2020

asmeurer commented Aug 10, 2020

codecov bot commented Aug 7, 2020 •

edited

Loading