Cannot access project data through signac interface

[klywang]

Description

Accessing arrays from project.data using signac returns a ValueError. The same arrays may be accessed usingh5py so I believe the data was written correctly. This error does not occur when accessing arrays using job.dataor when accessing scalars and strings from project.data.

To reproduce

In a project directory:

$ signac shell
>>> import numpy as np
>>> import h5py
>>> pr.data['a'] = np.zeros([10])
>>> with pr.data:
...     pr.data.a[:]
... 

This will return a ValueError.

Accessing the same arrays with h5py returns the correct values:

>>> f = h5py.File('signac_data.h5', mode='r')
>>> f['a']
<HDF5 dataset "a": shape (10,), type "<f8">
>>> f['a'][:]
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
>>> f.close()

Error output

Traceback (most recent call last):
  File "<console>", line 2, in <module>
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/Users/kelwang/miniconda3/envs/ENV/lib/python3.8/site-packages/h5py/_hl/dataset.py", line 489, in __getitem__
    if is_empty_dataspace(self.id):
  File "/Users/kelwang/miniconda3/envs/ENV/lib/python3.8/site-packages/h5py/_hl/base.py", line 87, in is_empty_dataspace
    if obj.get_space().get_simple_extent_type() == h5s.NULL:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 289, in h5py.h5d.DatasetID.get_space
ValueError: Not a dataset (not a dataset)

System configuration

Please complete the following information:

• Operating System [e.g. macOS]: macOS and Linux (Debian)
• Version of Python [e.g. 3.7]: Python 3.8.1
• Version of signac [e.g. 1.0]: 1.3.0

Or copy & paste the output of: python -c 'import platform; print(platform.platform()); import sys; print(sys.version); import signac; print(signac.__version__)'

[bdice]

I can reproduce this on my machine. I tried copying the signac_data.h5 file to a job directory and it seems like that worked fine. I am guessing that there is something different in the implementation of project.data vs. job.data that causes this issue.

[bdice]

I think I solved the problem. I think the underlying problem is that project.stores is returning a new instance of H5StoreManager in both first line with pr.data: and the second line pr.data.a[:].

The Job class keeps a single H5StoreManager in its private job._stores variable, but the Project class returns a new instance each time the project.stores attribute is called (in this case, it's called internally by project.data).

signac/signac/contrib/project.py

Lines 341 to 371 in 8ec1bc7

	@property
	def stores(self):
	"""Access HDF5-stores associated with this project.
	Use this property to access an HDF5 file within the project's root
	directory using the H5Store dict-like interface.
	This is an example for accessing an HDF5 file called ``'my_data.h5'``
	within the project's root directory:
	.. code-block:: python
	project.stores['my_data']['array'] = np.random((32, 4))
	This is equivalent to:
	.. code-block:: python
	H5Store(project.fn('my_data.h5'))['array'] = np.random((32, 4))
	Both the `project.stores` and the `H5Store` itself support attribute
	access. The above example could therefore also be expressed as:
	.. code-block:: python
	project.stores.my_data.array = np.random((32, 4))
	:return: The HDF5-Store manager for this project.
	:rtype: :class:`~..core.h5store.H5StoreManager`
	"""
	return H5StoreManager(self._rd)

@klywang, would you be willing to take these next steps?

1. Try that branch.
2. Add tests. Copy some of the testing logic from test_job.py into test_project.py since I don't think we're testing project.data or project.stores at all right now.
3. Make sure the tests fail on master and succeed on fix/project-data-interface.
4. Create a pull request, update changelog, etc.

[bdice]

@klywang Can you open a PR with your current progress on this?

[bdice]

Resolved by #278.