Merge f4142e3 into e0ef97a

.github/workflows/tests.yml

@@ -2,9 +2,9 @@ name: tests
 
 on:
   push:
-    branches: [ master ]
+    branches: [ main ]
   pull_request:
-    branches: [ master ]
+    branches: [ main ]
 
 env:
   PIP_CACHE_DIR: ~/.cache/pip
@@ -21,6 +21,8 @@ jobs:
     steps:
     - name: Checkout
       uses: actions/checkout@v2
+      with:
+        submodules: recursive
 
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v2
@@ -46,7 +48,7 @@ jobs:
         pip install -e .[tests]
 
     - name: Run tests
-      run: pytest -s -v -n 4
+      run: pytest -s -v -n auto
 
     - name: Upload coverage
       if: ${{ success() && matrix.python-version == '3.7' }}

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "data"]
+	path = tests/data
+	url = https://github.com/eX-Mech/pymech-test-data.git

README.md

@@ -17,7 +17,7 @@ Read the full documentation at [Pymech doc](http://pymech.readthedocs.io/en/stab
 For some quick wins, download some sample data
 
 ```sh
-curl -LO https://github.com/eX-Mech/pymech/raw/master/tests/nek/channel3D_0.f00001
+curl -fLO https://raw.githubusercontent.com/eX-Mech/pymech-test-data/main/nek/channel3D_0.f00001
 ```
 
 Fire up a Python / IPython console and execute:

docs/core.rst

@@ -23,10 +23,12 @@ element(s).  The general information is constituted by:
 ``time``     float                          simulation time of the file
 ``istep``    integer                        simulation time step of the file
 ``wdsz``     integer                        word size, i.e. double or single precision file
-``endian``   string                         endianness of the file (little/big)
+``endian``   string                         endianness_ of the file (little/big)
 ``lims``     :ref:`datalims`                extrema of all quantities stored
 ==========   ============================   ===============================================
 
+.. _endianness: https://wizardzines.com/comics/little-endian/
+
 The element by element data is stored in:
 
 ==========   ============================   ===============================================

docs/dev.md

@@ -9,10 +9,10 @@ Ready to contribute? Here's how to set up Pymech for local development.
 1. Fork the `pymech` repo on GitHub.
 2. Clone your fork locally:
     ```{tab} HTTPS
-        git clone https://github.com/your_name_here/pymech.git
+        git clone --recursive https://github.com/your_name_here/pymech.git
     ```
     ```{tab} SSH
-        git clone git@github.com:your_name_here/pymech.git
+        git clone --recursive git@github.com:your_name_here/pymech.git
     ```
 3. To set up for local development, first, create a [virtual
    environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments):

docs/internals.rst

@@ -11,7 +11,7 @@ Internals
 
 
 :mod:`pymech.neksuite`
----------------------
+----------------------
 
 .. automodule:: pymech.neksuite
    :noindex:
@@ -24,5 +24,5 @@ Internals
 .. automodule:: pymech.dataset
    :noindex:
    :no-members:
-   :private-members: _open_nek_dataset, _NekDataStore
+   :private-members:  can_open_nek_dataset, _open_nek_dataset, _NekDataStore, PymechXarrayBackend
 

docs/release.rst

@@ -12,7 +12,7 @@ Release checklist for maintainers
 
 - Ensure tests pass locally and on CI::
 
-      pytest --cov --numprocesses=4
+      pytest --numprocesses=auto
 
 - Compile documentation::
 
@@ -49,7 +49,7 @@ Release checklist for maintainers
 
 - Upload to repository::
 
-      git push --tags && git push
+      git push --atomic origin main $VERSION
 
 .. _twine: https://twine.readthedocs.io/en/latest/
 .. _TestPyPI: https://packaging.python.org/guides/using-testpypi/

docs/tests.rst

@@ -10,6 +10,9 @@ the tests::
     pip install -e '.[tests]'
     pytest -s -v
 
+The directory `tests/data` is a git submodule repository
+(https://github.com/eX-Mech/pymech-test-data.git), which houses all the data
+files required to run the test suite.
 
 Continuous Integration
 ----------------------

docs/usage.ipynb

@@ -36,13 +36,13 @@
       "remote: Total 1322 (delta 187), reused 253 (delta 132), pack-reused 1001\u001b[K\n",
       "Receiving objects: 100% (1322/1322), 12.38 MiB | 8.30 MiB/s, done.\n",
       "Resolving deltas: 100% (806/806), done.\n",
-      "/home/avmo/src/snek5000/pymech/docs/pymech/tests/nek\n"
+      "/home/avmo/src/snek5000/pymech/docs/pymech/tests/data/nek\n"
      ]
     }
    ],
    "source": [
-    "!git clone https://github.com/eX-Mech/pymech.git\n",
-    "%cd pymech/tests/nek"
+    "!git clone --recursive https://github.com/eX-Mech/pymech.git\n",
+    "%cd pymech/tests/data/nek"
    ]
   },
   {
@@ -158,18 +158,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Majority of the attributes store metadata, such as:\n",
-    "\n",
-    "- endian\n",
-    "- iteration\n",
-    "- element shape / polynomial order\n",
-    "- boundary conditions\n",
-    "- curvature\n",
-    "- number of dimensions\n",
-    "- number of elements\n",
-    "- time\n",
-    "- number of variables in (X: positions, U: velocities, P: pressure, T: temperature, S: scalars)\n",
-    "- word size / precision"
+    "Here `check_connectivity` and `merge` are methods, and `elem` and `elmap` are data objects. The rest of the attributes store metadata. See {ref}`core` to understand what they imply."
    ]
   },
   {
@@ -2059,7 +2048,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.5"
+   "version": "3.8.9"
   }
  },
  "nbformat": 4,

pymech/dataset.py

@@ -2,7 +2,26 @@
 
 .. _xarray: https://xarray.pydata.org
 
+Installing ``pymech`` also registers as a ``xarray`` backend. This means
+in addition to :func:`open_dataset`, a file can be directly opened via
+``xarray`` as follows:
+
+>>> import xarray as xr
+>>> xr.open_dataset("case0.f00001")  # let xarray choose the backend / engine
+>>> xr.open_dataset("case0.f00001", engine="pymech")  # or explicitly mention the *engine*
+
+.. seealso::
+
+    `The backend API of xarray
+    <https://xarray.pydata.org/en/stable/internals/how-to-add-new-backend.html>`__
+    and the implementation :class:`PymechXarrayBackend` (:ref:`internals`)
+
+.. todo::
+
+    Opening as a object is not supported by :func:`pymech.neksuite.readnek`
+
 """
+import re
 from pathlib import Path
 from glob import glob
 
@@ -12,7 +31,33 @@
 from .neksuite import readnek
 
 
-__all__ = ("open_dataset", "open_mfdataset")
+__all__ = (
+    "open_dataset",
+    "open_mfdataset",
+)
+
+
+nek_ext_pattern = re.compile(
+    r"""
+   .*          # one or more characters
+   \.          # character "."
+   f           # character "f"
+   (\d{5}|ld)  # 5 digits or the characters "ld"
+""",
+    re.VERBOSE,
+)
+
+
+def can_open_nek_dataset(path):
+    """A regular expression check of the file extension.
+
+    .. hint::
+
+        - Would not match: .f90 .f .fort .f0000
+        - Would match: .fld .f00001 .f12345
+
+    """
+    return nek_ext_pattern.match(str(path))
 
 
 def open_dataset(path, **kwargs):
@@ -27,11 +72,10 @@ def open_dataset(path, **kwargs):
             Keyword arguments passed on to the compatible open function.
 
     """
-    path = Path(path)
-    if path.suffix.startswith(".f"):
+    if can_open_nek_dataset(path):
         _open = _open_nek_dataset
     else:
-        raise NotImplementedError(f"Filetype: {path.suffix} is not supported.")
+        raise NotImplementedError(f"Filetype: {Path(path).suffix} is not supported.")
 
     return _open(path, **kwargs)
 
@@ -58,7 +102,7 @@ def open_mfdataset(
 
     .. todo::
 
-            To be removed when a backend entrypoint_ is implementated.
+        To be removed when a backend entrypoint_ is implementated.
 
     .. _implementation: https://github.com/pydata/xarray/blob/484d1ce5ff8969b6ca6fa942b344379725f33b9c/xarray/backends/api.py#L726
     .. _docs: https://xarray.pydata.org/en/v0.15.1/generated/xarray.open_mfdataset.html
@@ -118,7 +162,11 @@ def open_mfdataset(
             # Redo ordering from coordinates, ignoring how they were ordered
             # previously
             combined = xr.combine_by_coords(
-                datasets, compat=compat, data_vars=data_vars, coords=coords, join=join
+                datasets,
+                compat=compat,
+                data_vars=data_vars,
+                coords=coords,
+                join=join,
             )
         else:
             raise ValueError(
@@ -141,7 +189,7 @@ def open_mfdataset(
     return combined
 
 
-def _open_nek_dataset(path):
+def _open_nek_dataset(path, drop_variables=None):
     """Interface for converting Nek field files into xarray_ datasets."""
     field = readnek(path)
     if isinstance(field, int):
@@ -154,16 +202,32 @@ def _open_nek_dataset(path):
     ]
 
     # See: https://github.com/MITgcm/xmitgcm/pull/200
-    if xr.__version__ < "0.15.2":
-        ds = xr.combine_by_coords(elem_dsets)
-    else:
-        ds = xr.combine_by_coords(elem_dsets, combine_attrs="drop")
-
+    ds = xr.combine_by_coords(elem_dsets, combine_attrs="drop")
     ds.coords.update({"time": field.time})
 
+    if drop_variables:
+        ds = ds.drop_vars(drop_variables)
+
     return ds
 
 
+class PymechXarrayBackend(xr.backends.BackendEntrypoint):
+    def guess_can_open(self, filename_or_obj):
+        return can_open_nek_dataset(filename_or_obj)
+
+    def open_dataset(
+        self,
+        filename_or_obj,
+        *,
+        drop_variables=None,
+        # other backend specific keyword arguments
+        # `chunks` and `cache` DO NOT go here, they are handled by xarray
+    ):
+        return _open_nek_dataset(filename_or_obj, drop_variables)
+
+    open_dataset_parameters = ("filename_or_obj", "drop_variables")
+
+
 class _NekDataStore(xr.backends.common.AbstractDataStore):
     """Xarray store for a Nek field element.
 

pymech/vtksuite.py

@@ -53,7 +53,7 @@ def exa2vtk(field, downsample=False):
         dataset.plot()
 
     .. _MayaVi: https://docs.enthought.com/mayavi/mayavi/mlab.html
-    .. _PyVista: https://docs.pyvista.org/getting-started/simple.html
+    .. _PyVista: https://docs.pyvista.org/getting-started/index.html
 
     Parameters
     ----------

setup.cfg

@@ -9,7 +9,9 @@ license =  GNU General Public License v3 or later (GPLv3+)
 license_file = LICENSE
 url = https://github.com/eX-Mech/pymech
 project-urls =
-    documentation = https://pymech.readthedocs.io/en/stable
+    Bug Tracker = https://github.com/eX-Mech/pymech/issues
+    Documentation = https://pymech.readthedocs.io/en/stable
+    Changelog = https://pymech.readthedocs.io/en/stable/changelog.html
 classifiers =
     Intended Audience :: Science/Research
     Programming Language :: Python :: 3
@@ -57,6 +59,10 @@ dev =
     %(tests)s
     pre-commit
 
+[options.entry_points]
+xarray.backends =
+    pymech = pymech.dataset:PymechXarrayBackend
+
 [flake8]
 max-line-length = 88
 ignore = W503, E203, E501

tests/conftest.py

@@ -0,0 +1,8 @@
+from pathlib import Path
+
+import pytest
+
+
+@pytest.fixture
+def test_data_dir():
+    return Path(__file__).parent / "data"