Apm em sprint14 p2 (#117)

* Initial commit new em_nion swift project reader * Added code for parsing nsproj, ndata, and h5 file metadata from nszip, inspect get_display_items_from_nszip.ipynb in em_nion/development for the next steps * Added remaining examples from Benedikts Nion session and image concept dict * Added code for analyzing which metadata come with specific different types of display_items from swift and added code for classifying cases on unique NeXus concepts and recovery of dimension scale axes, next steps: i) create mapping table, ii) modify nx classes * Added close to jsonmap_reader mapping table exemplarily for concept NxImageRealSpace, next step i) complete this, ii) carry the code over to reader.py * Time zone converting from swift local time to utc and NxImageReaslSpace mapping table and code for interpreting modifiers working, next step, carry over to reader code to get a first nxs example with instances of NxImageRealSpace as event_data_em * Parser skeleton carried over from ipynb to reader.py tested working, i.e. reads zip file in place and reads from these the individual ndata and hdf5 files in place, next step implement assigning into the template * Support for NxImageRealSpace implemented, next steps i) parse eln_data.yaml to pass verification, ii) add default plots, iii) check if nd array correctly arranged * Mapping table for concept eels * Added generic ELN loading * CI fixes, intentional downgrading of tzlocal to 4.3 as nionswift is not explicit constraining this enough but using the default tzlocal today (5.0) in combination with py3.10 yields problems as tzlocal>4.3 and backport.zoneinfo is not supported any longer, seems that swift developers should use newer zoneinfo syntax * Generated dev-requirements within py310 to check if one can avoid the issue that backports.zoneinfo does not compile with 3.10 * Fixed buges with 3d axes, fixed default plotting, and instance indexing, added paths to default, successful linting and mypy, next step add and test the standalone ipynb for running the em_nion reader and ask Benedikt for permission to upload the multimodal imaging dataset so that we can make a complete nomad example * integrated changes from testing with local NOMAD OASIS, removed development ipynbs for em_nion * added standalone ipynb, updated all other standalone ipynb, next step upload example from Benedikt and i) upload zenodo reference, ii) test with nomad * Feature-development closed and feedback from tests with update-north branch of nomad integrated, also missing link to Benedikts dataset for the new em_nion reader added in the standalone example ipynb * Removed nionswift_tool as it seems that this is not called * Removed deprecated distutils for setuputils * More robust locating of NXtest example appdef * fixing linting error toomanylocals * Resolve merge conflict * Fix merge conflict --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de>
FAIRmat-NFDI · Jun 1, 2023 · 2f3f99f · 2f3f99f
1 parent 494109b
commit 2f3f99f
Show file tree

Hide file tree

Showing 38 changed files with 2,376 additions and 207 deletions.
diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -1,5 +1,5 @@
 #
-# This file is autogenerated by pip-compile with Python 3.8
+# This file is autogenerated by pip-compile with Python 3.10
 # by the following command:
 #
 #    pip-compile --extra=dev --output-file=dev-requirements.txt --resolver=backtracking pyproject.toml
@@ -124,6 +124,7 @@ h5py==3.7.0
     #   ifes-apt-tc-data-modeling
     #   jupyterlab-h5web
     #   kikuchipy
+    #   nionswift
     #   orix
     #   pyfai
     #   pynxtools (pyproject.toml)
@@ -143,17 +144,12 @@ imageio==2.22.1
     # via
     #   hyperspy
     #   kikuchipy
+    #   nionswift
+    #   nionswift-io
+    #   nionui
     #   scikit-image
 importlib-metadata==5.0.0
-    # via
-    #   hyperspy
-    #   jupyterlab-server
-    #   nbconvert
-    #   numba
-importlib-resources==5.9.0
-    # via
-    #   jsonschema
-    #   radioactivedecay
+    # via hyperspy
 iniconfig==1.1.1
     # via pytest
 ipykernel==6.16.0
@@ -312,6 +308,22 @@ networkx==2.6.3
     # via
     #   radioactivedecay
     #   scikit-image
+niondata==0.15.3
+    # via
+    #   nionswift
+    #   nionswift-io
+nionswift==0.16.8
+    # via pynxtools (pyproject.toml)
+nionswift-io==0.15.1
+    # via nionswift
+nionui==0.6.10
+    # via nionswift
+nionutils==0.4.6
+    # via
+    #   niondata
+    #   nionswift
+    #   nionswift-io
+    #   nionui
 notebook==6.5.2
     # via jupyterlab
 notebook-shim==0.2.2
@@ -344,6 +356,10 @@ numpy==1.21.6
     #   kikuchipy
     #   lmfit
     #   matplotlib
+    #   niondata
+    #   nionswift
+    #   nionswift-io
+    #   nionui
     #   numba
     #   numcodecs
     #   numexpr
@@ -407,13 +423,12 @@ pillow==9.2.0
     # via
     #   imageio
     #   matplotlib
+    #   nionswift
     #   scikit-image
 pint==0.18
     # via hyperspy
 pip-tools==6.13.0
     # via pynxtools (pyproject.toml)
-pkgutil-resolve-name==1.3.10
-    # via jsonschema
 platformdirs==2.5.2
     # via
     #   jupyter-core
@@ -487,8 +502,11 @@ python-dateutil==2.8.2
 pytz==2022.4
     # via
     #   babel
+    #   nionswift
     #   pandas
     #   pynxtools (pyproject.toml)
+pytz-deprecation-shim==0.1.0.post0
+    # via tzlocal
 pywavelets==1.3.0
     # via scikit-image
 pyxem==0.15.0
@@ -534,6 +552,8 @@ scipy==1.7.3
     #   hyperspy
     #   kikuchipy
     #   lmfit
+    #   niondata
+    #   nionswift
     #   orix
     #   pyfai
     #   pynxtools (pyproject.toml)
@@ -647,10 +667,14 @@ types-urllib3==1.26.25.5
     # via types-requests
 typing-extensions==4.3.0
     # via
-    #   astroid
     #   mypy
     #   numcodecs
-    #   pylint
+tzdata==2023.3
+    # via pytz-deprecation-shim
+tzlocal==4.3
+    # via
+    #   nionswift
+    #   pynxtools (pyproject.toml)
 uncertainties==3.1.7
     # via lmfit
 url-normalize==1.4.3
@@ -682,9 +706,7 @@ zarr==2.12.0
 zipfile37==0.1.3
     # via pynxtools (pyproject.toml)
 zipp==3.8.1
-    # via
-    #   importlib-metadata
-    #   importlib-resources
+    # via importlib-metadata
 
 # The following packages are considered to be unsafe in a requirements file:
 # pip

diff --git a/examples/apm/Write.NXapm.Example.1.ipynb b/examples/apm/Write.NXapm.Example.1.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Using dataconverter/apm for mapping atom probe microscopy data to NeXus/NXapm"
+    "## Using dataconverter/apm for mapping atom probe microscopy data to NeXus/HDF5/NXapm"
    ]
   },
   {
@@ -68,7 +68,7 @@
     "tags": []
    },
    "source": [
-    "Example data can be found on Zenodo http://dx.doi.org/10.5281/zenodo.6808516."
+    "Example data can be found on Zenodo https://www.zenodo.org/record/7908429."
    ]
   },
   {
@@ -79,7 +79,7 @@
    },
    "outputs": [],
    "source": [
-    "import shutil  # unpacks in current path unless an additional path argument is provided"
+    "import zipfile as zp"
    ]
   },
   {
@@ -88,8 +88,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "! curl --output APM.LEAP.Datasets.1.zip https://zenodo.org/record/6808516/files/APM.LEAP.Datasets.1.zip\n",
-    "shutil.unpack_archive(\"APM.LEAP.Datasets.1.zip\")"
+    "! curl --output usa_denton_smith_apav_si.zip https://zenodo.org/record/7908429/files/usa_denton_smith_apav_si.zip"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "zp.ZipFile(\"usa_denton_smith_apav_si.zip\").extractall(path=\"\", members=None, pwd=None)"
    ]
   },
   {
@@ -101,7 +109,7 @@
     "  The eln_data_apm.yaml file in the example can be edited with a text editor.<br>\n",
     "* A file with **reconstructed ion positions** in community, technology partner format with<br>\n",
     "  the ion positions and mass-to-charge state ratio values for the tomographic reconstruction.<br>\n",
-    "  POS, ePOS, or APT are allowed.<br>\n",
+    "  POS, ePOS, or APT are allowed. Inspect some of the above-mentioned examples on Zenodo.<br>\n",
     "* A file with **ranging definitions** in community, technology partner format with<br>\n",
     "  the definitions how mass-to-charge-state-ratio values map on ion species.<br>\n",
     "  RNG, RRNG and is possible. A MatLab script can be used to inject other representations<br>\n",
@@ -197,26 +205,34 @@
    "source": [
     "#parser-nexus/tests/data/tools/dataconverter/readers/em_om/\n",
     "eln_data_file_name = [\"eln_data_apm.yaml\"]\n",
-    "input_recon_file_name = [\"R31_06365-v02.pos\",\n",
+    "input_recon_file_name = [\"Si.apt\",\n",
+    "                         \"Si.epos\",\n",
+    "                         \"Si.pos\",\n",
+    "                         \"R31_06365-v02.pos\",\n",
     "                         \"R18_58152-v02.epos\",\n",
     "                         \"70_50_50.apt\"]\n",
     "#                         \"R56_01769-v01.pos\"]\n",
-    "input_range_file_name = [\"R31_06365-v02.rrng\",\n",
+    "input_range_file_name = [\"Si.RRNG\",\n",
+    "                         \"Si.RNG\",\n",
+    "                         \"Si.RNG\",\n",
+    "                         \"R31_06365-v02.rrng\",\n",
     "                         \"R31_06365-v02.rrng\",\n",
     "                         \"R31_06365-v02.rrng\"]\n",
     "#                         \"R56_01769.rng.fig.txt\"]\n",
     "output_file_name = [\"apm.case1.nxs\",\n",
     "                    \"apm.case2.nxs\",\n",
     "                    \"apm.case3.nxs\",\n",
-    "                    \"apm.case4.nxs\"]\n",
-    "for case_id in [0]:  # , 1, 2, 3]:\n",
+    "                    \"apm.case4.nxs\",\n",
+    "                    \"apm.case5.nxs\",\n",
+    "                    \"apm.case6.nxs\"]\n",
+    "for case_id in [0]:\n",
     "    ELN = eln_data_file_name[0]\n",
     "    INPUT_RECON = input_recon_file_name[case_id]\n",
     "    INPUT_RANGE = input_range_file_name[case_id]\n",
     "    OUTPUT = output_file_name[case_id]\n",
     "\n",
     "    ! dataconverter --reader apm --nxdl NXapm --input-file $ELN --input-file \\\n",
-    "    $INPUT_RECON --input-file $INPUT_RANGE --output $OUTPUT --output $OUTPUT"
+    "    $INPUT_RECON --input-file $INPUT_RANGE --output $OUTPUT"
    ]
   },
   {
@@ -242,10 +258,7 @@
    "outputs": [],
    "source": [
     "# H5Web(OUTPUT)\n",
-    "H5Web(\"apm.case1.nxs\")\n",
-    "# H5Web(\"apm.case2.nxs\")\n",
-    "# H5Web(\"apm.case3.nxs\")\n",
-    "# H5Web(\"apm.case4.nxs\")"
+    "H5Web(\"apm.case1.nxs\")"
    ]
   },
   {
@@ -258,7 +271,6 @@
   {
    "cell_type": "markdown",
    "metadata": {
-    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
@@ -432,6 +444,19 @@
     "The apm reader has a functionality to generate synthetic dataset which are meant for pursuing code development."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-danger\">\n",
+    "This functionality uses recent features of ase which demands an environment that is currently not supported<br>\n",
+    "by NOMAD OASIS. As the here exemplified settings for this example are configured to represent an environment<br>\n",
+    "matching close to NOMAD users who are interested in this developer functionality should do the following:<br>\n",
+    "Run this example in a standalone environment where ase is upgraded to the latest version and then use<br>\n",
+    "the generated NeXus files either as is or upload them to NOMAD OASIS.<br>\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

diff --git a/examples/em_nion/README.md b/examples/em_nion/README.md
@@ -0,0 +1,24 @@
+## em_nion reader
+
+This is an example how the em_nion parser/reader/data extractor can be used as a standalone
+tool to convert data and metadata from a compressed nionswift project into an NXem-formatted
+NeXus/HDF5 file. Further details to the functionalities of the parser are documented
+in the parsers sub-directory:
+
+```
+pynxtools/pynxtools/dataconverter/readers/em_nion
+```
+
+**Write.NXem_nion.Example.1.ipynb** is the Jupyter notebook which exemplies
+how the parser can be used as a standalone version, i.e. without NOMAD.
+
+**eln_data_em_nion.yaml** is a YAML/text file which contains relevant data which are not
+contained typically in files from technology partners. These data have been collected
+either by editing the file manually or by using an electronic lab notebook (ELN),
+such as the NOMAD ELN.
+A few example files from real atom probe reconstructions and ranging definitions are
+offered as downloads to run the example with the above-mentioned Juypter notebook.
+
+Every other ELN can be used with this parser provided that this ELN writes its data
+into a YAML file with the same keywords and structure as is exemplified in the
+above-mentioned YAML file.