Update docs

- Add some notes on IMAS-less installs
- Update installing.rst to match the README.md in
  https://git.iter.org/projects/IMAS/repos/imaspy/pull-requests/73/overview
- Add IMASPy in 5 minutes page
- Reduce default sphinx-build output verbosity (to prevent sphinx
  errors and warnings from being drowned out)

Author: maarten-ic

docs/Makefile

@@ -20,7 +20,7 @@ help:
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
 	@echo Running '$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)'
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -Tvv
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
 clean: Makefile
 	@echo Running '$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)'

docs/source/conf.py

@@ -75,7 +75,7 @@
 # unique name: (base URL, label prefix)
 extlinks = {
     "src": (blob_url + "%s", f"{src_group}/{src_project}/%s"),
-    "issue": (issue_url + "%s", "#%s"),
+    "issue": (issue_url + "%s", "%s"),
     "merge": (mr_url + "%s", "!%s"),
     "netcdf4": (netcdf4_docs + "%s", "netcdf4 %s"),
     "dd": (dd_url + "%s", "%s"),
@@ -104,6 +104,7 @@
 # ones.
 extensions = [
     "sphinx.ext.autodoc",  # To auto-generate docs from Python docstrings
+    "sphinx.ext.autosectionlabel",
     "sphinx.ext.todo",  # nature theme
     "sphinx.ext.githubpages",  # nature theme
     "sphinx.ext.napoleon",  # Support for NumPy and Google style docstrings

docs/source/index.rst

@@ -1,17 +1,19 @@
-.. Master "index". This will be converted to a landing index.html by sphinx. We define TOC here, but it'll be put in the sidebar by the theme
+.. 
+   Master "index". This will be converted to a landing index.html by sphinx. We
+   define TOC here, but it'll be put in the sidebar by the theme
 
-===========================
+=============
 IMASPy Manual
-===========================
-IMASPy is (yet another) pure-python library to handle arbitrarily nested
+=============
+
+IMASPy is a pure-python library to handle arbitrarily nested
 data structures. IMASPy is designed for, but not necessarily bound to,
 interacting with Interface Data Structures (IDSs) as defined by the
 Integrated Modelling & Analysis Suite (IMAS) Data Model.
 
 It provides:
 
 - An easy-to-install and easy-to-get started package by
-
   * Not requiring an IMAS installation
   * Not strictly requiring matching a Data Dictionary (DD) version
 - A pythonic alternative to the IMAS Python High Level Interface (HLI)
@@ -26,11 +28,14 @@ It provides:
 
    self
    installing
+   intro
    api_overview
    mdsplus
 
+
 README
------------
+------
+
 The README is best read on :src:`#imaspy`.
 
 For developers
@@ -47,14 +52,16 @@ For developers
    api-hidden
 
 
-
 LICENSE
 -------
-.. literalinclude:: ../../LICENSE
+
+.. literalinclude:: ../../LICENSE.md
+   :language: text
 
 
 Sitemap
 -------
+
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`

docs/source/installing.rst

@@ -1,110 +1,71 @@
 Installing IMASPy
 =================
 
-IMASPy is a Python package with a compiled component, specifically the
-:al_cython:`Cython layer of the IMAS Access Layer (AL) <browse>`. This component
-is optional, but necessary to interact with the IMAS-AL, and thus with data stored
-to disk by said AL.
+IMASPy is a pure Python package. For full functionality of the package you need
+an installation of `the IMAS Access Layer <https://imas.iter.org/>`_. See
+:ref:`IMASPy 5 minute introduction` for an overview of functionality which does
+(not) require the IMAS Access Layer available.
 
-IMASPy also needs the :al_lib:`IMAS Python helper functions <browse>`. Currently, both
-dependencies are pulled it using :std:doc:`GitPython <gitpython:index>`, until the
-:issue:`separate Python HLI parts are available as packages <IMAS-584>`.
 
-.. :std:doc:`Cython <cython:index>` of the
+Installing on the ITER cluster and EuroFusion gateway
+-----------------------------------------------------
 
-
-Python is very flexible, and so is it's install procedure. This is a double-edged sword:
-Because of it flexibility, many different install options are available and in use in the
-community. For an introduction to the topic, read the
-:pypa:`guide to installing Python packaging by the Python Packaging Authority (PyPA) <tutorials/installing-packages/>`.
-
-Installing on the ITER cluster
-------------------------------
-To interact with the AL, we need the IMAS environment
-activated. I'm assuming a bash shell for all these commands:
-
-.. code-block:: bash
-
-    module load IMAS-2020a/3.30.0-4.8.5-2020a
-
-This should give us all we need, namely at time of testing:
-
-* ``python`` with a recent version: ``python --version #Python 3.8.2``
-* The Python package installer ``pip``:
-  ``pip --version #pip 20.0.2 from /work/imas/opt/EasyBuild/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages/pip-20.0.2-py3.8.egg/pip (python 3.8)``
-
-We are all developers, install imaspy in `pip editable <https://pip.pypa.io/en/stable/reference/pip_install/#options>`_ /`setuptools develop <https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode>`_ mode. **We are not using PEP517 build isolation to link to the systems numpy.**, **We are not using PyPA's recommended environment management, but instead install into the USER_SITE, this is ~/.local by default.**
+There is a `module` available on the ITER and Eurofusion Gateway clusters, so
+you can run
 
 .. code-block:: bash
 
-    git clone git@git.iter.org/IMAS/imaspy.git
-    pip install --user -e imaspy
-
+    module load IMASPy
 
-We should now be able to run the tests
+Additionally, if you wish to use the MDSPlus backend, you should load
 
 .. code-block:: bash
 
-    pip install --user -r requirements_test.txt
-    pytest imaspy/ --mini
-
-Note that at the time of writing access layer version 4.8.5 has an issue which causes
-several tests to fail.
-
-Develop install
-^^^^^^^^^^^^^^^
+    module load MDSplus-Java/7.96.17-GCCcore-10.2.0-Java-11
 
-.. note:: Check if this is still needed
 
-pip install --user -e .[backends_al,backends_xarray,test]
+Local installation
+------------------
 
-Check if you have access to the AL repository. This is currently needed to pull 'secret' dependencies. This will be checked by `pip` too but better to know it now.
-
-* Access to ``libimas.a``, located in ``$IMAS_PREFIX/lib``:
-  ``ls $IMAS_PREFIX/lib/libimas #/work/imas/core/IMAS/3.28.1-4.8.3/lib/libimas.a``
-  in our ``LD_LIBRARY_PATH=$IMAS_PREFIX/lib:$LD_LIBRARY_PATH``
-* A ``UAL_VERSION``, which will be used to pull the low-level AL files from the ITER
-  repository. ``echo $UAL_VERSION #4.8.3``
+We recommend using a :external:py:mod:`venv`. Then, clone the IMASPy repository
+and run `pip install`:
 
 .. code-block:: bash
 
-    (git ls-remote ssh://git@git.iter.org/imas/access-layer.git > /dev/null) && echo 'Connection successful!' || echo 'Connection failed!'
-    # Connection successful!
+    python3 -m venv ./venv
+    . venv/bin/activate
+    git clone ssh://git@git.iter.org/imas/imaspy.git
+    cd imaspy
+    pip install --upgrade pip
+    pip install --upgrade wheel setuptools
+    pip install .
 
 
-Quick primer on Python packages
--------------------------------
-A :pypa:`Python package <glossary/#term-import-package>`, commonly just called 'package', is a collection of :pypa:`Python modules <glossary/#term-module>`; reusable pieces of Python code. After installation, these packages are importable in scripts of other users, with the ``import package_name`` statement. On HPC systems, packages available to the user come from the following common locations:
+Development installation
+------------------------
 
-1. From the globally installed Python packages. These are installed by the system administrator (e.g. someone with `sudo` rights). For example on the ITER cluster:
+For development an installation in editable mode may be more convenient, and you
+will need some extra dependencies to run the test suite and build documentation.
 
 .. code-block:: bash
 
-    module purge
-    module load Python/3.6.4-intel-2018a
-    python -c 'import site; print(site.getsitepackages())'
-    # ['/work/imas/opt/EasyBuild/software/Python/3.6.4-intel-2018a/lib/python3.6/site-packages']
+    pip install -e .[test, docs]
 
-2. From imported modules. These are usually centrally managed and also handled by the system administrator. For example on the ITER cluster:
+Test your installation by trying
 
 .. code-block:: bash
 
-    module purge
-    module load Python/3.6.4-intel-2018a PyAL
-    python -m site
-    # sys.path = [
-    #   <snip>
-    #   '/work/imas/opt/EasyBuild/software/Python/3.6.4-intel-2018a/lib/python3.6/site-packages/numpy-1.14.0-py3.6-linux-x86_64.egg'
-    #   <snip>
-    # ]
-
-3. From the local user environment, usually in the users' HOME directory:
+    cd ~
+    python -c "import imaspy; print(imaspy.__version__)"
 
-.. code-block:: bash
 
-    python -c 'import site; print(site.USER_SITE)'
-    # /home/ITER/vandepk/.local/lib/python3.6/site-packages
+Installation without ITER access
+--------------------------------
 
-4. From the current working directory. E.g. if I have a file called ``fancy_code.py`` in my current folder, I can call ``from fancy_code import *`` from my other Python files.
+The installation script tries to access the `ITER IMAS Core Data Dictionary
+repository <https://git.iter.org/projects/IMAS/repos/data-dictionary/browse>`_
+to fetch the latest versions. If you do not have git+ssh access there, you can
+try to find this repository elsewhere, and do a ``git fetch --tags``.
 
-Installing a Python package just means putting the Python files somewhere the python binary can find it. It does this by walking down the `sys.path` until a package with the right name is found. See :std:doc:`python:library/importlib`
+Alternatively you could try to obtain an ``IDSDef.zip`` and place it in
+``~/.config/imaspy/``.

docs/source/intro.rst

@@ -0,0 +1,155 @@
+IMASPy 5 minute introduction
+----------------------------
+
+.. contents:: Contents
+    :local:
+    :depth: 1
+
+
+Verify your IMAS installation
+'''''''''''''''''''''''''''''
+
+Before continuing, verify that your imaspy install is working. Check the
+:ref:`Installing IMASPy` page for installation instructions if below fails for
+you. Start python and import imaspy. Note that the version in below output may
+be outdated.
+
+.. code-block:: python
+
+    >>> import imaspy
+    >>> print(imaspy.__version__)
+    0.6.2
+
+.. note::
+
+    If you have an IMASPy install without the IMAS Access Layer, importing
+    IMASPy will display an error message. You can still use IMASPy, but not all
+    functionality is available.
+
+
+Create and use an IDS
+'''''''''''''''''''''
+
+To create an IDS, you must first make an :py:class:`~imaspy.ids_root.IDSRoot`
+object. The IDS root is necessary for specifying which version of the IMAS Data
+Dictionary you want to use (the last available one, by default). See
+:ref:`Loading multiple DD versions in the same environment` for more information
+on different Data Dictionary versions.
+
+.. code-block:: python
+
+    >>> import imaspy
+    >>> import numpy as np
+    >>> ids_root = imaspy.ids_root.IDSRoot()
+    10:26:51 [INFO] Generating IDS structures for version 3.38.1 @ids_root.py:130
+    >>> # Create an empty core_profiles IDS
+    >>> core_profiles = ids_root.core_profiles
+    >>> # Caution: doing this a second time does not create a new one:
+    >>> core_profiles2 = ids_root.core_profiles
+    >>> core_profiles is core_profiles2
+    True
+
+We can now use this ``core_profiles`` IDS and assign some data to it:
+
+.. code-block:: python
+
+    >>> core_profiles.ids_properties.comment = "Testing IMASPy"
+    >>> core_profiles.ids_properties.homogeneous_time = imaspy.ids_defs.IDS_TIME_MODE_HOMOGENEOUS
+    >>> # array quantities are automatically converted to the appropriate numpy arrays
+    >>> core_profiles.time = [1, 2, 3]
+    >>> # the python list of ints is converted to a 1D array of floats
+    >>> core_profiles.time.value
+    array([1., 2., 3.])
+    >>> # resize the profiles_1d array of structures to match the size of `time`
+    >>> core_profiles.profiles_1d.resize(3)
+    >>> len(core_profiles.profiles_1d.value)
+    3
+    >>> # assign some data for the first time slice
+    >>> core_profiles.profiles_1d[0].grid.rho_tor_norm = [0, 0.5, 1.0]
+    >>> core_profiles.profiles_1d[0].j_tor = [0, 0, 0]
+
+.. note::
+
+    Until :issue:`IMAS-4680` is addressed, you should use :code:`.value` to get the
+    value of a quantity in IMASPy.
+
+As you can see in above example, IMASPy automatically checks the data you try to
+assign to an IDS with the data type specified in the Data Dictionary. When
+possible, your data is automatically converted to the expected type. You will
+get an error message if this is not possible:
+
+.. code-block:: python
+
+    >>> core_profiles.time = "Cannot be converted"
+    ValueError: could not convert string to float: 'Cannot be converted'
+    >>> core_profiles.time = 1-1j
+    TypeError: can't convert complex to float
+    >>> core_profiles.ids_properties.source = 1-1j  # automatically converted to str
+    >>> core_profiles.ids_properties.source.value
+    '(1-1j)'
+
+
+Store an IDS to disk
+''''''''''''''''''''
+
+.. note::
+
+    - This functionality requires the IMAS Access Layer.
+    - This API will change when IMASPy is moving to Access Layer 5 (expected Q2
+      2023).
+
+To store an IDS to disk, we need to indicate the following information to the
+IMAS Access Layer. Please check the IMAS Access Layer documentation for more
+information on this.
+
+- ``shot``
+- ``run``
+- ``user``
+- ``tokamak`` (also known as database)
+- ``version`` (major version of the access layer, typically ``"3"``)
+- Optional: which backend to use (e.g. the default MDSplus or HDF5).
+
+In IMASPy you do this as follows:
+
+.. code-block:: python
+
+    >>> # you can specify shot=10 and run=2 when creating the IDSRoot object
+    >>> #ids_root = imaspy.ids_root.IDSRoot(s=10, r=2)
+    >>> # you can also set this after creating the ids_root object
+    >>> # as long as you do it before create_env_backend
+    >>> ids_root.shot = 10
+    >>> ids_root.run = 2
+    >>> # Create a new IMAS data entry for storing the core_profiles IDS we created earlier
+    >>> # Here we specify user, tokamak, version and the backend
+    >>> import os
+    >>> ids_root.create_env_backend(user=os.environ['USER'], tokamak="ITER", version="3", backend_type=imaspy.ids_defs.HDF5_BACKEND)
+    10:29:13 [INFO] Opening AL backend HDF5 for ITER (shot 10, run 2, user sebregm, ver 3, mode w) @ids_root.py:337
+    (0, 1)
+    >>> # now store the core_profiles IDS we just populated
+    >>> ids_root.core_profiles.put()
+
+
+Load an IDS from disk
+'''''''''''''''''''''
+
+.. note::
+
+    - This functionality requires the IMAS Access Layer.
+    - This API will change when IMASPy is moving to Access Layer 5 (expected Q2
+      2023).
+
+To load an IDS from disk, you need to specify the same information as
+when storing the IDS (see previous section). Once a data entry is opened, you
+can use ``<IDS>.get()`` to load IDS data from disk: 
+
+.. code-block:: python
+
+    >>> # Now load the core_profiles IDS back into a fresh ids_root object
+    >>> ids_root2 = imaspy.ids_root.IDSRoot(s=10, r=2)
+    10:29:56 [INFO] Generating IDS structures for version 3.38.1 @ids_root.py:130
+    >>> ids_root2.open_env_backend(user=os.environ['USER'], tokamak="ITER", version="3", backend_type=imaspy.ids_defs.HDF5_BACKEND)
+    10:30:07 [INFO] Opening AL backend HDF5 for ITER (shot 10, run 2, user sebregm, ver 3, mode r) @ids_root.py:337
+    (0, 2)
+    >>> ids_root2.core_profiles.get()
+    >>> print(ids_root2.core_profiles.ids_properties.comment.value)
+    Testing IMASPy