Merge 9d657ea into b76140f

README.md

@@ -18,9 +18,9 @@
 
 It aims at **providing modular and robust tools for the most common analyses needed with DEMs**, including both geospatial
 operations specific to DEMs and a wide range of 3D alignment and correction methods from published, peer-reviewed studies.
-The core manipulation of DEMs (e.g., vertical alignment, terrain analysis) are **conveniently centered around `DEM` and `dDEM` classes** (that, notably, re-implements all tools
+The core manipulation of DEMs (e.g., vertical alignment, terrain analysis) are **conveniently centered around a `DEM` class** (that, notably, re-implements all tools
 of [gdalDEM](https://gdal.org/programs/gdaldem.html)). More complex pipelines (e.g., 3D rigid coregistration, bias corrections, filtering) are **built around
-modular `Coreg`, `BiasCorr` and `Filter` classes that easily interface between themselves**. Finally, xDEM includes advanced
+modular `Coreg`, `BiasCorr` classes that easily interface between themselves**. Finally, xDEM includes advanced
 uncertainty analysis tools based on spatial statistics of [SciKit-GStat](https://scikit-gstat.readthedocs.io/en/latest/).
 
 Additionally, xDEM inherits many convenient functionalities from [GeoUtils](https://github.com/GlacioHack/geoutils) such as
@@ -49,7 +49,7 @@ See [mamba's documentation](https://mamba.readthedocs.io/en/latest/) to install
 
 When using a method implemented in xDEM, please **cite both the package and the related study**:
 
-Citing xDEM: [![Zenodo](https://zenodo.org/badge/doi/10.5281/zenodo.4809697.svg)](https://zenodo.org/record/4809698)
+Citing xDEM: [![Zenodo](https://zenodo.org/badge/doi/10.5281/zenodo.4809697.svg)](https://zenodo.org/doi/10.5281/zenodo.4809697)
 
 Citing the related study:
 

doc/source/about_xdem.md

@@ -2,72 +2,32 @@
 
 # About xDEM
 
+## What is xDEM?
 
-xDEM is a [Python](https://www.python.org/) package for the analysis of DEMs, with name standing for _cross-DEM analysis_[^sn1]
-and echoing its dependency on [xarray](https://docs.xarray.dev/en/stable/). It is designed for all Earth and planetary
-observation science, although our group currently has a strong focus on glaciological applications.
+xDEM is a Python package for the analysis of elevation data, and in particular that of digital elevation models (DEMs),
+with name standing for _cross-DEM analysis_[^sn1] and echoing its dependency on [Xarray](https://docs.xarray.dev/en/stable/).
 
-[^sn1]: The core features of xDEM rely on cross-analysis of surface elevation, for example for DEM alignment or error analysis.
+[^sn1]: Several core features of xDEM, in particular coregistration and uncertainty analysis, rely specifically on cross-analysis of elevation data over static surfaces.
 
+## Why use xDEM?
 
-```{epigraph}
-The core mission of xDEM is to be **easy-of-use**, **modular**, **robust**, **reproducible** and **fully open**.
+xDEM implements a wide range of high-level operations required for analyzing elevation data in a consistent framework
+tested to ensure the accuracy of these operations.
 
-Additionally, xDEM aims to be **efficient**, **scalable** and **state-of-the-art**.
-```
+It has three main focus points:
 
-```{important}
-:class: margin
-xDEM is in early stages of development and its features might evolve rapidly. Note the version you are working on for
-**reproducibility**!
-We are working on making features fully consistent for the first long-term release ``v0.1`` (planned early 2024).
-```
+1. Having an **easy and intuitive interface** based on the principle of least knowledge,
+2. Providing **statistically robust methods** for reliable quantitative analysis,
+3. Allowing **modular user input** to adapt to most applications.
 
-In details, those mean:
+Although modularity can sometimes hamper performance, we also aim to **preserve scalibility** as much as possible[^sn2].
 
-- **Ease-of-use:** all DEM basic operations or methods from published works should only require a few lines of code to be performed;
+[^sn2]: Out-of-memory, parallelizable computations relying on Dask are planned for late 2024!
 
-- **Modularity:** all DEM methods should be fully customizable, to allow both flexibility and inter-comparison;
+We particularly take to heart to verify the accuracy of our methods. For instance, our terrain attributes
+which have their own modular Python-based implementation, are tested to match exactly
+[gdaldem](https://gdal.org/programs/gdaldem.html) (slope, aspect, hillshade, roughness) and
+[RichDEM](https://richdem.readthedocs.io/en/latest/) (curvatures).
 
-- **Robustness:** all DEM methods should be tested within our continuous integration test-suite, to enforce that they always perform as expected;
-
-- **Reproducibility:** all code should be version-controlled and release-based, to ensure consistency of dependent
-  packages and works;
-
-- **Open-source:** all code should be accessible and re-usable to anyone in the community, for transparency and open governance.
-
-```{note}
-:class: margin
-Additional mission points, in particular **scalability**, are partly developed but not a priority until our first long-term release ``v0.1`` is reached. Those will be further developed specifically in a subsequent version ``v0.2``.
-```
-
-And, additionally:
-
-- **Efficiency**: all methods should be optimized at the lower-level, to function with the highest performance offered by Python packages;
-
-- **Scalability**: all methods should support both lazy processing and distributed parallelized processing, to work with high-resolution data on local machines as well as on HPCs;
-
-- **State-of-the-art**: all methods should be at the cutting edge of remote sensing science, to provide users with the most reliable and up-to-date tools.
-
-
-# The people behind xDEM
-
-```{margin}
-<sup>2</sup>More on our GlacioHack founder at [adehecq.github.io](https://adehecq.github.io/)!
-```
-
-xDEM was created during the [GlacioHack](https://github.com/GlacioHack) hackaton event, that was initiated by
-Amaury Dehecq<sup>2</sup> and took place online on November 8, 2020.
-
-```{margin}
-<sup>3</sup>Check-out [glaciology.ch](https://glaciology.ch) on our founding group of VAW glaciology!
-```
-
-The initial core development of xDEM was performed by members of the Glaciology group of the Laboratory of Hydraulics, Hydrology and
-Glaciology (VAW) at ETH Zürich<sup>3</sup>, with contributions by members of the University of Oslo, the University of Washington, and University
-Grenobles Alpes.
-
-We are not software developers but geoscientists, and we try our best to offer tools that can be useful to a larger group,
-documented, reliable and maintained. All development and maintenance is made on a voluntary basis and we welcome
-any new contributors. See some information on how to contribute in the dedicated page of our
-[GitHub repository](https://github.com/GlacioHack/xdem/blob/main/CONTRIBUTING.md).
+More details about the people behind xDEM and the package's objectives can be found on the **{ref}`background` reference
+page**.

doc/source/accuracy_precision.md

@@ -0,0 +1,115 @@
+(accuracy-precision)=
+
+# Grasping accuracy and precision
+
+Below, a small guide explaining what accuracy and precision are, and their relation to elevation data (or any spatial data!).
+
+## Why do we need to understand accuracy and precision?
+
+Elevation data comes from a wide range of instruments (optical, radar, lidar) acquiring in different conditions (ground,
+airborne, spaceborne) and relying on specific post-processing techniques (e.g., photogrammetry, interferometry).
+
+While some complexities are specific to certain instruments and methods, all elevation data generally possesses:
+
+- a [ground sampling distance](https://en.wikipedia.org/wiki/Ground_sample_distance), or pixel size, **that does not necessarily represent the underlying spatial resolution of the observations**,
+- a [georeferencing](https://en.wikipedia.org/wiki/Georeferencing) **that can be subject to shifts, tilts or other deformations** due to inherent instrument errors, noise, or associated processing schemes,
+- a large number of [outliers](https://en.wikipedia.org/wiki/Outlier) **that remain difficult to filter** as they can originate from various sources (e.g., blunders, clouds).
+
+All of these factors lead to difficulties in assessing the reliability of elevation data, required to
+perform additional quantitative analysis, which calls for defining the concepts relating to accuracy and precision for elevation data.
+
+## Accuracy and precision of elevation data
+
+### What are accuracy and precision?
+
+[Accuracy and precision](https://en.wikipedia.org/wiki/Accuracy_and_precision) describe random and systematic errors, respectively.
+A more accurate measurement is on average closer to the true value (less systematic error), while a more precise measurement has
+less spread in measurement error (less random error), as shown in the simple schematic below.
+
+*Note: sometimes "accuracy" is also used to describe both types of errors, while "trueness" refers to systematic errors, as defined
+in* [ISO 5725-1](https://www.iso.org/obp/ui/#iso:std:iso:5725:-1:ed-1:v1:en) *. Here, we use accuracy for systematic
+errors as, to our knowledge, it is a more commonly used terminology for remote sensing applications.*
+
+:::{figure} imgs/precision_accuracy.png
+:width: 80%
+
+Source: [antarcticglaciers.org](http://www.antarcticglaciers.org/glacial-geology/dating-glacial-sediments2/precision-and-accuracy-glacial-geology/), accessed 29.06.21.
+:::
+
+### Translating these concepts for elevation data
+
+However, elevation data rarely consists of a single independent measurement but of a **series of measurement** (image grid,
+ground track) **related to a spatial support** (horizontal georeferencing, independent of height), which complexifies
+the notion of accuracy and precision.
+
+Due to this, spatially consistent systematic errors can arise in elevation data independently of the error in elevation itself,
+such as **affine biases** (systematic georeferencing shifts), in addition to **specific biases** known to exist locally
+(e.g., signal penetration in land cover type).
+
+For random errors, a variability in error magnitude or **heteroscedasticity** is common in elevation data (e.g.,
+large errors on steep slopes). Finally, spatially structured yet random patterns of errors (e.g., along-track undulations)
+also exist and force us to consider the **spatial correlation of errors**.
+
+Translating the accuracy and precision concepts to elevation data, we can thus define:
+
+- **Elevation accuracy** (systematic error) describes how close an elevation data is to the true elevation on the Earth's surface, both for errors **common to the entire spatial support**
+(DEM grid, altimetric track) and errors **specific to a location** (pixel, footprint),
+- **Elevation precision** (random error) describes the random spread of elevation error in measurement, independently of a possible bias from the true positioning, both for errors **structured over the spatial support** and **specific to a location**.
+
+These categories are depicted in the figure below.
+
+:::{figure} https://github.com/rhugonnet/dem_error_study/blob/main/figures/fig_2.png?raw=true
+:width: 100%
+
+Source: [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922).
+:::
+
+### Absolute or relative elevation accuracy
+
+Accuracy is generally considered from two focus points:
+
+- **Absolute elevation accuracy** describes systematic errors to the true positioning, usually important when analysis focuses on the exact location of topographic features at a specific epoch.
+- **Relative elevation accuracy** describes systematic errors with reference to other elevation data that does not necessarily matches the true positioning, important for analyses interested in topographic change over time.
+
+## How to get the best accuracy and precision of your elevation data?
+
+### Quantifying and improving absolute and relative elevation accuracy
+
+Misalignments due to poor absolute or relative accuracy are common in elevation datasets, and are usually assessed and
+corrected by performing **three-dimensional elevation coregistration and bias corrections to an independent source
+of elevation data**.
+
+In the case of absolute accuracy, this independent source must be precise and accurate, such as altimetric data from
+[ICESat](https://icesat.gsfc.nasa.gov/icesat/) and [ICESat-2](https://icesat-2.gsfc.nasa.gov/) elevations, or coarser yet
+quality-controlled DEMs themselves aligned on altimetric data such as the
+[Copernicus DEM](https://portal.opentopography.org/raster?opentopoID=OTSDEM.032021.4326.3).
+
+To use coregistration and bias correction pipelines in xDEM, see the **feature pages on {ref}`coregistration` and {ref}`biascorr`**.
+
+```{eval-rst}
+.. minigallery:: xdem.coreg.Coreg
+    :add-heading: Examples that use coregistration and bias corrections
+```
+
+### Quantifying and improving assessing elevation precision
+
+While assessing accuracy is fairly straightforward as it consists of computing the mean differences (biases) between
+two or multiple datasets, assessing precision of elevation data can be much more complex. The spread in measurement
+errors cannot be quantified by a single difference and require statistical inference.
+
+Assessing precision usually means applying **spatial statistics combined to uncertainty quantification**,
+to account for the spatial variability and the spatial correlation in errors. For this it is usually necessary, as
+for coregistration, to **rely on an independent source of elevation data on static surfaces similarly**. More background
+on this topic is available on the **{ref}`spatial-stats` guide page**.
+
+To use spatial statistics for quantifying precision in xDEM, see **the feature page on {ref}`uncertainty`**.
+
+Additionally, improving the precision of elevation data is sometimes possible by correcting random structured
+errors using, as for accuracy, **bias correction methods but here applied to pseudo-periodic errors**.
+
+% Functions that are used in several examples create duplicate examples instead of being merged into the list.
+% Circumventing manually by selecting functions used only once in each example for now.
+```{eval-rst}
+.. minigallery:: xdem.spatialstats.infer_heteroscedasticity_from_stable xdem.spatialstats.get_variogram_model_func xdem.spatialstats.sample_empirical_variogram
+    :add-heading: Examples that use spatial statistics functions
+```

doc/source/api.md

@@ -20,7 +20,7 @@ documentation.
 
 ```{important}
 A {class}`~xdem.DEM` inherits all raster methods and attributes from the {class}`~geoutils.Raster` object of GeoUtils.
-Below, we only repeat the core attributes and methods of GeoUtils, see
+Below, we only repeat some core attributes and methods of GeoUtils, see
 [the Raster API in GeoUtils](https://geoutils.readthedocs.io/en/latest/api.html#raster) for the full list.
 ```
 
@@ -79,6 +79,23 @@ Below, we only repeat the core attributes and methods of GeoUtils, see
     DEM.vcrs
 ```
 
+### Other attributes
+
+#### Inherited from {class}`~geoutils.Raster`
+
+See the full list in [the Raster API of GeoUtils](https://geoutils.readthedocs.io/en/latest/api.html#raster).
+
+```{eval-rst}
+.. autosummary::
+    :toctree: gen_modules/
+
+    DEM.res
+    DEM.bounds
+    DEM.width
+    DEM.height
+    DEM.shape
+```
+
 ### Georeferencing
 
 #### Inherited from {class}`~geoutils.Raster`
@@ -115,19 +132,6 @@ See the full list of vector methods in [GeoUtils' documentation](https://geoutil
     DEM.proximity
 ```
 
-### Coregistration
-
-```{tip}
-To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d`, see the API of {ref}`api-geo-handle`.
-```
-
-```{eval-rst}
-.. autosummary::
-    :toctree: gen_modules/
-
-    DEM.coregister_3d
-```
-
 ### Terrain attributes
 
 ```{eval-rst}
@@ -148,6 +152,37 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
     DEM.fractal_roughness
 ```
 
+Or to get multiple related terrain attributes at once (for performance):
+
+```{eval-rst}
+.. autosummary::
+    :toctree: gen_modules/
+
+    DEM.get_terrain_attribute
+```
+
+### Coregistration and bias corrections
+
+```{tip}
+To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d`, see the API of {ref}`api-geo-handle`.
+```
+
+```{eval-rst}
+.. autosummary::
+    :toctree: gen_modules/
+
+    DEM.coregister_3d
+```
+
+### Uncertainty analysis
+
+```{eval-rst}
+.. autosummary::
+    :toctree: gen_modules/
+
+    DEM.estimate_uncertainty
+```
+
 ## dDEM
 
 ```{eval-rst}
@@ -175,8 +210,8 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
 **Overview of co-registration class structure**:
 
 ```{eval-rst}
-.. inheritance-diagram:: xdem.coreg.base xdem.coreg.affine xdem.coreg.biascorr
-        :top-classes: xdem.Coreg
+.. inheritance-diagram:: xdem.coreg.base.Coreg xdem.coreg.affine xdem.coreg.biascorr
+        :top-classes: xdem.coreg.Coreg
 ```
 
 ### Coregistration, pipeline and blockwise
@@ -209,10 +244,9 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
     xdem.coreg.Coreg.residuals
 ```
 
-### Affine coregistration methods
+### Affine coregistration
 
-
-**Generic parent class:**
+#### Parent object (to define custom methods)
 
 ```{eval-rst}
 .. autosummary::
@@ -221,7 +255,7 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
     xdem.coreg.AffineCoreg
 ```
 
-**Convenience classes for specific coregistrations:**
+#### Coregistration methods
 
 ```{eval-rst}
 .. autosummary::
@@ -233,9 +267,21 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
     xdem.coreg.Tilt
 ```
 
-### Bias-correction (including non-affine coregistration) methods
+#### Manipulating affine transforms
+
+```{eval-rst}
+.. autosummary::
+    :toctree: gen_modules/
+
+    xdem.coreg.AffineCoreg.from_matrix
+    xdem.coreg.AffineCoreg.to_matrix
+    xdem.coreg.apply_matrix
+    xdem.coreg.invert_matrix
+```
+
+### Bias-correction
 
-**Generic parent class:**
+#### Parent object (to define custom methods)
 
 ```{eval-rst}
 .. autosummary::
@@ -244,7 +290,7 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
     xdem.coreg.BiasCorr
 ```
 
-**Convenience classes for specific corrections:**
+#### Bias-correction methods
 
 ```{eval-rst}
 .. autosummary::
@@ -299,3 +345,12 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
 
     xdem.spatialstats
 ```
+
+## Stand-alone functions (moved)
+
+```{eval-rst}
+.. autosummary::
+    :toctree: gen_modules/
+
+    xdem.spatialstats.nmad
+```