Merge pull request #116 from digitalearthafrica/wsf-doc

Added evolutoin data

data_specs/World_Settlement_Footprint_specs.ipynb

@@ -25,7 +25,7 @@
    "id": "0a7cc6df-948d-4559-92f4-39d3f0b6aebc",
    "metadata": {},
    "source": [
-    "**Date modified:** 11 September 2023"
+    "**Date modified:** 21 November 2023"
    ]
   },
   {
@@ -57,14 +57,16 @@
     "\n",
     "The World Settlement Footprint 2019 features data from the Copernicus Sentinel-1 and Sentinel-2 missions and provides information on global human settlements with unprecedented detail and precision (Marconcini et al., 2021).\n",
     "\n",
-    "The World Settlement Footprint 2015 and 2019 data are now indexed in the DE Africa platform.\n",
+    "The World Settlement Footprint Evolution was produced by processing seven million images from the US Landsat satellite collected between 1985 and 2015 and shows the annual growth of human settlements globally (Mapping Our Human Footprint From Space, 2023).\n",
+    "\n",
+    "The World Settlement Footprint 2015, 2019, World Settlement Footprint Evolution data are now indexed in the DE Africa platform.\n",
     " \n",
     "#### Product User Manual\n",
     "\n",
     "[World Settlement Footprint](https://doi.org/10.1553/giscience2021_01_s33)\n",
     "\n",
     "\n",
-    "A [Jupyter Notebook](https://github.com/digitalearthafrica/deafrica-sandbox-notebooks/blob/main/Datasets/World_Settlement_Footprint.ipynb) which demonstrates loading and using the  datasets in the Sandbox is also available."
+    "A [Jupyter Notebook for the World Settlement Footprint 2015 and 2019 dataset](https://github.com/digitalearthafrica/deafrica-sandbox-notebooks/blob/main/Datasets/World_Settlement_Footprint.ipynb) and [Jupyter Notebook for the World Settlement Footprint Evolution](https://github.com/digitalearthafrica/deafrica-sandbox-notebooks/blob/main/Datasets/World_Settlement_Footprint-Evolution.ipynb) which demonstrates loading and using the  datasets in the Sandbox is also available."
    ]
   },
   {
@@ -90,7 +92,7 @@
    "id": "2f1fd227-5b30-4bcb-9609-ee82b2a89dd6",
    "metadata": {},
    "source": [
-    "The specific temporal and geographic extents for the product can be explored as an interactive map on the metadata for [2015](https://explorer.digitalearth.africa/products/wsf_2015) and [2019](https://explorer.digitalearth.africa/products/wsf_2019)."
+    "The specific temporal and geographic extents for the product can be explored as an interactive map on the metadata for [2015](https://explorer.digitalearth.africa/products/wsf_2015), [2019](https://explorer.digitalearth.africa/products/wsf_2019) and [Evolution](https://explorer.digitalearth.africa/products/wsf_evolution)."
    ]
   },
   {
@@ -106,15 +108,15 @@
    "id": "6c7c0d62-4edc-4105-b584-d39e45360653",
    "metadata": {},
    "source": [
-    "|Specification | | |\n",
-    "|----------|-------------|-------------|\n",
-    "|Product name|  `wsf_2015` | `wsf_2019` |\n",
-    "|Cell size - X |   10m | 10m | \n",
-    "|Cell size - Y |  10m | 10m | \n",
-    "|Coordinate reference system | `EPSG:4326`  | `EPSG:4326` | \n",
-    "|Temporal resolution | NA |  NA |\n",
-    "|Temporal range| 2015 |  2019 |\n",
-    "|Update frequency| NA | NA |"
+    "|Specification | | | |\n",
+    "|----------|-------------|-------------|-------------|\n",
+    "|Product name|  `wsf_2015` | `wsf_2019` | `wsf_evolution`\n",
+    "|Cell size - X |   10m | 10m | 30m |\n",
+    "|Cell size - Y |  10m | 10m | 30m |\n",
+    "|Coordinate reference system | `EPSG:4326`  | `EPSG:4326` | `EPSG:4326` | \n",
+    "|Temporal resolution | NA |  NA | NA |\n",
+    "|Temporal range| 2015 |  2019 | 1985 - 2015 |\n",
+    "|Update frequency| NA | NA | NA |"
    ]
   },
   {
@@ -165,7 +167,9 @@
     "| Band ID | Description | Units | Data type | No data$^\\dagger$ |\n",
     "|:------------------------------|:--------------|:--------|:--------|---------:|\n",
     "| wsf_2015 | World Settlement Footprint 2015 | 1 | uint8 | 0.0 |\n",
-    "| wsf_2019 | World Settlement Footprint 2019 | 1 | uint8 | 0.0 |"
+    "| wsf_2019 | World Settlement Footprint 2019 | 1 | uint8 | 0.0 |\n",
+    "| wsfevolution | World Settlement Footprint Evolution | 1 | uint8 | 0.0 |\n",
+    "| idcscore | Input Data Consistency score | 1 | uint8 | 0.0 |"
    ]
   },
   {
@@ -183,7 +187,17 @@
    "source": [
     "The **World Settlement Footprint WSF 2015 version 2** (WSF2015 v2) is a 10m resolution binary mask outlining the extent of human settlements globally for the year 2015. Specifically, the WSF2015 v2 is a pilot product generated by combining multiple datasets.The **World Settlement Footprint (WSF) 2015** derived at 10m spatial resolution by means of 2014-2015 multitemporal Landsat-8 and Sentinel-1 imagery (of which ~217K and ~107K scenes have been processed, respectively). The High Resolution Settlement Layer (HRSL) generated by the Connectivity Lab team at Facebook through the employment of 2016 DigitalGlobe VHR satellite imagery and publicly released at 30m spatial resolution for 214 countries (Marconcini et al., 2020).\n",
     "\n",
-    "The **World Settlement Footprint (WSF) 2019** is a 10m resolution binary mask outlining the extent of human settlements globally derived by means of 2019 multitemporal Sentinel-1 (S1) and Sentinel-2 (S2) imagery. Based on the hypothesis that settlements generally show a more stable behavior with respect to most land-cover classes, temporal statistics are calculated for both S1- and S2-based indices. In particular, a comprehensive analysis has been performed by exploiting a number of reference building outlines to identify the most suitable set of temporal features (ultimately including 6 from S1 and 25 from S2). Training points for the settlement and non-settlement class are then generated by thresholding specific features, which varies depending on the 30 climate types of the well-established Köppen Geiger scheme. Next, binary classification based on Random Forest is applied and, finally, a dedicated post-processing is performed where ancillary datasets are employed to further reduce omission and commission errors. Here, the whole classification process has been entirely carried out within the Google Earth Engine platform. To assess the high accuracy and reliability of the WSF2019, two independent crowd-sourcing-based validation exercises have been carried out with the support of Google and Mapswipe, respectively, where overall 1M reference labels have been collected based photointerpretation of very high-resolution optical imagery (Marconcini et al., 2021)."
+    "The **World Settlement Footprint (WSF) 2019** is a 10m resolution binary mask outlining the extent of human settlements globally derived by means of 2019 multitemporal Sentinel-1 (S1) and Sentinel-2 (S2) imagery. Based on the hypothesis that settlements generally show a more stable behavior with respect to most land-cover classes, temporal statistics are calculated for both S1- and S2-based indices. In particular, a comprehensive analysis has been performed by exploiting a number of reference building outlines to identify the most suitable set of temporal features (ultimately including 6 from S1 and 25 from S2). Training points for the settlement and non-settlement class are then generated by thresholding specific features, which varies depending on the 30 climate types of the well-established Köppen Geiger scheme. Next, binary classification based on Random Forest is applied and, finally, a dedicated post-processing is performed where ancillary datasets are employed to further reduce omission and commission errors. Here, the whole classification process has been entirely carried out within the Google Earth Engine platform. To assess the high accuracy and reliability of the WSF2019, two independent crowd-sourcing-based validation exercises have been carried out with the support of Google and Mapswipe, respectively, where overall 1M reference labels have been collected based photointerpretation of very high-resolution optical imagery (Marconcini et al., 2021).\n",
+    "\n",
+    "The **World Settlement Footprint (WSF Evolution)** is a 30m resolution dataset outlining the global settlement extent on a yearly basis from 1985 to 2015. Based on the assumption that settlement growth occurred over time, all pixels categorized as non-settlement in the WSF2015 (Marconcini et al., 2020) are excluded a priori from the analysis. Next, for each target year in the past, all available Landsat-5/7 scenes acquired over the given area of interest are gathered and key temporal statistics (i.e., temporal mean, minimum, maximum, etc.) are then extracted for different spectral indices. Among others, these include: the normalized difference built-up index (NDBI), normalized difference vegetation index (NDVI) and modified normalized difference water index (MNDWI). Temporal features proved generally robust if computed over at least 7 clear cloud-/cloud-shadow-free observations; accordingly, if for a given pixel in the target year this constraint is not satisfied, the time frame is enlarged backwards (at 1-year steps) as long as the condition is met.\n",
+    "\n",
+    "Starting backwards from the year 2015 - for which the WSF2015 is used as a reference - settlement and non-settlement training samples for the given target year t are iteratively extracted by applying morphological filtering to the settlement mask derived for the year t+1, as well as excluding potentially mislabeled samples by adaptively thresholding the temporal mean NDBI, MNDWI and NDVI. Finally, binary Random Forest classification in performed.\n",
+    "\n",
+    "To quantitatively assess the high accuracy and reliability of the dataset, an extensive campaign based on crowdsourcing photointerpretation of very high-resolution airborne and satellite historical imagery has been performed with the support of Google. In particular, for the years 1990, 1995, 2000, 2005, 2010 and 2015, ~200K reference cells of 30x30m size distributed over 100 sites around the world have been labelled, hence summing up to overall ~1.2M validation samples.\n",
+    "\n",
+    "It is worth noting that past Landsat-5/7 availability considerably varies across the world and over time. Independently from the implemented approach, this might then result in a lower quality of the final product where few/no scenes have been collected. Accordingly, to provide the users with a suitable and intuitive measure that accounts for the goodness of the Landsat imagery, we conceived the Input Data Consistency (IDC) score, which ranges from 6 to 1 with: 6) very good; 5) good; 4) fair; 3) moderate; 2) low; 1) very low. The IDC score is available on a yearly basis between 1985 and 2015 and supports a proper interpretation of the WSF evolution product.\n",
+    "\n",
+    "The WSF evolution and IDC score datasets are organized in 5138 GeoTIFF files (EPSG4326 projection) each one referring to a portion of 2 x 2 degree size (~222 x 222 km) on the ground. WSF evolution values range between 1985 and 2015 corresponding to the estimated year of settlement detection, whereas 0 is no data. A comprehensive publication with all technical details and accuracy figures is currently being finalized. For the time being, please refer to [Marconcini et al,. 2021](https://austriaca.at/0xc1aa5576%200x003c9b4c.pdf)."
    ]
   },
   {
@@ -226,6 +240,22 @@
     "<img src=\"../_static/data_specs/WSF/WorldSettlementFootprint_2.png\" alt=\"wsf 2015 and 2019\" width=\"600\" align=\"left\"/>"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "5b84728b-bcdf-42ab-9e0d-cc85191b0e78",
+   "metadata": {},
+   "source": [
+    "**Figure 4: World Settlement Footprint Evolution over Mansoura, Egypt with IDC score for the image**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8341d1ed-111b-4e5c-9248-7aad93f90a67",
+   "metadata": {},
+   "source": [
+    "<img src=\"../_static/data_specs/WSF/WorldSettlementFootprinEvolution.png\" alt=\"World Settlement Footprint Evolution\" width=\"600\" align=\"left\"/>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "7519d955-16f1-4f5a-908c-d8eb0964bbb0",
@@ -285,7 +315,7 @@
    "id": "d601693c-c60a-436b-81c4-64277a468fcf",
    "metadata": {},
    "source": [
-    "The World Settlement Footprint 10m 2015 and 2019 products are avaliable in AWS S3."
+    "The World Settlement Footprint 10m 2015 2019, Evolution products are avaliable in AWS S3."
    ]
   },
   {
@@ -331,7 +361,7 @@
    "id": "1b829dc7-a1dd-4c9e-b692-8acb125b7111",
    "metadata": {},
    "source": [
-    "The World Settlement Footprint 10m product `wsf_2015` and `wsf_2019` are available through the Digital Earth Africa's OWS. "
+    "The World Settlement Footprint 10m product `wsf_2015`, `wsf_2019`, `wsf_evolution` are available through the Digital Earth Africa's OWS. "
    ]
   },
   {
@@ -382,7 +412,7 @@
    "source": [
     "The World Settlement Footprint product can be accessed through the Digital Earth Africa ODC API, which is available through the [Digital Earth Africa Sandbox](https://sandbox.digitalearth.africa/hub/login).\n",
     "\n",
-    "**ODC product name:** `wsf_{year}`"
+    "**ODC product name:** `wsf_{year}` and `wsf_evolution`"
    ]
   },
   {
@@ -409,7 +439,8 @@
    "source": [
     "|Band name| Alternative names| Fill value |\n",
     "| :-: | :-: | :-: |\n",
-    "| wsf{year}                | NaN        |      `0.0` |"
+    "| wsf{year}                | NaN        |      `0.0` |\n",
+    "| wsfevolution                | NaN        |      `0.0` |"
    ]
   }
  ],

data_specs/index.rst

@@ -272,7 +272,7 @@ Land and water
 
        :doc:`World_Settlement_Footprint_specs`
 
-       The World Settlement Footprint WSF is a 10m resolution binary maskoutlining the extent of human settlements globally for the year 2015 and 2019 based on multiple datasets. 
+       The World Settlement Footprint WSF is a 10m-resolution binary mask outlining the extent of human settlements globally for the years 2015 and 2019 based on multiple datasets, and the World Settlement Footprint Evolution shows the annual growth of human settlements globally at 30m-resolution. 
 
        :doc:`More <World_Settlement_Footprint_specs>`