Data/unpack_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unpack the SEN12MSCR Dataset in the required format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stand alone notebook to download and correctly format the SEN12MSCR dataset for the DSen2-CR Model. Running this notebook in full will satisfy everything you need for getting and reformatting the dataset correctly.  However, it may be a better strategy to run the commands under \"Get dataset from online\" and \"Decompress the dataset into current folder\" within a terminal instead for ease and speed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the kernel of this notebook to the dsen2cr_env conda environment:\n",
    "\n",
    "Open a terminal with the dsen2cr_env environment activated and run:\n",
    "```conda install -c anaconda ipykernel```\\\n",
    "```python -m ipykernel install --user --name dsen2cr_env```\n",
    "\n",
    "Set the kernel to dsen2cr_env using the button in the top right corner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Please update this folder. It is the folder that the data will be distracted to. It should be the same folder set as input_data_folder variable in dsen2cr_main.py\n",
    "input_data_folder = '/path/to/input_data_folder'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get dataset from online.\n",
    "Note: these commands may run significantly quicker in parallel, with each being done in separate terminal at the same time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1158_spring_s1.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1158_spring_s1.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1158_spring_s2_cloudy.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1158_spring_s2_cloudy.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1158_spring_s2.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1158_spring_s2.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1868_summer_s1.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1868_summer_s1.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1868_summer_s2_cloudy.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1868_summer_s2_cloudy.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1868_summer_s2.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1868_summer_s2.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1970_fall_s1.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1970_fall_s1.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1970_fall_s2_cloudy.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1970_fall_s2_cloudy.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs1970_fall_s2.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs1970_fall_s2.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs2017_winter_s1.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs2017_winter_s1.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs2017_winter_s2_cloudy.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs2017_winter_s2_cloudy.tar.gz\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-check-certificate -O ROIs2017_winter_s2.tar.gz \"https://dataserv.ub.tum.de/s/m1554803/download?path=%2F&files=ROIs2017_winter_s2.tar.gz\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decompress the dataset into current folder\n",
    "Add v to the options to see verbose output from the tar command ( !tar xzvf instead of !tar xzf)\\\n",
    "Note: these commands may run significantly quicker in parallel, with each being done in separate terminal at the same time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1158_spring_s1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1158_spring_s2_cloudy.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1158_spring_s2.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1868_summer_s1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1868_summer_s2_cloudy.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1868_summer_s2.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1970_fall_s1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1970_fall_s2_cloudy.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs1970_fall_s2.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs2017_winter_s1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs2017_winter_s2_cloudy.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ROIs2017_winter_s2.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create new folders to move data into"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "s1_folder = os.path.join(input_data_folder, \"s1\")\n",
    "!mkdir -p {s1_folder}\n",
    "s2_cloudy_folder = os.path.join(input_data_folder, \"s2_cloudy\")\n",
    "!mkdir -p {s2_cloudy_folder}\n",
    "s2_cloudFree_folder = os.path.join(input_data_folder, \"s2_cloudFree\")\n",
    "!mkdir -p {s2_cloudFree_folder}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rename and move all data into the input_data_folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "s1_data = [\"ROIs1158_spring_s1\", \"ROIs1868_summer_s1\", \"ROIs1970_fall_s1\", \"ROIs2017_winter_s1\"]\n",
    "\n",
    "#unpack all files within the folders in the s1_data list\n",
    "for season_sentinel_folder in s1_data:\n",
    "    for sentinel_ROI_folder in os.listdir(season_sentinel_folder):\n",
    "        for filename in os.listdir(os.path.join(season_sentinel_folder,sentinel_ROI_folder)):\n",
    "            if(filename != \".ipynb_checkpoints\"):\n",
    "                #extract information from filename\n",
    "                season_id, season, sentinel, ROI, patch = filename.split(\"_\")\n",
    "                patch = patch[1:]\n",
    "                #create new filename\n",
    "                new_filename = f\"{season_id}_{season}_{ROI}_p{patch}\"\n",
    "                #rename from old filename to new filename, and put into correct folder\n",
    "                to_rename = os.path.join(season_sentinel_folder, sentinel_ROI_folder, filename)\n",
    "                new_name =  os.path.join(input_data_folder, \"s1\",new_filename) #CHECK THIS\n",
    "                shutil.move(to_rename, new_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2_cloudy_data = [\"ROIs1158_spring_s2_cloudy\", \"ROIs1868_summer_s2_cloudy\", \"ROIs1970_fall_s2_cloudy\", \"ROIs2017_winter_s2_cloudy\"]\n",
    "\n",
    "#unpack all files within the folders in the s2_cloudy_data list\n",
    "for season_sentinel_folder in s2_cloudy_data:\n",
    "    for sentinel_ROI_folder in os.listdir(season_sentinel_folder):\n",
    "        for filename in os.listdir(os.path.join(season_sentinel_folder, sentinel_ROI_folder)):\n",
    "            if(filename != \".ipynb_checkpoints\"):\n",
    "                #extract information from filename\n",
    "                season_id, season, sentinel1, sentinel2, ROI, patch = filename.split(\"_\")\n",
    "                patch = patch[1:]\n",
    "                #create new filename\n",
    "                new_filename = f\"{season_id}_{season}_{ROI}_p{patch}\"\n",
    "                #rename from old filename to new filename, and put into correct folder\n",
    "                to_rename = os.path.join(season_sentinel_folder, sentinel_ROI_folder, filename)\n",
    "                new_name =  os.path.join(input_data_folder, \"s2_cloudy\", new_filename) #CHECK THIS\n",
    "                shutil.move(to_rename, new_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2_data = [\"ROIs1158_spring_s2\", \"ROIs1868_summer_s2\", \"ROIs1970_fall_s2\", \"ROIs2017_winter_s2\"]\n",
    "\n",
    "#unpack all files within the folders in the s2_data list\n",
    "for season_sentinel_folder in s2_data:\n",
    "    for sentinel_ROI_folder in os.listdir(season_sentinel_folder):\n",
    "        for filename in os.listdir(os.path.join(season_sentinel_folder, sentinel_ROI_folder)):\n",
    "            if(filename != \".ipynb_checkpoints\"):\n",
    "                #extract information from filename\n",
    "                season_id, season, sentinel, ROI, patch = filename.split(\"_\")\n",
    "                patch = patch[1:]\n",
    "                #create new filename\n",
    "                new_filename = f\"{season_id}_{season}_{ROI}_p{patch}\"\n",
    "                #rename from old filename to new filename, and put into correct folder\n",
    "                to_rename = os.path.join(season_sentinel_folder,sentinel_ROI_folder, filename)\n",
    "                new_name =  os.path.join(input_data_folder, \"s2_cloudFree\", new_filename) #CHECK THIS\n",
    "                shutil.move(to_rename, new_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensure all files have been downloaded, extracted, and moved\n",
    "Get number of files in s1, s2_cloudy, and s2_cloudFree folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess\n",
    "cmd_p1 = subprocess.Popen([\"ls\", s1_folder], stdout=subprocess.PIPE)\n",
    "cmd_p2 = subprocess.Popen([\"wc\",\"-l\"], stdin=cmd_p1.stdout, stdout = subprocess.PIPE)\n",
    "len_s1 = cmd_p2.stdout.name\n",
    "cmd_p1.stdout.close()\n",
    "cmd_p2.stdout.close()\n",
    "\n",
    "cmd_p1 = subprocess.Popen([\"ls\", s2_cloudy_folder], stdout=subprocess.PIPE)\n",
    "cmd_p2 = subprocess.Popen([\"wc\",\"-l\"], stdin=cmd_p1.stdout, stdout = subprocess.PIPE)\n",
    "len_s2_cloudy = cmd_p2.stdout.name\n",
    "cmd_p1.stdout.close()\n",
    "cmd_p2.stdout.close()\n",
    "\n",
    "cmd_p1 = subprocess.Popen([\"ls\", s2_cloudFree_folder], stdout=subprocess.PIPE)\n",
    "cmd_p2 = subprocess.Popen([\"wc\",\"-l\"], stdin=cmd_p1.stdout, stdout = subprocess.PIPE)\n",
    "len_s2_cloudFree = cmd_p2.stdout.name\n",
    "cmd_p1.stdout.close()\n",
    "cmd_p2.stdout.close()\n",
    "\n",
    "print(\"Number of s1, s2_cloudy, and s2_cloudFree files: \", len_s1, \" \", len_s2_cloudy, \" \", len_s2_cloudFree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check whether there are the correct number of files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (len_s1 != 122218 or len_s2_cloudy != 122218 or len_s2_cloudFree != 122218):\n",
    "    print(\"errors with downloading, extracting, and moving data\")\n",
    "else:\n",
    "    print(\"Data Processed successfully! You may remove all of the generated files and folders.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}