AWS/AWS_Send_dataframe_to_S3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "30539360",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"8%\" alt=\"AWS.png\" src=\"https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/.github/assets/logos/AWS.png\" style=\"border-radius: 15%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ba2d382",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# AWS - Send dataframe to S3\n",
    "<a href=\"https://bit.ly/3JyWIk6\">Give Feedback</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=AWS+-+Send+dataframe+to+S3:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5a5aae6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #aws #cloud #storage #S3bucket #operations #snippet #dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b7e082b",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Maxime Jublou](https://www.linkedin.com/in/maximejublou/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c3e0f63-a870-49e7-8c3d-c3d112740f8d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Last update:** 2023-11-20 (Created: 2022-04-28)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30937e73-6d14-45f9-9540-5e77b0e80efa",
   "metadata": {
    "papermill": {},
    "tags": [
     "description"
    ]
   },
   "source": [
    "**Description:** This notebook demonstrates how to use AWS to send a dataframe to an S3 bucket."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb46a1ae",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4e0537a-4baf-4148-b8f9-de1333a1a0d6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d87447cd-3a50-44f3-9669-c2333a3af917",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "import naas\n",
    "try:\n",
    "    import awswrangler as wr\n",
    "except:\n",
    "    !pip install awswrangler --user\n",
    "    import awswrangler as wr\n",
    "from os import environ\n",
    "from datetime import date\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5495ae4-0c12-4793-ae0e-8e8144be57bf",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup variables\n",
    "**Mandatory**\n",
    "- `aws_access_key_id`: This variable is used to store the AWS access key ID.\n",
    "- `aws_secret_access_key`: This variable is used to store the AWS secret access key.\n",
    "- `bucket_path`: The name of the S3 bucket from which you want to list the files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "587741e3-0006-4db3-9780-1b0a909f61e6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Mandatory\n",
    "aws_access_key_id = naas.secret.get(\"AWS_ACCESS_KEY_ID\") or \"YOUR_AWS_ACCESS_KEY_ID\"\n",
    "aws_secret_access_key = naas.secret.get(\"AWS_SECRET_ACCESS_KEY\") or \"YOUR_AWS_SECRET_ACCESS_KEY\"\n",
    "bucket_path = f\"s3://naas-data-lake/example/\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612e695e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27e095d5-49e0-420c-9d6d-3fd90567867f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-21T06:42:00.638182Z",
     "iopub.status.busy": "2022-04-21T06:42:00.637939Z",
     "iopub.status.idle": "2022-04-21T06:42:00.843677Z",
     "shell.execute_reply": "2022-04-21T06:42:00.842680Z",
     "shell.execute_reply.started": "2022-04-21T06:42:00.638157Z"
    },
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Set environ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9aac758-d29b-421a-8417-1528e8462fa1",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "environ[\"AWS_ACCESS_KEY_ID\"] = aws_access_key_id\n",
    "environ[\"AWS_SECRET_ACCESS_KEY\"] = aws_secret_access_key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d77884",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ac4e6d3",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"id\": [1, 2],\n",
    "        \"value\": [\"foo\", \"boo\"],\n",
    "        \"date\": [date(2020, 1, 1), date(2020, 1, 2)],\n",
    "    }\n",
    ")\n",
    "\n",
    "# Display dataframe\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5fef003",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69333139",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Send dataset to S3\n",
    "Wrangler has 3 different write modes to store Parquet Datasets on Amazon S3.\n",
    "- **append** (Default) : Only adds new files without any delete.\n",
    "- **overwrite** : Deletes everything in the target directory and then add new files.\n",
    "- **overwrite_partitions** (Partition Upsert) : Only deletes the paths of partitions that should be updated and then writes the new partitions files. It's like a \"partition Upsert\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "601e9f52",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "wr.s3.to_parquet(df=df, path=bucket_path, dataset=True, mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "362e1dfd-606b-4552-8bc9-128a70b7bdf7",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "naas": {
   "notebook_id": "0d5fe4330975af3c53c528f914bd8d0124c25dea14afe09df1119b430a9bb2b2",
   "notebook_path": "AWS/AWS_Send_dataframe_to_S3.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.3.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}