docs/source/advanced.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"../\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subclassing schemas\n",
    "\n",
    "Subclassing schemas is a useful pattern for pipelines where every next function adds a few columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from strictly_typed_pandas import DataSet\n",
    "\n",
    "class SchemaA:\n",
    "    name: str\n",
    "\n",
    "class SchemaB(SchemaA):\n",
    "    id: int\n",
    "\n",
    "df = DataSet[SchemaA]({\"name\": [\"John\", \"Jane\", \"Jack\"]})\n",
    "\n",
    "def foo(df: DataSet[SchemaA]) -> DataSet[SchemaB]:\n",
    "    return (\n",
    "        df.assign(id=lambda df: range(df.shape[0]))\n",
    "        .pipe(DataSet[SchemaB])\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, you can use it when merging (or joining or concatenating) two datasets together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SchemaA:\n",
    "    id: int\n",
    "    name: str\n",
    "\n",
    "class SchemaB:\n",
    "    id: int\n",
    "    job: str\n",
    "\n",
    "class SchemaAB(SchemaA, SchemaB):\n",
    "    pass\n",
    "\n",
    "df1 = DataSet[SchemaA]({\"id\": [1, 2, 3], \"name\": [\"John\", \"Jane\", \"Jack\"]})\n",
    "df2 = DataSet[SchemaB]({\"id\": [1, 2, 3], \"job\": \"Data Scientist\"})\n",
    "(\n",
    "    df1.merge(df2, on=\"id\")\n",
    "    .pipe(DataSet[SchemaAB])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an empty DataSet\n",
    "Sometimes it's useful to create a DataSet without any rows. This can be easily done as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Schema:\n",
    "    id: int\n",
    "    name: str\n",
    "\n",
    "DataSet[Schema]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support for numpy and pandas data types\n",
    "We also support using numpy types and pandas types, as well as `typing.Any`. If you miss support for any other data type, drop us a line and we'll see if we can add it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from typing import Any\n",
    "\n",
    "class Schema:\n",
    "    name: pd.StringDtype\n",
    "    money: np.float64\n",
    "    eggs: np.int64\n",
    "    potatoes: Any\n",
    "\n",
    "df = DataSet[Schema](\n",
    "    {\n",
    "        \"name\": pd.Series([\"John\", \"Jane\", \"Jack\"], dtype=\"string\"),\n",
    "        \"money\": pd.Series([100.50, 1000.23, 123.45], dtype=np.float64),\n",
    "        \"eggs\": pd.Series([1, 2, 3], dtype=np.int64),\n",
    "        \"potatoes\": [\"1\", 0, np.nan]\n",
    "    }\n",
    ")\n",
    "\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IndexedDataSet\n",
    "\n",
    "If you'd like to also strictly type the index, you can use the IndexedDataSet class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from strictly_typed_pandas import IndexedDataSet\n",
    "\n",
    "class IndexSchema:\n",
    "    id: int\n",
    "    job: str\n",
    "\n",
    "class DataSchema:\n",
    "    name: str\n",
    "\n",
    "df = (\n",
    "    pd.DataFrame({\"id\": [1, 2, 3], \"name\": [\"John\", \"Jane\", \"Jack\"], \"job\": \"Data Scientist\"})\n",
    "    .set_index([\"id\", \"job\"])\n",
    "    .pipe(IndexedDataSet[IndexSchema, DataSchema])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reusing a variable (e.g. `df`) with different schemas\n",
    "Sometimes when building a pipeline, it's useful to reuse a variable (e.g. `df`) with different schemas. If we do that in the following way however, we'll get a mypy error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SchemaA:\n",
    "    name: str\n",
    "\n",
    "class SchemaB(SchemaA):\n",
    "    id: int\n",
    "\n",
    "def foo(df: DataSet[SchemaA]) -> DataSet[SchemaB]:\n",
    "    return (\n",
    "        df.assign(id=1)\n",
    "        .pipe(DataSet[SchemaB])\n",
    "    )\n",
    "\n",
    "df = DataSet[SchemaA]({\"name\": [\"John\", \"Jane\", \"Jack\"]})\n",
    "df = foo(df)\n",
    "# mypy(error): Incompatible types in assignment (expression has type \"DataSet[SchemaB]\", variable has type \"DataSet[SchemaA]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To avoid this error, we need to declare that `df` will be of the type `DataSet` (implying the the schema may be different at different points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df: DataSet\n",
    "df = DataSet[SchemaA]({\"name\": [\"John\", \"Jane\", \"Jack\"]})\n",
    "df = foo(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## No cloning\n",
    "\n",
    "When a `DataFrame` is cast to a `DataSet`, the underlying data isn't cloned (unless you use `DataSet[Schema](..., copy=True)`). This is great for memory purposes, but it does require some caution. For example, consider the following pandas script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.DataFrame({\"id\": [1, 2, 3], \"name\": [\"John\", \"Jane\", \"Jack\"]})\n",
    "df2 = pd.DataFrame(df1)\n",
    "df1.name = [1, 2, 3]\n",
    "df2.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, `df1` and `df2` essentially point to the same data, so changing one of them changes the other one too. This behaviour extends to `DataSet` as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Schema:\n",
    "    id: int\n",
    "    name: str\n",
    "\n",
    "df1 = pd.DataFrame({\"id\": [1, 2, 3], \"name\": [\"John\", \"Jane\", \"Jack\"]})\n",
    "df2 = DataSet[Schema](df1)\n",
    "\n",
    "df1.name = [1, 2, 3]\n",
    "df2.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is somewhat problematic, because we now made a change to the schema, without any error thrown whatsoever! However:\n",
    "\n",
    "* I essentially can't stop you from doing this (apart from forcing `DataSet` to copy the data when created, which I won't).\n",
    "\n",
    "* If this happens in your code, you have bigger problems that type checking.\n",
    "\n",
    "So the bottomline is: be careful when dealing with pointers!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "21955bae40816b58329a864495bd83642121ab031d49eff86d34b7b0569c6cea"
  },
  "kernelspec": {
   "display_name": "Python 3.8.5 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": ""
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2
}