notebooks/tutorials/hello-syft/01-hello-syft.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Hello Syft"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {},
   "source": [
    "PySyft is a python library containing a set of data serialization and remote code execution APIs which mimic existing popular Data Science tools while working interchangeably with existing popular data types. It enables data scientists query for their data related questions on sensitive or proprietary data in a secure and privacy-preserving way. The python package for PySyft is called `syft`. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "In this tutorial, we will cover the following workflows:\n",
    "\n",
    "- Data Owner Workflow - Part 1\n",
    "    - upload mock data\n",
    "- Data Scientist Workflow  - Part 1\n",
    "    - write query against mock data\n",
    "    - submit code for review on the data owner side\n",
    "- Data Owner Workflow - Part 2\n",
    "    - review code and approve\n",
    "    - share the real result with the data scientist\n",
    "- Data Scientist Workflow - Part 2\n",
    "    - fetch the real result\n",
    "\n",
    "For more detailed tutorials for each subject, please refer to `data-owner` and `data-scientist` tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3",
   "metadata": {},
   "source": [
    "## Install `syft`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "SYFT_VERSION = \">=0.8.2.b0,<0.9\"\n",
    "package_string = f'\"syft{SYFT_VERSION}\"'\n",
    "# %pip install {package_string} -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# syft absolute\n",
    "import syft as sy\n",
    "\n",
    "sy.requires(SYFT_VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6",
   "metadata": {},
   "source": [
    "## Launch a dummy server \n",
    "\n",
    "In this tutorial, for the sake of demonstration, we will be using in-memory workers as dummy servers. For details of deploying a server on your own using `syft` and `hagrid`, please refer to the `quickstart` tutorials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7",
   "metadata": {},
   "outputs": [],
   "source": [
    "node = sy.orchestra.launch(name=\"hello-syft-usa-server\", port=9000, reset=True)\n",
    "root_domain_client = node.login(email=\"info@openmined.org\", password=\"changethis\")\n",
    "root_domain_client.register(\n",
    "    name=\"Jane Doe\",\n",
    "    email=\"janedoe@caltech.edu\",\n",
    "    password=\"abc123\",\n",
    "    password_verify=\"abc123\",\n",
    "    institution=\"Caltech\",\n",
    "    website=\"https://www.caltech.edu/\",\n",
    ")\n",
    "\n",
    "ds_client = node.login(email=\"janedoe@caltech.edu\", password=\"abc123\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8",
   "metadata": {},
   "source": [
    "## Data owner - Part 1\n",
    "\n",
    "### Upload Data to Domain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# third party\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10",
   "metadata": {},
   "source": [
    "The first thing we do as a data owner is uploading our dataset. Based on the original data, the data owner will generate a synthetic or fake version of this dataset. They can add any amount of noise to the fake values. Let's say in this fake version, they are adding `+10` to each of the ages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = sy.Dataset(\n",
    "    name=\"usa-mock-data\",\n",
    "    description=\"Dataset of ages\",\n",
    "    asset_list=[\n",
    "        sy.Asset(\n",
    "            name=\"ages\",\n",
    "            data=pd.DataFrame(\n",
    "                {\n",
    "                    \"Patient_ID\": [\"011\", \"015\", \"022\", \"034\", \"044\"],\n",
    "                    \"Age\": [40, 39, 35, 60, 25],\n",
    "                }\n",
    "            ),\n",
    "            mock=pd.DataFrame(\n",
    "                {\"Patient_ID\": [\"1\", \"2\", \"3\", \"4\", \"5\"], \"Age\": [50, 49, 45, 70, 35]}\n",
    "            ),\n",
    "            mock_is_real=False,\n",
    "        )\n",
    "    ],\n",
    ")\n",
    "root_domain_client.upload_dataset(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12",
   "metadata": {},
   "source": [
    "## Data Scientist - Part 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13",
   "metadata": {},
   "source": [
    "### Load Mock Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14",
   "metadata": {},
   "source": [
    "The data scientist can get access to the `Assets` uploaded by the `Data Owner`, and the mock version of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15",
   "metadata": {},
   "outputs": [],
   "source": [
    "asset = ds_client.datasets[-1].assets[\"ages\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16",
   "metadata": {},
   "outputs": [],
   "source": [
    "asset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17",
   "metadata": {},
   "outputs": [],
   "source": [
    "mock = asset.mock\n",
    "mock"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18",
   "metadata": {},
   "source": [
    "### Write Query on Mock Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19",
   "metadata": {},
   "source": [
    "We can use the mock to develop against"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20",
   "metadata": {},
   "outputs": [],
   "source": [
    "age_sum = mock[\"Age\"].mean()\n",
    "print(age_sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21",
   "metadata": {},
   "source": [
    "When we are done, we wrap the code into a function decorated with a `syft_function`, in this case the most simple version, `syft_function_single_use`. Read more about syft_functions in the data scientist tutorials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22",
   "metadata": {},
   "outputs": [],
   "source": [
    "@sy.syft_function_single_use(df=asset)\n",
    "def get_mean_age(df):\n",
    "    return df[\"Age\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23",
   "metadata": {},
   "source": [
    "### Submit Code Request for Review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24",
   "metadata": {},
   "outputs": [],
   "source": [
    "req = ds_client.code.request_code_execution(get_mean_age)\n",
    "req"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25",
   "metadata": {},
   "source": [
    "The code request is successfully submitted!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26",
   "metadata": {},
   "source": [
    "## Data Owner - Part 2\n",
    "\n",
    "### Get Requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27",
   "metadata": {},
   "source": [
    "As a data owner, we can now view and approve the request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28",
   "metadata": {},
   "outputs": [],
   "source": [
    "root_domain_client.requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29",
   "metadata": {},
   "outputs": [],
   "source": [
    "request = root_domain_client.requests[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30",
   "metadata": {},
   "outputs": [],
   "source": [
    "str_changes = []\n",
    "for change in request.changes:\n",
    "    if change.id in request.current_change_state:\n",
    "        print(\"A\")\n",
    "        str_change = (\n",
    "            change.__repr_syft_nested__()\n",
    "            if hasattr(change, \"__repr_syft_nested__\")\n",
    "            else type(change)\n",
    "        )\n",
    "        str_change = f\"{str_change}. \"\n",
    "        str_changes.append(str_change)\n",
    "str_changes = \"\\n\".join(str_changes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31",
   "metadata": {},
   "source": [
    "### Review Code and Policies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32",
   "metadata": {},
   "source": [
    "Before we approve, we want to inspect the code and the policies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33",
   "metadata": {},
   "outputs": [],
   "source": [
    "usercode = request.code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34",
   "metadata": {},
   "outputs": [],
   "source": [
    "usercode"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35",
   "metadata": {},
   "source": [
    "### Execute function on real data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36",
   "metadata": {},
   "source": [
    "Now that we have seen the code we can run it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_mean_age_user_function = usercode.unsafe_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38",
   "metadata": {},
   "outputs": [],
   "source": [
    "asset = usercode.assets[0]\n",
    "real_result = get_mean_age_user_function(df=asset)\n",
    "print(real_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39",
   "metadata": {},
   "source": [
    "### Share the real result with the Data Scientist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = request.accept_by_depositing_result(real_result)\n",
    "print(result)\n",
    "assert isinstance(result, sy.SyftSuccess)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41",
   "metadata": {},
   "source": [
    "## Data Scientist - Part 2\n",
    "\n",
    "### Fetch Real Result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42",
   "metadata": {},
   "source": [
    "As a Data scientist, we can now fetch the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43",
   "metadata": {},
   "outputs": [],
   "source": [
    "asset = ds_client.datasets[0].assets[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_client.code[0].status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45",
   "metadata": {},
   "outputs": [],
   "source": [
    "result_ptr = ds_client.code.get_mean_age(df=asset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46",
   "metadata": {},
   "outputs": [],
   "source": [
    "real_result = result_ptr.get()\n",
    "print(real_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47",
   "metadata": {},
   "source": [
    "**That's a success!! The external data scientist was able to know the average age of breast cancer patients in a USA regional hospital, without having to access or even look at the real data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48",
   "metadata": {},
   "source": [
    "Once you are done with this tutorial, you can safely shut down the servers as following,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49",
   "metadata": {},
   "outputs": [],
   "source": [
    "node.land()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}