Examples-WUSS2018-HOW-Everything_Is_Better_With_Friends.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Everything is better with friends: Executing SAS code in Python scripts with SASPy\n",
    "\n",
    "#### Hands-on Workshop Examples; Western Users of SAS Software (WUSS) 2018 Conference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Python version (IL)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "import platform\n",
    "print(platform.sys.version)\n",
    "```\n",
    "Notes:\n",
    "\n",
    "1. Assuming a Python 3 kernel is associated with this Notebook, the Python version and operating-system information should be printed. \n",
    "2. Here, we're using three common Python coding conventions:\n",
    "  * A module object called `platform` is made available.\n",
    "  * In the call to the `print` function, so-called dot notation (from object-oriented programming) is used with `platform` invoking the sub-objects `sys` nested inside of it and `sys` invoking the sub-sub-object `version` nested inside it. (Think Russian nesting doll.)\n",
    "3. If an error is displayed, an incompatibility kernel has been chosen. This Notebook was developed using the Python 3.5 kernel provided with SAS University Edition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing available Python modules (IL)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "help('modules')\n",
    "```\n",
    "Notes:\n",
    "1. All Python modules available to the Notebook's kernel should be printed. This list will include standard library modules like `platform`, which was used above, and any third-party modules that have been installed.\n",
    "2. Due to the \"batteries included\" philosophy of Python's standard library, the list of modules is expected to be long. In addition, because Python is an open-source language, many third-party modules are actively developed and made freely available through popular websites like <https://github.com/> and <https://pypi.org/>.\n",
    "3. The third-party modules `pandas` and `saspy` will need to appear in the list generated by `help('modules')` for the remaining examples in this Notebook to work as expected. In addition, `saspy` will need to be pre-configured to connect to a SAS kernel with access to the SAS `sashelp` library as provided with SAS University Edition (or its equivalent)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python hello-world example with strings (IL)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "hello_world_str = 'Hello, Jupyter!'\n",
    "print(hello_world_str)\n",
    "print()\n",
    "print(type(hello_world_str))\n",
    "```\n",
    "Notes:\n",
    "1. A string object named `hello_world_str` is created, and then its value and type are printed with a blank line between them.\n",
    "2. Python objects can be defined virtually anywhere and can have many different types. This is fundamentally different from SAS, which only allows variables to be created in special places (like DATA steps) and restricts their types to string and numeric.\n",
    "3. Python syntax is sometimes more forgiving and sometimes less forgiving than SAS:\n",
    "  * In Python, single and double quotes are interchangeable, whereas they have different behavior in SAS when macro variables are involved.\n",
    "  * In Python, semicolons are optional at the end of each statement, unlike in SAS.\n",
    "  * In Python, capitalization matters (e.g., `hello_world_str` is distinct from `HELLO_WORLD_STR`), unlike in SAS.\n",
    "  * In Python, a single equals sign (=) only ever means assignment, and a double-equals (==) is used to test for equality, unlike in SAS where single-equals (=) can be used both for object assignment and equality testing.\n",
    "  * In Python, indentation is significant and used to determine scope (as will be seen in later examples), unlike SAS where things like do-groups are needed to delimit scope in DATA steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python hello-world example with DataFrames (IL)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "from pandas import DataFrame\n",
    "hello_world_df = DataFrame(['Hello', 'Notebook!'])\n",
    "print(hello_world_df)\n",
    "print()\n",
    "hello_world_df.info()\n",
    "```\n",
    "Notes:\n",
    "1. A DataFrame object named `hello_world_df` is created, and then its value, type, and row/column information are printed.\n",
    "2. DataFrames are two-dimensional arrays of values that can be thought of like a SAS dataset. However, while SAS datasets are typically only accessed from disk and processed row-by-row as needed, DataFrames are loaded into memory all at once. This means values in DataFrames can be randomly accessed, but it also means their size can't grow beyond available memory.\n",
    "3. Since the DataFrame object type is not built into Python, we first need to import it from the `pandas` module, after which objects can be created using the `DataFrame` constructor function. In this example, we're calling this constructor on the list `['Hello', 'Notebook!']` with the two comma-separated string values `'Hello'` and `'Notebook!'`. (If a list of lists _of the same length_ had been provided, each list would form a new column. Multiple lists of different lengths would result in an error.)\n",
    "4. The `hello_world_df` object calls its `info` method (meaning a function whose definition is nested inside it), which prints information about `hello_world_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using SASPy to connect to a SAS kernel (MS)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "from saspy import SASsession\n",
    "sas = SASsession()\n",
    "type(sas)\n",
    "```\n",
    "Notes:\n",
    "1. A SASsession object named `sas` is created, and then three things are printed to the log:\n",
    "  * Confirmation `saspy` is using a preconfigured SAS kernel connection profile (here, the `default` configuration)\n",
    "  * Confirmation a SAS session has been established with this kernel\n",
    "  * The type of object `sas`\n",
    "2. As with the DataFrame object type above, SASsession is not built into Python, so we first need to import it from the `saspy` module, after which objects can be created using the `SASsession` constructor function. In this example, we're calling this constructor with no argument, which is why SASPy's default configuration is used.\n",
    "3. The lines `from saspy import SASsession` and `sas = SASsession()` only need to be used to establish a SAS kernel connection once within a given Python session. Consequently, all subsequent cells in this Notebook will assume these lines have been executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using SASPy to get information about its SAS session (MS)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "sas_product_status = sas.submit('proc product_status; run;')\n",
    "print(type(sas_product_status))\n",
    "print(sas_product_status['LOG'])\n",
    "```\n",
    "Notes:\n",
    "1. The object `sas_product_status` is created and assigned the result of submitting the code `proc product_status; run;` to a SAS kernel, and then its type and the _SAS log portion_ of its value are printed with a blank line between them.\n",
    "2. The `sas` object represents a connection to a SAS session and was created when a previous cell was run. Here, `sas` calls its `submit` method to execute SAS code within this SAS session.\n",
    "3. Because SAS code execution typically creates both log output and results output, the `submit` method returns a Python dictionary (or `dict` for short) with the two keys `'LOG'` and `'LST'`, meaning we can write\n",
    "  * `sas_product_status['LOG']` to access log output and\n",
    "  * `sas_product_status['LST']` to access results output.\n",
    "4. Dictionaries are a fundamental Python data structure and are related to hash tables in SAS DATA steps. Both are more generally called _associative arrays_ or _maps_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using SASPy to connect to a SAS dataset (MS)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "fish_sds = sas.sasdata('fish','sashelp')\n",
    "print(type(fish_sds))\n",
    "fish_sds.columnInfo()\n",
    "```\n",
    "Notes:\n",
    "1. The object `fish_sds` is created and becomes a connection to the SAS dataset `fish` in the `sashelp` library, and then its type and summary information about the dataset's seven columns are printed. (Note that Python starts counting at zero, which is why the variables are indexed from 0 to 6.)\n",
    "2. The `sas` object represents a connection to a SAS session and was created when a previous cell was run. Here, `sas` calls its `sasdata` method to access a SAS library defined within this SAS session.\n",
    "3. The `fish_sds` object calls its _convenience method_ `columnInfo`, which implicitly invokes PROC CONTENTS on `sashelp.fish`.\n",
    "4. Additional convenience methods provided by SASPy can be found at <https://sassoftware.github.io/saspy/api.html#sas-data-object>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting SAS code generated by SASPy convenience methods (MS)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "sas.teach_me_SAS(True)\n",
    "fish_sds.means()\n",
    "sas.teach_me_SAS(False)\n",
    "```\n",
    "Notes:\n",
    "1. The `sas` object represents a connection to a SAS session, and the `fish_sds` object represents a connection to a SAS dataset. Both were created when previous cells were run.\n",
    "2. Here, `sas` calls its `teach_me_SAS` method twice, creating a \"teach me\" sandwich. (Think \"ODS sandwich\" in SAS.) Within this \"teach me\" sandwich, `fish_sds` calls its convenience method `means`.\n",
    "2. When the `teach_me_SAS` method is called with argument `True`, SAS output is suppressed, and the SAS code underlying SASPy convenience methods is returned instead. When `teach_me_SAS` is called with argument `False`, this behavior is turned off."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting arbitrary SAS code with SASPy (MS)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```\n",
    "%%SAS sas\n",
    "proc means data=sashelp.fish std mean min max;\n",
    "    class species;\n",
    "    var Weight;\n",
    "run;\n",
    "```\n",
    "Notes:\n",
    "1. The Jupyter magic command `%%SAS` is used to redirect all subsequent code in the cell to the SAS kernel associated with SASsession object `sas` created when a previous cell was run. (The magic command `%lsmagic` can be used to list all magic commands available within a Notebook session.)\n",
    "2. Magic commands like `%%SAS` allow code in different languages to be intermixed within the same Notebook without having to switch kernels and is particularly helpful when options aren't provided by SASPy convenience methods (like the `class` statement for PROC MEANS). However, if both SAS and Python code should be intermixed within the same cell, the `sas.submit` method might need to be used instead (as in the PROC PRODUCT_STATUS example above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using SASPy to convert a SAS dataset into a DataFrame (IL)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "fish_df = sas.sasdata2dataframe('fish','sashelp')\n",
    "print(type(fish_df))\n",
    "fish_df.describe()\n",
    "```\n",
    "Notes:\n",
    "1. The object `fish_df` is created from the SAS dataset `fish` in the `sashelp` library, and then its type and summary information about the dataset's six columns are printed.\n",
    "2. The `sas` object represents a connection to a SAS session and was created when a previous cell was run. Here, `sas` calls its `sasdata2dataframe` method to access the SAS library `sashelp` defined within this SAS session and to load the entire contents of SAS dataset `sashelp.fish` into DataFrame `fish_df`.\n",
    "3. The `fish_df` method `describe` is the Python parallel of the SASPy convenience method `means`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chaining DataFrame operations into \"one-liners\" (IL)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "fish_df.groupby('Species')['Weight'].agg(['count', 'std', 'mean', 'min', 'max'])\n",
    "```\n",
    "Notes:\n",
    "1. The `fish_df` object represents a DataFrame created from the SAS dataset `sashelp.fish` when a previous cell was run. Here, we are applying the following chain of operations to create a so-called \"one-liner\":\n",
    "  * First, the `groupby` method is used to group rows in `fish_df` by the values in column `'Species'`.\n",
    "  * Next, the column named `'Weight'` is extracted from the result of the group-by operation, using the same notation we saw above for selecting part of a dictionary.\n",
    "  * Next, the grouped values of the column `'Weight'` are aggregated using the functions whose names are given in the list `['count', 'std', 'mean', 'min', 'max']`.\n",
    "2. By chaining these operations together, we obtain the same result as the following SAS code, where the `class` statement is equivalent to the group-by operation and the `var` statement is equivalent to the column-extraction operation: \n",
    "```\n",
    "%%SAS\n",
    "proc means data=sashelp.fish std mean min max;\n",
    "        class species;\n",
    "        var Weight;\n",
    "run;\n",
    "```\n",
    "3. Because DataFrames are loaded entirely into memory, we have significantly more flexibility for arbitrarily accessing their values than for disk-based SAS datasets. As long as datasets are small enough to fit into memory, this allows us to chain together on-the-fly reshaping operations like grouping, column extraction, and aggregation in order to create highly customized data summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Python to imitate the SAS Macro Processor (IL)\n",
    "\n",
    "Type the following code into the cell labelled `In [ ]:` immediately below, and then run that cell:\n",
    "```Python\n",
    "from IPython.display import HTML\n",
    "sas_code_fragment = 'proc means data=sashelp.%s; run;'\n",
    "for dsn in ['fish','iris']:\n",
    "    display(HTML(sas.submit(sas_code_fragment%dsn)['LST']))\n",
    "```\n",
    "Notes:\n",
    "1. A string object named `sas_code_fragment` is created with templating placeholder `%s`, which will be filled using other strings in subsequent uses of `sas_code_fragment`.\n",
    "2. The output of PROC MEANS applied to SAS datasets `sashelp.fish` and `sashelp.iris` is then displayed by extracting the results output from SAS session code submissions and rendering their HTML using a combination of the `IPython.display.HTML` method and the built-in `display` function.\n",
    "3. The `sas` object represents a connection to a SAS session and was created when a previous cell was run. Here, `sas` calls its `submit` method for each value of the for-loop indexing variable `dsn`, and the `%s` portion of `sas_code_fragment` is replaced by `dsn`. In other words, the following SAS code is submitted to the SAS kernel:\n",
    "```\n",
    "%%SAS\n",
    "proc means data=sashelp.fish; run; \n",
    "proc means data=sashelp.iris; run; \n",
    "```\n",
    "4. The same outcome could also be achieved with the following SAS macro code:\n",
    "```\n",
    "%%SAS\n",
    "%macro loop(); \n",
    "        %let dsn_list = fish iris;\n",
    "        %do i = 1 %to 2; \n",
    "            %let dsn = %scan(&dsn_list.,&i.);\n",
    "            proc means data=sashelp.&dsn.;\n",
    "            run; \n",
    "        %end;\n",
    "%mend; \n",
    "%loop()\n",
    "```\n",
    "However, note the following differences:\n",
    "  * Python allows us to concisely repeat an arbitrary block of code by iterating over a list using a for-loop. In other words, the body of the for-loop (meaning everything indented underneath it, since Python uses indentation to determine scope) is repeated for each string in the list `['fish','iris']`.\n",
    "  * SAS only provides do-loops based on index variables (e.g., the macro variable `i` above) for its macro facility, so clever tricks like implicitly defined arrays (e.g., macro variables `dsn_list` above) need to used together with functions like `%scan` for extracting a sequence of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}