Descriptive_stats_vs_Graphs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 記述統計とグラフ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div name=\"html-admonition\" style=\"font-size: 0.8em\">\n",
    "<input type=\"button\" onclick=\"location.href='https://translate.google.com/translate?hl=&sl=ja&tl=en&u='+window.location;\" value=\"Google translation\" style=\"color:#ffffff;background-color:#008080; height:25px\" onmouseover=\"this.style.background='#99ccff'\" onmouseout=\"this.style.background='#008080'\"/> in English or the language of your choice.\n",
    "</div><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from statsmodels.formula.api import ols\n",
    "\n",
    "# 警告メッセージを非表示\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 説明"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Anscombeのデータセット**\n",
    "* 4つのデータセット\n",
    "    * それぞれ変数は`x`と`y`の2つ\n",
    "* 全てのデータセットで以下が殆ど同じ\n",
    "    * `x`と`y`の平均（mean）\n",
    "    * `x`と`y`の標準偏差（standard deviation）\n",
    "    * `x`と`y`の相関係数（correlation coefficient）\n",
    "    * 回帰線（regression line）\n",
    "    * 決定係数（coefficient of determination, $R^2$）\n",
    "* 図示（散布図）すると大きく異なる"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**＜本トピックの目的＞**\n",
    "\n",
    "データセットの質的な違いは記述統計だけでは確認できない。図示することが重要ということを示す例を紹介する。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 記述統計 vs 図"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ここでは`matplotlib`に基づいた`seaborn`パッケージを使う。このパッケージを使うことにより，`matplotlib`よりも簡単に，更により綺麗にできるようになる。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anscombeのデータセット"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1 = [10.0, 8.0,  13.0,  9.0,  11.0, 14.0, 6.0,  4.0,  12.0,  7.0,  5.0]\n",
    "y1 = [8.04, 6.95, 7.58,  8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]\n",
    "\n",
    "x2 = [10.0, 8.0,  13.0,  9.0,  11.0, 14.0, 6.0,  4.0,  12.0,  7.0,  5.0]\n",
    "y2 = [9.14, 8.14, 8.74,  8.77, 9.26, 8.10, 6.13, 3.10, 9.13,  7.26, 4.74]\n",
    "\n",
    "x3 = [10.0, 8.0,  13.0,  9.0,  11.0, 14.0, 6.0,  4.0,  12.0,  7.0,  5.0]\n",
    "y3 = [7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15,  6.42, 5.73]\n",
    "\n",
    "x4 = [8.0,  8.0,  8.0,   8.0,  8.0,  8.0,  8.0,  19.0,  8.0,  8.0,  8.0]\n",
    "y4 = [6.58, 5.76, 7.71,  8.84, 8.47, 7.04, 5.25, 12.50, 5.56, 7.91, 6.89]\n",
    "\n",
    "df1 = pd.DataFrame({'x':x1, 'y':y1})  # Dataset 1\n",
    "df2 = pd.DataFrame({'x':x2, 'y':y2})  # Dataset 2\n",
    "df3 = pd.DataFrame({'x':x3, 'y':y3})  # Dataset 3\n",
    "df4 = pd.DataFrame({'x':x4, 'y':y4})  # Dataset 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "散布図"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax1 = plt.subplot(221)  # ax1に図の座標の情報を挿入\n",
    "plt.scatter('x', 'y', data=df1)\n",
    "plt.xlim(2,20)  # 横軸の表示範囲\n",
    "plt.ylim(2,14)  # 縦軸の表示範囲\n",
    "plt.title('Dataset 1')\n",
    "\n",
    "plt.subplot(222, sharex= ax1, sharey=ax1)  # ax1の座標と同じに設定\n",
    "plt.scatter('x', 'y', data=df2)\n",
    "plt.title('Dataset 2')\n",
    "\n",
    "plt.subplot(223, sharex= ax1, sharey=ax1)  # ax1の座標と同じに設定\n",
    "plt.scatter('x', 'y', data=df3)\n",
    "plt.title('Dataset 3')\n",
    "\n",
    "plt.subplot(224, sharex= ax1, sharey=ax1)  # ax1の座標と同じに設定\n",
    "plt.scatter('x', 'y', data=df4)\n",
    "plt.title('Dataset 4')\n",
    "\n",
    "plt.tight_layout()  # レイアウトを見やすく調整\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "平均"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_list = [df1, df2, df3, df4]\n",
    "\n",
    "for df in df_list:\n",
    "    print('x:',df['x'].mean(), '    ', 'y:',df['y'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "標準偏差"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for df in df_list:\n",
    "    print('x:',df['x'].std(), '    ', 'y:',df['y'].std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相関係数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for df in df_list:\n",
    "    print(df.corr().iloc[0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "回帰直線の係数の推定値"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b0hat = []  # 切片の推定値を入れる空のリスト\n",
    "b1hat = []  # スロープの推定値を入れる空のリスト\n",
    "\n",
    "for df in df_list:\n",
    "    mod = ols('y ~ x', data=df).fit()  # OLSの推定\n",
    "    b0hat.append(mod.params.iloc[0])         # 空のリストに推定値を追加\n",
    "    b1hat.append(mod.params.iloc[1])         # 空のリストに推定値を追加\n",
    "    print('b0:',mod.params.iloc[0], '    ', 'b1:',mod.params[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "回帰直線の図示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xx = np.linspace(2,20,100)  # 回帰直線を描くための横軸の値\n",
    "\n",
    "ax1 = plt.subplot(221)\n",
    "plt.plot(xx,b0hat[0]+b1hat[0]*xx)  # 回帰直線\n",
    "plt.scatter('x', 'y', data=df1)\n",
    "plt.xlim(2,20)\n",
    "plt.ylim(2,14)\n",
    "plt.title('Dataset 1')\n",
    "\n",
    "plt.subplot(222, sharex= ax1, sharey=ax1)\n",
    "plt.plot(xx,b0hat[1]+b1hat[1]*xx)  # 回帰直線\n",
    "plt.scatter('x', 'y', data=df2)\n",
    "plt.title('Dataset 2')\n",
    "\n",
    "plt.subplot(223, sharex= ax1, sharey=ax1)\n",
    "plt.plot(xx,b0hat[2]+b1hat[2]*xx)  # 回帰直線\n",
    "plt.scatter('x', 'y', data=df3)\n",
    "plt.title('Dataset 3')\n",
    "\n",
    "plt.subplot(224, sharex= ax1, sharey=ax1)\n",
    "plt.plot(xx,b0hat[3]+b1hat[3]*xx)  # 回帰直線\n",
    "plt.scatter('x', 'y', data=df4)\n",
    "plt.title('Dataset 4')\n",
    "\n",
    "plt.tight_layout()\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "決定係数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for df in df_list:\n",
    "    mod = ols('y ~ x', data=df).fit()\n",
    "    print('R^2:',mod.rsquared)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}