9주차_PySpark_기본_4일차_1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "PySpark 기본: 4일차-1",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/learn-programmers/programmers_kdt_II/blob/main/9%EC%A3%BC%EC%B0%A8_PySpark_%EA%B8%B0%EB%B3%B8_4%EC%9D%BC%EC%B0%A8_1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xAqhTDfuWrcM"
      },
      "source": [
        "PySpark을 로컬머신에 설치하고 노트북을 사용하기 보다는 머신러닝 관련 다양한 라이브러리가 이미 설치되었고 좋은 하드웨어를 제공해주는 Google Colab을 통해 실습을 진행한다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XIA23YgbXKJd"
      },
      "source": [
        "이를 위해 pyspark과 Py4J 패키지를 설치한다. Py4J 패키지는 파이썬 프로그램이 자바가상머신상의 오브젝트들을 접근할 수 있게 해준다. Local Standalone Spark을 사용한다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NbT0rpGfVdiq",
        "outputId": "ca839239-845b-4a94-b3d2-ca2cdd40db42"
      },
      "source": [
        "!pip install pyspark==3.0.1 py4j==0.10.9 "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting pyspark==3.0.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f0/26/198fc8c0b98580f617cb03cb298c6056587b8f0447e20fa40c5b634ced77/pyspark-3.0.1.tar.gz (204.2MB)\n",
            "\u001b[K     |████████████████████████████████| 204.2MB 66kB/s \n",
            "\u001b[?25hCollecting py4j==0.10.9\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl (198kB)\n",
            "\u001b[K     |████████████████████████████████| 204kB 44.9MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.0.1-py2.py3-none-any.whl size=204612242 sha256=fcae897c3b2f7f51201ccbad04463d5646b7ff511a29581f41a53795ad57475b\n",
            "  Stored in directory: /root/.cache/pip/wheels/5e/bd/07/031766ca628adec8435bb40f0bd83bb676ce65ff4007f8e73f\n",
            "Successfully built pyspark\n",
            "Installing collected packages: py4j, pyspark\n",
            "Successfully installed py4j-0.10.9 pyspark-3.0.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3vm6tgcPXdnR"
      },
      "source": [
        "from pyspark.sql import SparkSession\n",
        "\n",
        "spark = SparkSession \\\n",
        "    .builder \\\n",
        "    .appName(\"Boston Housing Linear Regression example\") \\\n",
        "    .getOrCreate()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pyl0gES0KhkF"
      },
      "source": [
        "# 보스턴 주택 가격 예측 모델 만들기\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 217
        },
        "id": "LSs_1PYaYWxI",
        "outputId": "8cb1c271-6b5d-4ab4-9269-eeae293ef1d8"
      },
      "source": [
        "spark"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "            <div>\n",
              "                <p><b>SparkSession - in-memory</b></p>\n",
              "                \n",
              "        <div>\n",
              "            <p><b>SparkContext</b></p>\n",
              "\n",
              "            <p><a href=\"http://00f66ebc919b:4040\">Spark UI</a></p>\n",
              "\n",
              "            <dl>\n",
              "              <dt>Version</dt>\n",
              "                <dd><code>v3.0.1</code></dd>\n",
              "              <dt>Master</dt>\n",
              "                <dd><code>local[*]</code></dd>\n",
              "              <dt>AppName</dt>\n",
              "                <dd><code>Boston Housing Linear Regression example</code></dd>\n",
              "            </dl>\n",
              "        </div>\n",
              "        \n",
              "            </div>\n",
              "        "
            ],
            "text/plain": [
              "<pyspark.sql.session.SparkSession at 0x7f48e98614a8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vE8iL4vy6705",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e770f39d-1584-4845-cd2f-4950d9ff7e12"
      },
      "source": [
        "!wget https://s3-geospatial.s3-us-west-2.amazonaws.com/boston_housing.csv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-01-31 20:52:43--  https://s3-geospatial.s3-us-west-2.amazonaws.com/boston_housing.csv\n",
            "Resolving s3-geospatial.s3-us-west-2.amazonaws.com (s3-geospatial.s3-us-west-2.amazonaws.com)... 52.218.217.25\n",
            "Connecting to s3-geospatial.s3-us-west-2.amazonaws.com (s3-geospatial.s3-us-west-2.amazonaws.com)|52.218.217.25|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 36240 (35K) [text/csv]\n",
            "Saving to: ‘boston_housing.csv’\n",
            "\n",
            "\rboston_housing.csv    0%[                    ]       0  --.-KB/s               \rboston_housing.csv  100%[===================>]  35.39K  --.-KB/s    in 0.04s   \n",
            "\n",
            "2021-01-31 20:52:43 (963 KB/s) - ‘boston_housing.csv’ saved [36240/36240]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GfSzc03fOC6e",
        "outputId": "2ea998f3-2cf3-43f9-f70e-ea7d64b9b050"
      },
      "source": [
        "!ls -tl"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total 40\n",
            "-rw-r--r-- 1 root root 36240 Jan 31 01:46 boston_housing.csv\n",
            "drwxr-xr-x 1 root root  4096 Jan 20 17:27 sample_data\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-mH3roiKIOix"
      },
      "source": [
        "data = spark.read.csv('./boston_housing.csv', header=True, inferSchema=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SCty0kw6ITVi",
        "outputId": "fb5c21be-c2c8-4171-9438-6871c332a2ee"
      },
      "source": [
        "data.printSchema()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- crim: double (nullable = true)\n",
            " |-- zn: double (nullable = true)\n",
            " |-- indus: double (nullable = true)\n",
            " |-- chas: integer (nullable = true)\n",
            " |-- nox: double (nullable = true)\n",
            " |-- rm: double (nullable = true)\n",
            " |-- age: double (nullable = true)\n",
            " |-- dis: double (nullable = true)\n",
            " |-- rad: integer (nullable = true)\n",
            " |-- tax: integer (nullable = true)\n",
            " |-- ptratio: double (nullable = true)\n",
            " |-- b: double (nullable = true)\n",
            " |-- lstat: double (nullable = true)\n",
            " |-- medv: double (nullable = true)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nrlYYyVfIV1e",
        "outputId": "305db144-78d2-4d15-f562-e549b8bd208b"
      },
      "source": [
        "data.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-------+----+-----+----+-----+-----+-----+------+---+---+-------+------+-----+----+\n",
            "|   crim|  zn|indus|chas|  nox|   rm|  age|   dis|rad|tax|ptratio|     b|lstat|medv|\n",
            "+-------+----+-----+----+-----+-----+-----+------+---+---+-------+------+-----+----+\n",
            "|0.00632|18.0| 2.31|   0|0.538|6.575| 65.2|  4.09|  1|296|   15.3| 396.9| 4.98|24.0|\n",
            "|0.02731| 0.0| 7.07|   0|0.469|6.421| 78.9|4.9671|  2|242|   17.8| 396.9| 9.14|21.6|\n",
            "|0.02729| 0.0| 7.07|   0|0.469|7.185| 61.1|4.9671|  2|242|   17.8|392.83| 4.03|34.7|\n",
            "|0.03237| 0.0| 2.18|   0|0.458|6.998| 45.8|6.0622|  3|222|   18.7|394.63| 2.94|33.4|\n",
            "|0.06905| 0.0| 2.18|   0|0.458|7.147| 54.2|6.0622|  3|222|   18.7| 396.9| 5.33|36.2|\n",
            "|0.02985| 0.0| 2.18|   0|0.458| 6.43| 58.7|6.0622|  3|222|   18.7|394.12| 5.21|28.7|\n",
            "|0.08829|12.5| 7.87|   0|0.524|6.012| 66.6|5.5605|  5|311|   15.2| 395.6|12.43|22.9|\n",
            "|0.14455|12.5| 7.87|   0|0.524|6.172| 96.1|5.9505|  5|311|   15.2| 396.9|19.15|27.1|\n",
            "|0.21124|12.5| 7.87|   0|0.524|5.631|100.0|6.0821|  5|311|   15.2|386.63|29.93|16.5|\n",
            "|0.17004|12.5| 7.87|   0|0.524|6.004| 85.9|6.5921|  5|311|   15.2|386.71| 17.1|18.9|\n",
            "|0.22489|12.5| 7.87|   0|0.524|6.377| 94.3|6.3467|  5|311|   15.2|392.52|20.45|15.0|\n",
            "|0.11747|12.5| 7.87|   0|0.524|6.009| 82.9|6.2267|  5|311|   15.2| 396.9|13.27|18.9|\n",
            "|0.09378|12.5| 7.87|   0|0.524|5.889| 39.0|5.4509|  5|311|   15.2| 390.5|15.71|21.7|\n",
            "|0.62976| 0.0| 8.14|   0|0.538|5.949| 61.8|4.7075|  4|307|   21.0| 396.9| 8.26|20.4|\n",
            "|0.63796| 0.0| 8.14|   0|0.538|6.096| 84.5|4.4619|  4|307|   21.0|380.02|10.26|18.2|\n",
            "|0.62739| 0.0| 8.14|   0|0.538|5.834| 56.5|4.4986|  4|307|   21.0|395.62| 8.47|19.9|\n",
            "|1.05393| 0.0| 8.14|   0|0.538|5.935| 29.3|4.4986|  4|307|   21.0|386.85| 6.58|23.1|\n",
            "| 0.7842| 0.0| 8.14|   0|0.538| 5.99| 81.7|4.2579|  4|307|   21.0|386.75|14.67|17.5|\n",
            "|0.80271| 0.0| 8.14|   0|0.538|5.456| 36.6|3.7965|  4|307|   21.0|288.99|11.69|20.2|\n",
            "| 0.7258| 0.0| 8.14|   0|0.538|5.727| 69.5|3.7965|  4|307|   21.0|390.95|11.28|18.2|\n",
            "+-------+----+-----+----+-----+-----+-----+------+---+---+-------+------+-----+----+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cGMk-ULmLcVM"
      },
      "source": [
        "## 피쳐 벡터를 만들기"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vBYWOVQYIXEt"
      },
      "source": [
        "from pyspark.ml.feature import VectorAssembler\n",
        "\n",
        "feature_columns = data.columns[:-1]\n",
        "assembler = VectorAssembler(inputCols=feature_columns, outputCol=\"features\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SAkNEWkhO9Vk",
        "outputId": "9c7c2e5a-fc9b-4d21-9708-6ec4526a8be8"
      },
      "source": [
        "feature_columns"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['crim',\n",
              " 'zn',\n",
              " 'indus',\n",
              " 'chas',\n",
              " 'nox',\n",
              " 'rm',\n",
              " 'age',\n",
              " 'dis',\n",
              " 'rad',\n",
              " 'tax',\n",
              " 'ptratio',\n",
              " 'b',\n",
              " 'lstat']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8J0_WrOYIqA0"
      },
      "source": [
        "data_2 = assembler.transform(data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e0WZHfxmIuxr",
        "outputId": "f3ac324d-f68a-4702-91df-bde0e4237bf7"
      },
      "source": [
        "data_2.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-------+----+-----+----+-----+-----+-----+------+---+---+-------+------+-----+----+--------------------+\n",
            "|   crim|  zn|indus|chas|  nox|   rm|  age|   dis|rad|tax|ptratio|     b|lstat|medv|            features|\n",
            "+-------+----+-----+----+-----+-----+-----+------+---+---+-------+------+-----+----+--------------------+\n",
            "|0.00632|18.0| 2.31|   0|0.538|6.575| 65.2|  4.09|  1|296|   15.3| 396.9| 4.98|24.0|[0.00632,18.0,2.3...|\n",
            "|0.02731| 0.0| 7.07|   0|0.469|6.421| 78.9|4.9671|  2|242|   17.8| 396.9| 9.14|21.6|[0.02731,0.0,7.07...|\n",
            "|0.02729| 0.0| 7.07|   0|0.469|7.185| 61.1|4.9671|  2|242|   17.8|392.83| 4.03|34.7|[0.02729,0.0,7.07...|\n",
            "|0.03237| 0.0| 2.18|   0|0.458|6.998| 45.8|6.0622|  3|222|   18.7|394.63| 2.94|33.4|[0.03237,0.0,2.18...|\n",
            "|0.06905| 0.0| 2.18|   0|0.458|7.147| 54.2|6.0622|  3|222|   18.7| 396.9| 5.33|36.2|[0.06905,0.0,2.18...|\n",
            "|0.02985| 0.0| 2.18|   0|0.458| 6.43| 58.7|6.0622|  3|222|   18.7|394.12| 5.21|28.7|[0.02985,0.0,2.18...|\n",
            "|0.08829|12.5| 7.87|   0|0.524|6.012| 66.6|5.5605|  5|311|   15.2| 395.6|12.43|22.9|[0.08829,12.5,7.8...|\n",
            "|0.14455|12.5| 7.87|   0|0.524|6.172| 96.1|5.9505|  5|311|   15.2| 396.9|19.15|27.1|[0.14455,12.5,7.8...|\n",
            "|0.21124|12.5| 7.87|   0|0.524|5.631|100.0|6.0821|  5|311|   15.2|386.63|29.93|16.5|[0.21124,12.5,7.8...|\n",
            "|0.17004|12.5| 7.87|   0|0.524|6.004| 85.9|6.5921|  5|311|   15.2|386.71| 17.1|18.9|[0.17004,12.5,7.8...|\n",
            "|0.22489|12.5| 7.87|   0|0.524|6.377| 94.3|6.3467|  5|311|   15.2|392.52|20.45|15.0|[0.22489,12.5,7.8...|\n",
            "|0.11747|12.5| 7.87|   0|0.524|6.009| 82.9|6.2267|  5|311|   15.2| 396.9|13.27|18.9|[0.11747,12.5,7.8...|\n",
            "|0.09378|12.5| 7.87|   0|0.524|5.889| 39.0|5.4509|  5|311|   15.2| 390.5|15.71|21.7|[0.09378,12.5,7.8...|\n",
            "|0.62976| 0.0| 8.14|   0|0.538|5.949| 61.8|4.7075|  4|307|   21.0| 396.9| 8.26|20.4|[0.62976,0.0,8.14...|\n",
            "|0.63796| 0.0| 8.14|   0|0.538|6.096| 84.5|4.4619|  4|307|   21.0|380.02|10.26|18.2|[0.63796,0.0,8.14...|\n",
            "|0.62739| 0.0| 8.14|   0|0.538|5.834| 56.5|4.4986|  4|307|   21.0|395.62| 8.47|19.9|[0.62739,0.0,8.14...|\n",
            "|1.05393| 0.0| 8.14|   0|0.538|5.935| 29.3|4.4986|  4|307|   21.0|386.85| 6.58|23.1|[1.05393,0.0,8.14...|\n",
            "| 0.7842| 0.0| 8.14|   0|0.538| 5.99| 81.7|4.2579|  4|307|   21.0|386.75|14.67|17.5|[0.7842,0.0,8.14,...|\n",
            "|0.80271| 0.0| 8.14|   0|0.538|5.456| 36.6|3.7965|  4|307|   21.0|288.99|11.69|20.2|[0.80271,0.0,8.14...|\n",
            "| 0.7258| 0.0| 8.14|   0|0.538|5.727| 69.5|3.7965|  4|307|   21.0|390.95|11.28|18.2|[0.7258,0.0,8.14,...|\n",
            "+-------+----+-----+----+-----+-----+-----+------+---+---+-------+------+-----+----+--------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6kK7gG2JLjy_"
      },
      "source": [
        "## 훈련용과 테스트용 데이터를 나누고 Linear Regression 모델을 하나 만든다"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OEmZtDfzIyvc"
      },
      "source": [
        "train, test = data_2.randomSplit([0.7, 0.3])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y1cKT-0-I2rt"
      },
      "source": [
        "from pyspark.ml.regression import LinearRegression\n",
        "\n",
        "algo = LinearRegression(featuresCol=\"features\", labelCol=\"medv\")\n",
        "model = algo.fit(train)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELSupsQVLFK-"
      },
      "source": [
        "## 모델 성능 측정"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LDzlKgOTJkTJ"
      },
      "source": [
        "evaluation_summary = model.evaluate(test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "02BIJYu4JnNJ",
        "outputId": "15ca7666-d594-4f36-c47b-90c7f3b891b5"
      },
      "source": [
        "evaluation_summary"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyspark.ml.regression.LinearRegressionSummary at 0x7f48e6db5438>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GDSM4ZAeJrFa",
        "outputId": "3057dcd2-4ea8-4a79-e7c5-ef5d33622a96"
      },
      "source": [
        "evaluation_summary.meanAbsoluteError"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3.823400450912103"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "idZshiJqJuOl",
        "outputId": "5dab0a81-0cb9-46aa-87ee-8ac3ca7a7413"
      },
      "source": [
        "evaluation_summary.rootMeanSquaredError"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5.122713827311029"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3VnnpJqYJx0d",
        "outputId": "8f241852-9b22-434c-a41f-427e97f7f9cb"
      },
      "source": [
        "evaluation_summary.r2"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7148202490449094"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k1V_bYQmLKy2"
      },
      "source": [
        "## 모델 예측값 살펴보기 "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1pegKAalJz4n"
      },
      "source": [
        "predictions = model.transform(test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8kpfjtXJRVus",
        "outputId": "4cab1114-660d-4d87-878b-7c32a8d18c08"
      },
      "source": [
        "predictions.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-------+----+-----+----+------+-----+----+-------+---+---+-------+------+-----+----+--------------------+------------------+\n",
            "|   crim|  zn|indus|chas|   nox|   rm| age|    dis|rad|tax|ptratio|     b|lstat|medv|            features|        prediction|\n",
            "+-------+----+-----+----+------+-----+----+-------+---+---+-------+------+-----+----+--------------------+------------------+\n",
            "|0.01096|55.0| 2.25|   0| 0.389|6.453|31.9| 7.3073|  1|300|   15.3|394.72| 8.23|22.0|[0.01096,55.0,2.2...|26.571482297719307|\n",
            "|0.01301|35.0| 1.52|   0| 0.442|7.241|49.3| 7.0379|  1|284|   15.5|394.74| 5.49|32.7|[0.01301,35.0,1.5...|30.167506151964368|\n",
            "|0.01311|90.0| 1.22|   0| 0.403|7.249|21.9| 8.6966|  5|226|   17.9|395.93| 4.81|35.4|[0.01311,90.0,1.2...|30.335900587302184|\n",
            "| 0.0136|75.0|  4.0|   0|  0.41|5.888|47.6| 7.3197|  3|469|   21.1| 396.9| 14.8|18.9|[0.0136,75.0,4.0,...|15.228505739892949|\n",
            "|0.01501|90.0| 1.21|   1| 0.401|7.923|24.8|  5.885|  1|198|   13.6|395.52| 3.16|50.0|[0.01501,90.0,1.2...| 44.80928824019599|\n",
            "|0.01709|90.0| 2.02|   0|  0.41|6.728|36.1|12.1265|  5|187|   17.0|384.46|  4.5|30.1|[0.01709,90.0,2.0...|25.264400971073016|\n",
            "|0.02009|95.0| 2.68|   0|0.4161|8.034|31.9|  5.118|  4|224|   14.7|390.55| 2.88|50.0|[0.02009,95.0,2.6...| 41.68916053418553|\n",
            "|0.02177|82.5| 2.03|   0| 0.415| 7.61|15.7|   6.27|  2|348|   14.7|395.38| 3.11|42.3|[0.02177,82.5,2.0...|36.146191514394395|\n",
            "|0.02498| 0.0| 1.89|   0| 0.518| 6.54|59.7| 6.2669|  1|422|   15.9|389.96| 8.65|16.5|[0.02498,0.0,1.89...|22.746885437097184|\n",
            "|0.02543|55.0| 3.78|   0| 0.484|6.696|56.4| 5.7321|  5|370|   17.6| 396.9| 7.18|23.9|[0.02543,55.0,3.7...|27.769299940712642|\n",
            "|0.02763|75.0| 2.95|   0| 0.428|6.595|21.8| 5.4011|  3|252|   18.3|395.63| 4.32|30.8|[0.02763,75.0,2.9...| 30.72520875882616|\n",
            "|0.02875|28.0|15.04|   0| 0.464|6.211|28.9| 3.6659|  4|270|   18.2|396.33| 6.21|25.0|[0.02875,28.0,15....|28.649351219272933|\n",
            "|0.03041| 0.0| 5.19|   0| 0.515|5.895|59.6|  5.615|  5|224|   20.2|394.81|10.56|18.5|[0.03041,0.0,5.19...|19.578146685590635|\n",
            "| 0.0315|95.0| 1.47|   0| 0.403|6.975|15.3| 7.6534|  3|402|   17.0| 396.9| 4.56|34.9|[0.0315,95.0,1.47...|29.605091447377312|\n",
            "|0.03237| 0.0| 2.18|   0| 0.458|6.998|45.8| 6.0622|  3|222|   18.7|394.63| 2.94|33.4|[0.03237,0.0,2.18...|28.946444601423302|\n",
            "|0.03427| 0.0| 5.19|   0| 0.515|5.869|46.3| 5.2311|  5|224|   20.2| 396.9|  9.8|19.5|[0.03427,0.0,5.19...| 20.31754513618022|\n",
            "|0.03466|35.0| 6.06|   0|0.4379|6.031|23.3| 6.6407|  1|304|   16.9|362.25| 7.83|19.4|[0.03466,35.0,6.0...|22.979032509331866|\n",
            "|0.03578|20.0| 3.33|   0|0.4429| 7.82|64.5| 4.6947|  5|216|   14.9|387.31| 3.76|45.4|[0.03578,20.0,3.3...| 38.10797692996398|\n",
            "|0.03738| 0.0| 5.19|   0| 0.515| 6.31|38.5| 6.4584|  5|224|   20.2| 389.4| 6.75|20.7|[0.03738,0.0,5.19...| 21.98230064873198|\n",
            "|0.04294|28.0|15.04|   0| 0.464|6.249|77.3|  3.615|  4|270|   18.2| 396.9|10.59|20.6|[0.04294,28.0,15....| 26.75212862015282|\n",
            "+-------+----+-----+----+------+-----+----+-------+---+---+-------+------+-----+----+--------------------+------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uxpTKuQyJ8mH",
        "outputId": "f572d822-e0fb-4d91-8b4d-052726520ad5"
      },
      "source": [
        "predictions.select(predictions.columns[13:]).show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+----+--------------------+------------------+\n",
            "|medv|            features|        prediction|\n",
            "+----+--------------------+------------------+\n",
            "|22.0|[0.01096,55.0,2.2...|26.571482297719307|\n",
            "|32.7|[0.01301,35.0,1.5...|30.167506151964368|\n",
            "|35.4|[0.01311,90.0,1.2...|30.335900587302184|\n",
            "|18.9|[0.0136,75.0,4.0,...|15.228505739892949|\n",
            "|50.0|[0.01501,90.0,1.2...| 44.80928824019599|\n",
            "|30.1|[0.01709,90.0,2.0...|25.264400971073016|\n",
            "|50.0|[0.02009,95.0,2.6...| 41.68916053418553|\n",
            "|42.3|[0.02177,82.5,2.0...|36.146191514394395|\n",
            "|16.5|[0.02498,0.0,1.89...|22.746885437097184|\n",
            "|23.9|[0.02543,55.0,3.7...|27.769299940712642|\n",
            "|30.8|[0.02763,75.0,2.9...| 30.72520875882616|\n",
            "|25.0|[0.02875,28.0,15....|28.649351219272933|\n",
            "|18.5|[0.03041,0.0,5.19...|19.578146685590635|\n",
            "|34.9|[0.0315,95.0,1.47...|29.605091447377312|\n",
            "|33.4|[0.03237,0.0,2.18...|28.946444601423302|\n",
            "|19.5|[0.03427,0.0,5.19...| 20.31754513618022|\n",
            "|19.4|[0.03466,35.0,6.0...|22.979032509331866|\n",
            "|45.4|[0.03578,20.0,3.3...| 38.10797692996398|\n",
            "|20.7|[0.03738,0.0,5.19...| 21.98230064873198|\n",
            "|20.6|[0.04294,28.0,15....| 26.75212862015282|\n",
            "+----+--------------------+------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hiuqE26pKBYV"
      },
      "source": [
        "model.save(\"boston_housing_model\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-3LEk6_uSCxc",
        "outputId": "dbecf751-6b6f-4b3a-b728-bd52986e674c"
      },
      "source": [
        "!ls boston_housing_model"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "data  metadata\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xvfSY3ATX9sl"
      },
      "source": [
        "!ls -tl boston_housing_model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NgwgLq6xgLwi",
        "outputId": "d6e7d90e-88d5-4d7f-bb24-1c536cbf3d22"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PFEMhdWbgVnk"
      },
      "source": [
        "model_save_name = \"boston_housing_model\"\n",
        "path = F\"/content/gdrive/My Drive/boston_housing_model2\" \n",
        "model.save(path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GnIczTktX_mC"
      },
      "source": [
        "from pyspark.ml.regression import LinearRegressionModel\n",
        "\n",
        "loaded_model = LinearRegressionModel.load(path)  # \"boston_housing_model\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rUPqrP4Beb7P"
      },
      "source": [
        "predictions2 = loaded_model.transform(test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cXUj8afbfDLz",
        "outputId": "3b5f81bf-2664-4edd-d414-51c2848b35ba"
      },
      "source": [
        "predictions2.select(predictions.columns[13:]).show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+----+--------------------+------------------+\n",
            "|medv|            features|        prediction|\n",
            "+----+--------------------+------------------+\n",
            "|22.0|[0.01096,55.0,2.2...|26.571482297719307|\n",
            "|32.7|[0.01301,35.0,1.5...|30.167506151964368|\n",
            "|35.4|[0.01311,90.0,1.2...|30.335900587302184|\n",
            "|18.9|[0.0136,75.0,4.0,...|15.228505739892949|\n",
            "|50.0|[0.01501,90.0,1.2...| 44.80928824019599|\n",
            "|30.1|[0.01709,90.0,2.0...|25.264400971073016|\n",
            "|50.0|[0.02009,95.0,2.6...| 41.68916053418553|\n",
            "|42.3|[0.02177,82.5,2.0...|36.146191514394395|\n",
            "|16.5|[0.02498,0.0,1.89...|22.746885437097184|\n",
            "|23.9|[0.02543,55.0,3.7...|27.769299940712642|\n",
            "|30.8|[0.02763,75.0,2.9...| 30.72520875882616|\n",
            "|25.0|[0.02875,28.0,15....|28.649351219272933|\n",
            "|18.5|[0.03041,0.0,5.19...|19.578146685590635|\n",
            "|34.9|[0.0315,95.0,1.47...|29.605091447377312|\n",
            "|33.4|[0.03237,0.0,2.18...|28.946444601423302|\n",
            "|19.5|[0.03427,0.0,5.19...| 20.31754513618022|\n",
            "|19.4|[0.03466,35.0,6.0...|22.979032509331866|\n",
            "|45.4|[0.03578,20.0,3.3...| 38.10797692996398|\n",
            "|20.7|[0.03738,0.0,5.19...| 21.98230064873198|\n",
            "|20.6|[0.04294,28.0,15....| 26.75212862015282|\n",
            "+----+--------------------+------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DAyn3LSafJuv"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}