Spark_NLP_Udemy_MOOC/Healthcare_NLP/FeaturesAssembler.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"sXatvRX899i0"},"source":["![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"]},{"cell_type":"markdown","metadata":{"id":"AOn8d1tcBkK3"},"source":["# **FeaturesAssembler**"]},{"cell_type":"markdown","source":["This notebook will cover the different usages of `FeaturesAssembler` annotator. \n","\n","**📖 Learning Objectives:**\n","\n","1. Understand how to use `FeaturesAssembler`.\n","\n","2. Using `FeaturesAssembler` with several columns.\n","\n","3. Using `FeaturesAssembler` with embeddings.\n","\n","\n","**🔗 Helpful Links:**\n","\n","- Documentation : [FeaturesAssembler](https://nlp.johnsnowlabs.com/docs/en/licensed_annotators#featuresassembler)\n","\n","- Python Docs : [FeaturesAssembler](https://nlp.johnsnowlabs.com/licensed/api/python/reference/autosummary/sparknlp_jsl/annotator/feature_assembler/index.html#module-sparknlp_jsl.annotator.feature_assembler)\n","\n","- Scala Docs : [FeaturesAssembler](https://nlp.johnsnowlabs.com/licensed/api/com/johnsnowlabs/nlp/FeaturesAssembler.html)\n","\n","- For extended examples of usage, see the [Spark NLP Workshop repository](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/healthcare-nlp/08.2.Generic_Classifier.ipynb)."],"metadata":{"id":"WimihSwD3vtH"}},{"cell_type":"markdown","source":["## **📜 Background**\n"],"metadata":{"id":"qL9lcISyFSLv"}},{"cell_type":"markdown","source":["The `FeaturesAssembler` is used to collect features from different columns. It can collect features from single value columns (anything which can be cast to a float, if casts fail then the value is set to 0), array columns, or SparkNLP annotations (if the annotation is an embedding, it takes the embedding, otherwise tries to cast the result field). The output of the transformer is a `feature_vector` annotation (the numeric vector is in the embeddings field).\n","\n","The output of `FeaturesAssembler` (`feature_vector`) is used as input for the **`GenericClassifier, GenericSVMClassifier, GenericLogRegClassifier`** annotators."],"metadata":{"id":"TjDKOoZ4Fc8G"}},{"cell_type":"markdown","metadata":{"id":"MfkkKkbVF309"},"source":["## **🎬 Colab Setup**"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"iMkMQtZNF2n-","executionInfo":{"status":"ok","timestamp":1685715681839,"user_tz":-180,"elapsed":4984,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}}},"outputs":[],"source":["!pip install -q johnsnowlabs"]},{"cell_type":"code","source":["from google.colab import files\n","print('Please Upload your John Snow Labs License using the button below')\n","license_keys = files.upload()"],"metadata":{"id":"zbUfCKmmiVCG"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from johnsnowlabs import nlp\n","\n","nlp.install()"],"metadata":{"id":"NulWi4_f4GN5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from johnsnowlabs import nlp, medical\n","import pandas as pd\n","import numpy as np\n","\n","spark = nlp.start()"],"metadata":{"id":"2GLXxe1Q0Iln"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## **🖨️ Input/Output Annotation Types**"],"metadata":{"id":"9Fbbk1bqcuA5"}},{"cell_type":"markdown","source":["- Input: `NONE`\n","\n","- Output: `FEATURE_VECTOR`"],"metadata":{"id":"0yFIrr5acsiU"}},{"cell_type":"markdown","source":["## **🔎 Parameters**\n"],"metadata":{"id":"b2YJehUKMhb0"}},{"cell_type":"markdown","source":["The parameters below are used for `FeaturesAssembler`.\n","\n","\n","- `inputCols`: The name of the columns containing the input annotations. It can read either a String column name or an Array of strings (column names).\n","- `outputCol`: The name of the column in Document type that is generated. We can specify only one column here.\n","\n","\n","All the parameters can be set using the corresponding set method in the camel case. For example, `.setInputcols()`."],"metadata":{"id":"oidLDoS94asU"}},{"cell_type":"markdown","source":["## Using `FeaturesAssembler` with several columns"],"metadata":{"id":"f3JnKer_hWkb"}},{"cell_type":"markdown","source":["Let's load a dataset and create some float valued features to be used in `FeaturesAssembler` "],"metadata":{"id":"jgsSa5DULkTM"}},{"cell_type":"code","source":["!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/data/petfinder-mini.csv"],"metadata":{"id":"4f4GGRCoGKIs","executionInfo":{"status":"ok","timestamp":1685716532609,"user_tz":-180,"elapsed":491,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}}},"execution_count":5,"outputs":[]},{"cell_type":"code","source":["# prepare the data\n","dataframe = pd.read_csv('petfinder-mini.csv')\n","dataframe['target'] = np.where(dataframe['AdoptionSpeed']==4, 0, 1) # In the original dataset \"4\" indicates the pet was not adopted.\n","dataframe.Description = dataframe.Description.fillna('- no description -')\n","dataframe.head(3)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":257},"id":"mkQCOuVMGKGI","executionInfo":{"status":"ok","timestamp":1685716532610,"user_tz":-180,"elapsed":15,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"f4d8aac1-50b5-4904-9e13-41959f866702"},"execution_count":6,"outputs":[{"output_type":"execute_result","data":{"text/plain":["  Type  Age                Breed1 Gender Color1 Color2 MaturitySize FurLength  \\\n","0  Cat    3                 Tabby   Male  Black  White        Small     Short   \n","1  Cat    1  Domestic Medium Hair   Male  Black  Brown       Medium    Medium   \n","2  Dog    1           Mixed Breed   Male  Brown  White       Medium    Medium   \n","\n","  Vaccinated Sterilized   Health  Fee  \\\n","0         No         No  Healthy  100   \n","1   Not Sure   Not Sure  Healthy    0   \n","2        Yes         No  Healthy    0   \n","\n","                                         Description  PhotoAmt  AdoptionSpeed  \\\n","0  Nibble is a 3+ month old ball of cuteness. He ...         1              2   \n","1  I just found it alone yesterday near my apartm...         2              0   \n","2  Their pregnant mother was dumped by her irresp...         7              3   \n","\n","   target  \n","0       1  \n","1       1  \n","2       1  "],"text/html":["\n","  <div id=\"df-192bb3a4-ae8e-448c-998d-93d443dda3a8\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Type</th>\n","      <th>Age</th>\n","      <th>Breed1</th>\n","      <th>Gender</th>\n","      <th>Color1</th>\n","      <th>Color2</th>\n","      <th>MaturitySize</th>\n","      <th>FurLength</th>\n","      <th>Vaccinated</th>\n","      <th>Sterilized</th>\n","      <th>Health</th>\n","      <th>Fee</th>\n","      <th>Description</th>\n","      <th>PhotoAmt</th>\n","      <th>AdoptionSpeed</th>\n","      <th>target</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Cat</td>\n","      <td>3</td>\n","      <td>Tabby</td>\n","      <td>Male</td>\n","      <td>Black</td>\n","      <td>White</td>\n","      <td>Small</td>\n","      <td>Short</td>\n","      <td>No</td>\n","      <td>No</td>\n","      <td>Healthy</td>\n","      <td>100</td>\n","      <td>Nibble is a 3+ month old ball of cuteness. He ...</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Cat</td>\n","      <td>1</td>\n","      <td>Domestic Medium Hair</td>\n","      <td>Male</td>\n","      <td>Black</td>\n","      <td>Brown</td>\n","      <td>Medium</td>\n","      <td>Medium</td>\n","      <td>Not Sure</td>\n","      <td>Not Sure</td>\n","      <td>Healthy</td>\n","      <td>0</td>\n","      <td>I just found it alone yesterday near my apartm...</td>\n","      <td>2</td>\n","      <td>0</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Dog</td>\n","      <td>1</td>\n","      <td>Mixed Breed</td>\n","      <td>Male</td>\n","      <td>Brown</td>\n","      <td>White</td>\n","      <td>Medium</td>\n","      <td>Medium</td>\n","      <td>Yes</td>\n","      <td>No</td>\n","      <td>Healthy</td>\n","      <td>0</td>\n","      <td>Their pregnant mother was dumped by her irresp...</td>\n","      <td>7</td>\n","      <td>3</td>\n","      <td>1</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-192bb3a4-ae8e-448c-998d-93d443dda3a8')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-192bb3a4-ae8e-448c-998d-93d443dda3a8 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-192bb3a4-ae8e-448c-998d-93d443dda3a8');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":6}]},{"cell_type":"markdown","source":["To create numerical features we will use OneHotEncoder and TF-IDF methods. Because `FeaturesAssembler` can not cast string values to folat."],"metadata":{"id":"G1HMobXiL3PY"}},{"cell_type":"code","source":["from sklearn.compose import make_column_transformer\n","# from sklearn.compose import ColumnTransformer\n","from sklearn.feature_extraction.text import TfidfVectorizer\n","from sklearn.preprocessing import OneHotEncoder, StandardScaler\n","from sklearn.linear_model import LogisticRegression\n","from sklearn.model_selection import train_test_split\n","\n","column_trans = make_column_transformer(\n","     (OneHotEncoder(), ['Type', 'Breed1', 'Gender', 'Color1', 'Color2', 'MaturitySize',\n","       'FurLength', 'Vaccinated', 'Sterilized', 'Health']),\n","     (TfidfVectorizer(max_features=15,  norm='l2', ngram_range=(1, 3)), 'Description'),\n","     remainder=StandardScaler())\n","\n","X = column_trans.fit_transform(dataframe.drop(['target'], axis=1))\n","\n","y = dataframe.target"],"metadata":{"id":"2UONE7L4GKDj","executionInfo":{"status":"ok","timestamp":1685716540803,"user_tz":-180,"elapsed":8206,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}}},"execution_count":7,"outputs":[]},{"cell_type":"code","source":["df = pd.DataFrame.sparse.from_spmatrix(X)\n","\n","feature_columns = ['col_{}'.format(i) for i in range(X.shape[1])]\n","\n","df.columns = feature_columns\n","\n","df['target']= y\n","\n","df.head(3)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":237},"id":"hCH6RigoGKAx","executionInfo":{"status":"ok","timestamp":1685716540804,"user_tz":-180,"elapsed":21,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"f15b6396-2150-4c0b-f00e-bfa99bee2eaa"},"execution_count":8,"outputs":[{"output_type":"execute_result","data":{"text/plain":["   col_0  col_1  col_2  col_3  col_4  col_5  col_6  col_7  col_8  col_9  ...  \\\n","0    1.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  ...   \n","1    1.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  ...   \n","2    0.0    1.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  ...   \n","\n","    col_209   col_210  col_211  col_212  col_213   col_214   col_215  \\\n","0  0.381339  0.299670      0.0      0.0      0.0 -0.452479  0.950288   \n","1  0.000000  0.826697      0.0      0.0      0.0 -0.555981 -0.299388   \n","2  0.366811  0.720633      0.0      0.0      0.0 -0.555981 -0.299388   \n","\n","    col_216   col_217  target  \n","0 -0.829762 -0.414688       1  \n","1 -0.511871 -2.119392       1  \n","2  1.077583  0.437664       1  \n","\n","[3 rows x 219 columns]"],"text/html":["\n","  <div id=\"df-73f81114-58d7-498d-a199-289175742414\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>col_0</th>\n","      <th>col_1</th>\n","      <th>col_2</th>\n","      <th>col_3</th>\n","      <th>col_4</th>\n","      <th>col_5</th>\n","      <th>col_6</th>\n","      <th>col_7</th>\n","      <th>col_8</th>\n","      <th>col_9</th>\n","      <th>...</th>\n","      <th>col_209</th>\n","      <th>col_210</th>\n","      <th>col_211</th>\n","      <th>col_212</th>\n","      <th>col_213</th>\n","      <th>col_214</th>\n","      <th>col_215</th>\n","      <th>col_216</th>\n","      <th>col_217</th>\n","      <th>target</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>0.381339</td>\n","      <td>0.299670</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-0.452479</td>\n","      <td>0.950288</td>\n","      <td>-0.829762</td>\n","      <td>-0.414688</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>0.000000</td>\n","      <td>0.826697</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-0.555981</td>\n","      <td>-0.299388</td>\n","      <td>-0.511871</td>\n","      <td>-2.119392</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0.0</td>\n","      <td>1.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>0.366811</td>\n","      <td>0.720633</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-0.555981</td>\n","      <td>-0.299388</td>\n","      <td>1.077583</td>\n","      <td>0.437664</td>\n","      <td>1</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>3 rows × 219 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-73f81114-58d7-498d-a199-289175742414')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-73f81114-58d7-498d-a199-289175742414 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-73f81114-58d7-498d-a199-289175742414');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":8}]},{"cell_type":"code","source":["spark_df = spark.createDataFrame(df)\n","spark_df.select(spark_df.columns[-10:]).show(2)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2fA9PPleGJ6o","executionInfo":{"status":"ok","timestamp":1685716565874,"user_tz":-180,"elapsed":25089,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"beb8dae3-dcaa-4027-fd11-2627360f8c99"},"execution_count":9,"outputs":[{"output_type":"stream","name":"stdout","text":["+-------------------+------------------+-------+-------+-------+-------------------+--------------------+-------------------+--------------------+------+\n","|            col_209|           col_210|col_211|col_212|col_213|            col_214|             col_215|            col_216|             col_217|target|\n","+-------------------+------------------+-------+-------+-------+-------------------+--------------------+-------------------+--------------------+------+\n","|0.38133910553960104|0.2996700768545381|    0.0|    0.0|    0.0|-0.4524794726808656|  0.9502875792756131|-0.8297616989552165|-0.41468778162984526|     1|\n","|                0.0|0.8266974703081306|    0.0|    0.0|    0.0| -0.555981017719065|-0.29938816657135553|-0.5118709929431844| -2.1193921951924772|     1|\n","+-------------------+------------------+-------+-------+-------+-------------------+--------------------+-------------------+--------------------+------+\n","only showing top 2 rows\n","\n"]}]},{"cell_type":"code","source":["# dispaly feature columns' names\n","print(feature_columns)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"CQ7u8U59NgBS","executionInfo":{"status":"ok","timestamp":1685716565875,"user_tz":-180,"elapsed":65,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"e93620cf-05d0-4ab2-bf2b-4e4680e00d81"},"execution_count":10,"outputs":[{"output_type":"stream","name":"stdout","text":["['col_0', 'col_1', 'col_2', 'col_3', 'col_4', 'col_5', 'col_6', 'col_7', 'col_8', 'col_9', 'col_10', 'col_11', 'col_12', 'col_13', 'col_14', 'col_15', 'col_16', 'col_17', 'col_18', 'col_19', 'col_20', 'col_21', 'col_22', 'col_23', 'col_24', 'col_25', 'col_26', 'col_27', 'col_28', 'col_29', 'col_30', 'col_31', 'col_32', 'col_33', 'col_34', 'col_35', 'col_36', 'col_37', 'col_38', 'col_39', 'col_40', 'col_41', 'col_42', 'col_43', 'col_44', 'col_45', 'col_46', 'col_47', 'col_48', 'col_49', 'col_50', 'col_51', 'col_52', 'col_53', 'col_54', 'col_55', 'col_56', 'col_57', 'col_58', 'col_59', 'col_60', 'col_61', 'col_62', 'col_63', 'col_64', 'col_65', 'col_66', 'col_67', 'col_68', 'col_69', 'col_70', 'col_71', 'col_72', 'col_73', 'col_74', 'col_75', 'col_76', 'col_77', 'col_78', 'col_79', 'col_80', 'col_81', 'col_82', 'col_83', 'col_84', 'col_85', 'col_86', 'col_87', 'col_88', 'col_89', 'col_90', 'col_91', 'col_92', 'col_93', 'col_94', 'col_95', 'col_96', 'col_97', 'col_98', 'col_99', 'col_100', 'col_101', 'col_102', 'col_103', 'col_104', 'col_105', 'col_106', 'col_107', 'col_108', 'col_109', 'col_110', 'col_111', 'col_112', 'col_113', 'col_114', 'col_115', 'col_116', 'col_117', 'col_118', 'col_119', 'col_120', 'col_121', 'col_122', 'col_123', 'col_124', 'col_125', 'col_126', 'col_127', 'col_128', 'col_129', 'col_130', 'col_131', 'col_132', 'col_133', 'col_134', 'col_135', 'col_136', 'col_137', 'col_138', 'col_139', 'col_140', 'col_141', 'col_142', 'col_143', 'col_144', 'col_145', 'col_146', 'col_147', 'col_148', 'col_149', 'col_150', 'col_151', 'col_152', 'col_153', 'col_154', 'col_155', 'col_156', 'col_157', 'col_158', 'col_159', 'col_160', 'col_161', 'col_162', 'col_163', 'col_164', 'col_165', 'col_166', 'col_167', 'col_168', 'col_169', 'col_170', 'col_171', 'col_172', 'col_173', 'col_174', 'col_175', 'col_176', 'col_177', 'col_178', 'col_179', 'col_180', 'col_181', 'col_182', 'col_183', 'col_184', 'col_185', 'col_186', 'col_187', 'col_188', 'col_189', 'col_190', 'col_191', 'col_192', 'col_193', 'col_194', 'col_195', 'col_196', 'col_197', 'col_198', 'col_199', 'col_200', 'col_201', 'col_202', 'col_203', 'col_204', 'col_205', 'col_206', 'col_207', 'col_208', 'col_209', 'col_210', 'col_211', 'col_212', 'col_213', 'col_214', 'col_215', 'col_216', 'col_217']\n"]}]},{"cell_type":"markdown","source":["`FeaturesAssembler` will combine all `feature_columns` above into one column which is a `feature_vector` type."],"metadata":{"id":"6-3RTVPYN2pk"}},{"cell_type":"code","source":["features_asm = medical.FeaturesAssembler()\\\n","    .setInputCols(feature_columns)\\\n","    .setOutputCol(\"features\")"],"metadata":{"id":"aSy4nROrGJ2h","executionInfo":{"status":"ok","timestamp":1685716565875,"user_tz":-180,"elapsed":6,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}}},"execution_count":11,"outputs":[]},{"cell_type":"code","source":["result = features_asm.transform(spark_df)"],"metadata":{"id":"5Yki1gFrKub_","executionInfo":{"status":"ok","timestamp":1685716567279,"user_tz":-180,"elapsed":1409,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}}},"execution_count":12,"outputs":[]},{"cell_type":"code","source":["result.show(2)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Zuka1-ClNqpA","executionInfo":{"status":"ok","timestamp":1685716569578,"user_tz":-180,"elapsed":2301,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"c2f845ca-05f8-4ee7-b7d6-765c27bdd9b5"},"execution_count":13,"outputs":[{"output_type":"stream","name":"stdout","text":["+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------------------+-------+------------------+------------------+-------+----------------+------------------+------------------+------------------+-------+-------------------+------------------+-------+-------+-------+-------------------+--------------------+-------------------+--------------------+------+--------------------+\n","|col_0|col_1|col_2|col_3|col_4|col_5|col_6|col_7|col_8|col_9|col_10|col_11|col_12|col_13|col_14|col_15|col_16|col_17|col_18|col_19|col_20|col_21|col_22|col_23|col_24|col_25|col_26|col_27|col_28|col_29|col_30|col_31|col_32|col_33|col_34|col_35|col_36|col_37|col_38|col_39|col_40|col_41|col_42|col_43|col_44|col_45|col_46|col_47|col_48|col_49|col_50|col_51|col_52|col_53|col_54|col_55|col_56|col_57|col_58|col_59|col_60|col_61|col_62|col_63|col_64|col_65|col_66|col_67|col_68|col_69|col_70|col_71|col_72|col_73|col_74|col_75|col_76|col_77|col_78|col_79|col_80|col_81|col_82|col_83|col_84|col_85|col_86|col_87|col_88|col_89|col_90|col_91|col_92|col_93|col_94|col_95|col_96|col_97|col_98|col_99|col_100|col_101|col_102|col_103|col_104|col_105|col_106|col_107|col_108|col_109|col_110|col_111|col_112|col_113|col_114|col_115|col_116|col_117|col_118|col_119|col_120|col_121|col_122|col_123|col_124|col_125|col_126|col_127|col_128|col_129|col_130|col_131|col_132|col_133|col_134|col_135|col_136|col_137|col_138|col_139|col_140|col_141|col_142|col_143|col_144|col_145|col_146|col_147|col_148|col_149|col_150|col_151|col_152|col_153|col_154|col_155|col_156|col_157|col_158|col_159|col_160|col_161|col_162|col_163|col_164|col_165|col_166|col_167|col_168|col_169|col_170|col_171|col_172|col_173|col_174|col_175|col_176|col_177|col_178|col_179|col_180|col_181|col_182|col_183|col_184|col_185|col_186|col_187|col_188|col_189|col_190|col_191|col_192|col_193|col_194|col_195|col_196|col_197|col_198|            col_199|col_200|           col_201|           col_202|col_203|         col_204|           col_205|           col_206|           col_207|col_208|            col_209|           col_210|col_211|col_212|col_213|            col_214|             col_215|            col_216|             col_217|target|            features|\n","+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------------------+-------+------------------+------------------+-------+----------------+------------------+------------------+------------------+-------+-------------------+------------------+-------+-------+-------+-------------------+--------------------+-------------------+--------------------+------+--------------------+\n","|  1.0|  0.0|  0.0|  0.0|  0.0|  0.0|  0.0|  0.0|  0.0|  0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    1.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    1.0|    1.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    1.0|    0.0|    0.0|    0.0|    1.0|    0.0|    0.0|    1.0|    1.0|    0.0|    0.0|    1.0|    0.0|    0.0|    1.0|    0.0|    0.0|0.29305349389990176|    0.0|0.3457686156544891|0.2605937347155163|    0.0|             0.0|0.4404315292913929|0.3251645222253951|0.4378596266895194|    0.0|0.38133910553960104|0.2996700768545381|    0.0|    0.0|    0.0|-0.4524794726808656|  0.9502875792756131|-0.8297616989552165|-0.41468778162984526|     1|[{feature_vector,...|\n","|  1.0|  0.0|  0.0|  0.0|  0.0|  0.0|  0.0|  0.0|  0.0|  0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   1.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|   0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    1.0|    1.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    1.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    0.0|    1.0|    0.0|    0.0|    1.0|    0.0|    0.0|    1.0|    0.0|    0.0|    1.0|    0.0|    1.0|    0.0|    0.0|                0.0|    0.0|               0.0|               0.0|    0.0|0.56264668539514|               0.0|               0.0|               0.0|    0.0|                0.0|0.8266974703081306|    0.0|    0.0|    0.0| -0.555981017719065|-0.29938816657135553|-0.5118709929431844| -2.1193921951924772|     1|[{feature_vector,...|\n","+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------------------+-------+------------------+------------------+-------+----------------+------------------+------------------+------------------+-------+-------------------+------------------+-------+-------+-------+-------------------+--------------------+-------------------+--------------------+------+--------------------+\n","only showing top 2 rows\n","\n"]}]},{"cell_type":"code","source":["result.select(\"features\").show(10, truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"znE6fr11K2_b","executionInfo":{"status":"ok","timestamp":1685716569579,"user_tz":-180,"elapsed":8,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"2cb6906e-541a-4b92-9c75-6178458b1cc0"},"execution_count":14,"outputs":[{"output_type":"stream","name":"stdout","text":["+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","|features                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |\n","+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.2930535, 0.0, 0.34576863, 0.26059374, 0.0, 0.0, 0.44043154, 0.32516453, 0.43785962, 0.0, 0.3813391, 0.29967007, 0.0, 0.0, 0.0, -0.45247948, 0.9502876, -0.8297617, -0.41468778]}]                        |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5626467, 0.0, 0.0, 0.0, 0.0, 0.0, 0.82669747, 0.0, 0.0, 0.0, -0.55598104, -0.29938817, -0.511871, -2.1193922]}]                                                                 |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.2818889, 0.0, 0.3325957, 0.0, 0.21963617, 0.196184, 0.2118261, 0.15638828, 0.0, 0.0, 0.36681104, 0.7206335, 0.0, 0.0, 0.0, -0.55598104, -0.29938817, 1.0775825, 0.43766442]}]                            |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.6571836, 0.0, 0.43398422, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.40623134, 0.0, 0.46340427, -0.4007287, 1.5751255, 1.3954732, -0.41468778]}]                                                     |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.088158086, 0.1443838, 0.3120485, 0.7839335, 0.0, 0.12270939, 0.13249323, 0.19563584, 0.13171953, 0.0, 0.22943339, 0.18029706, 0.0, 0.0, 0.29338247, -0.55598104, -0.29938817, -0.19398029, -0.41468778]}]|\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.73534626, 0.0, 0.0, 0.0, 0.6776916, 0.0, 0.0, 0.0, -0.45247948, -0.29938817, -0.511871, -0.41468778]}]                                                                |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.19371788, 0.2860385, 0.57776, 0.0, 0.33545375, 0.65902925, 0.0, 0.0, 0.0, 0.01327748, 3.449639, -0.19398029, -1.26704]}]                                                   |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.42674515, 0.0, 0.0, 0.0, 0.0, 0.0, 0.64135736, 0.0, 0.63761216, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.50423026, -0.29938817, 0.75969183, -1.26704]}]                                                          |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.16864297, 0.27620056, 0.5969365, 0.0, 0.26279926, 0.23473826, 0.25345436, 0.0, 0.0, 0.0, 0.0, 0.1724506, 0.491987, 0.0, 0.28061458, 0.01327748, -0.29938817, -0.511871, 1.2900167]}]                     |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.50423026, -0.29938817, 1.0775825, -1.26704]}]                                                                                |\n","+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","only showing top 10 rows\n","\n"]}]},{"cell_type":"markdown","source":["Now we generated `features` column and it can be used in `GenericClassifier, GenericSVMClassifier, GenericLogRegClassifier` annotators as an input column."],"metadata":{"id":"bpfFeVCvPIyq"}},{"cell_type":"markdown","source":["## Using `FeaturesAssembler` with embeddings\n","\n","\n"],"metadata":{"id":"EymDTiSEQFiV"}},{"cell_type":"markdown","source":["Load some data to extract embeddings."],"metadata":{"id":"aSU7rowFjvAm"}},{"cell_type":"code","source":["!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/data/mtsamples_classifier.csv"],"metadata":{"id":"LMSlHGXXQIU8","executionInfo":{"status":"ok","timestamp":1685716570105,"user_tz":-180,"elapsed":532,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}}},"execution_count":15,"outputs":[]},{"cell_type":"code","source":["spark_df = spark.read.csv(\"mtsamples_classifier.csv\", header = True)\n","\n","spark_df.show(10,truncate=100)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"tsmDTY87Qbxf","executionInfo":{"status":"ok","timestamp":1685716571365,"user_tz":-180,"elapsed":1262,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"18fdbd18-1151-4663-bd63-fd39e847d358"},"execution_count":16,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------------+----------------------------------------------------------------------------------------------------+\n","|        category|                                                                                                text|\n","+----------------+----------------------------------------------------------------------------------------------------+\n","|Gastroenterology| PROCEDURES PERFORMED: Colonoscopy. INDICATIONS: Renewed symptoms likely consistent with active f...|\n","|Gastroenterology| OPERATION 1. Ivor-Lewis esophagogastrectomy. 2. Feeding jejunostomy. 3. Placement of two right-s...|\n","|Gastroenterology| PREOPERATIVE DIAGNOSES: 1. Gastroesophageal reflux disease. 2. Chronic dyspepsia. POSTOPERATIVE ...|\n","|Gastroenterology| PROCEDURE: Colonoscopy. PREOPERATIVE DIAGNOSES: Rectal bleeding and perirectal abscess. POSTOPER...|\n","|Gastroenterology| PREOPERATIVE DIAGNOSIS: Right colon tumor. POSTOPERATIVE DIAGNOSES: 1. Right colon cancer. 2. As...|\n","|Gastroenterology| PREOPERATIVE DIAGNOSIS (ES): 1. Cholelithiasis. 2. Cholecystitis. POSTOPERATIVE DIAGNOSIS (ES): ...|\n","|Gastroenterology| The patient was taken to the GI lab and placed in the left lateral supine position. Continuous p...|\n","|Gastroenterology| PREOPERATIVE DIAGNOSIS: Gastrostomy (gastrocutaneous fistula). POSTOPERATIVE DIAGNOSIS: Gastrost...|\n","|Gastroenterology| PREOPERATIVE DIAGNOSIS: Acute acalculous cholecystitis. POSTOPERATIVE DIAGNOSIS: Acute hemorrhag...|\n","|Gastroenterology| HISTORY OF PRESENT ILLNESS: This is a 10-year-old who was found with biliary atresia and underwe...|\n","+----------------+----------------------------------------------------------------------------------------------------+\n","only showing top 10 rows\n","\n"]}]},{"cell_type":"markdown","source":["Now let's create a pipeline that extracts sentence embeddings and creates a `feature_vector` at the end using `FeaturesAssembler`."],"metadata":{"id":"ufsMsr68Q6Ad"}},{"cell_type":"code","source":["document_assembler = nlp.DocumentAssembler()\\\n","        .setInputCol(\"text\")\\\n","        .setOutputCol(\"document\")\n","\n","tokenizer = nlp.Tokenizer() \\\n","            .setInputCols([\"document\"]) \\\n","            .setOutputCol(\"token\")\n","\n","word_embeddings = nlp.WordEmbeddingsModel.pretrained(\"embeddings_healthcare_100d\",\"en\",\"clinical/models\")\\\n","        .setInputCols([\"document\",\"token\"])\\\n","        .setOutputCol(\"word_embeddings\")\n","\n","sentence_embeddings = nlp.SentenceEmbeddings() \\\n","        .setInputCols([\"document\", \"word_embeddings\"]) \\\n","        .setOutputCol(\"sentence_embeddings\") \\\n","        .setPoolingStrategy(\"AVERAGE\")\n","\n","features_asm =medical.FeaturesAssembler()\\\n","    .setInputCols([\"sentence_embeddings\"])\\\n","    .setOutputCol(\"features\")\n","\n","embeddings_pipeline = nlp.Pipeline(\n","    stages = [\n","        document_assembler,\n","        tokenizer,\n","        word_embeddings,\n","        sentence_embeddings,\n","        features_asm\n","    ])"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"RSFL0WiKQe-w","executionInfo":{"status":"ok","timestamp":1685716595330,"user_tz":-180,"elapsed":23969,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"50aece88-c80c-4d90-f582-41937fb4b62c"},"execution_count":17,"outputs":[{"output_type":"stream","name":"stdout","text":["embeddings_healthcare_100d download started this may take some time.\n","Approximate size to download 475.8 MB\n","[OK!]\n"]}]},{"cell_type":"code","source":["result = embeddings_pipeline.fit(spark_df).transform(spark_df)"],"metadata":{"id":"q0qUktZqRS8L","executionInfo":{"status":"ok","timestamp":1685716595855,"user_tz":-180,"elapsed":543,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}}},"execution_count":18,"outputs":[]},{"cell_type":"code","source":["result.show(2)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1685716597855,"user_tz":-180,"elapsed":2002,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"aae460b9-4561-4f0b-ef6b-c33ff34a15c4","id":"VCkXc5PNReNn"},"execution_count":19,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n","|        category|                text|            document|               token|     word_embeddings| sentence_embeddings|            features|\n","+----------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n","|Gastroenterology| PROCEDURES PERFO...|[{document, 0, 30...|[{token, 1, 10, P...|[{word_embeddings...|[{sentence_embedd...|[{feature_vector,...|\n","|Gastroenterology| OPERATION 1. Ivo...|[{document, 0, 59...|[{token, 1, 9, OP...|[{word_embeddings...|[{sentence_embedd...|[{feature_vector,...|\n","+----------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n","only showing top 2 rows\n","\n"]}]},{"cell_type":"code","source":["result.select(\"sentence_embeddings.embeddings\").show(5, truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"11DPEk3JUr86","executionInfo":{"status":"ok","timestamp":1685716597856,"user_tz":-180,"elapsed":5,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"c93a0bf3-c7a5-43d0-dd66-7b35068ce5cd"},"execution_count":20,"outputs":[{"output_type":"stream","name":"stdout","text":["+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","|embeddings                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n","+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","|[[-0.00896873, 0.011731416, 0.12154201, 0.1149235, -0.14689414, 0.0103584975, 0.053073216, -0.056412186, -0.05143186, -0.0118978135, -0.12175384, -0.035894137, 0.11812756, 0.094671555, 0.15838866, 0.15260744, -0.004094441, -0.13675772, -0.07472433, -0.035856977, -0.026730005, -0.21840473, 0.029632289, -0.011515695, -0.20407394, -0.07848257, 0.040990185, 0.23028605, 0.077140555, 0.066990435, 0.015219222, -0.10295644, 0.038072545, 0.10786369, 0.121525764, -0.09569349, -0.06309264, 0.2778952, 0.06462455, -0.10851931, -0.14370486, -0.1466352, 0.08354363, -0.078758985, -0.08377953, 0.12384644, -0.23281692, -0.25607574, 0.16399069, -0.07780675, -0.18302177, -0.18325584, -0.12128636, -0.0010129504, 0.0070792097, 0.20506753, 0.034964647, 0.058425985, 0.19572404, -0.103953235, -0.20159312, -0.099047214, -0.07337802, -0.03713124, -0.055443633, 0.11107734, 0.048563413, -0.038048305, -0.020617828, 0.17082842, 0.069010496, 0.08457101, -0.038229663, 0.073144384, -0.092326105, -0.10054428, -4.3286112E-4, -0.046703782, -0.080231875, 0.02524295, 0.01368699, -0.19783853, -0.03501917, 0.13324805, 0.09053264, -0.0958231, -0.0032442473, 0.19218525, -0.027179888, 0.030672349, 0.12848215, -0.014700146, -0.089054875, 0.13839856, -0.15778734, 0.07103226, -0.060303356, 0.20854644, -0.008389737, -0.1473986]]|\n","|[[0.05308125, -0.015051386, 0.14103058, 0.10856558, -0.2342213, 0.003566569, 0.04778356, -0.004497177, -0.0535818, 0.0026637823, -0.08539946, -0.053373538, 0.09551983, 0.14680131, 0.19040567, 0.12977865, -0.033773836, -0.1319684, -0.040731516, -0.008372502, -0.006487461, -0.22866476, 0.052934043, -0.08824037, -0.26920623, -0.06628319, -0.024219586, 0.24569845, 0.0031692924, 0.057381, 0.028687425, -0.16567227, 0.02853736, 0.10968278, 0.17362879, -0.113311104, -0.062397536, 0.3374604, 0.029003158, -0.06641296, -0.1424927, -0.20313045, 0.031945523, -0.08096304, -0.060245346, 0.20948681, -0.2579919, -0.307684, 0.06597106, -0.06554805, -0.20020092, -0.21681747, -0.13199064, 0.03438211, -0.017373739, 0.28633022, 0.020454949, 0.012209459, 0.16254629, -0.17099468, -0.21445248, -0.042360373, -0.04671074, -0.081191055, -0.16396351, 0.1635775, 0.07605838, -0.055918057, -0.056702156, 0.11127068, 0.04990405, 0.06818552, -0.13117148, 0.18798989, -0.07119257, -0.15537563, 0.042161517, -0.079932965, -0.12116789, 0.08042555, 0.049063064, -0.1816178, -0.02954805, 0.21658188, 0.09907657, -0.10253809, -0.012852975, 0.18569835, -0.03166329, 0.017464636, 0.091833346, 0.035398744, -0.20562796, 0.15094037, -0.20429432, 0.049046826, -0.04301696, 0.2677997, -0.025445838, -0.070918486]]                      |\n","|[[0.0054126224, 0.024950134, 0.11474294, 0.10822452, -0.2555815, 0.006825694, 0.0849884, -0.013415727, -0.067355156, -0.051267654, -0.1150451, -0.10311411, 0.12731767, 0.10249366, 0.16624388, 0.15492792, -0.05668665, -0.15464978, -0.036532804, -0.051754355, -0.007137836, -0.2513172, 0.08712005, -0.045029067, -0.18400155, -0.10452683, 0.027711688, 0.269648, 0.115273856, 0.09415018, 0.007242371, -0.14897408, 0.041124213, 0.08991895, 0.17528482, -0.0702524, -0.095246, 0.26005113, 0.08287458, -0.12394875, -0.1654959, -0.17744197, 0.12454772, -0.10536424, -0.058388017, 0.14923076, -0.24283154, -0.31404442, 0.1759314, -0.109916285, -0.18941985, -0.22670338, -0.102816485, 0.07351742, -0.0046835076, 0.22211693, 0.07862632, 0.06029365, 0.18198037, -0.15607189, -0.21938755, -0.0648467, -0.06229591, -0.061715987, -0.051075175, 0.13917515, 0.042945575, -0.06531585, -0.04472695, 0.13748632, 0.06643436, 0.1394113, -0.11119953, 0.11860371, -0.11418141, -0.12697497, 0.0050508482, -0.084115386, -0.105697244, -0.025213558, 0.033772327, -0.17188449, -0.058286436, 0.17098169, 0.058491495, -0.14792554, -0.014757689, 0.21050486, -0.028120184, 0.071195774, 0.13185748, 0.008545133, -0.1287004, 0.189709, -0.15126283, 0.070789315, -0.03831092, 0.25867718, 0.03666226, -0.15486968]]                           |\n","|[[0.008756783, 0.05039759, 0.16213615, 0.097373165, -0.23011582, 0.05685288, 0.045038182, -0.043422982, -0.10621742, 0.021861993, -0.103197925, -0.10453307, 0.1128905, 0.105824575, 0.19338556, 0.1283347, -0.009834639, -0.1456381, -0.09109473, -0.015304626, -0.02358634, -0.28288808, 0.075918205, -0.11495868, -0.21861364, -0.042715643, -0.0021146596, 0.28538048, 0.1250176, 0.06596709, 0.046708006, -0.14905003, 0.018370079, 0.12479546, 0.17377286, -0.12793261, -0.08494885, 0.3624782, 0.06705852, -0.07460713, -0.15057686, -0.19423227, 0.032626033, -0.10774493, -0.08319685, 0.2038898, -0.24785702, -0.32097295, 0.1538476, -0.06367145, -0.20836613, -0.23597735, -0.12798108, 0.07948215, -0.027333453, 0.26723576, 0.016597308, 0.08272391, 0.16125162, -0.14834878, -0.2491029, -0.051455367, -0.043988984, -0.06324682, -0.119092226, 0.14168923, 0.13753091, -0.043976754, -0.06759758, 0.14472446, 0.07076088, 0.05267864, -0.07459972, 0.08896091, -0.06585022, -0.13084342, 0.0100373, -0.047292944, -0.14638452, -0.029784478, 0.033027586, -0.20373267, 0.024487026, 0.19487573, 0.09436645, -0.14655155, 0.0019440161, 0.18331449, -0.044212725, -0.0066170166, 0.0941082, 0.0058092014, -0.18064514, 0.16326123, -0.18090741, 0.08155668, -0.060303602, 0.30489156, -0.03252674, -0.13991456]]                       |\n","|[[0.028661652, 0.0113291005, 0.15641849, 0.10562915, -0.18895394, 0.010692466, 0.028517308, -0.046971526, -0.06753073, 0.0015370062, -0.0794119, -0.05521748, 0.109319516, 0.12964095, 0.1794887, 0.1544829, -0.011513941, -0.1308672, -0.06253322, -2.441166E-4, -0.015246368, -0.28145686, 0.052070346, -0.08519331, -0.2572522, -0.061911665, -7.287748E-4, 0.26109117, 0.05577541, 0.0594182, 0.026305387, -0.13174595, 0.03511284, 0.14048967, 0.16914414, -0.12223352, -0.02627304, 0.32095695, 0.046736695, -0.076408096, -0.1273522, -0.17759354, 0.038809806, -0.10235544, -0.06325414, 0.18265921, -0.25014886, -0.2927684, 0.12070944, -0.059869632, -0.21926697, -0.21340933, -0.11169393, 0.059311472, -0.025239093, 0.27021858, 0.028728224, 0.03955428, 0.18724075, -0.14486238, -0.25880367, -0.0442654, -0.041866943, -0.056578852, -0.1482131, 0.1593034, 0.10229411, -0.01818499, -0.06094723, 0.10570261, 0.056156162, 0.070567705, -0.123730645, 0.15145405, -0.0942186, -0.14194241, 0.020219717, -0.07873223, -0.13622506, 0.02423812, 0.038878553, -0.1428635, -0.008074558, 0.2080821, 0.09682035, -0.117740385, -0.002011929, 0.20160528, -0.011861206, 0.014873792, 0.07865043, 0.018721405, -0.1766168, 0.15204018, -0.18189679, 0.06212848, -0.050278958, 0.26429534, -0.036394075, -0.091129705]]                       |\n","+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","only showing top 5 rows\n","\n"]}]},{"cell_type":"code","source":["result.select(\"features\").show(5, truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1685716598349,"user_tz":-180,"elapsed":496,"user":{"displayName":"Halil SAGLAMLAR","userId":"07259164328506563794"}},"outputId":"c1275882-fe5e-4860-b286-977594fa592f","id":"67leyDO4ReNn"},"execution_count":21,"outputs":[{"output_type":"stream","name":"stdout","text":["+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","|features                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n","+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [-0.00896873, 0.011731416, 0.12154201, 0.1149235, -0.14689414, 0.0103584975, 0.053073216, -0.056412186, -0.05143186, -0.0118978135, -0.12175384, -0.035894137, 0.11812756, 0.094671555, 0.15838866, 0.15260744, -0.004094441, -0.13675772, -0.07472433, -0.035856977, -0.026730005, -0.21840473, 0.029632289, -0.011515695, -0.20407394, -0.07848257, 0.040990185, 0.23028605, 0.077140555, 0.066990435, 0.015219222, -0.10295644, 0.038072545, 0.10786369, 0.121525764, -0.09569349, -0.06309264, 0.2778952, 0.06462455, -0.10851931, -0.14370486, -0.1466352, 0.08354363, -0.078758985, -0.08377953, 0.12384644, -0.23281692, -0.25607574, 0.16399069, -0.07780675, -0.18302177, -0.18325584, -0.12128636, -0.0010129504, 0.0070792097, 0.20506753, 0.034964647, 0.058425985, 0.19572404, -0.103953235, -0.20159312, -0.099047214, -0.07337802, -0.03713124, -0.055443633, 0.11107734, 0.048563413, -0.038048305, -0.020617828, 0.17082842, 0.069010496, 0.08457101, -0.038229663, 0.073144384, -0.092326105, -0.10054428, -4.3286112E-4, -0.046703782, -0.080231875, 0.02524295, 0.01368699, -0.19783853, -0.03501917, 0.13324805, 0.09053264, -0.0958231, -0.0032442473, 0.19218525, -0.027179888, 0.030672349, 0.12848215, -0.014700146, -0.089054875, 0.13839856, -0.15778734, 0.07103226, -0.060303356, 0.20854644, -0.008389737, -0.1473986]}]|\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [0.05308125, -0.015051386, 0.14103058, 0.10856558, -0.2342213, 0.003566569, 0.04778356, -0.004497177, -0.0535818, 0.0026637823, -0.08539946, -0.053373538, 0.09551983, 0.14680131, 0.19040567, 0.12977865, -0.033773836, -0.1319684, -0.040731516, -0.008372502, -0.006487461, -0.22866476, 0.052934043, -0.08824037, -0.26920623, -0.06628319, -0.024219586, 0.24569845, 0.0031692924, 0.057381, 0.028687425, -0.16567227, 0.02853736, 0.10968278, 0.17362879, -0.113311104, -0.062397536, 0.3374604, 0.029003158, -0.06641296, -0.1424927, -0.20313045, 0.031945523, -0.08096304, -0.060245346, 0.20948681, -0.2579919, -0.307684, 0.06597106, -0.06554805, -0.20020092, -0.21681747, -0.13199064, 0.03438211, -0.017373739, 0.28633022, 0.020454949, 0.012209459, 0.16254629, -0.17099468, -0.21445248, -0.042360373, -0.04671074, -0.081191055, -0.16396351, 0.1635775, 0.07605838, -0.055918057, -0.056702156, 0.11127068, 0.04990405, 0.06818552, -0.13117148, 0.18798989, -0.07119257, -0.15537563, 0.042161517, -0.079932965, -0.12116789, 0.08042555, 0.049063064, -0.1816178, -0.02954805, 0.21658188, 0.09907657, -0.10253809, -0.012852975, 0.18569835, -0.03166329, 0.017464636, 0.091833346, 0.035398744, -0.20562796, 0.15094037, -0.20429432, 0.049046826, -0.04301696, 0.2677997, -0.025445838, -0.070918486]}]                      |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [0.0054126224, 0.024950134, 0.11474294, 0.10822452, -0.2555815, 0.006825694, 0.0849884, -0.013415727, -0.067355156, -0.051267654, -0.1150451, -0.10311411, 0.12731767, 0.10249366, 0.16624388, 0.15492792, -0.05668665, -0.15464978, -0.036532804, -0.051754355, -0.007137836, -0.2513172, 0.08712005, -0.045029067, -0.18400155, -0.10452683, 0.027711688, 0.269648, 0.115273856, 0.09415018, 0.007242371, -0.14897408, 0.041124213, 0.08991895, 0.17528482, -0.0702524, -0.095246, 0.26005113, 0.08287458, -0.12394875, -0.1654959, -0.17744197, 0.12454772, -0.10536424, -0.058388017, 0.14923076, -0.24283154, -0.31404442, 0.1759314, -0.109916285, -0.18941985, -0.22670338, -0.102816485, 0.07351742, -0.0046835076, 0.22211693, 0.07862632, 0.06029365, 0.18198037, -0.15607189, -0.21938755, -0.0648467, -0.06229591, -0.061715987, -0.051075175, 0.13917515, 0.042945575, -0.06531585, -0.04472695, 0.13748632, 0.06643436, 0.1394113, -0.11119953, 0.11860371, -0.11418141, -0.12697497, 0.0050508482, -0.084115386, -0.105697244, -0.025213558, 0.033772327, -0.17188449, -0.058286436, 0.17098169, 0.058491495, -0.14792554, -0.014757689, 0.21050486, -0.028120184, 0.071195774, 0.13185748, 0.008545133, -0.1287004, 0.189709, -0.15126283, 0.070789315, -0.03831092, 0.25867718, 0.03666226, -0.15486968]}]                           |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [0.008756783, 0.05039759, 0.16213615, 0.097373165, -0.23011582, 0.05685288, 0.045038182, -0.043422982, -0.10621742, 0.021861993, -0.103197925, -0.10453307, 0.1128905, 0.105824575, 0.19338556, 0.1283347, -0.009834639, -0.1456381, -0.09109473, -0.015304626, -0.02358634, -0.28288808, 0.075918205, -0.11495868, -0.21861364, -0.042715643, -0.0021146596, 0.28538048, 0.1250176, 0.06596709, 0.046708006, -0.14905003, 0.018370079, 0.12479546, 0.17377286, -0.12793261, -0.08494885, 0.3624782, 0.06705852, -0.07460713, -0.15057686, -0.19423227, 0.032626033, -0.10774493, -0.08319685, 0.2038898, -0.24785702, -0.32097295, 0.1538476, -0.06367145, -0.20836613, -0.23597735, -0.12798108, 0.07948215, -0.027333453, 0.26723576, 0.016597308, 0.08272391, 0.16125162, -0.14834878, -0.2491029, -0.051455367, -0.043988984, -0.06324682, -0.119092226, 0.14168923, 0.13753091, -0.043976754, -0.06759758, 0.14472446, 0.07076088, 0.05267864, -0.07459972, 0.08896091, -0.06585022, -0.13084342, 0.0100373, -0.047292944, -0.14638452, -0.029784478, 0.033027586, -0.20373267, 0.024487026, 0.19487573, 0.09436645, -0.14655155, 0.0019440161, 0.18331449, -0.044212725, -0.0066170166, 0.0941082, 0.0058092014, -0.18064514, 0.16326123, -0.18090741, 0.08155668, -0.060303602, 0.30489156, -0.03252674, -0.13991456]}]                       |\n","|[{feature_vector, 0, 0, , {sentence -> 0}, [0.028661652, 0.0113291005, 0.15641849, 0.10562915, -0.18895394, 0.010692466, 0.028517308, -0.046971526, -0.06753073, 0.0015370062, -0.0794119, -0.05521748, 0.109319516, 0.12964095, 0.1794887, 0.1544829, -0.011513941, -0.1308672, -0.06253322, -2.441166E-4, -0.015246368, -0.28145686, 0.052070346, -0.08519331, -0.2572522, -0.061911665, -7.287748E-4, 0.26109117, 0.05577541, 0.0594182, 0.026305387, -0.13174595, 0.03511284, 0.14048967, 0.16914414, -0.12223352, -0.02627304, 0.32095695, 0.046736695, -0.076408096, -0.1273522, -0.17759354, 0.038809806, -0.10235544, -0.06325414, 0.18265921, -0.25014886, -0.2927684, 0.12070944, -0.059869632, -0.21926697, -0.21340933, -0.11169393, 0.059311472, -0.025239093, 0.27021858, 0.028728224, 0.03955428, 0.18724075, -0.14486238, -0.25880367, -0.0442654, -0.041866943, -0.056578852, -0.1482131, 0.1593034, 0.10229411, -0.01818499, -0.06094723, 0.10570261, 0.056156162, 0.070567705, -0.123730645, 0.15145405, -0.0942186, -0.14194241, 0.020219717, -0.07873223, -0.13622506, 0.02423812, 0.038878553, -0.1428635, -0.008074558, 0.2080821, 0.09682035, -0.117740385, -0.002011929, 0.20160528, -0.011861206, 0.014873792, 0.07865043, 0.018721405, -0.1766168, 0.15204018, -0.18189679, 0.06212848, -0.050278958, 0.26429534, -0.036394075, -0.091129705]}]                       |\n","+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n","only showing top 5 rows\n","\n"]}]},{"cell_type":"markdown","source":["Now `features` column is generated from embeddings and it can be used in `GenericClassifier, GenericSVMClassifier, GenericLogRegClassifier` annotators as an input column."],"metadata":{"id":"idA3loWkReNn"}}],"metadata":{"colab":{"machine_shape":"hm","provenance":[{"file_id":"1T7asEUpoJuH04ikR6-vCkOVLeEGF418K","timestamp":1685695933589},{"file_id":"1uRpqjOSXr26Fl9fZCVUJpGxLInZ0Icbi","timestamp":1684556881706},{"file_id":"1L8GdQ-yorjVk_V8Um6Rw6aEzFVT31OCO","timestamp":1683516961200},{"file_id":"1VVV4jTagH47UZiKFqXoP-Abq5ozZM1BV","timestamp":1672918708428},{"file_id":"https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/2.Text_Preprocessing_with_SparkNLP_Annotators_Transformers.ipynb","timestamp":1671914287039}],"toc_visible":true},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"}},"nbformat":4,"nbformat_minor":0}