paw_demo/Step_3_Attaching_Additional_Metadata.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Add this directory to the path and load our functions\n",
    "import sys\n",
    "sys.path.append(\"../src/\")\n",
    "\n",
    "import paware\n",
    "\n",
    "from data_processing import sentiment_data\n",
    "\n",
    "import polars as pl\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding Metadata\n",
    "\n",
    "## Agree and Disagree Distances\n",
    "\n",
    "We can use our embedding to compute an \"agree distance\" and a \"disagree distance\" for every text chunk.\n",
    "\n",
    "We do this by embedding the following statements:\n",
    "\n",
    "```python\n",
    "agree_statements = [\"This is correct\", \"This is true\", \"I agree\", \"This is helpful\"]\n",
    "\n",
    "disagree_statements = [\"This is incorrect\", \"This is false\", \"I disagree\", \"This is not helpful\"]\n",
    "```\n",
    "\n",
    "Then, we compute the average distance between the vector representation of each text chunk and each of these statements.\n",
    "\n",
    "Then, we look at the the replies to any given post, and aggregate information about these distances for those replies.\n",
    "\n",
    "This is all automated by our embedding class. Here, we continue with the same embedding we generated in [Step 1](\"../paw_demo/Step_1_Loading_and_Embedding_Data.ipynb\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
      "sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n"
     ]
    }
   ],
   "source": [
    "## Define the embedding tool\n",
    "embedding_tool = paware.PawEmbedding(\n",
    "    CONFIG_NAME = \"demo\",\n",
    "    RAW_DATA_DIR = \"../paw_demo/raw_subs/\",\n",
    "    EMBEDDED_SAVE_DIR =  \"../paw_demo/embedded_subs/\",\n",
    "    BATCH_SIZE = 100000,\n",
    "    CHUNK_WITH_METADATA = True,\n",
    "    CHUNK_SIZE = 512,\n",
    "    CHUNK_OVERLAP_PCT = 0.2\n",
    ")\n",
    "\n",
    "## Embed the data\n",
    "embedding_tool.add_agree_disagree_distances()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment of Replies\n",
    "\n",
    "Having generated the sentiment data, we can now attached it to our embedded data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentiment_data.copy_sentiment_data(vec_dir = \"../paw_demo/embedded_subs/config_demo/\",\n",
    "                                   sen_dir = \"../paw_demo/raw_sentiment/\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verifying the Results\n",
    "\n",
    "We can load the data now, and verify that it includes the additional engineered metadata data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 32)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>aware_post_type</th><th>aware_created_ts</th><th>reddit_id</th><th>reddit_name</th><th>reddit_created_utc</th><th>reddit_author</th><th>reddit_text</th><th>reddit_permalink</th><th>reddit_title</th><th>reddit_url</th><th>reddit_subreddit</th><th>reddit_link_id</th><th>reddit_parent_id</th><th>reddit_submission</th><th>reply_ids</th><th>is_short_question</th><th>text_chunk</th><th>chunk_agree_distance</th><th>chunk_disagree_distance</th><th>chunk_agree_distance_avg</th><th>chunk_disagree_distance_avg</th><th>agree_distance_avg</th><th>disagree_distance_avg</th><th>reply_agree_distances</th><th>reply_disagree_distances</th><th>top_reply_agree_distance</th><th>top_reply_disagree_distance</th><th>avg_reply_agree_distance</th><th>avg_reply_disagree_distance</th><th>summed_sentiments</th><th>absolute_summed_sentiment</th><th>vector</th></tr><tr><td>str</td><td>str</td><td>str</td><td>str</td><td>i64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>list[str]</td><td>bool</td><td>str</td><td>list[f64]</td><td>list[f64]</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>list[f64]</td><td>list[f64]</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>i64</td><td>i32</td><td>list[f64]</td></tr></thead><tbody><tr><td>&quot;submission&quot;</td><td>&quot;2024-01-05T10:…</td><td>&quot;18z96ud&quot;</td><td>&quot;t3_18z96ud&quot;</td><td>1704467742</td><td>&quot;Jmill2000&quot;</td><td>&quot;So I know GM m…</td><td>&quot;/r/GeneralMoto…</td><td>&quot;References&quot;</td><td>&quot;https://www.re…</td><td>&quot;GeneralMotors&quot;</td><td>null</td><td>null</td><td>null</td><td>[&quot;t1_kgfznkx&quot;, &quot;t1_kgg05vt&quot;]</td><td>false</td><td>&quot;General Motors…</td><td>[0.283116, 0.300825, … 0.254619]</td><td>[0.27071, 0.273402, … 0.261558]</td><td>0.282383</td><td>0.275515</td><td>0.276075</td><td>0.273053</td><td>[0.249591, 0.238443]</td><td>[0.23238, 0.247751]</td><td>0.249591</td><td>0.23238</td><td>0.244017</td><td>0.240066</td><td>-2</td><td>2</td><td>[0.018657, 0.037279, … 0.037011]</td></tr><tr><td>&quot;submission&quot;</td><td>&quot;2024-01-05T10:…</td><td>&quot;18z96ud&quot;</td><td>&quot;t3_18z96ud&quot;</td><td>1704467742</td><td>&quot;Jmill2000&quot;</td><td>&quot;So I know GM m…</td><td>&quot;/r/GeneralMoto…</td><td>&quot;References&quot;</td><td>&quot;https://www.re…</td><td>&quot;GeneralMotors&quot;</td><td>null</td><td>null</td><td>null</td><td>[&quot;t1_kgfznkx&quot;, &quot;t1_kgg05vt&quot;]</td><td>false</td><td>&quot;General Motors…</td><td>[0.266819, 0.28803, … 0.250238]</td><td>[0.260055, 0.269114, … 0.254161]</td><td>0.269768</td><td>0.270592</td><td>0.276075</td><td>0.273053</td><td>[0.249591, 0.238443]</td><td>[0.23238, 0.247751]</td><td>0.249591</td><td>0.23238</td><td>0.244017</td><td>0.240066</td><td>-2</td><td>2</td><td>[0.019537, 0.014495, … 0.008045]</td></tr><tr><td>&quot;comment&quot;</td><td>&quot;2024-01-05T10:…</td><td>&quot;kgfznkx&quot;</td><td>&quot;t1_kgfznkx&quot;</td><td>1704468972</td><td>&quot;warwolf0&quot;</td><td>&quot;Yea, that’s a …</td><td>&quot;/r/GeneralMoto…</td><td>null</td><td>null</td><td>&quot;GeneralMotors&quot;</td><td>&quot;t3_18z96ud&quot;</td><td>&quot;t3_18z96ud&quot;</td><td>&quot;18z96ud&quot;</td><td>null</td><td>false</td><td>&quot;General Motors…</td><td>[0.248872, 0.249956, … 0.255405]</td><td>[0.229161, 0.224686, … 0.235727]</td><td>0.249591</td><td>0.23238</td><td>0.249591</td><td>0.23238</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>0</td><td>0</td><td>[0.0408, 0.030147, … -0.004045]</td></tr><tr><td>&quot;comment&quot;</td><td>&quot;2024-01-05T10:…</td><td>&quot;kgg05vt&quot;</td><td>&quot;t1_kgg05vt&quot;</td><td>1704469154</td><td>&quot;tossedawaytheq…</td><td>&quot;Search Socrate…</td><td>&quot;/r/GeneralMoto…</td><td>null</td><td>null</td><td>&quot;GeneralMotors&quot;</td><td>&quot;t3_18z96ud&quot;</td><td>&quot;t3_18z96ud&quot;</td><td>&quot;18z96ud&quot;</td><td>[&quot;t1_kgg0rjf&quot;, &quot;t1_kgg79j0&quot;]</td><td>false</td><td>&quot;General Motors…</td><td>[0.230886, 0.252112, … 0.221663]</td><td>[0.234782, 0.238319, … 0.247299]</td><td>0.238443</td><td>0.247751</td><td>0.238443</td><td>0.247751</td><td>[0.257003, 0.262228]</td><td>[0.239812, 0.258668]</td><td>0.257003</td><td>0.239812</td><td>0.259616</td><td>0.24924</td><td>-2</td><td>2</td><td>[0.02871, 0.029767, … 0.022664]</td></tr><tr><td>&quot;comment&quot;</td><td>&quot;2024-01-05T10:…</td><td>&quot;kgg0rjf&quot;</td><td>&quot;t1_kgg0rjf&quot;</td><td>1704469370</td><td>&quot;Ripinpasta69&quot;</td><td>&quot;Only works if …</td><td>&quot;/r/GeneralMoto…</td><td>null</td><td>null</td><td>&quot;GeneralMotors&quot;</td><td>&quot;t3_18z96ud&quot;</td><td>&quot;t1_kgg05vt&quot;</td><td>&quot;18z96ud&quot;</td><td>null</td><td>false</td><td>&quot;General Motors…</td><td>[0.251144, 0.250768, … 0.261878]</td><td>[0.216444, 0.228874, … 0.249519]</td><td>0.257003</td><td>0.239812</td><td>0.257003</td><td>0.239812</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>0</td><td>0</td><td>[0.012741, 0.031151, … 0.01205]</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 32)\n",
       "┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐\n",
       "│ aware_pos ┆ aware_cre ┆ reddit_id ┆ reddit_na ┆ … ┆ avg_reply ┆ summed_se ┆ absolute_ ┆ vector   │\n",
       "│ t_type    ┆ ated_ts   ┆ ---       ┆ me        ┆   ┆ _disagree ┆ ntiments  ┆ summed_se ┆ ---      │\n",
       "│ ---       ┆ ---       ┆ str       ┆ ---       ┆   ┆ _distance ┆ ---       ┆ ntiment   ┆ list[f64 │\n",
       "│ str       ┆ str       ┆           ┆ str       ┆   ┆ ---       ┆ i64       ┆ ---       ┆ ]        │\n",
       "│           ┆           ┆           ┆           ┆   ┆ f64       ┆           ┆ i32       ┆          │\n",
       "╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡\n",
       "│ submissio ┆ 2024-01-0 ┆ 18z96ud   ┆ t3_18z96u ┆ … ┆ 0.240066  ┆ -2        ┆ 2         ┆ [0.01865 │\n",
       "│ n         ┆ 5T10:15:4 ┆           ┆ d         ┆   ┆           ┆           ┆           ┆ 7, 0.037 │\n",
       "│           ┆ 2         ┆           ┆           ┆   ┆           ┆           ┆           ┆ 279, …   │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ 0.037011 │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ ]        │\n",
       "│ submissio ┆ 2024-01-0 ┆ 18z96ud   ┆ t3_18z96u ┆ … ┆ 0.240066  ┆ -2        ┆ 2         ┆ [0.01953 │\n",
       "│ n         ┆ 5T10:15:4 ┆           ┆ d         ┆   ┆           ┆           ┆           ┆ 7, 0.014 │\n",
       "│           ┆ 2         ┆           ┆           ┆   ┆           ┆           ┆           ┆ 495, …   │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ 0.008045 │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ ]        │\n",
       "│ comment   ┆ 2024-01-0 ┆ kgfznkx   ┆ t1_kgfznk ┆ … ┆ null      ┆ 0         ┆ 0         ┆ [0.0408, │\n",
       "│           ┆ 5T10:36:1 ┆           ┆ x         ┆   ┆           ┆           ┆           ┆ 0.030147 │\n",
       "│           ┆ 2         ┆           ┆           ┆   ┆           ┆           ┆           ┆ , … -0.0 │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ 04045]   │\n",
       "│ comment   ┆ 2024-01-0 ┆ kgg05vt   ┆ t1_kgg05v ┆ … ┆ 0.24924   ┆ -2        ┆ 2         ┆ [0.02871 │\n",
       "│           ┆ 5T10:39:1 ┆           ┆ t         ┆   ┆           ┆           ┆           ┆ , 0.0297 │\n",
       "│           ┆ 4         ┆           ┆           ┆   ┆           ┆           ┆           ┆ 67, …    │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ 0.022664 │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ ]        │\n",
       "│ comment   ┆ 2024-01-0 ┆ kgg0rjf   ┆ t1_kgg0rj ┆ … ┆ null      ┆ 0         ┆ 0         ┆ [0.01274 │\n",
       "│           ┆ 5T10:42:5 ┆           ┆ f         ┆   ┆           ┆           ┆           ┆ 1, 0.031 │\n",
       "│           ┆ 0         ┆           ┆           ┆   ┆           ┆           ┆           ┆ 151, …   │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ 0.01205] │\n",
       "└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_dir = \"../paw_demo/embedded_subs/config_demo/\"\n",
    "test = pl.read_parquet(\n",
    "    embedding_dir+\"vectorized_generalmotors_data_complete.parquet\")\n",
    "\n",
    "test.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "aware",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}