how_to_query_a_database_EX_edition.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "how to query a database EX edition.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Interpause/pseudo-text/blob/master/how_to_query_a_database_EX_edition.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lpSAuFfXe_lr",
        "colab_type": "code",
        "outputId": "959bcd49-fe1f-416b-d08b-17163f7ca2da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "%matplotlib inline\n",
        "%unload_ext google.colab.data_table\n",
        "%tensorflow_version 2.x\n",
        "from google.colab import drive #drive.flush_and_unmount()\n",
        "import os\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from collections import OrderedDict\n",
        "drive.mount('/content/drive')\n",
        "os.chdir('/content/drive/My Drive/CSIT Internship/code')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The google.colab.data_table extension is not loaded.\n",
            "TensorFlow 2.x selected.\n",
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g18L43S-jl2U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = pd.read_pickle('./raw/myNewsFinal.pkl')\n",
        "#df1.join(df2, how='outer').sort_index(1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FxvfJauGf6gv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def query(sent,vectorizer,dist,embeds,index,db,top_n=10,reverse=False):\n",
        "    aquery = vectorizer(sent)\n",
        "    reply = OrderedDict(sorted(zip(index,dist(aquery,embeds)),key = lambda x: x[1],reverse=reverse))\n",
        "    for ind,dist in list(reply.items())[:10]:\n",
        "        print('%s\\t%f\\t%s'%(str(ind)[:4],dist,db.df.loc[db.df.uuid==ind].title.values[0]))\n",
        "    print(df.loc[df.uuid==list(reply.keys())[0]].text.values[0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XEIStyv8coxg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title TS_SS stuff {display-mode: \"form\"}\n",
        "#@markdown https://github.com/taki0112/Vector_Similarity (i stole this)\n",
        "class TS_SS:\n",
        "    def Cosine(self, vec1: np.ndarray, vec2: np.ndarray):\n",
        "        return np.dot(vec1, vec2.T)/(np.linalg.norm(vec1) * np.linalg.norm(vec2))\n",
        "\n",
        "    def VectorSize(self, vec: np.ndarray):\n",
        "        return np.linalg.norm(vec)\n",
        "\n",
        "    def Euclidean(self, vec1: np.ndarray, vec2: np.ndarray):\n",
        "        return np.linalg.norm(vec1-vec2)\n",
        "\n",
        "    def Theta(self, vec1: np.ndarray, vec2: np.ndarray):\n",
        "        return np.arccos(self.Cosine(vec1, vec2)) + np.radians(10) #apparently the \"best\" value to solve overlapping problem (seems arbitrary tho)\n",
        "\n",
        "    def Triangle(self, vec1: np.ndarray, vec2: np.ndarray):\n",
        "        theta = np.radians(self.Theta(vec1, vec2))\n",
        "        return (self.VectorSize(vec1) * self.VectorSize(vec2) * np.sin(theta))/2\n",
        "\n",
        "    def Magnitude_Difference(self, vec1: np.ndarray, vec2: np.ndarray):\n",
        "        return abs(self.VectorSize(vec1) - self.VectorSize(vec2))\n",
        "\n",
        "    def Sector(self, vec1: np.ndarray, vec2: np.ndarray):\n",
        "        ED = self.Euclidean(vec1, vec2)\n",
        "        MD = self.Magnitude_Difference(vec1, vec2)\n",
        "        theta = self.Theta(vec1, vec2)\n",
        "        return np.pi * (ED + MD)**2 * theta/360\n",
        "\n",
        "    def __call__(self, vec1: np.ndarray, vec2: np.ndarray):\n",
        "        return self.Triangle(vec1, vec2) * self.Sector(vec1, vec2)\n",
        "tsss = TS_SS()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7AUlc_PZCD-o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sklearn.metrics\n",
        "def cosine(vec1,samples):\n",
        "    ans = sklearn.metrics.pairwise.cosine_similarity(np.array([vec1]),samples)\n",
        "    return 1 - ans[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r1ilrVQNoPrV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import functools\n",
        "def distwrap(f):\n",
        "    @functools.wraps(f)\n",
        "    def dist(query,docs):\n",
        "        return 1 - f(np.array([query]),docs)[0]\n",
        "    return dist\n",
        "\n",
        "@distwrap\n",
        "def dis(docs1,docs2):\n",
        "    return sklearn.metrics.pairwise.cosine_similarity(docs1,docs2)\n",
        "#just a little fun smth. nice to know it actually worked"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UnBjrnpBfJPH",
        "colab_type": "text"
      },
      "source": [
        "# Future approaches\n",
        "- BERTsumEXT sentences before embedding\n",
        "- Factor in date\n",
        "\n",
        "##TODO:\n",
        "- Try hashingVectorizer?\n",
        "- bertEXT instead of textRank\n",
        "- gensim sent vec, doc vec instead of sent-transformers\n",
        "- Try out flair's stacked embeddings\n",
        "- spacy's NER? flair's NER? well it will be needed for active scrape\n",
        "- NER based search approach\n",
        "- Try LDA. the idea of a probability distribution a document should be in the same genre... sounds smart"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CNJXB_hGo5lS",
        "colab_type": "text"
      },
      "source": [
        "#TFIDF Base"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_7OLMkox7wL-",
        "colab_type": "code",
        "outputId": "32dc1e7c-ba90-47fe-9da5-81218903dd40",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "import string\n",
        "import spacy\n",
        "import nltk\n",
        "from collections import OrderedDict\n",
        "nltk.download('stopwords')\n",
        "removables = nltk.corpus.stopwords.words('english')\n",
        "replacements = OrderedDict({'barack obama':'obama','donald trump':'trump','health care':'healthcare','hillary clinton':'hillary','president obama':'obama','barack':'obama','donald':'trump','gov ':'government ','said':'say'})\n",
        "class LemmaTokenizer(object):\n",
        "    def __init__(self):\n",
        "        self.nlp = spacy.load('en', disable=['parser', 'ner'])\n",
        "    def __call__(self, articles):\n",
        "        for rep,val in replacements.items(): articles = articles.replace(rep,val)\n",
        "        return [str(t.lemma_) for t in self.nlp(articles) if t.text not in removables]\n",
        "lemma = LemmaTokenizer()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2XGkW-tllscc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from collections import OrderedDict\n",
        "text_vectorizer = TfidfVectorizer(lowercase=True,\n",
        "                            #tokenizer=lemma, #lemmatizer somehow accents word sense disambiguation problems\n",
        "                            stop_words='english',\n",
        "                            ngram_range=(1,2),\n",
        "                            max_features=25000,\n",
        "                            norm=None)\n",
        "title_vectorizer = TfidfVectorizer(lowercase=True,\n",
        "                            #tokenizer=lemma, #lemmatizer somehow accents word sense disambiguation problems (well it makes sense that if you reduce words to root forms...)\n",
        "                            stop_words='english',\n",
        "                            ngram_range=(1,2),\n",
        "                            max_features=25000,\n",
        "                            norm=None)\n",
        "#TODO, increase ngram_range since word2vecs support phrases (underscores i think?)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N6CVMAN_TKiM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "database = pd.concat({'df':df.copy(),\n",
        "                 'tfidf_text':pd.DataFrame.sparse.from_spmatrix(text_vectorizer.fit_transform(df.text),index=df.index),\n",
        "                 'tfidf_title':pd.DataFrame.sparse.from_spmatrix(title_vectorizer.fit_transform(df.title),index=df.index)\n",
        "                 },\n",
        "                axis=1)#,copy=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9xe5GJSpgXqV",
        "colab_type": "code",
        "outputId": "8888f357-6955-44ef-c6e6-20da95c5c0bd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 904
        }
      },
      "source": [
        "test_query = 'vaccines cause autism'\n",
        "print('Cosine text')\n",
        "query(test_query,\n",
        "      lambda s: text_vectorizer.transform([s]).toarray()[0],\n",
        "      cosine,\n",
        "      database.tfidf_text.values,\n",
        "      database.df.uuid,\n",
        "      database)\n",
        "print('\\nTS-SS text')\n",
        "query(test_query,\n",
        "      lambda s: text_vectorizer.transform([s]).toarray()[0],\n",
        "      tsss,\n",
        "      database.tfidf_text.values,\n",
        "      database.df.uuid,\n",
        "      database)\n",
        "print('\\nCosine title')\n",
        "query(test_query,\n",
        "      lambda s: title_vectorizer.transform([s]).toarray()[0],\n",
        "      cosine,\n",
        "      database.tfidf_title.values,\n",
        "      database.df.uuid,\n",
        "      database)\n",
        "print('\\nTS-SS title')\n",
        "query(test_query,\n",
        "      lambda s: title_vectorizer.transform([s]).toarray()[0],\n",
        "      tsss,\n",
        "      database.tfidf_title.values,\n",
        "      database.df.uuid,\n",
        "      database)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cosine text\n",
            "245b\t0.607406\ti'm thankful for autism, eight reasons this thanksgiving i am giving thanks\n",
            "ba97\t0.652245\tautism and adhd share genes\n",
            "aeb0\t0.660046\tvaccines fast facts\n",
            "946e\t0.729204\ta parent-to-parent campaign to get vaccine rates up\n",
            "ecfd\t0.729218\ther son died. and then anti-vaxers attacked her\n",
            "56cf\t0.741302\tautism news\n",
            "7a57\t0.774695\tautism news\n",
            "14a6\t0.809315\tsamoa measles deaths rise above 50, mostly small children\n",
            "6aa5\t0.809315\tsamoa measles deaths rise above 50, mostly small children\n",
            "312a\t0.814439\tsamoa measles deaths rise above 50, mostly small children\n",
            "i never really thought i would utter the words \"i'm thankful for autism\" though i have and while i certainly have moments of self-doubt, feelings of being overwhelmed, wants of wishing life was easier for my daughter, there are so many things that i am thankful for, because of autism. first, autism has allowed me to gain true friends. choosing to be a friend to a person or with a family who lives with a loved one on the spectrum can be overwhelming. there have been countless friends that have come and gone but autism has allowed me to see who truly values myself, my family and my daughter for who we are. they listen when things get tough, the pick us up when we are down and they never ask for an apology when we have to leave someplace early because of well, autism. these friends, some old and some new choose to say \"yes\" to us even when saying \"no\" would be easier. for them, i am thankful. second, autism has given myself and my entire family a better understanding of what is most important in life. we no longer pine for the most expensive cars, best vacations or the most gifts under the tree, instead we are grateful for small successes (which extend outside our daughter with special needs). we choose to realize that small moments truly are what life is all about and find ourselves recounting the baby steps we all take even if its something small like an \"a\" on a math test or learning to use the potty. for that, i am thankful. third, we realize how lucky we are. autism is hard. there is no doubt, just ask any one of my kids as they are skirting away from their sister when she's having an outburst. that being said, each of them will tell you the fact that they are lucky, at their young age they understand more than i could have ever taught them without autism in our lives. my children value others for who they are because of this journey. they each value unique abilities, the opportunity to find success in their own right. they also value health. hearing the stories of our trips to the doctors or hospitals or the many facebook pages i am on allow them to see how autism, while challenging and life altering is truly still easier than watching their sibling suffer in pain. for that, i am thankful. fourth, autism has given me a voice. i have always loved writing, sharing and learning however as a mother, staying at home i have felt lost. i wanted to work, but where? instead, my work i realize is in the home and learning to share our with hope this journey could make life better for others who come behind us. it may be a blog, a conversation, a friend reaching out for a family member, no matter what it is, i am here. i believe that this is my reason for writing, speaking and sharing. i am constantly willing to learn, willing to listen and willing to share what i have learned so that one day someone may have less of a struggle than we first did. for that, i am thankful. fifth, autism has gotten me outside of myself and given me purpose. i realize that my life isn't lived for myself alone nor for my own satisfaction but for the opportunity to serve others. i truly believe that we were chosen to love, nurture and raise my daughter with extra needs to better ourselves and help others not just in the special needs community but anyone we encounter in our lives that need a little extra support. for this, i am thankful. sixth, the path we are on is better than the one we were on. i said it. i won't deny there are moments i miss competitive gymnastics or the small catholic school we attended but reality is, this journey is better. certainly it is harder. it is more challenging and scary at times but it forces us slow down, we see the world around us, from our daughters eyes and it reminds us that life is best when lived not raced through with some tangible goal in mind instead asking us to focus on the intrinsic gifts we once missed. for that, i am thankful. seventh, autism has given us an understanding of others and allowed us to release our own judgement of others. i won't lie, we were a bit judgemental. from a young age, i knew i was a better parent, i expected my children would be gifted and that following the \"right\" path i would get what i wanted when i wanted it. sadly, until seraphina, i believe i was raising my children to follow that path which i now realize was wrong. autism has been a humbling experience that has lead me to realize that judgement of anyone has never gotten someone anywhere and so now, i am open to listening, learning, understanding and supporting. always. i may not always agree, but i accept others for who they are and my children do too. for that, i am thankful. eight, my family. our family was never complete. autism has made it that way. this year, my parents chose to move nearly eight hours to share in this journey with me. not only did they choose to adopt me and raise me with my baggage but to accept who i am with my family in tow changing their own life trajectory to live on this path with us. no words can ever describe the thanks i have nor the opportunity to spend their golden years with them. without autism, perhaps we would not have them just a hop, skip and a jump away and for that i am most thankful. autism isn't the path i expected or wanted. it isn't the journey i chose but it is the path we are on, this year, i can finally say, i am thankful for autism not because of the challenges but for the beauty it has brought to my life. happy thanksgiving to all my friends and family and a huge hug to those walking, living and supporting this journey we are on. every day i wake up i realize, i choose her. always. for that, i am thankful.\n",
            "\n",
            "TS-SS text\n",
            "ecfd\t352277461960.622192\ther son died. and then anti-vaxers attacked her\n",
            "aeb0\t352619931751.854858\tvaccines fast facts\n",
            "ba97\t352659747420.822327\tautism and adhd share genes\n",
            "245b\t352708454931.329407\ti'm thankful for autism, eight reasons this thanksgiving i am giving thanks\n",
            "946e\t352810863861.518860\ta parent-to-parent campaign to get vaccine rates up\n",
            "5563\t353029959683.090149\twhy sri lanka should go back on your bucket list\n",
            "ebcf\t353098235667.996521\tscientists are using mri scans to reveal the physical makeup of our thoughts and feelings\n",
            "453b\t353146108145.312012\tthis is what its like to experience the crushing silence of the intensive care waiting room day in, day out\n",
            "56cf\t353179322798.336426\tautism news\n",
            "66cc\t353183554285.791870\tu.s. officials deny access to doctors seeking to give flu shots to migrant children\n",
            "(cnn) not long ago, a 4-year-old boy died of the flu. his mother, under doctor's orders, watched his two little brothers like a hawk, terrified they might get sick and die, too. grieving and frightened, just days after her son's death she checked her facebook page hoping to read messages of comfort from family and friends. instead, she found dozens of hateful comments: you're a terrible mother. you killed your child. you deserved what happened to your son. this is all fake - your child doesn't exist. bewildered and rattled, she closed her facebook app. a few days later she received a text message from someone named ron. expect more like this, ron warned. expect more. the attacks were from those who oppose vaccination, and this mother, who lives in the midwest, doesn't want her name used for fear the attention would only encourage more messages. nothing too cruel interviews with mothers who've lost children and with those who spy on anti-vaccination groups, reveal a tactic employed by anti-vaxers: when a child dies, members of the group sometimes encourage each other to go on that parent's facebook page. the anti-vaxers then post messages telling the parents they're lying and their child never existed, or that the parent murdered them, or that vaccines killed the child, or some combination of all of those. nothing is considered too cruel. just days after their children died, mothers say anti-vaxers on social media called them whores, the c-word and baby killers. the mother in the midwest, who wants to remain anonymous, isn't alone. jill promoli, who lives outside toronto, lost her son to flu. she believes the anti-vaxers are trying to silence the very people who can make the strongest argument for vaccinations: those whose children died of vaccine-preventable illnesses. jude mcgee, who died of the flu at age two. his mother, jill promoli, has suffered abuse from anti-vaxers. flu took the life of promoli's 2-year-old son, jude mcgee, three years ago. she's since started a campaign in his name for flu prevention, including vaccination. \"i know that these people are really trying to hurt me, and i understand that the reason they're doing it is because they want me to stop,\" she said. anti-vaxers respond larry cook, the founder of stop mandatory vaccination, doesn't deny that such attacks on mothers of dead children exist. in an email to cnn, he wrote that members of his group make more than half a million comments on the group's facebook page each month. \"any discussions about parents who lose their children after those children are vaccinated would be minor in number, and even smaller would be the number of members reaching out to parents in private message to share their concerns that vaccines may have played a role in a death,\" cook wrote. \"i do not condone violent behavior or tone and encourage decorum during discussion,\" cook wrote, adding that anyone \"who deliberately engage[s] in the politics of advocating for compulsory vaccination where children may be further damaged through government vaccine mandates can expect push back and resistance, alongside knowledgable discussions about vaccine risk in social media commentary.\" cook said some of the more than 160,000 members of his group have been targets of \"harassment campaigns,\" and that \"police have actually showed up at my members' doorsteps.\" he added that he's been threatened and included a screenshot of a private facebook message that said, \"finally found where you live. finally i'll be able to put a bullet in you. you are dead.\" another anti-vaccination leader blamed the postings on grieving parents' pages on \"infiltrators\" who try to \"create incendiary situations.\" \"i tell everybody that you should look at the person you're talking to and those on the other side of this discussion and recognize that they care about children, too,\" said del bigtree, chief executive officer of the informed consent action network. during the public comment period at a meeting last month of the us center for disease control's advisory committee on immunization practices, more than a dozen people spoke about their concerns about vaccines. some said they or their loved ones had been injured by vaccines. \"my older son suffered documented illness and regression following vaccination,\" said jackie martin-sebell. \"these vaccines are not safe for everyone.\" another speaker, rilei cherry, said her son developed autism as a result of vaccines. \"we owe it to our children to truly look at the long term effects of vaccines and be honest about what could happen to our children,\" she told the cdc committee. despite the speakers' concerns, more than a dozen studies have shown that vaccines do not cause autism. the american academy of pediatrics says \"vaccines are safe. vaccines are effective. vaccines save lives.\" jill promoli with her twin boys, jude and thomas mcgee. jude died at age two of the flu. the mothers on may 6, 2016, promoli put her toddlers jude and his twin brother thomas, down for an afternoon nap in their home. jude had a low-grade fever, but he was laughing and singing when he went down for his nap. when his mother went to check on him two hours later, he was dead. promoli said the next few weeks were \"a living hell.\" \"having to go in and plan a funeral and find the ability somehow to even take steps to walk into a funeral home, to make plans and decide whether to bury or cremate your child -- it was just all so horrifying,\" she said. when an autopsy came back showing jude had died of the flu, promoli started her flu prevention campaign. just watched teen opts to get vaccinated despite mother's objection replay more videos ... must watch teen opts to get vaccinated despite mother's objection 04:58 that's when the online attacks began. some anti-vaxers told her she'd murdered jude and made up a story about the flu to cover up her crime. others said vaccines had killed her son. some called her the c-word. the worst ones -- the ones that would sometimes make her cry -- were the posts that said she was advocating for flu shots so that other children would die from the shots and their parents would be miserable like she was. \"the first time it made me feel really sick because i couldn't fathom how anybody could even come up with such a terrible claim,\" promoli said. \"it caught me off guard in its cruelty. what kind of a person does this?\" twisted logic that relies on scientific lies doesn't bother promoli so much anymore. she's continued with her flu shot campaign, persuading canadian prime minister justin trudeau to publicize his flu shot. got this last night. get your flu shot - for your protection and to help protect the more vulnerable people around you. #forjudeforeveryone pic.twitter.com/2efzjwudkv justin trudeau (@justintrudeau) november 22, 2018 \"i've had to grow some very thick skin,\" she said. she said no matter how many nasty messages she's received -- and she says she's received hundreds -- she'll continue her campaign. \"the work that we're doing might mean that somebody else doesn't have to go plan a funeral for their toddler, and that is everything,\" she said. other mothers have also persevered despite attacks from anti-vaxers. serese marotta lost her 5-year-old son, joseph, to the flu in 2009, and is now chief operating officer of families fighting flu, a group that encourages flu awareness and prevention, including vaccination. in 2017, she posted a video on the eighth anniversary of her son's death to reinforce the importance of getting the flu vaccine. \"slut,\" one person commented. \"pharma whore.\" \"may you rot in hell for all the damages you do!\" a facebook user wrote on another one of her posts. she says a facebook user in australia sent her a death threat. \"she called me a lot of names i won't repeat and used the go-to conspiracy theories about government and big pharma, and i responded, 'i lost a child,' and questioned where she was coming from, and she continued to attack me,\" said marotta, who lives in syracuse, new york. catherine and greg hughes, an australian couple who lost their 1-month old son, riley, to whooping cough, have also received online abuse. too young to be vaccinated, riley relied on herd immunity -- the vaccinations of others -- to protect him. as his mother held his hand, riley hughes was baptized hours before he passed away. but herd immunity didn't protect him, since the area where the hughes family lived in perth has some of the lowest vaccination rates in australia. \"riley's death was a very inconvenient truth for anti-vaccine activists,\" catherine said. \"the nasty messages started 24 hours after he died. they called us baby killers and said we would have the blood of other babies on our hands. we've been told to kill ourselves.\" the couple started a vaccination campaign , light for riley. catherine said they still receive vile comments years after riley's death. \"[f**k] you, hughes family,\" one facebook user wrote on the light for riley page. \"what a [f**king] evil whore you really are,\" another user wrote to them in a private facebook message. another facebook user was more succinct. \"please die,\" the user wrote in a private message. \"a lot of them come from the position that they have children that were vaccine-injured,\" catherine said. \"but a fair chunk of them are just haters.\" the professor grieving mothers aren't the only targets of anti-vaxer abuse. dorit reiss, a professor at uc hastings school of law, has received countless vile messages, and as with the mothers, many of the messages are gender-oriented. over the years, she's become pretty blase about it. \"'whore' is pretty normal,\" said reiss, a pro-vaccine advocate who has written extensively about vaccines. \"i've also been called a [c**t].\" sometimes reiss, who is jewish, receives comments that mention the holocaust. one facebook user made a meme with a photo of her father with \"proud supporter of the vaccine holocaust.\" reiss says her father has nothing to do with vaccines. another meme shows a photo of reiss holding her infant son and it says that reiss is \"force-injecting\" her baby with vaccines. below the photo is written: \"because one holocaust wasn't enough.\" other facebook users have said her children look sick in photos, and that vaccines are clearly to blame. \"the boy [sic] appears lethargic,\" one user wrote, with \"dark circles under his eyes. these are common precipitants of vaccine induced immune damage and suppression, as well as vaccine induced transient ischemic adverse event.\" of all the various messages from anti-vaxers, reiss said the one that angered her the most was a voicemail left for her husband on his work phone. \"if i hear or see anything written by your wife after today, i will release your phone number, your work phone number, your work address, her work address, her work phone number,\" the caller said, correctly reciting their phone numbers and home address. the doctors three pediatricians who are vocal vaccine advocates have also been frequent targets of anti-vaxers. all three now have security escorts when they speak publicly. dr. paul offit keeps a fat folder of nasty messages he's received so that \"if someone kills me, my wife can give it to the police.\" he does not laugh when he says this. \"rot in hell you baby killer,\" one user wrote in an email to offit, who is director of the vaccine education center at the children's hospital of philadelphia and professor of pediatrics at the perelman school of medicine at the university of pennsylvania. \"go [f**king] kill yourself,\" another wrote. dr. peter hotez at the baylor college of medicine in houston is a vaccine researcher who wrote a book about his daughter called \" vaccines did not cause rachel's autism .\" \"you have no morals whatsoever and you know that you are a [f**king] liar. i hope you rot in hell,\" one anti-vaxer emailed hotez, a professor of pediatrics and dean of the national school of tropical medicine at baylor. users on the social media platform mewe discussed an upcoming public meeting hotez was expected to attend. \"maybe if we cause him enough stress he'll have a heart attack before [wednesday],\" one woman wrote, adding, \"#sorrynotsorry.\" dr. richard pan, a pediatrician and a california state senator, spearheaded a successful attempt to get rid of vaccine exemptions for personal or religious reasons for schoolchildren in his state. he's frequently been the target of race-driven abuse by anti-vaxers on facebook. he says he's received thousands of hateful messages from anti-vaxers. \"chinese garbage,\" one user wrote on pan's facebook page, followed by vomiting and devil emojis. \"most ignorant [a**hole].\" \"i hope they stone you to death,\" another facebook user wrote. \"i'll make a special trip to happily watch your head crack. the parents of the children you are destroying should each get a chance. like a nazi pinata.\" some of the professionals and the mothers interviewed for this story said they reported the abusive messages they received to facebook. most of those who made reports said they received an automated response and in the end nothing changed. others said after they made a report the sender was suspended from facebook for a short period of time, or their offensive messages were removed. others said they didn't report to facebook because the process was onerous or they'd heard that it wouldn't change anything. a facebook spokesperson responded to these concerns: \"we try to empower our users with controls, such as blocking other users and moderating comments, so they can limit their exposure to unwanted, offensive or hurtful content. we also encourage people to report bullying behavior on our platform, so we can review the content and take proper action,\" the spokesperson wrote in an email. \"we want members of our community to feel safe and respected on facebook and will remove material that appears to purposefully target private individuals with the intention of degrading or shaming them.\" erin costello creates fake facebook accounts so she can join anti-vaxer groups to see if members are planning to attack mothers of dead children. the spy erin costello, a former bartender and current stay-at-home mom in utica, new york, is the \"ron\" who texted the grieving mother in the midwest, warning her to expect more anti-vaxer attacks. costello is an administrator for the pro-vaccine facebook page \" what's the harm? costello is one of several vaccine advocates who've set up so-called \"sock puppet\" or fake facebook accounts and then joins the closed anti-vaxer groups to spy on them. she said she often sees members discussing posts by parents of sick or dead children, sometimes suggesting that members \"educate\" those parents by posting on their page. for example, a member of the anti-vaccine group stop mandatory vaccination said a mother had posted that her baby had suffered seizures after receiving vaccines. the anti-vaxer urged others in the group to \"comment for her to read! i want to win this mom over and she really trusts her pediatrician but at the same time she is scared!\" another member of stop mandatory vaccination re-posted a post by catherine hughes, the mother who'd lost her child to a vaccine-preventable disease, calling for others to vaccinate their children. \"anyone want to chime in on this post?\" the anti-vaxer wrote. another member responded, \"i feel sorry for the lost baby and her other children but someone needs to inject her with vaccines until she dies.\" a mother's greatest fear when she sees anti-vaxers talking about parents in their closed groups, costello, the online pro-vaccine spy, gets in touch with those parents to warn them they may be getting nasty messages from the anti-vaxers. when costello reached out to the mother in the midwest, she explained why she was contacting her. \"i know you're likely getting many horrible messages on facebook right now,\" costello wrote to the mother. \"children such as [yours] are the reason why i do my part to fight for overwhelming acceptance of vaccines as well as fight against the lies and misinformation that are recklessly spread around against vaccines.\" get cnn health's weekly newsletter sign up here to get the results are in with dr. sanjay gupta every tuesday from the cnn health team. the mother wrote back. \"i appreciate the strong role you take in helping protect families like mine,\" she said. after hundreds of facebook comments from anti-vaxers, the mother turned off comments on her page, and deleted many of the ones she received. some are still in her head, though. she weeps as she remembers the one that was hardest to read. \"the ones that said this was a fake story. that he wasn't real. that my child didn't exist,\" she said. \"because when your child dies, that's the biggest fear -- that he will be forgotten.\"\n",
            "\n",
            "Cosine title\n",
            "1f58\t0.445545\tvaccines news\n",
            "7a57\t0.640472\tautism news\n",
            "56cf\t0.640472\tautism news\n",
            "aeb0\t0.657763\tvaccines fast facts\n",
            "ba97\t0.715462\tautism and adhd share genes\n",
            "1fb4\t0.726611\tnew york ends religious exemptions for required vaccines\n",
            "c187\t0.748254\tphiladelphia eagles open sensory room for fans with autism\n",
            "245b\t0.777244\ti'm thankful for autism, eight reasons this thanksgiving i am giving thanks\n",
            "a2dd\t0.781161\tnurse takes in man with autism so he can get a new heart\n",
            "9d19\t0.788909\tn.y.c. weather: snow and sleet cause havoc for travelers\n",
            "read more a new study has found severe pneumonia decreases by 35 per cent in children who receive a vaccine against a pneumonia-causing ... read more a newly developed experimental vaccine was more than eighty percent effective in protecting mice from succumbing to staphylococcus aureus infection. s. aureus causes more than 30,000 deaths from ... read more within seconds after an infected mosquito bites, the malaria parasite navigates the host skin and blood vessels to invade the liver, where it will stay embedded until thousands of infected cells ... read more scientists have made a breakthrough in the hunt for a new vaccine for killer hospital bug clostridioides difficile (c. ... read more a new study indicates a way for cancer immunotherapy to spur a more robust immune response. such knowledge could lead to the development of better cancer vaccines and more effective immunotherapy ... read more a school-based human papillomavirus (hpv) immunization program in british columbia, canada, is dramatically reducing rates of cervical pre-cancer in b.c. women, according to a new ... read more a single dose of a highly diluted vsv-ebola virus (ebov) vaccine -- approximately one-millionth of what is in the vaccine being used to help control the ongoing ebola outbreak in the democratic ... read more there is no effective vaccine currently available to prevent lyme disease in humans. experts from academia, government, and industry recently convened to tackle this public health challenge. now, a ... read more predicting and controlling disease outbreaks would be easier and more reliable with the wider application of mathematical modelling, according to a new ... read more scientists show, on the molecular level, how an experimental vaccine offers long-term protection against the ... read more a vaccine for the common and sometimes deadly rsv (respiratory syncytial virus) has been elusive, but scientists say a new discovery puts them much closer to ... read more scientists have found a common, new source of tumor mutations that could offer three levels of therapy with a cancer vaccine: 1) a broadly protective, or pan-cancer vaccine 2) cancer-type specific ... read more mit chemical engineers have developed a new type of lipid nanoparticle that can be used to deliver rna vaccines. the particles also trigger the release of powerful immune signaling proteins that ... read more a new type of vaccine that can be stored at warmer temperatures, removing the need for refrigeration, has been developed for mosquito-borne virus chikungunya in a major advance in vaccine technology. ... read more the biological clock influences immune response efficacy. indeed, cd8 t cells, which are essential to fight infections and cancers, function very differently according to the time of ... read more a novel vaccine protected almost all mice and guinea pigs exposed to a new herpes virus. this may lead to the vaccine being tested in human ... read more researchers have discovered that the dengue virus changes its shape through mutations in envelope protein to evade vaccines and therapeutics. the study also gives insights on the types of treatment ... read more a new needle-free flu vaccine patch revved up the immune system much like a traditional flu shot without any negative side effects. the research represents an important step toward a technology that ... read more despite its proven success at preventing cancer, many adolescents are still not getting the hpv vaccine. a new study from shows that physicians' delivery and communication practices must improve ...\n",
            "\n",
            "TS-SS title\n",
            "1fb4\t364239628.547191\tnew york ends religious exemptions for required vaccines\n",
            "1f58\t364239628.547191\tvaccines news\n",
            "aeb0\t364239628.547191\tvaccines fast facts\n",
            "1ed7\t364239628.547191\tafter child deaths, doctors pressure border patrol to let them administer flu vaccines\n",
            "c187\t364327666.640174\tphiladelphia eagles open sensory room for fans with autism\n",
            "7a57\t364327666.640174\tautism news\n",
            "ba97\t364327666.640174\tautism and adhd share genes\n",
            "a2dd\t364327666.640174\tnurse takes in man with autism so he can get a new heart\n",
            "9196\t364327666.640174\tmall santa claus calms boy with autism who'd been too nervous to meet him for 6 years\n",
            "afdc\t364327666.640174\tafter a teen with autism missed a ride on the \"polar express,\" the conductor visited him for a magical experience\n",
            "new york ends religious exemptions for required vaccines enlarge this image toggle caption paul vernon/ap paul vernon/ap updated at 4:33 p.m. et friday new york gov. andrew cuomo signed a bill thursday ending vaccination exemptions based on religious beliefs, the latest attempt to address the growing measles outbreak, the worst the u.s. has experienced in decades. cuomo said plugging the loophole should help contain the spike in measles cases in new york, the state hardest hit by the uptick in the contagious virus due to low vaccination rates in ultra-orthodox communities. \"the science is crystal clear: vaccines are safe, effective and the best way to keep our children safe,\" cuomo said after signing the bill. \"while i understand and respect freedom of religion, our first job is to protect the public health and by signing this measure into law, we will help prevent further transmissions and stop this outbreak right in its tracks.\" the democratic-controlled legislature approved the measure, which also eliminates other nonmedical exemptions for schoolchildren across the state. \"we are facing an unprecedented public health crisis,\" said sen. brad hoylman, the legislation's sponsor. \"the atrocious peddlers of junk science and fraudulent medicine who we know as anti-vaxxers have spent years sowing unwarranted doubt and fear, but it is time for legislators to confront them head-on.\" the exemption, which exists in some form in most states, allows parents of schoolchildren to cite their religious beliefs in opting their kids out of required vaccines. supporters of keeping the religious exemptions say religious freedom should not be overpowered by state laws. after the final vote tally was announced in the assembly, howling protesters, including the parents of unvaccinated children, filled the chamber, hurling expletives and chanting \"shame on you\" until lawmakers moved to recess. some backers of the bill have seen the measles outbreak up close, both in their districts and in their own homes, such as assembly member kenneth zebrowski, who represents rockland county. there, he noted, there have already been more than 266 confirmed measles cases, including 16 hospitalizations. his 1-year-old daughter had to get her first vaccine shot at 6 months of age before her regularly scheduled immunization when she turned 1 recently. \"we had to get our kids overvaccinated,\" he said of his and other families in the district. \"because of this epidemic. i'm not particularly thrilled.\" to those who question whether the recent measles outbreak in new york is indeed an epidemic, zebrowski said that should not be the focus. the job of lawmakers, he said, is not to react to epidemics. \"our job as legislators is to prevent epidemics,\" he said. the centers for disease control and prevention reported last week that the number of new measles cases this year has exceeded 1,000, the highest count in 27 years. most of those new measles cases have been concentrated in ultra-orthodox areas of new york, including rockland county and parts of brooklyn, adding urgency to the statewide debate around religious exemptions to vaccines. the new york assembly narrowly passed the bill by a 77-53 vote. it needed 76 votes for passage. lawmakers in the state senate advanced the measure by a tally of 36-26. a small number of other states including california, mississippi and arizona have already passed laws banning vaccine exemptions on religious grounds. (arizona law currently allows for a parent to opt a child out of mandatory vaccinations because of \"personal beliefs,\" but not on religious grounds. a proposed bill in the arizona legislature had sought to expand the vaccination exemptions to include religious beliefs, but it has stalled.) in new york, about 96% of students have been immunized against measles, mumps and rubella, yet \"a measles outbreak continues to affect communities in several parts of the state where the rate is lower,\" according to state health officials. in the 2017-2018 school year, 26,217 students in new york, including those in public and private schools and children in day care and prekindergarten, had religious exemptions from vaccinations, officials said. \"although the state can claim high immunization rates overall, preventable diseases like measles remain a public health threat when administrative loopholes allow children to go unvaccinated, carrying the potential to harm communities and especially our most vulnerable residents,\" said dr. howard zucker, the commissioner of the state department of health. new york state sen. john liu, who represents queens, said while he thinks removing the religious exemption is the right move, he has heard from constituents who hold \"deep and sincere\" religious beliefs who would be \"absolutely outraged that anyone would suggest that they don't care about the health of their children.\" liu suggested that the tenor of the debate on both sides could be more civil. \"we can respectfully disagree,\" liu said. the law eliminating religious exemptions takes effect immediately. unvaccinated students will have up to 30 days to show school officials they have received their first dose of each required immunization. in april, new york city health officials declared a public health emergency because of the measles outbreak. parents of unvaccinated children could be fined $1,000 for not complying with the order.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ISvGy7RgRIKM",
        "colab_type": "text"
      },
      "source": [
        "## Brief analysis\n",
        "TS-SS seems better than cosine on average. (probably not confirmation bias). Title seems better than text on average too but perhaps thats me judging a book by the cover."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kmik4vG6TS1K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pickle\n",
        "with open('./tfidf_vectorizer.bin','wb') as f:\n",
        "    pickle.dump({\n",
        "        'tfidf_text':text_vectorizer,\n",
        "        'tfidf_title':title_vectorizer\n",
        "    },f)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6BtM2V0f571G",
        "colab_type": "text"
      },
      "source": [
        "# Reload Vectorizers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4A1GrMOa5FKM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pickle\n",
        "with open('./tfidf_vectorizer.bin','rb') as f:\n",
        "    temp = pickle.load(f)\n",
        "    text_vectorizer = temp['tfidf_text']\n",
        "    title_vectorizer = temp['tfidf_title']\n",
        "    del temp"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S6xvtawn53tB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "database = pd.concat({'df':df.copy(),\n",
        "                 'tfidf_text':pd.DataFrame.sparse.from_spmatrix(text_vectorizer.transform(df.text),index=df.index),\n",
        "                 'tfidf_title':pd.DataFrame.sparse.from_spmatrix(title_vectorizer.transform(df.title),index=df.index)\n",
        "                 },\n",
        "                axis=1)#,copy=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DWZNi04zLkV5",
        "colab_type": "text"
      },
      "source": [
        "# tfidf + GloVe + tsss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JwpKkC5UPqUx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class vectorGetter():\n",
        "    def __init__(self,word2vec,vocab2ind):\n",
        "        self.word2vec = word2vec\n",
        "        self.ind2vocab = {i:w for w,i in vocab2ind.items()}\n",
        "        self.vec_dim = word2vec.vector_size\n",
        "\n",
        "    def __call__(self,t):\n",
        "        w = '_'.join(self.ind2vocab[t].split())\n",
        "        try:\n",
        "            emb = self.word2vec[w]\n",
        "        except KeyError:\n",
        "            embs = []\n",
        "            try:\n",
        "                for sw in w.split('_'): embs.append(self.word2vec[sw])\n",
        "                emb = np.mean(embs,axis=0)\n",
        "            except KeyError:\n",
        "                return np.zeros(self.vec_dim,dtype=np.float32)\n",
        "        return emb"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6gWkilh3Lpkw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.preprocessing import normalize\n",
        "class embedtsss(vectorGetter):\n",
        "    def __init__(self,word2vec,vocab2ind):\n",
        "        super(embedtsss,self).__init__(word2vec,vocab2ind)\n",
        "        #self.tsss = TS_SS()\n",
        "        self.tsss = cosine\n",
        "\n",
        "    def convert_tfidf(self,w):\n",
        "        con = {i:{} for i in range(len(w))}\n",
        "        for y,x in np.column_stack(np.nonzero(w)):\n",
        "            con[y][x] = w[y][x]\n",
        "        return con\n",
        "\n",
        "    def gen_weights(self,docs):\n",
        "        weights = np.zeros((len(docs),self.vec_dim),dtype=np.float32)\n",
        "        for i,freq in docs.items():\n",
        "            for t,f in freq.items():\n",
        "                weights[i] += f * super().__call__(t)\n",
        "        return weights\n",
        "    \n",
        "    def __call__(self,query,samples):\n",
        "        dists = []\n",
        "        doc1 = self.convert_tfidf(normalize((query,),norm='l2'))\n",
        "        docs = self.convert_tfidf(normalize(samples,norm='l2'))\n",
        "        w1 = self.gen_weights(doc1)[0]\n",
        "        w2 = self.gen_weights(docs)\n",
        "        return self.tsss(w1,w2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FU2Wj9NXRDdo",
        "colab_type": "code",
        "outputId": "b934df96-04e7-4dda-8784-1eefb2962e68",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 258
        }
      },
      "source": [
        "test_query = 'vaccines cause autism'\n",
        "etsss1 = embedtsss(glove,text_vectorizer.vocabulary_)\n",
        "etsss2 = embedtsss(glove,title_vectorizer.vocabulary_)\n",
        "#^recalculated weights werent cached...\n",
        "print('TS-SS text')\n",
        "query(test_query,\n",
        "      lambda s: text_vectorizer.transform([s]).toarray()[0],\n",
        "      etsss1, #i changed from tsss to cosine in the class declaration temporarily\n",
        "      database.tfidf_text.values,\n",
        "      database.df.uuid,\n",
        "      database)\n",
        "'''\n",
        "print('\\nTS-SS title')\n",
        "query(test_query,\n",
        "      lambda s: title_vectorizer.transform([s]).toarray()[0],\n",
        "      etsss2,\n",
        "      database.tfidf_title.values,\n",
        "      database.df.uuid,\n",
        "      database)\n",
        "'''"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TS-SS text\n",
            "ba97\t0.225629\tautism and adhd share genes\n",
            "1f58\t0.261112\tvaccines news\n",
            "140d\t0.268623\tbird flu research news\n",
            "0984\t0.273196\tbird flu news\n",
            "2028\t0.277309\tinfluenza news\n",
            "a2a3\t0.279930\tjapan launches human trial of new ebola vaccine\n",
            "9275\t0.283978\tmalaysias first polio case in 27 years is a child who wasnt vaccinated. here are 3 things to know about the incurable virus\n",
            "78a1\t0.284980\tnew vaccine will stop the spread of bovine tb\n",
            "0333\t0.285415\tbuilding a better flu shot\n",
            "9380\t0.287054\tpolio scare: malaysia begins vaccination campaign as first case in 27 years reported\n",
            "researchers from the national psychiatric project ipsych have found that autism and adhd share changes in the same genes. the new knowledge relates directly to the biological causes of the two child psychiatric disorders. in denmark, approximately one per cent of schoolchildren have autism and two to three per cent are diagnosed with adhd. autism and adhd are different developmental disorders, but they can have certain common symptoms. for example, children with autism can demonstrate violent or aggressive behaviour, be impulsive, and have problems in school and with social relations -- and these same symptoms can be shared by children with adhd. researchers from ipsych, denmark's largest research project in the field of psychiatry, have now discovered that the similarities between the two diagnoses can be linked to changes in the same genes. the new study is the largest study to date of rare mutations in the genome of people with adhd and autism. \"the very fact that mutations are found to the same extent and in the same genes in children with autism and in children with adhd, points towards the same biological mechanisms being involved,\" says anders brglum, who is professor at aarhus university, principal investigator at ipsych and one of the leading researchers behind the study. the study was carried out in collaboration with the broad institute and harvard. first time a gene is linked to adhd and autism the results have just been published in the international journal nature neuroscience. advertisement \"this is the first time that the genome has been mapped so comprehensively for both adhd and autism, and the discovery that children with adhd have the same amount of deleterious gene mutations in their dna as children with autism is both striking and quite surprising,\" says anders brglum. the findings point directly towards which biological causal mechanisms are shared by adhd and autism. \"in the study, the gene that is most frequently affected by mutations in people with adhd or autism is the so-called map1a gene. the gene is involved in the formation of the physical structure of nerve cells -- their inner 'skeleton', so to speak -- and is important for the development of the brain,\" explains anders brglum. this is the first time that this gene has been linked to the development of adhd and autism, and the mutations found by the researchers give a significantly increased risk of developing autism and/or adhd. \"we discovered an increased burden of mutations that destroy or severely affect the map1a gene in those with adhd and autism, while very few of the control subjects had such changes in the gene,\" explains anders brglum. this means that the risk of adhd and autism increases by more than 15 times for people who carry such a mutation. the researchers have analysed the genes of approximately 8,000 people with autism and/or adhd, and 5,000 people without either of the two disorders, all from denmark. the study also incorporated additional data from approx. 45,000 international control subjects who did not have any psychiatric disorders. \"the study shows that many more genes for adhd and autism can be identified directly by studying more people in a similar way with extensive dna sequencing, thereby providing a more complete picture of the biological causal mechanisms and possible approaches to medical treatment,\" explains the researcher. as part of the project, anders brglum and his colleagues have had access to samples from the danish national biobank at the statens serum institut and to data from the danish psychiatric central research register. people born in denmark between 1981 and 2005 were matched with diagnoses of autism, adhd and mental retardation.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"\\nprint('\\nTS-SS title')\\nquery(test_query,\\n      lambda s: title_vectorizer.transform([s]).toarray()[0],\\n      etsss2,\\n      database.tfidf_title.values,\\n      database.df.uuid,\\n      database)\\n\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 127
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ByIsgK3iXQ71",
        "colab_type": "text"
      },
      "source": [
        "Too general. L2 norm was very general, L1 norm was slightly better... how about max norm? okay it was much worse. Perhaps it has to do with naively comparing distances between relatively low-dimensional word embeddings (seen from quite low distances when using cosine). Question then would be if so, why would the same approach but using WMD instead be so successful?\n",
        "\n",
        "Cosine distance is better than tsss in this case. Cosine L2 better than cosine L1 or max? what?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "d_npQsnL-zhF"
      },
      "source": [
        "# tfidf + GloVe + WMD"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I1GPxx59onrh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#!pip install wmd\n",
        "import wmd\n",
        "#^needed to get libwmdrelax\n",
        "import libwmdrelax\n",
        "class wmdistance(vectorGetter):\n",
        "    def __init__(self,word2vec,vocab2ind):\n",
        "        super(wmdistance,self).__init__(word2vec,vocab2ind)\n",
        "    \n",
        "    def convert_tfidf(self,w):\n",
        "        return {t:w[t] for t in np.nonzero(w)[0]}\n",
        "\n",
        "    def get_vocab(self,doc1,doc2):\n",
        "        return {t:i for i,t in enumerate(sorted(set(doc1).union(doc2)))}\n",
        "\n",
        "    def gen_weights(self,doc,vocab):\n",
        "        w = np.zeros(len(vocab),dtype=np.float32)\n",
        "        if(len(doc) == 0): return w\n",
        "        for t,f in doc.items():\n",
        "            w[vocab[t]] = f\n",
        "        w /= w.sum()\n",
        "        return w\n",
        "\n",
        "    def __call__(self,query,samples):\n",
        "        dists = []\n",
        "        doc1 = self.convert_tfidf(query)\n",
        "        for doc in samples:\n",
        "            doc2 = self.convert_tfidf(doc)\n",
        "            dists.append(self.calc_dist(doc1,doc2))\n",
        "        return dists\n",
        "    \n",
        "    #In summary, for only the set of words present in the compared docs:\n",
        "    # weight of words determined by input\n",
        "    # distance between words calculated from embeddings\n",
        "    # calculates earth mover's distance using the above\n",
        "    # as tfidf may have been too loose for keyword extraction\n",
        "    # Key is to strategically deweigh/remove unimportant words\n",
        "    # ^find better weighing system?\n",
        "    def calc_dist(self,doc1,doc2):\n",
        "        vocab = self.get_vocab(doc1,doc2)\n",
        "        w1 = self.gen_weights(doc1, vocab)\n",
        "        w2 = self.gen_weights(doc2, vocab)\n",
        "        if np.all(w1==0) or np.all(w2==0): return np.inf\n",
        "        evec = np.zeros((len(vocab),self.vec_dim),dtype=np.float32)\n",
        "        for t,i in vocab.items():\n",
        "            evec[i] = super().__call__(t)\n",
        "        evec_sqr = (evec*evec).sum(axis=1)\n",
        "        dists = evec_sqr - 2*evec.dot(evec.T)+evec_sqr[:, np.newaxis]\n",
        "        dists[dists < 0] = 0\n",
        "        dists = np.sqrt(dists)\n",
        "        try:\n",
        "            ans = libwmdrelax.emd(w1,w2,dists)\n",
        "        except RuntimeError:\n",
        "            print(dists)\n",
        "            print(w1)\n",
        "            print(w2)\n",
        "            raise RuntimeError\n",
        "        return ans"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Od6Khpa_Wkw",
        "colab_type": "code",
        "outputId": "5eee87ee-75ee-46d5-bd99-bcbc73e1fe8b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        }
      },
      "source": [
        "wmd1 = wmdistance(glove,text_vectorizer.vocabulary_)\n",
        "#cannot cache weights, wmd weights dependent on context of both docs. At most find faster version of np.nonzero given that sparse matrix is used\n",
        "query('vaccines cause autism',\n",
        "      lambda s: text_vectorizer.transform([s]).toarray()[0],\n",
        "      wmd1,\n",
        "      database.tfidf_text.values,\n",
        "      database.df.uuid,\n",
        "      database)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ba97\t0.991549\tautism and adhd share genes\n",
            "0984\t1.050522\tbird flu news\n",
            "1f58\t1.055506\tvaccines news\n",
            "0333\t1.062309\tbuilding a better flu shot\n",
            "140d\t1.062313\tbird flu research news\n",
            "90fc\t1.067561\tcancer patients, survivors face increased risk of heart disease deaths\n",
            "524d\t1.068516\tadd and adhd news\n",
            "b76a\t1.071482\tbuilding a better flu vaccine -- one you don't have to get every year\n",
            "1fd4\t1.071544\tthe 2019-2020 flu season is off to an early start\n",
            "c21e\t1.071725\tinfectious diseases news\n",
            "researchers from the national psychiatric project ipsych have found that autism and adhd share changes in the same genes. the new knowledge relates directly to the biological causes of the two child psychiatric disorders. in denmark, approximately one per cent of schoolchildren have autism and two to three per cent are diagnosed with adhd. autism and adhd are different developmental disorders, but they can have certain common symptoms. for example, children with autism can demonstrate violent or aggressive behaviour, be impulsive, and have problems in school and with social relations -- and these same symptoms can be shared by children with adhd. researchers from ipsych, denmark's largest research project in the field of psychiatry, have now discovered that the similarities between the two diagnoses can be linked to changes in the same genes. the new study is the largest study to date of rare mutations in the genome of people with adhd and autism. \"the very fact that mutations are found to the same extent and in the same genes in children with autism and in children with adhd, points towards the same biological mechanisms being involved,\" says anders brglum, who is professor at aarhus university, principal investigator at ipsych and one of the leading researchers behind the study. the study was carried out in collaboration with the broad institute and harvard. first time a gene is linked to adhd and autism the results have just been published in the international journal nature neuroscience. advertisement \"this is the first time that the genome has been mapped so comprehensively for both adhd and autism, and the discovery that children with adhd have the same amount of deleterious gene mutations in their dna as children with autism is both striking and quite surprising,\" says anders brglum. the findings point directly towards which biological causal mechanisms are shared by adhd and autism. \"in the study, the gene that is most frequently affected by mutations in people with adhd or autism is the so-called map1a gene. the gene is involved in the formation of the physical structure of nerve cells -- their inner 'skeleton', so to speak -- and is important for the development of the brain,\" explains anders brglum. this is the first time that this gene has been linked to the development of adhd and autism, and the mutations found by the researchers give a significantly increased risk of developing autism and/or adhd. \"we discovered an increased burden of mutations that destroy or severely affect the map1a gene in those with adhd and autism, while very few of the control subjects had such changes in the gene,\" explains anders brglum. this means that the risk of adhd and autism increases by more than 15 times for people who carry such a mutation. the researchers have analysed the genes of approximately 8,000 people with autism and/or adhd, and 5,000 people without either of the two disorders, all from denmark. the study also incorporated additional data from approx. 45,000 international control subjects who did not have any psychiatric disorders. \"the study shows that many more genes for adhd and autism can be identified directly by studying more people in a similar way with extensive dna sequencing, thereby providing a more complete picture of the biological causal mechanisms and possible approaches to medical treatment,\" explains the researcher. as part of the project, anders brglum and his colleagues have had access to samples from the danish national biobank at the statens serum institut and to data from the danish psychiatric central research register. people born in denmark between 1981 and 2005 were matched with diagnoses of autism, adhd and mental retardation.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5fAgff3XEczI",
        "colab_type": "code",
        "outputId": "b13e68dc-bbe2-4baa-cbc9-6e9a7f22fe70",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        }
      },
      "source": [
        "wmd2 = wmdistance(glove,title_vectorizer.vocabulary_)\n",
        "query('vaccines cause autism',\n",
        "      lambda s: title_vectorizer.transform([s]).toarray()[0],\n",
        "      wmd2,\n",
        "      database.tfidf_title.values,\n",
        "      database.df.uuid,\n",
        "      database)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7a57\t0.681598\tautism news\n",
            "56cf\t0.681598\tautism news\n",
            "1f58\t0.690510\tvaccines news\n",
            "aeb0\t0.765396\tvaccines fast facts\n",
            "ba97\t0.806746\tautism and adhd share genes\n",
            "a2dd\t0.876639\tnurse takes in man with autism so he can get a new heart\n",
            "78a1\t0.912511\tnew vaccine will stop the spread of bovine tb\n",
            "b76a\t0.951569\tbuilding a better flu vaccine -- one you don't have to get every year\n",
            "245b\t0.952104\ti'm thankful for autism, eight reasons this thanksgiving i am giving thanks\n",
            "ccbd\t0.961914\tside effects of pediatric medications for anxiety, ocd\n",
            "read more down syndrome, due to an extra chromosome 21, occurs in 250,000 children and adults in the united states, making it the country's most common chromosomal disorder. inherited heart defects, ... read more a research team has found exposure to sound -- not sound reduction -- during early development of mice engineered to have fragile x syndrome, or fxs, restores molecular, cellular, and functional ... read more the annual transition to and from daylight saving time (dst) has clinical implications that last longer than the days where clocks 'fall back' or 'spring ... read more zebrafish research is a promising way to understand the neural and genetic causes of eye movement problems in people, according to multi-university ... read more severe infections during pregnancy have been connected to a range of psychiatric disorders by different studies in humans and animals. now, researchers have shown in mice how the infections affect ... read more a team of psychology researchers is providing one of the first comprehensive literature reviews on arrogance, as well as a way to classify the condition on different levels across a spectrum, ... read more when people become stressed, their bodies can respond by sweating. now, researchers are monitoring how much adolescents severely affected by autism sweat in order to better understand when behavioral ... read more communicating with others is one of the biggest difficulties for autistic children. asking for help, initiating social interaction and sharing their emotions become skills that require learning ... read more researchers have devised a new way to rapidly image synaptic proteins at high resolution. using fluorescent nucleic acid probes, they can label and image as many as 12 different proteins in neuronal ... read more the expression of many genes that have previously been associated with autism is abnormal also in violent psychopathy, a new study shows. the researchers used stem cell technology to analyse the ... read more autism rates among black and hispanic children are catching up to and, in many states, exceeding rates among white youth, who have historically had higher prevalence, a new study ... read more a specific gene associated with autism appears to undergo changes in the sperm of men who use marijuana, according to new research. the gene change occurs through a process called dna methylation, ... read more pivotal response treatment involving parents works better than other existing therapies at motivating children with autism and significant speech delays to talk, according to the results of a large ... read more neurodevelopmental disorders arising from rare genetic mutations can cause atypical cognitive function, intellectual disability, and developmental delays, yet it is unclear why and how this happens. ... read more scientist have identified a link between exposure to high levels of estrogen sex hormones in the womb and the likelihood of developing ... read more neuroscientists have identified a brain region linked to altered social interactions in an autism model. restoring activity of a specific forebrain region in a mouse model also reversed social traits ... read more oxytocin is referred to as the love hormone and is important in the regulation of social and maternal behavior. the oxytocin system in the brain may be key to new treatments for many mental health ... read more for the first time researchers have been able to demonstrate that the brains of a patient and therapist become synchronized during a music therapy session, a breakthrough that could improve future ... read more for the brain to function smoothly, its cells must carefully regulate which proteins are produced and when. by studying gene regulation, researchers are now shedding light on complex brain conditions ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LN_rGQKHDibw",
        "colab_type": "text"
      },
      "source": [
        "## Brief Analysis\n",
        "Even only using global vectors (static), seems to be extremely on point with queries... except it took forever to run (for text at least). Try compiling it? Well, seems definitely much more on point then the other base tfidf text. Should be better on tfidf title since title is better topic indicator (see previous analysis). however, it did miss the really good article tfidf ts-ss text recommended as first.\n",
        "\n",
        "Title processed faster but seems to only be on par. So I guess tfidf title first, and then wmd text (though it takes forever). I have yet to try NER.\n",
        "\n",
        "On better understanding of how WMD works, as tfidf may have been too loose for keyword extraction, strategically deweighing/removing unimportant words using other techniques (NER?) might help calculate better distances."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S6XnXPnnguiC",
        "colab_type": "text"
      },
      "source": [
        "# Sent2Vec Base"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G4hqvWZigxf7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#!pip install -U sentence-transformers\n",
        "promising = (\n",
        "#'bert-base-wikipedia-sections-mean-tokens', #crap\n",
        "'distilbert-base-nli-stsb-mean-tokens',\n",
        "'roberta-base-nli-stsb-mean-tokens',\n",
        "#'roberta-base-nli-mean-tokens', #crap\n",
        "#'distilbert-base-nli-mean-tokens'\n",
        ")\n",
        "from sentence_transformers import SentenceTransformer"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mWm5e-C_E0M7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from gensim.summarization import summarize, keywords\n",
        "def encode_document(sent,model):\n",
        "    return model.encode([sent])[0]\n",
        "    #try:\n",
        "        #ext = summarize(sent,split=True)\n",
        "        #if len(ext) <= 1:\n",
        "        #    return model.encode([sent])[0]\n",
        "        #else:\n",
        "        #    return np.mean(model.encode(ext),axis=0)\n",
        "    #except ValueError:\n",
        "    #    return model.encode([sent])[0]\n",
        "def encode_documents(sents,model):\n",
        "    return np.stack(pd.Series(sents).apply(encode_document,args=(model,)))\n",
        "#np.mean(model.encode(summarize(df.loc[31].text,split=True)),axis=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E51c-XB7ItVd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "all_enc = {}\n",
        "for model_name in promising:\n",
        "    model = SentenceTransformer(model_name)\n",
        "    all_enc[model_name+'_title'] = pd.DataFrame(encode_documents(df.title,model),index=df.index)\n",
        "    all_enc[model_name+'_text'] = pd.DataFrame(encode_documents(df.text,model),index=df.index)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lc9TmX3lC-PX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pickle\n",
        "with open('./raw/sentTransData.bin','wb') as f:\n",
        "    pickle.dump(all_enc,f)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gm5ZqYOSiXtZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pickle\n",
        "with open('./raw/sentTransData.bin','rb') as f:\n",
        "    all_enc = pickle.load(f)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t73yh9NxNS4v",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "all_enc['df'] = df.copy()\n",
        "db2 = pd.concat(all_enc,axis=1)#,copy=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0QqSISX3y9uM",
        "colab_type": "code",
        "outputId": "47fb49e0-76ad-49fb-f5c4-b838946fff86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 989
        }
      },
      "source": [
        "test_query = \"vaccines cause autism\"\n",
        "for model_name in promising:\n",
        "    model = SentenceTransformer(model_name)\n",
        "    print(model_name)\n",
        "    print('Title')\n",
        "    query(test_query,\n",
        "        lambda s: encode_document(s,model),\n",
        "        tsss,\n",
        "        db2[model_name+'_title'].values,\n",
        "        db2.df.uuid,\n",
        "        db2)\n",
        "    print('\\nText')\n",
        "    query(test_query,\n",
        "        lambda s: encode_document(s,model),\n",
        "        tsss,\n",
        "        db2[model_name+'_text'].values,\n",
        "        db2.df.uuid,\n",
        "        db2)\n",
        "    print('\\n')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "distilbert-base-nli-stsb-mean-tokens\n",
            "Title\n",
            "1f58\t98861336.000000\tvaccines news\n",
            "7a57\t98914192.000000\tautism news\n",
            "56cf\t98914192.000000\tautism news\n",
            "aeb0\t98951768.000000\tvaccines fast facts\n",
            "2981\t99053704.000000\tthere's a promising new vaccine for one of the world's top health threats\n",
            "5634\t99097680.000000\tan experimental genetic test gives early warning for kids at risk of type 1 diabetes\n",
            "ba97\t99102272.000000\tautism and adhd share genes\n",
            "ccbd\t99142496.000000\tside effects of pediatric medications for anxiety, ocd\n",
            "2028\t99154816.000000\tinfluenza news\n",
            "5f06\t99155096.000000\tspecialized immune cells could help repair inflammatory bowel disease damage in children\n",
            "read more a new study has found severe pneumonia decreases by 35 per cent in children who receive a vaccine against a pneumonia-causing ... read more a newly developed experimental vaccine was more than eighty percent effective in protecting mice from succumbing to staphylococcus aureus infection. s. aureus causes more than 30,000 deaths from ... read more within seconds after an infected mosquito bites, the malaria parasite navigates the host skin and blood vessels to invade the liver, where it will stay embedded until thousands of infected cells ... read more scientists have made a breakthrough in the hunt for a new vaccine for killer hospital bug clostridioides difficile (c. ... read more a new study indicates a way for cancer immunotherapy to spur a more robust immune response. such knowledge could lead to the development of better cancer vaccines and more effective immunotherapy ... read more a school-based human papillomavirus (hpv) immunization program in british columbia, canada, is dramatically reducing rates of cervical pre-cancer in b.c. women, according to a new ... read more a single dose of a highly diluted vsv-ebola virus (ebov) vaccine -- approximately one-millionth of what is in the vaccine being used to help control the ongoing ebola outbreak in the democratic ... read more there is no effective vaccine currently available to prevent lyme disease in humans. experts from academia, government, and industry recently convened to tackle this public health challenge. now, a ... read more predicting and controlling disease outbreaks would be easier and more reliable with the wider application of mathematical modelling, according to a new ... read more scientists show, on the molecular level, how an experimental vaccine offers long-term protection against the ... read more a vaccine for the common and sometimes deadly rsv (respiratory syncytial virus) has been elusive, but scientists say a new discovery puts them much closer to ... read more scientists have found a common, new source of tumor mutations that could offer three levels of therapy with a cancer vaccine: 1) a broadly protective, or pan-cancer vaccine 2) cancer-type specific ... read more mit chemical engineers have developed a new type of lipid nanoparticle that can be used to deliver rna vaccines. the particles also trigger the release of powerful immune signaling proteins that ... read more a new type of vaccine that can be stored at warmer temperatures, removing the need for refrigeration, has been developed for mosquito-borne virus chikungunya in a major advance in vaccine technology. ... read more the biological clock influences immune response efficacy. indeed, cd8 t cells, which are essential to fight infections and cancers, function very differently according to the time of ... read more a novel vaccine protected almost all mice and guinea pigs exposed to a new herpes virus. this may lead to the vaccine being tested in human ... read more researchers have discovered that the dengue virus changes its shape through mutations in envelope protein to evade vaccines and therapeutics. the study also gives insights on the types of treatment ... read more a new needle-free flu vaccine patch revved up the immune system much like a traditional flu shot without any negative side effects. the research represents an important step toward a technology that ... read more despite its proven success at preventing cancer, many adolescents are still not getting the hpv vaccine. a new study from shows that physicians' delivery and communication practices must improve ...\n",
            "\n",
            "Text\n",
            "ba97\t78549096.000000\tautism and adhd share genes\n",
            "aeb0\t78550160.000000\tvaccines fast facts\n",
            "528a\t78553984.000000\tfacebook debuts vaccine education pop-up windows\n",
            "245b\t78570584.000000\ti'm thankful for autism, eight reasons this thanksgiving i am giving thanks\n",
            "d54c\t78589792.000000\twhat to do when your baby wont stop coughing\n",
            "5634\t78591504.000000\tan experimental genetic test gives early warning for kids at risk of type 1 diabetes\n",
            "020a\t78599216.000000\tleukemia: could germ-free childhoods could trigger illness?\n",
            "cf48\t78600800.000000\tinfant and preschool learning news\n",
            "9b30\t78601944.000000\tit's not just you: flu season strikes earlier than usual\n",
            "e8e9\t78604424.000000\ttick bites: 2 sickened newborns could signal increase in infections\n",
            "researchers from the national psychiatric project ipsych have found that autism and adhd share changes in the same genes. the new knowledge relates directly to the biological causes of the two child psychiatric disorders. in denmark, approximately one per cent of schoolchildren have autism and two to three per cent are diagnosed with adhd. autism and adhd are different developmental disorders, but they can have certain common symptoms. for example, children with autism can demonstrate violent or aggressive behaviour, be impulsive, and have problems in school and with social relations -- and these same symptoms can be shared by children with adhd. researchers from ipsych, denmark's largest research project in the field of psychiatry, have now discovered that the similarities between the two diagnoses can be linked to changes in the same genes. the new study is the largest study to date of rare mutations in the genome of people with adhd and autism. \"the very fact that mutations are found to the same extent and in the same genes in children with autism and in children with adhd, points towards the same biological mechanisms being involved,\" says anders brglum, who is professor at aarhus university, principal investigator at ipsych and one of the leading researchers behind the study. the study was carried out in collaboration with the broad institute and harvard. first time a gene is linked to adhd and autism the results have just been published in the international journal nature neuroscience. advertisement \"this is the first time that the genome has been mapped so comprehensively for both adhd and autism, and the discovery that children with adhd have the same amount of deleterious gene mutations in their dna as children with autism is both striking and quite surprising,\" says anders brglum. the findings point directly towards which biological causal mechanisms are shared by adhd and autism. \"in the study, the gene that is most frequently affected by mutations in people with adhd or autism is the so-called map1a gene. the gene is involved in the formation of the physical structure of nerve cells -- their inner 'skeleton', so to speak -- and is important for the development of the brain,\" explains anders brglum. this is the first time that this gene has been linked to the development of adhd and autism, and the mutations found by the researchers give a significantly increased risk of developing autism and/or adhd. \"we discovered an increased burden of mutations that destroy or severely affect the map1a gene in those with adhd and autism, while very few of the control subjects had such changes in the gene,\" explains anders brglum. this means that the risk of adhd and autism increases by more than 15 times for people who carry such a mutation. the researchers have analysed the genes of approximately 8,000 people with autism and/or adhd, and 5,000 people without either of the two disorders, all from denmark. the study also incorporated additional data from approx. 45,000 international control subjects who did not have any psychiatric disorders. \"the study shows that many more genes for adhd and autism can be identified directly by studying more people in a similar way with extensive dna sequencing, thereby providing a more complete picture of the biological causal mechanisms and possible approaches to medical treatment,\" explains the researcher. as part of the project, anders brglum and his colleagues have had access to samples from the danish national biobank at the statens serum institut and to data from the danish psychiatric central research register. people born in denmark between 1981 and 2005 were matched with diagnoses of autism, adhd and mental retardation.\n",
            "\n",
            "\n",
            "roberta-base-nli-stsb-mean-tokens\n",
            "Title\n",
            "0ecf\t286062304.000000\tresearch looks at side effects for pediatric medications\n",
            "2981\t286070240.000000\tthere's a promising new vaccine for one of the world's top health threats\n",
            "ccbd\t286119200.000000\tside effects of pediatric medications for anxiety, ocd\n",
            "1f58\t286184672.000000\tvaccines news\n",
            "9275\t286306912.000000\tmalaysias first polio case in 27 years is a child who wasnt vaccinated. here are 3 things to know about the incurable virus\n",
            "020a\t286398176.000000\tleukemia: could germ-free childhoods could trigger illness?\n",
            "9380\t286404864.000000\tpolio scare: malaysia begins vaccination campaign as first case in 27 years reported\n",
            "10c8\t286431008.000000\thealth & medicine news\n",
            "99a0\t286431008.000000\thealth & medicine news\n",
            "fdb2\t286431008.000000\thealth & medicine news\n",
            "jeffrey mills, associate professor in the department of economics at the uc lindner college of business, and dr. jeffrey strawn, associate professor in the department of psychiatry and behavioral neuroscience at the university of cincinnati college of medicine. credit: university of cincinnati sometimes, the medications needed to function and live a quality life cause side effects that can make life quite uncomfortable. dr. jeffrey strawn, associate professor in the department of psychiatry and behavioral neuroscience at the university of cincinnati college of medicine, and jeffrey mills, associate professor in the department of economics at the uc lindner college of business, published a study in the journal of the american academy of child & adolescent psychiatry looking specifically at side effects that impact children and adolescents being treated for anxiety disorders and obsessive-compulsive disorder (ocd). strawn says this is one of the first studies examining side effects of these medications in youth that doesn't just focus on suicidal thinking or discontinuation of medication. \"for youth with anxiety disorders and ocd, these medications improve symptoms and functional outcomes,\" he says. \"over the past two decades, selective serotonin reuptake inhibitors, known as ssris, and serotonin-norepinephrine reuptake inhibitors, known as snris, have become the standard medication treatments for pediatric patients with these conditions.\" the uc study assessed quality-of-life issues. \"evaluations of antidepressant tolerability focus almost entirely on discontinuation of the medicine or suicidality. we wanted to examine side effects commonly reported in pediatric patients treated with antidepressants, including agitation, nausea, abdominal pain, insomnia, headache and fatigue, in addition to suicidality and discontinuation of medication.\" ssris increase levels of serotonin in the brain. snris block the reabsorption of the neurotransmitters serotonin and norepinephrine in the brain. in this study, mills and strawn looked at academic peer-reviewed articles through march 1, 2019, and identified ssri and snri studies in patients under 18 with ocd and anxiety disorders, specifically noting side effect rates. they used statistical tools known as bayesian hierarchical models created by mills that enable results from different studies to be combined while taking into account variations across patients and those studies. \"out of 18 trials, which included more than 2,500 patients who were treated and then compared to patients taking a placebo, ssris produced more side effects; agitation was more common with ssri use,\" mills says. \"ssris and snris were not associated with suicidal thoughts. this finding is consistent with earlier studies that suggest that suicidality relates to the condition being treated,\" strawn adds. \"medication side effects are important for clinicians to consider, particularly in light of data suggesting these medications also differ in terms of how effective they are. ssris are a better option compared to snris and are the treatment of choice for children and adolescents with anxiety.\" explore further new study may help guide treatment of pediatric anxiety more information: jeffrey a. mills et al. antidepressant tolerability in pediatric anxiety and obsessive-compulsive disorders: a bayesian hierarchical modeling meta-analysis, journal of the american academy of child & adolescent psychiatry (2019). journal information: journal of the american academy of child & adolescent psychiatry jeffrey a. mills et al. antidepressant tolerability in pediatric anxiety and obsessive-compulsive disorders: a bayesian hierarchical modeling meta-analysis,(2019). doi: 10.1016/j.jaac.2019.10.013\n",
            "\n",
            "Text\n",
            "ba97\t237754208.000000\tautism and adhd share genes\n",
            "b3a8\t237785264.000000\tallergy news\n",
            "aeb0\t237806528.000000\tvaccines fast facts\n",
            "8ce1\t237963904.000000\thelper protein worsens diabetic eye disease: blocking second protein could lead to better treatment\n",
            "6628\t237976288.000000\tbrain function abnormal in children with type 1 diabetes\n",
            "9601\t237980816.000000\tstem cells news\n",
            "bdd1\t237989328.000000\tbrain function irregular in children with type 1 diabetes, study says\n",
            "c21e\t237993424.000000\tinfectious diseases news\n",
            "2b1d\t237999264.000000\tstem cells news\n",
            "7b21\t238008288.000000\tmachine-guided engineering of aav capsids\n",
            "researchers from the national psychiatric project ipsych have found that autism and adhd share changes in the same genes. the new knowledge relates directly to the biological causes of the two child psychiatric disorders. in denmark, approximately one per cent of schoolchildren have autism and two to three per cent are diagnosed with adhd. autism and adhd are different developmental disorders, but they can have certain common symptoms. for example, children with autism can demonstrate violent or aggressive behaviour, be impulsive, and have problems in school and with social relations -- and these same symptoms can be shared by children with adhd. researchers from ipsych, denmark's largest research project in the field of psychiatry, have now discovered that the similarities between the two diagnoses can be linked to changes in the same genes. the new study is the largest study to date of rare mutations in the genome of people with adhd and autism. \"the very fact that mutations are found to the same extent and in the same genes in children with autism and in children with adhd, points towards the same biological mechanisms being involved,\" says anders brglum, who is professor at aarhus university, principal investigator at ipsych and one of the leading researchers behind the study. the study was carried out in collaboration with the broad institute and harvard. first time a gene is linked to adhd and autism the results have just been published in the international journal nature neuroscience. advertisement \"this is the first time that the genome has been mapped so comprehensively for both adhd and autism, and the discovery that children with adhd have the same amount of deleterious gene mutations in their dna as children with autism is both striking and quite surprising,\" says anders brglum. the findings point directly towards which biological causal mechanisms are shared by adhd and autism. \"in the study, the gene that is most frequently affected by mutations in people with adhd or autism is the so-called map1a gene. the gene is involved in the formation of the physical structure of nerve cells -- their inner 'skeleton', so to speak -- and is important for the development of the brain,\" explains anders brglum. this is the first time that this gene has been linked to the development of adhd and autism, and the mutations found by the researchers give a significantly increased risk of developing autism and/or adhd. \"we discovered an increased burden of mutations that destroy or severely affect the map1a gene in those with adhd and autism, while very few of the control subjects had such changes in the gene,\" explains anders brglum. this means that the risk of adhd and autism increases by more than 15 times for people who carry such a mutation. the researchers have analysed the genes of approximately 8,000 people with autism and/or adhd, and 5,000 people without either of the two disorders, all from denmark. the study also incorporated additional data from approx. 45,000 international control subjects who did not have any psychiatric disorders. \"the study shows that many more genes for adhd and autism can be identified directly by studying more people in a similar way with extensive dna sequencing, thereby providing a more complete picture of the biological causal mechanisms and possible approaches to medical treatment,\" explains the researcher. as part of the project, anders brglum and his colleagues have had access to samples from the danish national biobank at the statens serum institut and to data from the danish psychiatric central research register. people born in denmark between 1981 and 2005 were matched with diagnoses of autism, adhd and mental retardation.\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_dSwKdiT4Je",
        "colab_type": "text"
      },
      "source": [
        "## Brief Analysis\n",
        "Actually loses out to tfidf based on this search query. Seems more capable than tfidf at gathered articles that share similar contexts - except they are too broad. For example, using the PIE expressway query, it doesn't pull up results related to pie crusts. Corroborates that a tfidf weighted approach might be best."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tWGcwPz1hQ_o",
        "colab_type": "text"
      },
      "source": [
        "# GENSIM\n",
        "used for word embeddings (tfidf as weights approach), provides distance formulae like cosine and WMD"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sn3019v8hDhK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gensim\n",
        "from gensim.models import KeyedVectors\n",
        "#word_vectors.distance(\"media\", \"media\") #cosine"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O6WqhVluUgkK",
        "colab_type": "text"
      },
      "source": [
        "## Conversion code for saving ignore if already run"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OVa7z-RBD8it",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#model.syn0norm = model.syn0\n",
        "#newsve.init_sims(replace=True)\n",
        "#glove.init_sims(replace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rLK9miWyJAr2",
        "colab_type": "code",
        "outputId": "f8a69762-1be1-469e-cd78-0e86fb30b1de",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "#newsve.save('GoogleNews-vectors-gensim-normed.bin')\n",
        "#glove.save('gensim_glove_vectors_normed.bin')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rweIb3ZRUs1y",
        "colab_type": "text"
      },
      "source": [
        "## Load vectors"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7T4kePErigDu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "glove_loc = '/content/drive/My Drive/CSIT Internship/code/gensim_glove_vectors_normed.bin' #Common crawl uncased 42 billion tokens 300 dimensions\n",
        "word2vec_loc = '/content/drive/My Drive/CSIT Internship/code/GoogleNews-vectors-gensim-normed.bin' #google news 3 million tokens 300 dimensions"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B5RknIaSiEkm",
        "colab_type": "code",
        "outputId": "a38d17bb-eae5-4834-b2b9-a3a47166c1f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "source": [
        "#newsve = KeyedVectors.load(word2vec_loc, mmap='r')\n",
        "#newsve.syn0norm = newsve.syn0\n",
        "glove = KeyedVectors.load(glove_loc, mmap='r')\n",
        "glove.syn0norm = glove.syn0\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `syn0` (Attribute will be removed in 4.0.0, use self.wv.vectors instead).\n",
            "  \n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `syn0norm` (Attribute will be removed in 4.0.0, use self.wv.vectors_norm instead).\n",
            "  \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gccq7MjjdJrj",
        "colab_type": "text"
      },
      "source": [
        "# Flair Embedddings\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kgx3Z-0PdLuX",
        "colab_type": "code",
        "outputId": "7214c38d-9f4e-4b39-ed2b-de3d8faebf52",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "#!pip install https://github.com/zalandoresearch/flair/archive/GH-1221-fix-tokenization-problems.zip\n",
        "from flair.embeddings import DocumentPoolEmbeddings,WordEmbeddings, Sentence, XLNetEmbeddings\n",
        "import torch\n",
        "torch.cuda.empty_cache()\n",
        "# init embedding\n",
        "xlnet_embedding = XLNetEmbeddings(\"xlnet-base-cased\")#, layers=\"0,1,2,3,4,5,6,7,8,9,10,11,12\",pooling_operation=\"first\", use_scalar_mix=True)\n",
        "#flair_forward_embedding = FlairEmbeddings('mix-forward')\n",
        "#flair_backward_embedding = FlairEmbeddings('mix-backward')\n",
        "base_embedding = WordEmbeddings('en') #fasttext without subwords\n",
        "#bpe_embedding = BytePairEmbeddings('en')\n",
        "#roberta_embedding = RoBERTaEmbeddings('roberta-base')\n",
        "\n",
        "#and henceforth, Im doomed to mix and match for all of friday.\n",
        "#https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/90426\n",
        "\n",
        "stacked_embedder = DocumentPoolEmbeddings(embeddings=[xlnet_embedding, base_embedding])"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tHOZiWLTfp5e",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def temp1(s):\n",
        "    temp = Sentence(s)\n",
        "    #print(temp)\n",
        "    try:\n",
        "        stacked_embedder.embed(temp)\n",
        "        ans = temp.embedding.cpu().detach().numpy()\n",
        "    except RuntimeError:\n",
        "        torch.cuda.empty_cache()\n",
        "        try:\n",
        "            stacked_embedder.embed(temp)\n",
        "            ans = temp.embedding.cpu().detach().numpy()\n",
        "        except RuntimeError:\n",
        "            ans = np.zeros(stacked_embedder.embedding_length,dtype=np.float32)\n",
        "            torch.cuda.empty_cache()\n",
        "            print(s)\n",
        "    del temp\n",
        "    return ans"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iwbusqJNlOLV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "db3 = pd.concat({'df':df.copy(),\n",
        "                 'flair_text':pd.DataFrame(np.stack(df.text.apply(temp1).values),index=df.index),\n",
        "                 'flair_title':pd.DataFrame(np.stack(df.title.apply(temp1).values),index=df.index)\n",
        "                 },\n",
        "                axis=1)#,copy=False)\n",
        "#^ for the record this took approximately more than 3 hours [flair forward, backward, XLNET, base]\n",
        "#^ 11 mins for [XLnNet, Base]\n",
        "db3.to_pickle('./raw/db3.pkl')\n",
        "#tends to drop sentences. use a empty_cache, retry, on fail give up style"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UoahC3f5x7dc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "db3 = pd.read_pickle('./raw/db3.pkl')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zUycE1M0_gFZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 904
        },
        "outputId": "11108204-1912-486a-b940-d265d8325c8b"
      },
      "source": [
        "test_query = Sentence('vaccines cause autism')\n",
        "stacked_embedder.embed(test_query)\n",
        "print('Cosine text')\n",
        "query(test_query,\n",
        "      lambda s: test_query.embedding.cpu().detach().numpy(),\n",
        "      cosine,\n",
        "      db3.flair_text.values,\n",
        "      db3.df.uuid,\n",
        "      db3)\n",
        "print('\\nTS-SS text')\n",
        "query(test_query,\n",
        "      lambda s: test_query.embedding.cpu().detach().numpy(),\n",
        "      tsss,\n",
        "      db3.flair_text.values,\n",
        "      db3.df.uuid,\n",
        "      db3)\n",
        "print('\\nCosine title')\n",
        "query(test_query,\n",
        "      lambda s: test_query.embedding.cpu().detach().numpy(),\n",
        "      cosine,\n",
        "      db3.flair_title.values,\n",
        "      db3.df.uuid,\n",
        "      db3)\n",
        "print('\\nTS-SS title')\n",
        "query(test_query,\n",
        "      lambda s: test_query.embedding.cpu().detach().numpy(),\n",
        "      tsss,\n",
        "      db3.flair_title.values,\n",
        "      db3.df.uuid,\n",
        "      db3)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cosine text\n",
            "64db\t0.614733\tprogrammes in place at public hospitals to counter antibiotic resistance\n",
            "9fae\t0.620411\tconcussion recovery not clear cut for children\n",
            "aeb0\t0.629850\tvaccines fast facts\n",
            "c968\t0.642287\tdementia study reveals how proteins interact to stop brain signals\n",
            "0333\t0.644414\tbuilding a better flu shot\n",
            "140d\t0.648763\tbird flu research news\n",
            "1f58\t0.649355\tvaccines news\n",
            "4ffa\t0.649600\ttype 2 diabetes: drinking this type of tea could lower blood sugar\n",
            "fa8d\t0.650068\tgenetic brain disorder fixed in mice using precision epigenome editing\n",
            "bcf9\t0.651480\tcalifornia considers calling thc in pot a risk to moms-to-be\n",
            "we are pleased to see the increasing awareness of antibiotic resistance and applaud the study by the singapore general hospital on not using antibiotics to treat non-bacterial infection (stopping antibiotics early may help patients: sgh, nov 27). antibiotics work on bacteria by stopping their growth or killing them. they do not work on infections caused by viruses. the basic principles of appropriate antibiotic use include not using antibiotics for non-bacterial infections and using the right antibiotics for the target bacteria for an appropriate duration. this reduces overuse of antibiotics that can accelerate the development of resistance against the effects of antibiotics. antibiotic resistance makes infections harder to treat and requires the use of more potent and costly antibiotics, sometimes with more side effects. in some cases, we may run out of antibiotic options. as the world becomes better connected through travel, trade and commerce, antibiotic resistance can spread quickly. globally, antibiotic resistance is increasing and singapore has not been spared. a comprehensive programme to reduce the development of antibiotic resistance employs multiple strategies to achieve this, including educating and raising awareness among prescribers, surveillance to monitor the trends of antibiotic resistance and antibiotic use, and providing antibiotic recommendations to medical teams. one of the ongoing initiatives singapore has taken was the setting up of antimicrobial stewardship programmes (asp) in all public hospitals in 2011. an asp promotes appropriate antibiotic use with the aim of improving patient outcomes while reducing the risk of bacteria developing resistance. studies have shown that antimicrobial stewardship is safe, effective and improves patient outcomes while reducing the use of antibiotics and healthcare costs. the typical asp team, consisting of infectious diseases physicians and pharmacists, reviews patients systematically, taking into account clinical history, physical findings and diagnostic test results. advice on appropriate antibiotic use is given to medical teams to stop, or adjust antibiotic choice, dose or duration. the fight against antibiotic resistance is not restricted to hospitals or prescribers. public awareness of sound antibiotic use, such as not using antibiotics for viral infections and adhering to recommended duration of antibiotic treatment, plays a part in reducing the risk of antibiotic resistance. dr lee tau hong head antimicrobial resistance coordinating office national centre for infectious diseases\n",
            "\n",
            "TS-SS text\n",
            "a035\t90576432.000000\tNone\n",
            "5a8e\t91438200.000000\tpoverty related diseases\n",
            "64db\t91447792.000000\tprogrammes in place at public hospitals to counter antibiotic resistance\n",
            "9fae\t91486408.000000\tconcussion recovery not clear cut for children\n",
            "90fc\t91490056.000000\tcancer patients, survivors face increased risk of heart disease deaths\n",
            "f0ef\t91516776.000000\tthe genetic signature of memory: unique memory genes are expressed in different areas of the brain\n",
            "3f32\t91518264.000000\tnew bone healing mechanism has potential therapeutic applications\n",
            "1e7a\t91526576.000000\tinadequacies in current early-stage lung cancer treatment revealed\n",
            "f734\t91528904.000000\ttype 2 diabetes: best foods to include in your diet to lower blood sugar\n",
            "c968\t91530720.000000\tdementia study reveals how proteins interact to stop brain signals\n",
            "None\n",
            "\n",
            "Cosine title\n",
            "7a57\t0.341082\tautism news\n",
            "56cf\t0.341082\tautism news\n",
            "1f58\t0.352840\tvaccines news\n",
            "aeb0\t0.422530\tvaccines fast facts\n",
            "ba97\t0.428820\tautism and adhd share genes\n",
            "a6d3\t0.464081\thealth officials warn the measles outbreak is accelerating\n",
            "1fb4\t0.473172\tnew york ends religious exemptions for required vaccines\n",
            "3d00\t0.488002\tscientists uncover structure of key pneumonia virus enzyme\n",
            "1ed7\t0.497164\tafter child deaths, doctors pressure border patrol to let them administer flu vaccines\n",
            "0786\t0.497586\twashington state senate passes bill removing exemption for measles vaccine\n",
            "read more down syndrome, due to an extra chromosome 21, occurs in 250,000 children and adults in the united states, making it the country's most common chromosomal disorder. inherited heart defects, ... read more a research team has found exposure to sound -- not sound reduction -- during early development of mice engineered to have fragile x syndrome, or fxs, restores molecular, cellular, and functional ... read more the annual transition to and from daylight saving time (dst) has clinical implications that last longer than the days where clocks 'fall back' or 'spring ... read more zebrafish research is a promising way to understand the neural and genetic causes of eye movement problems in people, according to multi-university ... read more severe infections during pregnancy have been connected to a range of psychiatric disorders by different studies in humans and animals. now, researchers have shown in mice how the infections affect ... read more a team of psychology researchers is providing one of the first comprehensive literature reviews on arrogance, as well as a way to classify the condition on different levels across a spectrum, ... read more when people become stressed, their bodies can respond by sweating. now, researchers are monitoring how much adolescents severely affected by autism sweat in order to better understand when behavioral ... read more communicating with others is one of the biggest difficulties for autistic children. asking for help, initiating social interaction and sharing their emotions become skills that require learning ... read more researchers have devised a new way to rapidly image synaptic proteins at high resolution. using fluorescent nucleic acid probes, they can label and image as many as 12 different proteins in neuronal ... read more the expression of many genes that have previously been associated with autism is abnormal also in violent psychopathy, a new study shows. the researchers used stem cell technology to analyse the ... read more autism rates among black and hispanic children are catching up to and, in many states, exceeding rates among white youth, who have historically had higher prevalence, a new study ... read more a specific gene associated with autism appears to undergo changes in the sperm of men who use marijuana, according to new research. the gene change occurs through a process called dna methylation, ... read more pivotal response treatment involving parents works better than other existing therapies at motivating children with autism and significant speech delays to talk, according to the results of a large ... read more neurodevelopmental disorders arising from rare genetic mutations can cause atypical cognitive function, intellectual disability, and developmental delays, yet it is unclear why and how this happens. ... read more scientist have identified a link between exposure to high levels of estrogen sex hormones in the womb and the likelihood of developing ... read more neuroscientists have identified a brain region linked to altered social interactions in an autism model. restoring activity of a specific forebrain region in a mouse model also reversed social traits ... read more oxytocin is referred to as the love hormone and is important in the regulation of social and maternal behavior. the oxytocin system in the brain may be key to new treatments for many mental health ... read more for the first time researchers have been able to demonstrate that the brains of a patient and therapist become synchronized during a music therapy session, a breakthrough that could improve future ... read more for the brain to function smoothly, its cells must carefully regulate which proteins are produced and when. by studying gene regulation, researchers are now shedding light on complex brain conditions ...\n",
            "\n",
            "TS-SS title\n",
            "7a57\t323663936.000000\tautism news\n",
            "56cf\t323663936.000000\tautism news\n",
            "1f58\t323742816.000000\tvaccines news\n",
            "aeb0\t324672160.000000\tvaccines fast facts\n",
            "a0e7\t324845504.000000\tarthritis news\n",
            "b35c\t324872960.000000\tschizophrenia news\n",
            "1de1\t324872960.000000\tschizophrenia news\n",
            "4d21\t324879136.000000\tdementia news\n",
            "6f6b\t324903456.000000\tdiabetes news\n",
            "f524\t324903456.000000\tdiabetes news\n",
            "read more down syndrome, due to an extra chromosome 21, occurs in 250,000 children and adults in the united states, making it the country's most common chromosomal disorder. inherited heart defects, ... read more a research team has found exposure to sound -- not sound reduction -- during early development of mice engineered to have fragile x syndrome, or fxs, restores molecular, cellular, and functional ... read more the annual transition to and from daylight saving time (dst) has clinical implications that last longer than the days where clocks 'fall back' or 'spring ... read more zebrafish research is a promising way to understand the neural and genetic causes of eye movement problems in people, according to multi-university ... read more severe infections during pregnancy have been connected to a range of psychiatric disorders by different studies in humans and animals. now, researchers have shown in mice how the infections affect ... read more a team of psychology researchers is providing one of the first comprehensive literature reviews on arrogance, as well as a way to classify the condition on different levels across a spectrum, ... read more when people become stressed, their bodies can respond by sweating. now, researchers are monitoring how much adolescents severely affected by autism sweat in order to better understand when behavioral ... read more communicating with others is one of the biggest difficulties for autistic children. asking for help, initiating social interaction and sharing their emotions become skills that require learning ... read more researchers have devised a new way to rapidly image synaptic proteins at high resolution. using fluorescent nucleic acid probes, they can label and image as many as 12 different proteins in neuronal ... read more the expression of many genes that have previously been associated with autism is abnormal also in violent psychopathy, a new study shows. the researchers used stem cell technology to analyse the ... read more autism rates among black and hispanic children are catching up to and, in many states, exceeding rates among white youth, who have historically had higher prevalence, a new study ... read more a specific gene associated with autism appears to undergo changes in the sperm of men who use marijuana, according to new research. the gene change occurs through a process called dna methylation, ... read more pivotal response treatment involving parents works better than other existing therapies at motivating children with autism and significant speech delays to talk, according to the results of a large ... read more neurodevelopmental disorders arising from rare genetic mutations can cause atypical cognitive function, intellectual disability, and developmental delays, yet it is unclear why and how this happens. ... read more scientist have identified a link between exposure to high levels of estrogen sex hormones in the womb and the likelihood of developing ... read more neuroscientists have identified a brain region linked to altered social interactions in an autism model. restoring activity of a specific forebrain region in a mouse model also reversed social traits ... read more oxytocin is referred to as the love hormone and is important in the regulation of social and maternal behavior. the oxytocin system in the brain may be key to new treatments for many mental health ... read more for the first time researchers have been able to demonstrate that the brains of a patient and therapist become synchronized during a music therapy session, a breakthrough that could improve future ... read more for the brain to function smoothly, its cells must carefully regulate which proteins are produced and when. by studying gene regulation, researchers are now shedding light on complex brain conditions ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HMsyKDEQK_RT",
        "colab_type": "text"
      },
      "source": [
        "Embedding TS-SS title seems conciously aware that it is looking for fact pages without being indicated to do so somehow. Other than, accuracy is pretty low, a lot of other health articles being mixed in. Perhaps the usage of flair embedders, which encode structure of sentences, without one more layer to interpret that properly into the vector, distorts the results because now sentence with similar structure, though different meanings, are evaluated higher.\n",
        "\n",
        "After testing without Flair embeddings, it does not seem to have gotten better. Perhaps finetuning & investigating suitable output layer is necessary. Alternatively, what is word mover's distance was added to this?"
      ]
    }
  ]
}