OSPemul.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Social Platform Emulator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## authors: A. Giovanidis, B. Baynat, C. Magnien, A. Vendeville"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "08 mai 2020"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The emulator uses a tweeter trace as input. It calculates the percentage of time that posts of a specific origin occupy the first position on a Wall of another user. This is defined as the influence Q[user][author]. By finding all these values, we can derive the Psi-score for all users in the trace. Note here that for the emulator, we just use the direct information of what posts enter a Wall, at what time, and when the next post entering pushes the current post from its top position. In other words, occupancy periods divided by the total time period of the trace determine the influence. No further information on the social graph is necessary.\n",
    "\n",
    "Users are anonymised treated as a number. Tweetids are also anonymised and the content is removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a twitter trace as input with the following Data Format in \"TextTweet\"\n",
    "\n",
    "tweetid timestamp userid retweetid\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = 1 # Wall list size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wall is finite for all users. We apply FIFO replacement principle. The Newsfeed is not needed, because we have no information on the policy of the OSP or the user policy to choose among many. We know: 1) which post is tweeted-retweeted and the time-stamp, and the tweet origin.\n",
    "\n",
    "For each user we create a Wall list of max size $K$.\n",
    "\n",
    "We use $K=1$, to find the frequency of posts of some origin, \n",
    "occupying the most recent position of the user's Wall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "Wall = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** retweetid, is the tweetid from the origin . Leader information is lost. This means that when a user (userid) retweets some tweet at time (timestamp), then the new produced tweet takes the id (tweetid) and the trace informs us that the origin of this tweet is (retweetid). Then we can trace back from the trace the author of this original tweetid, but we do not know the id of the user-leader from whom the tweet was retweeted/shared. This brings essentially problems in the creation of the user leader-follower graph, but not in the calculation of user influence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3754043\n",
      "405236\n"
     ]
    }
   ],
   "source": [
    "# Open file containing the origin\n",
    "# Count the number of tweets (tc) contained in trace. \n",
    "f = open(\"/Users/Fishbone/Desktop/NEWSFEEDfresh/PYTHON/NewResults/PsiEmul.tronc.txt\")\n",
    "tc = 0\n",
    "tc0 = 0\n",
    "for lign in f:\n",
    "    lign = lign.split()\n",
    "    tc +=1\n",
    "    l1 = float(lign[1])\n",
    "    if l1!=0:\n",
    "        tc0+=1\n",
    "f.close()\n",
    "print(tc)\n",
    "print(tc0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5804715\n",
      "405236\n"
     ]
    }
   ],
   "source": [
    "# The trace is sorted in increasing order of timestamps\n",
    "f = open(\"/Users/Fishbone/Desktop/NEWSFEEDfresh/PYTHON/NewResults/Psi.tronque.sorted22gr.0.5804715.txt\")\n",
    "tc = 0\n",
    "tc0 = 0\n",
    "for lign in f:\n",
    "    lign = lign.split()\n",
    "    tc +=1\n",
    "    l1 = float(lign[1])\n",
    "    if l1!=0:\n",
    "        tc0+=1\n",
    "f.close()\n",
    "print(tc)\n",
    "print(tc0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#f = open(\"TextTweet.txt\")\n",
    "f = open(\"tweets.parsed.sorted22n.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create dictionary for Authors\n",
    "Author = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assign each tweetid to its author and fill in the dictionary\n",
    "for lign in f:\n",
    "    lign = lign.split()\n",
    "    tweetid = int(lign[0])\n",
    "    userid = int(lign[2])\n",
    "    Author[tweetid] = userid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Author"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27916100\n"
     ]
    }
   ],
   "source": [
    "# Number of authors\n",
    "print(len(Author))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n"
     ]
    }
   ],
   "source": [
    "# Some users are not included in the Authors\n",
    "print(74400 in Author)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert 74400 not in Author"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We introduce a dictionary Q, which contains many dictionaries, each related to a userid. $Q[userid][auth-out]$ saves the time periods that a post from $[auth-out]$ stays on the Wall of $[userid]$. Since the Wall is of size $K=1$ we actually count the period that a post of origin (auth-out) stays on the first position of the Wall of (userid), before being pushed to a lower position by a new post."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q = {}\n",
    "FirstP = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Important!**\n",
    "\n",
    "if the tweet is original then the person who posted is the author.\n",
    "    else it is a retweet. If the original post was observed and saved in \"Authors\",\n",
    "   we know the author.\n",
    "   else the retweet has origin outside the dataset. In this case we assume that\n",
    "   every user with this retweet is considered as the original author."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'tweets.parsed.sorted22n.txt'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-47a53d8949d2>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#f = open(\"TextTweet.txt\")\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mAmphitrions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"tweets.parsed.sorted22n.txt\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mnb_neg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m# debug\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'tweets.parsed.sorted22n.txt'"
     ]
    }
   ],
   "source": [
    "#f = open(\"TextTweet.txt\")\n",
    "Amphitrions = {}\n",
    "f = open(\"tweets.parsed.sorted22n.txt\")\n",
    "nb_neg = 0\n",
    "# debug\n",
    "i=0\n",
    "j=0\n",
    "# Time0 marks the beginning of time in the trace\n",
    "Time0 = None\n",
    "timestamp_old = 0\n",
    "count_ties = 0\n",
    "for lign in f:\n",
    "    #if j<6:\n",
    "    #    j+=1\n",
    "    #    print(\"original=\",lign)\n",
    "    lign = lign.split()\n",
    "    tweetid = int(lign[0])\n",
    "    tstamp = int(lign[1])\n",
    "    userid = int(lign[2])\n",
    "    rtid = int(lign[3])  \n",
    "    if Time0 ==None:\n",
    "        Time0 = tstamp\n",
    "    if tstamp == timestamp_old:\n",
    "        count_ties+=1\n",
    "    timestamp_old = tstamp\n",
    "    if userid not in Amphitrions:\n",
    "        Amphitrions[userid] = {}\n",
    "    #if j<6:\n",
    "    #    print(\"split=\",tweetid, tstamp, userid, rtid)  \n",
    "    # if the tweet is original then the person who posted is the author.\n",
    "    # else it is a retweet. If the original post was observed and saved in \"Authors\",\n",
    "    # we know the author.\n",
    "    # else the retweet has origin outside the dataset. In this case we assume that\n",
    "    # the first retweeter is the original author.\n",
    "    if rtid == -1:\n",
    "        auth = userid\n",
    "    elif rtid in Author:\n",
    "        auth = Author[rtid] # it is the tweet origin ID\n",
    "    else:\n",
    "        assert rtid not in Author\n",
    "        auth = userid\n",
    "        Author[tweetid] = userid\n",
    "    if userid not in Wall:\n",
    "        Wall[userid] = []\n",
    "        Q[userid] = {}\n",
    "    if len(Wall[userid])==0:\n",
    "        FirstP[userid] = tstamp\n",
    "    Wall[userid].append((tweetid,tstamp,auth))\n",
    "    if len(Wall[userid])>K:\n",
    "        auth_out = Wall[userid][0][2]\n",
    "        # Use time_wall to count the period that a post stays at position K=1, \n",
    "        # or find time_wall =0 to identify simultaneous posts with same timestamp\n",
    "        time_wall = tstamp-Wall[userid][0][1]\n",
    "        #if time_wall < 0:\n",
    "        #    nb_neg += 1\n",
    "        #    # debug\n",
    "        #    if i<6:\n",
    "        #        i+=1\n",
    "        #        #print(userid, tstamp)\n",
    "        #if userid not in Q:\n",
    "        #    Q[userid] = {}\n",
    "        if auth_out not in Q[userid]:\n",
    "            Q[userid][auth_out] = 0\n",
    "        # CORRECTION\n",
    "        #Q[userid][auth_out] += time_wall\n",
    "        #\n",
    "        # CORRECTION\n",
    "        # In our data-sets we found many tweets with the same time-stamp, so we need to account for this as well.\n",
    "        # The issue arises when the same author posts or retweets instantly several tweets (e.g. a bot).\n",
    "        # In this case we assume that the posts that simultaneously enter a Wall will split uniformly among them the\n",
    "        # occupancy time till the next arrival of a post at a fresh time-stamp.\n",
    "        if time_wall == 0:\n",
    "            if auth_out not in Amphitrions[userid]:\n",
    "                Amphitrions[userid][auth_out] = 0 \n",
    "            Amphitrions[userid][auth_out] += 1\n",
    "        else: \n",
    "            if auth_out not in Amphitrions[userid]:\n",
    "                Amphitrions[userid][auth_out] = 0 \n",
    "            Amphitrions[userid][auth_out] += 1\n",
    "            Ndt = 0\n",
    "            for author in Amphitrions[userid]:\n",
    "                Ndt += Amphitrions[userid][author]\n",
    "            for author in Amphitrions[userid]:\n",
    "                Q[userid][author] += time_wall/Ndt*Amphitrions[userid][author]\n",
    "            Amphitrions[userid] = {}\n",
    "        #### END of CORRECTION\n",
    "        #if userid == 3474:\n",
    "        #    print(auth_out, time_wall)\n",
    "        Wall[userid] = Wall[userid][1:]\n",
    "        assert( len(Wall[userid]) == K )\n",
    "#print(nb_neg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The number of entries with clone time-stamp found in our trace\n",
    "print(count_ties)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1398895201\n"
     ]
    }
   ],
   "source": [
    "print(Time0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23889642"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "27916100-4026458"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27916100\n"
     ]
    }
   ],
   "source": [
    "print(len(Author))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(Wall,'\\n')\n",
    "#print('Q=',Q)\n",
    "#print('FirstP=',FirstP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note1:** EndPoint of simulation is the time of last arrival."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1406843993\n"
     ]
    }
   ],
   "source": [
    "EndP = tstamp\n",
    "print(EndP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the parsing of the trace, there are still posts that stay on the place $K=1$ of all Walls. \n",
    "We need to count their part in the influence, so we need to calculate the period they occupy this post, \n",
    "from the time instant they entered the first Wall position, till the end of the simulation (because they have not \n",
    "been ejected by some other post)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note2:** Post-processing: Treat extra users with Wall content less than $K$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "for u in Wall:\n",
    "    if len(Wall[u])>0:\n",
    "        auth_out = Wall[u][0][2]\n",
    "        time_wall = EndP-Wall[u][0][1]\n",
    "        if u not in Q:\n",
    "            Q[u] = {}\n",
    "        if auth_out not in Q[u]:\n",
    "            Q[u][auth_out] = 0\n",
    "        #Q[u][auth_out] += time_wall\n",
    "        # CORRECTION\n",
    "        if time_wall == 0:\n",
    "            if auth_out not in Amphitrions[u]:\n",
    "                Amphitrions[u][auth_out] = 0 \n",
    "            Amphitrions[u][auth_out] += 1\n",
    "        else: \n",
    "            if auth_out not in Amphitrions[u]:\n",
    "                Amphitrions[u][auth_out] = 0 \n",
    "            Amphitrions[u][auth_out] += 1\n",
    "            Ndt = 0\n",
    "            for author in Amphitrions[u]:\n",
    "                Ndt += Amphitrions[u][author]\n",
    "            for author in Amphitrions[u]:\n",
    "                # Q[u][author] += time_wall/Ndt*Amphitrions[u][author]\n",
    "                ## CORRECTION 2: circularite\n",
    "                Q[u][author] += (time_wall+FirstP[u]-Time0)/Ndt*Amphitrions[u][author]\n",
    "            Amphitrions[u] = {}\n",
    "        #### END of CORRECTION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(Q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of what has been saved in entry (userid=3474). There are posts from 7 users that entered the Wall\n",
    "and for each user-origin, the dictionary gives the period that their posts occupied position 1 of the wall of user 3474 cumulatively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{2083: 6.5,\n",
       " 327: 6.5,\n",
       " 408: 114161.0,\n",
       " 3474: 926459.0,\n",
       " 2660: 666122.0,\n",
       " 16559: 5431010.0,\n",
       " 23742: 811027.0}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q[3474]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{2083: 6.5,\n",
       " 327: 6.5,\n",
       " 408: 114161.0,\n",
       " 3474: 926459.0,\n",
       " 2660: 666122.0,\n",
       " 16559: 5431010.0,\n",
       " 23742: 811027.0}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q[3474]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Estimated Qs**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**EstQ[user][leader]:** is the proportion of time that posts of \"leader\" is on 1-st position of \"user\" Wall. Hence: sum{leaders}EstQ[user][leader] = 1. \n",
    "\n",
    "To find these proportions we just need to divide the occupancy time by the total period (EndP-Time0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "EstQ = {}\n",
    "nb0 = 0\n",
    "for u in Q:\n",
    "    EstQ[u] = {}\n",
    "    for j in Q[u]:\n",
    "        if EndP-FirstP[u] == 0:\n",
    "            nb0 += 1\n",
    "            assert len(Q[u]) == 1\n",
    "            EstQ.pop(u)\n",
    "        else:\n",
    "            #EstQ[u][j] = Q[u][j]/(EndP-FirstP[u])\n",
    "            # CORRECTION 2: circularity\n",
    "            EstQ[u][j] = Q[u][j]/(EndP-Time0)\n",
    "#print(EstQ)\n",
    "print(nb0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6020228\n"
     ]
    }
   ],
   "source": [
    "N = len(EstQ)\n",
    "print(N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a test. For user=2500 there are posts from 2 users on his Wall: 33.85% of time these were self-posts of origin  user=2500 and 66.15% of time these were reposts of origin user=1548. These sum up to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1548: 0.6614825246402221, 2500: 0.33851747535977794}\n"
     ]
    }
   ],
   "source": [
    "testU =2500\n",
    "print(EstQ[testU])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "Psi = 0\n",
    "for user in EstQ[testU]:\n",
    "    Psi+=EstQ[testU][user]\n",
    "print(Psi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export as .py\n",
    "#jupyter nbconvert --to script OSPemul.ipynb\n",
    "#through command line convert to .py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculate Influence of user on his followers**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have found for each user Wall the percentage of time that posts of different origin occupy the first position on his Wall. To calculate user Influence, we need to use this information and find the various time periods that posts for a specific user occupy the first position of Walls of all other users. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "Influence = {}\n",
    "for follower in EstQ:\n",
    "    for leader in EstQ[follower]:\n",
    "        if leader not in Influence:\n",
    "            Influence[leader] = {}\n",
    "        Influence[leader][follower] = EstQ[follower][leader]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the Influence of user 158410 is a dictionary.  This user influences 5 users, e.g. his posts have occupied 10.5% of the total time of user's 17700 wall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{17700: 0.10512050132900698, 158410: 0.9339905988230665, 172286: 0.26822465602320456, 444382: 5.032211183787424e-07, 1876907: 0.10557893576785}\n"
     ]
    }
   ],
   "source": [
    "print(Influence[158410])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, user 5 influences 4 users. Specifically his posts occupy 100% of time of his Wall, which means that this user does not repost. But they also occupy 100% of time the Wall of user 1816833 which means that this user only reposts posts of origin 5. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{5: 1.0, 10728: 0.6129629508483805, 57295: 0.7196136217930976, 1816833: 1.0}\n"
     ]
    }
   ],
   "source": [
    "print(Influence[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Psi-score is calculated by the trace directly, by adding the influence from a user to all others and then dividing by (N-1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "Psi = {}\n",
    "for user in Influence:\n",
    "    Psi[user] = 0\n",
    "    for follower in Influence[user]:\n",
    "        if follower != user:\n",
    "            Psi[user]+=Influence[user][follower]\n",
    "    Psi[user] = Psi[user]/(N-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, the Psi-score of user 158410 on the platform is the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.955258104738906e-08"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Psi[158410]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.305334665951965e-08"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Psi[1108115]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.725805124780818e-12"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Psi[21586] #4138, 15698, 17896, 19678, 20116, 20198, 20374, 20376, 21586"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.555219818877829e-06"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Psi[266]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.018266619353711842"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Psi[11604]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write the results to external file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip\n",
    "f = open('PsiCorrCirc20190208','w')\n",
    "for u in Psi:\n",
    "    f.write(\"%d %g\\n\"%(u, Psi[u]))\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import gzip\n",
    "#f = open('Psi','w')\n",
    "#for u in Psi:\n",
    "#    f.write(\"%d %g\\n\"%(u, Psi[u]))\n",
    "#f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## **Part II: Extracting Info from data to feed the analysis**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use the trace to extract information to be used for the model and the analytical formula, rather than the emulator. The information we need is: the rate of tweets and re-tweets per user, as well as the leader graph. We use Ntweet and Nrtweet to count the number of tweets and re-tweets of users during the trace. The Ntweet and Nrtweet are both dictionaries, having as keys the author information.\n",
    "\n",
    "For the LeadGraph, which is also a dictionary, we note for each user all his leaders. Here we use the Star-graph approach, based on which a user x is assumed to follow another user y , if he has retweeted a post of origin y. Note that this approach gives rise to a star-graph, with a user having several leaders, and direct influence without paths of several hops. The reason we take such approach is that we do not have any additional information about the user from whom a post is re-tweeted. Only the post origin. We would not need to go for such an approach in case we knew the real friendship graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "Ntweet = {}\n",
    "Nrtweet = {}\n",
    "LeadGraph = {}\n",
    "FirstT = None\n",
    "LastT = None\n",
    "el=0\n",
    "el2=0\n",
    "f = open(\"tweets.parsed.sorted22n.txt\")\n",
    "for lign in f:\n",
    "    lign = lign.split()\n",
    "    tstamp = int(lign[1])\n",
    "    userid = int(lign[2])\n",
    "    rtid = int(lign[3])\n",
    "    #if el<21:\n",
    "    #    print(userid, rtid)\n",
    "    #    el+=1\n",
    "    if FirstT == None:\n",
    "        FirstT = tstamp\n",
    "    if userid not in Ntweet:\n",
    "        Ntweet[userid] = 0\n",
    "        Nrtweet[userid] = 0\n",
    "        LeadGraph[userid] = set() \n",
    "\n",
    "        #If the retweetid is -1 this indicates a self-post\n",
    "    if rtid == -1:\n",
    "        Ntweet[userid] += 1\n",
    "    else: \n",
    "        if rtid in Author:\n",
    "            LeadGraph[userid].add(Author[rtid])\n",
    "            Nrtweet[userid] += 1\n",
    "        else:\n",
    "            Ntweet[userid] += 1\n",
    "LastT = tstamp\n",
    "f.close()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The Rtweet and Rrtweet are dictionaries which contain the tweet and retweet rates. One simply needs to divide the \n",
    "# count from Ntweet and Nrtweet by the total trace period (LastT-FirstT)\n",
    "Rtweet = {}\n",
    "Rrtweet = {}\n",
    "for user in Ntweet:\n",
    "    Rtweet[user] = Ntweet[user]/(LastT-FirstT)\n",
    "    Rrtweet[user] = Nrtweet[user]/(LastT-FirstT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1142"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Ntweet[11604]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3039 T= 3.887383139475785e-05 R= 0.0\n",
      "set() \n",
      "\n",
      "790 T= 6.54187453892365e-06 R= 0.0\n",
      "set() \n",
      "\n",
      "4806 T= 8.30314845324925e-06 R= 0.0\n",
      "set() \n",
      "\n",
      "175 T= 8.164762645695094e-05 R= 0.0\n",
      "set() \n",
      "\n",
      "6 T= 0.0004423313630549145 R= 0.0\n",
      "set() \n",
      "\n",
      "831 T= 0.00012253434232522377 R= 0.0\n",
      "set() \n",
      "\n",
      "25 T= 0.0004180509440931402 R= 0.0\n",
      "set() \n",
      "\n",
      "374 T= 5.459949134409354e-05 R= 0.0\n",
      "set() \n",
      "\n",
      "2646 T= 1.1322475163521703e-05 R= 0.0\n",
      "set() \n",
      "\n",
      "250 T= 9.699587056750258e-05 R= 0.0\n",
      "set() \n",
      "\n",
      "194 T= 0.0010933736849574123 R= 0.0\n",
      "set() \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# An example of the result for the 10 first users.\n",
    "k = 0\n",
    "for user in Rtweet:\n",
    "    print(user, 'T=', Rtweet[user],'R=', Rrtweet[user])\n",
    "    print(LeadGraph[user],'\\n')\n",
    "    if k==10:\n",
    "        break\n",
    "    else: \n",
    "        k+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6020228"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(Rtweet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.1386897531096547e-06"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Rtweet[17700]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3416701"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([0 for u in Rtweet if Rtweet[u]==0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#jupyter nbconvert --to script OSPemul.ipynb\n",
    "#through command line convert to .py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}