01_get_captions.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personality and its Transformations ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An analysis of prof. Jordan Peterson's collection of lectures from University of Toronto personality course."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lectures in video form provided at: https://www.youtube.com/watch?v=UGLsnu5RLe8&list=PLYNhvBtnVUK4aUJ6onJbylGBeeqlJUpN1     |    More about prof. Peterson at https://www.jordanbpeterson.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Get captions from a playlist</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# main imports\n",
    "import os\n",
    "from constants import *\n",
    "\n",
    "try:\n",
    "    from pytube import Playlist\n",
    "    from youtube_transcript_api import YouTubeTranscriptApi\n",
    "except ModuleNotFoundError:\n",
    "    os.system('pip install pytube') \n",
    "    os.system('pip install youtube-transcript-api') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***pytube*** is a lightweight, Pythonic, dependency-free, library (and command-line utility) for downloading YouTube Videos.    \n",
    "Documentation available at https://pytube.io/en/latest/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***YouTube Transcript/Subtitle API*** is a python API which allows you to get the transcript/subtitles for a given YouTube video    \n",
    "Documentation available at: https://pypi.org/project/youtube-transcript-api/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_playlist(play_list_name: str) -> list:\n",
    "    \"\"\"Generate list of youtube urls based on a given playlist name\n",
    "\n",
    "    Args:\n",
    "        play_list_name (str): name of a playlist\n",
    "\n",
    "    Returns:\n",
    "        list: Outputs a list of youtube urls\n",
    "    \"\"\"\n",
    "    all_urls = [] # list of all urls\n",
    "    playlist = Playlist(play_list_name)\n",
    "    \n",
    "    for url in playlist:\n",
    "        all_urls.append(url)\n",
    "\n",
    "    return all_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PLAYLIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.youtube.com/watch?v=UGLsnu5RLe8',\n",
       " 'https://www.youtube.com/watch?v=ajtnhtEg76k',\n",
       " 'https://www.youtube.com/watch?v=PH67HpFD2Ew',\n",
       " 'https://www.youtube.com/watch?v=G3fWuMQ5K8I',\n",
       " 'https://www.youtube.com/watch?v=IO6NvcGKZ20',\n",
       " 'https://www.youtube.com/watch?v=BSh37_x5RNY',\n",
       " 'https://www.youtube.com/watch?v=3uJkd54p9dY',\n",
       " 'https://www.youtube.com/watch?v=WjpV9mja3Wc',\n",
       " 'https://www.youtube.com/watch?v=539UQF6eT6I',\n",
       " 'https://www.youtube.com/watch?v=f511uRzsHhQ',\n",
       " 'https://www.youtube.com/watch?v=RNxlEQSvh_w',\n",
       " 'https://www.youtube.com/watch?v=q15eTySnWxc',\n",
       " 'https://www.youtube.com/watch?v=qRFxulvRC7I']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jp_videos = get_playlist(PLAYLIST)\n",
    "jp_videos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(jp_videos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_captions(urls: list) -> list:\n",
    "    \"\"\"Get captions (transcripts) from all videos in a given playlist\n",
    "\n",
    "    Args:\n",
    "        urls (list): list of YouTube video\n",
    "\n",
    "    Returns:\n",
    "        list: list of all captions from all videos\n",
    "    \"\"\"\n",
    "    all_captions = []\n",
    "\n",
    "    for url in urls:\n",
    "        # a shorter version of a given YouTube link\n",
    "        short_url = url.split('watch?v=')[1]\n",
    "        try:\n",
    "            captions = YouTubeTranscriptApi.get_transcript(short_url,\\\n",
    "                 languages=['en'])\n",
    "\n",
    "            preprocessed_captions = ' '.join([entry['text'].replace('\\n', ' ') \\\n",
    "                for entry in captions])\n",
    "\n",
    "            all_captions.append(preprocessed_captions)\n",
    "\n",
    "        # print the short url of a video that doesn't have captions\n",
    "        except:\n",
    "            print(f'Transcripts disabled for video {short_url}')\n",
    "    return all_captions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transcripts disabled for video q15eTySnWxc\n"
     ]
    }
   ],
   "source": [
    "all_captions = get_captions(jp_videos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(all_captions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Well, after all that. So, welcome to Psychology 230. Nice to see you all here. So, what I’m going to do today—how I’m going to start—is I’m going to give you an overview of the content of the course and then I’ll give you an overview of the class requirements right at the end. But I think we might as well jump right into the content to begin with. So, there’s a website—I don’t really like Blackboard so I have my own website. You can go to jordanbpeterson.com and underneath there there’s a menu that lists all the courses and the full syllabus is listed there. So, all the information that you’re going to need about the course can be found there, including most of the readings, although there is also a textbook, which I presume the majority of you have already purchased. So, it also lists the other things you need to know like what days the tests are going to be and what the assignments are and I’ll go over that anyways at the end of the class. So, but to begin with I’m going to tell you '"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# preview of the first 1000  characters of the first caption\n",
    "all_captions[0][:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save captions to a file\n",
    "with open(f'{OUTPUT_FOLDER}\\{CAPTIONS_FILE_NAME}', 'w', encoding='utf-8') as f:\n",
    "    for line in all_captions:\n",
    "        f.write(f\"{line}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "f5dc7221bd57f443bd21d92d893b3958d3081f94c6c945d5f95188cb4cde5b4f"
  },
  "kernelspec": {
   "display_name": "Python 3.7.5 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}