CHAPTER 12 exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CHAPTER 12 exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1: Change the socket program socket1.py to prompt the user for the\n",
    "URL so it can read any web page. You can use split(’/’) to break the URL into\n",
    "its component parts so you can extract the host name for the socket connect call.\n",
    "Add error checking using try and except to handle the condition where the user\n",
    "enters an improperly formatted or non-existent URL.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HTTP/1.1 200 OK\r\n",
      "Date: Sat, 06 Apr 2019 10:54:13 GMT\r\n",
      "Server: Apache/2.4.18 (Ubuntu)\r\n",
      "Last-Modified: Sat, 13 May 2017 11:22:22 GMT\r\n",
      "ETag: \"a7-54f6609245537\"\r\n",
      "Accept-Ranges: bytes\r\n",
      "Content-Length: 167\r\n",
      "Cache-Control: max-age=0, no-cache, no-store, must-revalidate\r\n",
      "Pragma: no-cache\r\n",
      "Expires: Wed, 11 Jan 1984 05:00:00 GMT\r\n",
      "Connection: close\r\n",
      "Content-Type: text/plain\r\n",
      "\r\n",
      "But soft what light through yonder window breaks\n",
      "It is the east and Juliet is the sun\n",
      "Arise fair sun and kill the envious moon\n",
      "Who is already sick and pale with grief\n"
     ]
    }
   ],
   "source": [
    "#socket1.py\n",
    "\n",
    "import socket\n",
    "\n",
    "mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
    "mysock.connect(('data.pr4e.org', 80))\n",
    "cmd = 'GET http://data.pr4e.org/romeo.txt HTTP/1.0\\r\\n\\r\\n'.encode()\n",
    "mysock.send(cmd)\n",
    "while True:\n",
    "    data = mysock.recv(20)\n",
    "    if len(data) < 1:\n",
    "        break\n",
    "    print(data.decode(),end='')\n",
    "    \n",
    "mysock.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter a URL to retrieve data:http://data.pr4e.org/romeo.txt\n",
      "HTTP/1.1 200 OK\n",
      "Date: Sat, 06 Apr 2019 10:54:18 GMT\n",
      "Server: Apache/2.4.18 (Ubuntu)\n",
      "Last-Modified: Sat, 13 May 2017 11:22:22 GMT\n",
      "ETag: \"a7-54f6609245537\"\n",
      "Accept-Ranges: bytes\n",
      "Content-Length: 167\n",
      "Cache-Control: max-age=0, no-cache, no-store, must-revalidate\n",
      "Pragma: no-cache\n",
      "Expires: Wed, 11 Jan 1984 05:00:00 GMT\n",
      "Connection: close\n",
      "Content-Type: text/plain\n",
      "\n",
      "But soft what light through yonder window breaks\n",
      "It is the east and Juliet is the sun\n",
      "Arise fair sun and kill the envious moon\n",
      "Who is already sick and pale with grief\n"
     ]
    }
   ],
   "source": [
    "import socket\n",
    "url = input ('Enter a URL to retrieve data:')\n",
    "surl = url.split('/')\n",
    "\n",
    "try:\n",
    "    server = surl[2]\n",
    "    mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
    "    mysock.connect((str(server), 80))\n",
    "    url = 'GET ' + url + ' HTTP/1.0\\r\\n\\r\\n'\n",
    "    cmd = url.encode()\n",
    "    mysock.send(cmd)\n",
    "    while True:\n",
    "        data = mysock.recv(512)  \n",
    "        if len(data) < 1:\n",
    "            break\n",
    "        print(data.decode(),end='')\n",
    "    \n",
    "    mysock.close()\n",
    "\n",
    "except:\n",
    "    print ('Invalid URL')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2: Change your socket program so that it counts the number of characters\n",
    "it has received and stops displaying any text after it has shown 3000 characters.\n",
    "The program should retrieve the entire document and count the total number of \n",
    "characters and display the count of the number of characters at the end of the\n",
    "document.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter a URL to retrieve data:http://data.pr4e.org/romeo.txt\n",
      "HTTP/1.1 200 OK\n",
      "Date: Sat, 06 Apr 2019 10:54:24 GMT\n",
      "Server: Apache/2.4.18 (Ubuntu)\n",
      "Last-Modified: Sat, 13 May 2017 11:22:22 GMT\n",
      "ETag: \"a7-54f6609245537\"\n",
      "Accept-Ranges: bytes\n",
      "Content-Length: 167\n",
      "Cache-Control: max-age=0, no-cache, no-store, must-revalidate\n",
      "Pragma: no-cache\n",
      "Expires: Wed, 11 Jan 1984 05:00:00 GMT\n",
      "Connection: close\n",
      "Content-Type: text/plain\n",
      "\n",
      "But soft what light through yonder window breaks\n",
      "It is the east and Juliet is the sun\n",
      "Arise fair sun and kill the envious moon\n",
      "Who is already sick and pale with grief\n",
      "\n",
      "536\n"
     ]
    }
   ],
   "source": [
    "import socket\n",
    "url = input ('Enter a URL to retrieve data:')\n",
    "surl = url.split('/')\n",
    "server = surl[2]\n",
    "data_total = ''\n",
    "\n",
    "mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
    "mysock.connect((str(server), 80))\n",
    "url = 'GET ' + url + ' HTTP/1.0\\r\\n\\r\\n'\n",
    "cmd = url.encode()\n",
    "mysock.send(cmd)\n",
    "while True:\n",
    "    data = mysock.recv(512)    \n",
    "    if len(data) < 1:\n",
    "        break\n",
    "    data_total = data_total + str(data.decode())\n",
    "    \n",
    "mysock.close()\n",
    "print(data_total[:3000])\n",
    "print(len(data_total[:3000]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3: Use urllib to replicate the previous exercise of (1) retrieving the\n",
    "document from a URL, (2) displaying up to 3000 characters, and (3) counting the\n",
    "overall number of characters in the document. Don’t worry about the headers for\n",
    "this exercise, simply show the first 3000 characters of the document contents.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please enter a URL to retrieve data:http://data.pr4e.org/romeo.txt\n",
      "But soft what light through yonder window breaks\n",
      "It is the east and Juliet is the sun\n",
      "Arise fair sun and kill the envious moon\n",
      "Who is already sick and pale with grief\n",
      "\n",
      "167\n"
     ]
    }
   ],
   "source": [
    "import urllib.request, urllib.parse, urllib.error\n",
    "url = input ('Please enter a URL to retrieve data:')\n",
    "fhand = urllib.request.urlopen(url)\n",
    "counts = dict()\n",
    "data_total = ''\n",
    "for line in fhand:\n",
    "    words = line.decode()\n",
    "    data_total = data_total + words\n",
    "print (data_total[:3000])\n",
    "print (len(data_total[:3000]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4: Change the urllinks.py program to extract and count paragraph (p)\n",
    "tags from the retrieved HTML document and display the count of the paragraphs\n",
    "as the output of your program. Do not display the paragraph text, only count\n",
    "them. Test your program on several small web pages as well as some larger web\n",
    "pages.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request, urllib.parse, urllib.error\n",
    "from bs4 import BeautifulSoup\n",
    "import ssl\n",
    "\n",
    "ctx = ssl.create_default_context()\n",
    "ctx.check_hostname = False\n",
    "ctx.verify_mode = ssl.CERT_NONE\n",
    "\n",
    "url = input('Enter a URL:')\n",
    "html = urllib.request.urlopen(url, context=ctx).read()\n",
    "soup = BeautifulSoup(html, 'html.parser')\n",
    "\n",
    "tags = soup('p')\n",
    "count = 0\n",
    "for tag in tags:\n",
    "    print('TAG:', tag)\n",
    "    count += 1\n",
    "\n",
    "print (count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5: (Advanced) Change the socket program so that it only shows data\n",
    "after the headers and a blank line have been received. Remember that recv is\n",
    "receiving characters (newlines and all), not lines.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "But soft what light through yonder window breaks\n",
      "It is the east and Juliet is the sun\n",
      "Arise fair sun and kill the envious moon\n",
      "Who is already sick and pale with grief\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import socket\n",
    "\n",
    "mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
    "mysock.connect(('data.pr4e.org', 80))\n",
    "cmd = 'GET http://data.pr4e.org/romeo.txt HTTP/1.0\\r\\n\\r\\n'.encode()\n",
    "mysock.send(cmd)\n",
    "total_bdata = b''\n",
    "while True:\n",
    "    data = mysock.recv(512)\n",
    "    if len(data) < 1:\n",
    "        break\n",
    "\n",
    "    total_bdata = total_bdata + data\n",
    "    \n",
    "total_bdata = total_bdata.split(b\"\\r\\n\\r\\n\")\n",
    "total_bdata = total_bdata [1]\n",
    "total_bdata = total_bdata.decode()\n",
    "print (total_bdata)\n",
    "\n",
    "mysock.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}