Advertools/Advertools_Check_status_code_in_bulk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5da81fdb-1607-4dcd-8945-78492d41c20c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"8%\" alt=\"Advertools.png\" src=\"https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/.github/assets/logos/Advertools.png\" style=\"border-radius: 15%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96049d01-11e9-4538-9012-57992898281e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# Advertools - Check status code in bulk\n",
    "<a href=\"https://bit.ly/3JyWIk6\">Give Feedback</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=Advertools+-+Check+status+code+in+bulk:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b0bd6e-4a5c-472f-a473-a315e3738936",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #advertools #adviz #website #analyze #audit #seo #status_code #response_headers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1089c7e4-b7a9-48ab-8320-e17847ad3b30",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Elias Dabbas](https://www.linkedin.com/in/eliasdabbas/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8913d2a4-4370-4933-bb80-0c6995fa9f0e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Last update:** 2023-07-31 (Created: 2023-07-20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d93cd7a5-1d4f-4bc6-94a2-b74461c7f538",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Description:** This notebook runs an automated status code checker with response headers using the HTTP `HEAD` method.\n",
    "\n",
    "* Bulk and concurrent checking of status codes for a known list of URLs\n",
    "* Get all available response headers from all URLs\n",
    "* Set speed, number of concurent request and various other crawling options\n",
    "* Does NOT download the full HTML of a page, saving a lot of time, energy, and resources, and enabling an extreemely fast and light process"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d84a0cf-6ba8-456a-8b42-1bf63e3a17f2",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**References:**\n",
    "- [advertools `crawl_headers` function](https://advertools.readthedocs.io/en/master/advertools.header_spider.html)\n",
    "- Scrapy [throttling](https://docs.scrapy.org/en/latest/topics/autothrottle.html) and [custom settings](https://docs.scrapy.org/en/latest/topics/settings.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75ef3a2-b261-45a8-bd64-52cb78b5f502",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b54cbe9-d040-4624-a5fb-3d127baed4c5",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Install libraries\n",
    "If running it on naas, run the code below to uninstall (bug) and install the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0ffd4ba7-df94-47b9-87d2-4910edaf1e4b",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip uninstall -y scrapy attrs\n",
    "# !pip install advertools adviz pandas==1.5.3 --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f530c66d-1a5b-47d1-8d80-01f21a83ac3e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1e8d108a-44de-44b4-941f-f12795838c5c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import adviz\n",
    "except ModuleNotFoundError:\n",
    "    !pip install adviz --user\n",
    "    import adviz\n",
    "try:\n",
    "    import advertools as adv\n",
    "except ModuleNotFoundError:\n",
    "    !pip install advertools --user\n",
    "    import advertools as adv\n",
    "from datetime import datetime\n",
    "import plotly.express as px\n",
    "import pandas as pd\n",
    "pd.options.display.max_columns = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d35a9e6f-71c4-4cb2-b9bc-32c23b18c4ba",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup Variables\n",
    "- `url_list`: List of URLs to check the status codes\n",
    "- `CONCURRENT_REQUESTS_PER_DOMAIN`: Defaults to 8. Most likely you will need to slow it down, because the crawling is extremely fast.\n",
    "- `DEFAULT_REQUEST_HEADERS`: A dictionary where you can set custom request headers.\n",
    "- `USER_AGENT`: Set a different/custom user agent if you need to. By default, we are using the one in advertools.\n",
    "- `AUTOTHROTTLE_ENABLED`: Whether or not to dynamically change the pace of crawling to be nice to servers as much as possible. It's usually good to enable this extension, as this type of crawling is extremely fast, often leading to 429 (too many requests code), or being blocked.\n",
    "- `LOG_FILE`: Log file path. Please update it everytime you checked a new list.\n",
    "- `output_file`: The path to the file for saving the output. It has to be in the '.jl' format. Note that new lines to the file are 'appended' to the end and the file is not overwritten while crawling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cebcf1a5-23ce-49ca-aee9-aac7ff28a2e4",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Inputs\n",
    "url_list = [\n",
    "    \"https://www.naas.ai/\",\n",
    "    \"https://www.naas.ai/free-forever\",\n",
    "    \"https://app.naas.ai/user/florent@naas.ai\",\n",
    "    \"https://www.naas.ai/test\",\n",
    "]\n",
    "domain = 'naas.ai'\n",
    "CONCURRENT_REQUESTS_PER_DOMAIN = 4\n",
    "DEFAULT_REQUEST_HEADERS = {}\n",
    "USER_AGENT = adv.spider.user_agent\n",
    "AUTOTHROTTLE_ENABLED = True\n",
    "LOG_FILE = 'crawl_headers_output.log'\n",
    "\n",
    "# Outputs\n",
    "output_file = f'{datetime.now().strftime(\"%Y%m%d%H%M%S\")}_crawl_headers_output.jl'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ad6e918-ae61-4a78-9b71-a6ad5f530f53",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dafe9992-9fcb-4849-a69e-ced13c995aff",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Check status codes and retrieve response headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2393668d-9f22-45a0-9f79-5901b1c88496",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create custom settings dict\n",
    "custom_settings = {\n",
    "    # optionally change request headers:\n",
    "    'DEFAULT_REQUEST_HEADERS': DEFAULT_REQUEST_HEADERS,\n",
    "    'AUTOTHROTTLE_ENABLED': AUTOTHROTTLE_ENABLED,\n",
    "    'CONCURRENT_REQUESTS_PER_DOMAIN': CONCURRENT_REQUESTS_PER_DOMAIN,\n",
    "    'USER_AGENT': USER_AGENT,\n",
    "    'LOG_FILE': LOG_FILE,\n",
    "}\n",
    "\n",
    "# Crawl headers\n",
    "adv.crawl_headers(\n",
    "    url_list=url_list,\n",
    "    output_file=output_file,\n",
    "    custom_settings=custom_settings\n",
    ")\n",
    "\n",
    "# Open the crawl output file\n",
    "headers_df = pd.read_json(output_file, lines=True)\n",
    "headers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "550110a8-50b2-49a0-9752-5132256356b9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd817594-1b7f-4d02-a245-9e1dcd6ad2b6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Visualize Status codes OK and KO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5839cc92-18ea-4a57-b9bb-5fa8d0db279c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "adviz.status_codes(\n",
    "    headers_df['status'],\n",
    "    height=400\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bab2898f-1c33-452e-b73c-bd1a28290f48",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-18T18:40:54.104130Z",
     "iopub.status.busy": "2023-07-18T18:40:54.103899Z",
     "iopub.status.idle": "2023-07-18T18:40:54.108578Z",
     "shell.execute_reply": "2023-07-18T18:40:54.108020Z",
     "shell.execute_reply.started": "2023-07-18T18:40:54.104107Z"
    },
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Visualize status codes count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "552d3abb-4e7c-4c0e-8e68-5777ac8cc24b",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "adviz.value_counts_plus(\n",
    "    headers_df['status'],\n",
    "    name='Status codes',\n",
    "    size=14\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50a39376-9dab-4f3e-a87d-6006a61165ff",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Visualize URL structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0d536efd-9d10-480c-abdd-0353d6f38c95",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "adviz.url_structure(\n",
    "    headers_df['url'],\n",
    "    domain=domain,\n",
    "    items_per_level=50,\n",
    "    theme='plotly_dark'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7cb3852-bc53-4f24-83fb-65fbe3d42764",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get redirects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3132e337-c431-4145-8ba1-f58b08a0cea5",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "if 'redirect_urls' in headers_df:\n",
    "    redirect_df = headers_df.filter(regex='^url$|redirect_').dropna(thresh=4)\n",
    "    redirect_df['redirect_urls'] = redirect_df['redirect_urls'].str.split('@@')\n",
    "    redirect_df['redirect_reasons'] = redirect_df['redirect_reasons'].astype(int).astype(str).str.split('@@')\n",
    "    redirect_df['redirect_chain'] = (redirect_df['redirect_urls']\n",
    "                                     .str.join('@@')\n",
    "                                     .add('@@')\n",
    "                                     .add(redirect_df['url']).str.split('@@'))\n",
    "    from IPython.display import display\n",
    "    display(redirect_df)\n",
    "else:\n",
    "    print('No redirects found in this dataset')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea195054-23c4-498a-b75e-9929270d89d9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Visualize download latency\n",
    "The same charts can be used for any other `float` columns in `headers_df` if available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0b13dd7e-d0a5-44da-9032-6bc0e2255425",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "fig = px.histogram(\n",
    "    x=headers_df['download_latency'],\n",
    "    template='none',\n",
    "    labels={'x': 'Latency (seconds)<br>'},\n",
    "    height=500,\n",
    "    # experiment with different values for nbins:\n",
    "    # nbins=10,\n",
    "    title='<b>Download latency distribution</b>')\n",
    "\n",
    "fig.add_annotation(\n",
    "    yref='y domain',\n",
    "    xref='paper',\n",
    "    x=0,\n",
    "    y=-0.25,\n",
    "    showarrow=False,\n",
    "    align='right',\n",
    "    text='<b>Latency:</b> time elapsed between establishing the TCP connection and receiving the HTTP headers.')\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eeab37b-5de9-42f9-b0e8-edf43ccbe62c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Create helper visualizations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "55e5e098-20dc-47fc-b8fe-fd169e8e2c01",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def ecdf(df, column, template='none', width=None, height=500):\n",
    "    df = df.sort_values(column)\n",
    "    df['count below'] = range(len(df))\n",
    "    df['count above'] = range(len(df)-1, -1, -1)\n",
    "    df['total count'] = len(df)\n",
    "    fig = px.ecdf(\n",
    "        df.dropna(subset=[column]),\n",
    "        x=column,\n",
    "        markers=True, lines=False,\n",
    "        template=template,\n",
    "        height=height,\n",
    "        width=width,\n",
    "        hover_data=['count below', 'count above', 'total count'],\n",
    "        title=f\"Cumulative distribution of <b>{column.replace('_', ' ')}</b>\",\n",
    "        ecdfnorm='percent')\n",
    "    fig.data[0].hovertemplate = f'<b>{column.replace(\"_\", \" \").title()}</b><br><br>{column}' + ': %{x}<br>percent: %{y}<br><br>count below: %{customdata[0]:,}<br>count above: %{customdata[1]:,}<br>total count: %{customdata[2]:,}<extra></extra>'\n",
    "    fig.layout.yaxis.ticksuffix = '%'\n",
    "    fig.layout.yaxis.showspikes = True\n",
    "    fig.layout.xaxis.showspikes = True\n",
    "    return fig\n",
    "\n",
    "ecdf(df=headers_df, column='download_latency')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "naas": {
   "notebook_id": "3c5612a68165853c27d59d0068f4ee37bc3ca2172bb0a1383abe8544664b9524",
   "notebook_path": "Advertools/Advertools_Check_status_code_in_bulk.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.4.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}