python/use-cases/Single_Name_Screen_NucleusAPI_Use_Case.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><center>  Single Names' Screen - Nucleus APIs Use Cases</center></h1>\n",
    "\n",
    "\n",
    "<h1><center>  SumUp Analytics, Proprietary & Confidential</center></h1>\n",
    "<h1><center>  Disclaimers and Terms of Service available at www.sumup.ai</center></h1>\n",
    "\n",
    "\n",
    "#  \n",
    "\n",
    " \n",
    "\n",
    "\n",
    "## Objective: \n",
    "-\tDevelop a ranking of single name equities using content published by corporations\n",
    "\n",
    "\n",
    "## Data:\n",
    "-\tA chosen list of corporations, for instance within the same industry sector, or with similar market capitalization\n",
    " - \tCompany reports such as SEC filings\n",
    " - \tPress releases\n",
    " - \tEarning call transcripts\n",
    "\n",
    "\n",
    "\n",
    "## Nucleus APIs used:\n",
    "-\tDataset creation API\n",
    " - \t*api_instance.post_upload_file(file, dataset)*\n",
    " - \t*nucleus_helper.import_files(api_instance, dataset, file_iters, processes=1)*\n",
    "\n",
    "        nucleus_helper.import_files leverages api_instance.post_upload_file with parallel execution to speed-up the dataset creation\n",
    "\n",
    "\n",
    "-\tTopic Modeling API\n",
    " - \t*api_instance.post_topic_api(payload)*\n",
    "\n",
    "\n",
    "-\tTopic Sentiment API\n",
    " - \t*api_instance.post_topic_sentiment_api(payload)*\n",
    "\n",
    "\n",
    "-\tDocInfo API\n",
    " - \t*api_instance.post_doc_info(payload)*\n",
    "\n",
    "\n",
    "-\tDatasetInfo API\n",
    " - \t*api_instance.post_dataset_info(payload)*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach:\n",
    "\n",
    "### 1.\tDataset Preparation\n",
    "-\tCreate a Nucleus dataset containing all relevant documents over a chosen historical period\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import json\n",
    "import numpy as np\n",
    "import datetime\n",
    "import regex as re\n",
    "import nucleus_api.api.nucleus_api as nucleus_helper\n",
    "import nucleus_api\n",
    "from nucleus_api.rest import ApiException\n",
    "\n",
    "configuration = nucleus_api.Configuration()\n",
    "configuration.host = 'UPDATE-WITH-API-SERVER-HOSTNAME'\n",
    "configuration.api_key['x-api-key'] = 'UPDATE-WITH-API-KEY'\n",
    "\n",
    "# Create API instance\n",
    "api_instance = nucleus_api.NucleusApi(nucleus_api.ApiClient(configuration))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('--------- Append all files from local folder to dataset in parallel -----------')\n",
    "folder = 'Corporate_documents'         \n",
    "dataset = 'Corporate_docs'# str | Destination dataset where the file will be inserted.\n",
    "\n",
    "# build file iterable from a folder recursively. \n",
    "# Each item in the iterable is in the format below:\n",
    "# {'filename': filename,   # filename to be uploaded. REQUIRED\n",
    "#  'metadata': {           # metadata for the file. Optional\n",
    "#      'key1': val1,       # keys can have arbiturary names as long as the names only\n",
    "#      'key2': val2        # contain alphanumeric (0-9|a-z|A-Z) and underscore (_)\n",
    "#   } \n",
    "# }\n",
    "file_iter = []\n",
    "for root, dirs, files in os.walk(folder):\n",
    "    for file in files:\n",
    "        if Path(file).suffix == '.pdf': # .txt .doc .docx .rtf .html .csv also supported\n",
    "            file_dict = {'filename': os.path.join(root, file),\n",
    "                         'metadata': {'ticker': 'AAPL',\n",
    "                                      'company': 'Apple',\n",
    "                                      'category': 'Earning Calls',\n",
    "                                      'date': '2019-01-01'}}\n",
    "            file_iter.append(file_dict)\n",
    "\n",
    "file_props = nucleus_helper.upload_files(api_instance, dataset, file_iter, processes=4)\n",
    "for fp in file_props:\n",
    "    print(fp.filename, '(', fp.size, 'bytes) has been added to dataset', dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-\tAn alternative using SEC filings and the embedded Nucleus datafeed "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = \"Corporate_docs\" \n",
    "period_start = \"2010-01-01\" \n",
    "period_end= \"2019-06-01\"\n",
    "\n",
    "payload = nucleus_api.EdgarQuery(destination_dataset=dataset,\n",
    "                                tickers=[\"FB\", \"AMZN\", \"INTL\", \"IBM\", \"NFLX\", \"GOOG\"], \n",
    "                                filing_types=[\"10-K\", \"10-K/A\", \"10-Q\", \"10-Q/A\"], \n",
    "                                sections=[\"Quantitative and Qualitative Disclosures about Market Risk\",\n",
    "                                          \"Management's Discussion and Analysis of Financial Condition and Results of Operations\",\n",
    "                                          \"Risk Factors\"],\n",
    "                                period_start=period_start,\n",
    "                                period_end=period_end)\n",
    "\n",
    "api_response = api_instance.post_create_dataset_from_sec_filings(payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You can subsequently work on specific time periods within your dataset directly in the APIs, as illustrated below**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.\tSentiment and Topic Contribution = Screen Analysis\n",
    "- Identify and Extract key topics at a given point in time on the subset of documents at that date \n",
    "\n",
    "\n",
    "- Measure the sentiment on each topic to classify all key topics into ‘good’ and ‘bad’ topics\n",
    "\n",
    "\n",
    "- Determine the exposure of each company to each topic\n",
    "\n",
    "\n",
    "- Aggregate the exposures of a given company across key topics based on the ‘good’ or ‘bad’ nature of the topics, to derive a ranking of the companies\n",
    " - The top company is the one with the most exposure to good topics and/or the least exposure to bad topics\n",
    " \n",
    " \n",
    "- Further down, we discuss how to refine this analysis by leveraging the different parameters available to the user\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determine which companies are associated to the documents contributing to the topics\n",
    "payload = nucleus_api.DocInfo(dataset='Corporate_docs')\n",
    "api_response = api_instance.post_doc_info(payload)\n",
    "\n",
    "company_sources = []\n",
    "for res in api_response.result:        \n",
    "    company_sources.append(res.attribute['ticker']) \n",
    "\n",
    "company_list = np.unique(company_sources)\n",
    "\n",
    "\n",
    "print('-------- Get topic sentiment and exposure per firm ----------------')\n",
    "\n",
    "payload = nucleus_api.TopicSentimentModel(dataset='Corporate_docs',          \n",
    "                                query='interest rates',                   \n",
    "                                num_topics=20,\n",
    "                                num_keywords=8)\n",
    "try:\n",
    "    api_response = api_instance.post_topic_sentiment_api(payload)    \n",
    "    api_ok = True\n",
    "except ApiException as e:\n",
    "    api_error = json.loads(e.body)\n",
    "    print('ERROR:', api_error['message'])\n",
    "    api_ok = False\n",
    "\n",
    "if api_ok:\n",
    "    company_rankings = np.zeros(len(company_list), len(enumerate(api_response.result))\n",
    "    for i, res in enumerate(api_response.result):\n",
    "        print('Topic', i, 'sentiment:')\n",
    "        print('    Keywords:', res.keywords)\n",
    "\n",
    "        # Aggregate all document exposures within a topic into a company exposure, using the dataset metadata\n",
    "        payload = nucleus_api.DocInfo(dataset='Corporate_docs', doc_ids = res.doc_ids)\n",
    "        try:\n",
    "            api_response1 = api_instance.post_doc_info(payload)\n",
    "            api_ok = True\n",
    "        except ApiException as e:\n",
    "            api_error = json.loads(e.body)\n",
    "            print('ERROR:', api_error['message'])\n",
    "            api_ok = False\n",
    "\n",
    "        if api_ok:\n",
    "            company_sources = [] # This list will be much shorter than the whole dataset because not all documents contribute to a given topic\n",
    "            for res1 in api_response1.result:        \n",
    "                company_sources.append(res1.attribute['ticker']) \n",
    "\n",
    "            company_contributions = np.zeros([len(company_list), 1])\n",
    "            for j in range(len(company_list)):\n",
    "                for k in range(len(company_sources)):\n",
    "                    if company_sources[k] == company_list[j]:\n",
    "                        company_contributions[j] += json.loads(res.doc_topic_exposures[0])[k]\n",
    "\n",
    "            company_rankings[:, i] = [x[0] for x in  float(res.strength) * float(res.sentiment) * company_contributions[:]]  \n",
    "\n",
    "            print('---------------')\n",
    "\n",
    "\n",
    "    # Add up the ranking of companies per topic into the final credit screen\n",
    "    Corporate_screen = np.mean(company_rankings, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-\tRepeat the above tasks for each date in the historical period to get the complete history of your single name screen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('------------ Retrieve all companies found in the dataset ----------')\n",
    "\n",
    "payload = nucleus_api.DocInfo(dataset='Corporate_docs')\n",
    "api_response = api_instance.post_doc_info(payload)\n",
    "\n",
    "company_sources = []\n",
    "for res in api_response.result:        \n",
    "    company_sources.append(res.attribute['ticker']) \n",
    "\n",
    "company_list = np.unique(company_sources)\n",
    "\n",
    "\n",
    "print('--------------- Retrieve the time range of the dataset -------------')\n",
    "\n",
    "payload = nucleus_api.DatasetInfo(dataset='Corporate_docs', query='')\n",
    "api_response = api_instance.post_dataset_info(payload)\n",
    "\n",
    "first_date = datetime.datetime.fromtimestamp(float(api_response.result.time_range[0]))\n",
    "last_date = datetime.datetime.fromtimestamp(float(api_response.result.time_range[1]))\n",
    "delta = last_date - first_date\n",
    "\n",
    "# Now loop through time and at each date, compute the ranking of companies\n",
    "T = 90 # The look-back period in days\n",
    "\n",
    "Corporate_screen = []\n",
    "for i in range(delta.days):  \n",
    "    if i == 0:\n",
    "        end_date = first_date + datetime.timedelta(days=T)\n",
    " \n",
    "    # first and last date used for the lookback period of T days\n",
    "    start_date = end_date - datetime.timedelta(days=T)\n",
    "    start_date_str = start_date.strftime(\"%Y-%m-%d 00:00:00\")\n",
    "\n",
    "    # We want a daily indicator\n",
    "    end_date = end_date + datetime.timedelta(days=1) \n",
    "    end_date_str = end_date.strftime(\"%Y-%m-%d 00:00:00\")\n",
    "\n",
    "    payload = nucleus_api.TopicSentimentModel(dataset=\"Corporate_docs\",      \n",
    "                                query='',                   \n",
    "                                num_topics=20,\n",
    "                                num_keywords=8,\n",
    "                                period_start=start_date_str,\n",
    "                                period_end=end_date_str)\n",
    "    api_response = api_instance.post_topic_sentiment_api(payload)\n",
    "\n",
    "    company_rankings = np.zeros(len(company_list), len(enumerate(api_response.result))\n",
    "    for l, res in enumerate(api_response.result):\n",
    "        # Aggregate all document exposures within a topic into a company exposure, using the dataset metadata\n",
    "        payload = nucleus_api.DocInfo(dataset='Corporate_docs', doc_ids=res.doc_ids)\n",
    "        api_response1 = api_instance.post_doc_info(payload)\n",
    "\n",
    "        company_sources = [] # This list will be much shorter than the whole dataset because not all documents contribute to a given topic\n",
    "        for res1 in api_response1.result:        \n",
    "            company_sources.append(res1.attribute['ticker']) \n",
    "\n",
    "        company_contributions = np.zeros([len(company_list), 1])\n",
    "        for j in range(len(company_list)):\n",
    "            for k in range(len(company_sources)):\n",
    "                if company_sources[k] == company_list[j]:\n",
    "                    company_contributions[j] += json.loads(res.doc_topic_exposures[0])[k]\n",
    "\n",
    "        company_rankings[:, l] = [x[0] for x in  float(res.strength) * float(res.sentiment) * company_contributions[:]]       \n",
    "\n",
    "    # Add up the ranking of companies per topic into the final credit screen\n",
    "    Corporate_screen.append(np.mean(company_rankings, axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.\tResults Interpretation\n",
    "-\tPlot the time series of the company rankings against the beta-adjusted equity returns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.\tFine Tuning\n",
    "\n",
    "#### a.\tTailoring the topics\n",
    "-\tSee whether some tailoring may be applied to your single name screen by excluding certain topics considered not impactful. This is achieved by using the custom_stop_words parameter in input to the Topic Sentiment API\n",
    "\n",
    "\n",
    "-\tIdentify and Extract key topics on the subset of documents and print their keywords\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('------------- Get list of topics from dataset --------------')\n",
    "\n",
    "payload = nucleus_api.Topics(dataset='Corporate_docs',                       \n",
    "                                query='',                       \n",
    "                                num_topics=20, \n",
    "                                num_keywords=8,\n",
    "                                period_start=\"2018-11-01 00:00:00\",\n",
    "                                period_end=\"2019-01-01 00:00:00\")\n",
    "try:\n",
    "    api_response = api_instance.post_topic_api(payload)        \n",
    "    api_ok = True\n",
    "except ApiException as e:\n",
    "    api_error = json.loads(e.body)\n",
    "    print('ERROR:', api_error['message'])\n",
    "    api_ok = False\n",
    "\n",
    "if api_ok:    \n",
    "    for i, res in enumerate(api_response.result.topics):\n",
    "        print('Topic', i, ' keywords: ', res.keywords)    \n",
    "        print('---------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then tailor the screen analysis by creating a custom_stop_words variable. Initialize the variable as follows, for instance, and pass it in the payload of the main code of section 2: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_stop_words = [\"call\",\"report\"] # str | List of stop words. (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b.\tFocusing the screen analysis on certain subjects\n",
    "In case you decide to focus the screen analysis, for instance on financial health and corporate actions subjects, simply substitute the query variable in the main code of section 2. with: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = '(earnings OR debt OR competition OR lawsuit OR restructuring)' # str | Fulltext query, using mysql MATCH boolean query format. Example: \"(word1 OR word2) AND (word3 OR word4)\" (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c.\tExploring the impact of the type of documents, the lookback period, the number of topics being extracted\n",
    "**num_topics**: You can compute the single names' screen using different breadth of topics by changing the variable num_topics in the payload in the main code of section 2. A larger value will provide more breadth in establishing rankings while a smaller value will provide a shallower measure. If num_topics is too large, some very marginal topics may bring in a lot of noise in measuring company rankings.\n",
    "\n",
    "**T**: You can compute the single names' screen with different speeds of propagation by changing the variable T (lookback) in the main code of section 2. A larger value will provide a slowly changing ranking while a smaller value will lead to a very responsive ranking. If T is too small, too few documents may be used and this may lead to a lot of noise in ranking companies. If T is too long, the rankings won’t reflect quickly enough important new information. \n",
    "\n",
    "**Document types**: You can investigate how the single names' screen changes if it is measured using only one type of document among company reports, press releases, earning call transcripts compared to capturing the whole content by leveraging the metadata selector provided during the construction of the dataset. Rerun the main code of section 2. on a subset of the whole corpus. Create a variable metadata_selection and pass it in to the payload:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_selection = {\"category\": \"Report\"}   # str | json object of {\\\"metadata_field\\\":[\\\"selected_values\\\"]} (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.\tNext Steps\n",
    "-\tPossible extension: repeat the above tasks for different industry sectors\n",
    " - This gives you a broad market screen to have views within every important industry, which you can use for security selection \n",
    " - If you blend all companies across industries together in a single dataset, you could then derive industry rankings using the main code in section 2 and an extra metadata argument being the industry sector attached to a company/document. Then you can apply this industry screen to sector allocation\n",
    "\n",
    "\n",
    "-\tPerform a correlation analysis between the time series of the ranking of companies and their beta-adjusted equity returns\n",
    " - Several time horizons for price impact can be studied: 1 day, 7 days, a few weeks, perhaps even longer lasting impact\n",
    " - Several time lags for price impact can be studied: following day, 2 to 3 days gap before market starts adjusting, a week, perhaps even longer gap before markets start incorporating the information from the corporate content\n",
    " - You could also conduct such correlation analysis by bucket of dividend yield or P/E ratio within your universe of single name stocks. Are the companies trading at higher dividend yields showing a different price impact than companies trading at lower dividend yields?\n",
    "\n",
    "\n",
    "-\tPossible extension: explore simple transformation of the indicator \n",
    " - You could rescale and smooth these rankings using a cross-sectional score \n",
    "\n",
    "        Score(Company i) = ( Rank(Company i) – Average(Ranks, [Companies]) ) / Std(Ranks, [Companies])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) 2019 SumUp Analytics, Inc. All Rights Reserved.\n",
    "\n",
    "NOTICE: All information contained herein is, and remains the property of SumUp Analytics Inc. and its suppliers, if any. The intellectual and technical concepts contained herein are proprietary to SumUp Analytics Inc. and its suppliers and may be covered by U.S. and Foreign Patents, patents in process, and are protected by trade secret or copyright law.\n",
    "\n",
    "Dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from SumUp Analytics Inc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}