notebooks/pydata_nyc_2019.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each part of this tutorial is meant to help you work through some part of the data science take home challenge that I have seen many applicants struggle with.  The data set we are working with today is the citibike trip data which can be found [here](https://www.citibikenyc.com/system-data).  The data has the following variables:\n",
    "\n",
    "- Trip Duration (seconds)\n",
    "- Start Time and Date\n",
    "- Stop Time and Date\n",
    "- Start Station Name\n",
    "- End Station Name\n",
    "- Station ID\n",
    "- Station Lat/Long\n",
    "- Bike ID\n",
    "- User Type (Customer = 24-hour pass or 3-day pass user; Subscriber = Annual Member)\n",
    "- Gender (Zero=unknown; 1=male; 2=female)\n",
    "- Year of Birth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run these first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir data\n",
    "!mkdir data/raw\n",
    "!wget -P data/raw https://s3.amazonaws.com/tripdata/201909-citibike-tripdata.csv.zip\n",
    "!wget -P data/raw https://s3.amazonaws.com/tripdata/201908-citibike-tripdata.csv.zip\n",
    "!wget -P data/raw https://s3.amazonaws.com/tripdata/201907-citibike-tripdata.csv.zip\n",
    "!unzip -q data/raw/201909-citibike-tripdata.csv.zip -d data/raw/\n",
    "!unzip -q data/raw/201908-citibike-tripdata.csv.zip -d data/raw/\n",
    "!unzip -q data/raw/201907-citibike-tripdata.csv.zip -d data/raw/\n",
    "cb_csv = [x for x in os.listdir(\"data/raw\") if x.endswith(\".csv\")]\n",
    "df_cb_list = []\n",
    "for csv in cb_csv:\n",
    "    df_cb_list.append(pd.read_csv(os.path.join(\"data/raw\",csv)))\n",
    "df_cb = pd.concat(df_cb_list, ignore_index=True, sort=False)\n",
    "del df_cb_list\n",
    "df_cb = df_cb.rename(columns={x:x.replace(\" \", \"_\") for x in df_cb.columns})\n",
    "df_cb.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data = pd.read_csv(\"https://raw.githubusercontent.com/MichoelSnow/pydata_nyc_2019/master/downloads/citibike_data.csv\", low_memory=False)\n",
    "df_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://github.com/MichoelSnow/pydata_nyc_2019/raw/master/downloads/model_predictions.zip\n",
    "!unzip model_predictions.zip    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Working with Trick Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "People who create these datasets will sometimes purposefully create \"bad data\" within the data set.  As most data in the wild has some internal discrepancies, you must carefully examine the data before you can model it.  Only once you have identified and dealt with, or decided to ignore, the problems with the data can you feel confident in analyzing it. To this end I have modified a portion of the citibike data set, which already had some problems with it, and added in a whole host of errors.\n",
    "\n",
    "The variable `df_data`, which you created above, is the modified data set. Your task is to find all the different problems with the data, and determine how you are going to rectify them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Create an Outline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have an understanding of the data, here is your challenge.\n",
    "\n",
    "\n",
    "*Using the citibike data set predict how long a ride will be given the information available when the rider starts their ride.  You should spend about 5 hours working on this challenge.  Please make sure to include clear data analysis and visualization, well-documented and interpretable code, appropriate use of statistical/machine-learning models, and explanation of the results.*\n",
    "\n",
    "Do not start coding immediately.  First you should develop a plan of attack for the challenge. For each of the following sections, write a brief outline (no more than 1 paragraph) which includes what you are going to do and how long it should take.  Be as specific as reasonable, e.g., for a figure describe the axes and the type of plot.  Try to have your time estimates stay within the suggested time limit.  I would suggest you allot 80% of the time provided with a 20% buffer.  \n",
    "\n",
    "While I believe any report should contain all the following sections, you will want to vary the amount of time you spend on each depending on the challenge and the company.  For example, the more modeling the position the more time should be spent on that. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA (single variable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Relationships (between different variables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Result Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Exploratory Data Analysis (EDA) and Data Relationships"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the above outline, create at least one figure from both the EDA and Data Relationships section.  Each figure should be presentation ready before you move on to the next one.  Your goal for any figure, is to have it as understandable as possible without requiring outside explanation. Now is not the time to be learning a new plotting library; to get these done quickly and beautifully you should use the library you are most familiar with.  \n",
    "\n",
    "Use [pandas profiling](https://pandas-profiling.github.io/pandas-profiling/docs/) to get a headstart on all your analyses.  Click [here](https://michoelsnow.github.io/html_files/citibike_data_report) for a link the output report for the citibike data.\n",
    "\n",
    "Your first step, before you begin to code, is to sketch out what you plan on plotting.  Consider both the data and the type of plot.  Is this the most informative variables, should you engineer.\n",
    "\n",
    "As a guide, here is a table from Bertin's Semiology of graphics as seen in Jake Vanderplas's talk at PyCon 2019 talk [How to Think about Data Visualization](https://www.youtube.com/watch?v=vTingdk_pVM).  This is a quick reference for how to plot the three main data types:\n",
    "\n",
    "- Quantitative (Q)\n",
    "    - Numerical data\n",
    "- Ordinal (O)\n",
    "    - Ordered categorical data \n",
    "- Nominal (N)\n",
    "    - Unordered categorical data\n",
    "\n",
    "|               |       |       |       |\n",
    "|-------------\t|---\t|---\t|---\t|\n",
    "| Position    \t| N \t| O \t| Q \t|\n",
    "| Size        \t| N \t| O \t| Q \t|\n",
    "| Color Value \t| N \t| O \t| q \t|\n",
    "| Texture     \t| N \t| o \t|   \t|\n",
    "| Color Hue   \t| N \t|   \t|   \t|\n",
    "| Angle       \t| N \t|   \t|   \t|\n",
    "| Shape       \t| N \t|   \t|   \t|\n",
    "\n",
    "\n",
    "Here are links to sample plots for the various libraries (for inspiration and reference):\n",
    "- [Matplotlib](https://matplotlib.org/3.1.1/tutorials/introductory/sample_plots.html)\n",
    "- [plotly](https://plot.ly/python/)\n",
    "- [Seaborn](https://seaborn.pydata.org/examples/index.html)\n",
    "- [plotnine](https://plotnine.readthedocs.io/en/stable/gallery.html)\n",
    "- [Altair](https://altair-viz.github.io/gallery/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4: Model Explanation & Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A common mistake is to model your data and present your results with either just a score or a confusion matrix.  This result is the culmination of your work, it should be presented with as much thought as the rest of your notebook. In the cell below a dataframe is loaded which contains the test data along with the predicted ride duration as the column `predicted_duration`. Create a figure, or two at most, which visually demonstrates the results.  Then spend a paragraph explaining where the model has succeeded and where it still needs more work.  While a good model is important, good communication is essential to a well received data challenge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_results = pd.read_feather(\"model_predictions.feather\")\n",
    "df_results.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5: Musings and Addenda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Structuring your data science challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the sections discussed in the outline, all challenges should start off with a summary and a readme section.\n",
    "\n",
    "The summary section should be a short summary of what you did and what you found.  It should be between one to three paragraphs. Put anything you want to be sure the reviewer reads into an executive summary. Write the summary, as well as the rest of the code, as if it was a project you were presenting to people unfamiliar with the work you are doing.  Assume they are knowledgeable about the topic but not the specifics of your task or the underlying data.  Use hyperlinks to link to specific figures and sections you are referencing in the notebook.  Comment about positive findings and pertinent negatives, e.g. always check for missing values, but only reference them in the summary if there is something unique about them.  see [here](https://michoelsnow.github.io/html_files/citibike#Summary) for an example summary\n",
    "\n",
    "The Readme section should contain any technical information required to run your code.  At least list the version of python your code was running on, as well as the package versions.  If your code uses a lot of memory, takes a long time to run or some other technical nuance that you want to inform the reader about, place it here. See [here](https://michoelsnow.github.io/html_files/citibike#Readme) for an example Readme."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Things to keep in mind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like any presentation, spelling and formatting matter.  To that end I would recommend using a notebook spellchecker and formatter.  Below are the two that I use:\n",
    "\n",
    "- Black for jupyter notebooks\n",
    "    - https://github.com/drillan/jupyter-black\n",
    "- spellcheck for jupyter notebooks (part of [nbextensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/index.html))\n",
    "    - https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/spellchecker/README.html\n",
    " \n",
    " \n",
    "**Other musings** \n",
    "- Tell me a Story\n",
    "    - Your challenge should have a beginning, middle, and end.  It should tell a compelling, cohesive story.\n",
    "- Ask Questions\n",
    "    - If something is not clear from the challenge (and feel free to ask a friend or colleague to make sure you are not missing something simple) ask.\n",
    "    - It is possible they made the instructions vague on purpose to make sure you are someone who follows up on unclear requirements\n",
    "    - Be upfront about prior commitments or other time constraints \n",
    "- Explain and Justify\n",
    "    - Don't randomly select algorithms to throw at the data. \n",
    "        - Why did you choose each algorithm? \n",
    "        - Why were you choices valid and reasonable? \n",
    "        - What else would you try if you had more time? \n",
    "    - You will have to answer these questions later.\n",
    "- Random seeds\n",
    "    - Always set you random seed for code reproducibility\n",
    "- Tailor your response\n",
    "    - Consider the job description and tailor your approach to match the skills needed for the job (i.e., if this is a very ML heavy job, spend more time on modeling than on EDA)\n",
    "    - Where you are in the company's interview process, e.g. screening vs last step\n",
    "- Submit an IPython notebook and an HTML download\n",
    "    - Just in case they can't get the notebook running or don't have time to start up a kernel, giving them an HTML version of your notebook ensures that what you are giving is what they are seeing. \n",
    "- DO NOT use a neural network    \n",
    "    - Barring some outliers, you should never use a neural network as your first model and only use it as a second model when you have a really good reason to.\n",
    "- Create a portfolio\n",
    "    - Find your own datasets and create data science challenge projects on your own and add to your portfolio.\n",
    "    - Forks/Classes are not the best showcases, and might be ignored.\n",
    "\n",
    "**Grading Rubric** (the order of importance will depend on the grader)\n",
    "- Does your code run?\n",
    "- What is the ratio of text to code?\n",
    "- Do you explain what significant choices you made and why you made them?\n",
    "- Are there any spelling mistakes?\n",
    "- Is your code pythonic (e.g., how often you use loops)?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List of good newsletters/Data science articles:\n",
    "\n",
    "- NYC data Jobs and Events by Josh Laurito\n",
    "    - https://tinyletter.com/nycdatajobs\n",
    "- Normcore Tech by Vicki Boykis\n",
    "    - https://vicki.substack.com/\n",
    "- Data Science Roundup by Tristan Handy\n",
    "    - http://roundup.fishtownanalytics.com/\n",
    "- The Algorithm from MIT Technology Review\n",
    "    - https://forms.technologyreview.com/newsletters/\n",
    "- PyCoder's Weekly     \n",
    "    - https://pycoders.com/\n",
    "- Natural Language Processing News by Sebastian Ruder\n",
    "    - http://newsletter.ruder.io/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}