Big data analytics allows you to analyse data at scale. It has applications in almost every industry in the world. Let’s consider an unconventional application that you wouldn’t ordinarily encounter. New York City is a thriving metropolis. Just like most other metros its size, one of the biggest problems its citizens face is parking. The classic combination of a huge number of cars and cramped geography leads to a huge number of parking tickets. In an attempt to scientifically analyse this phenomenon, the NYC Police Department has collected data for parking tickets. Of these, the data files for multiple years are publicly available on Kaggle. We will try and perform some exploratory analysis on a part of this data. Spark will allow us to analyse the full files at high speeds as opposed to taking a series of random samples that will approximate the population. For the scope of this analysis, we will analyse the parking tickets over the year 2017. The purpose of this case study is to conduct an exploratory data analysis that will help you understand the data.
- Reading Data
- Cleaning Data
- Aggregation tasks along with answering a lot of business related questions
- NYC+Parking+Tickets+-+An+Exploratory+Analysis+(1) (1).ipynb : The python file showing coding and data analysis in PySpark
- https://www.kaggle.com/new-york-city/nyc-parking-tickets/data#Parking_Violations_Issued_-_Fiscal_Year_2017.csv (Parking_Violations_Issued_-_Fiscal_Year_2017.csv) : Data worked on