Spark-Training

This repository stores my training course work on Databricks platform. It includes two parts:

• DataFrame Lab:
  • Practiced read, write data using DataFrameReader and DataFrameWriter
  • Using DataFrame API, perform transformation and action to analyze data.
• Structured Streaming:
  • Practiced read, write streams from file and messaging system Kafka using DataStreamReader and DataStreamWriter.
  • Using DataFrame API to perform ETL jobs.
  • Built a Twitter realtime data pipeline to get information like top most tweeted hashtag in last 5 minute and where they came from.