Skip to content

Project INCREMENT 2

Jump to bottom
System.out.println("Ahamed-Raziuddin") edited this page May 7, 2020 · 7 revisions

AIM : Running queries in Itversity Cloud on different big data tools (HIve-MR, Hive-Tez,SparkSQL,SparkDF) and comparing the runtimes of the queries

TEAM # 2

TEAM LEADER NAME : BHAVAZ AKULA, ID : 16282638, Class ID : 3

Name: Ahamed Raziuddin, Id: 16275183. Class id: 15

Name : Dinesh Kumar Reddy Kusam, ID : 16291692, Class Id : 9

Name: Acsenvil Alex Student Id: 2251438 Class id: 2

  • We want to check the run time of queries like ’like’ , ‘groupby’ ,’orderby’ , ‘count’ , ‘pattern-match’ , etc and help the users tell which one is best for which query and save their time from selecting which tool to be used.
  • I’m using the flight data set called ‘data expo 09’
  • I’m setting up a hive, spark environment in ITversity labs CREDENTIALS : Username: apandit7 Password : Ik0i0a6ti4aenguezei2Chie3Eupie0d
  • It has around 400 GB of ram and it just cost me just around 29$.

  • I have runned the Group-by, Order-by,Count, Patten-Match query in Hive , Hive with Tez , SparkSQL and SparkDataFrame.

  • I have run these queries approximately around 100 times to get accurate average runtime value. Its difficult to take the screenshots of all the 100 executions so i have mentioned only few here.

  • And I Just showed you the Pattern-Match Query in the wiki so that i Can the entire project in the Final Review.

Running queries in HIVE environment

RUNNING QUERIES IN HIVE using TEZ ENGINE

RUNNING QUERIES USING SPARK SQL

Running queries in SparkDF

We are using Plotly Dash to visualize the outputs.

I'm still learning dash will show the entire output by final submission.

later we are gonna deploy dash page in Heroku server so that our Dynamic web server is available for everyone

PLOTLY - DASH :

Dash is Plotly's main product which is an open-source Python and R framework for building web-based analytic applications

Heroku :

Heroku is a cloud platform as a service (PaaS) supporting several programming languages. One of the first cloud platforms, Heroku has been in development since June 2007, when it supported only the Ruby programming language, but now supports Java, Node.js, Scala, Clojure, Python, PHP, and For this reason, Heroku is said to be a polyglot platform as it has features for a developer to build, run and scale applications in a similar manner across most languages.

References :

My directory spark : cloudproj/spark my hive databse : cloud_flight_data

Clone this wiki locally