Project INCREMENT 1

AIM : Running queries in Itversity Cloud on different big data tools (HIve-MR, Hive-Tez,SparkSQL,SparkDF) and comparing the runtimes of the queries

TEAM # 2

TEAM LEADER NAME : BHAVAZ AKULA, ID : 16282638, Class ID : 3

Name: Ahamed Raziuddin, Id: 16275183, Class id: 15

Name : Dinesh Kumar Reddy Kusam, ID : 16291692, Class Id : 9

Name: Acsenvil Alex Student Id: 2251438 Class id: 2

We want to check the run time of queries like ’like’ , ‘groupby’ ,’orderby’ , ‘count’ , ‘pattern-match’ , etc and help the users tell which one is best for which query and save their time from selecting which tool to be used.
I’m using the flight data set called ‘data expo 09’
I’m setting up a hive, spark environment in ITversity labs CREDENTIALS : Username: apandit7 Password : Ik0i0a6ti4aenguezei2Chie3Eupie0d
It has around 400 GB of ram and it just cost me just around 29$.

I have runned the like query in Hive , Spark SQL as of now Query : select * from df where Origin like "I%"
I have run these queries approximately around 100 times to get accurate average runtime value. Its difficult to take the screenshots of all the 100 executions so i have mentioned only few here.

References :

You can view my work in https://labs.itversity.com/user/lab using my credentails mentioned above

My directory spark : cloudproj/spark my hive databse : cloud_flight_data