Big Data Analytics - Analyze with Hadoop & Spark

環境

• http://micron.largitdata.com:8080/#/login
• 帳號：admin
• 密碼：admin

Name Node 管理介面

• http://micron.largitdata.com:50070

Hive Advanced

• https://github.com/ywchiu/micronbigdata2/blob/master/HiveAdvanced/CreateManagedTable.sql
• https://github.com/ywchiu/micronbigdata2/blob/master/HiveAdvanced/Partitioning.sql
• https://github.com/ywchiu/micronbigdata2/blob/master/HiveAdvanced/Bucketing.sql
• https://github.com/ywchiu/micronbigdata2/blob/master/HiveAdvanced/windowing.sql

Custom Python

• https://github.com/ywchiu/micronbigdata2/blob/master/HiveAdvanced/function.py
• https://github.com/ywchiu/micronbigdata2/blob/master/HiveAdvanced/RunningCustomPythonscript.sql

投影片

• https://www.slideshare.net/secret/hAQy6APouvIQrZ

Docker on Cloud

• https://hortonworks.com/downloads/#sandbox

Zeppelin Notebook

• http://micron.largitdata.com:8080/#/

Zepl

• https://www.zepl.com/explore
• https://raw.githubusercontent.com/ywchiu/micronbigdata/master/Spark/Demo20190313.json

MapReduce 範例

• https://github.com/ywchiu/micronbigdata/blob/master/MapReduce.md

Hive 範例

• https://github.com/ywchiu/micronbigdata/blob/master/Hive/hive-exercise1.txt
• https://github.com/ywchiu/micronbigdata/blob/master/Hive/hive-exercise2.txt

Sqoop 範例

• https://github.com/ywchiu/micronbigdata/blob/master/Sqoop/sqoop-exercises.txt

Pig 範例

• https://github.com/ywchiu/micronbigdata/blob/master/Pig/pig-exercises.txt

Flume 範例

• https://github.com/ywchiu/micronbigdata/blob/master/Flume/flume-exercises.txt

Spark 範例

• https://raw.githubusercontent.com/ywchiu/micronbigdata/master/Spark/20160704Demo.json
• https://raw.githubusercontent.com/ywchiu/micronbigdata/master/Spark/Spark%20SQL.json

HBase 投影片與範例

• https://github.com/ywchiu/micronbigdata/blob/master/HBase/20160707%20-%20HBase%20%E7%B0%A1%E4%BB%8B%E8%88%87%E5%AF%A6%E4%BD%9C.pdf

Hive Batch 與Spark Streaming 簡介

• https://github.com/ywchiu/micronbigdata/blob/master/Slides/20161201%20%E5%B7%A8%E9%87%8F%E8%B3%87%E6%96%99%E5%AF%A6%E6%88%B0%E8%AA%B2%E7%A8%8B(I)%20%E2%80%93%20Hive%20Batch%20%E8%88%87Spark%20Streaming%20%E7%B0%A1%E4%BB%8B.pdf

Kafka 與 HBase 簡介

• https://github.com/ywchiu/micronbigdata/blob/master/Slides/20161208%20%E5%B7%A8%E9%87%8F%E8%B3%87%E6%96%99%E5%AF%A6%E6%88%B0%E8%AA%B2%E7%A8%8B(II)%20%E2%80%93%20Kafka%20%E8%88%87%20HBase%20%E7%B0%A1%E4%BB%8B.pdf

建立即時分析儀表板

• https://github.com/ywchiu/micronbigdata/blob/master/Slides/20161209%20%E5%B7%A8%E9%87%8F%E8%B3%87%E6%96%99%E5%AF%A6%E6%88%B0%E8%AA%B2%E7%A8%8B(III)%20%E2%80%93%20%E5%BB%BA%E7%AB%8B%E5%8D%B3%E6%99%82%E5%88%86%E6%9E%90%E5%84%80%E8%A1%A8%E6%9D%BF.pdf

Recommendation System

• https://youtu.be/RoRixPm4XyQ
• https://www.youtube.com/watch?v=l0TemN7lpCs