-
Notifications
You must be signed in to change notification settings - Fork 0
Home
Gukie edited this page Apr 5, 2017
·
7 revisions
Welcome to the logViewer wiki!
This project is mainly to make a real-time demo system to do following things:
- collect log data
- parse the log data to a uniform format
- analyze them
- give a report about the log.
Main flow:
数据采集 --> 数据接入 --> 流式计算 -->数据持久化 --> 数据展现
The tech list to use:
- LogStash/Flume: 数据采集
- Kafka: 数据采集速度 跟 数据处理速度有可能是不同步的；所以使用Kafka做一下缓冲
- Storm: 对采集到的数据进行 实时分析
- Elastic: 数据持久化
- Kibana: 数据展现
refer: 有道云笔记: <大数据架构：flume-ng+Kafka+Storm+HDFS 实时系统组合>