New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python 10 min系列二之日志可视化的进阶 #2

Open
shengxinjing opened this Issue Dec 2, 2015 · 54 comments

Comments

Projects
None yet
@shengxinjing
Owner

shengxinjing commented Dec 2, 2015

日志可视化 🍄

本文需要有一定的python和前端基础,如果没基础的,请关注我后续的基础教程系列博客
本文所有的demo,都是浏览器下展示的

同步发布在博客

项目地址,跪求右上角star ⭐️ 🌟 🌠
基于python,前端基于echarts,力求用简单的代码说明原理

提纲

  • access日志按照什么维度展示数据
  • 和web结合,更好的体验,只需记住URL
  • 分页排序和搜索,表格展示体验的提升
  • 一图胜万言,汇总信息更友好的展示
  • 逼格满满,让IP信息在地图上展示
  • 后续展望--机房网络流量可视化,机器、机柜、机房3D展示

背景

老板要看日志数据汇总

  • 本文重点:如何做可视化
  • 目标:如何用友好的方式去展现沉闷繁冗的数据
  • 下个月就发年终奖!!!
  • 为了说明可视化的方式,用一个简单的log举例子

原材料


  • 一个标准的access_log日志 大概2W行
  • 老板想要这个日志的分析结果,每个url,ip,status分别访问多少次,把前几名统计出来看看
  • 分析出统计数据 展现结果

很普通的日志,大概长这样

为了方便展示,切割了一下,大概2W行

61.159.140.123 - - [23/Aug/2014:00:01:42 +0800] "GET /favicon.ico HTTP/1.1" 404 \ "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36 LBBROWSER" "-"
61.159.140.123 - - [23/Aug/2014:00:01:42 +0800] "GET /favicon.ico HTTP/1.1" 404 \ "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36 LBBROWSER" "-"
61.159.140.123 - - [23/Aug/2014:00:01:42 +0800] "GET /favicon.ico HTTP/1.1" 404 \ "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36 LBBROWSER" "-"
61.159.140.123 - - [23/Aug/2014:00:01:42 +0800] "GET /favicon.ico HTTP/1.1" 404 \ "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36 LBBROWSER" "-"
66.249.64.5 - - [23/Aug/2014:00:02:16 +0800] "GET /data/uploads/2013/0519/09/small_51982ba18e012.jpg HTTP/1.1" 200 \ "-" "Googlebot-Image/1.0" "-"
66.249.64.10 - - [23/Aug/2014:00:02:54 +0800] "GET /data/uploads/2013/0319/08/middle_5147b116e93b4.jpg HTTP/1.1" 200 \ "-" "Googlebot-Image/1.0" "-"

目标

  • 初级展现
  • 友好交互
  • 饼图汇总
  • 更进一步
  • 后续展望

级别1


  • 数据处理,命令行展现
  • 打开文件,处理完数据后排序
  • 打印前10

talk is cheap, show me the money code!

# coding=utf-8
f = open('www_access_20140823.log')
res = {}
for l in f:
    arr = l.split(' ')
    # 获取ip url 和status
    ip = arr[0]
    url = arr[6]
    status = arr[8]
    # ip url 和status当key,每次统计+1
    res[(ip,url,status)] = res.get((ip,url,status),0)+1
# 生成一个临时的list
res_list = [(k[0],k[1],k[2],v) for k,v in res.items()]
# 按照统计数量排序,打印前10
for k in sorted(res_list,key=lambda x:x[3],reverse=True)[:10]:
    print k

处理结果

('222.86.153.12', '/images/cursor_minify.cur', '404', 60)
('222.86.153.12', '/images/cursor_zoom.cur', '404', 32)
('58.253.6.133', '/images/cursor_minify.cur', '404', 32)
('111.85.34.165', '/%3Ca%20href=', '404', 28)
('58.253.6.133', '/images/cursor_zoom.cur', '404', 27)
('218.29.111.117', '/images/cursor_zoom.cur', '404', 27)
('218.29.111.117', '/images/cursor_minify.cur', '404', 26)
('117.63.146.40', '/public/js/common.js?20110824', '200', 19)
('117.63.146.40', '/favicon.ico', '404', 18)
('117.63.146.40', '/public/js/weibo.js?20110824', '200', 16)

任务完成

  • 下一步粘到邮件里,或者生成一个csv文件发出去
  • 然而这是一个看脸的社会,运维也逃脱不了这个魔咒

级别2

浏览器端展现

生成list之后,拼接sql,存入数据库

talk is cheap, show me the money code!

import MySQLdb as mysql
con = mysql.connect(user='root',\
                    passwd='',\
                    db='log',\
                    host='localhost')
con.autocommit(True)
cur = con.cursor()
# 处理文件省略
for s in res_list:
    sql = 'insert log values ("%s","%s",%s,%s)' % s
    try:
        # 入库
        cur.execute(sql)
    except Exception, e:
        pass

前端展现

读库 展现页面

talk is cheap, show me the money code!

from flask import Flask,request,render_template
app = Flask(__name__)
import MySQLdb as mysql
con = mysql.connect(user='xx',\
                    passwd='xx',\
                    db='xx')
cur = con.cursor()
@app.route('/')
def index():
    table = '<table border="1">'
    cur.execute('select * from log order by value desc limit 20; ')
    for c in cur.fetchall():
        table += '<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>'%c
    table +='</table>'
    return table

if __name__ == '__main__':
    app.run(host='0.0.0.0',port=9092)

给老板一个url即可,老板想看随时能看

表格

但是老板表示有点丑

老板

完整的思路

表格

前端展现上做一些优化

  • 分页
  • 表格排序
  • 搜索
  • 控制每页显示数量
  • 数据多了之后,前端交互和后端数据的接口配合

不动戳大

表格

我们的console页面,提供几个汇总信息,那就更好啦

比如根据http的status来个汇总

老板

难不倒我

一句sql搞定

select status,sum(value) from log group by status
+--------+------------+
| status | sum(value) |
+--------+------------+
|    200 |      15529 |
|    206 |          6 |
|    301 |          2 |
|    304 |       3549 |
|    403 |          1 |
|    404 |        847 |
+--------+------------+
6 rows in set (0.02 sec)

汇总信息可视化

根据https状态汇总

其他功能

  • 图例开关
  • 图表转换
  • 数据视图
  • 直接导出图片

可视化并不仅限于此

上面只是举得小栗子

如果你对时间更感兴趣,我们的log里也是有时间信息的,可以像下面这样

统计量,时间轴拖动,保存图片,etc

更进一步

如何让日志数据更加一目了然,让老板觉得你很有逼格呢

IP都是有地址位置的,定位每个ip的位置,画个地图出来汇总

经纬度坐标系统

  • 地球坐标(WGS84)
    • 国际标准,从 GPS 设备中取出的数据的坐标系
    • 国际地图提供商使用的坐标系
  • 火星坐标(GCJ-02)
    • 中国标准,从国行移动设备中定位获取的坐标数据使用这个坐标系
  • 百度坐标(BD-09)
    • 百度标准,百度 SDK,百度地图,Geocoding 使用

应用场景

  • WGS84坐标系:
    • 国际标准,谷歌国外地图、osm地图等国外的地图一般都是这个
  • 火星坐标系:
    • iOS 地图
    • Gogole地图
    • 搜搜、阿里云、高德地图
  • 百度坐标系:
    • 当然只有百度地图

地图是需要经纬度的,用第三方的ip库转换一下

http://developer.baidu.com/map/index.php?title=webapi/ip-api

talk is cheap, show me the money code!

import urllib2
import json
key = 'q5mTrTGzCSVq5QmGpI9y18Bo'
ipurl = 'http://api.map.baidu.com/location/ip?ak='+key+'&coor=bd09ll&ip='
sqlarr = []
def getGeo(ip):
    try:
        u = urllib2.urlopen(ipurl+ip)
        page = json.load(u)
        if 'content' in page:
            point = page['content'].get('point')
            print 'ip %s has geoX %s and geoY %s' % (ip,point['x'],point['y']) 
    except:
        print 'error'
getGeo('202.198.16.3')
# ip 202.198.16.3 has geoX 125.31364243 and geoY 43.89833761

就想玩网游时候,坐标可以定位一个人,经纬度可以再地图上定位一个点,画图展现

刚才那个图仅关注区域,进阶一下,还要关注访问量

可以根据value筛选

进阶:多台机器的日志

  • 获取每个机器的hostname和ip,和日志数据一起存在数据库里
  • 一个表存日志,带上一个机器的id
  • 机器的id=>ip和经纬度
  • 最终统计访问量

后续扩展

  • 日志数据
  • 前端展现场景

怎么实践

  • 这次分享的主题关注与可视化
  • 我们用了一个很小的静态日志,目的是说明可视化的思路
  • 实际工作中日志数据应该怎么处理

elk

Logstash+ElasticSearch+Kibana4

  • logstash
    • 日志进行收集、分析,并将其存储供使用
  • ElasticSearch
    • 开源分布式搜索引擎,
  • Kibana4
    • 日志分析友好的 Web 界面
  • 其他
    • Kafka scribe等

常见日志处理架构

  • ELK
  • logstash+Hadoop
  • scribe+hadoop
  • 线上数据->Flume->Kafka->Hdfs->Map/Reduce
  • 线上数据->flume->kafka->storm
  • 在上面的基础上定制化二次开发,比如MR平台上写代码,我们的代码就可以直接拿来用
  • 日志的数据处理架构详情,请见下回分解

前端展现场景

  • 展现逼格更高一些,数据一样,效果更好
  • 运维人员权限树
  • 流量图
  • 年终数据统计
  • 区域点击统计图
  • 3D机房

大家具体需要哪个,可以继续扩展,给大家展现几个假数据的demo

高逼格饼图展示状态汇总(假数据,可以替换为http_status)

人员权限树(假数据,可以作为运维人员权限展示)

流量图(假数据,可以作为机房之间,或者网卡的流量)

日志统计汇总(假数据,可以用来展示年日志数据汇总)

区域数据汇总饼图展示(假数据,可选择省份,生成饼图)

3D展示机房(网上盗图,后续会做一个类似的开源)

json生成,实时查看机器状态,点击时间

3D展示不止于此 有图有XX

谢谢!

如果您觉得有我写的东西对你帮助,可以打赏点钱给我支付宝支持我,谢谢

@shengxinjing shengxinjing added the 前端 label Dec 2, 2015

@shengxinjing shengxinjing changed the title from 日志可视化(未完成) to 周日分享--日志可视化(未完成) Dec 2, 2015

@auxten

This comment has been minimized.

Show comment
Hide comment
@auxten

auxten Dec 3, 2015

蜗牛最牛逼

auxten commented Dec 3, 2015

蜗牛最牛逼

@caozhenqiang

This comment has been minimized.

Show comment
Hide comment
@caozhenqiang

caozhenqiang Dec 3, 2015

蜗牛摸摸大

蜗牛摸摸大

@anbaoyong

This comment has been minimized.

Show comment
Hide comment
@anbaoyong

anbaoyong Dec 3, 2015

好酷炫,膜拜

好酷炫,膜拜

@shengxinjing shengxinjing added 未完成 and removed 前端 labels Dec 4, 2015

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 4, 2015

Owner

@anbaoyong 快去给star

Owner

shengxinjing commented Dec 4, 2015

@anbaoyong 快去给star

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
Owner

shengxinjing commented Dec 4, 2015

@caozhenqiang 摸摸大

@shengxinjing shengxinjing changed the title from 周日分享--日志可视化(未完成) to 日志可视化(未完成) Dec 4, 2015

@shengxinjing shengxinjing changed the title from 日志可视化(未完成) to 日志可视化 Dec 6, 2015

@shengxinjing shengxinjing added python 运维 and removed 未完成 labels Dec 6, 2015

@shengxinjing shengxinjing changed the title from 日志可视化 to 日志可视化的进阶 Dec 6, 2015

@Milef

This comment has been minimized.

Show comment
Hide comment
@Milef

Milef Dec 6, 2015

为什么这么牛逼

Milef commented Dec 6, 2015

为什么这么牛逼

@yanweiran123

This comment has been minimized.

Show comment
Hide comment

greate!

@gxbjtu

This comment has been minimized.

Show comment
Hide comment

gxbjtu commented Dec 6, 2015

硬!

@xmdan

This comment has been minimized.

Show comment
Hide comment
@xmdan

xmdan Dec 7, 2015

膝盖送上!

xmdan commented Dec 7, 2015

膝盖送上!

@nqzhang

This comment has been minimized.

Show comment
Hide comment
@nqzhang

nqzhang Dec 7, 2015

为什么 逼格这么高

nqzhang commented Dec 7, 2015

为什么 逼格这么高

@seerjk

This comment has been minimized.

Show comment
Hide comment
@seerjk

seerjk Dec 7, 2015

蜗牛大大 👍

seerjk commented Dec 7, 2015

蜗牛大大 👍

@mujiatong

This comment has been minimized.

Show comment
Hide comment

不错

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 7, 2015

Owner

@mujiatong 么么哒 求个star 😄

Owner

shengxinjing commented Dec 7, 2015

@mujiatong 么么哒 求个star 😄

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 7, 2015

Owner

@Milef 么么哒

Owner

shengxinjing commented Dec 7, 2015

@Milef 么么哒

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 7, 2015

Owner

@seerjk 么么哒

Owner

shengxinjing commented Dec 7, 2015

@seerjk 么么哒

@iselu

This comment has been minimized.

Show comment
Hide comment
@iselu

iselu Dec 7, 2015

蜗牛哒哒最醋~~

iselu commented Dec 7, 2015

蜗牛哒哒最醋~~

@songkane

This comment has been minimized.

Show comment
Hide comment
@songkane

songkane Dec 8, 2015

蜗牛么么哒

songkane commented Dec 8, 2015

蜗牛么么哒

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 8, 2015

Owner

@iselu 么么哒 快给star

Owner

shengxinjing commented Dec 8, 2015

@iselu 么么哒 快给star

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 8, 2015

Owner

@songkane 摸摸大 快给star

Owner

shengxinjing commented Dec 8, 2015

@songkane 摸摸大 快给star

@wangycc

This comment has been minimized.

Show comment
Hide comment
@wangycc

wangycc Dec 9, 2015

这是只真wo牛

wangycc commented Dec 9, 2015

这是只真wo牛

@xinjian0728

This comment has been minimized.

Show comment
Hide comment
@xinjian0728

xinjian0728 Dec 9, 2015

已经star,大神膜拜啊~

已经star,大神膜拜啊~

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 9, 2015

Owner

@xinjian0728 嘿嘿 谢谢夸奖

Owner

shengxinjing commented Dec 9, 2015

@xinjian0728 嘿嘿 谢谢夸奖

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
Owner

shengxinjing commented Dec 9, 2015

@wangycc 必须的

@shengxinjing shengxinjing added the 前端 label Dec 11, 2015

@yuzhiquan

This comment has been minimized.

Show comment
Hide comment
@yuzhiquan

yuzhiquan Dec 11, 2015

下面的画图感觉已经是前端了,不过做的确实NB

下面的画图感觉已经是前端了,不过做的确实NB

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 11, 2015

Owner

@yuzhiquan 就是为了说明,相同的一份数据,怎么展现更优雅,运维很大的问题就在于,做的东西不能量化展示

Owner

shengxinjing commented Dec 11, 2015

@yuzhiquan 就是为了说明,相同的一份数据,怎么展现更优雅,运维很大的问题就在于,做的东西不能量化展示

@shengxinjing shengxinjing changed the title from 日志可视化的进阶 to python 10 min系列二之日志可视化的进阶 Dec 11, 2015

@lweitsinghua

This comment has been minimized.

Show comment
Hide comment
@lweitsinghua

lweitsinghua Dec 11, 2015

厉害啊,师兄。膜拜下

厉害啊,师兄。膜拜下

@xEasy

This comment has been minimized.

Show comment
Hide comment

xEasy commented Dec 11, 2015

卧槽

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 11, 2015

Owner

@lweitsinghua 快去给star 哈哈

Owner

shengxinjing commented Dec 11, 2015

@lweitsinghua 快去给star 哈哈

@lweitsinghua

This comment has been minimized.

Show comment
Hide comment
@lweitsinghua

lweitsinghua Dec 11, 2015

已给,在哪高就

已给,在哪高就

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Dec 11, 2015

Owner

@lweitsinghua 其他方式私聊吧,这里挺多人关注的

Owner

shengxinjing commented Dec 11, 2015

@lweitsinghua 其他方式私聊吧,这里挺多人关注的

@mingyun

This comment has been minimized.

Show comment
Hide comment
@mingyun

mingyun Dec 27, 2015

star,太酷了

mingyun commented Dec 27, 2015

star,太酷了

@shengxinjing shengxinjing added the web label Jan 6, 2016

@redfish88

This comment has been minimized.

Show comment
Hide comment
@redfish88

redfish88 Feb 4, 2016

没有看到这个项目代码啊

没有看到这个项目代码啊

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Feb 15, 2016

Owner

@redfish88 就是提供一个思路,完整代码没放上来,建议大家自己敲 哈哈

Owner

shengxinjing commented Feb 15, 2016

@redfish88 就是提供一个思路,完整代码没放上来,建议大家自己敲 哈哈

@linzhi

This comment has been minimized.

Show comment
Hide comment
@linzhi

linzhi Feb 15, 2016

已star......

linzhi commented Feb 15, 2016

已star......

@GuoJianzhu

This comment has been minimized.

Show comment
Hide comment
@GuoJianzhu

GuoJianzhu Mar 3, 2016

没有完整代码!!!!!!!!!!

没有完整代码!!!!!!!!!!

@lwhile

This comment has been minimized.

Show comment
Hide comment
@lwhile

lwhile Mar 3, 2016

@GuoJianzhu 作者就提供个思路而已

lwhile commented Mar 3, 2016

@GuoJianzhu 作者就提供个思路而已

@yoCruzer

This comment has been minimized.

Show comment
Hide comment
@yoCruzer

yoCruzer Apr 16, 2016

hi,博主,可以考虑将「前端展现上做一些优化」部分的内容细化地讲解一下么?或者是录个视频分享一下呢?

hi,博主,可以考虑将「前端展现上做一些优化」部分的内容细化地讲解一下么?或者是录个视频分享一下呢?

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Apr 17, 2016

Owner

@yoCruzer 这部分就是用了jquery+bootstrap+datatable 你可以尝试一下,不太难的,这部分不是这个文章的重点,就没说太细

Owner

shengxinjing commented Apr 17, 2016

@yoCruzer 这部分就是用了jquery+bootstrap+datatable 你可以尝试一下,不太难的,这部分不是这个文章的重点,就没说太细

@yoCruzer

This comment has been minimized.

Show comment
Hide comment
@yoCruzer

yoCruzer Apr 17, 2016

@shengxinjing 好的,我自己去试试看,谢谢你的思路~

@shengxinjing 好的,我自己去试试看,谢谢你的思路~

@whoistr

This comment has been minimized.

Show comment
Hide comment
@whoistr

whoistr May 17, 2016

来膜拜大大的

whoistr commented May 17, 2016

来膜拜大大的

@lycciam

This comment has been minimized.

Show comment
Hide comment
@lycciam

lycciam Jun 28, 2016

talk is cheap,请收下我的膝盖还有雪糕钱!

lycciam commented Jun 28, 2016

talk is cheap,请收下我的膝盖还有雪糕钱!

@imlinhao

This comment has been minimized.

Show comment
Hide comment
@imlinhao

imlinhao Jul 7, 2016

画图的代码怎么没有?

imlinhao commented Jul 7, 2016

画图的代码怎么没有?

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Jul 28, 2016

Owner

@imlinhao 本文提供了思路,代码啥的没提交

Owner

shengxinjing commented Jul 28, 2016

@imlinhao 本文提供了思路,代码啥的没提交

@lycciam

This comment has been minimized.

Show comment
Hide comment
@lycciam

lycciam Sep 17, 2016

话说有部分图看不到了...是图床被墙了还是出什么问题呢?求瞧瞧

lycciam commented Sep 17, 2016

话说有部分图看不到了...是图床被墙了还是出什么问题呢?求瞧瞧

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
@shengxinjing

shengxinjing Sep 18, 2016

Owner

@lycciam 七牛云账号欠费了,刚充值了 可以看到了

Owner

shengxinjing commented Sep 18, 2016

@lycciam 七牛云账号欠费了,刚充值了 可以看到了

@jxauwxj

This comment has been minimized.

Show comment
Hide comment
@jxauwxj

jxauwxj Oct 13, 2016

请问这个炫的动态图,是用什么软件做出来的?

jxauwxj commented Oct 13, 2016

请问这个炫的动态图,是用什么软件做出来的?

@shengxinjing

This comment has been minimized.

Show comment
Hide comment
Owner

shengxinjing commented Oct 24, 2016

@jxauwxj Echarts

@orangle

This comment has been minimized.

Show comment
Hide comment
@orangle

orangle Nov 20, 2016

echarts 地图相关的api过了一遍

orangle commented Nov 20, 2016

echarts 地图相关的api过了一遍

@Everfighting

This comment has been minimized.

Show comment
Hide comment
@Everfighting

Everfighting Jan 19, 2017

帅爆啦~!

帅爆啦~!

@disanji789

This comment has been minimized.

Show comment
Hide comment
@disanji789

disanji789 Mar 24, 2017

蜗牛大大 牛掰

蜗牛大大 牛掰

@2015qyliang

This comment has been minimized.

Show comment
Hide comment

牛牛牛

@Cian-Liu

This comment has been minimized.

Show comment
Hide comment
@Cian-Liu

Cian-Liu May 23, 2017

膜拜大蟒蛇~

膜拜大蟒蛇~

@tianbao666

This comment has been minimized.

Show comment
Hide comment
@tianbao666

tianbao666 Sep 10, 2017

我操,好厉害啊哦

我操,好厉害啊哦

@snake3342

This comment has been minimized.

Show comment
Hide comment

屌炸天

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment