-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
1115 lines (714 loc) · 314 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>茶包哥的迷你仓</title>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
<meta name="description" content="专注于云/大数据/容器化方向">
<meta property="og:type" content="website">
<meta property="og:title" content="茶包哥的迷你仓">
<meta property="og:url" content="https://theodorew.github.io/wxsblog.github.io/index.html">
<meta property="og:site_name" content="茶包哥的迷你仓">
<meta property="og:description" content="专注于云/大数据/容器化方向">
<meta property="og:locale">
<meta property="article:author" content="Xinsheng Wang">
<meta name="twitter:card" content="summary">
<link rel="alternative" href="/atom.xml" title="茶包哥的迷你仓" type="application/atom+xml">
<link rel="icon" href="/favicon.png">
<link rel="stylesheet" href="/wxsblog.github.io/css/style.css">
<!--[if lt IE 9]><script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7/html5shiv.min.js"></script><![endif]-->
<meta name="generator" content="Hexo 6.0.0"></head>
<body>
<div id="container">
<div id="wrap">
<header id="header">
<div id="banner"></div>
<div id="header-outer" class="outer">
<div id="header-title" class="inner">
<h1 id="logo-wrap">
<a href="/wxsblog.github.io/" id="logo">茶包哥的迷你仓</a>
</h1>
</div>
<div id="header-inner" class="inner">
<nav id="main-nav">
<a id="main-nav-toggle" class="nav-icon"></a>
<a class="main-nav-link" href="/wxsblog.github.io/">Home</a>
<a class="main-nav-link" href="/wxsblog.github.io/archives">Archives</a>
</nav>
<nav id="sub-nav">
<a id="nav-rss-link" class="nav-icon" href="/atom.xml" title="RSS Feed"></a>
<a id="nav-search-btn" class="nav-icon" title="Search"></a>
</nav>
<div id="search-form-wrap">
<form action="//www.baidu.com/baidu" method="get" accept-charset="utf-8" class="search-form">
<input type="search" name="word" maxlength="20" class="search-form-input" placeholder="Search">
<input type="submit" value="" class="search-form-submit">
<input name=tn type=hidden value="bds">
<input name=cl type=hidden value="3">
<input name=ct type=hidden value="2097152">
<input type="hidden" name="si" value="theodorew.github.io/wxsblog.github.io">
</form>
</div>
</div>
</div>
</header>
<div class="outer">
<section id="main">
<article id="post-2022-11-11-AzureMySQLFlexibleServerNetworking" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2022/11/11/2022-11-11-AzureMySQLFlexibleServerNetworking/" class="article-date">
<time datetime="2022-11-11T06:45:17.000Z" itemprop="datePublished">2022-11-11</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/DataBase/">DataBase</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/DataBase/MySQL/">MySQL</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/DataBase/MySQL/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2022/11/11/2022-11-11-AzureMySQLFlexibleServerNetworking/">Azure MySQL Flexible Server 内网 DNS 集成方案</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p>最近一直在和客户聊云上资源迁移的相关事宜，借着客户的需求测试了一下 Azure MySQL Flexible Server ( <strong>后文简称 MySQLFS</strong> ) 集成 Custom DNS 的场景，其实<a target="_blank" rel="noopener" href="https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-networking">官方文档</a>是有一些说明的，不过为了使方案更直观，还是记录下来方便以后查询，包括其中遇到的“坑”。</p>
<br>
<h2 id="2-需求背景介绍"><a href="#2-需求背景介绍" class="headerlink" title="2. 需求背景介绍"></a>2. 需求背景介绍</h2><p>客户原本用的是 Azure MySQL Single Server，现要迁移 MySQLFS，由于 MySQLFS 的网络集成模型和 Single Server 有一些区别，所以在迁移的过程中顺便改造成更安全的内网集成方案。与之前不同的是，MySQLFS 支持 VNet integration 这种 Private access 网络集成方式，这种方式可以让 Azure MySQL 实例只拥有一个 Private IP，以 VNet Injection 的方式集成到 VNet 中，具体的网络部署模型可以参考以下这张图：</p>
<p><a href="Vnet-Diagram.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/Vnet-Diagram.png" height="600" width="600"/></a></p>
<p>需求是在 VNet Peering 互联的前提下，能否在对端 VNet 中的 MySQL Client 使用 Custom DNS 配置来成功访问 VNet Integration MySQLFS，那答案肯定是可以的，不然这篇 Blog 也就没有存在的意义了，好了废话不多说了，直接开搞。</p>
<br>
<h2 id="3-Azure-MySQL-Flexible-Server-内网集成-Custom-DNS-测试"><a href="#3-Azure-MySQL-Flexible-Server-内网集成-Custom-DNS-测试" class="headerlink" title="3. Azure MySQL Flexible Server 内网集成 Custom DNS 测试"></a>3. Azure MySQL Flexible Server 内网集成 Custom DNS 测试</h2><p>具体的测试环境如下图所示：</p>
<p><a href="Arch1.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/Arch1.png" height="800" width="800"/></a></p>
<p>简单介绍下架构: VNet1 和 VNet2 通过 VNet Peering 打通，分别划分两个 Subnet for DNS Server 和 MySQLFS 实例，每个 Subnet 分别挂 NSG，每个 VNet 有 Custom DNS Server，对应 <strong>vnet1-vmsn1-dns1 10.11.1.4</strong> 和 <strong>vnet2-vmsn1-dns1 10.12.1.4</strong>，系统使用 CentOS 7.9，Custom Domain 为 <strong>example.com</strong>，所有资源都部署在 EastUS 进行测试，具体信息如以下几张图所示：</p>
<p><a href="VNet1-Subnets.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet1-Subnets.jpg" height="800" width="800"/></a></p>
<p><a href="VNet2-Subnets.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet2-Subnets.jpg" height="800" width="800"/></a></p>
<p><a href="VNet-Peering.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet-Peering.jpg" height="800" width="800"/></a></p>
<p><font color=green><strong>接下来我们来测试两个场景：</strong></font><br><font color=red><strong>1.</strong></font> <strong>启用 Private Access Azure MySQLFS 实例在对端 VNet Peering Client 进行访问；</strong><br><font color=red><strong>2.</strong></font> <strong>使用自定义域名访问 Private Access Azure MySQLFS 实例；</strong></p>
<h3 id="3-1-启用-Private-Access-Azure-MySQLFS-实例在对端-VNet-Peering-Client-进行访问"><a href="#3-1-启用-Private-Access-Azure-MySQLFS-实例在对端-VNet-Peering-Client-进行访问" class="headerlink" title="3.1 启用 Private Access Azure MySQLFS 实例在对端 VNet Peering Client 进行访问"></a>3.1 启用 Private Access Azure MySQLFS 实例在对端 VNet Peering Client 进行访问</h3><p>首先先创建一个 Azure MySQLFS Private 实例，Azure 默认给实例分配的内网地址为 <strong>10.12.2.4</strong>（ DB Subnet 第一个内网地址 ），同样地，Azure 也会给实例分配一个 Private DNS Zone，本实验采用创建实例时 Azure 默认分配的 Zone：<strong>vnet2-dbsn1-mysqlfs1.private.mysql.database.azure.com</strong>。 其实本质上，Azure 也是通过域名解析来集成 MySQLFS 内网服务的，当创建好实例之后，就会在域名解析服务中增加一条 A Record，如下图所示：</p>
<p><a href="MySQLFS-DNS-Record.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/MySQLFS-DNS-Record.jpg" height="800" width="800"/></a></p>
<p>此实例的 Server Name 前缀名为：<strong>vnet2-dbsn1-mysqlfs1</strong>，部署在 VNet2 DB Subnet 中，也就是说和 <strong>vnet2-vmsn1-dns1</strong> 所处在一个 VNet 中。再不考虑 NSG 的前提条件下（ 也就是说同 VNet 流量出入都放行 ），Client 直接就可以解析 Azure MySQLFS 的域名了。那下一个问题可能又来了，Azure MySQLFS 本身有一个 FQDN，那和这个 Private 域名有什么关系呢？我们来验证一下：</p>
<p><a href="VNet2-Client-DNS-Resolution.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet2-Client-DNS-Resolution.jpg" height="800" width="800"/></a></p>
<p>通过 nslookup 的解析结果能看到，默认的 FQDN：<strong>vnet2-dbsn1-mysqlfs1.mysql.database.azure.com</strong> 是一个 CNAME 指向了 Azure MySQLFS 的内网域名上：<strong>vnet2-dbsn1-mysqlfs1.vnet2-dbsn1-mysqlfs1.private.mysql.database.azure.com</strong>，然后内网域名在 Azure Private DNS Zone 有一条 A 记录解析到了实际的内网地址 <strong>10.12.2.4</strong>。那启用了 VNet Peering 的对端 Client 的解析结果又如何呢？在 <strong>vnet1-vmsn1-dns1 10.11.1.4</strong> 上测试下：</p>
<p><a href="VNet1-Client-DNS-Resolution-Without-Private-DNS-Link.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet1-Client-DNS-Resolution-Without-Private-DNS-Link.jpg" height="800" width="800"/></a></p>
<p>很明显，只做了 VNet Peering 的对端 Client 没法直接解析，究其原因是对端 Client 并不知道 private zone 是什么，所以需要将 <strong>VNet1</strong> 和 <strong>vnet2-dbsn1-mysqlfs1.private.mysql.database.azure.com</strong> Link 起来，VNet Peering 对端的 Client 就可以和 VNet2 内一样解析了。</p>
<p><a href="VNet-Private-DNS-Link.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet-Private-DNS-Link.jpg" height="800" width="800"/></a></p>
<p>同理，也就明白了为什么 VNet2 默认就可以解析，也是因为 Azure 在创建 Private DNS Zone 的同时加了一条 VNet2 Link，如图标红出所示。</p>
<h3 id="3-2-使用自定义域名访问-Private-Access-Azure-MySQLFS-实例"><a href="#3-2-使用自定义域名访问-Private-Access-Azure-MySQLFS-实例" class="headerlink" title="3.2 使用自定义域名访问 Private Access Azure MySQLFS 实例"></a>3.2 使用自定义域名访问 Private Access Azure MySQLFS 实例</h3><p>3.1 我们验证了使用默认的 Azure DNS Domain 来进行 Azure MySQLFS 的内网访问，那如果使用 Custom Domain 比如 <strong>example.com</strong> 是否可以做集成呢？答案也是可以的，主要验证在对端 Peering VNet 使用 Custom DNS Server 的集成情况。首先在 <strong>vnet1-vmsn1-dns1 10.11.1.4</strong> 上部署 <strong>DNS Server Linux Bind</strong>。Bind 就不多做介绍了，很经典的 Linux DNS Server Solution，部署过程不赘述了，不了解的同学们自行 Google 吧。重点主要在 3 个配置文件的配置上：</p>
<p><font color=blue><strong>1. /etc/named.conf</strong></font><br><a href="Named-Config-File.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/Named-Config-File.jpg" height="800" width="800"/></a><br>DNS Forwarders 需要转发给 Azure Default DNS：168.63.129.16。</p>
<p><font color=blue><strong>2. /etc/resolv.conf</strong></font><br><a href="VNet1-Resolv-Conf.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet1-Resolv-Conf.jpg" height="800" width="800"/></a><br>系统 DNS Resolve File 要 search example.com domain 以及修改 nameserver 为 Local DNS Server 地址 10.11.1.4。</p>
<p><font color=blue><strong>3. /var/named/example.com.zone</strong></font><br><a href="VNet1-DNS1-Bind-CNAME.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet1-DNS1-Bind-CNAME.jpg" height="800" width="800"/></a><br>正向解析文件 <strong>example.com.zone</strong> 需要加一条 CNAME Record，如上图标红框所示。在 3.1 的基础上，可以直接通过 Custom Domain 的内网域名来做测试了：</p>
<p><a href="VNet1-Custom-Domain-DNS-Lookup.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/VNet1-Custom-Domain-DNS-Lookup.jpg" height="800" width="800"/></a></p>
<p>能够看出来，自定义域名 <strong>vnet2-dbsn1-mysqlfs1.example.com</strong> CNAME 指向了 Azure MySQLFS FQDN，然后 FQDN CNAME 指向了最终的内网域名 <strong>vnet2-dbsn1-mysqlfs1.private.mysql.database.azure.com</strong> 并最终解析成内网地址 10.12.2.4。</p>
<br>
<h2 id="4-总结"><a href="#4-总结" class="headerlink" title="4. 总结"></a>4. 总结</h2><p>至此，Azure MySQLFS 内网集成的两个场景就测试完毕了，其实大家应该能够感觉到 VNet Injection 这种网络模型其实基本上就是 VNet 里面的 VM，只是封装了一层变成 PaaS 服务暴露给用户，所以这么看的话大家是不是也知道网络层面的管控可以怎么做了？没错，就可以当作 VM 一样，通过 NSG 来限制出入口流量，这个测试下来也是没问题的，不过本篇就不做详细测试了，有需求的同学自己来做测试吧。最后再稍微提几个注意事项给到大家：</p>
<p><font color=Red><strong>1. 关于 Read Replica</strong></font><br><a href="MySQLFS-Read-Replica.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/MySQLFS-Read-Replica.jpg" height="800" width="800"/></a><br><a href="MySQLFS-Read-Replica-DNS-Lookup.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2022-11-11-AzureMySQLFlexibleServerNetworking/MySQLFS-Read-Replica-DNS-Lookup.jpg" height="800" width="800"/></a><br>创建了两个只读副本，和 Master 一样，Azure 会根据 DHCP 继续向下分配内网地址给到只读副本，然后生成 A Record 解析。</p>
<p><font color=Red><strong>2. 关于访问方式</strong></font><br>和研发和后台支持的同学们聊下来，无论是使用 FQDN 亦或 Custom DNS Domain Name，都强烈不建议使用内网 IP 来访问实例，在进行一些平台层面的维护或者启用 HA 的实例，都会发生 IP Floating，所以生产环境一定要用域名。</p>
<p><font color=Red><strong>3. 关于数据迁移</strong></font><br>Azure DMS Service 针对 SingleServer - Flexbile Server 又推出了一个 online 的迁移方式，本来 online 这个功能都下线了，可能是为了方便客户做迁移，单独针对 MySQLFS 上线了，这个功能在这里提一句，后面我们也测试看看实际效果怎么样。</p>
<p><font color=Red><strong>4. 域名解析</strong></font><br>以 VNet Injection 方式嵌入的 Azure MySQLFS 在域名解析上和其他云厂商有些许不同，经过测试发现非 VNet 环境是无法解析 MySQLFS 的内网域名的，像 AWS RDS 就不一样了，是可以在公网解析的，解析成 Private IP，只是无法访问罢了。</p>
<p>暂时就能想到这些，先写这么多给大家参考吧，后面有啥补充我再查缺补漏，谢谢大家。</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2022/11/11/2022-11-11-AzureMySQLFlexibleServerNetworking/" data-id="clat72drt006jyofkgi200wb8" class="article-share-link" data-share="baidu" data-title="Azure MySQL Flexible Server 内网 DNS 集成方案">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/DataBase/" rel="tag">DataBase</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/MySQL/" rel="tag">MySQL</a></li></ul>
</footer>
</div>
</article>
<article id="post-2021-07-26-AzureImageBuilderCLSIGI" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2021/07/26/2021-07-26-AzureImageBuilderCLSIGI/" class="article-date">
<time datetime="2021-07-26T15:33:56.000Z" itemprop="datePublished">2021-07-26</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/">infrastructure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/Automation/">Automation</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/Automation/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2021/07/26/2021-07-26-AzureImageBuilderCLSIGI/">Azure Image Builder（二）之自动化构建自定义托管镜像 CentOS 7.7 并集成 Azure Shared Image Gallery 做全球分发</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p><a href="https://theodorew.github.io/wxsblog.github.io/2021/07/25/2021-07-25-AzureImageBuilderCLMI/">上一篇</a>我们测试了 AIB 自动化构建 CentOS 7.7 自定义镜像，相信大家也体会到了云上原生服务带来的自动化便利度。其实，AIB 能够做到的远远不止这些，除此之外还可以集成 Vnet 以及 RHEL 等 License 等功能，尤其值得一提的是和 Azure Shared Image Gallery（后文简称 SIG ）的服务集成。对于 SIG 本文不多做介绍，不了解的同学们通过<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/virtual-machines/shared-image-galleries">官方文档</a>自行科普吧，简单来说就是一个可以做到全球管理分发虚拟机镜像的 Azure 云服务，该服务构成可以参考下图：</p>
<p><a href="Shared-Image-Gallery.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2021-07-26-AzureImageBuilderCLSIGI/Shared-Image-Gallery.png" height="700" width="700"/></a></p>
<p>本文，我们来测试下 AIB 和 SIG 的集成，验证下自动化构建全球虚拟机镜像的功能。</p>
<br>
<h2 id="2-前期准备工作"><a href="#2-前期准备工作" class="headerlink" title="2. 前期准备工作"></a>2. 前期准备工作</h2><p>和第一篇博客相同，需要准备好 Global Azure 账户，配置好 Azure CLI，该测试在 Windows Subsystem v2 Ubuntu 18.04 上运行，需要注意该实验同样需要在一个 Session 内运行，因为要继承所有设置的自定义变量。</p>
<br>
<h2 id="3-AIB-构建自定义-CentOS-7-7-镜像并集成-Azure-Shared-Image-Gallery-做全球分发"><a href="#3-AIB-构建自定义-CentOS-7-7-镜像并集成-Azure-Shared-Image-Gallery-做全球分发" class="headerlink" title="3. AIB 构建自定义 CentOS 7.7 镜像并集成 Azure Shared Image Gallery 做全球分发"></a>3. AIB 构建自定义 CentOS 7.7 镜像并集成 Azure Shared Image Gallery 做全球分发</h2><h3 id="3-1-AIB-x2F-VM-x2F-Storage-Feature-注册"><a href="#3-1-AIB-x2F-VM-x2F-Storage-Feature-注册" class="headerlink" title="3.1 AIB / VM / Storage Feature 注册"></a>3.1 AIB / VM / Storage Feature 注册</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Feature Registration</span></span><br><span class="line">az feature register --namespace Microsoft.VirtualMachineImages --name VirtualMachineTemplatePreview</span><br><span class="line">az provider register -n Microsoft.VirtualMachineImages</span><br><span class="line"></span><br><span class="line">az feature show --namespace Microsoft.VirtualMachineImages --name VirtualMachineTemplatePreview | grep state</span><br><span class="line"></span><br><span class="line"><span class="comment"># register and enable for shared image gallery</span></span><br><span class="line">az feature register --namespace Microsoft.Compute --name GalleryPreview</span><br><span class="line">az provider register -n Microsoft.Compute</span><br><span class="line"></span><br><span class="line">az feature show --namespace Microsoft.Compute --name GalleryPreview | grep state</span><br><span class="line"></span><br><span class="line"><span class="comment"># wait until it says registered</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># check you are registered for the providers</span></span><br><span class="line"></span><br><span class="line">az provider show -n Microsoft.VirtualMachineImages | grep registrationState</span><br><span class="line">az provider show -n Microsoft.Storage | grep registrationState</span><br><span class="line">az provider show -n Microsoft.Compute | grep registrationState</span><br><span class="line">az provider show -n Microsoft.KeyVault | grep registrationState</span><br></pre></td></tr></table></figure>
<p>如果命令输出结果显示相关 feature 没注册，则运行以下命令：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">az provider register -n Microsoft.VirtualMachineImages</span><br><span class="line">az provider register -n Microsoft.Storage</span><br><span class="line">az provider register -n Microsoft.Compute</span><br><span class="line">az provider register -n Microsoft.KeyVault</span><br></pre></td></tr></table></figure>
<p>创建 RG 并设置相关环境变量：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Create SIG resource group</span></span><br><span class="line">sigResourceGroup=aibsigrg</span><br><span class="line"></span><br><span class="line"><span class="comment"># location of SIG (see possible locations in main docs)</span></span><br><span class="line">location=southeastasia</span><br><span class="line"></span><br><span class="line"><span class="comment"># additional region to replication image to</span></span><br><span class="line">additionalregion=eastus</span><br><span class="line"></span><br><span class="line"><span class="comment"># your subscription</span></span><br><span class="line"><span class="comment"># get the current subID : 'az account show | grep id'</span></span><br><span class="line">subscriptionID=$(az account show | grep <span class="built_in">id</span> | <span class="built_in">tr</span> -d <span class="string">'",'</span> | <span class="built_in">cut</span> -c7-)</span><br><span class="line"></span><br><span class="line"><span class="comment"># name of the shared image gallery, e.g. myCorpGallery</span></span><br><span class="line">sigName=aibsig01</span><br><span class="line"></span><br><span class="line"><span class="comment"># name of the image definition to be created, e.g. ProdImages</span></span><br><span class="line">imageDefName=aib01sig01centos77image01def01</span><br><span class="line"></span><br><span class="line"><span class="comment"># image distribution metadata reference name</span></span><br><span class="line">runOutputName=aib01sig01centos77image01ro01</span><br><span class="line"></span><br><span class="line"><span class="comment"># create resource group</span></span><br><span class="line">az group create -n <span class="variable">$sigResourceGroup</span> -l <span class="variable">$location</span></span><br></pre></td></tr></table></figure>
<h3 id="3-2-创建-User-Assigned-Managed-Identity-并赋权"><a href="#3-2-创建-User-Assigned-Managed-Identity-并赋权" class="headerlink" title="3.2 创建 User-Assigned Managed Identity 并赋权"></a>3.2 创建 User-Assigned Managed Identity 并赋权</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># create user assigned identity for image builder to access the storage account where the script is located</span></span><br><span class="line">idenityName=aibsig01uami01</span><br><span class="line">az identity create -g <span class="variable">$sigResourceGroup</span> -n <span class="variable">$idenityName</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># get identity id</span></span><br><span class="line">aibsig01uami01id=$(az identity show -g <span class="variable">$sigResourceGroup</span> -n <span class="variable">$idenityName</span> | grep <span class="string">"clientId"</span> | <span class="built_in">cut</span> -c16- | <span class="built_in">tr</span> -d <span class="string">'",'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># get the user identity URI, needed for the template</span></span><br><span class="line">aibsig01uami01uri=/subscriptions/<span class="variable">$subscriptionID</span>/resourcegroups/<span class="variable">$sigResourceGroup</span>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<span class="variable">$idenityName</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># download preconfigured role definition example</span></span><br><span class="line">curl https://raw.githubusercontent.com/TheoDoreW/wxsblog.github.io/master/2021/07/26/2021-07-26-AzureImageBuilderCLSIGI/template/AIBRoleImageCreation.json -o AIBRoleImageCreation.json</span><br><span class="line"></span><br><span class="line">imageRoleDefName=<span class="string">"Azure Image Builder Image Def01"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># update the definition</span></span><br><span class="line">sed -i -e <span class="string">"s/<subscriptionID>/<span class="variable">$subscriptionID</span>/g"</span> AIBRoleImageCreation.json</span><br><span class="line">sed -i -e <span class="string">"s/<rgName>/<span class="variable">$imageResourceGroup</span>/g"</span> AIBRoleImageCreation.json</span><br><span class="line">sed -i -e <span class="string">"s/Azure Image Builder Service Image Creation Role/<span class="variable">$imageRoleDefName</span>/g"</span> AIBRoleImageCreation.json</span><br><span class="line"></span><br><span class="line"><span class="comment"># create role definitions</span></span><br><span class="line">az role definition create --role-definition ./AIBRoleImageCreation.json</span><br><span class="line"></span><br><span class="line"><span class="comment"># grant role definition to the user assigned identity</span></span><br><span class="line">az role assignment create \</span><br><span class="line"> --assignee <span class="variable">$aibsig01uami01id</span> \</span><br><span class="line"> --role <span class="string">"<span class="variable">$imageRoleDefName</span>"</span> \</span><br><span class="line"> --scope /subscriptions/<span class="variable">$subscriptionID</span>/resourceGroups/<span class="variable">$sigResourceGroup</span></span><br></pre></td></tr></table></figure>
<p>创建 Azure Shared Image Gallery：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># create SIG</span></span><br><span class="line">az sig create \</span><br><span class="line"> -g <span class="variable">$sigResourceGroup</span> \</span><br><span class="line"> --gallery-name <span class="variable">$sigName</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># create SIG image definition</span></span><br><span class="line">az sig image-definition create \</span><br><span class="line"> -g <span class="variable">$sigResourceGroup</span> \</span><br><span class="line"> --gallery-name <span class="variable">$sigName</span> \</span><br><span class="line"> --gallery-image-definition <span class="variable">$imageDefName</span> \</span><br><span class="line"> --publisher aibpublisher \</span><br><span class="line"> --offer aiboffer \</span><br><span class="line"> --sku 7.7 \</span><br><span class="line"> --os-type Linux</span><br></pre></td></tr></table></figure>
<h3 id="3-3-修改-AIB-SIG-CentOS77-模板文件"><a href="#3-3-修改-AIB-SIG-CentOS77-模板文件" class="headerlink" title="3.3 修改 AIB SIG CentOS77 模板文件"></a>3.3 修改 AIB SIG CentOS77 模板文件</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># download the example and configure it with your vars</span></span><br><span class="line">curl https://raw.githubusercontent.com/TheoDoreW/wxsblog.github.io/master/2021/07/26/2021-07-26-AzureImageBuilderCLSIGI/template/SIGCentOS77AIBTemplate.json -o SIGCentOS77AIBTemplate.json</span><br><span class="line"></span><br><span class="line">sed -i -e <span class="string">"s/<subscriptionID>/<span class="variable">$subscriptionID</span>/g"</span> SIGCentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<rgName>/<span class="variable">$sigResourceGroup</span>/g"</span> SIGCentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<imageDefName>/<span class="variable">$imageDefName</span>/g"</span> SIGCentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<sharedImageGalName>/<span class="variable">$sigName</span>/g"</span> SIGCentOS77AIBTemplate.json</span><br><span class="line"></span><br><span class="line">sed -i -e <span class="string">"s/<region1>/<span class="variable">$location</span>/g"</span> SIGCentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<region2>/<span class="variable">$additionalregion</span>/g"</span> SIGCentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<runOutputName>/<span class="variable">$runOutputName</span>/g"</span> SIGCentOS77AIBTemplate.json</span><br><span class="line"></span><br><span class="line">sed -i -e <span class="string">"s%<aibsig01uami01uri>%<span class="variable">$aibsig01uami01uri</span>%g"</span> SIGCentOS77AIBTemplate.json</span><br></pre></td></tr></table></figure>
<h3 id="3-4-创建-AIB-SIG-CentOS-7-7-镜像"><a href="#3-4-创建-AIB-SIG-CentOS-7-7-镜像" class="headerlink" title="3.4 创建 AIB SIG CentOS 7.7 镜像"></a>3.4 创建 AIB SIG CentOS 7.7 镜像</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># submit the image confiuration to the VM Image Builder Service</span></span><br><span class="line"></span><br><span class="line">az resource create \</span><br><span class="line"> --resource-group <span class="variable">$sigResourceGroup</span> \</span><br><span class="line"> --properties @SIGCentOS77AIBTemplate.json \</span><br><span class="line"> --is-full-object \</span><br><span class="line"> --resource-type Microsoft.VirtualMachineImages/imageTemplates \</span><br><span class="line"> -n SIG01CentOS77AIB01</span><br><span class="line"></span><br><span class="line"><span class="comment"># wait approx 1-3mins, depending on external links</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># start the image build</span></span><br><span class="line"></span><br><span class="line">az resource invoke-action \</span><br><span class="line"> --resource-group <span class="variable">$sigResourceGroup</span> \</span><br><span class="line"> --resource-type Microsoft.VirtualMachineImages/imageTemplates \</span><br><span class="line"> -n SIG01CentOS77AIB01 \</span><br><span class="line"> --action Run </span><br><span class="line"></span><br><span class="line"><span class="comment"># wait approx 15mins</span></span><br></pre></td></tr></table></figure>
<h3 id="3-5-创建-CentOS-7-7-VM-并登陆验证"><a href="#3-5-创建-CentOS-7-7-VM-并登陆验证" class="headerlink" title="3.5 创建 CentOS 7.7 VM 并登陆验证"></a>3.5 创建 CentOS 7.7 VM 并登陆验证</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">az vm create \</span><br><span class="line"> --resource-group <span class="variable">$sigResourceGroup</span> \</span><br><span class="line"> --name centos01 \</span><br><span class="line"> --admin-username centos \</span><br><span class="line"> --location <span class="variable">$location</span> \</span><br><span class="line"> --image <span class="string">"/subscriptions/<span class="variable">$subscriptionID</span>/resourceGroups/<span class="variable">$sigResourceGroup</span>/providers/Microsoft.Compute/galleries/<span class="variable">$sigName</span>/images/<span class="variable">$imageDefName</span>/versions/latest"</span> \</span><br><span class="line"> --ssh-key-value @id_rsa.pub</span><br><span class="line"></span><br><span class="line"><span class="comment"># and login...</span></span><br><span class="line"></span><br><span class="line">ssh centos@<pubIp></span><br><span class="line"></span><br><span class="line">You should see the image was customized with a Message of the Day as soon as your SSH connection is established!</span><br><span class="line"></span><br><span class="line">******************************************************************</span><br><span class="line">** This VM was built from the: **</span><br><span class="line">** !! AZURE VM IMAGE BUILDER Custom CentOS 7.7 Image !! **</span><br><span class="line">** You have just been Customized :-) **</span><br><span class="line">******************************************************************</span><br></pre></td></tr></table></figure>
<br>
<h2 id="4-总结"><a href="#4-总结" class="headerlink" title="4. 总结"></a>4. 总结</h2><p>至此，第二篇自动化构建自定义托管镜像 CentOS 7.7 并集成 Azure Shared Image Gallery 做全球分发的示例就完成了，希望能够给大家一些参考。</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2021/07/26/2021-07-26-AzureImageBuilderCLSIGI/" data-id="clat72drf000pyofk4rty4628" class="article-share-link" data-share="baidu" data-title="Azure Image Builder（二）之自动化构建自定义托管镜像 CentOS 7.7 并集成 Azure Shared Image Gallery 做全球分发">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Automation/" rel="tag">Automation</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/infrastructure/" rel="tag">infrastructure</a></li></ul>
</footer>
</div>
</article>
<article id="post-2021-07-25-AzureImageBuilderCLMI" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2021/07/25/2021-07-25-AzureImageBuilderCLMI/" class="article-date">
<time datetime="2021-07-25T07:07:53.000Z" itemprop="datePublished">2021-07-25</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/">infrastructure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/Automation/">Automation</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/Automation/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2021/07/25/2021-07-25-AzureImageBuilderCLMI/">Azure Image Builder（一）之自动化构建自定义托管镜像 CentOS 7.7</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p>云计算的自动化一直都是一个热门话题，无论 IaaS / PaaS / SaaS 都正在以 Everything As A Code 的方向发展，同时也诞生了很多优秀的云原生或第三方开源自动化工具，<a target="_blank" rel="noopener" href="https://www.packer.io/">HashiCorp Packer</a> 就是其中之一，一个镜像即代码的自动化构建镜像工具。相比开源界，云厂商更没有闲着，我软也不例外，最近在钻研 <a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/aks/">Azure Kubernetes Service</a> 的同时发现了 Global Azure 目前在 Preview 的一个服务：<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/virtual-machines/image-builder-overview">Azure Image Builder</a>（后文简称AIB）。该服务是构建在 Packer 上的 Azure 云托管服务，用户从基于 Windows 或 Linux 的 Azure 市场映像、现有自定义映像开始，添加自己的自定义项，只需提供一个描述映像的配置，将其提交给该服务，即可生成映像并进行分发，具体 Pipeline 如下图所示。好了，话不多说，第一篇博客我们来测测如何利用这个服务来构建自定义托管的 CentOS 7.7 镜像吧。</p>
<p><a href="Image-Builder-Flow.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2021-07-25-AzureImageBuilderCLMI/Image-Builder-Flow.png" height="800" width="800"/></a></p>
<br>
<h2 id="2-前期准备工作"><a href="#2-前期准备工作" class="headerlink" title="2. 前期准备工作"></a>2. 前期准备工作</h2><p>目前 AIB 依然是 Global Azure Preview Service，所以还存在一些限制，包括 Location 以及 OS 版本等，具体在使用 AIB 之前建议充分查看<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/virtual-machines/image-builder-overview">官方文档</a>。本文我们在 Global Azure Region Southeastasia 测试 CentOS 7.7，需要准备好 Global Azure 账户，配置好 Azure CLI，该测试在 Windows Subsystem v2 Ubuntu 18.04 上运行，需要注意该实验需要在一个 Session 内运行，因为要继承所有设置的自定义变量。</p>
<br>
<h2 id="3-AIB-构建自定义-CentOS-7-7-镜像"><a href="#3-AIB-构建自定义-CentOS-7-7-镜像" class="headerlink" title="3. AIB 构建自定义 CentOS 7.7 镜像"></a>3. AIB 构建自定义 CentOS 7.7 镜像</h2><h3 id="3-1-AIB-x2F-VM-x2F-Storage-Feature-注册"><a href="#3-1-AIB-x2F-VM-x2F-Storage-Feature-注册" class="headerlink" title="3.1 AIB / VM / Storage Feature 注册"></a>3.1 AIB / VM / Storage Feature 注册</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Feature Registration</span></span><br><span class="line">az feature register --namespace Microsoft.VirtualMachineImages --name VirtualMachineTemplatePreview</span><br><span class="line">az provider register -n Microsoft.VirtualMachineImages</span><br><span class="line"></span><br><span class="line">az feature show --namespace Microsoft.VirtualMachineImages --name VirtualMachineTemplatePreview | grep state</span><br><span class="line"></span><br><span class="line">az feature show --namespace Microsoft.KeyVault --name VirtualMachineTemplatePreview | grep state</span><br><span class="line"></span><br><span class="line"><span class="comment"># wait until it says registered</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># check you are registered for the providers</span></span><br><span class="line"></span><br><span class="line">az provider show -n Microsoft.VirtualMachineImages | grep registrationState</span><br><span class="line">az provider show -n Microsoft.Storage | grep registrationState</span><br><span class="line">az provider show -n Microsoft.Compute | grep registrationState</span><br><span class="line">az provider show -n Microsoft.KeyVault | grep registrationState</span><br></pre></td></tr></table></figure>
<p>如果命令输出结果显示相关 feature 没注册，则运行以下命令：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">az provider register -n Microsoft.VirtualMachineImages</span><br><span class="line">az provider register -n Microsoft.Storage</span><br><span class="line">az provider register -n Microsoft.Compute</span><br><span class="line">az provider register -n Microsoft.KeyVault</span><br></pre></td></tr></table></figure>
<p>创建 RG 并设置相关环境变量：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># destination image resource group</span></span><br><span class="line">imageResourceGroup=aibrg</span><br><span class="line"></span><br><span class="line"><span class="comment"># location (see possible locations in main docs)</span></span><br><span class="line">location=southeastasia</span><br><span class="line"></span><br><span class="line"><span class="comment"># your subscription</span></span><br><span class="line"><span class="comment"># get the current subID : 'az account show | grep id'</span></span><br><span class="line">subscriptionID=$(az account show | grep <span class="built_in">id</span> | <span class="built_in">tr</span> -d <span class="string">'",'</span> | <span class="built_in">cut</span> -c7-)</span><br><span class="line"></span><br><span class="line"><span class="comment"># name of the image to be created</span></span><br><span class="line">imageName=centos77image01</span><br><span class="line"></span><br><span class="line"><span class="comment"># image distribution metadata reference name</span></span><br><span class="line">runOutputName=centos77image01ro</span><br><span class="line"></span><br><span class="line"><span class="comment"># create resource group</span></span><br><span class="line">az group create -n <span class="variable">$imageResourceGroup</span> -l <span class="variable">$location</span></span><br></pre></td></tr></table></figure>
<p>创建 User-Assigned Managed Identity 并赋权：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># create user assigned identity for image builder to access the storage account where the script is located</span></span><br><span class="line">idenityName=aibuami01</span><br><span class="line">az identity create -g <span class="variable">$imageResourceGroup</span> -n <span class="variable">$idenityName</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># get identity id</span></span><br><span class="line">aibuami01id=$(az identity show -g <span class="variable">$imageResourceGroup</span> -n <span class="variable">$idenityName</span> | grep <span class="string">"clientId"</span> | <span class="built_in">cut</span> -c16- | <span class="built_in">tr</span> -d <span class="string">'",'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># get the user identity URI, needed for the template</span></span><br><span class="line">aibuami01uri=/subscriptions/<span class="variable">$subscriptionID</span>/resourcegroups/<span class="variable">$imageResourceGroup</span>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<span class="variable">$idenityName</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># download preconfigured role definition example</span></span><br><span class="line">curl https://raw.githubusercontent.com/TheoDoreW/wxsblog.github.io/master/2021/07/25/2021-07-25-AzureImageBuilderCLMI/template/AIBRoleImageCreation.json -o AIBRoleImageCreation.json</span><br><span class="line"></span><br><span class="line">imageRoleDefName=<span class="string">"Azure Image Builder Image Def01"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># update the definition</span></span><br><span class="line">sed -i -e <span class="string">"s/<subscriptionID>/<span class="variable">$subscriptionID</span>/g"</span> AIBRoleImageCreation.json</span><br><span class="line">sed -i -e <span class="string">"s/<rgName>/<span class="variable">$imageResourceGroup</span>/g"</span> AIBRoleImageCreation.json</span><br><span class="line">sed -i -e <span class="string">"s/Azure Image Builder Service Image Creation Role/<span class="variable">$imageRoleDefName</span>/g"</span> AIBRoleImageCreation.json</span><br><span class="line"></span><br><span class="line"><span class="comment"># create role definitions</span></span><br><span class="line">az role definition create --role-definition ./AIBRoleImageCreation.json</span><br><span class="line"></span><br><span class="line"><span class="comment"># grant role definition to the user assigned identity</span></span><br><span class="line">az role assignment create \</span><br><span class="line"> --assignee <span class="variable">$aibuami01id</span> \</span><br><span class="line"> --role <span class="string">"<span class="variable">$imageRoleDefName</span>"</span> \</span><br><span class="line"> --scope /subscriptions/<span class="variable">$subscriptionID</span>/resourceGroups/<span class="variable">$imageResourceGroup</span></span><br></pre></td></tr></table></figure>
<h3 id="3-2-修改-CentOS-7-7-模板定义文件"><a href="#3-2-修改-CentOS-7-7-模板定义文件" class="headerlink" title="3.2 修改 CentOS 7.7 模板定义文件"></a>3.2 修改 CentOS 7.7 模板定义文件</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># download the example and configure it with your vars</span></span><br><span class="line"></span><br><span class="line">curl https://raw.githubusercontent.com/TheoDoreW/wxsblog.github.io/master/2021/07/25/2021-07-25-AzureImageBuilderCLMI/template/CentOS77AIBTemplate.json -o CentOS77AIBTemplate.json</span><br><span class="line"></span><br><span class="line">sed -i -e <span class="string">"s/<subscriptionID>/<span class="variable">$subscriptionID</span>/g"</span> CentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<rgName>/<span class="variable">$imageResourceGroup</span>/g"</span> CentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<region>/<span class="variable">$location</span>/g"</span> CentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<imageName>/<span class="variable">$imageName</span>/g"</span> CentOS77AIBTemplate.json</span><br><span class="line">sed -i -e <span class="string">"s/<runOutputName>/<span class="variable">$runOutputName</span>/g"</span> CentOS77AIBTemplate.json</span><br><span class="line"></span><br><span class="line">sed -i -e <span class="string">"s%<imgBuilderId>%<span class="variable">$aibuami01uri</span>%g"</span> CentOS77AIBTemplate.json</span><br></pre></td></tr></table></figure>
<h3 id="3-3-创建-CentOS-7-7-镜像"><a href="#3-3-创建-CentOS-7-7-镜像" class="headerlink" title="3.3 创建 CentOS 7.7 镜像"></a>3.3 创建 CentOS 7.7 镜像</h3><p>该过程会创建一些中间资源，比如 “IT_DestinationResourceGroup_TemplateName” 的 RG 以及 D1v2 大小的 Staging VM，这些资源都是 AIB 自动化创建</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># submit the image confiuration to the VM Image Builder Service</span></span><br><span class="line"></span><br><span class="line">az resource create \</span><br><span class="line"> --resource-group <span class="variable">$imageResourceGroup</span> \</span><br><span class="line"> --properties @CentOS77AIBTemplate.json \</span><br><span class="line"> --is-full-object \</span><br><span class="line"> --resource-type Microsoft.VirtualMachineImages/imageTemplates \</span><br><span class="line"> -n CentOS77AIB01</span><br><span class="line"></span><br><span class="line"><span class="comment"># wait approx 1-3mins, depending on external links</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># start the image build</span></span><br><span class="line"></span><br><span class="line">az resource invoke-action \</span><br><span class="line"> --resource-group <span class="variable">$imageResourceGroup</span> \</span><br><span class="line"> --resource-type Microsoft.VirtualMachineImages/imageTemplates \</span><br><span class="line"> -n CentOS77AIB01 \</span><br><span class="line"> --action Run </span><br><span class="line"></span><br><span class="line"><span class="comment"># wait approx 15mins</span></span><br></pre></td></tr></table></figure>
<h3 id="3-4-创建-CentOS-7-7-VM-并登陆验证"><a href="#3-4-创建-CentOS-7-7-VM-并登陆验证" class="headerlink" title="3.4 创建 CentOS 7.7 VM 并登陆验证"></a>3.4 创建 CentOS 7.7 VM 并登陆验证</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">az vm create \</span><br><span class="line"> --resource-group <span class="variable">$imageResourceGroup</span> \</span><br><span class="line"> --name centos01 \</span><br><span class="line"> --image <span class="variable">$imageName</span> \</span><br><span class="line"> --location <span class="variable">$location</span> \</span><br><span class="line"> --admin-username centos \</span><br><span class="line"> --ssh-key-value @id_rsa.pub</span><br><span class="line"></span><br><span class="line"><span class="comment"># and login...</span></span><br><span class="line"></span><br><span class="line">ssh centos@<pubIp></span><br><span class="line"></span><br><span class="line">You should see the image was customized with a Message of the Day as soon as your SSH connection is established!</span><br><span class="line"></span><br><span class="line">******************************************************************</span><br><span class="line">** This VM was built from the: **</span><br><span class="line">** !! AZURE VM IMAGE BUILDER Custom CentOS 7.7 Image !! **</span><br><span class="line">** You have just been Customized :-) **</span><br><span class="line">******************************************************************</span><br></pre></td></tr></table></figure>
<br>
<h2 id="4-总结"><a href="#4-总结" class="headerlink" title="4. 总结"></a>4. 总结</h2><p>至此，通过 AIB 自动化构建 CentOS 7.7 镜像的示例就完成了，希望能够给大家一些参考，把这个服务有机的集成利用起来。</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2021/07/25/2021-07-25-AzureImageBuilderCLMI/" data-id="clat72drf000nyofke5yffk6u" class="article-share-link" data-share="baidu" data-title="Azure Image Builder（一）之自动化构建自定义托管镜像 CentOS 7.7">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Automation/" rel="tag">Automation</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/infrastructure/" rel="tag">infrastructure</a></li></ul>
</footer>
</div>
</article>
<article id="post-2021-03-17-AzureDatabricksMonitor" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2021/03/17/2021-03-17-AzureDatabricksMonitor/" class="article-date">
<time datetime="2021-03-17T03:17:27.000Z" itemprop="datePublished">2021-03-17</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/">Bigdata</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Monitor/">Monitor</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Monitor/Databricks/">Databricks</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Monitor/Databricks/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2021/03/17/2021-03-17-AzureDatabricksMonitor/">Azure Databricks 系列 Blog（四）之通过 Azure Monitor 做集群监控</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p>前面几篇系列博客分别给大家介绍了 Azure Databricks 的概念以及最常用的流计算及批处理框架，作为托管在 Azure 云上的 Spark 商业化产品，能看到 Azure Databricks 带给用户层面的便捷性还是非常具有吸引力的。可是在实际使用过程中，还有一点也非常的重要，就是集群的可靠性。可靠性大概应该分成两个层面，一是集群的可靠性 —— 虽然 Spark Cluster 的分布式架构已经充分考虑了可靠性，但是依然有故障的可能性；二是 Job 的可靠性 —— 提交的 Job 正常运行会和业务影响息息相关。针对这两个层面，可靠性的保证就需要依赖强大的监控系统，一个好的监控系统可以做到出现问题的时候及时通知用户，更好地可以做到未雨绸缪。目前做 Spark 监控的技术方案有很多，比如说 Prometheus、Nagios，包括 Azure Databricks 默认集成的 Ganglia，这些监控工具都有自己的优势和劣势。不过从云原生的角度讲，Azure Databricks 作为微软 Azure 云上的 PaaS 服务也可以和 Azure Monitor 做原生集成。本文就给大家介绍一下通过 Azure Monitor Log Analysis（下文简称 LA ）来监控 Azure Databricks Cluster & Job，所有的测试都在 Global Azure US East 2 上测试实现。</p>
<br>
<h2 id="2-前期准备工作"><a href="#2-前期准备工作" class="headerlink" title="2. 前期准备工作"></a>2. 前期准备工作</h2><p>本文的实验是基于 Github 开源代码的实现，原文可以看<a target="_blank" rel="noopener" href="https://github.com/mspnp/spark-monitoring">这里</a>。具体原理是通过编译生成 spark-listeners-loganalytics 和 spark-listeners JAR 包并分发到所有计算节点上，然后 Job 通过 Spark 原生监控系统 Dropwizard Metrics Library 提供的指标集成 LA SDK 将 Cluster & Job 的 Metric & Logs 送到 LA 中，最后通过 LA Kusto 做监控分析告警。在实验开始之前，需要一些基础的准备工作，包括一台编译服务器，本文选择 CentOS 7，并安装 Azure Databricks CLI, JDK 8, Scala 2.11/12, Apache Maven 3.x。需要注意的是，Databricks CLI 的权限验证可以通过 Personal Token 或者 AAD，本文采用 AAD 认证，不过具体过程不赘述，请大家自行参考<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/databricks/dev-tools/cli/">官方文档</a>。安装好这些工具包之后，可以通过一些简单的测试比如 java -verison, mvn -V 以及 databricks clusters list 来进行验证。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Testing</span></span><br><span class="line">[root@centos01 ~]<span class="comment"># java -verison</span></span><br><span class="line">[root@centos01 ~]<span class="comment"># mvn -V</span></span><br><span class="line">[root@centos01 ~]<span class="comment"># databricks clusters list</span></span><br></pre></td></tr></table></figure>
<br>
<h2 id="3-编译-Azure-Databricks-monitoring-library"><a href="#3-编译-Azure-Databricks-monitoring-library" class="headerlink" title="3. 编译 Azure Databricks monitoring library"></a>3. 编译 Azure Databricks monitoring library</h2><p>编译 spark-listeners-loganalytics and spark-listeners 这两个 JAR 的时候有两种方法，一种通过 Docker Image 编译打包，另外一种通过 Maven，本文采用 Docker 来编译打包，速度会比较快。具体过程如下：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># To build all profiles:</span></span><br><span class="line">[root@centos01 ~]<span class="comment"># git clone https://github.com/mspnp/spark-monitoring.git</span></span><br><span class="line">[root@centos01 ~]<span class="comment"># cd spark-monitoring</span></span><br><span class="line">[root@centos01 ~]<span class="comment"># chmod +x ./build.sh</span></span><br><span class="line">[root@centos01 ~]<span class="comment"># docker run -it --rm -v `pwd`:/spark-monitoring -v "$HOME/.m2":/root/.m2 maven:3.5.4-jdk-8 /spark-monitoring/build.sh</span></span><br></pre></td></tr></table></figure>
<p>编译好之后，所有的 JAR 都在 src/target 下：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">[root@centos01 spark-monitoring]<span class="comment"># ll src/target/</span></span><br><span class="line">total 952</span><br><span class="line">-rw-r--r-- 1 root root 206859 Mar 16 15:30 spark-listeners_2.4.3_2.11-1.0.0.jar</span><br><span class="line">-rw-r--r-- 1 root root 206860 Mar 16 15:32 spark-listeners_2.4.5_2.11-1.0.0.jar</span><br><span class="line">-rw-r--r-- 1 root root 150449 Mar 16 15:35 spark-listeners_3.0.0_2.12-1.0.0.jar</span><br><span class="line">-rw-r--r-- 1 root root 150449 Mar 16 15:38 spark-listeners_3.0.1_2.12-1.0.0.jar</span><br><span class="line">-rw-r--r-- 1 root root 73214 Mar 16 15:31 spark-listeners-loganalytics_2.4.3_2.11-1.0.0.jar</span><br><span class="line">-rw-r--r-- 1 root root 73213 Mar 16 15:32 spark-listeners-loganalytics_2.4.5_2.11-1.0.0.jar</span><br><span class="line">-rw-r--r-- 1 root root 51211 Mar 16 15:36 spark-listeners-loganalytics_3.0.0_2.12-1.0.0.jar</span><br><span class="line">-rw-r--r-- 1 root root 51212 Mar 16 15:38 spark-listeners-loganalytics_3.0.1_2.12-1.0.0.jar</span><br></pre></td></tr></table></figure>
<p>具体的 JAR 都指定对应了 Spark 和 Scala 版本，创建集群的时候需要注意指定对应的 JAR 。</p>
<br>
<h2 id="4-配置-Databricks-workspace"><a href="#4-配置-Databricks-workspace" class="headerlink" title="4. 配置 Databricks workspace"></a>4. 配置 Databricks workspace</h2><p>通过 Azure Databricks CLI 在 DBFS 上创建一个目录 dbfs:/databricks/spark-monitoring：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">dbfs mkdirs dbfs:/databricks/spark-monitoring</span><br></pre></td></tr></table></figure>
<p>修改目录下 src/spark-listeners/scripts/spark-monitoring.sh 添加相关配置信息进入配置文件，包括 LA ID/Key, RG Name 等等，具体的每个配置项按照实际的属性值填入，这些配置会作为 HTTP 请求报头中的信息。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># LA ID/Key</span></span><br><span class="line"><span class="built_in">export</span> LOG_ANALYTICS_WORKSPACE_ID=</span><br><span class="line"><span class="built_in">export</span> LOG_ANALYTICS_WORKSPACE_KEY=</span><br><span class="line"><span class="comment"># Environment Variables</span></span><br><span class="line"><span class="built_in">export</span> AZ_SUBSCRIPTION_ID=</span><br><span class="line"><span class="built_in">export</span> AZ_RSRC_GRP_NAME=</span><br><span class="line"><span class="built_in">export</span> AZ_RSRC_PROV_NAMESPACE=Microsoft.Databricks</span><br><span class="line"><span class="built_in">export</span> AZ_RSRC_TYPE=workspaces</span><br><span class="line"><span class="built_in">export</span> AZ_RSRC_NAME=</span><br></pre></td></tr></table></figure>
<p>配置结束后，最后把相关的监控脚本和 JAR 拷贝到 dbfs:/databricks/spark-monitoring 中：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@centos01 spark-monitoring]<span class="comment"># dbfs cp src/spark-listeners/scripts/spark-monitoring.sh dbfs:/databricks/spark-monitoring/spark-monitoring.sh</span></span><br><span class="line">[root@centos01 spark-monitoring]<span class="comment"># ll src/target/ | awk '{print $9}' |sed '1d' > jarlist</span></span><br><span class="line">[root@centos01 spark-monitoring]<span class="comment"># while read line; do dbfs cp src/target/$line dbfs:/databricks/spark-monitoring/; done < jarlist</span></span><br><span class="line">[root@centos01 spark-monitoring]<span class="comment"># rm -f jarlist</span></span><br></pre></td></tr></table></figure>
<p>此时，所有准备工作就准备完毕。下面就启动集群并提交运行 Sample Job 然后来看监控数据了。需要注意的是 Databricks Runtime Version 和对应的 Spark 及 Scala 版本以及在初始化运行中需要在 Advanced Options 中指定 Init Scripts 为 dbfs:/databricks/spark-monitoring/spark-monitoring.sh。</p>
<p><a href="Databricks-Cluster-Creation.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2021-03-17-AzureDatabricksMonitor/Databricks-Cluster-Creation.jpg" height="700" width="700"/></a></p>
<br>
<h2 id="5-运行-Sample-Job"><a href="#5-运行-Sample-Job" class="headerlink" title="5. 运行 Sample Job"></a>5. 运行 Sample Job</h2><p>这个 Github Repository 同样包含了 Sample Job，可以通过该 Job 测试发送 Metric & Logs 到 LA。构建 Job 的时候，需要注意指定 Databricks Runtime 版本。</p>
<table>
<thead>
<tr>
<th align="center">Databricks Runtime(s)</th>
<th align="center">Maven Profile</th>
</tr>
</thead>
<tbody><tr>
<td align="center">5.5</td>
<td align="center">scala-2.11_spark-2.4.3</td>
</tr>
<tr>
<td align="center">6.4 - 6.6</td>
<td align="center">scala-2.11_spark-2.4.5</td>
</tr>
<tr>
<td align="center">7.0 - 7.2</td>
<td align="center">scala-2.12_spark-3.0.0</td>
</tr>
<tr>
<td align="center">7.3 - 7.5</td>
<td align="center">scala-2.12_spark-3.0.1</td>
</tr>
</tbody></table>
<p>本文的 Databricks Runtime 为 7.3 LTS，所以对应的 Maven Profile 为 scala-2.12_spark-3.0.1。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -it --<span class="built_in">rm</span> -v `<span class="built_in">pwd</span>`/sample/spark-sample-job:/spark-sample-job -v <span class="string">"<span class="variable">$HOME</span>/.m2"</span>:/root/.m2 -w /spark-sample-job maven:3.5.4-jdk-8 mvn install -P scala-2.12_spark-3.0.1</span><br></pre></td></tr></table></figure>
<p>编译结束后，会在 spark-monitoring/sample/spark-sample-job/target 目录下生成一个 spark-monitoring-sample-1.0.0.jar，需要在提交 Job 的时候指定该 JAR 并指定 Main Class 为 com.microsoft.pnp.samplejob.StreamingQueryListenerSampleJob。</p>
<p><a href="Databricks-Sample-Job.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2021-03-17-AzureDatabricksMonitor/Databricks-Sample-Job.jpg" height="700" width="700"/></a></p>
<br>
<h2 id="6-在-Azure-Monitor-查询验证"><a href="#6-在-Azure-Monitor-查询验证" class="headerlink" title="6. 在 Azure Monitor 查询验证"></a>6. 在 Azure Monitor 查询验证</h2><p>在 Job 提交运行之后，就可以到 LA 查看监控数据了。能够看到 LA 中出现了 3 个 custom table：SparkListenerEvent_CL、SparkLoggingEvent_CL、SparkMetric_CL 对应 Metric 和 Log。我们这里运行一个示例 Kusto 查询，来查询下产生的所有 Event：</p>
<p><a href="Databricks-LA-Kusto.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2021-03-17-AzureDatabricksMonitor/Databricks-LA-Kusto.jpg" height="900" width="900"/></a></p>
<br>
<h2 id="7-总结"><a href="#7-总结" class="headerlink" title="7. 总结"></a>7. 总结</h2><p>至此，通过 Azure Monitor 监控 Azure Databricks 示例就完成了，需要注意在实际场景集成的时候，Job 依然要加载相应的类来做集成。衷心希望本文可以给各位读者一些参考并能在实际场景中有机的结合起来。</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2021/03/17/2021-03-17-AzureDatabricksMonitor/" data-id="clat72drf000jyofk2txb4c9n" class="article-share-link" data-share="baidu" data-title="Azure Databricks 系列 Blog（四）之通过 Azure Monitor 做集群监控">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Bigdata/" rel="tag">Bigdata</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Databricks/" rel="tag">Databricks</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Monitor/" rel="tag">Monitor</a></li></ul>
</footer>
</div>
</article>
<article id="post-2020-12-17-AzureDatabricksSparkSQLonDeltaLake" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2020/12/17/2020-12-17-AzureDatabricksSparkSQLonDeltaLake/" class="article-date">
<time datetime="2020-12-17T11:09:13.000Z" itemprop="datePublished">2020-12-17</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/">Bigdata</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Databricks/">Databricks</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Databricks/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2020/12/17/2020-12-17-AzureDatabricksSparkSQLonDeltaLake/">Azure Databricks 系列 Blog（三）之 SparkSQL on Delta Lake</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p><a href="https://theodorew.github.io/wxsblog.github.io/2020/11/06/2020-11-06-AzureDatabricksStructureStreaming/">上一篇</a>介绍了 Azure Databricks 流计算框架 Structure Streaming，通过一个 Demo 演示了 Spark 的实时流计算框架 Structure Streaming。但是在实际业务中，光有实时数据流是不够的，离线数据的计算和分析也非常重要，并且往往离线分析才能获取业务上所需要的更多的 Insights，而一般来说离线的数据分析都会跑一些 OLAP SQL 查询，所以基于此场景，本文就来介绍并演示下 SparkSQL，一个可以通过 Spark 来进行 SQL 分析的可实时可离线的计算框架。本文场景基于集成 Azure Datalake Gen2 并 Enable Databricks Delta Lake 作为外部存储实现计算存储分离，通过 SparkSQL 新冠肺炎实时统计数据，资源依然沿用前面 Demo 所创建的资源组，具体如何实现，且听下文分解。</p>
<br>
<h2 id="2-Delta-Lake-介绍及-Azure-Datalake-Gen2-集成"><a href="#2-Delta-Lake-介绍及-Azure-Datalake-Gen2-集成" class="headerlink" title="2. Delta Lake 介绍及 Azure Datalake Gen2 集成"></a>2. Delta Lake 介绍及 Azure Datalake Gen2 集成</h2><p>Delta Lake 是可提高数据湖可靠性的开源存储层，由 Databricks 开发并开源出来。 Delta Lake 提供 ACID 事务和可缩放的元数据处理，并可以统一流处理和批数据处理。 Delta Lake 在现有 Data Lake 的顶层运行，并且可以与 Apache Spark API 完全兼容。具体而言，Delta Lake 提供：</p>
<p><strong>Spark 上的 ACID 事务：</strong>可序列化的隔离级别可避免读者看到不一致的数据。<br><strong>可缩放的元数据处理：</strong>利用 Spark 的分布式处理能力，轻松处理包含数十亿文件的 PB 级表的所有元数据。<br><strong>流式处理和批处理统一：</strong>Delta Lake 中的表是批处理表，也是流式处理源和接收器。 流式处理数据引入、批处理历史回填、交互式查询功能都是现成的。<br><strong>架构强制：</strong>自动处理架构变体，以防在引入过程中插入错误的记录。<br><strong>按时间顺序查看：</strong>数据版本控制支持回滚、完整的历史审核线索和可重现的机器学习试验。<br><strong>更新插入和删除：</strong>支持合并、更新和删除操作，以启用复杂用例，如更改数据捕获、渐变维度 (SCD) 操作、流式处理更新插入等。</p>
<p>总的来说，Delta Lake 不仅仅在需要事务的 ACID 场景给予支持，并且针对性能也做了相当大的代码优化，本文的实现也是基于 Delta Lake，需要注意数据格式需要转换成 delta。</p>
<p><a href="Delta-Lake-marketecture.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-12-17-AzureDatabricksSparkSQLonDeltaLake/Delta-Lake-marketecture.png" height="750" width="750"/></a></p>
<p>正常 Azure Datalake Gen2 与 Delta Lake 的集成需要在 Spark 中增加配置项 <strong>“spark.delta.logStore.class=org.apache.spark.sql.delta.storage.AzureLogStore”</strong>, 并且集群需要在 lib 库中支持 hadoop-azure-datalake/hadoop-azure/wildfly-openssl JAR，具体过程可以见<a target="_blank" rel="noopener" href="https://docs.delta.io/latest/delta-storage.html#azure-data-lake-storage-gen2">这里</a>。所以这时候 PaaS 的优势就显示出来了，Azure Databricks 在 Cluster 创建好的同时就已经在集群上启用了该库，直接调用即可。 Azure Datalake Gen2 的创建过程本文不赘述，具体见<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/storage/blobs/create-data-lake-storage-account">官方文档</a>。本文介绍两种常见的集成方式，如下图所示并做几点说明：</p>
<p><a href="Delta-Lake-Azure-DataLake-Gen2.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-12-17-AzureDatabricksSparkSQLonDeltaLake/Delta-Lake-Azure-DataLake-Gen2.png" height="850" width="850"/></a></p>
<p><font color=blue><strong>cmd2：非挂载方式</strong></font><br>cmd2 中通过 scala 代码演示 Azure Databricks 集成 Azure Datalake Gen2 的认证配置项，通过创建拥有 IAM Role “Storage Blob Data Contributor” 的 Service Principle 来做认证，验证通过后就可以直接调用 Datalake 里面的文件了，注意相关变量需要替换。</p>
<p><font color=blue><strong>cmd3：挂载方式</strong></font><br>cmd3 中通过 Python 代码演示 Azure Databricks 如何挂载 Azure Datalake Gen2，身份验证过程和 cmd2 的方式一样。这种方式的好处是直接可以可以把远端的 Azure Datalake 挂载到 Azure Databricks 上，就好像在使用本地磁盘一样使用 Datalake，本文更推荐并且采用该模式，同时注意相关变量需要替换。</p>
<p>另外，本文的环境由于都是在 Azure 中国区，所以相关的域名都是 <a target="_blank" rel="noopener" href="https://login.partner.microsoftonline.cn,/">https://login.partner.microsoftonline.cn，</a> 如果在 Azure Global，那么注意域名是 <a target="_blank" rel="noopener" href="https://login.microsoftonline.com./">https://login.microsoftonline.com。</a> 其实从最佳实践的角度来说，本文并不是实现的最好方式，因为在 Notebook 里面的认证信息全部都是明文出现，最好的方式应该集成 Azure Key Vault 来隐藏密码，该方式不再本文赘述，有兴趣的同学自行研究吧，附上<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/azure-datalake-gen2">链接</a>供参考。</p>
<br>
<h2 id="3-引入数据源并运行-SparkSQL"><a href="#3-引入数据源并运行-SparkSQL" class="headerlink" title="3. 引入数据源并运行 SparkSQL"></a>3. 引入数据源并运行 SparkSQL</h2><p>前面提到，本文所采用的示例数据是和新冠肺炎实时统计数据相关，讲到这里介绍一个微软提供的一个开放数据集 Repo，链接在<a target="_blank" rel="noopener" href="https://azure.microsoft.com/zh-cn/services/open-datasets/catalog/">这里</a>，包括各种场景下的 csv/json/parquet 格式的原始和脱敏数据，本文所使用的新冠肺炎数据也是在这里获取的，选择 csv 格式数据作为数据源，下载好之后 upload 到 Azure DataLake 上，在 container data 下面再创建个 source folder 作为存放原始数据的目录径。</p>
<p><a href="Delta-Lake-Azure-DataLake-Container.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-12-17-AzureDatabricksSparkSQLonDeltaLake/Delta-Lake-Azure-DataLake-Container.jpg" height="850" width="850"/></a></p>
<p>原始数据确定好之后，就可以通过 Notebook 来做 SparkSQL 分析了，本文用到的示例 Notebook 已经上传到了<a target="_blank" rel="noopener" href="https://github.com/TheoDoreW/wxsblog.github.io/blob/master/2020/12/17/2020-12-17-AzureDatabricksSparkSQLonDeltaLake/DemoNotebook/DeltaLake-SparkSQL.ipynb">这里</a>，几个重要配置简单说明下：</p>
<p><a href="Delta-Lake-Azure-DataLake-SparkSQL.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-12-17-AzureDatabricksSparkSQLonDeltaLake/Delta-Lake-Azure-DataLake-SparkSQL.jpg" height="850" width="850"/></a></p>
<p><strong>COVIDDF：</strong>COVIDDF 为定义的 DataFrame，通过 spark.read.format(‘csv’) 格式并指定挂载路径引入数据；<br><strong>COVIDDF.write.mode(“append”).format(“delta”)：</strong>通过 append 追加的方式写入目标 Table 中，注意这里的格式 delta，也就是前面提到的需要做格式转换，写入的表也需要指定路径，并会在 Azure Dalalake 中生成相应的文件；<br><strong>ChinaCOVID 及 ChinaCOVID1：</strong>经过上面的建表过程之后就可以运行 SparkSQL 来做 SQL 查询了；</p>
<br>
<h2 id="4-总结"><a href="#4-总结" class="headerlink" title="4. 总结"></a>4. 总结</h2><p>一个通过集成 Azure Datalake Gen2 作为外部存储并采用 Delta Lake 格式进行 SparkSQL 分析的示例就完成了，本文基本上使用了 PySpark 来实现，其实除此之外通过 SQL 或者 Scala 也是一样可以实现的，有兴趣的同学可以自己再深入研究下吧。</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2020/12/17/2020-12-17-AzureDatabricksSparkSQLonDeltaLake/" data-id="clat72dre000hyofka1mu3jrz" class="article-share-link" data-share="baidu" data-title="Azure Databricks 系列 Blog（三）之 SparkSQL on Delta Lake">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Bigdata/" rel="tag">Bigdata</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Databricks/" rel="tag">Databricks</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li></ul>
</footer>
</div>
</article>
<article id="post-2020-11-06-AzureDatabricksStructureStreaming" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2020/11/06/2020-11-06-AzureDatabricksStructureStreaming/" class="article-date">
<time datetime="2020-11-06T04:35:53.000Z" itemprop="datePublished">2020-11-06</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/">Bigdata</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Databricks/">Databricks</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Databricks/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2020/11/06/2020-11-06-AzureDatabricksStructureStreaming/">Azure Databricks 系列 Blog（二）之流计算 Structure Streaming</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p>经过<a href="https://theodorew.github.io/wxsblog.github.io/2020/09/04/2020-09-04-AzureDatabricksIntroduction/">开箱</a>介绍后，想必都对 Azure Databricks 有了大致的了解，那乘胜追击，本文介绍一下 Azure Databricks 的流计算架构。基于的场景是设计一个实时告警系统：“针对一家杂货店的购物者的购物异常习惯进行实时采样和告警，主要针对的产品是止咳糖浆，如果采样发现一次性购物超过 10 瓶就进行实时告警”，毕竟需要防止下一个沃尔特·怀特出现。需要用到的服务全部部署在中国区 Azure 上，后文对具体每项服务功能不多做介绍，请读者到<a target="_blank" rel="noopener" href="https://www.azure.cn/">官网</a>自行搜索查询。</p>
<br>
<h2 id="2-流计算架构说明"><a href="#2-流计算架构说明" class="headerlink" title="2. 流计算架构说明"></a>2. 流计算架构说明</h2><p>依据本文模拟的场景，设计的云上架构如下图所示：</p>
<p><a href="Databricks-Structure-Streaming-Arch.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-11-06-AzureDatabricksStructureStreaming/Databricks-Structure-Streaming-Arch.png" height="750" width="750"/></a></p>
<p>用到的相关服务组件功能做如下说明：<br><strong>Azure VM：</strong>数据源，扮演 Producer 生产者，通过 Python 代码模拟客户购买行为，生成示例数据通过 SDK 发送数据到 Azure Eventhub。<br><strong>Azure Eventhub：</strong>消息队列，做上下游生产者消费者服务的解耦，Entity ingestion 接收 Producer 发送的数据，Entity alerting 接收经过 Databricks 实时计算后的数据。<br><strong>Azure Databricks：</strong>订阅 Eventhub Entity ingestion 作为数据源，通过 Structure Streaming 对数据进行实时处理后发送 Entity alerting。<br><strong>Azure LogicApp：</strong>订阅 Eventhub Entity alerting 并做邮件实时告警。</p>
<p><font color=blue><strong>整个数据流：Producer 生产者发送数据 &rarr; Eventhub Entity ingestion &rarr; Databricks Structured Streaming &rarr; Eventhub Entity alerting &rarr; Logic App</strong></font></p>
<br>
<h2 id="3-Azure-Databrick-Structure-Streaming-实现"><a href="#3-Azure-Databrick-Structure-Streaming-实现" class="headerlink" title="3. Azure Databrick Structure Streaming 实现"></a>3. Azure Databrick Structure Streaming 实现</h2><h3 id="3-1-Terraform-自动化部署"><a href="#3-1-Terraform-自动化部署" class="headerlink" title="3.1 Terraform 自动化部署"></a>3.1 Terraform 自动化部署</h3><p>通过 Terraform 部署的服务组件包括 Azure Virtual Machine、 Azure Databricks、Eventhub、Logic App，具体的 tf 文件和变量见<a target="_blank" rel="noopener" href="https://github.com/TheoDoreW/wxsblog.github.io/tree/master/2020/11/06/2020-11-06-AzureDatabricksStructureStreaming/Terraform">这里</a>，每一项服务 Terraform Azure Provider 都有 Resource 支持，具体可以参考<a target="_blank" rel="noopener" href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs">官方文档</a>。部署完成之后的资源清单如下图所示，所有资源都部署在中国北二区域。</p>
<p><a href="Databricks-Structure-Streaming-Resources.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-11-06-AzureDatabricksStructureStreaming/Databricks-Structure-Streaming-Resources.png" height="950" width="950"/></a></p>
<h3 id="3-2-Producer-代码发布"><a href="#3-2-Producer-代码发布" class="headerlink" title="3.2 Producer 代码发布"></a>3.2 Producer 代码发布</h3><p>模拟的生产者代码通过 VM 发布，通过调用 Azure Eventhub 的 SDK 将 Messages 进行写入，具体代码见<a target="_blank" rel="noopener" href="https://github.com/TheoDoreW/wxsblog.github.io/blob/master/2020/11/06/2020-11-06-AzureDatabricksStructureStreaming/PythonProducer/Producer.py">这里</a>，几个重要配置简单说明下：<br><strong>azure.eventhub：</strong>Azure Eventhub SDK 包，需要通过 pip3 install azure.eventhub 来指定安装。<br><strong>create_batch()：</strong>通过该方法 batch 发送数据，本次示例以 1 条消息为 1 个 batch 发送到 Eventhub Entity ingestion。<br><strong>CONNECTION_STR：</strong>Azure Eventhub Endpoint，该连接字符串可以在 Portal 上 Shared access policies 的 Connection string–primary key 查看。<br><strong>EVENTHUB_NAME：</strong>写入的 Eventhub Entity Name。</p>
<h3 id="3-3-Azure-Databricks-集群配置及-Structure-Streaming-Notebook-集成"><a href="#3-3-Azure-Databricks-集群配置及-Structure-Streaming-Notebook-集成" class="headerlink" title="3.3 Azure Databricks 集群配置及 Structure Streaming Notebook 集成"></a>3.3 Azure Databricks 集群配置及 Structure Streaming Notebook 集成</h3><p>Azure Databricks 的创建过程是先需要在 Azure 上创建一个 Databricks 实体，然后在此基础上在实体内部创建 Workspace 以及 Cluster，再提交 Job 等等。针对 Databricks 资源，都会有唯一的 ID 和 Endpoint 与之对应，以便能够进行 Restful API 调用，集群通过 Databricks Portal 创建即可。本示例创建 1 Driver 2 Worker 共计 3 个节点的 Standard Cluster，Databricks 版本为 6.4 (includes Apache Spark 2.4.5, Scala 2.11)。如果需要做机器学习相关的计算，可以启用集成 GPU/ML 框架的版本，详细说明见<a target="_blank" rel="noopener" href="https://docs.microsoft.com/zh-cn/azure/databricks/release-notes/runtime/releases">官方文档</a>，不做赘述。</p>
<p><a href="Databricks-Clusters.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-11-06-AzureDatabricksStructureStreaming/Databricks-Clusters.jpg" height="900" width="900"/></a></p>
<p>集群状态变为 Running 后就意味着 Ready 可以使用了，不过在导入 Python Notebook 之前，需要通过 Maven Central 安装 com.microsoft.azure:azure-eventhubs-spark 库文件以便安装 Spark 连接 Azure Eventhub Connector，需要注意库文件的版本要匹配。</p>
<p><a href="Databricks-Eventhub-Maven-Central-Installation.png"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-11-06-AzureDatabricksStructureStreaming/Databricks-Eventhub-Maven-Central-Installation.png" height="700" width="700"/></a></p>
<p>Notebook 可以直接在 Portal 里新建写入，也可以在 VS Code 等 IDE 编写完之后发布，本文采用第二种模式，原因是 IDE 丰富的插件可以提高效率。具体的 Notebook 本文不做展示，放在<a href="">这里</a>，有需要的读者可以自行查看。通过 import 导入后，附上导入后的截图并做几点说明：</p>
<p><a href="Databricks-Streaming-Notebook.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-11-06-AzureDatabricksStructureStreaming/Databricks-Streaming-Notebook.jpg" height="900" width="900"/></a></p>
<p>整个 Notebook 分为三个 stage：<br><strong>第一阶段：</strong>从 Eventhub Entity ingestion 读取 Producer 写入的数据，通过 Streaming DataFrames 的 spark.readStream() 创建。<br><strong>第二阶段：</strong>通过 DataFrame 丰富的函数做字段筛选，筛选出来我们需要的字段。<br><strong>第二阶段：</strong>回写 Eventhub Entity alerting，通过 Streaming DataFrames 的 spark.writeStream() 流式写入，注意利用 checkpoint 方便任务终止再运行。</p>
<p><a href="Databricks-Streaming-Status.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-11-06-AzureDatabricksStructureStreaming/Databricks-Streaming-Status.jpg" height="900" width="900"/></a></p>
<p>当 Producer 运行起来的时候，Eventhub 就会不断有数据写入，所以能看到 Spark 的 Input Records 的图像。对于每一个 Job，都能看到对于该任务分配的资源和 Spark 参数配置项。</p>
<h3 id="3-4-Logic-APP-配置邮件告警"><a href="#3-4-Logic-APP-配置邮件告警" class="headerlink" title="3.4 Logic APP 配置邮件告警"></a>3.4 Logic APP 配置邮件告警</h3><p>经过 Azure Databricks 的数据筛选后，筛选出来的 Messages 都写入了 Eventhub Entity alerting 中，此时通过 LogicApp 来定义一个自动化的 workflow 来进行邮件告警。具体创建过程选择 Blank 然后自己创建 Step 即可，当然 Azure Portal 上的示例模板也可以用来参考，如下图所示：</p>
<p><a href="LogicApp-Alerting-Designer.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-11-06-AzureDatabricksStructureStreaming/LogicApp-Alerting-Designer.jpg" height="800" width="800"/></a></p>
<p>第一步订阅 Eventhub Entity alerting，第二步集成 Outlook 邮件接口发送告警邮件。所以当目标消息被筛选出来之后，LogicApp 就按照定义的邮件内容(本文是 Messages 内容和时间戳)来发送邮件，发送邮件的截图如下：</p>
<p><a href="LogicApp-Email-Alert.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-11-06-AzureDatabricksStructureStreaming/LogicApp-Email-Alert.jpg" height="700" width="700"/></a></p>
<br>
<h2 id="4-总结"><a href="#4-总结" class="headerlink" title="4. 总结"></a>4. 总结</h2><p>总体上，一个通过消息队列 Azure Eventhub 以及 Databricks 做流计算处理的示例就完成了。如果消息生产者 Producer 不断产生消息，那么整个任务就会一直运行下去，当出现目标消息的时候就会不断的持续告警。在 Spark 推出 Structure Streaming 以后，也解决了 Spark Streaming microbatch 的局限性。介绍完流式计算，下一篇聊聊 Azure Databricks SparkSQL ，敬请期待吧~</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2020/11/06/2020-11-06-AzureDatabricksStructureStreaming/" data-id="clat72dre000eyofk26fpe0yt" class="article-share-link" data-share="baidu" data-title="Azure Databricks 系列 Blog（二）之流计算 Structure Streaming">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Bigdata/" rel="tag">Bigdata</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Databricks/" rel="tag">Databricks</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li></ul>
</footer>
</div>
</article>
<article id="post-2020-09-04-AzureDatabricksIntroduction" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2020/09/04/2020-09-04-AzureDatabricksIntroduction/" class="article-date">
<time datetime="2020-09-04T05:46:32.000Z" itemprop="datePublished">2020-09-04</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/">Bigdata</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Databricks/">Databricks</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Databricks/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2020/09/04/2020-09-04-AzureDatabricksIntroduction/">Azure Databricks 系列 Blog（一）之开箱</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p>随着云计算技术的高速发展以及相关云服务的产品化完善，Cloud 已经从互联网应用上云的 1.0 时代逐渐衍变成了 Cloud + AI + 5G 的 2.0 时代，伴随着的是企业的云化加深、核心业务上云，同时对于云计算 1.0 时代遗留的问题的改造也都提上了日程，其中比较典型的就是各行各业都开始数字化转型，那数据对企业的价值更是不言而喻了。可如果数据只保存不分析，那再多的数据也是徒劳，除了增加成本外毫无价值可言。由此可见，数据要想有价值，那就需要找到合适的好的工具，把收集的数据有机地结合起来做分析，从数据源头跟踪到最后有价值的输出。</p>
<p>目前，在数据分析领域已经很多成熟的方案和技术，比如数据仓库 Azure Synapse / Impala / Presto，大数据的计算框架 Spark / Flink 等，每一项技术都有自己所擅长解决的问题和场景以便供用户选择，本文不会对这些技术方案做详解和说明，不熟悉的读者请自行搜索科普。本文主要介绍微软 Azure 云上新发布的产品 Azure Databricks —— “一款基于纯 Spark 技术栈的大数据解决方案”。通过本文的介绍，希望可以帮助选择该技术方案解决业务需求的用户提供一些意见和参考。Azure Databricks 很早就已经在 Global Azure GA，最近刚刚在国内 Mooncake Azure Preview，如果要在国内 Azure 上使用需要申请。好，话不多说，下面就来揭开它的神秘面纱。</p>
<br>
<h2 id="2-什么是-Azure-Databricks"><a href="#2-什么是-Azure-Databricks" class="headerlink" title="2. 什么是 Azure Databricks"></a>2. 什么是 Azure Databricks</h2><p>Azure Databricks 是基于 Apache Spark 的分析平台，已针对 Microsoft Azure 云服务平台进行优化。 Databricks 是由 Spark 原作者团队创办的一家做 Spark 商业化产品的公司。微软与其合作，将 Databricks 与 Azure 集成以提供一键式部署等简化工作，从而能够帮助用户更专注地做基于业务的数据分析或者科学计算。Azure Databricks 和原生云服务产品的集成的缩略图如下所示：</p>
<p><a href="Azure-Databricks-Overview.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-09-04-AzureDatabricksIntroduction/Azure-Databricks-Overview.jpg" height="600" width="600"/></a></p>
<p>由此可见，Azure Databricks 可以和很多 Azure 云上服务做集成，支持例如 Azure Blob，DataLake，CosmosDB，Synapse，Eventhub 等作为其上下游服务进行对接，通过 Spark 计算框架提供的流批计算技术进行大数据计算，同时也可以集成机器学习来提取并探索数据中所包含的 insights。Azure Databricks 按照层次结构划分由底而上大致分为：</p>
<p><strong>Databricks IO：</strong>Databricks I/O 模块，又称 DBIO，利用垂直集成堆栈显著提高 Spark 在云上的性能。<br><strong>Databricks Runtime：</strong> 除原生 Spark 外，还提供其他组件来提高大数据分析的可用性、性能和安全性。<br><strong>Databricks Workspace：</strong>工作区是可以提供访问所有 Azure Databricks 资源的一组环境，包括 Notebook / Libraries / Experiments 等对象组织到文件夹中，并提供其对数据对象和计算资源的访问权限。<br><strong>Databricks Enterprise Security：</strong>Databricks 企业安全（DBES）模块添加了诸如静态和动态数据加密、细粒度数据访问控制和审核等功能，以满足标准合规性（例如 HIPAA、SOC2）和最严格的安全要求。</p>
<p>同时，Azure Databricks 也是 100% 兼容 Apache Spark 提供的集群技术和功能的，具体所包含的 Spark 组件如下图所示：</p>
<p><a href="Apache-Spark-Ecosystem-Databricks.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-09-04-AzureDatabricksIntroduction/Apache-Spark-Ecosystem-Databricks.jpg" height="600" width="600"/></a></p>
<p><strong>Spark SQL：</strong>SparkSQL 是用于处理结构化数据的 Spark 模块，基于 Spark DataFrames 分布式数据集合，可以把它在概念上理解为关系型数据库中的表。<br><strong>Streaming：</strong>实时数据处理和分析，适用于实时分析与交互式应用程序以及构建实时数仓，可以与 HDFS、Flume 和 Kafka 等集成。<br><strong>MLlib：</strong>由常见机器学习算法和工具（包括分类、回归、筛选、维数约简以及底层优化基元等）组成的机器学习库。<br><strong>GraphX：</strong>图形和图形计算，适用于从认知分析到数据探索的广泛用例。<br><strong>Spark Core API：</strong>包含对 R、SQL、Python、Scala 和 Java 的支持。</p>
<br>
<h2 id="3-Azure-Databricks-架构及特性介绍"><a href="#3-Azure-Databricks-架构及特性介绍" class="headerlink" title="3. Azure Databricks 架构及特性介绍"></a>3. Azure Databricks 架构及特性介绍</h2><p>Azure Databricks 是基于 Apache Spark 的快速、简单、协作型分析服务，具体的 Architecture 如图所示：</p>
<p><a href="Azure-Databricks-Arch.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-09-04-AzureDatabricksIntroduction/Azure-Databricks-Arch.jpg" height="600" width="600"/></a></p>
<p>Azure Databricks 整体分为 Control Plane 和 Data Plane。Control Plane 由 Azure 托管，负责管理集群主节点、提交的任务调度、账号权限等。Data Plane 交由用户管理，可以通过 Databricks 提供的接口来对集群进行操作，常见的一般有 UI / API / CLI。具体地说，当启动 Azure Databricks Cluster 时，需要指定要使用的 VM 类型和数量，同时部署出来集成 VNet / NSG / Storage Account 的托管资源组。部署完毕后，用户就可以通过 Azure Databricks UI 来管理集群。所有元数据（如计划作业）都存储在具有异地复制功能的 Azure 数据库中。值得一提的是，Azure Databricks 和容器的集成也逐渐深度了起来，在创建集群的时候也可以指定容器镜像，方便用户打包自己的定制化镜像从而进行集成以及 CI/CD 的可能性。总体上说，目前 Databricks 和 Azure 的集成基本上有以下几个方面：</p>
<p><strong>VM 类型的多样性：</strong>可集成所有现有的 VM 类型，例如 CPU 优化的 F 系列、内存优化的 E 系列、通用的 D 系列等等。<br><strong>网络拓扑的灵活性：</strong>提供 Azure Databricks 与 VNET 集成来满足多样化的基础架构网络的访问需求。<br><strong>Azure 存储和数据湖集成：</strong>通过 DBFS 向 Databricks 公开，以提供对现有数据的缓存和优化分析。<br><strong>Azure Power BI：</strong>可以使用 JDBC 将 Power BI 直接连接到 Azure Databricks 来进行大规模交互方式查询数据。<br><strong>Azure AD：</strong>通过 Azure AD 来做集群的访问控制和权限认证。<br><strong>Azure SQL 数据仓库、Azure SQL DB 和 Azure CosmosDB</strong>：可以与 Azure 上其余的 Data Services 进行快速便捷地集成，方便用户快速建立端到端的云上数据架构。</p>
<p>除此之外，还有很多特性，文本不再赘述，大家可以通过<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/databricks/scenarios/what-is-azure-databricks">官方文档</a>自行查看。</p>
<br>
<h2 id="4-后续篇幅计划及总结"><a href="#4-后续篇幅计划及总结" class="headerlink" title="4. 后续篇幅计划及总结"></a>4. 后续篇幅计划及总结</h2><p>介绍完开箱，想必都体会到 Azure Databricks 的优秀了吧？是不是都想动手实战下？那心动不如行动~后续会在此基础上分享一系列关于 Azure Databricks 在真实场景下落地的最佳实践，具体包括：</p>
<p><a href="">Azure Databricks 系列 Blog（二）之 流计算 Structure Streaming</a><br><a href="">Azure Databricks 系列 Blog（三）之 批处理 Spark SQL</a><br><a href="">Azure Databricks 系列 Blog（四）之 安全访问控制</a><br><a href="">Azure Databricks 系列 Blog（五）之 机器学习 Machine Learning</a></p>
<p>拭目以待吧！</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2020/09/04/2020-09-04-AzureDatabricksIntroduction/" data-id="clat72drd000dyofk9oet8e8x" class="article-share-link" data-share="baidu" data-title="Azure Databricks 系列 Blog（一）之开箱">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Bigdata/" rel="tag">Bigdata</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Databricks/" rel="tag">Databricks</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li></ul>
</footer>
</div>
</article>
<article id="post-2020-09-03-GithubCDN" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2020/09/03/2020-09-03-GithubCDN/" class="article-date">
<time datetime="2020-09-03T13:19:01.000Z" itemprop="datePublished">2020-09-03</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/CDN/">CDN</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/CDN/Github/">Github</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/CDN/Github/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2020/09/03/2020-09-03-GithubCDN/">使用 jsDelivr CDN 加速 Github Pages</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p>Github Pages 是个非常好的用来托管静态网页的功能，所以很多同学用来把自己的 Blog 放在上面去做托管，这么酷的功能既能让你心满意足地写博客，又可以同时找到 Coding 感觉，配合着静态站点生成器比如说 <a target="_blank" rel="noopener" href="https://jekyllrb.com/">Jekyll</a>、<a target="_blank" rel="noopener" href="https://hexo.io/">Hexo</a>，自己还不需要做维护，何乐而不为~ 不过这里面有个不太方面国内使用的问题，Github 服务器都在海外，在国内访问 Github Pages Blog 会出现访问比较慢的问题，尤其是 Blog 中需要加载的图片，稍微大一点的需要加载很长时间，但是 Github 在微软收购之后依然良心， 此时本文的主角登场，<a target="_blank" rel="noopener" href="https://www.jsdelivr.com/">jsDelivr CDN</a>。</p>
<br>
<h2 id="2-jsDelivr-CDN-介绍"><a href="#2-jsDelivr-CDN-介绍" class="headerlink" title="2. jsDelivr CDN 介绍"></a>2. jsDelivr CDN 介绍</h2><p>jsDelivr 是国外的一家优秀的公共 CDN 服务提供商，也是首个「打通中国大陆（网宿公司运营）与海外的免费 CDN 服务」，同时也为 npm、GitHub、WordPress 插件和具有特殊要求的其他几个项目的自定义终结点提供镜像。</p>
<p><a href="CDN-Introduction.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-09-03-GithubCDN/CDN-Introduction.jpg" height="800" width="800"/></a></p>
<p>能够看出来，POP 点也是分布全球比较均匀的，最重要的是标红的那句，能够在中国使用！是不是很令人激动~ 本文就给大家演示下如何用 jsDelivr CDN 加速 Github Pages 的图床，这样图片就可以秒加载啦！</p>
<br>
<h2 id="3-使用-jsDelivr-CDN-加速-Github-Pages-图床"><a href="#3-使用-jsDelivr-CDN-加速-Github-Pages-图床" class="headerlink" title="3. 使用 jsDelivr CDN 加速 Github Pages 图床"></a>3. 使用 jsDelivr CDN 加速 Github Pages 图床</h2><h3 id="3-1-Github-配置"><a href="#3-1-Github-配置" class="headerlink" title="3.1 Github 配置"></a>3.1 Github 配置</h3><p>首先需要创建一个 Github Repo，然后生成一个 Token，使得使用这个 key 可以有权限控制该仓库。如下面两张图所示：</p>
<p><a href="Github-Generate-Token.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-09-03-GithubCDN/Github-Generate-Token.jpg" height="800" width="800"/></a></p>
<p><a href="Github-Token-Access.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-09-03-GithubCDN/Github-Token-Access.jpg" height="800" width="800"/></a></p>
<p>记录好 Token，下面会用得到。</p>
<h3 id="3-2-下载-PicGo-并配置"><a href="#3-2-下载-PicGo-并配置" class="headerlink" title="3.2 下载 PicGo 并配置"></a>3.2 下载 PicGo 并配置</h3><p>下载地址点击<a target="_blank" rel="noopener" href="https://github.com/Molunerfinn/PicGo/releases">这里</a>，建议使用 GA 的版本，具体根据对应的平台下载相应的安装包即可，我们这里的环境是 Windows 10，那就选择 PicGo-Setup-2.2.2.exe 下载安装即可。安装好后，打开界面如图所示，有几个配置项，一一解释下：</p>
<p><a href="PicGo-Config.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-09-03-GithubCDN/PicGo-Config.jpg" height="800" width="800"/></a></p>
<p><strong>设定仓库名：</strong>存放图床的 Repo 名字，一般都是 xxx/xxx 格式；<br><strong>设定分支名：</strong>具体的分支，一般都是 master；<br><strong>设定 Token：</strong>3.1章节生成的 Token，填进去；<br><strong>指定存储路径：</strong>具体的 Repo 下面存储图片的路径；<br><strong>指定存储路径：</strong>一般都是 <a target="_blank" rel="noopener" href="https://cdn.jsdelivr.net/gh/xxx/xxx@master">https://cdn.jsdelivr.net/gh/xxx/xxx@master</a> ，域名是 jsDelivr CDN 本身发布的域名，gh 代表 Github，xxx/xxx@master 代表 Repo 以及 master 分支；</p>
<p>配置好之后，就可以切换到 “图片上传” 选择 “图片上传 - GitHub图床” 后就可以上传图片了，支持拖拽、点击、剪贴板上传，上传后，图片链接直接复制的你的剪贴板中。</p>
<p><a href="Github-PicGo.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-09-03-GithubCDN/Github-PicGo.jpg" height="800" width="800"/></a></p>
<p>能够看出来，Github 可以识别出来是通过 PicGo 上传的。</p>
<br>
<h2 id="4-总结"><a href="#4-总结" class="headerlink" title="4. 总结"></a>4. 总结</h2><p>至此，Github Pages 使用 jsDelivr CDN 加速图片的教程就完成了。其实可以做的更细致，比如说 js 静态文件也通过 CDN 做加速，都可以配置的，剩余的工作交由感兴趣的同学们自己来完成啦。</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2020/09/03/2020-09-03-GithubCDN/" data-id="clat72drd000ayofk087ib5nc" class="article-share-link" data-share="baidu" data-title="使用 jsDelivr CDN 加速 Github Pages">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/CDN/" rel="tag">CDN</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Github/" rel="tag">Github</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li></ul>
</footer>
</div>
</article>
<article id="post-2020-01-23-AzureMonitorSNMP" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2020/01/23/2020-01-23-AzureMonitorSNMP/" class="article-date">
<time datetime="2020-01-23T02:07:50.000Z" itemprop="datePublished">2020-01-23</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/">Monitor</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Fortigate/">Fortigate</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Fortigate/SNMP/">SNMP</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Fortigate/SNMP/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2020/01/23/2020-01-23-AzureMonitorSNMP/">Azure Monitor 通过 SNMP 协议监控网络设备 Fortigate Firewall</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p>快过年了，先在这里祝大家鼠年大吉。最近真是爱上了 Azure Monitor 这个服务，不仅功能强大而且还是云厂商托管滴，真的是让用户只需要关注业务场景的监控需求落地即可。这篇博客跟大家分享一下实际生产过程当中的一个案例，许多用户都会在混合云或者公有云多 Region 场景中选择自建防火墙来做更深入的定制化网络设计，而网络设备的监控也是非常必要的，这关系到整个基础架构的稳定性和可靠性，那本文来介绍下如何通过 SNMP 协议监控网络设备 Fortigate Firewall。</p>
<p>&nbsp;</p>
<h2 id="2-实验架构图及相关组件介绍"><a href="#2-实验架构图及相关组件介绍" class="headerlink" title="2. 实验架构图及相关组件介绍"></a>2. 实验架构图及相关组件介绍</h2><p>本文的实验的具体架构图如下所示：</p>
<p><a href="Azure-Monitor-Arch.jpg"><img src="Azure-Monitor-Arch.jpg" height="800" width="800"/></a></p>
<p>解释一下具体使用的一些产品和服务：<br><strong>Fortigate：</strong>飞塔防火墙设备，Azure上可以通过marketplace来做部署，分为PAYG和BYOL，分别代表两种lic激活模式。由于订阅的一些限制，本文不能通过此方式部署，只能通过vhd的方式部署，具体部署过程不进行赘述，大家可以参阅<a target="_blank" rel="noopener" href="https://docs.fortinet.com/document/fortigate/6.0.0/deploying-fortigate-on-azure/867066/running-azure-powershell-to-deploy-fortigate-vm">这里</a>。<br><strong>Collectd：</strong>Collectd是一个守护 (daemon) 进程，用来定期收集系统和应用程序的性能指标，同时提供了以不同的方式来存储这些指标值的机制。Collectd从各种来源收集指标，例如 操作系统，应用程序，日志文件和外部设备，并存储此信息或通过网络使其可用。 这些统计数据可用于监控系统、查找性能瓶颈（即性能分析）并预测未来的系统负载（即容量规划）等。支持插件，所有的插件可以看<a href="1.3.6.1.4.1.12356.101.4.1.8.0">这里</a>。</p>
<h2 id="3-Azure-Monitor-通过-SNMP-协议监控网络设备-Fortigate-Firewall"><a href="#3-Azure-Monitor-通过-SNMP-协议监控网络设备-Fortigate-Firewall" class="headerlink" title="3. Azure Monitor 通过 SNMP 协议监控网络设备 Fortigate Firewall"></a>3. Azure Monitor 通过 SNMP 协议监控网络设备 Fortigate Firewall</h2><h3 id="3-1-创建-Fortigate-Firewall-并启用-SNMP-监听"><a href="#3-1-创建-Fortigate-Firewall-并启用-SNMP-监听" class="headerlink" title="3.1 创建 Fortigate Firewall 并启用 SNMP 监听"></a>3.1 创建 Fortigate Firewall 并启用 SNMP 监听</h3><p>具体过程不进行赘述了，配置好之后，如图所示：</p>
<p><a href="Azure-Fortigate-SNMP.jpg"><img src="Azure-Fortigate-SNMP.jpg" height="800" width="800"/></a></p>
<h3 id="3-2-部署-CentOS-7-7-并配置-Collectd-Service-将数据转发-Azure-Log-Analysis-Workspace"><a href="#3-2-部署-CentOS-7-7-并配置-Collectd-Service-将数据转发-Azure-Log-Analysis-Workspace" class="headerlink" title="3.2 部署 CentOS 7.7 并配置 Collectd Service 将数据转发 Azure Log Analysis Workspace"></a>3.2 部署 CentOS 7.7 并配置 Collectd Service 将数据转发 Azure Log Analysis Workspace</h3><p>首先，通过 Terraform 自动化部署 CentOS 7.7 实例，不贴具体 tf 文件了，具体可以参考<a href="https://theodorew.github.io/wxsblog.github.io/2020/01/18/2020-01-18-AzureMonitorRestfulAPI/">这里</a>。通过 Terraform 进行部署的过程大概需要10分钟左右。创建完成后，登陆节点测试 snmpwalk 是否能够拉取 metric：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">$ snmpwalk -c public -v 2c 10.11.0.5 |more</span><br><span class="line">SNMPv2-MIB::sysDescr.0 = STRING: fortigate</span><br><span class="line">SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.12356.101.1.90081</span><br><span class="line">DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (647990) 1:47:59.90</span><br><span class="line">SNMPv2-MIB::sysContact.0 = STRING: </span><br><span class="line">SNMPv2-MIB::sysName.0 = STRING: FGVM08TM20000228</span><br><span class="line">SNMPv2-MIB::sysLocation.0 = STRING: </span><br><span class="line">SNMPv2-MIB::sysServices.0 = INTEGER: 78</span><br><span class="line">SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) 0:00:00.00</span><br><span class="line">SNMPv2-MIB::sysORIndex.1 = INTEGER: 1</span><br><span class="line">SNMPv2-MIB::sysORID.1 = OID: SNMPv2-SMI::zeroDotZero.0</span><br><span class="line">SNMPv2-MIB::sysORDescr.1 = STRING: </span><br><span class="line">SNMPv2-MIB::sysORUpTime.1 = Timeticks: (0) 0:00:00.00</span><br><span class="line">IF-MIB::ifNumber.0 = INTEGER: 2</span><br><span class="line">IF-MIB::ifIndex.1 = INTEGER: 1</span><br><span class="line">IF-MIB::ifIndex.2 = INTEGER: 2</span><br><span class="line">IF-MIB::ifDescr.1 = STRING: </span><br><span class="line">IF-MIB::ifDescr.2 = STRING: </span><br><span class="line">IF-MIB::ifType.1 = INTEGER: ethernetCsmacd(6)</span><br><span class="line">IF-MIB::ifType.2 = INTEGER: tunnel(131)</span><br><span class="line">IF-MIB::ifMtu.1 = INTEGER: 1500</span><br><span class="line">IF-MIB::ifMtu.2 = INTEGER: 1500</span><br><span class="line">IF-MIB::ifSpeed.1 = Gauge32: 1000000000</span><br><span class="line">IF-MIB::ifSpeed.2 = Gauge32: 0</span><br><span class="line">...</span><br></pre></td></tr></table></figure>
<p>10.11.0.5 是 Fortigate 的内网地址，SNMP 监控的 Metric 定义可以根据 Fortigate 的 MiB 库进行定制，具体可以参考<a target="_blank" rel="noopener" href="http://www.oidview.com/mibs/12356/FORTINET-FORTIGATE-MIB.html">这里</a>。部署配置 Collectd，采用分向式配置文件：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br></pre></td><td class="code"><pre><span class="line">$ yum install epel-release -y</span><br><span class="line">$ yum install collectd collectd-snmp -y</span><br><span class="line">$ vi /etc/collectd.d/snmp.conf</span><br><span class="line">LoadPlugin snmp</span><br><span class="line"><Plugin snmp></span><br><span class="line"> <Data <span class="string">"session_count"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"session_count"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.8.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"cpu_usage"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"cpu_usage"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.3.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"version"</span>></span><br><span class="line"> Type <span class="string">"absolute"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"version"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.1.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"memory_usage"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"memory_usage"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.4.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"disk_usage"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"disk_usage"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.6.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"fgSysSesRate1"</span>></span><br><span class="line"> <span class="comment"># The average session setup rate over the past minute.</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"fgSysSesRate1"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.11.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"fgSysSesRate10"</span>></span><br><span class="line"> <span class="comment"># The average session setup rate over the past 10 minute.</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"fgSysSesRate10"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.12.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"fgSysSesRate30"</span>></span><br><span class="line"> <span class="comment"># The average session setup rate over the past 30 minute.</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"fgSysSesRate30"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.13.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"fgSysSesRate60"</span>></span><br><span class="line"> <span class="comment"># The average session setup rate over the past 60 minute.</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"fgSysSesRate60"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.1.14.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"fgHwSensorCount"</span>></span><br><span class="line"> <span class="comment"># The number of entries in fgHwSensorTable</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"fgHwSensorCount"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.4.3.1.0"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"leePrdVpnStatus"</span>></span><br><span class="line"> <span class="comment"># The number of entries in fgHwSensorTable</span></span><br><span class="line"> <span class="comment"># 1.3.6.1.4.1.12356.101.12.2.2.1.3.2 </span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"leePrdVpnStatus"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.12.2.2.1.20.2"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"leeIntVpnStatus"</span>></span><br><span class="line"> <span class="comment"># The number of entries in fgHwSensorTable</span></span><br><span class="line"> <span class="comment"># 1.3.6.1.4.1.12356.101.12.2.2.1.3.3</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"leeIntVpnStatus"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.12.2.2.1.20.3"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"leePpdVpnStatus"</span>></span><br><span class="line"> <span class="comment"># The number of entries in fgHwSensorTable</span></span><br><span class="line"> <span class="comment"># 1.3.6.1.4.1.12356.101.12.2.2.1.3.4</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"leePpdVpnStatus"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.12.2.2.1.20.4"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"AMMPrdVpnStatus"</span>></span><br><span class="line"> <span class="comment"># 1.3.6.1.4.1.12356.101.12.2.2.1.3.5</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"AMMPrdVpnStatus"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.12.2.2.1.20.5"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"AMMAdmVpnStatus"</span>></span><br><span class="line"> <span class="comment"># 1.3.6.1.4.1.12356.101.12.2.2.1.3.6</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"AMMAdmVpnStatus"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.12.2.2.1.20.6"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"AMMIntVpnStatus"</span>></span><br><span class="line"> <span class="comment"># 1.3.6.1.4.1.12356.101.12.2.2.1.3.7</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"AMMIntVpnStatus"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.12.2.2.1.20.7"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"AMMPpdVpnStatus"</span>></span><br><span class="line"> <span class="comment"># 1.3.6.1.4.1.12356.101.12.2.2.1.3.8</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"AMMPddVpnStatus"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.12.2.2.1.20.8"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"CSTVpnStatus"</span>></span><br><span class="line"> <span class="comment"># 1.3.6.1.4.1.12356.101.12.2.2.1.3.4</span></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"CSTVpnStatus"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.4.1.12356.101.12.2.2.1.20.4"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"ifInOctetsmgmt1"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"ifInOctetsmgmt1"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.2.1.2.2.1.10.1"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"ifInOctetsmgmt2"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"ifInOctetsmgmt2"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.2.1.2.2.1.10.2"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"ifInOctetsAMEWAN1"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"ifInOctetsAMEWAN1"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.2.1.2.2.1.10.15"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"ifInOctetsAMELAN1"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"ifInOctetsAMELAN1"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.2.1.2.2.1.10.16"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"ifInOctetsAMELANPRD1"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"ifInOctetsAMELANPRD1"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.2.1.2.2.1.10.17"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"ifInOctetsAMELANPPD1"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"ifInOctetsAMELANPPD1"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.2.1.2.2.1.10.18"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"ifInOctetsAMELANINT1"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"ifInOctetsAMELANINT1"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.2.1.2.2.1.10.19"</span></span><br><span class="line"> </Data> </span><br><span class="line"> <Data <span class="string">"ifInOctetsAMELANADMIN1"</span>></span><br><span class="line"> Type <span class="string">"counter"</span></span><br><span class="line"> Table <span class="literal">false</span></span><br><span class="line"> Instance <span class="string">"ifInOctetsAMELANADMIN1"</span></span><br><span class="line"> Values <span class="string">"1.3.6.1.2.1.2.2.1.10.23"</span></span><br><span class="line"> </Data> </span><br><span class="line"></span><br><span class="line"> <Host <span class="string">"fortigate0001"</span>></span><br><span class="line"> Address <span class="string">"10.11.0.5"</span></span><br><span class="line"> Version 2c</span><br><span class="line"> Community <span class="string">"public"</span></span><br><span class="line"> Collect <span class="string">"session_count"</span> <span class="string">"version"</span> <span class="string">"cpu_usage"</span> <span class="string">"memory_usage"</span> <span class="string">"disk_usage"</span> <span class="string">"fgSysSesRate1"</span> <span class="string">"fgSysSesRate10"</span> <span class="string">"fgSysSesRate30"</span> <span class="string">"fgSysSesRate60"</span> <span class="string">"fgHwSensorCount"</span> <span class="string">"leePrdVpnStatus"</span> <span class="string">"leeIntVpnStatus"</span> <span class="string">"leePpdVpnStatus"</span> <span class="string">"AMMPrdVpnStatus"</span><span class="string">"AMMAdmVpnStatus"</span> <span class="string">"AMMIntVpnStatus"</span> <span class="string">"AMMPpdVpnStatus"</span> <span class="string">"CSTVpnStatus"</span> <span class="string">"ifInOctetsmgmt1"</span> <span class="string">"ifInOctetsmgmt2"</span> <span class="string">"ifInOctetsAMEWAN1"</span> <span class="string">"ifInOctetsAMELAN1"</span> <span class="string">"ifInOctetsAMELANPRD1"</span> <span class="string">"ifInOctetsAMELANPPD1"</span> <span class="string">"ifInOctetsAMELANINT1"</span> <span class="string">"ifInOctetsAMELANADMIN1"</span></span><br><span class="line"> Interval 300</span><br><span class="line"> </Host></span><br><span class="line"></Plugin></span><br></pre></td></tr></table></figure>
<p>保存退出后，配置 Azure Log Analysis Agent，如图所示：</p>
<p><img src=”LA-Installation.jpg” “height:800px” width=”800px” div align=center/></p>
<p>显示绿色图标，代表连接正常。</p>
<h3 id="3-2-部署单节点-MongoDB-Server-4-2-0"><a href="#3-2-部署单节点-MongoDB-Server-4-2-0" class="headerlink" title="3.2 部署单节点 MongoDB Server 4.2.0"></a>3.2 部署单节点 MongoDB Server 4.2.0</h3><h4 id="3-2-1-配置-CentOS-7-7-MongoDB-4-x-Yum-源"><a href="#3-2-1-配置-CentOS-7-7-MongoDB-4-x-Yum-源" class="headerlink" title="3.2.1 配置 CentOS 7.7 MongoDB 4.x Yum 源"></a>3.2.1 配置 CentOS 7.7 MongoDB 4.x Yum 源</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> /etc/yum.repos.d/</span><br><span class="line">$ vi mongodb.repo </span><br><span class="line">[mongodb-org-4.2]</span><br><span class="line">name=MongoDB Repository</span><br><span class="line">baseurl=https://repo.mongodb.org/yum/redhat/<span class="variable">$releasever</span>/mongodb-org/4.2/x86_64/</span><br><span class="line">gpgcheck=1</span><br><span class="line">enabled=1</span><br><span class="line">gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc</span><br></pre></td></tr></table></figure>
<p>保存退出，Yum 源配置完成。</p>
<h4 id="3-2-2-安装单节点-MongoDB-Server-4-2-0"><a href="#3-2-2-安装单节点-MongoDB-Server-4-2-0" class="headerlink" title="3.2.2 安装单节点 MongoDB Server 4.2.0"></a>3.2.2 安装单节点 MongoDB Server 4.2.0</h4><p>安装 MongoDB Server 4.2.0：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y mongodb-org-4.2.0 mongodb-org-server-4.2.0 mongodb-org-shell-4.2.0 mongodb-org-mongos-4.2.0 mongodb-org-tools-4.2.0</span><br></pre></td></tr></table></figure>
<p>检查安装包及版本：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ rpm -qa |grep mongodb</span><br><span class="line">mongodb-org-tools-4.2.0-1.el7.x86_64</span><br><span class="line">mongodb-org-mongos-4.2.0-1.el7.x86_64</span><br><span class="line">mongodb-org-shell-4.2.0-1.el7.x86_64</span><br><span class="line">mongodb-org-4.2.0-1.el7.x86_64</span><br><span class="line">mongodb-org-server-4.2.0-1.el7.x86_64</span><br></pre></td></tr></table></figure>
<p>修改 MongoDB Server 配置文件：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">$ vi /etc/mongod.conf</span><br><span class="line"><span class="comment"># mongod.conf</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># for documentation of all options, see:</span></span><br><span class="line"><span class="comment"># http://docs.mongodb.org/manual/reference/configuration-options/</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># where to write logging data.</span></span><br><span class="line">systemLog:</span><br><span class="line"> destination: file</span><br><span class="line"> logAppend: <span class="literal">true</span></span><br><span class="line"> path: /var/log/mongodb/mongod.log</span><br><span class="line"></span><br><span class="line"><span class="comment"># Where and how to store data.</span></span><br><span class="line">storage:</span><br><span class="line"> dbPath: /var/lib/mongo</span><br><span class="line"> journal:</span><br><span class="line"> enabled: <span class="literal">true</span></span><br><span class="line"><span class="comment"># engine:</span></span><br><span class="line"><span class="comment"># wiredTiger:</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># how the process runs</span></span><br><span class="line">processManagement:</span><br><span class="line"> fork: <span class="literal">true</span> <span class="comment"># fork and run in background</span></span><br><span class="line"> pidFilePath: /var/run/mongodb/mongod.pid <span class="comment"># location of pidfile</span></span><br><span class="line"> timeZoneInfo: /usr/share/zoneinfo</span><br><span class="line"></span><br><span class="line"><span class="comment"># network interfaces</span></span><br><span class="line">net:</span><br><span class="line"> port: 27017</span><br><span class="line"> bindIp: 0.0.0.0 <span class="comment"># Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#security:</span></span><br><span class="line"><span class="comment">#operationProfiling:</span></span><br><span class="line">replication:</span><br><span class="line"> oplogSizeMB: <span class="string">"20480"</span></span><br><span class="line"> replSetName: repconfig </span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>启动 MongoDB Sever 服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">systemctl start mongod && systemctl <span class="built_in">enable</span> mongod</span><br></pre></td></tr></table></figure>
<p>MongoDB 初始化：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">$ mongo --port 27017</span><br><span class="line">> repconfig = { _id : <span class="string">"repconfig"</span>, members : [ {_id : 0, host : <span class="string">"10.11.0.4:27017"</span> , priority: 1 } ] }</span><br><span class="line">{</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"repconfig"</span>,</span><br><span class="line"> <span class="string">"members"</span> : [</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"_id"</span> : 0,</span><br><span class="line"> <span class="string">"host"</span> : <span class="string">"10.11.0.4:27017"</span>,</span><br><span class="line"> <span class="string">"priority"</span> : 1</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line">}</span><br><span class="line">> rs.initiate(repconfig);</span><br><span class="line">{</span><br><span class="line"> <span class="string">"ok"</span> : 1,</span><br><span class="line"> <span class="string">"<span class="variable">$clusterTime</span>"</span> : {</span><br><span class="line"> <span class="string">"clusterTime"</span> : Timestamp(1579339466, 1),</span><br><span class="line"> <span class="string">"signature"</span> : {</span><br><span class="line"> <span class="string">"hash"</span> : BinData(0,<span class="string">"AAAAAAAAAAAAAAAAAAAAAAAAAAA="</span>),</span><br><span class="line"> <span class="string">"keyId"</span> : NumberLong(0)</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"operationTime"</span> : Timestamp(1579339466, 1)</span><br><span class="line">}</span><br><span class="line">repconfig:SECONDARY> </span><br><span class="line">repconfig:PRIMARY></span><br></pre></td></tr></table></figure>
<p>初始化结束后，可以通过 Mongo Shell rs.status() 来查 Replica 信息。</p>
<h3 id="3-3-通过-Python-调用-Azure-Monitor-Restful-API-来监控-Mongodb-Server-4-2-0"><a href="#3-3-通过-Python-调用-Azure-Monitor-Restful-API-来监控-Mongodb-Server-4-2-0" class="headerlink" title="3.3 通过 Python 调用 Azure Monitor Restful API 来监控 Mongodb Server 4.2.0"></a>3.3 通过 Python 调用 Azure Monitor Restful API 来监控 Mongodb Server 4.2.0</h3><h4 id="3-3-1-创建-Azure-Log-Analysis-Workspace"><a href="#3-3-1-创建-Azure-Log-Analysis-Workspace" class="headerlink" title="3.3.1 创建 Azure Log Analysis Workspace"></a>3.3.1 创建 Azure Log Analysis Workspace</h4><p>具体操作过程不再赘述，通过 Portal 或者 Cli 等方式 Step by Step 点击创建即可，如图：</p>
<p><img src=”LA-Creation.jpg” “height:800px” width=”600px” div align=center/></p>
<p>创建好了需要的几个信息，如图：</p>
<p><img src=”LA-Info.jpg” “height:800px” width=”800px” div align=center/></p>
<p>“WORKSPACE ID” 为脚本中的 customer_id，”PRIMARY KEY” 为脚本中的 shared_key。</p>
<h4 id="3-3-2-Python-调用-Azure-Monitor-Restful-API-向-Azure-Log-Analysis-Workspace-传送数据"><a href="#3-3-2-Python-调用-Azure-Monitor-Restful-API-向-Azure-Log-Analysis-Workspace-传送数据" class="headerlink" title="3.3.2 Python 调用 Azure Monitor Restful API 向 Azure Log Analysis Workspace 传送数据"></a>3.3.2 Python 调用 Azure Monitor Restful API 向 Azure Log Analysis Workspace 传送数据</h4><p>Azure Monitor Restful API 不做赘述，具体可以参考<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api">这里</a>。本实验先通过 pymongo 收集 MongoDB Metrics，然后送到 Azure Log Analysis Workspace 中，先安装 pymongo：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum install python-pip y</span><br><span class="line">$ pip install pymongo requests </span><br></pre></td></tr></table></figure>
<p>具体的 Python 脚本如下：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br><span class="line">239</span><br><span class="line">240</span><br><span class="line">241</span><br><span class="line">242</span><br><span class="line">243</span><br><span class="line">244</span><br><span class="line">245</span><br><span class="line">246</span><br><span class="line">247</span><br><span class="line">248</span><br><span class="line">249</span><br><span class="line">250</span><br><span class="line">251</span><br><span class="line">252</span><br><span class="line">253</span><br><span class="line">254</span><br><span class="line">255</span><br><span class="line">256</span><br><span class="line">257</span><br><span class="line">258</span><br><span class="line">259</span><br><span class="line">260</span><br><span class="line">261</span><br><span class="line">262</span><br><span class="line">263</span><br><span class="line">264</span><br><span class="line">265</span><br><span class="line">266</span><br><span class="line">267</span><br><span class="line">268</span><br><span class="line">269</span><br><span class="line">270</span><br><span class="line">271</span><br><span class="line">272</span><br><span class="line">273</span><br><span class="line">274</span><br><span class="line">275</span><br><span class="line">276</span><br><span class="line">277</span><br><span class="line">278</span><br><span class="line">279</span><br><span class="line">280</span><br><span class="line">281</span><br><span class="line">282</span><br><span class="line">283</span><br><span class="line">284</span><br><span class="line">285</span><br><span class="line">286</span><br><span class="line">287</span><br><span class="line">288</span><br><span class="line">289</span><br><span class="line">290</span><br><span class="line">291</span><br><span class="line">292</span><br><span class="line">293</span><br><span class="line">294</span><br><span class="line">295</span><br><span class="line">296</span><br><span class="line">297</span><br><span class="line">298</span><br><span class="line">299</span><br><span class="line">300</span><br><span class="line">301</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">Date: 01/18/2020</span></span><br><span class="line"><span class="string">Author: Xinsheng Wang</span></span><br><span class="line"><span class="string">Description: A custom script to get MongoDB metrics and send data to Azure Monitor </span></span><br><span class="line"><span class="string">Requires: MongoClient in python</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> calendar <span class="keyword">import</span> timegm</span><br><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> gmtime</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> pymongo <span class="keyword">import</span> MongoClient, errors</span><br><span class="line"><span class="keyword">from</span> sys <span class="keyword">import</span> exit</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> json</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">import</span> datetime</span><br><span class="line"><span class="keyword">import</span> hashlib</span><br><span class="line"><span class="keyword">import</span> hmac</span><br><span class="line"><span class="keyword">import</span> base64</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">from</span> glob <span class="keyword">import</span> glob</span><br><span class="line"></span><br><span class="line"><span class="comment"># Update the customer ID to your Log Analytics workspace ID</span></span><br><span class="line">customer_id = <span class="string">'86df0cbc-076c-4483-8a32-c59c6550a771'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># For the shared key, use either the primary or the secondary Connected Sources client authentication key </span></span><br><span class="line">shared_key = <span class="string">"b3uLsEOXBFBqTiAHDGp9boTeKR6v86f/9cLPWWsWUvs+LcjBIqjDp9CDJL+7vxlKDDRxqXIf1jjjKcZbdV0H/Q=="</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># The log type is the name of the event that is being submitted</span></span><br><span class="line">log_type = <span class="string">'MongoDBMonitorLog'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">MongoDB</span>(<span class="title class_ inherited__">object</span>):</span><br><span class="line"> <span class="string">"""main script class"""</span></span><br><span class="line"> <span class="comment"># pylint: disable=too-many-instance-attributes</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">delete_temporary_files</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""delete temporary files"""</span></span><br><span class="line"> <span class="keyword">for</span> file <span class="keyword">in</span> glob(<span class="string">'/tmp/mongometrics000*'</span>):</span><br><span class="line"> os.remove(file)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self</span>):</span><br><span class="line"> self.mongo_host = <span class="string">"10.11.0.4"</span></span><br><span class="line"> self.mongo_port = <span class="number">27017</span></span><br><span class="line"> self.mongo_db = [<span class="string">"admin"</span>, ]</span><br><span class="line"> self.mongo_user = <span class="literal">None</span></span><br><span class="line"> self.mongo_password = <span class="literal">None</span></span><br><span class="line"> self.__conn = <span class="literal">None</span></span><br><span class="line"> self.__dbnames = <span class="literal">None</span></span><br><span class="line"> self.__metrics = []</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">connect</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""Connect to MongoDB"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">if</span> self.mongo_user <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> self.__conn = MongoClient(<span class="string">'mongodb://%s:%s'</span> %</span><br><span class="line"> (self.mongo_host,</span><br><span class="line"> self.mongo_port))</span><br><span class="line"> <span class="keyword">except</span> errors.PyMongoError <span class="keyword">as</span> py_mongo_error:</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Error in MongoDB connection: %s'</span> %</span><br><span class="line"> <span class="built_in">str</span>(py_mongo_error))</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> self.__conn = MongoClient(<span class="string">'mongodb://%s:%s@%s:%s'</span> %</span><br><span class="line"> (self.mongo_user,</span><br><span class="line"> self.mongo_password,</span><br><span class="line"> self.mongo_host,</span><br><span class="line"> self.mongo_port))</span><br><span class="line"> <span class="keyword">except</span> errors.PyMongoError <span class="keyword">as</span> py_mongo_error:</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Error in MongoDB connection: %s'</span> %</span><br><span class="line"> <span class="built_in">str</span>(py_mongo_error))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">add_metrics</span>(<span class="params">self, k, v</span>):</span><br><span class="line"> <span class="string">"""add each metric to the metrics list"""</span></span><br><span class="line"> <span class="keyword">global</span> body</span><br><span class="line"> dict_metrics = {}</span><br><span class="line"> dict_metrics[<span class="string">"key"</span>] = k</span><br><span class="line"> dict_metrics[<span class="string">"value"</span>] = v</span><br><span class="line"> self.__metrics.append(dict_metrics)</span><br><span class="line"> dic = json.dumps(dict_metrics, sort_keys=<span class="literal">True</span>, indent=<span class="number">4</span>, separators=(<span class="string">','</span>, <span class="string">':'</span>)).replace(<span class="string">'}'</span>, <span class="string">'},'</span>)</span><br><span class="line"> </span><br><span class="line"> f = <span class="built_in">open</span>(<span class="string">'/tmp/mongometrics0001.txt'</span>,<span class="string">'a'</span>)</span><br><span class="line"> f.write(dic)</span><br><span class="line"> f.close</span><br><span class="line"> </span><br><span class="line"> os.system(<span class="string">"cat /tmp/mongometrics0001.txt |sed '$s/\}\,/\}\]/g;1s/{/[{/' > /tmp/mongometrics0002.txt"</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">with</span> <span class="built_in">open</span>(<span class="string">'/tmp/mongometrics0002.txt'</span>,<span class="string">'r'</span>) <span class="keyword">as</span> src:</span><br><span class="line"> body = src.read()</span><br><span class="line"> <span class="built_in">print</span>(body)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_db_names</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get a list of DB names"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> db_handler = self.__conn[self.mongo_db[<span class="number">0</span>]]</span><br><span class="line"></span><br><span class="line"> master = db_handler.command(<span class="string">'isMaster'</span>)[<span class="string">'ismaster'</span>]</span><br><span class="line"> dict_metrics = {}</span><br><span class="line"> dict_metrics[<span class="string">'key'</span>] = <span class="string">'mongodb.ismaster'</span></span><br><span class="line"> <span class="keyword">if</span> master:</span><br><span class="line"> dict_metrics[<span class="string">'value'</span>] = <span class="number">1</span></span><br><span class="line"> db_names = self.__conn.database_names()</span><br><span class="line"> self.__dbnames = db_names</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> dict_metrics[<span class="string">'value'</span>] = <span class="number">0</span></span><br><span class="line"> self.__metrics.append(dict_metrics)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_mongo_db_lld</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""print DB list in json format, to be used for mongo db discovery"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__dbnames <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> db_names = self.get_db_names()</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> db_names = self.__dbnames</span><br><span class="line"> dict_metrics = {}</span><br><span class="line"> db_list = []</span><br><span class="line"> dict_metrics[<span class="string">'key'</span>] = <span class="string">'mongodb.discovery'</span></span><br><span class="line"> dict_metrics[<span class="string">'value'</span>] = {<span class="string">"data"</span>: db_list}</span><br><span class="line"> <span class="keyword">if</span> db_names <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">for</span> db_name <span class="keyword">in</span> db_names:</span><br><span class="line"> dict_lld_metric = {}</span><br><span class="line"> dict_lld_metric[<span class="string">'{#MONGODBNAME}'</span>] = db_name</span><br><span class="line"> db_list.append(dict_lld_metric)</span><br><span class="line"> dict_metrics[<span class="string">'value'</span>] = <span class="string">'{"data": '</span> + json.dumps(db_list) + <span class="string">'}'</span></span><br><span class="line"> self.__metrics.insert(<span class="number">0</span>, dict_metrics)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_oplog</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get replica set oplog information"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> db_handler = self.__conn[<span class="string">'local'</span>]</span><br><span class="line"></span><br><span class="line"> coll = db_handler.oplog.rs</span><br><span class="line"></span><br><span class="line"> op_first = (coll.find().sort(<span class="string">'$natural'</span>, <span class="number">1</span>).limit(<span class="number">1</span>))</span><br><span class="line"> op_last = (coll.find().sort(<span class="string">'$natural'</span>, -<span class="number">1</span>).limit(<span class="number">1</span>))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># if host is not a member of replica set, without this check we will</span></span><br><span class="line"> <span class="comment"># raise StopIteration as guided in</span></span><br><span class="line"> <span class="comment"># http://api.mongodb.com/python/current/api/pymongo/cursor.html</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> op_first.count() > <span class="number">0</span> <span class="keyword">and</span> op_last.count() > <span class="number">0</span>:</span><br><span class="line"> op_fst = (op_first.<span class="built_in">next</span>())[<span class="string">'ts'</span>].time</span><br><span class="line"> op_last_st = op_last[<span class="number">0</span>][<span class="string">'ts'</span>]</span><br><span class="line"> op_lst = (op_last.<span class="built_in">next</span>())[<span class="string">'ts'</span>].time</span><br><span class="line"></span><br><span class="line"> status = <span class="built_in">round</span>(<span class="built_in">float</span>(op_lst - op_fst), <span class="number">1</span>)</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.oplog'</span>, status)</span><br><span class="line"></span><br><span class="line"> current_time = timegm(gmtime())</span><br><span class="line"> oplog = <span class="built_in">int</span>(((<span class="built_in">str</span>(op_last_st).split(<span class="string">'('</span>))[<span class="number">1</span>].split(<span class="string">','</span>))[<span class="number">0</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.oplog-sync'</span>, (current_time - oplog))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_maintenance</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get replica set maintenance info"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> db_handler = self.__conn</span><br><span class="line"></span><br><span class="line"> fsync_locked = <span class="built_in">int</span>(db_handler.is_locked)</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.fsync-locked'</span>, fsync_locked)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> config = db_handler.admin.command(<span class="string">"replSetGetConfig"</span>, <span class="number">1</span>)</span><br><span class="line"> connstring = (self.mongo_host + <span class="string">':'</span> + <span class="built_in">str</span>(self.mongo_port))</span><br><span class="line"> connstrings = <span class="built_in">list</span>()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">0</span>, <span class="built_in">len</span>(config[<span class="string">'config'</span>][<span class="string">'members'</span>])):</span><br><span class="line"> host = config[<span class="string">'config'</span>][<span class="string">'members'</span>][i][<span class="string">'host'</span>]</span><br><span class="line"> connstrings.append(host)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> connstring <span class="keyword">in</span> host:</span><br><span class="line"> priority = config[<span class="string">'config'</span>][<span class="string">'members'</span>][i][<span class="string">'priority'</span>]</span><br><span class="line"> hidden = <span class="built_in">int</span>(config[<span class="string">'config'</span>][<span class="string">'members'</span>][i][<span class="string">'hidden'</span>])</span><br><span class="line"></span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.priority'</span>, priority)</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.hidden'</span>, hidden)</span><br><span class="line"> <span class="keyword">except</span> errors.PyMongoError:</span><br><span class="line"> <span class="built_in">print</span> (<span class="string">'Error while fetching replica set configuration.'</span></span><br><span class="line"> <span class="string">'Not a member of replica set?'</span>)</span><br><span class="line"> <span class="keyword">except</span> UnboundLocalError:</span><br><span class="line"> <span class="built_in">print</span> (<span class="string">'Cannot use this mongo host: must be one of '</span> + <span class="string">','</span>.join(connstrings))</span><br><span class="line"> exit(<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_server_status_metrics</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get server status"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> db_handler = self.__conn[self.mongo_db[<span class="number">0</span>]]</span><br><span class="line"> ss = db_handler.command(<span class="string">'serverStatus'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># db info</span></span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.version'</span>, ss[<span class="string">'version'</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.storageEngine'</span>, ss[<span class="string">'storageEngine'</span>][<span class="string">'name'</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.uptime'</span>, <span class="built_in">int</span>(ss[<span class="string">'uptime'</span>]))</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.okstatus'</span>, <span class="built_in">int</span>(ss[<span class="string">'ok'</span>]))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># asserts</span></span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'asserts'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.asserts.'</span> + k, v)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># operations</span></span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'opcounters'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.operation.'</span> + k, v)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># connections</span></span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'connections'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.connection.'</span> + k, v)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># extra info</span></span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.page.faults'</span>,</span><br><span class="line"> ss[<span class="string">'extra_info'</span>][<span class="string">'page_faults'</span>])</span><br><span class="line"></span><br><span class="line"> <span class="comment">#wired tiger</span></span><br><span class="line"> <span class="keyword">if</span> ss[<span class="string">'storageEngine'</span>][<span class="string">'name'</span>] == <span class="string">'wiredTiger'</span>:</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.used-cache'</span>,</span><br><span class="line"> ss[<span class="string">'wiredTiger'</span>][<span class="string">'cache'</span>]</span><br><span class="line"> [<span class="string">"bytes currently in the cache"</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.total-cache'</span>,</span><br><span class="line"> ss[<span class="string">'wiredTiger'</span>][<span class="string">'cache'</span>]</span><br><span class="line"> [<span class="string">"maximum bytes configured"</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.dirty-cache'</span>,</span><br><span class="line"> ss[<span class="string">'wiredTiger'</span>][<span class="string">'cache'</span>]</span><br><span class="line"> [<span class="string">"tracked dirty bytes in the cache"</span>])</span><br><span class="line"></span><br><span class="line"> <span class="comment"># global lock</span></span><br><span class="line"> lock_total_time = ss[<span class="string">'globalLock'</span>][<span class="string">'totalTime'</span>]</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.globalLock.totalTime'</span>, lock_total_time)</span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'globalLock'</span>][<span class="string">'currentQueue'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.globalLock.currentQueue.'</span> + k, v)</span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'globalLock'</span>][<span class="string">'activeClients'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.globalLock.activeClients.'</span> + k, v)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_db_stats_metrics</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get DB stats for each DB"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> <span class="keyword">if</span> self.__dbnames <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.get_db_names()</span><br><span class="line"> <span class="keyword">if</span> self.__dbnames <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">for</span> mongo_db <span class="keyword">in</span> self.__dbnames:</span><br><span class="line"> db_handler = self.__conn[mongo_db]</span><br><span class="line"> dbs = db_handler.command(<span class="string">'dbstats'</span>)</span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> dbs.items():</span><br><span class="line"> <span class="keyword">if</span> k <span class="keyword">in</span> [<span class="string">'storageSize'</span>, <span class="string">'ok'</span>, <span class="string">'avgObjSize'</span>, <span class="string">'indexes'</span>,</span><br><span class="line"> <span class="string">'objects'</span>, <span class="string">'collections'</span>, <span class="string">'fileSize'</span>,</span><br><span class="line"> <span class="string">'numExtents'</span>, <span class="string">'dataSize'</span>, <span class="string">'indexSize'</span>,</span><br><span class="line"> <span class="string">'nsSizeMB'</span>]:</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.stats.'</span> + k +</span><br><span class="line"> <span class="string">'['</span> + mongo_db + <span class="string">']'</span>, <span class="built_in">int</span>(v))</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">close</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""close connection to mongo"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> self.__conn.close()</span><br><span class="line"></span><br><span class="line"><span class="comment"># Build the API signature</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">build_signature</span>(<span class="params">customer_id, shared_key, date, content_length, method, content_type, resource</span>):</span><br><span class="line"> x_headers = <span class="string">'x-ms-date:'</span> + date</span><br><span class="line"> string_to_hash = method + <span class="string">"\n"</span> + <span class="built_in">str</span>(content_length) + <span class="string">"\n"</span> + content_type + <span class="string">"\n"</span> + x_headers + <span class="string">"\n"</span> + resource</span><br><span class="line"> bytes_to_hash = <span class="built_in">bytes</span>(string_to_hash).encode(<span class="string">'utf-8'</span>) </span><br><span class="line"> decoded_key = base64.b64decode(shared_key)</span><br><span class="line"> encoded_hash = base64.b64encode(hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest())</span><br><span class="line"> authorization = <span class="string">"SharedKey {}:{}"</span>.<span class="built_in">format</span>(customer_id,encoded_hash)</span><br><span class="line"> <span class="keyword">return</span> authorization</span><br><span class="line"></span><br><span class="line"><span class="comment"># Build and send a request to the POST API</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">mongodb_azuremonitor_loganalysis_post_data</span>(<span class="params">customer_id, shared_key, body, log_type</span>):</span><br><span class="line"> method = <span class="string">'POST'</span></span><br><span class="line"> content_type = <span class="string">'application/json'</span></span><br><span class="line"> resource = <span class="string">'/api/logs'</span></span><br><span class="line"> rfc1123date = datetime.datetime.utcnow().strftime(<span class="string">'%a, %d %b %Y %H:%M:%S GMT'</span>)</span><br><span class="line"> content_length = <span class="built_in">len</span>(body)</span><br><span class="line"> signature = build_signature(customer_id, shared_key, rfc1123date, content_length, method, content_type, resource)</span><br><span class="line"> uri = <span class="string">'https://'</span> + customer_id + <span class="string">'.ods.opinsights.azure.com'</span> + resource + <span class="string">'?api-version=2016-04-01'</span></span><br><span class="line"></span><br><span class="line"> headers = {</span><br><span class="line"> <span class="string">'content-type'</span>: content_type,</span><br><span class="line"> <span class="string">'Authorization'</span>: signature,</span><br><span class="line"> <span class="string">'Log-Type'</span>: log_type,</span><br><span class="line"> <span class="string">'x-ms-date'</span>: rfc1123date</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> response = requests.post(uri,data=body, headers=headers)</span><br><span class="line"> <span class="keyword">if</span> (response.status_code >= <span class="number">200</span> <span class="keyword">and</span> response.status_code <= <span class="number">299</span>):</span><br><span class="line"> <span class="built_in">print</span> <span class="string">'Accepted'</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="built_in">print</span> <span class="string">"Response code: {}"</span>.<span class="built_in">format</span>(response.status_code)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> mongodb = MongoDB()</span><br><span class="line"> mongodb.delete_temporary_files()</span><br><span class="line"> mongodb.get_db_names()</span><br><span class="line"> mongodb.get_mongo_db_lld()</span><br><span class="line"> mongodb.get_oplog()</span><br><span class="line"> mongodb.get_maintenance()</span><br><span class="line"> mongodb.get_server_status_metrics()</span><br><span class="line"> mongodb.get_db_stats_metrics()</span><br><span class="line"> mongodb.close()</span><br><span class="line"> mongodb_azuremonitor_loganalysis_post_data(customer_id, shared_key, body, log_type)</span><br></pre></td></tr></table></figure>
<p>注意相应的 customer_id 和 shared_key 替换成图中的 key，log_type 为收集到 Log Analysis Workspace 的 namespace 名字。保存脚本名字为 AzureMonitor_LogAnalysis_MongodbMetrics.py，执行脚本：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">python AzureMonitor_LogAnalysis_MongodbMetrics.py</span><br></pre></td></tr></table></figure>
<p>通过 print 可以打印一些交互式信息，观察数据是否已经送到 Azure Log Analysis Workspace。</p>
<h4 id="3-3-3-利用-Kusto-查询-MongoDB-Custom-Logs"><a href="#3-3-3-利用-Kusto-查询-MongoDB-Custom-Logs" class="headerlink" title="3.3.3 利用 Kusto 查询 MongoDB Custom Logs"></a>3.3.3 利用 Kusto 查询 MongoDB Custom Logs</h4><p>在 Azure Portal 查看并通过 KQL 查询 MongoDB 的数据：</p>
<p><img src=”Azure-Monitor-Restful-API-MongoDB.jpg” “height:800px” width=”800px” div align=center/></p>
<p>如图可见，数据已经成功送到了 Azure Log Analysis Workspace，后续可以写更复杂的 Kusto 来做相关的查询，并可以考虑结合调度或者 Linux Crontab 等服务做定时的数据拉取和告警了。</p>
<h2 id="4-总结"><a href="#4-总结" class="headerlink" title="4. 总结"></a>4. 总结</h2><p>测试至此，Azure Monitor Restful API 监控 MongoDB 的示例完成了，希望能够给想要通过 Azure Monitor 做二次开发的同学们一些参考。但本文还有一些可以优化之处，比如 Python 脚本没有考虑更多的 Log Output 以及执行失败时候的 Retry 逻辑，在实际生产中，为了严谨，还是希望大家要充分考虑逻辑在上线，不过这些功能有待大家自己添加了。</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2020/01/23/2020-01-23-AzureMonitorSNMP/" data-id="clat72drc0009yofkdpwpc4eb" class="article-share-link" data-share="baidu" data-title="Azure Monitor 通过 SNMP 协议监控网络设备 Fortigate Firewall">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Fortigate/" rel="tag">Fortigate</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Monitor/" rel="tag">Monitor</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/SNMP/" rel="tag">SNMP</a></li></ul>
</footer>
</div>
</article>
<article id="post-2020-01-18-AzureMonitorRestfulAPI" class="article article-type-post" itemscope itemprop="blogPost">
<div class="article-meta">
<a href="/wxsblog.github.io/2020/01/18/2020-01-18-AzureMonitorRestfulAPI/" class="article-date">
<time datetime="2020-01-18T07:45:29.000Z" itemprop="datePublished">2020-01-18</time>
</a>
<div class="article-category">
<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/">Monitor</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Database/">Database</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Database/MongoDB/">MongoDB</a>►<a class="article-category-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Database/MongoDB/Linux/">Linux</a>
</div>
</div>
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/wxsblog.github.io/2020/01/18/2020-01-18-AzureMonitorRestfulAPI/">集成 Microsoft Azure Monitor Restful API 来监控 MongoDB</a>
</h1>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a>1. 前言</h2><p>随时 IT 技术的蓬勃发展，出现了越来越多地以各种技术栈为核心的应用程序，同时应用架构和服务的调用关系也就变得越来越复杂，所以建立一个强大的监控系统的必要性就很自然的体现了出来。一般在实际的生产环境中，监控对象类型丰富多样，工具平台也有很多可以来选择。比如从最早的 Smokeping，Cacti，Nagios，Ganglia 发展到现在比较流行功能也更加强大的 Zabbix，Prometheus，Openfalcon 以及偏业务层做代码追踪的监控平台 CAT，Zipkin 等，这些都是很不错的工具和平台，只要运用合适，都可以解决很多场景和不同维度的监控需求。但是维护这些监控平台并不是一件容易的事情，从可靠性、性能到定制化开发，都需要花费很多的人力物力来做底层代码层面的修改，不然直接上生产有非常大的风险。依据此背景，Azure 也推出自己的监控服务 Azure Monitor，帮助客户不需要担心监控系统的可靠性，只需要关注业务的可靠性和监控需求的落地即可。Azure Monitor 支持从基础架构层、应用层甚至于容器层面的监控。本文主要介绍通过 Azure Monitor Restful API 来收集 MongoDB 的指标，从而来模拟通过 Azure Monitor 来做二开监控平台的场景。</p>
<br>
<h2 id="2-Azure-Monitor-介绍"><a href="#2-Azure-Monitor-介绍" class="headerlink" title="2. Azure Monitor 介绍"></a>2. Azure Monitor 介绍</h2><p>Azure Monitor 提供用于收集、分析和处理来自云与本地环境的遥测数据的综合解决方案，可将应用程序和服务的可用性和性能最大化。它可以帮助你了解应用程序的性能，并主动识别影响应用程序及其所依赖资源的问题。下图提供了 Azure Monitor 的概要视图。示意图的中心是用于存储指标和日志 ( Azure Monitor 使用的两种基本类型的数据 ) 的数据存储，左侧是用于填充这些数据存储的监视数据源，右侧是 Azure Monitor 针对这些收集的数据执行的不同功能，例如分析、警报和流式传输到外部系统。</p>
<p><a href="Azure-Monitor-Arch.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-01-18-AzureMonitorRestfulAPI/Azure-Monitor-Arch.jpg" height="800" width="800"/></a></p>
<p>Azure Monitor 支持监控的数据类型的层级都很多，具体可以参考<a target="_blank" rel="noopener" href="https://docs.microsoft.com/zh-cn/azure/azure-monitor/overview">官方文档</a>，就不在此赘述了。对于收集上来的数据，Azure Monitor 使用 Azure Data Explorer 使用的 Kusto 查询语言来做查询，同时包括高级功能，例如聚合、联接和智能分析等。话不多说，直接开干。</p>
<br>
<h2 id="3-Azure-Monitor-通过-Restful-API-监控-MongoDB"><a href="#3-Azure-Monitor-通过-Restful-API-监控-MongoDB" class="headerlink" title="3. Azure Monitor 通过 Restful API 监控 MongoDB"></a>3. Azure Monitor 通过 Restful API 监控 MongoDB</h2><h3 id="3-1-创建测试服务器-CentOS-Server"><a href="#3-1-创建测试服务器-CentOS-Server" class="headerlink" title="3.1 创建测试服务器 CentOS Server"></a>3.1 创建测试服务器 CentOS Server</h3><p>首先，通过 Terraform 自动化部署 CentOS 7.7 实例，具体的 .tf 文件内容如下，其中变量引用</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Configure the Microsoft Azure Provider</span></span><br><span class="line">provider <span class="string">"azurerm"</span> {</span><br><span class="line"> subscription_id = <span class="string">"<span class="variable">${var.subscription_id}</span>"</span></span><br><span class="line"> client_id = <span class="string">"<span class="variable">${var.client_id}</span>"</span></span><br><span class="line"> client_secret = <span class="string">"<span class="variable">${var.client_secret}</span>"</span></span><br><span class="line"> tenant_id = <span class="string">"<span class="variable">${var.tenant_id}</span>"</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create a resource group if it doesn’t exist</span></span><br><span class="line">resource <span class="string">"azurerm_resource_group"</span> <span class="string">"myterraformgroup"</span> {</span><br><span class="line"> name = <span class="string">"<span class="variable">${var.lab_namespace}</span>rg0001"</span></span><br><span class="line"> location = <span class="string">"<span class="variable">${var.location}</span>"</span></span><br><span class="line"> tags = {</span><br><span class="line"> environment = <span class="string">"Azure Terraform Automation"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create virtual network</span></span><br><span class="line">resource <span class="string">"azurerm_virtual_network"</span> <span class="string">"myterraformnetwork"</span> {</span><br><span class="line"> name = <span class="string">"<span class="variable">${var.lab_namespace}</span>vnet0001"</span></span><br><span class="line"> address_space = [<span class="string">"10.11.0.0/16"</span>]</span><br><span class="line"> location = <span class="string">"<span class="variable">${var.location}</span>"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"></span><br><span class="line"> tags = {</span><br><span class="line"> environment = <span class="string">"Azure Terraform Automation"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create Network Security Group and rule</span></span><br><span class="line">resource <span class="string">"azurerm_network_security_group"</span> <span class="string">"publicnsg"</span> {</span><br><span class="line"> name = <span class="string">"public-nsg"</span></span><br><span class="line"> location = <span class="string">"<span class="variable">${var.location}</span>"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"> </span><br><span class="line"> security_rule {</span><br><span class="line"> name = <span class="string">"SSH"</span></span><br><span class="line"> priority = 1001</span><br><span class="line"> direction = <span class="string">"Inbound"</span></span><br><span class="line"> access = <span class="string">"Allow"</span></span><br><span class="line"> protocol = <span class="string">"Tcp"</span></span><br><span class="line"> source_port_range = <span class="string">"*"</span></span><br><span class="line"> destination_port_range = <span class="string">"22"</span></span><br><span class="line"> source_address_prefix = <span class="string">"*"</span></span><br><span class="line"> destination_address_prefix = <span class="string">"*"</span></span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> security_rule {</span><br><span class="line"> name = <span class="string">"RDP"</span></span><br><span class="line"> priority = 1002</span><br><span class="line"> direction = <span class="string">"Inbound"</span></span><br><span class="line"> access = <span class="string">"Allow"</span></span><br><span class="line"> protocol = <span class="string">"Tcp"</span></span><br><span class="line"> source_port_range = <span class="string">"*"</span></span><br><span class="line"> destination_port_range = <span class="string">"3389"</span></span><br><span class="line"> source_address_prefix = <span class="string">"*"</span></span><br><span class="line"> destination_address_prefix = <span class="string">"*"</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> tags = {</span><br><span class="line"> environment = <span class="string">"Azure Terraform Automation"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line">resource <span class="string">"azurerm_network_security_group"</span> <span class="string">"opnsg"</span> {</span><br><span class="line"> name = <span class="string">"op-nsg"</span></span><br><span class="line"> location = <span class="string">"<span class="variable">${var.location}</span>"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"> </span><br><span class="line"> security_rule {</span><br><span class="line"> name = <span class="string">"SSH"</span></span><br><span class="line"> priority = 1001</span><br><span class="line"> direction = <span class="string">"Inbound"</span></span><br><span class="line"> access = <span class="string">"Allow"</span></span><br><span class="line"> protocol = <span class="string">"Tcp"</span></span><br><span class="line"> source_port_range = <span class="string">"*"</span></span><br><span class="line"> destination_port_range = <span class="string">"22"</span></span><br><span class="line"> source_address_prefix = <span class="string">"*"</span></span><br><span class="line"> destination_address_prefix = <span class="string">"*"</span></span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> security_rule {</span><br><span class="line"> name = <span class="string">"RDP"</span></span><br><span class="line"> priority = 1002</span><br><span class="line"> direction = <span class="string">"Inbound"</span></span><br><span class="line"> access = <span class="string">"Allow"</span></span><br><span class="line"> protocol = <span class="string">"Tcp"</span></span><br><span class="line"> source_port_range = <span class="string">"*"</span></span><br><span class="line"> destination_port_range = <span class="string">"3389"</span></span><br><span class="line"> source_address_prefix = <span class="string">"*"</span></span><br><span class="line"> destination_address_prefix = <span class="string">"*"</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> tags = {</span><br><span class="line"> environment = <span class="string">"Azure Terraform Automation"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create subnet</span></span><br><span class="line">resource <span class="string">"azurerm_subnet"</span> <span class="string">"publicsubnet"</span> {</span><br><span class="line"> name = <span class="string">"publicsubnet0001"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"> virtual_network_name = <span class="string">"<span class="variable">${azurerm_virtual_network.myterraformnetwork.name}</span>"</span></span><br><span class="line"> address_prefix = <span class="string">"10.11.0.0/23"</span></span><br><span class="line"> network_security_group_id = <span class="string">"<span class="variable">${azurerm_network_security_group.publicnsg.id}</span>"</span></span><br><span class="line">}</span><br><span class="line">resource <span class="string">"azurerm_subnet"</span> <span class="string">"opsubnet"</span> {</span><br><span class="line"> name = <span class="string">"opsubnet0001"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"> virtual_network_name = <span class="string">"<span class="variable">${azurerm_virtual_network.myterraformnetwork.name}</span>"</span></span><br><span class="line"> address_prefix = <span class="string">"10.11.2.0/23"</span></span><br><span class="line"> network_security_group_id = <span class="string">"<span class="variable">${azurerm_network_security_group.opnsg.id}</span>"</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create Public IP</span></span><br><span class="line">resource <span class="string">"azurerm_public_ip"</span> <span class="string">"centospips"</span> {</span><br><span class="line"> name = <span class="string">"centos0001pip"</span></span><br><span class="line"> location = <span class="string">"<span class="variable">${var.location}</span>"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"> allocation_method = <span class="string">"Static"</span></span><br><span class="line"></span><br><span class="line"> tags = {</span><br><span class="line"> environment = <span class="string">"Azure Terraform Automation"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create Network interface</span></span><br><span class="line">resource <span class="string">"azurerm_network_interface"</span> <span class="string">"centosnics"</span> {</span><br><span class="line"> name = <span class="string">"centos0001nic0001"</span></span><br><span class="line"> location = <span class="string">"<span class="variable">${var.location}</span>"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"></span><br><span class="line"> ip_configuration {</span><br><span class="line"> name = <span class="string">"centos0001nic0001"</span></span><br><span class="line"> subnet_id = <span class="string">"<span class="variable">${azurerm_subnet.publicsubnet.id}</span>"</span></span><br><span class="line"> private_ip_address_allocation = <span class="string">"Static"</span></span><br><span class="line"> private_ip_address = <span class="string">"10.11.0.4"</span></span><br><span class="line"> public_ip_address_id = <span class="string">"<span class="variable">${azurerm_public_ip.centospips.id}</span>"</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> tags = {</span><br><span class="line"> environment = <span class="string">"Azure Terraform Automation"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create storage account for boot diagnostics</span></span><br><span class="line">resource <span class="string">"azurerm_storage_account"</span> <span class="string">"mystorageaccount"</span> {</span><br><span class="line"> name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>sa0001"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"> location = <span class="string">"<span class="variable">${var.location}</span>"</span></span><br><span class="line"> account_kind = <span class="string">"StorageV2"</span></span><br><span class="line"> account_tier = <span class="string">"Standard"</span></span><br><span class="line"> account_replication_type = <span class="string">"LRS"</span></span><br><span class="line"></span><br><span class="line"> tags = {</span><br><span class="line"> environment = <span class="string">"Azure Terraform Automation"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create virtual machine</span></span><br><span class="line">resource <span class="string">"azurerm_virtual_machine"</span> <span class="string">"centos0001"</span> {</span><br><span class="line"> name = <span class="string">"centos0001"</span></span><br><span class="line"> location = <span class="string">"<span class="variable">${var.location}</span>"</span></span><br><span class="line"> resource_group_name = <span class="string">"<span class="variable">${azurerm_resource_group.myterraformgroup.name}</span>"</span></span><br><span class="line"> network_interface_ids = [<span class="string">"<span class="variable">${azurerm_network_interface.centosnics.id}</span>"</span>]</span><br><span class="line"> vm_size = <span class="string">"Standard_D2s_v3"</span></span><br><span class="line"></span><br><span class="line"> storage_os_disk {</span><br><span class="line"> name = <span class="string">"centos0001OsDisk"</span></span><br><span class="line"> caching = <span class="string">"ReadWrite"</span></span><br><span class="line"> create_option = <span class="string">"FromImage"</span></span><br><span class="line"> managed_disk_type = <span class="string">"Standard_LRS"</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> storage_image_reference {</span><br><span class="line"> <span class="built_in">id</span> = <span class="string">"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Compute/images/centos77image0001"</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> os_profile {</span><br><span class="line"> computer_name = <span class="string">"centos0001"</span></span><br><span class="line"> admin_username = <span class="string">"<span class="variable">${var.admin_username}</span>"</span></span><br><span class="line"> </span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> os_profile_linux_config {</span><br><span class="line"> disable_password_authentication = <span class="literal">true</span></span><br><span class="line"> ssh_keys {</span><br><span class="line"> path = <span class="string">"/home/<span class="variable">${var.admin_username}</span>/.ssh/authorized_keys"</span></span><br><span class="line"> key_data = <span class="string">"<span class="variable">${var.ssh_public_key_data}</span>"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> boot_diagnostics {</span><br><span class="line"> enabled = <span class="string">"true"</span></span><br><span class="line"> storage_uri = <span class="string">"<span class="variable">${azurerm_storage_account.mystorageaccount.primary_blob_endpoint}</span>"</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> tags = {</span><br><span class="line"> environment = <span class="string">"Azure Terraform Automation"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>整个部署过程大概需要10分钟左右完成。</p>
<h3 id="3-2-部署单节点-MongoDB-Server-4-2-0"><a href="#3-2-部署单节点-MongoDB-Server-4-2-0" class="headerlink" title="3.2 部署单节点 MongoDB Server 4.2.0"></a>3.2 部署单节点 MongoDB Server 4.2.0</h3><h4 id="3-2-1-配置-CentOS-7-7-MongoDB-4-x-Yum-源"><a href="#3-2-1-配置-CentOS-7-7-MongoDB-4-x-Yum-源" class="headerlink" title="3.2.1 配置 CentOS 7.7 MongoDB 4.x Yum 源"></a>3.2.1 配置 CentOS 7.7 MongoDB 4.x Yum 源</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> /etc/yum.repos.d/</span><br><span class="line">$ vi mongodb.repo </span><br><span class="line">[mongodb-org-4.2]</span><br><span class="line">name=MongoDB Repository</span><br><span class="line">baseurl=https://repo.mongodb.org/yum/redhat/<span class="variable">$releasever</span>/mongodb-org/4.2/x86_64/</span><br><span class="line">gpgcheck=1</span><br><span class="line">enabled=1</span><br><span class="line">gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc</span><br></pre></td></tr></table></figure>
<p>保存退出，Yum 源配置完成。</p>
<h4 id="3-2-2-安装单节点-MongoDB-Server-4-2-0"><a href="#3-2-2-安装单节点-MongoDB-Server-4-2-0" class="headerlink" title="3.2.2 安装单节点 MongoDB Server 4.2.0"></a>3.2.2 安装单节点 MongoDB Server 4.2.0</h4><p>安装 MongoDB Server 4.2.0：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y mongodb-org-4.2.0 mongodb-org-server-4.2.0 mongodb-org-shell-4.2.0 mongodb-org-mongos-4.2.0 mongodb-org-tools-4.2.0</span><br></pre></td></tr></table></figure>
<p>检查安装包及版本：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ rpm -qa |grep mongodb</span><br><span class="line">mongodb-org-tools-4.2.0-1.el7.x86_64</span><br><span class="line">mongodb-org-mongos-4.2.0-1.el7.x86_64</span><br><span class="line">mongodb-org-shell-4.2.0-1.el7.x86_64</span><br><span class="line">mongodb-org-4.2.0-1.el7.x86_64</span><br><span class="line">mongodb-org-server-4.2.0-1.el7.x86_64</span><br></pre></td></tr></table></figure>
<p>修改 MongoDB Server 配置文件：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">$ vi /etc/mongod.conf</span><br><span class="line"><span class="comment"># mongod.conf</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># for documentation of all options, see:</span></span><br><span class="line"><span class="comment"># http://docs.mongodb.org/manual/reference/configuration-options/</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># where to write logging data.</span></span><br><span class="line">systemLog:</span><br><span class="line"> destination: file</span><br><span class="line"> logAppend: <span class="literal">true</span></span><br><span class="line"> path: /var/log/mongodb/mongod.log</span><br><span class="line"></span><br><span class="line"><span class="comment"># Where and how to store data.</span></span><br><span class="line">storage:</span><br><span class="line"> dbPath: /var/lib/mongo</span><br><span class="line"> journal:</span><br><span class="line"> enabled: <span class="literal">true</span></span><br><span class="line"><span class="comment"># engine:</span></span><br><span class="line"><span class="comment"># wiredTiger:</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># how the process runs</span></span><br><span class="line">processManagement:</span><br><span class="line"> fork: <span class="literal">true</span> <span class="comment"># fork and run in background</span></span><br><span class="line"> pidFilePath: /var/run/mongodb/mongod.pid <span class="comment"># location of pidfile</span></span><br><span class="line"> timeZoneInfo: /usr/share/zoneinfo</span><br><span class="line"></span><br><span class="line"><span class="comment"># network interfaces</span></span><br><span class="line">net:</span><br><span class="line"> port: 27017</span><br><span class="line"> bindIp: 0.0.0.0 <span class="comment"># Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#security:</span></span><br><span class="line"><span class="comment">#operationProfiling:</span></span><br><span class="line">replication:</span><br><span class="line"> oplogSizeMB: <span class="string">"20480"</span></span><br><span class="line"> replSetName: repconfig </span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>启动 MongoDB Sever 服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">systemctl start mongod && systemctl <span class="built_in">enable</span> mongod</span><br></pre></td></tr></table></figure>
<p>MongoDB 初始化：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">$ mongo --port 27017</span><br><span class="line">> repconfig = { _id : <span class="string">"repconfig"</span>, members : [ {_id : 0, host : <span class="string">"10.11.0.4:27017"</span> , priority: 1 } ] }</span><br><span class="line">{</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"repconfig"</span>,</span><br><span class="line"> <span class="string">"members"</span> : [</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"_id"</span> : 0,</span><br><span class="line"> <span class="string">"host"</span> : <span class="string">"10.11.0.4:27017"</span>,</span><br><span class="line"> <span class="string">"priority"</span> : 1</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line">}</span><br><span class="line">> rs.initiate(repconfig);</span><br><span class="line">{</span><br><span class="line"> <span class="string">"ok"</span> : 1,</span><br><span class="line"> <span class="string">"<span class="variable">$clusterTime</span>"</span> : {</span><br><span class="line"> <span class="string">"clusterTime"</span> : Timestamp(1579339466, 1),</span><br><span class="line"> <span class="string">"signature"</span> : {</span><br><span class="line"> <span class="string">"hash"</span> : BinData(0,<span class="string">"AAAAAAAAAAAAAAAAAAAAAAAAAAA="</span>),</span><br><span class="line"> <span class="string">"keyId"</span> : NumberLong(0)</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"operationTime"</span> : Timestamp(1579339466, 1)</span><br><span class="line">}</span><br><span class="line">repconfig:SECONDARY> </span><br><span class="line">repconfig:PRIMARY></span><br></pre></td></tr></table></figure>
<p>初始化结束后，可以通过 Mongo Shell rs.status() 来查 Replica 信息。</p>
<h3 id="3-3-通过-Python-调用-Azure-Monitor-Restful-API-来监控-Mongodb-Server-4-2-0"><a href="#3-3-通过-Python-调用-Azure-Monitor-Restful-API-来监控-Mongodb-Server-4-2-0" class="headerlink" title="3.3 通过 Python 调用 Azure Monitor Restful API 来监控 Mongodb Server 4.2.0"></a>3.3 通过 Python 调用 Azure Monitor Restful API 来监控 Mongodb Server 4.2.0</h3><h4 id="3-3-1-创建-Azure-Log-Analysis-Workspace"><a href="#3-3-1-创建-Azure-Log-Analysis-Workspace" class="headerlink" title="3.3.1 创建 Azure Log Analysis Workspace"></a>3.3.1 创建 Azure Log Analysis Workspace</h4><p>具体操作过程不再赘述，通过 Portal 或者 Cli 等方式 Step by Step 点击创建即可，如图：</p>
<p><a href="LA-Creation.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-01-18-AzureMonitorRestfulAPI/LA-Creation.jpg" height="800" width="600"/></a></p>
<p>创建好了需要的几个信息，如图：</p>
<p><a href="LA-Info.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-01-18-AzureMonitorRestfulAPI/LA-Info.jpg" height="800" width="800"/></a></p>
<p>“WORKSPACE ID” 为脚本中的 customer_id，”PRIMARY KEY” 为脚本中的 shared_key。</p>
<h4 id="3-3-2-Python-调用-Azure-Monitor-Restful-API-向-Azure-Log-Analysis-Workspace-传送数据"><a href="#3-3-2-Python-调用-Azure-Monitor-Restful-API-向-Azure-Log-Analysis-Workspace-传送数据" class="headerlink" title="3.3.2 Python 调用 Azure Monitor Restful API 向 Azure Log Analysis Workspace 传送数据"></a>3.3.2 Python 调用 Azure Monitor Restful API 向 Azure Log Analysis Workspace 传送数据</h4><p>Azure Monitor Restful API 不做赘述，具体可以参考<a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api">这里</a>。本实验先通过 pymongo 收集 MongoDB Metrics，然后送到 Azure Log Analysis Workspace 中，先安装 pymongo：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum install python-pip y</span><br><span class="line">$ pip install pymongo requests </span><br></pre></td></tr></table></figure>
<p>具体的 Python 脚本如下：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br><span class="line">239</span><br><span class="line">240</span><br><span class="line">241</span><br><span class="line">242</span><br><span class="line">243</span><br><span class="line">244</span><br><span class="line">245</span><br><span class="line">246</span><br><span class="line">247</span><br><span class="line">248</span><br><span class="line">249</span><br><span class="line">250</span><br><span class="line">251</span><br><span class="line">252</span><br><span class="line">253</span><br><span class="line">254</span><br><span class="line">255</span><br><span class="line">256</span><br><span class="line">257</span><br><span class="line">258</span><br><span class="line">259</span><br><span class="line">260</span><br><span class="line">261</span><br><span class="line">262</span><br><span class="line">263</span><br><span class="line">264</span><br><span class="line">265</span><br><span class="line">266</span><br><span class="line">267</span><br><span class="line">268</span><br><span class="line">269</span><br><span class="line">270</span><br><span class="line">271</span><br><span class="line">272</span><br><span class="line">273</span><br><span class="line">274</span><br><span class="line">275</span><br><span class="line">276</span><br><span class="line">277</span><br><span class="line">278</span><br><span class="line">279</span><br><span class="line">280</span><br><span class="line">281</span><br><span class="line">282</span><br><span class="line">283</span><br><span class="line">284</span><br><span class="line">285</span><br><span class="line">286</span><br><span class="line">287</span><br><span class="line">288</span><br><span class="line">289</span><br><span class="line">290</span><br><span class="line">291</span><br><span class="line">292</span><br><span class="line">293</span><br><span class="line">294</span><br><span class="line">295</span><br><span class="line">296</span><br><span class="line">297</span><br><span class="line">298</span><br><span class="line">299</span><br><span class="line">300</span><br><span class="line">301</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">Date: 01/18/2020</span></span><br><span class="line"><span class="string">Author: Xinsheng Wang</span></span><br><span class="line"><span class="string">Description: A custom script to get MongoDB metrics and send data to Azure Monitor </span></span><br><span class="line"><span class="string">Requires: MongoClient in python</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> calendar <span class="keyword">import</span> timegm</span><br><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> gmtime</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> pymongo <span class="keyword">import</span> MongoClient, errors</span><br><span class="line"><span class="keyword">from</span> sys <span class="keyword">import</span> exit</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> json</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">import</span> datetime</span><br><span class="line"><span class="keyword">import</span> hashlib</span><br><span class="line"><span class="keyword">import</span> hmac</span><br><span class="line"><span class="keyword">import</span> base64</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">from</span> glob <span class="keyword">import</span> glob</span><br><span class="line"></span><br><span class="line"><span class="comment"># Update the customer ID to your Log Analytics workspace ID</span></span><br><span class="line">customer_id = <span class="string">'86df0cbc-076c-4483-8a32-c59c6550a771'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># For the shared key, use either the primary or the secondary Connected Sources client authentication key </span></span><br><span class="line">shared_key = <span class="string">"b3uLsEOXBFBqTiAHDGp9boTeKR6v86f/9cLPWWsWUvs+LcjBIqjDp9CDJL+7vxlKDDRxqXIf1jjjKcZbdV0H/Q=="</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># The log type is the name of the event that is being submitted</span></span><br><span class="line">log_type = <span class="string">'MongoDBMonitorLog'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">MongoDB</span>(<span class="title class_ inherited__">object</span>):</span><br><span class="line"> <span class="string">"""main script class"""</span></span><br><span class="line"> <span class="comment"># pylint: disable=too-many-instance-attributes</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">delete_temporary_files</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""delete temporary files"""</span></span><br><span class="line"> <span class="keyword">for</span> file <span class="keyword">in</span> glob(<span class="string">'/tmp/mongometrics000*'</span>):</span><br><span class="line"> os.remove(file)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self</span>):</span><br><span class="line"> self.mongo_host = <span class="string">"10.11.0.4"</span></span><br><span class="line"> self.mongo_port = <span class="number">27017</span></span><br><span class="line"> self.mongo_db = [<span class="string">"admin"</span>, ]</span><br><span class="line"> self.mongo_user = <span class="literal">None</span></span><br><span class="line"> self.mongo_password = <span class="literal">None</span></span><br><span class="line"> self.__conn = <span class="literal">None</span></span><br><span class="line"> self.__dbnames = <span class="literal">None</span></span><br><span class="line"> self.__metrics = []</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">connect</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""Connect to MongoDB"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">if</span> self.mongo_user <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> self.__conn = MongoClient(<span class="string">'mongodb://%s:%s'</span> %</span><br><span class="line"> (self.mongo_host,</span><br><span class="line"> self.mongo_port))</span><br><span class="line"> <span class="keyword">except</span> errors.PyMongoError <span class="keyword">as</span> py_mongo_error:</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Error in MongoDB connection: %s'</span> %</span><br><span class="line"> <span class="built_in">str</span>(py_mongo_error))</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> self.__conn = MongoClient(<span class="string">'mongodb://%s:%s@%s:%s'</span> %</span><br><span class="line"> (self.mongo_user,</span><br><span class="line"> self.mongo_password,</span><br><span class="line"> self.mongo_host,</span><br><span class="line"> self.mongo_port))</span><br><span class="line"> <span class="keyword">except</span> errors.PyMongoError <span class="keyword">as</span> py_mongo_error:</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Error in MongoDB connection: %s'</span> %</span><br><span class="line"> <span class="built_in">str</span>(py_mongo_error))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">add_metrics</span>(<span class="params">self, k, v</span>):</span><br><span class="line"> <span class="string">"""add each metric to the metrics list"""</span></span><br><span class="line"> <span class="keyword">global</span> body</span><br><span class="line"> dict_metrics = {}</span><br><span class="line"> dict_metrics[<span class="string">"key"</span>] = k</span><br><span class="line"> dict_metrics[<span class="string">"value"</span>] = v</span><br><span class="line"> self.__metrics.append(dict_metrics)</span><br><span class="line"> dic = json.dumps(dict_metrics, sort_keys=<span class="literal">True</span>, indent=<span class="number">4</span>, separators=(<span class="string">','</span>, <span class="string">':'</span>)).replace(<span class="string">'}'</span>, <span class="string">'},'</span>)</span><br><span class="line"> </span><br><span class="line"> f = <span class="built_in">open</span>(<span class="string">'/tmp/mongometrics0001.txt'</span>,<span class="string">'a'</span>)</span><br><span class="line"> f.write(dic)</span><br><span class="line"> f.close</span><br><span class="line"> </span><br><span class="line"> os.system(<span class="string">"cat /tmp/mongometrics0001.txt |sed '$s/\}\,/\}\]/g;1s/{/[{/' > /tmp/mongometrics0002.txt"</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">with</span> <span class="built_in">open</span>(<span class="string">'/tmp/mongometrics0002.txt'</span>,<span class="string">'r'</span>) <span class="keyword">as</span> src:</span><br><span class="line"> body = src.read()</span><br><span class="line"> <span class="built_in">print</span>(body)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_db_names</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get a list of DB names"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> db_handler = self.__conn[self.mongo_db[<span class="number">0</span>]]</span><br><span class="line"></span><br><span class="line"> master = db_handler.command(<span class="string">'isMaster'</span>)[<span class="string">'ismaster'</span>]</span><br><span class="line"> dict_metrics = {}</span><br><span class="line"> dict_metrics[<span class="string">'key'</span>] = <span class="string">'mongodb.ismaster'</span></span><br><span class="line"> <span class="keyword">if</span> master:</span><br><span class="line"> dict_metrics[<span class="string">'value'</span>] = <span class="number">1</span></span><br><span class="line"> db_names = self.__conn.database_names()</span><br><span class="line"> self.__dbnames = db_names</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> dict_metrics[<span class="string">'value'</span>] = <span class="number">0</span></span><br><span class="line"> self.__metrics.append(dict_metrics)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_mongo_db_lld</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""print DB list in json format, to be used for mongo db discovery"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__dbnames <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> db_names = self.get_db_names()</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> db_names = self.__dbnames</span><br><span class="line"> dict_metrics = {}</span><br><span class="line"> db_list = []</span><br><span class="line"> dict_metrics[<span class="string">'key'</span>] = <span class="string">'mongodb.discovery'</span></span><br><span class="line"> dict_metrics[<span class="string">'value'</span>] = {<span class="string">"data"</span>: db_list}</span><br><span class="line"> <span class="keyword">if</span> db_names <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">for</span> db_name <span class="keyword">in</span> db_names:</span><br><span class="line"> dict_lld_metric = {}</span><br><span class="line"> dict_lld_metric[<span class="string">'{#MONGODBNAME}'</span>] = db_name</span><br><span class="line"> db_list.append(dict_lld_metric)</span><br><span class="line"> dict_metrics[<span class="string">'value'</span>] = <span class="string">'{"data": '</span> + json.dumps(db_list) + <span class="string">'}'</span></span><br><span class="line"> self.__metrics.insert(<span class="number">0</span>, dict_metrics)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_oplog</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get replica set oplog information"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> db_handler = self.__conn[<span class="string">'local'</span>]</span><br><span class="line"></span><br><span class="line"> coll = db_handler.oplog.rs</span><br><span class="line"></span><br><span class="line"> op_first = (coll.find().sort(<span class="string">'$natural'</span>, <span class="number">1</span>).limit(<span class="number">1</span>))</span><br><span class="line"> op_last = (coll.find().sort(<span class="string">'$natural'</span>, -<span class="number">1</span>).limit(<span class="number">1</span>))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># if host is not a member of replica set, without this check we will</span></span><br><span class="line"> <span class="comment"># raise StopIteration as guided in</span></span><br><span class="line"> <span class="comment"># http://api.mongodb.com/python/current/api/pymongo/cursor.html</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> op_first.count() > <span class="number">0</span> <span class="keyword">and</span> op_last.count() > <span class="number">0</span>:</span><br><span class="line"> op_fst = (op_first.<span class="built_in">next</span>())[<span class="string">'ts'</span>].time</span><br><span class="line"> op_last_st = op_last[<span class="number">0</span>][<span class="string">'ts'</span>]</span><br><span class="line"> op_lst = (op_last.<span class="built_in">next</span>())[<span class="string">'ts'</span>].time</span><br><span class="line"></span><br><span class="line"> status = <span class="built_in">round</span>(<span class="built_in">float</span>(op_lst - op_fst), <span class="number">1</span>)</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.oplog'</span>, status)</span><br><span class="line"></span><br><span class="line"> current_time = timegm(gmtime())</span><br><span class="line"> oplog = <span class="built_in">int</span>(((<span class="built_in">str</span>(op_last_st).split(<span class="string">'('</span>))[<span class="number">1</span>].split(<span class="string">','</span>))[<span class="number">0</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.oplog-sync'</span>, (current_time - oplog))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_maintenance</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get replica set maintenance info"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> db_handler = self.__conn</span><br><span class="line"></span><br><span class="line"> fsync_locked = <span class="built_in">int</span>(db_handler.is_locked)</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.fsync-locked'</span>, fsync_locked)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> config = db_handler.admin.command(<span class="string">"replSetGetConfig"</span>, <span class="number">1</span>)</span><br><span class="line"> connstring = (self.mongo_host + <span class="string">':'</span> + <span class="built_in">str</span>(self.mongo_port))</span><br><span class="line"> connstrings = <span class="built_in">list</span>()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">0</span>, <span class="built_in">len</span>(config[<span class="string">'config'</span>][<span class="string">'members'</span>])):</span><br><span class="line"> host = config[<span class="string">'config'</span>][<span class="string">'members'</span>][i][<span class="string">'host'</span>]</span><br><span class="line"> connstrings.append(host)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> connstring <span class="keyword">in</span> host:</span><br><span class="line"> priority = config[<span class="string">'config'</span>][<span class="string">'members'</span>][i][<span class="string">'priority'</span>]</span><br><span class="line"> hidden = <span class="built_in">int</span>(config[<span class="string">'config'</span>][<span class="string">'members'</span>][i][<span class="string">'hidden'</span>])</span><br><span class="line"></span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.priority'</span>, priority)</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.hidden'</span>, hidden)</span><br><span class="line"> <span class="keyword">except</span> errors.PyMongoError:</span><br><span class="line"> <span class="built_in">print</span> (<span class="string">'Error while fetching replica set configuration.'</span></span><br><span class="line"> <span class="string">'Not a member of replica set?'</span>)</span><br><span class="line"> <span class="keyword">except</span> UnboundLocalError:</span><br><span class="line"> <span class="built_in">print</span> (<span class="string">'Cannot use this mongo host: must be one of '</span> + <span class="string">','</span>.join(connstrings))</span><br><span class="line"> exit(<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_server_status_metrics</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get server status"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> db_handler = self.__conn[self.mongo_db[<span class="number">0</span>]]</span><br><span class="line"> ss = db_handler.command(<span class="string">'serverStatus'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># db info</span></span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.version'</span>, ss[<span class="string">'version'</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.storageEngine'</span>, ss[<span class="string">'storageEngine'</span>][<span class="string">'name'</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.uptime'</span>, <span class="built_in">int</span>(ss[<span class="string">'uptime'</span>]))</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.okstatus'</span>, <span class="built_in">int</span>(ss[<span class="string">'ok'</span>]))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># asserts</span></span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'asserts'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.asserts.'</span> + k, v)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># operations</span></span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'opcounters'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.operation.'</span> + k, v)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># connections</span></span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'connections'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.connection.'</span> + k, v)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># extra info</span></span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.page.faults'</span>,</span><br><span class="line"> ss[<span class="string">'extra_info'</span>][<span class="string">'page_faults'</span>])</span><br><span class="line"></span><br><span class="line"> <span class="comment">#wired tiger</span></span><br><span class="line"> <span class="keyword">if</span> ss[<span class="string">'storageEngine'</span>][<span class="string">'name'</span>] == <span class="string">'wiredTiger'</span>:</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.used-cache'</span>,</span><br><span class="line"> ss[<span class="string">'wiredTiger'</span>][<span class="string">'cache'</span>]</span><br><span class="line"> [<span class="string">"bytes currently in the cache"</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.total-cache'</span>,</span><br><span class="line"> ss[<span class="string">'wiredTiger'</span>][<span class="string">'cache'</span>]</span><br><span class="line"> [<span class="string">"maximum bytes configured"</span>])</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.dirty-cache'</span>,</span><br><span class="line"> ss[<span class="string">'wiredTiger'</span>][<span class="string">'cache'</span>]</span><br><span class="line"> [<span class="string">"tracked dirty bytes in the cache"</span>])</span><br><span class="line"></span><br><span class="line"> <span class="comment"># global lock</span></span><br><span class="line"> lock_total_time = ss[<span class="string">'globalLock'</span>][<span class="string">'totalTime'</span>]</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.globalLock.totalTime'</span>, lock_total_time)</span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'globalLock'</span>][<span class="string">'currentQueue'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.globalLock.currentQueue.'</span> + k, v)</span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> ss[<span class="string">'globalLock'</span>][<span class="string">'activeClients'</span>].items():</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.globalLock.activeClients.'</span> + k, v)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">get_db_stats_metrics</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""get DB stats for each DB"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.connect()</span><br><span class="line"> <span class="keyword">if</span> self.__dbnames <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> self.get_db_names()</span><br><span class="line"> <span class="keyword">if</span> self.__dbnames <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">for</span> mongo_db <span class="keyword">in</span> self.__dbnames:</span><br><span class="line"> db_handler = self.__conn[mongo_db]</span><br><span class="line"> dbs = db_handler.command(<span class="string">'dbstats'</span>)</span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> dbs.items():</span><br><span class="line"> <span class="keyword">if</span> k <span class="keyword">in</span> [<span class="string">'storageSize'</span>, <span class="string">'ok'</span>, <span class="string">'avgObjSize'</span>, <span class="string">'indexes'</span>,</span><br><span class="line"> <span class="string">'objects'</span>, <span class="string">'collections'</span>, <span class="string">'fileSize'</span>,</span><br><span class="line"> <span class="string">'numExtents'</span>, <span class="string">'dataSize'</span>, <span class="string">'indexSize'</span>,</span><br><span class="line"> <span class="string">'nsSizeMB'</span>]:</span><br><span class="line"> self.add_metrics(<span class="string">'mongodb.stats.'</span> + k +</span><br><span class="line"> <span class="string">'['</span> + mongo_db + <span class="string">']'</span>, <span class="built_in">int</span>(v))</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">close</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="string">"""close connection to mongo"""</span></span><br><span class="line"> <span class="keyword">if</span> self.__conn <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> self.__conn.close()</span><br><span class="line"></span><br><span class="line"><span class="comment"># Build the API signature</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">build_signature</span>(<span class="params">customer_id, shared_key, date, content_length, method, content_type, resource</span>):</span><br><span class="line"> x_headers = <span class="string">'x-ms-date:'</span> + date</span><br><span class="line"> string_to_hash = method + <span class="string">"\n"</span> + <span class="built_in">str</span>(content_length) + <span class="string">"\n"</span> + content_type + <span class="string">"\n"</span> + x_headers + <span class="string">"\n"</span> + resource</span><br><span class="line"> bytes_to_hash = <span class="built_in">bytes</span>(string_to_hash).encode(<span class="string">'utf-8'</span>) </span><br><span class="line"> decoded_key = base64.b64decode(shared_key)</span><br><span class="line"> encoded_hash = base64.b64encode(hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest())</span><br><span class="line"> authorization = <span class="string">"SharedKey {}:{}"</span>.<span class="built_in">format</span>(customer_id,encoded_hash)</span><br><span class="line"> <span class="keyword">return</span> authorization</span><br><span class="line"></span><br><span class="line"><span class="comment"># Build and send a request to the POST API</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">mongodb_azuremonitor_loganalysis_post_data</span>(<span class="params">customer_id, shared_key, body, log_type</span>):</span><br><span class="line"> method = <span class="string">'POST'</span></span><br><span class="line"> content_type = <span class="string">'application/json'</span></span><br><span class="line"> resource = <span class="string">'/api/logs'</span></span><br><span class="line"> rfc1123date = datetime.datetime.utcnow().strftime(<span class="string">'%a, %d %b %Y %H:%M:%S GMT'</span>)</span><br><span class="line"> content_length = <span class="built_in">len</span>(body)</span><br><span class="line"> signature = build_signature(customer_id, shared_key, rfc1123date, content_length, method, content_type, resource)</span><br><span class="line"> uri = <span class="string">'https://'</span> + customer_id + <span class="string">'.ods.opinsights.azure.com'</span> + resource + <span class="string">'?api-version=2016-04-01'</span></span><br><span class="line"></span><br><span class="line"> headers = {</span><br><span class="line"> <span class="string">'content-type'</span>: content_type,</span><br><span class="line"> <span class="string">'Authorization'</span>: signature,</span><br><span class="line"> <span class="string">'Log-Type'</span>: log_type,</span><br><span class="line"> <span class="string">'x-ms-date'</span>: rfc1123date</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> response = requests.post(uri,data=body, headers=headers)</span><br><span class="line"> <span class="keyword">if</span> (response.status_code >= <span class="number">200</span> <span class="keyword">and</span> response.status_code <= <span class="number">299</span>):</span><br><span class="line"> <span class="built_in">print</span> <span class="string">'Accepted'</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="built_in">print</span> <span class="string">"Response code: {}"</span>.<span class="built_in">format</span>(response.status_code)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> mongodb = MongoDB()</span><br><span class="line"> mongodb.delete_temporary_files()</span><br><span class="line"> mongodb.get_db_names()</span><br><span class="line"> mongodb.get_mongo_db_lld()</span><br><span class="line"> mongodb.get_oplog()</span><br><span class="line"> mongodb.get_maintenance()</span><br><span class="line"> mongodb.get_server_status_metrics()</span><br><span class="line"> mongodb.get_db_stats_metrics()</span><br><span class="line"> mongodb.close()</span><br><span class="line"> mongodb_azuremonitor_loganalysis_post_data(customer_id, shared_key, body, log_type)</span><br></pre></td></tr></table></figure>
<p>注意相应的 customer_id 和 shared_key 替换成图中的 key，log_type 为收集到 Log Analysis Workspace 的 namespace 名字。保存脚本名字为 AzureMonitor_LogAnalysis_MongodbMetrics.py，执行脚本：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">python AzureMonitor_LogAnalysis_MongodbMetrics.py</span><br></pre></td></tr></table></figure>
<p>通过 print 可以打印一些交互式信息，观察数据是否已经送到 Azure Log Analysis Workspace。</p>
<h4 id="3-3-3-利用-Kusto-查询-MongoDB-Custom-Logs"><a href="#3-3-3-利用-Kusto-查询-MongoDB-Custom-Logs" class="headerlink" title="3.3.3 利用 Kusto 查询 MongoDB Custom Logs"></a>3.3.3 利用 Kusto 查询 MongoDB Custom Logs</h4><p>在 Azure Portal 查看并通过 KQL 查询 MongoDB 的数据：</p>
<p><a href="Azure-Monitor-Restful-API-MongoDB.jpg"><img src="https://cdn.jsdelivr.net/gh/TheoDoreW/CDN_Images@master/images/2020-01-18-AzureMonitorRestfulAPI/Azure-Monitor-Restful-API-MongoDB.jpg" height="800" width="800"/></a></p>
<p>如图可见，数据已经成功送到了 Azure Log Analysis Workspace，后续可以写更复杂的 Kusto 来做相关的查询，并可以考虑结合调度或者 Linux Crontab 等服务做定时的数据拉取和告警了。</p>
<br>
<h2 id="4-总结"><a href="#4-总结" class="headerlink" title="4. 总结"></a>4. 总结</h2><p>测试至此，Azure Monitor Restful API 监控 MongoDB 的示例完成了，希望能够给想要通过 Azure Monitor 做二次开发的同学们一些参考。但本文还有一些可以优化之处，比如 Python 脚本没有考虑更多的 Log Output 以及执行失败时候的 Retry 逻辑，在实际生产中，为了严谨，还是希望大家要充分考虑逻辑在上线，不过这些功能有待大家自己添加了。</p>
</div>
<footer class="article-footer">
<a data-url="https://theodorew.github.io/wxsblog.github.io/2020/01/18/2020-01-18-AzureMonitorRestfulAPI/" data-id="clat72drb0006yofk9l52h96p" class="article-share-link" data-share="baidu" data-title="集成 Microsoft Azure Monitor Restful API 来监控 MongoDB">Share</a>
<a class="article-reward-link">Reward</a>
<ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Database/" rel="tag">Database</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/MongoDB/" rel="tag">MongoDB</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/wxsblog.github.io/tags/Monitor/" rel="tag">Monitor</a></li></ul>
</footer>
</div>
</article>
<nav id="page-nav">
<span class="page-number current">1</span><a class="page-number" href="/wxsblog.github.io/page/2/">2</a><a class="extend next" rel="next" href="/wxsblog.github.io/page/2/">Next &raquo;</a>
</nav>
</section>
<aside id="sidebar">
<div class="widget-wrap">
<h3 class="widget-title">Categories</h3>
<div class="widget">
<ul class="category-list"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/">All Blogs</a><span class="category-list-count">14</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Linux/">Linux</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Linux/Windows-10/">Windows 10</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Linux/Windows-10/WSL/">WSL</a><span class="category-list-count">1</span></li></ul></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/">Microsoft Azure</a><span class="category-list-count">13</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/">Bigdata</a><span class="category-list-count">4</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Databricks/">Databricks</a><span class="category-list-count">3</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Databricks/Linux/">Linux</a><span class="category-list-count">3</span></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Monitor/">Monitor</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Monitor/Databricks/">Databricks</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Bigdata/Monitor/Databricks/Linux/">Linux</a><span class="category-list-count">1</span></li></ul></li></ul></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/CDN/">CDN</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/CDN/Github/">Github</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/CDN/Github/Linux/">Linux</a><span class="category-list-count">1</span></li></ul></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/DataBase/">DataBase</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/DataBase/MySQL/">MySQL</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/DataBase/MySQL/Linux/">Linux</a><span class="category-list-count">1</span></li></ul></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Database/">Database</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Database/MySQL/">MySQL</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Database/MySQL/Linux/">Linux</a><span class="category-list-count">1</span></li></ul></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Kubernetes/">Kubernetes</a><span class="category-list-count">2</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Kubernetes/Bigdata/">Bigdata</a><span class="category-list-count">2</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Kubernetes/Bigdata/Flink/">Flink</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Kubernetes/Bigdata/Flink/Linux/">Linux</a><span class="category-list-count">1</span></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Kubernetes/Bigdata/Spark/">Spark</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Kubernetes/Bigdata/Spark/Linux/">Linux</a><span class="category-list-count">1</span></li></ul></li></ul></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/">Monitor</a><span class="category-list-count">2</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Database/">Database</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Database/MongoDB/">MongoDB</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Database/MongoDB/Linux/">Linux</a><span class="category-list-count">1</span></li></ul></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Fortigate/">Fortigate</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Fortigate/SNMP/">SNMP</a><span class="category-list-count">1</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/Monitor/Fortigate/SNMP/Linux/">Linux</a><span class="category-list-count">1</span></li></ul></li></ul></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/">infrastructure</a><span class="category-list-count">2</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/Automation/">Automation</a><span class="category-list-count">2</span><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/wxsblog.github.io/categories/All-Blogs/Microsoft-Azure/infrastructure/Automation/Linux/">Linux</a><span class="category-list-count">2</span></li></ul></li></ul></li></ul></li></ul></li></ul>
</div>
</div>
<div class="widget-wrap">
<h3 class="widget-title">Tags</h3>
<div class="widget">
<ul class="tag-list" itemprop="keywords"><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Automation/" rel="tag">Automation</a><span class="tag-list-count">2</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Bigdata/" rel="tag">Bigdata</a><span class="tag-list-count">6</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/CDN/" rel="tag">CDN</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/DataBase/" rel="tag">DataBase</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Database/" rel="tag">Database</a><span class="tag-list-count">2</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Databricks/" rel="tag">Databricks</a><span class="tag-list-count">4</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Flink/" rel="tag">Flink</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Fortigate/" rel="tag">Fortigate</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Github/" rel="tag">Github</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Kubernetes/" rel="tag">Kubernetes</a><span class="tag-list-count">2</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Linux/" rel="tag">Linux</a><span class="tag-list-count">14</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Microsoft-Azure/" rel="tag">Microsoft Azure</a><span class="tag-list-count">13</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/MongoDB/" rel="tag">MongoDB</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Monitor/" rel="tag">Monitor</a><span class="tag-list-count">3</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/MySQL/" rel="tag">MySQL</a><span class="tag-list-count">2</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/SNMP/" rel="tag">SNMP</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Spark/" rel="tag">Spark</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/WSL/" rel="tag">WSL</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/Windows-10/" rel="tag">Windows 10</a><span class="tag-list-count">1</span></li><li class="tag-list-item"><a class="tag-list-link" href="/wxsblog.github.io/tags/infrastructure/" rel="tag">infrastructure</a><span class="tag-list-count">2</span></li></ul>
</div>
</div>
<div class="widget-wrap">
<h3 class="widget-title">Tag Cloud</h3>
<div class="widget tagcloud">
<a href="/wxsblog.github.io/tags/Automation/" style="font-size: 11.67px;">Automation</a> <a href="/wxsblog.github.io/tags/Bigdata/" style="font-size: 16.67px;">Bigdata</a> <a href="/wxsblog.github.io/tags/CDN/" style="font-size: 10px;">CDN</a> <a href="/wxsblog.github.io/tags/DataBase/" style="font-size: 10px;">DataBase</a> <a href="/wxsblog.github.io/tags/Database/" style="font-size: 11.67px;">Database</a> <a href="/wxsblog.github.io/tags/Databricks/" style="font-size: 15px;">Databricks</a> <a href="/wxsblog.github.io/tags/Flink/" style="font-size: 10px;">Flink</a> <a href="/wxsblog.github.io/tags/Fortigate/" style="font-size: 10px;">Fortigate</a> <a href="/wxsblog.github.io/tags/Github/" style="font-size: 10px;">Github</a> <a href="/wxsblog.github.io/tags/Kubernetes/" style="font-size: 11.67px;">Kubernetes</a> <a href="/wxsblog.github.io/tags/Linux/" style="font-size: 20px;">Linux</a> <a href="/wxsblog.github.io/tags/Microsoft-Azure/" style="font-size: 18.33px;">Microsoft Azure</a> <a href="/wxsblog.github.io/tags/MongoDB/" style="font-size: 10px;">MongoDB</a> <a href="/wxsblog.github.io/tags/Monitor/" style="font-size: 13.33px;">Monitor</a> <a href="/wxsblog.github.io/tags/MySQL/" style="font-size: 11.67px;">MySQL</a> <a href="/wxsblog.github.io/tags/SNMP/" style="font-size: 10px;">SNMP</a> <a href="/wxsblog.github.io/tags/Spark/" style="font-size: 10px;">Spark</a> <a href="/wxsblog.github.io/tags/WSL/" style="font-size: 10px;">WSL</a> <a href="/wxsblog.github.io/tags/Windows-10/" style="font-size: 10px;">Windows 10</a> <a href="/wxsblog.github.io/tags/infrastructure/" style="font-size: 11.67px;">infrastructure</a>
</div>
</div>
<div class="widget-wrap">
<h3 class="widget-title">Archives</h3>
<div class="widget">
<ul class="archive-list"><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2022/11/">November 2022</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2021/07/">July 2021</a><span class="archive-list-count">2</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2021/03/">March 2021</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2020/12/">December 2020</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2020/11/">November 2020</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2020/09/">September 2020</a><span class="archive-list-count">2</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2020/01/">January 2020</a><span class="archive-list-count">2</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2019/12/">December 2019</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2019/11/">November 2019</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2019/10/">October 2019</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/wxsblog.github.io/archives/2019/01/">January 2019</a><span class="archive-list-count">1</span></li></ul>
</div>
</div>
<div class="widget-wrap">
<h3 class="widget-title">Recent Posts</h3>
<div class="widget">
<ul>
<li>
<a href="/wxsblog.github.io/2022/11/11/2022-11-11-AzureMySQLFlexibleServerNetworking/">Azure MySQL Flexible Server 内网 DNS 集成方案</a>
</li>
<li>
<a href="/wxsblog.github.io/2021/07/26/2021-07-26-AzureImageBuilderCLSIGI/">Azure Image Builder（二）之自动化构建自定义托管镜像 CentOS 7.7 并集成 Azure Shared Image Gallery 做全球分发</a>
</li>
<li>
<a href="/wxsblog.github.io/2021/07/25/2021-07-25-AzureImageBuilderCLMI/">Azure Image Builder（一）之自动化构建自定义托管镜像 CentOS 7.7</a>
</li>
<li>
<a href="/wxsblog.github.io/2021/03/17/2021-03-17-AzureDatabricksMonitor/">Azure Databricks 系列 Blog（四）之通过 Azure Monitor 做集群监控</a>
</li>