/
search.xml
1088 lines (523 loc) · 693 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>我的服务器系列：总体设计</title>
<link href="//my-server-overall-design/"/>
<url>//my-server-overall-design/</url>
<content type="html"><![CDATA[<p>不定期更新的服务器攻略指南。<br>记录服务器的配置管理、软件安装等信息，和附带文章索引。</p><span id="more"></span><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>2022.4更新：把笔记本扔了，买了nas服务器（并升级到16g内存）<br>本文记录对服务器（2云+1NAS）的规划、管理过程，并分享一些经验心得。</p><h2 id="一-资源总览"><a href="#一-资源总览" class="headerlink" title="一 资源总览"></a>一 资源总览</h2><h3 id="1-域名"><a href="#1-域名" class="headerlink" title="1 域名"></a>1 域名</h3><p>域名即入口，现有已备案域名一个： linshenkx.cn </p><h3 id="2-云服务器"><a href="#2-云服务器" class="headerlink" title="2 云服务器"></a>2 云服务器</h3><p>云服务器都是买了3年的，有公网ip</p><table><thead><tr><th>名称</th><th>资源</th><th>说明</th></tr></thead><tbody><tr><td>uc</td><td>2c8g1m,40g</td><td>主服务器</td></tr><tr><td>tx</td><td>2c4g3m,50g+400g</td><td>存储服务器</td></tr></tbody></table><h3 id="3-服务器"><a href="#3-服务器" class="headerlink" title="3 服务器"></a>3 服务器</h3><p>闲置笔记本因为性能孱弱，且年久失修，不经折腾，便领了盒饭光荣退休了。</p><p>新上任的是威联通的ts262c，默认2c2g我给它升级到16g，能装docker，功耗又低，就当是迷你服务器了。</p><h2 id="二-组网方案（VPN）"><a href="#二-组网方案（VPN）" class="headerlink" title="二 组网方案（VPN）"></a>二 组网方案（VPN）</h2><p>以前是用蒲公英的跨网组网，现在全部升级到 TailScale。方便又好用。<br>开源版本可以使用 HeadScale,参考：<a href="https://fuckcloudnative.io/posts/how-to-set-up-or-migrate-headscale">https://fuckcloudnative.io/posts/how-to-set-up-or-migrate-headscale</a> 。<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/20220327172113.png"></p><h2 id="三-域名管理（dns）"><a href="#三-域名管理（dns）" class="headerlink" title="三 域名管理（dns）"></a>三 域名管理（dns）</h2><h3 id="域名解析配置"><a href="#域名解析配置" class="headerlink" title="域名解析配置"></a>域名解析配置</h3><ol><li>主域名<br>指向个人网站</li><li>tailscale次级域名<br>如 tx.linshenkx.cn、uc.linshenkx.cn等配置为tailscale分配的IP，只能通过vpn访问。</li><li>公网服务次级域名<br>如 derper、wiznote等都指向云服务的公网ip</li><li>其他通配符域名<br>指向个人网站</li></ol><h3 id="Nginx"><a href="#Nginx" class="headerlink" title="Nginx"></a>Nginx</h3><p>由swag负责证书申请、使用以及续签等，并内置nginx实现代理、跳转等。</p><p>Nginx服务器接收到请求后，统一将http 301跳到https访问。<br>而在https内，则根据二级域名的具体名称转发到对应的服务。</p><p>整个过程对于服务应用来说是透明的，应用只需确保监听端口即可，无需考虑域名和ssl配置。</p><h2 id="二-角色分配"><a href="#二-角色分配" class="headerlink" title="二 角色分配"></a>二 角色分配</h2><table><thead><tr><th>机器</th><th>角色（软件）</th><th>说明（博客链接）</th></tr></thead><tbody><tr><td>nas</td><td>nas服务器</td><td>家用服务器，有较高的网络带宽，负责文件存储、家庭影音等</td></tr><tr><td></td><td>clash(clash-web)</td><td>网络畅通是一切行动的前提：<a href="https://www.linshenkx.cn/archives/clashdocker">我的服务器系列：clash-docker使用并实现订阅链接自动更新</a></td></tr><tr><td></td><td>clash-tracing</td><td>clash汇总监控：<a href="https://www.linshenkx.cn/archives/clash-tracing-windows">我的服务器系列：使用clash-tracing统计windows下clash使用情况</a></td></tr><tr><td></td><td>swag</td><td>letsencrypt+nginx,再也不要看到ip和端口，万物皆在https+域名之下：文章待整理</td></tr><tr><td></td><td>ddns</td><td>动态域名解析</td></tr><tr><td></td><td>docker-registry</td><td>docker私有仓库（原本使用nexus，太重了，没必要）</td></tr><tr><td></td><td>git-lfs-server</td><td>git大文件存储（原本使用nexus，太重了，没必要）</td></tr><tr><td></td><td>jellyfin</td><td>家庭影音：照片墙+播放器</td></tr><tr><td></td><td>auto-bangumi</td><td>家庭影音：自动追番</td></tr><tr><td></td><td>chinesesubfinder</td><td>字幕自动查找下载</td></tr><tr><td></td><td>flaresolverr</td><td>绕开网站Cloudflare保护</td></tr><tr><td></td><td>iyuu</td><td>自动辅种工具</td></tr><tr><td></td><td>qbit</td><td>下载工具</td></tr><tr><td></td><td>nas-tools</td><td>nas工具</td></tr><tr><td></td><td>syncthing</td><td>文件同步工具</td></tr><tr><td></td><td>watchtower</td><td>docker镜像自动更新</td></tr><tr><td>tx</td><td>主云服务器</td><td>网络带宽和磁盘空间较大，但cpu和内存不高，存放供公网访问的服务，如：个人网站、数据库、http文件服务器等，</td></tr><tr><td></td><td>swag</td><td>常规必备，参考如上</td></tr><tr><td></td><td>http-file-server</td><td><a href="https://github.com/patrickdappollonio/http-server">简单好用的http文件服务器</a></td></tr><tr><td></td><td>derper</td><td>tailscale自定义derper服务器</td></tr><tr><td></td><td>halo</td><td>个人博客: <a href="https://www.linshenkx.cn/archives/myblogsystemdesign">个人博客系统设计(支持hexo和halo同步) </a></td></tr><tr><td></td><td>mysql</td><td>自动备份的数据库：<a href="https://github.com/linshenkx/mysql-cron">预装cron服务的mysql镜像</a></td></tr><tr><td>uc</td><td>备用云服务器</td><td>存放个人笔记等</td></tr><tr><td></td><td>swag</td><td>常规必备，参考如上</td></tr><tr><td></td><td>halo</td><td>备用</td></tr><tr><td></td><td>mysql</td><td>备用</td></tr><tr><td></td><td>haloSyncServer</td><td><a href="https://www.linshenkx.cn/s/open-source-project">Halo博客同步器 </a></td></tr><tr><td></td><td>wiznote</td><td>从印象笔记到可以私有部署的为知笔记：文章待整理 （也可以看看joplin，优点是支持webdav备份，支持裁剪，缺点是丑，导入印象笔记目录结构会丢失）</td></tr><tr><td></td><td>random-image-api</td><td>颜值即正义：<a href="https://github.com/linshenkx/random-image-api">对接了阿里oss的随机图api</a></td></tr></tbody></table><h2 id="三-管理工程"><a href="#三-管理工程" class="headerlink" title="三 管理工程"></a>三 管理工程</h2><p>为了使每个服务器使用的软件及其部署过程有迹可循，我开了一个私有仓库来跟踪记录。</p><p>因为我几乎所有软件都是以docker容器形式安装运行的，<br>所以安装、管理过程都可以简化为docker命令记录在readme.md里面。<br>而个别需要额外配置文件的也放在同级目录下。<br>这样以后服务器迁移的时候十分方便，<br>只需要根据docker目录把挂载目录进行迁移再运行docker命令即可。<br>另外docker命令也可以使用docker-compose配置文件进行取代，本质一样。</p><p>其结构如下：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">myserver:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">服务器1:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">server.md</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">软件1</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">readme.md</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">软件1的配置文件</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">软件2</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">readme.md</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">软件2的配置文件</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">软件3</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">readme.md</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">服务器2:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">软件4</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">readme.md</span><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 我的服务器系列 </category>
</categories>
<tags>
<tag> 生产力 </tag>
<tag> 我的服务器系列 </tag>
</tags>
</entry>
<entry>
<title>基于k3s-helm-controller实现k8s api离线部署helm chart的方案</title>
<link href="//k3s-helm-controller-k8s-api-offline-helm-chart-deploy/"/>
<url>//k3s-helm-controller-k8s-api-offline-helm-chart-deploy/</url>
<content type="html"><![CDATA[<p>本文介绍了在内网环境下，使用k8s api(java fabric8io/kubernetes-client)实现helm组件部署的方法。使用k3s-helm-controller规避helm命令行操作。并记录helm仓库的简易</p><span id="more"></span><h2 id="需求"><a href="#需求" class="headerlink" title="需求"></a>需求</h2><p>我想在代码里通过helm部署 <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack">kube-prometheus-stack</a>，经过考察，k3s-helm-controller可以比较好地满足我的需求。它可以将HelmChart作为k8s的crd，所以可以使用k8s的api来对helm chart进行管理。</p><p>在这个基础上，我还想实现内网部署，除了私有docker镜像仓库，因为使用到了helm，还需要helm仓库。但由于我只需要部署kube-prometheus-stack一个charts，不想增加一个组件，所以直接在代码里提供文件api来充当helm仓库（也可以是任意符合格式要求的http文件服务器），本文记录这个过程的注意事项。</p><h2 id="部署k3s-helm-controller"><a href="#部署k3s-helm-controller" class="headerlink" title="部署k3s-helm-controller"></a>部署k3s-helm-controller</h2><h3 id="创建kube-config的configmap"><a href="#创建kube-config的configmap" class="headerlink" title="创建kube-config的configmap"></a>创建kube-config的configmap</h3><p>部署时需要读取/root/.kube/config文件，下面创建kube-config的脚本不一定适用于你的环境</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 获取 Kubernetes 主节点的 IP 地址</span><br>K8S_MASTER_IP=$(kubectl get nodes -o wide | awk 'NR==2 {print $6}')<br><span class="hljs-meta">#</span><span class="bash"> 读取 config 文件内容到变量中</span><br>CONFIG_CONTENT=$(cat /root/.kube/config)<br><span class="hljs-meta">#</span><span class="bash"> 使用 sed 命令将其中的 127.0.0.1 替换为 Kubernetes 主节点的 IP 地址</span><br>NEW_CONFIG_CONTENT=$(echo "$CONFIG_CONTENT" | sed "s/127.0.0.1/$K8S_MASTER_IP/g")<br><span class="hljs-meta">#</span><span class="bash"> 使用 kubectl create configmap 命令将其发布为 ConfigMap</span><br>echo -e "$NEW_CONFIG_CONTENT" | kubectl -n=kube-system create configmap kube-config --from-file=config=/dev/stdin<br></code></pre></td></tr></table></figure><h3 id="部署helm-controller"><a href="#部署helm-controller" class="headerlink" title="部署helm-controller"></a>部署helm-controller</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat <<EOF >./deploy-cluster-scoped.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br> name: helm-controller<br> labels:<br> app: helm-controller<br>spec:<br> replicas: 1<br> selector:<br> matchLabels:<br> app: helm-controller<br> template:<br> metadata:<br> labels:<br> app: helm-controller<br> spec:<br> containers:<br> - name: helm-controller<br> image: rancher/helm-controller:v0.15.4<br> command: ["helm-controller"]<br> args: ["--kubeconfig", "/root/.kube/config"]<br> volumeMounts: <br> - name: kube-config <br> mountPath: /root/.kube/config<br> subPath: config<br> volumes:<br> - name: kube-config<br> configMap:<br> name: kube-config<br><br>EOF<br><br>kubectl -n=kube-system create -f deploy-cluster-scoped.yaml<br><br></code></pre></td></tr></table></figure><h3 id="测试部署helmchart"><a href="#测试部署helmchart" class="headerlink" title="测试部署helmchart"></a>测试部署helmchart</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 部署</span><br>cat <<EOF | kubectl create -f -<br>apiVersion: helm.cattle.io/v1<br>kind: HelmChart<br>metadata:<br> name: traefik <br> namespace: kube-system<br>spec:<br> chart: stable/traefik<br> set:<br> rbac.enabled: "true"<br> ssl.enabled: "true"<br>EOF<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 删除</span><br>kubectl -n=kube-system delete helmchart traefik<br><br></code></pre></td></tr></table></figure><h2 id="使用helm-controller部署Prometheus"><a href="#使用helm-controller部署Prometheus" class="headerlink" title="使用helm-controller部署Prometheus"></a>使用helm-controller部署Prometheus</h2><p>如下，使用国内helm镜像仓库在monitoring命名空间安装kube-prometheus-stack的chart<br>需要修改的参数可以都放到set里面</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat <<EOF >./prometheus-helm-chart.yaml<br>apiVersion: helm.cattle.io/v1<br>kind: HelmChart<br>metadata:<br> name: prometheus-stack <br> namespace: monitoring<br>spec:<br><span class="hljs-meta"> #</span><span class="bash">repo: https://charts.grapps.cn</span><br> repo: https://helm-charts.itboon.top/prometheus-community<br> chart: kube-prometheus-stack<br> targetNamespace: monitoring<br> set:<br> namespaceOverride: "monitoring"<br> alertmanager.service.type: "NodePort"<br> grafana.defaultDashboardsTimezone: "Asia/Shanghai"<br> grafana.adminPassword: "1qaz@WSX"<br> grafana.sidecar.dashboards.folderAnnotation: "folder"<br> grafana.sidecar.dashboards.provider.allowUiUpdates: "true"<br> grafana.service.nodePort: "30902"<br> grafana.service.type: "NodePort"<br> prometheus.service.nodePort: "30900"<br> prometheus.service.type: "NodePort"<br> prometheus.prometheusSpec.additionalScrapeConfigsSecret.enabled: "true"<br> prometheus.prometheusSpec.additionalScrapeConfigsSecret.name: "additional-scrape-configs"<br> prometheus.prometheusSpec.additionalScrapeConfigsSecret.key: "prometheus-additional.yaml"<br> kubelet.serviceMonitor.cAdvisorMetricRelabelings: ""<br><br>EOF<br><br>kubectl apply -f prometheus-helm-chart.yaml<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 卸载</span> <br><span class="hljs-meta">#</span><span class="bash"> kubectl -n=monitoring delete helmchart prometheus-stack</span> <br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> Kubernetes </category>
</categories>
<tags>
<tag> helm </tag>
<tag> api </tag>
</tags>
</entry>
<entry>
<title>GithubAction配置private仓库自动从public仓库同步的方法</title>
<link href="//github-private-repo-sync-from-public-repo/"/>
<url>//github-private-repo-sync-from-public-repo/</url>
<content type="html"><![CDATA[<p>本文介绍一种基于github action的解决方案用于实现private项目自动从public项目同步</p><span id="more"></span><h3 id="需求"><a href="#需求" class="headerlink" title="需求"></a>需求</h3><p>GitHub 提供了许多用于实现仓库同步的 Action，有些只支持 fork，而有些则支持两个独立的仓库。<br>对于 fork 项目，建议直接使用 <a href="https://github.com/apps/pull">GitHub Pull</a>。本文将讨论非 fork 项目的同步问题。</p><p>有些开源项目，出于备份或私有部署的目的，需要克隆一份。 然而，由于 fork 的项目不能设置为 private，这就引发了隐私问题， 因为你可能不希望所有人知道你 fork 了某个项目。 一般的做法是将 public 项目克隆下来，然后推送到自己的 private 仓库。 这一过程可以直接使用 GitHub 的 import repository 功能实现。但这样由于失去了 fork 关系，导致无法使用 “Sync fork” 进行项目同步。</p><p>为了解决这个问题，我们可以使用 <a href="https://github.com/aormsby/Fork-Sync-With-Upstream-action">aormsby/Fork-Sync-With-Upstream-action</a> 来实现独立项目的同步。 虽然有很多独立项目同步的 Action，但它们大多基于 Git 实现，这导致上游项目的所有变更都会同步到下游，包括 <code>.github/workflows</code> 下的文件。 这可能会导致 GitHub 任务在对 Action 进行操作时报错，因为这超出了默认的 GITHUB_TOKEN 的权限范围。</p><p>如果是 fork 项目，可以直接手动执行 “Sync Fork” 操作。否则，就需要创建更高权限的 token。</p><p>独立项目的同步action还有很多，不过都有一个问题，因为他们的实现都是基于git的，所以上游项目的所有变更都会同步到下游，这其中就包括了 .github/workflows 下的文件。会导致github认为该action在对action进行操作，这超出了默认的GITHUB_TOKEN的权限范围</p><p>缺少权限时的报错信息：(refusing to allow a GitHub App to create or update workflow <code>.github/workflows/xxx.yml</code> without <code>workflows</code> permission)<br>这时应使用有workflow权限的自定义token</p><h3 id="设计"><a href="#设计" class="headerlink" title="设计"></a>设计</h3><p>仓库同步可以基于 <a href="https://github.com/aormsby/Fork-Sync-With-Upstream-action">aormsby/Fork-Sync-With-Upstream-action</a> 实现。<br>由于不想在 Action 中配置源仓库的地址，可以约定仓库名称为 <code>${repoName}-syncfrom-${userName}</code>，将远程库的信息存在项目名中，实现 Action 的通用化。<br>当然，这取决于个人需求。</p><h3 id="ACTION-TOKEN-配置说明"><a href="#ACTION-TOKEN-配置说明" class="headerlink" title="ACTION_TOKEN 配置说明"></a>ACTION_TOKEN 配置说明</h3><ol><li>在个人设置中创建具有完整 repo、workflow 权限的 Personal Access Token。</li><li>在仓库的设置中，添加名为 <code>ACTION_TOKEN</code> 的 secret，值为上一步获取的 Personal Access Token。</li></ol><p>此外，需要在项目的设置中配置 Action：</p><ol><li>将 Actions 权限设置为 Allow all actions and reusable workflows。</li><li>将 Workflow 权限设置为 Read and write permissions。</li></ol><h3 id="action示例"><a href="#action示例" class="headerlink" title="action示例"></a>action示例</h3><p>Action 文件的名称应独特，不与上游仓库的 Action 文件名重复，例如 <code>.github/workflows/super-sync-from.yml</code>。<br>如果上游仓库对 <code>.github/workflows/</code> 下的文件进行变更，则需要在 <code>actions/checkout</code> 步骤中使用具有 workflow 权限的 token。</p><h4 id="无需修改版本（适用于项目名-repoName-syncfrom-userName-）"><a href="#无需修改版本（适用于项目名-repoName-syncfrom-userName-）" class="headerlink" title="无需修改版本（适用于项目名${repoName}-syncfrom-${userName}）"></a>无需修改版本（适用于项目名${repoName}-syncfrom-${userName}）</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">name:</span> <span class="hljs-string">Public</span> <span class="hljs-string">Upstream</span> <span class="hljs-string">Sync</span><br><br><span class="hljs-attr">on:</span><br> <span class="hljs-attr">schedule:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">cron:</span> <span class="hljs-string">"0 0 * * *"</span><br> <span class="hljs-attr">workflow_dispatch:</span><br><span class="hljs-attr">jobs:</span><br> <span class="hljs-attr">sync_latest_from_upstream:</span><br> <span class="hljs-attr">name:</span> <span class="hljs-string">Sync</span> <span class="hljs-string">latest</span> <span class="hljs-string">commits</span> <span class="hljs-string">from</span> <span class="hljs-string">upstream</span> <span class="hljs-string">repo</span><br> <span class="hljs-attr">runs-on:</span> <span class="hljs-string">ubuntu-latest</span><br><br> <span class="hljs-attr">steps:</span><br> <span class="hljs-comment"># 标准签出</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Checkout</span> <span class="hljs-string">target</span> <span class="hljs-string">repo</span><br> <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/checkout@v4</span><br><span class="hljs-comment"># 如果上游仓库有对.github/workflows/下的文件进行变更，则需要使用有workflow权限的token</span><br><span class="hljs-comment"># with:</span><br><span class="hljs-comment"># token: ${{ secrets.ACTION_TOKEN }}</span><br> <br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Extract</span> <span class="hljs-string">Repo</span> <span class="hljs-string">and</span> <span class="hljs-string">User</span><br> <span class="hljs-attr">id:</span> <span class="hljs-string">extract</span><br> <span class="hljs-attr">run:</span> <span class="hljs-string">|</span><br><span class="hljs-string"> repository_name=${{ github.event.repository.name }}</span><br><span class="hljs-string"> if echo "$repository_name" | grep -qE '(.+)-syncfrom-(.+)'; then</span><br><span class="hljs-string"> repoName=$(echo "$repository_name" | sed -E 's/(.+)-syncfrom-(.+)/\1/')</span><br><span class="hljs-string"> userName=$(echo "$repository_name" | sed -E 's/(.+)-syncfrom-(.+)/\2/')</span><br><span class="hljs-string"> echo "RepoName: $repoName"</span><br><span class="hljs-string"> echo "UserName: $userName"</span><br><span class="hljs-string"> echo "repoName=$repoName" >> $GITHUB_ENV</span><br><span class="hljs-string"> echo "userName=$userName" >> $GITHUB_ENV</span><br><span class="hljs-string"> else</span><br><span class="hljs-string"> echo '无法从仓库名中提取 repoName 和 userName，格式应为 ${repoName}-syncfrom-${userName}，当前为：'$GITHUB_REPOSITORY</span><br><span class="hljs-string"> exit 1</span><br><span class="hljs-string"> fi</span><br><span class="hljs-string"></span><br> <span class="hljs-comment"># 获取分支名</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Get</span> <span class="hljs-string">branch</span> <span class="hljs-string">name</span> <span class="hljs-string">(merge)</span><br> <span class="hljs-attr">if:</span> <span class="hljs-string">github.event_name</span> <span class="hljs-type">!=</span> <span class="hljs-string">'pull_request'</span><br> <span class="hljs-attr">shell:</span> <span class="hljs-string">bash</span><br> <span class="hljs-attr">run:</span> <span class="hljs-string">echo</span> <span class="hljs-string">"BRANCH_NAME=$(echo ${GITHUB_REF#refs/heads/} | tr / -)"</span> <span class="hljs-string">>></span> <span class="hljs-string">$GITHUB_ENV</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Get</span> <span class="hljs-string">branch</span> <span class="hljs-string">name</span> <span class="hljs-string">(pull</span> <span class="hljs-string">request)</span><br> <span class="hljs-attr">if:</span> <span class="hljs-string">github.event_name</span> <span class="hljs-string">==</span> <span class="hljs-string">'pull_request'</span><br> <span class="hljs-attr">shell:</span> <span class="hljs-string">bash</span><br> <span class="hljs-attr">run:</span> <span class="hljs-string">echo</span> <span class="hljs-string">"BRANCH_NAME=$(echo ${GITHUB_HEAD_REF} | tr / -)"</span> <span class="hljs-string">>></span> <span class="hljs-string">$GITHUB_ENV</span><br><br> <span class="hljs-comment"># 运行同步动作</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Sync</span> <span class="hljs-string">upstream</span> <span class="hljs-string">changes</span><br> <span class="hljs-attr">id:</span> <span class="hljs-string">sync</span><br> <span class="hljs-attr">uses:</span> <span class="hljs-string">aormsby/Fork-Sync-With-Upstream-action@v3.4</span><br> <span class="hljs-attr">with:</span><br> <span class="hljs-attr">upstream_sync_repo:</span> <span class="hljs-string">${{</span> <span class="hljs-string">env.userName</span> <span class="hljs-string">}}/${{</span> <span class="hljs-string">env.repoName</span> <span class="hljs-string">}}</span><br> <span class="hljs-attr">upstream_sync_branch:</span> <span class="hljs-string">${{</span> <span class="hljs-string">env.BRANCH_NAME</span> <span class="hljs-string">}}</span><br> <span class="hljs-attr">upstream_pull_args:</span> <span class="hljs-string">--allow-unrelated-histories</span> <span class="hljs-string">--no-edit</span><br> <span class="hljs-attr">target_sync_branch:</span> <span class="hljs-string">${{</span> <span class="hljs-string">env.BRANCH_NAME</span> <span class="hljs-string">}}</span><br> <span class="hljs-attr">target_repo_token:</span> <span class="hljs-string">${{</span> <span class="hljs-string">secrets.GITHUB_TOKEN</span> <span class="hljs-string">}}</span><br> <span class="hljs-attr">test_mode:</span> <span class="hljs-literal">false</span><br><br></code></pre></td></tr></table></figure><h4 id="定制化版本"><a href="#定制化版本" class="headerlink" title="定制化版本"></a>定制化版本</h4><p>对于已有的不想改名的，直接配置上流仓库名即可</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">name:</span> <span class="hljs-string">Public</span> <span class="hljs-string">Upstream</span> <span class="hljs-string">Sync</span><br><br><span class="hljs-attr">on:</span><br> <span class="hljs-attr">schedule:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">cron:</span> <span class="hljs-string">"0 0 * * *"</span><br> <span class="hljs-attr">workflow_dispatch:</span><br><span class="hljs-attr">jobs:</span><br> <span class="hljs-attr">sync_latest_from_upstream:</span><br> <span class="hljs-attr">name:</span> <span class="hljs-string">Sync</span> <span class="hljs-string">latest</span> <span class="hljs-string">commits</span> <span class="hljs-string">from</span> <span class="hljs-string">upstream</span> <span class="hljs-string">repo</span><br> <span class="hljs-attr">runs-on:</span> <span class="hljs-string">ubuntu-latest</span><br><br> <span class="hljs-attr">steps:</span><br> <span class="hljs-comment"># 标准签出</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Checkout</span> <span class="hljs-string">target</span> <span class="hljs-string">repo</span><br> <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/checkout@v4</span><br><span class="hljs-comment"># 如果上游仓库有对.github/workflows/下的文件进行变更，则需要使用有workflow权限的token</span><br><span class="hljs-comment"># with:</span><br><span class="hljs-comment"># token: ${{ secrets.ACTION_TOKEN }}</span><br> <br> <span class="hljs-comment"># 获取分支名</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Get</span> <span class="hljs-string">branch</span> <span class="hljs-string">name</span> <span class="hljs-string">(merge)</span><br> <span class="hljs-attr">if:</span> <span class="hljs-string">github.event_name</span> <span class="hljs-type">!=</span> <span class="hljs-string">'pull_request'</span><br> <span class="hljs-attr">shell:</span> <span class="hljs-string">bash</span><br> <span class="hljs-attr">run:</span> <span class="hljs-string">echo</span> <span class="hljs-string">"BRANCH_NAME=$(echo ${GITHUB_REF#refs/heads/} | tr / -)"</span> <span class="hljs-string">>></span> <span class="hljs-string">$GITHUB_ENV</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Get</span> <span class="hljs-string">branch</span> <span class="hljs-string">name</span> <span class="hljs-string">(pull</span> <span class="hljs-string">request)</span><br> <span class="hljs-attr">if:</span> <span class="hljs-string">github.event_name</span> <span class="hljs-string">==</span> <span class="hljs-string">'pull_request'</span><br> <span class="hljs-attr">shell:</span> <span class="hljs-string">bash</span><br> <span class="hljs-attr">run:</span> <span class="hljs-string">echo</span> <span class="hljs-string">"BRANCH_NAME=$(echo ${GITHUB_HEAD_REF} | tr / -)"</span> <span class="hljs-string">>></span> <span class="hljs-string">$GITHUB_ENV</span><br><br> <span class="hljs-comment"># 运行同步动作</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Sync</span> <span class="hljs-string">upstream</span> <span class="hljs-string">changes</span><br> <span class="hljs-attr">id:</span> <span class="hljs-string">sync</span><br> <span class="hljs-attr">uses:</span> <span class="hljs-string">aormsby/Fork-Sync-With-Upstream-action@v3.4</span><br> <span class="hljs-attr">with:</span><br> <span class="hljs-attr">upstream_sync_repo:</span> [<span class="hljs-string">替换成目标仓库名称</span>]<br> <span class="hljs-attr">upstream_sync_branch:</span> <span class="hljs-string">${{</span> <span class="hljs-string">env.BRANCH_NAME</span> <span class="hljs-string">}}</span><br> <span class="hljs-attr">target_sync_branch:</span> <span class="hljs-string">${{</span> <span class="hljs-string">env.BRANCH_NAME</span> <span class="hljs-string">}}</span><br> <span class="hljs-attr">target_repo_token:</span> <span class="hljs-string">${{</span> <span class="hljs-string">secrets.ACTION_TOKEN</span> <span class="hljs-string">}}</span><br> <span class="hljs-attr">upstream_pull_args:</span> <span class="hljs-string">--allow-unrelated-histories</span> <span class="hljs-string">--no-edit</span> <br> <span class="hljs-attr">test_mode:</span> <span class="hljs-literal">false</span><br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 程序员杂记 </category>
</categories>
<tags>
<tag> github </tag>
</tags>
</entry>
<entry>
<title>kafka-jaas环境下常用命令示例（含压测）</title>
<link href="//kafka-jaas-benchmark-command-examples/"/>
<url>//kafka-jaas-benchmark-command-examples/</url>
<content type="html"><![CDATA[<p>本文记录本人使用kafka过程中常用的运维指令。</p><span id="more"></span><h2 id="一-准备客户端环境"><a href="#一-准备客户端环境" class="headerlink" title="一 准备客户端环境"></a>一 准备客户端环境</h2><p>安装好jdk和kafka</p><h3 id="1-配置producer-consumer-properties"><a href="#1-配置producer-consumer-properties" class="headerlink" title="1. 配置producer/consumer.properties"></a>1. 配置producer/consumer.properties</h3><p>修改config/producer.properties和config/consumer.properties，添加以下内容</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs text">security.protocol=SASL_PLAINTEXT<br>sasl.mechanism=PLAIN<br>sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='1qaz@WSX';"<br><br><br></code></pre></td></tr></table></figure><h3 id="2-配置静态jaas文件（可选）"><a href="#2-配置静态jaas文件（可选）" class="headerlink" title="2. 配置静态jaas文件（可选）"></a>2. 配置静态jaas文件（可选）</h3><p>如果使用的kafka版本太旧，或者properties文件的sasl.jaas.config无法生效的时候，可配置静态jaas文件<br>添加config/kafka_jaas.conf文件，然后配置环境变量使之启用</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">export KAFKA_OPTS="-Djava.security.auth.login.config=$KAFKA_HOME/config/kafka_jaas.conf"<br><br></code></pre></td></tr></table></figure><p>账号密码认证方式，文件格式如下</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs text">KafkaClient {<br> org.apache.kafka.common.security.plain.PlainLoginModule required<br> username="admin"<br> password="123456";<br>};<br><br></code></pre></td></tr></table></figure><p>如果是keytab的话，格式应该是这样</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs text">KafkaClient {<br> com.sun.security.auth.module.Krb5LoginModule required<br> useKeyTab=true<br> storeKey=true<br> serviceName="kafka"<br> keyTab="/mnt/XXX.keytab"<br> principal="XXX/XXX@XXX.COM";<br>};<br></code></pre></td></tr></table></figure><h2 id="二-常用命令"><a href="#二-常用命令" class="headerlink" title="二 常用命令"></a>二 常用命令</h2><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 查看主题列表：注意使用zk连接，也可自行替换成--bootstrap-server</span><br>bin/kafka-topics.sh --zookeeper $HOSTNAME:$ZK_CLIENT_PORT/kafka --list <br><span class="hljs-meta">#</span><span class="bash"> 创建主题：注意使用zk连接，也可自行替换成--bootstrap-server</span><br>bin/kafka-topics.sh --zookeeper $HOSTNAME:$ZK_CLIENT_PORT/kafka --create --topic $TOPIC_NAME --partitions 1 --replication-factor 1<br><span class="hljs-meta">#</span><span class="bash"> 生产消息</span><br>bin/kafka-console-producer.sh --bootstrap-server $KAFKA_HOSTNAME:$KAFKA_PORT --topic $TOPIC_NAME --producer.config config/producer.properties<br><span class="hljs-meta">#</span><span class="bash"> 消费消息：从头开始接收数据并打印时间戳</span><br>bin/kafka-console-consumer.sh --bootstrap-server $KAFKA_HOSTNAME:$KAFKA_PORT --from-beginning --topic $TOPIC_NAME --consumer.config config/consumer.properties --property print.timestamp=true<br><span class="hljs-meta">#</span><span class="bash"> 接收1条最新的数据</span><br>bin/kafka-console-consumer.sh --bootstrap-server $HOSTNAME:$KAFKA_PORT --max-messages 1 --topic $TOPIC_NAME --consumer.config config/consumer.properties<br><span class="hljs-meta">#</span><span class="bash"> 查看消费组信息列表</span><br>bin/kafka-consumer-groups.sh --bootstrap-server $KAFKA_HOSTNAME:$KAFKA_PORT --list --command-config config/consumer.properties<br><span class="hljs-meta">#</span><span class="bash"> 查看指定消费组的消费/堆积情况</span><br>bin/kafka-consumer-groups.sh --bootstrap-server $KAFKA_HOSTNAME:$KAFKA_PORT --describe --group $groupname --command-config config/consumer.properties<br><br></code></pre></td></tr></table></figure><h2 id="三-压测命令"><a href="#三-压测命令" class="headerlink" title="三 压测命令"></a>三 压测命令</h2><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 压测 预设吞吐量为5w/s,总共发送100w条数据，每条数据10字节(自动生成)</span><br>bin/kafka-producer-perf-test.sh --topic $TOPIC_NAME \<br>--num-record 1000000 \<br>--throughput 50000 \<br>--record-size=10 \<br>--producer-props bootstrap.servers=$KAFKA_HOSTNAME:$KAFKA_PORT \<br>--producer.config config/producer.properties<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 压测 预设吞吐量为1w/s,总共发送150w条数据，数据来自data.txt（按行切割）</span><br>bin/kafka-producer-perf-test.sh --topic $TOPIC_NAME \<br>--num-record 1500000 \<br>--throughput 10000 \<br>--payload-file=data.txt \<br>--producer-props bootstrap.servers=$KAFKA_HOSTNAME:$KAFKA_PORT \<br>--producer.config config/producer.properties<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 压测数据可使用如下方法获取</span><br><span class="hljs-meta">#</span><span class="bash"> 从指定主题中获取1000条数据写入到 data.txt文件中</span><br>bin/kafka-console-consumer.sh --bootstrap-server $KAFKA_HOSTNAME:$KAFKA_PORT \<br>--topic $TOPIC_NAME \<br>--max-messages=1000 \<br>--consumer.config config/consumer.properties >> data.txt<br><br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> kafka </tag>
</tags>
</entry>
<entry>
<title>我的服务器系列：使用clash-tracing统计windows下clash使用情况</title>
<link href="//clash-tracing-windows/"/>
<url>//clash-tracing-windows/</url>
<content type="html"><![CDATA[<p>本文记录clash-tracing的部署使用（踩坑经验），以及windows下clash的trace数据采集方法。</p><span id="more"></span><h2 id="一-前言说明"><a href="#一-前言说明" class="headerlink" title="一 前言说明"></a>一 前言说明</h2><p>clash-tracing（<a href="https://github.com/Dreamacro/clash-tracing">https://github.com/Dreamacro/clash-tracing</a> ）是clash作者推出的clash数据采集展示工具。<br>核心是一个docker-compose配置，包含组件有：<br>loki、grafana、vector、traffic_scraper（websocat）、tracing_scraper（websocat）<br>其工作流程如下：</p><ol><li>两个websocat将clash数据转发到vector</li><li>vector将数据进行路由、转换等操作，然后写入loki</li><li>loki存储数据并提供搜索功能</li><li>grafana提供面板访问，数据源来自loki</li></ol><p>由上面的工作流程可以看出，如果要对接多个clash，只需要提供多个websocat即可。</p><p>目前本人有3个地方安装了clash：pc、手机、服务器。 其中服务器是多个服务器共同使用一个clash实例。</p><p>因为服务器是长时间运行的，可直接用部署的websocat进行数据采集。<br>而pc则不是一直使用的，不想为此多部署两个websocat容器，所以使用任务计划在系统启动时自动启动websocat程序即可。</p><h2 id="二-部署clash-tracing"><a href="#二-部署clash-tracing" class="headerlink" title="二 部署clash-tracing"></a>二 部署clash-tracing</h2><p>主要看官方的：<a href="https://github.com/Dreamacro/clash-tracing">https://github.com/Dreamacro/clash-tracing</a><br>以下是个人的修改配置，可参考：</p><ol><li><p>把grafana/grafana-oss:latest换成grafana/grafana-oss:latest-ubuntu<br>grafana/grafana-oss:latest使用镜像不支持no_proxy环境变量,会导致大量不需要代理的连接进入clash影响统计。<br>比方说tailscale的连接，因为我的grafana是通过tailscale访问的。<br>见：<a href="https://linshenkx.github.io/clash_docker/#6-no-proxy%E7%9B%B8%E5%85%B3%E8%AF%B4%E6%98%8E">https://linshenkx.github.io/clash_docker/#6-no-proxy%E7%9B%B8%E5%85%B3%E8%AF%B4%E6%98%8E</a></p></li><li><p>在docker-compose.yml里面把loki的端口也放出来。<br>再把clash-tracing/grafana/provisioning/datasources/clash.yaml以及clash-tracing/vector/vector.toml中的 loki:3100 换成 域名:3100，因网络代理，loki可能无法解析</p></li><li><p>在docker-compose.yml里面把<code>user: root</code>换成 <code>user: "0:0"</code></p></li><li><p>loki的版本使用2.4.1避免“too many outstanding requests”报错，见：<a href="https://github.com/grafana/loki/issues/4613">https://github.com/grafana/loki/issues/4613</a></p></li><li><p>两个websocat合并成一个</p></li></ol><p>然后去执行 <code>docker-compose up -d</code> 就行了</p><p>注意默认数据是存储在clash-tracing的各个组件目录下，工具loki的默认配置数据只存储168h，但还是要注意空间占用</p><p>docker-compose.yml参考如下：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">version:</span> <span class="hljs-string">'3'</span><br><span class="hljs-attr">services:</span><br> <span class="hljs-attr">loki:</span><br> <span class="hljs-attr">image:</span> <span class="hljs-string">grafana/loki:2.4.1</span><br> <span class="hljs-attr">container_name:</span> <span class="hljs-string">loki</span><br> <span class="hljs-attr">restart:</span> <span class="hljs-string">always</span><br> <span class="hljs-attr">user:</span> <span class="hljs-string">"0:0"</span><br> <span class="hljs-attr">volumes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">./loki/data:/loki</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">./loki/config.yaml:/etc/loki/local-config.yaml</span><br> <span class="hljs-attr">ports:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">"3100:3100"</span><br> <span class="hljs-attr">grafana:</span><br> <span class="hljs-attr">image:</span> <span class="hljs-string">grafana/grafana-oss:latest-ubuntu</span><br> <span class="hljs-attr">container_name:</span> <span class="hljs-string">grafana</span><br> <span class="hljs-attr">restart:</span> <span class="hljs-string">always</span><br> <span class="hljs-attr">user:</span> <span class="hljs-string">"0:0"</span><br> <span class="hljs-attr">volumes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">./grafana/data:/var/lib/grafana</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">./grafana/panels:/etc/dashboards</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources</span><br> <span class="hljs-attr">ports:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">"3000:3000"</span><br> <span class="hljs-attr">vector:</span><br> <span class="hljs-attr">image:</span> <span class="hljs-string">timberio/vector:0.X-alpine</span><br> <span class="hljs-attr">container_name:</span> <span class="hljs-string">vector</span><br> <span class="hljs-attr">restart:</span> <span class="hljs-string">always</span><br> <span class="hljs-attr">volumes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">./vector/vector.toml:/etc/vector/vector.toml</span><br> <span class="hljs-attr">ports:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">"39000:9000"</span><br> <span class="hljs-attr">depends_on:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">loki</span><br> <span class="hljs-attr">scraper:</span><br> <span class="hljs-attr">image:</span> <span class="hljs-string">vi0oss/websocat:0.10.0</span><br> <span class="hljs-attr">container_name:</span> <span class="hljs-string">scraper</span><br> <span class="hljs-attr">restart:</span> <span class="hljs-string">always</span><br> <span class="hljs-attr">entrypoint:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">/bin/sh</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">-c</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">|</span><br><span class="hljs-string"> /usr/local/bin/websocat -v --autoreconnect-delay-millis 15000 autoreconnect:ws://$CLASH_HOST/traffic?token=$CLASH_TOKEN autoreconnect:tcp:vector:9000 &</span><br><span class="hljs-string"> /usr/local/bin/websocat -v --autoreconnect-delay-millis 15000 autoreconnect:ws://$CLASH_HOST/profile/tracing?token=$CLASH_TOKEN autoreconnect:tcp:vector:9000</span><br><span class="hljs-string"></span> <span class="hljs-attr">depends_on:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">vector</span><br><br><br></code></pre></td></tr></table></figure><h2 id="三-windows使用websocat采集clash数据"><a href="#三-windows使用websocat采集clash数据" class="headerlink" title="三 windows使用websocat采集clash数据"></a>三 windows使用websocat采集clash数据</h2><h3 id="1-编写脚本"><a href="#1-编写脚本" class="headerlink" title="1 编写脚本"></a>1 编写脚本</h3><p>在 <a href="https://github.com/vi/websocat">https://github.com/vi/websocat</a> 下载对应的webscocat执行程序。<br>编写 run.vbs 脚本如下</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs shell">Dim clashServer,clashToken,vectorServer,command<br>clashServer="localhost:29090"<br>clashToken="myToken"<br>vectorServer="my.vector:39000"<br>command= "D:\Program1\websocat\websocat -v --autoreconnect-delay-millis 15000 autoreconnect:ws://" & clashServer & "/profile/tracing?token=" & clashToken & " autoreconnect:tcp:" & vectorServer<br>command2= "D:\Program1\websocat\websocat -v --autoreconnect-delay-millis 15000 autoreconnect:ws://" & clashServer & "/traffic?token=" & clashToken & " autoreconnect:tcp:" & vectorServer<br><br>' 输出指令进行调试<br>' WScript.Echo command<br>' 把0改成1，使得窗口不自动隐藏进行调试 <br>WScript.CreateObject("WScript.Shell").run command , 0<br>WScript.CreateObject("WScript.Shell").run command2 , 0<br><br></code></pre></td></tr></table></figure><h3 id="2-配置开机执行"><a href="#2-配置开机执行" class="headerlink" title="2 配置开机执行"></a>2 配置开机执行</h3><p>打开“任务计划程序”，创建任务，重要的配置项如下：</p><ul><li>触发器-新建-登陆时</li><li>操作-启动程序-“程序或脚本”设置为run.vbs路径</li></ul><h2 id="三-效果验证"><a href="#三-效果验证" class="headerlink" title="三 效果验证"></a>三 效果验证</h2><p><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/202304071500435.png" alt="clash统计界面"><br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/202304071503040.png" alt="clash日志"></p>]]></content>
<categories>
<category> 我的服务器系列 </category>
</categories>
<tags>
<tag> 我的服务器系列 </tag>
<tag> clash-tracing </tag>
<tag> tracing </tag>
<tag> clash </tag>
</tags>
</entry>
<entry>
<title>WSL开发系列-gui篇(wslg)</title>
<link href="//wsl-dev-wslg/"/>
<url>//wsl-dev-wslg/</url>
<content type="html"><![CDATA[<p>直到2022年，wslg的体验仍较差，难以用于日常开发，大部分情况下不建议使用</p><span id="more"></span><h2 id="导航"><a href="#导航" class="headerlink" title="导航"></a>导航</h2><ul><li><a href="https://linshenkx.github.io/wsl-dev-base/">基础篇</a></li><li><a href="https://linshenkx.github.io/wsl-dev-static-ip/">网络篇</a></li><li><a href="https://linshenkx.github.io/wsl-dev-wslg/">gui篇</a></li><li><a href="https://linshenkx.github.io/wsl2_idea2021/">idea篇</a></li></ul><h2 id="0-1-为什么我不推荐使用wslg"><a href="#0-1-为什么我不推荐使用wslg" class="headerlink" title="0-1 为什么我不推荐使用wslg"></a>0-1 为什么我不推荐使用wslg</h2><ol><li>当前wslg不成熟，bug较多，如idea频繁卡死（冻结）导致文件修改丢失、全屏偏移等</li><li>正经人谁用linux是为了gui？</li></ol><h2 id="0-2-wslg有哪些应用场景"><a href="#0-2-wslg有哪些应用场景" class="headerlink" title="0-2 wslg有哪些应用场景"></a>0-2 wslg有哪些应用场景</h2><p>以我浅薄的开发经历来说，只有一次，<br>就是使用wsl的chrome浏览器访问被kerberos保护的hadoop web界面。</p><p>虽然理论上windows的浏览器也可以做到，但是麻烦得多。<br>而在linux系统中，kerberos认证就是kinit一个指令的事情。</p><h2 id="1-中文环境及输入法"><a href="#1-中文环境及输入法" class="headerlink" title="1 中文环境及输入法"></a>1 中文环境及输入法</h2><p>这个教程比较多，而且写得都不一样，但大同小异，需要注意的是<br>输入法没有切换处理很可能是快捷键被windows系统的覆盖了，<br>需要手动调出来输入法的配置界面，修改快捷键<br>比方说fcix的是fcitx-config-gtk3命令<br>然后我的配置是这样的，就是shift键，保持和windows使用习惯统一<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1636899123(1).png"></p><p>这里的教程可以参考：<a href="https://monkeywie.cn/2021/09/26/wsl2-gui-idea-config/">https://monkeywie.cn/2021/09/26/wsl2-gui-idea-config/</a><br>搜狗输入法参考：<a href="https://zhuanlan.zhihu.com/p/142206571">https://zhuanlan.zhihu.com/p/142206571</a><br>搜狗输入法配置命令：sogouIme-configtool<br>我的配置和里面的不一样， 也没有遇到idea切不出输入法的问题<br>如果有的话参考文章里的方法或者看：<a href="https://github.com/microsoft/wslg/issues/278">https://github.com/microsoft/wslg/issues/278</a></p><h2 id="2-手动添加软件的快捷方式到开始菜单"><a href="#2-手动添加软件的快捷方式到开始菜单" class="headerlink" title="2 手动添加软件的快捷方式到开始菜单"></a>2 手动添加软件的快捷方式到开始菜单</h2><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">cd ~/.local/share/applications<br>sudo cp *.desktop /usr/share/applications<br><br></code></pre></td></tr></table></figure><p>参考链接：<a href="https://juejin.cn/post/6966630345915498526">https://juejin.cn/post/6966630345915498526</a></p><h2 id="3-wslg处理快捷方式无法以root用户运行应用"><a href="#3-wslg处理快捷方式无法以root用户运行应用" class="headerlink" title="3 wslg处理快捷方式无法以root用户运行应用"></a>3 wslg处理快捷方式无法以root用户运行应用</h2><p>很多linux的软件默认不支持root用户启动，如chrome，会报错：</p><blockquote><p>Running as root without –no-sandbox is not supported. See <a href="https://crbug.com/638180">https://crbug.com/638180</a></p></blockquote><p>这是报错原理：<br>wslg的应用通过windows快捷方式启动时，用的是wsl默认用户，<br>而我设置的默认用户是root，会导致闪退。<br>因为wslg我使用频率很低，故没必要特地去把默认用户改为普通用户。</p><p>这里介绍下几种处理方法</p><h3 id="（0）-修改默认用户"><a href="#（0）-修改默认用户" class="headerlink" title="（0） 修改默认用户"></a>（0） 修改默认用户</h3><p>使用范围：平常习惯使用sudo命令操作的<br>有效期：永久</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">Ubuntu-20.04 config --default-user 系统安装时创建的用户<br><br></code></pre></td></tr></table></figure><h3 id="（1）-修改linux桌面图标配置-原理：修改启动命令"><a href="#（1）-修改linux桌面图标配置-原理：修改启动命令" class="headerlink" title="（1） 修改linux桌面图标配置(原理：修改启动命令)"></a>（1） 修改linux桌面图标配置(原理：修改启动命令)</h3><p>适用范围：可以在启动命令中添加特定选项规避问题的，如chrome可使用–no-sandbox<br>有效期：直到软件更新（图标刷新）</p><p>以下为例子</p><p>去 /usr/share/applications 目录对chrome的快捷方式进行编辑，修改其启动命令</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">vim /usr/share/applications/google-chrome.desktop<br><br></code></pre></td></tr></table></figure><p>在其Exec列添加 <em>–no-sandbox</em> ，如下</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">Exec=/usr/bin/google-chrome-stable --no-sandbox %U<br></code></pre></td></tr></table></figure><h3 id="（2）-修改windows-wslg桌面图标配置-原理：修改调用用户"><a href="#（2）-修改windows-wslg桌面图标配置-原理：修改调用用户" class="headerlink" title="（2） 修改windows-wslg桌面图标配置(原理：修改调用用户)"></a>（2） 修改windows-wslg桌面图标配置(原理：修改调用用户)</h3><p>适用范围：所有<br>有效期：重启失效</p><p>有些软件（<strong>如firefox</strong>）就是不支持root用户运行，<br>这个时候需要修改其windows图标对应启动命令，指定用户启动</p><p>如下，linux的图标所在文件夹类似：C:\Users\用户名\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Ubuntu-20.04<br>将快捷方式-属性-目标由“C:\Windows\System32\wslg.exe ~ -d Ubuntu-20.04 firefox”改为<br>“C:\Windows\System32\wslg.exe ~ -d Ubuntu-20.04 su 用户名 -c firefox”</p><p><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1660206303479.png" alt="linux图标"><br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/20220811162605.png" alt="firefox属性"></p><h4 id="（3）直接shell启动"><a href="#（3）直接shell启动" class="headerlink" title="（3）直接shell启动"></a>（3）直接shell启动</h4><p>适用范围：所有<br>有效期：shell会话</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">su 普通用户<br>google-chrome/firefox<br><br></code></pre></td></tr></table></figure><h2 id="4-在wslg使用chrome访问kerberos页面"><a href="#4-在wslg使用chrome访问kerberos页面" class="headerlink" title="4 在wslg使用chrome访问kerberos页面"></a>4 在wslg使用chrome访问kerberos页面</h2><p>wslg-chrome安装见：<a href="https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps#run-linux-gui-apps">https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps#run-linux-gui-apps</a></p><ol><li>配置kerberos<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /etc/opt/chrome/policies/managed/<br>cat << 'EOF' > /etc/opt/chrome/policies/managed/kerberos.json<br>{ "AuthServerAllowlist": "*",<br> "AuthNegotiateDelegateAllowlist": "*" }<br>EOF<br><br></code></pre></td></tr></table></figure></li><li>启动<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">kinit -kt /home/admin02.service.keytab admin02/adp@HADOOP.COM<br>google-chrome --no-sandbox<br><br></code></pre></td></tr></table></figure></li></ol><h2 id="其他"><a href="#其他" class="headerlink" title="其他"></a>其他</h2><h3 id="直接使用windows访问kerberos页面"><a href="#直接使用windows访问kerberos页面" class="headerlink" title="直接使用windows访问kerberos页面"></a>直接使用windows访问kerberos页面</h3><p>参考 <a href="https://blog.csdn.net/sinat_20554629/article/details/105283518">https://blog.csdn.net/sinat_20554629/article/details/105283518</a> ，<br>在Windows上，必须使用Firefox + MIT Kerberos Windows客户端进行认证<br>不支持edge、chrome、ie等</p><p>firefox访问kerberos参考：<br><a href="https://blog.csdn.net/IUNIQUE/article/details/108615090">https://blog.csdn.net/IUNIQUE/article/details/108615090</a><br><a href="https://www.cnblogs.com/kischn/p/7443343.html">https://www.cnblogs.com/kischn/p/7443343.html</a></p><p>具体操作流程参考如下：<br>1.下载并安装mit kerberos客户端<br>下载地址：<a href="https://web.mit.edu/kerberos/dist/">https://web.mit.edu/kerberos/dist/</a><br>2.修改PATH<br>安装完了kerberos就自动的在 PATH 里面加上了自己的目录,但是在最后。<br>jdk也自带了kerberos工具，所以需要跳转PATH里kerberos的优先级在前面<br>3.将krb5.conf文件内容复制到krb5.ini<br>完整默认路径：C:\ProgramData\MIT\Kerberos5\krb5.ini<br>4.kinit登录<br>5.修改firefox配置<br> 1. 输入 about:config<br> 2. 修改配置项 network.negotiate-auth.trusted-uris 的值为： master01,worker01<br> 3. 修改配置项 network.auth.use-sspi 的值为： false<br> 4. 重启firefox<br>6.使用firefox打开kerberos页面</p><p>注意：机器重启会导致kinit信息失效</p>]]></content>
<categories>
<category> WSL开发系列 </category>
</categories>
<tags>
<tag> 生产力 </tag>
<tag> WSL </tag>
</tags>
</entry>
<entry>
<title>WSL开发系列-基础篇</title>
<link href="//wsl-dev-base/"/>
<url>//wsl-dev-base/</url>
<content type="html"><![CDATA[<p>本人使用win11+idea+docker+wsl2作为日常开发测试环境，<br>随着版本的迭代完善，加上本人丰富的踩坑经验，wsl已基本能满足我对linux单机使用的需求。<br>故重新整理wsl开发环境相关配置、优化、使用集成，形成本系列文章。<br>本文为WSL开发系列的开篇，也是目录，建议阅读此篇，再根据需求跳转。</p><span id="more"></span><h2 id="导航"><a href="#导航" class="headerlink" title="导航"></a>导航</h2><ul><li><a href="https://linshenkx.github.io/wsl-dev-base/">基础篇</a></li><li><a href="https://linshenkx.github.io/wsl-dev-static-ip/">网络篇</a></li><li><a href="https://linshenkx.github.io/wsl-dev-wslg/">gui篇</a></li><li><a href="https://linshenkx.github.io/wsl2_idea2021/">idea篇</a></li></ul><h2 id="一-问答"><a href="#一-问答" class="headerlink" title="一 问答"></a>一 问答</h2><ul><li><p>问：为什么不买一台linux服务器？</p><p>答：因为没钱，且交不起电费。事实上已经买了nas了，但性能孱弱，无法用于开发。</p></li><li><p>问：为什么不买云主机？</p><p>答：因为没钱，且交不起电费。事实上已经买了两台了，加起来不到8c16g（我笔记本的一半）。</p></li><li><p>问：为什么不装双系统系统？</p><p>答：体验割裂，垃圾。</p></li><li><p>问：为什么不直接装linux系统/黑苹果？</p><p>答：生态匮乏（要玩游戏）。</p></li><li><p>问：和虚拟机相比除了性能有什么优势？</p><p>答：虚拟机使用上相当于多了台服务器，开发上并不方便。<br> 而wsl可以实现一个工程，一套源码，在windows下开发，在linux下运行。<br> idea自带支持wsl，虽然也支持ssh，但体验上差一些。</p><h2 id="二-安装和基础配置"><a href="#二-安装和基础配置" class="headerlink" title="二 安装和基础配置"></a>二 安装和基础配置</h2><p>建议优先看官方文档，会不断更新，是最可靠的。<br><a href="https://learn.microsoft.com/zh-cn/windows/wsl/">https://learn.microsoft.com/zh-cn/windows/wsl/</a><br>下面的一些操作也是，我会尽可能给出官方链接，对比着来看。<br>我的是自己的实际操作或补充，但可能失效了。</p></li></ul><h3 id="1-安装WSL"><a href="#1-安装WSL" class="headerlink" title="1 安装WSL"></a>1 安装WSL</h3><p>见：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/install">https://learn.microsoft.com/zh-cn/windows/wsl/install</a><br>现在已经可以一行命令安装了：<code>wsl --install</code><br>我用的是Ubuntu-20.04，对应的命令应该是 <code>wsl --install Ubuntu-20.04</code><br>安装过程会提示输入一个默认用户和对应密码，注意不能用root（已存在）</p><h3 id="2-安装windows-terminal"><a href="#2-安装windows-terminal" class="headerlink" title="2 安装windows terminal"></a>2 安装windows terminal</h3><p>在应用商店下载安装，自行美化即可，新版本已可直接通过选项配置，不需要直接操作配置文件</p><p>另外建议在terminal配置文件中，找到Ubuntu的配置项，修改其启动目录为 “//wsl$/Ubuntu-20.04/home”</p><h3 id="3-修改默认用户为root"><a href="#3-修改默认用户为root" class="headerlink" title="3 修改默认用户为root"></a>3 修改默认用户为root</h3><p>见：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/basic-commands#change-the-default-user-for-a-distribution">https://learn.microsoft.com/zh-cn/windows/wsl/basic-commands#change-the-default-user-for-a-distribution</a></p><ol><li><p>查看当前wsl列表：<br>之所以要这一步是因为如果装了多个版本的ubuntu，其标识可能不一样</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">wsl -l <br></code></pre></td></tr></table></figure></li><li><p>修改对应wsl默认用户</p><p> 注意，操作标识不含特殊符号，如Ubuntu-20.04,则以下命令为Ubuntu2004（或全小写 ubuntu2004）</p> <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">Ubuntu-20.04 config --default-user root<br></code></pre></td></tr></table></figure><p> <img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1660199927570.png" alt="截图"></p><p> 不过需要注意，很多linux的软件默认不支持root用户启动，如chrome，相关处理方法见下文</p><h3 id="4-将linux根目录映射到网络驱动器"><a href="#4-将linux根目录映射到网络驱动器" class="headerlink" title="4 将linux根目录映射到网络驱动器"></a>4 将linux根目录映射到网络驱动器</h3><p>见：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/filesystems#view-your-current-directory-in-windows-file-explorer">https://learn.microsoft.com/zh-cn/windows/wsl/filesystems#view-your-current-directory-in-windows-file-explorer</a><br>在文件地址栏输入 \wsl$ ，可以看到已安装的wsl文件系统，如 Ubuntu<br>右键-映射网络驱动器，选择要映射的盘符即可。<br>另外注意，可以用<code>explorer.exe . </code>命令使用windows文件资源管理器打开当前路径（这个功能用得比较少）</p></li></ol><h3 id="5-配置国内软件源"><a href="#5-配置国内软件源" class="headerlink" title="5 配置国内软件源"></a>5 配置国内软件源</h3><p>参考：<a href="https://mirrors.tuna.tsinghua.edu.cn/help/ubuntu/">https://mirrors.tuna.tsinghua.edu.cn/help/ubuntu/</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 备份</span><br>cp /etc/apt/sources.list /etc/apt/sources.list.backup<br><span class="hljs-meta">#</span><span class="bash"> 替换</span><br>sudo sed -i "s@http://.*archive.ubuntu.com@https://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list<br>sudo sed -i "s@http://.*security.ubuntu.com@https://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list<br><span class="hljs-meta">#</span><span class="bash"> 更新</span><br>apt-get update -y && apt-get upgrade -y<br></code></pre></td></tr></table></figure><h2 id="三-可选进阶配置"><a href="#三-可选进阶配置" class="headerlink" title="三 可选进阶配置"></a>三 可选进阶配置</h2><h3 id="1-配置代理"><a href="#1-配置代理" class="headerlink" title="1 配置代理"></a>1 配置代理</h3><p>如果代理软件是装在windows系统的，参考WSL开发系列-网络篇，使用my.win就行了</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat << 'EOF' >> /etc/profile<br>export http_proxy="http://my.win:7890"<br>export https_proxy="http://my.win:7890"<br>export ftp_proxy="http://my.win:7890"<br>export PROXY_HOST="my.win"<br>export PROXY_PORT=7890<br>export no_proxy="*.internal,localhost,172.*,127.*"<br>EOF<br><br></code></pre></td></tr></table></figure><h3 id="2-配置ssh"><a href="#2-配置ssh" class="headerlink" title="2 配置ssh"></a>2 配置ssh</h3><p>除非确实需要，不然不建议启用ssh<br>应用场景：idea支持ssh运行环境但不支持wsl的时候</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 更新软件</span></span><br>apt -y update && apt -y upgrade<br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 安装ssh（一般已安装）</span></span><br>apt install -y openssh-server<br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 配置允许密码登录</span></span><br>sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config<br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 配置允许root登录</span></span><br>sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config<br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 启动/重启</span></span><br>service ssh restart<br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 修改root密码</span></span><br>passwd<br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 链接root目录(一般不需要执行！)</span></span><br>ln -s /root /home/root<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 测试ssh(使用刚设置的root密码登录)</span></span><br>ssh root@localhost<br><br></code></pre></td></tr></table></figure><h3 id="3-在wsl中启用systemd支持"><a href="#3-在wsl中启用systemd支持" class="headerlink" title="3 在wsl中启用systemd支持"></a>3 在wsl中启用systemd支持</h3><p>wsl.conf配置参考：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/wsl-config">https://learn.microsoft.com/zh-cn/windows/wsl/wsl-config</a></p><p>在/etc/wsl.conf（文件不存在则新建）中添加如下配置</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">[boot]<br>systemd=true<br></code></pre></td></tr></table></figure><p>需要<code>wsl --shutdown</code>重启生效<br>然后验证：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">systemctl list-unit-files --type=service<br><br></code></pre></td></tr></table></figure><h3 id="4-禁止添加Windows-PATH"><a href="#4-禁止添加Windows-PATH" class="headerlink" title="4 禁止添加Windows-PATH"></a>4 禁止添加Windows-PATH</h3><p>windows和wsl同时安装java，但wsl的hadoop需要使用wsl的java而非windows的</p><p>为了避免开发环境冲突，建议关闭Windows-PATH。<br>需要在wsl中使用windows软件的时候用路径全称即可，一般用得少。</p><p>在/etc/wsl.conf（文件不存在则新建）中添加如下配置</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 不加载Windows中的PATH内容</span><br>[interop]<br>appendWindowsPath = false<br></code></pre></td></tr></table></figure><h3 id="5-wsl-docker（ext4-vhdx）磁盘空间回收"><a href="#5-wsl-docker（ext4-vhdx）磁盘空间回收" class="headerlink" title="5 wsl-docker（ext4.vhdx）磁盘空间回收"></a>5 wsl-docker（ext4.vhdx）磁盘空间回收</h3><p>问题描述：<br>Docker Desktop for Windows v2 使用 WSL2，<br>将所有映像和容器文件存储在单独的虚拟卷 (vhdx) 中。<br>当需要更多空间（达到一定限度）时，这个虚拟硬盘文件会自动增长。<br>不幸的是，如果您回收一些空间，即通过删除未使用的图像，vhdx 不会自动缩小。</p><p>处理方法：<br>暂时没有好的处理方法，得手动执行命令，或者利用Docker Desktop的Troubleshoot完全清除。<br>参考： <a href="https://github.com/microsoft/WSL/issues/4699">https://github.com/microsoft/WSL/issues/4699</a></p><p>ext4.vhdx的完整路径获取方法：<br>见：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/disk-space?source=recommendations#how-to-locate-the-vhdx-file-and-disk-path-for-your-linux-distribution">https://learn.microsoft.com/zh-cn/windows/wsl/disk-space?source=recommendations#how-to-locate-the-vhdx-file-and-disk-path-for-your-linux-distribution</a></p><figure class="highlight powershell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs powershell">(<span class="hljs-built_in">Get-ChildItem</span> <span class="hljs-literal">-Path</span> HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss | <span class="hljs-built_in">Where-Object</span> { <span class="hljs-variable">$_</span>.GetValue(<span class="hljs-string">"DistributionName"</span>) <span class="hljs-operator">-eq</span> <span class="hljs-string">'Ubuntu-20.04'</span> }).GetValue(<span class="hljs-string">"BasePath"</span>) + <span class="hljs-string">"\ext4.vhdx"</span><br></code></pre></td></tr></table></figure><p>执行命令前注意先停止Docker Desktop和WSL2<br>windows专业版执行命令：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">wsl --shutdown<br>optimize-vhd -Path C:\…\…\ext4.vhdx -Mode full<br></code></pre></td></tr></table></figure><p>windows家庭版执行命令：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs shell">wsl --shutdown<br>diskpart<br><span class="hljs-meta">#</span><span class="bash"> open window Diskpart</span><br>select vdisk file="C:\…\…\ext4.vhdx"<br>attach vdisk readonly<br>compact vdisk<br>detach vdisk<br>exit<br></code></pre></td></tr></table></figure><h3 id="6-在wsl中以管理员身份调用powershell"><a href="#6-在wsl中以管理员身份调用powershell" class="headerlink" title="6 在wsl中以管理员身份调用powershell"></a>6 在wsl中以管理员身份调用powershell</h3><blockquote><p>2023.4更新：在更新wsl之后，调用wudo失败（socket.timeout: timed out），<br>5个open的issues有3个是与此相关的，可见不是小概率事件。且无较好解决方法，故本步骤不推荐使用。</p></blockquote><p>wsl本身就有对windows软件的交互能力，但默认情况下没有管理员权限</p><p>参考：<a href="https://github.com/Chronial/wsl-sudo">https://github.com/Chronial/wsl-sudo</a><br>前提：已安装python3，且版本>= 3.5</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 下载py脚本</span><br>wget https://raw.githubusercontent.com/Chronial/wsl-sudo/master/wsl-sudo.py<br><span class="hljs-meta">#</span><span class="bash"> 替换成完整路径（如果你的wsl没有关闭使用windows的环境变量，则可忽略这一步）</span><br>sed -i "s#\"powershell.exe\"#\"/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe\"#g" wsl-sudo.py<br><span class="hljs-meta">#</span><span class="bash"> 修改名称并授权</span><br>mv wsl-sudo.py wudo<br>chmod a+x wudo<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 测试：报错<span class="hljs-string">"拒绝访问"</span></span><br>/mnt/c/WINDOWS/system32/net.exe sessions<br><span class="hljs-meta">#</span><span class="bash"> 测试：无报错</span><br>./wudo /mnt/c/WINDOWS/system32/net.exe sessions<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 拷贝到系统路径下</span><br>mv wudo /usr/bin/<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 再次测试</span><br>wudo /mnt/c/WINDOWS/system32/net.exe sessions<br><br></code></pre></td></tr></table></figure><p>以后在wsl里面需要以admin身份调用powershell指令只需要加上前缀 <strong>wudo</strong> 即可</p><h3 id="7-wsl存储位置迁移"><a href="#7-wsl存储位置迁移" class="headerlink" title="7 wsl存储位置迁移"></a>7 wsl存储位置迁移</h3><p>wsl镜像导出导入命令参考：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/enterprise">https://learn.microsoft.com/zh-cn/windows/wsl/enterprise</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 关闭wsl</span><br>wsl --shutdown<br><span class="hljs-meta">#</span><span class="bash"> 确保为stop状态</span><br>wsl -l -v<br><span class="hljs-meta">#</span><span class="bash"> 文件夹需提前创建</span><br>wsl --export Ubuntu-20.04 E:\UbuntuWSL\ubuntu.tar<br><span class="hljs-meta">#</span><span class="bash"> 注销</span><br>wsl --unregister Ubuntu-20.04<br><span class="hljs-meta">#</span><span class="bash"> 确定已注销</span><br>wsl -l -v<br><span class="hljs-meta">#</span><span class="bash"> 执行导入(如果失败可再次尝试执行)</span><br><span class="hljs-meta">#</span><span class="bash"> 执行完会在E:\UbuntuWSL\创建ext4.vhdx，然后ubuntu.tar就可以删掉了</span><br>wsl --import Ubuntu-20.04 E:\UbuntuWSL\ E:\UbuntuWSL\ubuntu.tar<br>wsl -l -v<br></code></pre></td></tr></table></figure><h3 id="7-windows-docker存储位置迁移"><a href="#7-windows-docker存储位置迁移" class="headerlink" title="7 windows-docker存储位置迁移"></a>7 windows-docker存储位置迁移</h3><p>除了常规的用户linux系统（如ubuntu），新版的windows-docker也是基于wsl2的</p><p>参考：<a href="https://docs.docker.com/desktop/wsl/">https://docs.docker.com/desktop/wsl/</a></p><blockquote><p>Docker Desktop installs two special-purpose internal Linux distros docker-desktop and docker-desktop-data.<br>The first (docker-desktop) is used to run the Docker engine (dockerd) while the second (docker-desktop-data) stores containers and images.<br>Neither can be used for general development.</p></blockquote><p>即docker使用的磁盘目录无法通过data-root来指定，而是在名为docker-desktop-data的wsl上。<br>其ext4.vhdx具体位置可通过以下命令获取：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">(Get-ChildItem -Path HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss | Where-Object { $_.GetValue("DistributionName") -eq 'docker-desktop-data' }).GetValue("BasePath") + "\ext4.vhdx"<br></code></pre></td></tr></table></figure><p>所以其目录迁移和普通的wsl迁移一致</p><p>执行下面命令之前，先关闭windows-docker</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 关闭wsl</span><br>wsl --shutdown<br><span class="hljs-meta">#</span><span class="bash"> 确保为stop状态</span><br>wsl -l -v<br><span class="hljs-meta">#</span><span class="bash"> 文件夹需提前创建</span><br>wsl --export docker-desktop-data E:\dockerData\docker-desktop-data.tar<br><span class="hljs-meta">#</span><span class="bash"> 注销</span><br>wsl --unregister docker-desktop-data<br><span class="hljs-meta">#</span><span class="bash"> 确定已注销</span><br>wsl -l -v<br><span class="hljs-meta">#</span><span class="bash"> 执行导入(如果失败可再次尝试执行)</span><br>wsl --import docker-desktop-data E:\dockerData\ E:\dockerData\docker-desktop-data.tar<br>wsl -l -v<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 再次查看位置，确认修改成功</span><br> (Get-ChildItem -Path HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss | Where-Object { $_.GetValue("DistributionName") -eq 'docker-desktop-data' }).GetValue("BasePath") + "\ext4.vhdx"<br><br></code></pre></td></tr></table></figure><p>再次启动windows-docker</p><p>参考：<a href="https://stackoverflow.com/questions/62441307/how-can-i-change-the-location-of-docker-images-when-using-docker-desktop-on-wsl2">https://stackoverflow.com/questions/62441307/how-can-i-change-the-location-of-docker-images-when-using-docker-desktop-on-wsl2</a></p>]]></content>
<categories>
<category> WSL开发系列 </category>
</categories>
<tags>
<tag> 生产力 </tag>
<tag> WSL </tag>
</tags>
</entry>
<entry>
<title>WSL开发系列-网络篇（wsl固定ip）</title>
<link href="//wsl-dev-static-ip/"/>
<url>//wsl-dev-static-ip/</url>
<content type="html"><![CDATA[<p>wsl网络情况较复杂，本篇除了官方的网络配置事项，主要添加了：<br>1.wsl自动获取宿主机ip的脚本。<br>2.wsl设置固定（静态）ip的配置方法。</p><p>2023.4更新：因wudo异常，wsl网段的windows静态ip的设置使用bat脚本+任务计划程序实现</p><span id="more"></span><h2 id="导航"><a href="#导航" class="headerlink" title="导航"></a>导航</h2><ul><li><a href="https://linshenkx.github.io/wsl-dev-base/">基础篇</a></li><li><a href="https://linshenkx.github.io/wsl-dev-static-ip/">网络篇</a></li><li><a href="https://linshenkx.github.io/wsl-dev-wslg/">gui篇</a></li><li><a href="https://linshenkx.github.io/wsl2_idea2021/">idea篇</a></li></ul><h2 id="一-基于官方说明"><a href="#一-基于官方说明" class="headerlink" title="一 基于官方说明"></a>一 基于官方说明</h2><p>建议先阅读官方文档：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/networking">https://learn.microsoft.com/zh-cn/windows/wsl/networking</a></p><p>通过官方配置和以下教程，可以达到如下效果，无需记住任何ip</p><ul><li>windows访问wsl：使用localhost即可（但是不能使用127.0.0.1）</li><li>wsl访问windows：使用自动配置的my.win即可（见下）</li></ul><p>如果系统是专业版（可开启Hyper-V），可参考 <a href="https://github.com/luxzg/WSL2-fixes">https://github.com/luxzg/WSL2-fixes</a> 配置wsl桥接网络。<br>本文相关配置在使用wsl桥接网络的情况下可能无效或没必要，不过桥接网络本身还没有被wsl正式支持的。<br>且我用的是家庭版，虽然也可以开启，但可能有风险，故暂不使用。</p><h3 id="1-wsl自动获取宿主机ip"><a href="#1-wsl自动获取宿主机ip" class="headerlink" title="1 wsl自动获取宿主机ip"></a>1 wsl自动获取宿主机ip</h3><blockquote><p>注意：如果采用了下面设置静态ip的方法，会一并设置windows静态ip，就不需要使用这里的脚本了！</p></blockquote><p>见：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/networking#accessing-windows-networking-apps-from-linux-host-ip">https://learn.microsoft.com/zh-cn/windows/wsl/networking#accessing-windows-networking-apps-from-linux-host-ip</a></p><p>这里的脚本主要是负责自动获取win-ip、添加host映射：<br>1.wsl的hosts文件默认情况下是从windows继承过来的，每次都会重新生成（可通过wsl.conf控制），所以需先删除原来的my.win配置<br>2.sed结果需要使用/tmp/hosts作为中转，不能直接输出到/etc/hosts，否则/etc/hosts会被清空</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat << 'EOF' >> /etc/profile<br>host_ip=$(cat /etc/resolv.conf |grep "nameserver" |cut -f 2 -d " ")<br>sed -e "/my.win/d" /etc/hosts > /tmp/hosts<br>cat /tmp/hosts > /etc/hosts<br>echo "$host_ip my.win" >>/etc/hosts<br>EOF<br><br></code></pre></td></tr></table></figure><p>另外在windows的hosts文件中配置 <code>127.0.0.1 my.win</code>,<br>这样，在windows上启动的服务，不管在wsl还是windows上，都可以通过my.win访问到了</p><h3 id="2-局域网访问wsl2的操作示例"><a href="#2-局域网访问wsl2的操作示例" class="headerlink" title="2 局域网访问wsl2的操作示例"></a>2 局域网访问wsl2的操作示例</h3><p>见：<a href="https://learn.microsoft.com/zh-cn/windows/wsl/networking#accessing-a-wsl-2-distribution-from-your-local-area-network-lan">https://learn.microsoft.com/zh-cn/windows/wsl/networking#accessing-a-wsl-2-distribution-from-your-local-area-network-lan</a></p><p>需要使用netsh interface portproxy功能，按照help提示操作就行</p><p>以下为示例,局域网其他机器只需要连接宿主机的19909端口即可访问wsl2内19909端口的内容:</p><p>在powershell管理员模式下执行命令，</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 重置端口代理</span><br>netsh interface portproxy reset<br><span class="hljs-meta">#</span><span class="bash"> 最好一并设置v4tov6，例如springboot网络默认是v6的</span><br>netsh interface portproxy add v4tov6 listenport=19909 listenaddress=0.0.0.0 connectport=19909 connectaddress=localhost<br>netsh interface portproxy add v4tov4 listenport=19909 listenaddress=0.0.0.0 connectport=19909 connectaddress=localhost<br><span class="hljs-meta">#</span><span class="bash"> 查看端口配置</span><br>netsh interface portproxy show all<br><br></code></pre></td></tr></table></figure><h2 id="二-wsl配置固定ip"><a href="#二-wsl配置固定ip" class="headerlink" title="二 wsl配置固定ip"></a>二 wsl配置固定ip</h2><p>除了docker访问wsl，其他大部分场景下不需要这个功能</p><p>windows的docker不支持使用host网络，也就是不能通过localhost访问wsl<br>ip是可以的，但wsl的ip每次启动都会变，所以比较麻烦，这里通过使用固定ip解决</p><p>参考：<a href="https://www.cnblogs.com/Likfees/p/16750300.html">https://www.cnblogs.com/Likfees/p/16750300.html</a></p><p>2023.4更新：wudo有bug，不推荐使用，WIN_IP的设置还是放到windows下去执行</p><h3 id="1-使用Shell启动脚本设置wsl静态ip"><a href="#1-使用Shell启动脚本设置wsl静态ip" class="headerlink" title="1 使用Shell启动脚本设置wsl静态ip"></a>1 使用Shell启动脚本设置wsl静态ip</h3><p><del>注意此处调用了wudo（见<a href="https://linshenkx.github.io/wsl-dev-base/">WSL开发系列-基础篇</a>）以使用管理员权限调用netsh工具</del></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat << 'EOF' > setup-static-ip.sh <br><span class="hljs-meta">#</span><span class="bash">!/bin/bash</span><br><br>WSL_IP=192.168.95.100<br>WIN_IP=192.168.95.101<br>WSL_BROADCAST=192.168.95.255<br><br>WSL_IP_SET_RESULT=$(ip addr show eth0 | grep -sc "$WSL_IP/24" )<br>if [ $WSL_IP_SET_RESULT == 0 ]; then<br> ip addr add $WSL_IP/24 broadcast $WSL_BROADCAST dev eth0 label eth0:staticip<br> echo "set WSL_IP $WSL_IP success "<br>fi<br>echo "check and set WSL_IP($WSL_IP) WIN_IP($WIN_IP) success "<br>EOF<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 2023.4更新：以下内容已移除，WIN_IP的设置放到windows下去执行</span><br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash">WIN_IP_SET_RESULT=$(/mnt/c/WINDOWS/system32/netsh.exe interface ip show addresses <span class="hljs-string">"vEthernet (WSL)"</span> | grep -sc <span class="hljs-string">"<span class="hljs-variable">$WIN_IP</span>"</span> )</span><br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-keyword">if</span> [ <span class="hljs-variable">$WIN_IP_SET_RESULT</span> == 0 ]; <span class="hljs-keyword">then</span></span><br><span class="hljs-meta">#</span><span class="bash"> wudo /mnt/c/WINDOWS/system32/netsh.exe interface ip add address <span class="hljs-string">"vEthernet (WSL)"</span> address=<span class="hljs-variable">$WIN_IP</span>/24</span><br><span class="hljs-meta">#</span><span class="bash"> <span class="hljs-built_in">echo</span> <span class="hljs-string">"set WIN_IP <span class="hljs-variable">$WIN_IP</span> success "</span></span><br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-keyword">fi</span></span><br></code></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 设置权限</span><br>chmod a+x setup-static-ip.sh<br><span class="hljs-meta">#</span><span class="bash"> 测试执行</span><br>source setup-static-ip.sh<br><span class="hljs-meta">#</span><span class="bash"> 验证结果:wsl</span><br>ip addr show eth0<br><span class="hljs-meta">#</span><span class="bash"> 拷贝到系统路径下</span><br>mv setup-static-ip.sh /usr/bin/<br><span class="hljs-meta">#</span><span class="bash"> 再次测试</span><br>setup-static-ip.sh<br>echo "setup-static-ip.sh" >> /etc/profile<br></code></pre></td></tr></table></figure><h3 id="2-使用Bat启动脚本设置win静态ip"><a href="#2-使用Bat启动脚本设置win静态ip" class="headerlink" title="2 使用Bat启动脚本设置win静态ip"></a>2 使用Bat启动脚本设置win静态ip</h3><h4 id="1-新建winip-bat"><a href="#1-新建winip-bat" class="headerlink" title="1 新建winip.bat"></a>1 新建winip.bat</h4><figure class="highlight powershell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs powershell">@<span class="hljs-built_in">echo</span> off<br>setlocal enabledelayedexpansion<br>::检查宿主机有没有我需要的IP<br><span class="hljs-built_in">set</span> WIN_IP=<span class="hljs-number">192.168</span>.<span class="hljs-number">95.101</span><br>netsh interface ip show addresses <span class="hljs-string">"vEthernet (WSL)"</span> | findstr %WIN_IP% <br><span class="hljs-keyword">if</span> !errorlevel! equ <span class="hljs-number">0</span> (<br> <span class="hljs-built_in">echo</span> windows ip has <span class="hljs-built_in">set</span><br>) <span class="hljs-keyword">else</span> (<br> ::IP不存在则绑定IP<br> netsh interface ip add address <span class="hljs-string">"vEthernet (WSL)"</span> address=%WIN_IP%/<span class="hljs-number">24</span><br> <span class="hljs-built_in">echo</span> <span class="hljs-built_in">set</span> windows ip success: %WIN_IP%<br>)<br>pause<br><br></code></pre></td></tr></table></figure><h4 id="2-配置开机执行"><a href="#2-配置开机执行" class="headerlink" title="2 配置开机执行"></a>2 配置开机执行</h4><p>打开“任务计划程序”，创建任务，重要的配置项如下：</p><ul><li>常规-更改用户或组-高级-立即查找-选择“SYSTEM”用户-确定：使用SYSTEM而非当前用户运行就不会有个cmd黑框一闪而过</li><li>常规-勾选使用最高权限运行</li><li>触发器-新建-启动时</li><li>触发器-新建-登陆时-配置：勾选延迟任务时间并设置为1分钟，勾选重复任务间隔并设置为5分钟，持续时间为15分钟：多次触发执行避免因意外执行失败</li><li>操作-启动程序-“程序或脚本”设置为winip.bat路径</li></ul><p>创建完可选择任务右键点击运行，状态会变成“正在运行”，刷新一下即恢复“准备就绪”说明执行完成</p><h4 id="3-验证结果"><a href="#3-验证结果" class="headerlink" title="3. 验证结果"></a>3. 验证结果</h4><figure class="highlight cmd"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs cmd">netsh.exe interface ip show addresses "vEthernet (WSL)"<br><br></code></pre></td></tr></table></figure><h3 id="3-配置hosts文件"><a href="#3-配置hosts文件" class="headerlink" title="3 配置hosts文件"></a>3 配置hosts文件</h3><p>现在直接在windows配置hosts就行了（wsl默认情况下会复制windows的hosts）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">192.168.95.101 my.win<br>192.168.95.100 my.wsl<br><br></code></pre></td></tr></table></figure><h3 id="4-验证"><a href="#4-验证" class="headerlink" title="4 验证"></a>4 验证</h3><p>分别在windows/wsl下ping my.win和my.wsl即可，然后重启，再测试一次，看能否自动设置<br>my.wsl需启动wsl实例后才会自动配置，才能ping通</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">ping my.win<br>ping my.wsl<br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> WSL开发系列 </category>
</categories>
<tags>
<tag> 生产力 </tag>
<tag> WSL </tag>
</tags>
</entry>
<entry>
<title>gitlab-ci-k8s-runner部署及缓存配置说明</title>
<link href="//gitlab-ci-k8s-runner-cache/"/>
<url>//gitlab-ci-k8s-runner-cache/</url>
<content type="html"><![CDATA[<p>因gitlab-ci的k8s-runner会将job里的每个stage都交由单独的pod来执行，导致默认的本地缓存无法使用。<br>本文记录了基于k8s pv的gitlab-ci的缓存配置方案，无需使用S3分布式存储。<br>并记录了gitlab-ci-k8s-runner相关的配置、部署步骤等。</p><span id="more"></span><h2 id="gitlab-ci-k8s-runner部署及缓存配置说明"><a href="#gitlab-ci-k8s-runner部署及缓存配置说明" class="headerlink" title="gitlab-ci-k8s-runner部署及缓存配置说明"></a>gitlab-ci-k8s-runner部署及缓存配置说明</h2><h3 id="一-gitlab-ci-cache机制说明"><a href="#一-gitlab-ci-cache机制说明" class="headerlink" title="一 gitlab-ci-cache机制说明"></a>一 gitlab-ci-cache机制说明</h3><p>参考：<a href="https://blog.csdn.net/xichenguan/article/details/101439395">https://blog.csdn.net/xichenguan/article/details/101439395</a></p><blockquote><p>Gitlab cache 机制可以大大加快 CI/CD Job 的执行速度。基础知识可以参看 <a href="https://docs.gitlab.com/ee/ci/yaml/index.html#cache">Gitlab Cache</a>。<br>下面直接总结在 Kubernetes 环境中的三种 Cache 的解决方案。</p><h4 id="1-Distributed-runners-caching"><a href="#1-Distributed-runners-caching" class="headerlink" title="1. Distributed runners caching"></a>1. <a href="https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching">Distributed runners caching</a></h4><p>gitlab runner job执行前从分布式存储中检查下载解压 cache 文件，job执行后，压缩上传 cache 文件到分布式存储。这是 gitlab 提供的通用的正宗的方法，在非 Kubernetes 环境中也可以使用；</p><h4 id="2-给-Job-Executor-Pod-挂载同一个-volume"><a href="#2-给-Job-Executor-Pod-挂载同一个-volume" class="headerlink" title="2. 给 Job Executor Pod 挂载同一个 volume"></a>2. 给 Job Executor Pod 挂载同一个 volume</h4><p>gitlab runner 提供了本地存储 cache 的方式，如果远程存储没有配置，gitlab runner 照样会压缩 cache 文件，然后按照目录规则存储到指定的目录。这种方式的原理是将此目录配置为 Kubernetes Volume，每个 Job 执行时都挂载此 Volume ，这样就相当于所有的 Job 有了一个集中式的存储。可以参考 <a href="https://help.aliyun.com/document_detail/106968.html?aly_as=b8lTvr8V">使用GitLab CI在Kubernetes服务上运行GitLab Runner并执行Pipeline</a> 尝试配置。</p><h4 id="3-不使用缓存，在-Kubernetes-集群中安装各类仓库的私服"><a href="#3-不使用缓存，在-Kubernetes-集群中安装各类仓库的私服" class="headerlink" title="3. 不使用缓存，在 Kubernetes 集群中安装各类仓库的私服"></a>3. 不使用缓存，在 Kubernetes 集群中安装各类仓库的私服</h4><p>Gitlab CI/CI Job 执行时，配置为使用这些私服，速度也很快。</p></blockquote><p>注意：方式1一般指S3存储，建议使用方式2</p><h3 id="二-准备缓存pv、pvc"><a href="#二-准备缓存pv、pvc" class="headerlink" title="二 准备缓存pv、pvc"></a>二 准备缓存pv、pvc</h3><h4 id="1-配置runner-nfs缓存目录"><a href="#1-配置runner-nfs缓存目录" class="headerlink" title="1.配置runner nfs缓存目录"></a>1.配置runner nfs缓存目录</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell">export CACHE_DIR=/home/runner-cache<br>mkdir -p $CACHE_DIR<br>chmod -R 777 $CACHE_DIR<br>echo "$CACHE_DIR *(rw,sync,no_root_squash,all_squash,no_subtree_check,anonuid=0,anongid=0)" >> /etc/exports<br>exportfs -a<br>showmount -e localhost<br><br></code></pre></td></tr></table></figure><h4 id="2-创建pv、pvc"><a href="#2-创建pv、pvc" class="headerlink" title="2.创建pv、pvc"></a>2.创建pv、pvc</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl -n=命名空间 apply -f gitlab-runner-pv-pvc.yaml<br></code></pre></td></tr></table></figure><h4 id="3-创建imagePullSecret"><a href="#3-创建imagePullSecret" class="headerlink" title="3. 创建imagePullSecret"></a>3. 创建imagePullSecret</h4><p>参考：<a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account">https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl -n=命名空间 create secret docker-registry myregistrykey --docker-server=DUMMY_SERVER \<br> --docker-username=DUMMY_USERNAME --docker-password=DUMMY_DOCKER_PASSWORD <br>kubectl -n=命名空间 get secret myregistrykey<br>kubectl -n=命名空间 get secret myregistrykey --output=json<br><br></code></pre></td></tr></table></figure><h4 id="4-可选：将gitlab-runner、gitlab-runner-helper镜像上传到私有仓库"><a href="#4-可选：将gitlab-runner、gitlab-runner-helper镜像上传到私有仓库" class="headerlink" title="4. 可选：将gitlab-runner、gitlab-runner-helper镜像上传到私有仓库"></a>4. 可选：将gitlab-runner、gitlab-runner-helper镜像上传到私有仓库</h4><p>gitlab-runner镜像</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker tag registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.8.0 私有仓库/gitlab-runner:alpine-v15.8.0<br>docker push 私有仓库/gitlab-runner:alpine-v15.8.0<br><br></code></pre></td></tr></table></figure><p>gitlab-runner-helper镜像</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker tag registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144 私有仓库/gitlab-runner-helper:x86_64-12335144<br>docker push 私有仓库/gitlab-runner-helper:x86_64-12335144<br><br></code></pre></td></tr></table></figure><h3 id="三-根据官方helm生成k8s-runner部署文件"><a href="#三-根据官方helm生成k8s-runner部署文件" class="headerlink" title="三 根据官方helm生成k8s-runner部署文件"></a>三 根据官方helm生成k8s-runner部署文件</h3><p>参考：<a href="https://docs.gitlab.com/runner/install/kubernetes.html">https://docs.gitlab.com/runner/install/kubernetes.html</a><br>默认values.yaml文件：<a href="https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/main/values.yaml">https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/main/values.yaml</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">helm repo add gitlab https://charts.gitlab.io<br>helm repo update gitlab<br>helm install --dry-run --debug --namespace 命名空间 \<br>gitlab-runner -f values.yaml gitlab/gitlab-runner > gitlab-runner-all.yaml<br><br>kubectl -n=命名空间 apply -f gitlab-runner-all.yaml<br></code></pre></td></tr></table></figure><h3 id="四-附：docker-runner部署"><a href="#四-附：docker-runner部署" class="headerlink" title="四 附：docker-runner部署"></a>四 附：docker-runner部署</h3><h4 id="运行"><a href="#运行" class="headerlink" title="运行"></a>运行</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker run -d --name gitlab-runner --restart always \<br> -v /home/gitlab-runner/config:/etc/gitlab-runner \<br> -v /var/run/docker.sock:/var/run/docker.sock \<br> gitlab/gitlab-runner:latest<br><br></code></pre></td></tr></table></figure><h4 id="注册"><a href="#注册" class="headerlink" title="注册"></a>注册</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker run --rm -it -v /home/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register<br><br></code></pre></td></tr></table></figure><h4 id="重启"><a href="#重启" class="headerlink" title="重启"></a>重启</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker restart gitlab-runner<br>docker logs -f gitlab-runner<br><br></code></pre></td></tr></table></figure><h4 id="修改配置"><a href="#修改配置" class="headerlink" title="修改配置"></a>修改配置</h4><p>在gitlab修改配置runner，支持运行未标记的作业</p><h3 id="五-value-yaml和pv-pvc文件参考"><a href="#五-value-yaml和pv-pvc文件参考" class="headerlink" title="五 value.yaml和pv-pvc文件参考"></a>五 value.yaml和pv-pvc文件参考</h3><h4 id="gitlab-runner-pv-pvc-yaml"><a href="#gitlab-runner-pv-pvc-yaml" class="headerlink" title="gitlab-runner-pv-pvc.yaml"></a>gitlab-runner-pv-pvc.yaml</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">v1</span><br><span class="hljs-attr">kind:</span> <span class="hljs-string">PersistentVolume</span><br><span class="hljs-attr">metadata:</span><br> <span class="hljs-attr">name:</span> <span class="hljs-string">gitlab-runner-cache-pv</span><br><span class="hljs-attr">spec:</span><br> <span class="hljs-attr">storageClassName:</span> <span class="hljs-string">gitlab-runner</span><br> <span class="hljs-attr">capacity:</span><br> <span class="hljs-attr">storage:</span> <span class="hljs-string">100Gi</span><br> <span class="hljs-attr">accessModes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">ReadWriteMany</span><br> <span class="hljs-attr">nfs:</span><br> <span class="hljs-attr">server:</span> <span class="hljs-string">$NFS_SERVER</span><br> <span class="hljs-attr">path:</span> <span class="hljs-string">"$NFS_PATH"</span><br><span class="hljs-meta">---</span><br><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">v1</span><br><span class="hljs-attr">kind:</span> <span class="hljs-string">PersistentVolumeClaim</span><br><span class="hljs-attr">metadata:</span><br> <span class="hljs-attr">name:</span> <span class="hljs-string">gitlab-runner-cache-pvc</span><br><span class="hljs-attr">spec:</span><br> <span class="hljs-attr">accessModes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">ReadWriteMany</span><br> <span class="hljs-attr">storageClassName:</span> <span class="hljs-string">gitlab-runner</span><br> <span class="hljs-attr">resources:</span><br> <span class="hljs-attr">requests:</span><br> <span class="hljs-attr">storage:</span> <span class="hljs-string">100Gi</span><br><br></code></pre></td></tr></table></figure><h4 id="gitlab-k8s-runner-helm的value-yaml文件配置"><a href="#gitlab-k8s-runner-helm的value-yaml文件配置" class="headerlink" title="gitlab-k8s-runner-helm的value.yaml文件配置"></a>gitlab-k8s-runner-helm的value.yaml文件配置</h4><p>参考：</p><ol><li>默认values.yaml文件：<a href="https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/main/values.yaml">https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/main/values.yaml</a></li><li><a href="https://docs.gitlab.com/runner/executors/kubernetes.html#overwriting-container-resources">https://docs.gitlab.com/runner/executors/kubernetes.html#overwriting-container-resources</a></li><li><a href="https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerskubernetes-section">https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerskubernetes-section</a><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">image:</span><br> <span class="hljs-attr">registry:</span> <span class="hljs-string">"自行填写"</span><br> <span class="hljs-attr">image:</span> <span class="hljs-string">"自行填写"</span><br> <span class="hljs-attr">tag:</span> <span class="hljs-string">"自行填写"</span><br><span class="hljs-attr">imagePullPolicy:</span> <span class="hljs-string">IfNotPresent</span><br><span class="hljs-attr">imagePullSecrets:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">"自行填写"</span><br><span class="hljs-attr">gitlabUrl:</span> <span class="hljs-string">"自行填写"</span><br><span class="hljs-attr">runnerRegistrationToken:</span> <span class="hljs-string">"自行填写"</span><br><span class="hljs-comment"># runnerToken: ""</span><br><span class="hljs-comment"># unregisterRunners: true</span><br><span class="hljs-attr">terminationGracePeriodSeconds:</span> <span class="hljs-number">3600</span><br><span class="hljs-comment"># certsSecretName:</span><br><span class="hljs-attr">concurrent:</span> <span class="hljs-number">10</span><br><span class="hljs-attr">checkInterval:</span> <span class="hljs-number">30</span><br><span class="hljs-comment"># logLevel:</span><br><span class="hljs-comment"># logFormat:</span><br><span class="hljs-comment"># sentryDsn:</span><br><span class="hljs-attr">preEntrypointScript:</span> <span class="hljs-string">|</span><br><span class="hljs-string"> echo "hello"</span><br><span class="hljs-string"></span><span class="hljs-attr">sessionServer:</span><br> <span class="hljs-attr">enabled:</span> <span class="hljs-literal">false</span><br> <span class="hljs-comment"># annotations: {}</span><br> <span class="hljs-comment"># timeout: 1800</span><br> <span class="hljs-comment"># internalPort: 8093</span><br> <span class="hljs-comment"># externalPort: 9000</span><br> <span class="hljs-comment"># publicIP: ""</span><br> <span class="hljs-comment"># loadBalancerSourceRanges:</span><br> <span class="hljs-comment"># - 1.2.3.4/32</span><br><br><span class="hljs-comment">## For RBAC support:</span><br><span class="hljs-attr">rbac:</span><br> <span class="hljs-attr">create:</span> <span class="hljs-literal">true</span><br> <span class="hljs-attr">rules:</span> []<br> <span class="hljs-comment"># - resources: ["configmaps", "pods", "pods/attach", "secrets", "services"]</span><br> <span class="hljs-comment"># verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]</span><br> <span class="hljs-comment"># - apiGroups: [""]</span><br> <span class="hljs-comment"># resources: ["pods/exec"]</span><br> <span class="hljs-comment"># verbs: ["create", "patch", "delete"]</span><br> <span class="hljs-attr">clusterWideAccess:</span> <span class="hljs-literal">false</span><br> <span class="hljs-comment"># serviceAccountName: default</span><br> <span class="hljs-comment"># serviceAccountAnnotations: {}</span><br> <span class="hljs-attr">podSecurityPolicy:</span><br> <span class="hljs-attr">enabled:</span> <span class="hljs-literal">false</span><br> <span class="hljs-attr">resourceNames:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-string">gitlab-runner</span><br><br> <span class="hljs-comment">## Specify one or more imagePullSecrets used for pulling the runner image</span><br> <span class="hljs-comment">##</span><br> <span class="hljs-comment">## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account</span><br> <span class="hljs-comment">##</span><br> <span class="hljs-attr">imagePullSecrets:</span> []<br><br><span class="hljs-attr">metrics:</span><br> <span class="hljs-attr">enabled:</span> <span class="hljs-literal">false</span><br> <span class="hljs-attr">portName:</span> <span class="hljs-string">metrics</span><br> <span class="hljs-attr">port:</span> <span class="hljs-number">9252</span><br> <span class="hljs-attr">serviceMonitor:</span><br> <span class="hljs-attr">enabled:</span> <span class="hljs-literal">false</span><br> <span class="hljs-comment">## labels: {}</span><br> <span class="hljs-comment"># interval: ""</span><br> <span class="hljs-comment"># scheme: "http"</span><br> <span class="hljs-comment"># tlsConfig: {}</span><br> <span class="hljs-comment"># path: "/metrics"</span><br> <span class="hljs-comment"># metricRelabelings: []</span><br> <span class="hljs-comment">## relabelings: []</span><br><span class="hljs-attr">service:</span><br> <span class="hljs-attr">enabled:</span> <span class="hljs-literal">false</span><br> <span class="hljs-comment"># labels: {}</span><br> <span class="hljs-comment"># annotations: {}</span><br> <span class="hljs-comment"># clusterIP: ""</span><br> <span class="hljs-comment"># externalIPs: []</span><br> <span class="hljs-comment"># loadBalancerIP: ""</span><br> <span class="hljs-comment"># loadBalancerSourceRanges: []</span><br> <span class="hljs-attr">type:</span> <span class="hljs-string">ClusterIP</span><br> <span class="hljs-comment"># metrics:</span><br> <span class="hljs-comment"># nodePort: ""</span><br> <span class="hljs-comment"># additionalPorts: []</span><br><br><span class="hljs-attr">runners:</span><br> <span class="hljs-comment"># runner configuration, where the multi line strings is evaluated as</span><br> <span class="hljs-comment"># template so you can specify helm values inside of it.</span><br> <span class="hljs-comment">#</span><br> <span class="hljs-comment"># tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function</span><br> <span class="hljs-comment"># runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html</span><br> <span class="hljs-comment"># 见：1. https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerskubernetes-section</span><br> <span class="hljs-comment"># 2. https://docs.gitlab.com/runner/executors/kubernetes.html#overwriting-container-resources</span><br> <span class="hljs-attr">config:</span> <span class="hljs-string">|</span><br><span class="hljs-string"> [[runners]]</span><br><span class="hljs-string"> cache_dir = "/cache"</span><br><span class="hljs-string"> [runners.kubernetes]</span><br><span class="hljs-string"> namespace = "自行填写"</span><br><span class="hljs-string"> image = "自行填写"</span><br><span class="hljs-string"> privileged = true</span><br><span class="hljs-string"> image_pull_secrets = ["自行填写"]</span><br><span class="hljs-string"> image_pull_policy = "if-not-present"</span><br><span class="hljs-string"> helper_image = "自行填写"</span><br><span class="hljs-string"> helper_cpu_limit = "1000m"</span><br><span class="hljs-string"> helper_memory_limit = "2048Mi"</span><br><span class="hljs-string"> helper_cpu_requests = "200m"</span><br><span class="hljs-string"> helper_memory_requests = "512Mi"</span><br><span class="hljs-string"> service_cpu_limit = "1000m"</span><br><span class="hljs-string"> service_memory_limit = "2048Mi"</span><br><span class="hljs-string"> service_cpu_requests = "200m"</span><br><span class="hljs-string"> service_memory_requests = "512Mi"</span><br><span class="hljs-string"> cpu_limit = "1000m"</span><br><span class="hljs-string"> cpu_limit_overwrite_max_allowed = "5000m"</span><br><span class="hljs-string"> memory_limit = "2048Mi"</span><br><span class="hljs-string"> memory_limit_overwrite_max_allowed = "10240Mi"</span><br><span class="hljs-string"> cpu_requests = "200m"</span><br><span class="hljs-string"> cpu_requests_overwrite_max_allowed = "5000m"</span><br><span class="hljs-string"> memory_requests = "512Mi"</span><br><span class="hljs-string"> memory_requests_overwrite_max_allowed = "10240Mi"</span><br><span class="hljs-string"> [[runners.kubernetes.volumes.host_path]]</span><br><span class="hljs-string"> name = "docker"</span><br><span class="hljs-string"> mount_path = "/var/run/docker.sock"</span><br><span class="hljs-string"> read_only = true</span><br><span class="hljs-string"> host_path = "/var/run/docker.sock"</span><br><span class="hljs-string"> [[runners.kubernetes.volumes.pvc]]</span><br><span class="hljs-string"> name = "gitlab-runner-cache-pvc"</span><br><span class="hljs-string"> mount_path = "/cache"</span><br><span class="hljs-string"></span><br><span class="hljs-attr">securityContext:</span><br> <span class="hljs-attr">allowPrivilegeEscalation:</span> <span class="hljs-literal">false</span><br> <span class="hljs-attr">readOnlyRootFilesystem:</span> <span class="hljs-literal">false</span><br> <span class="hljs-attr">runAsNonRoot:</span> <span class="hljs-literal">true</span><br> <span class="hljs-attr">privileged:</span> <span class="hljs-literal">false</span><br> <span class="hljs-attr">capabilities:</span><br> <span class="hljs-attr">drop:</span> [<span class="hljs-string">"ALL"</span>]<br><br><span class="hljs-attr">podSecurityContext:</span><br> <span class="hljs-attr">runAsUser:</span> <span class="hljs-number">100</span><br> <span class="hljs-comment"># runAsGroup: 65533</span><br> <span class="hljs-attr">fsGroup:</span> <span class="hljs-number">65533</span><br> <span class="hljs-comment"># supplementalGroups: [65533]</span><br> <span class="hljs-comment">## <span class="hljs-doctag">Note:</span> values for the ubuntu image:</span><br> <span class="hljs-comment"># runAsUser: 999</span><br> <span class="hljs-comment"># fsGroup: 999</span><br><br><span class="hljs-comment">## Configure resource requests and limits</span><br><span class="hljs-comment">## ref: http://kubernetes.io/docs/user-guide/compute-resources/</span><br><span class="hljs-comment">##</span><br><span class="hljs-attr">resources:</span><br> <span class="hljs-attr">limits:</span><br> <span class="hljs-attr">memory:</span> <span class="hljs-string">2048Mi</span><br> <span class="hljs-attr">cpu:</span> <span class="hljs-string">1000m</span><br> <span class="hljs-attr">requests:</span><br> <span class="hljs-attr">memory:</span> <span class="hljs-string">512Mi</span><br> <span class="hljs-attr">cpu:</span> <span class="hljs-string">200m</span><br><br><span class="hljs-attr">affinity:</span> {}<br><span class="hljs-attr">nodeSelector:</span> {}<br><span class="hljs-attr">tolerations:</span> []<br><span class="hljs-comment"># envVars:</span><br><span class="hljs-comment"># - name: RUNNER_EXECUTOR</span><br><span class="hljs-comment"># value: kubernetes</span><br><span class="hljs-attr">hostAliases:</span> []<br> <span class="hljs-comment"># Example:</span><br> <span class="hljs-comment"># - ip: "127.0.0.1"</span><br> <span class="hljs-comment"># hostnames:</span><br> <span class="hljs-comment"># - "foo.local"</span><br> <span class="hljs-comment"># - "bar.local"</span><br> <span class="hljs-comment"># - ip: "10.1.2.3"</span><br> <span class="hljs-comment"># hostnames:</span><br> <span class="hljs-comment"># - "foo.remote"</span><br><span class="hljs-comment"># - "bar.remote"</span><br><br><span class="hljs-attr">podAnnotations:</span> {}<br> <span class="hljs-comment"># Example:</span><br><span class="hljs-comment"># iam.amazonaws.com/role: <my_role_arn></span><br><span class="hljs-attr">podLabels:</span> {}<br> <span class="hljs-comment"># Example:</span><br> <span class="hljs-comment"># owner.team: <my_cool_team></span><br><br><span class="hljs-attr">priorityClassName:</span> <span class="hljs-string">""</span><br><br><span class="hljs-attr">secrets:</span> []<br> <span class="hljs-comment"># Example:</span><br> <span class="hljs-comment"># - name: my-secret</span><br> <span class="hljs-comment"># - name: myOtherSecret</span><br> <span class="hljs-comment"># items:</span><br> <span class="hljs-comment"># - key: key_one</span><br><span class="hljs-comment"># path: path_one</span><br><br><span class="hljs-attr">configMaps:</span> {}<br><br><span class="hljs-attr">volumeMounts:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">cache</span><br> <span class="hljs-attr">mountPath:</span> <span class="hljs-string">/cache</span><br><span class="hljs-attr">volumes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">cache</span><br> <span class="hljs-attr">persistentVolumeClaim:</span><br> <span class="hljs-attr">claimName:</span> <span class="hljs-string">gitlab-runner-cache-pvc</span><br></code></pre></td></tr></table></figure></li></ol>]]></content>
<categories>
<category> Kubernetes </category>
</categories>
<tags>
<tag> gitlab </tag>
<tag> ci </tag>
</tags>
</entry>
<entry>
<title>grafana第三方集成以及使用jwt(jwks)实现免登录访问</title>
<link href="//grafana-integration-jwt-jwks/"/>
<url>//grafana-integration-jwt-jwks/</url>
<content type="html"><![CDATA[<p>对于第三方集成grafana，需要可以通过链接直接访问管理界面，跳过登录页面。<br>传统方法是使用代理认证，而最新的jwt方法可以更好地实现这个需求。<br>不过网络上相关教程文档较少，缺乏详细说明，故此记录。</p><span id="more"></span><h2 id="一-控制台-面板-分享"><a href="#一-控制台-面板-分享" class="headerlink" title="一 控制台/面板-分享"></a>一 控制台/面板-分享</h2><p>grafana本身支持面板的内嵌分享，见 <a href="https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels">https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels</a><br>注意修改以下配置，这里以环境变量的写法表示：<br>domain需要填写外部访问的host<br>port这里也修改成非默认端口，建议直接修改端口号而不是反向代理形式来隐藏默认端口<br>下面两个是允许内嵌和允许匿名访问，是关键。</p><figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs ini"><span class="hljs-attr">GF_SERVER_DOMAIN</span>=<span class="hljs-string">"192.168.10.192"</span><br><span class="hljs-attr">GF_SERVER_HTTP_PORT</span>=<span class="hljs-number">33303</span><br><span class="hljs-attr">GF_SECURITY_ALLOW_EMBEDDING</span>=<span class="hljs-literal">true</span><br><span class="hljs-attr">GF_AUTH_ANONYMOUS_ENABLED</span>=<span class="hljs-literal">true</span><br><br></code></pre></td></tr></table></figure><h2 id="二-免登录访问"><a href="#二-免登录访问" class="headerlink" title="二 免登录访问"></a>二 免登录访问</h2><p>对于第三方集成grafana，需要可以通过链接直接访问管理界面，跳过登录页面。传统方法是使用代理认证，而最新的jwt方法可以更好地实现这个需求。</p><h3 id="1-认证代理方式资料"><a href="#1-认证代理方式资料" class="headerlink" title="1 认证代理方式资料"></a>1 认证代理方式资料</h3><p>传统的方法是使用认证代理（Auth Proxy），需和nginx、apache等集成，如：</p><ul><li>openresty： <a href="http://blog.helongfei.com/2019/%E4%BD%BF%E7%94%A8-openresty-%E5%AE%9E%E7%8E%B0-grafana-%E5%85%8D%E7%99%BB%E5%BD%95/">http://blog.helongfei.com/2019/%E4%BD%BF%E7%94%A8-openresty-%E5%AE%9E%E7%8E%B0-grafana-%E5%85%8D%E7%99%BB%E5%BD%95/</a></li><li>java代理程序：<a href="https://blog.51cto.com/u_15127641/4530608">https://blog.51cto.com/u_15127641/4530608</a></li><li>apache：<a href="https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/auth-proxy/">https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/auth-proxy/</a><h3 id="2-jwt方式资料"><a href="#2-jwt方式资料" class="headerlink" title="2 jwt方式资料"></a>2 jwt方式资料</h3></li><li><a href="https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/jwt/">https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/jwt/</a></li><li><a href="https://github.com/grafana/grafana-iframe-oauth-sample">https://github.com/grafana/grafana-iframe-oauth-sample</a></li><li><a href="https://www.cnblogs.com/yinxy/p/14893595.html">https://www.cnblogs.com/yinxy/p/14893595.html</a><h2 id="三-jwt免密访问步骤"><a href="#三-jwt免密访问步骤" class="headerlink" title="三 jwt免密访问步骤"></a>三 jwt免密访问步骤</h2><h3 id="1-获取jwks文件"><a href="#1-获取jwks文件" class="headerlink" title="1 获取jwks文件"></a>1 获取jwks文件</h3>访问 <a href="https://mkjwk.org/">https://mkjwk.org/</a> ，参考如下值，keyId自行填写<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1667786431218.png"><br>将中间部分（含public、private的）保存为jwks.json文件，用于生成token。<br>右边的public key保存为jwks-public.json，用于对token进行认证。注意外围也要加上<code>{"keys": [ ]}</code>，否则grafana会找不到keys。<h3 id="2-启动grafana"><a href="#2-启动grafana" class="headerlink" title="2 启动grafana"></a>2 启动grafana</h3>这里以容器方式启动，注意，以下配置适用于9.3.2及以后版本<br>注意这里使用的是jwks-public.json<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker run \<br> -p 33303:33303 \<br> --user root \<br> --name my-grafana \<br> --restart=always \<br> --env GF_SERVER_DOMAIN="localhost" \<br> --env GF_SERVER_HTTP_PORT=33303 \<br> --env GF_SECURITY_ADMIN_PASSWORD="123456" \<br> --env GF_SECURITY_ALLOW_EMBEDDING=true \<br> --env GF_AUTH_ANONYMOUS_ENABLED=true \<br> --env GF_USERS_DEFAULT_THEME=light \<br> --env GF_AUTH_JWT_ENABLED=true \<br> --env GF_AUTH_JWT_ENABLE_LOGIN_TOKEN=true \<br> --env GF_AUTH_JWT_HEADER_NAME=GF-JWT \<br> --env GF_AUTH_JWT_USERNAME_CLAIM=sub \<br> --env GF_AUTH_JWT_EMAIL_CLAIM=sub \<br> --env GF_AUTH_JWT_JWK_SET_FILE='/home/jwks.json' \<br> --env GF_AUTH_JWT_CACHE_TTL=60m \<br> --env GF_AUTH_JWT_ROLE_ATTRIBUTE_PATH=role \<br> --env GF_AUTH_JWT_AUTO_SIGN_UP=true \<br> --env GF_AUTH_JWT_URL_LOGIN=true \<br> --env GF_AUTH_JWT_ALLOW_ASSIGN_GRAFANA_ADMIN=true \<br> --env GF_LOG_LEVEL=info \<br> -v /tmp/grafana:/var/lib/grafana \<br> -v /tmp/jwks-public.json:/home/jwks.json \<br> -d grafana/grafana<br><br>docker logs -f my-grafana<br><br></code></pre></td></tr></table></figure>GF_AUTH_JWT_HEADER_NAME：token在header的名称，如果是在url中，则固定是auth_token<br>GF_AUTH_JWT_USERNAME_CLAIM：grafana将使用sub（即subject）字段作为用户名<br>GF_AUTH_JWT_ROLE_ATTRIBUTE_PATH：grafana将使用role字段作为用户角色<br>GF_AUTH_JWT_AUTO_SIGN_UP：新用户字段注册</li></ul><h3 id="3-生成token"><a href="#3-生成token" class="headerlink" title="3 生成token"></a>3 生成token</h3><p>如下，使用D:\tmp\jwks.json文件，生成用户名为user1，角色为Admin，有效期为1小时的token。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-keyword">package</span> jwk;<br><br><span class="hljs-keyword">import</span> cn.hutool.core.io.FileUtil;<br><span class="hljs-keyword">import</span> com.nimbusds.jose.*;<br><span class="hljs-keyword">import</span> com.nimbusds.jose.jwk.JWKSet;<br><span class="hljs-keyword">import</span> com.nimbusds.jose.jwk.source.ImmutableJWKSet;<br><span class="hljs-keyword">import</span> com.nimbusds.jose.mint.DefaultJWSMinter;<br><span class="hljs-keyword">import</span> com.nimbusds.jwt.JWTClaimsSet;<br><span class="hljs-keyword">import</span> org.junit.jupiter.api.Test;<br><br><span class="hljs-keyword">import</span> java.time.Instant;<br><span class="hljs-keyword">import</span> java.time.temporal.ChronoUnit;<br><span class="hljs-keyword">import</span> java.util.Date;<br><span class="hljs-keyword">import</span> java.util.Map;<br><br><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">JwtTest</span> </span>{<br><br> <span class="hljs-meta">@Test</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">test</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> Exception </span>{<br> System.out.println(getJwt());<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> String <span class="hljs-title">getJwt</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> Exception </span>{<br> JWKSet jwkSet = JWKSet.load(FileUtil.file(<span class="hljs-string">"D:\\tmp\\jwks.json"</span>));<br> DefaultJWSMinter minter = <span class="hljs-keyword">new</span> DefaultJWSMinter();<br> minter.setJWKSource(<span class="hljs-keyword">new</span> ImmutableJWKSet(jwkSet));<br><br> JWSHeader header = <span class="hljs-keyword">new</span> JWSHeader.Builder(JWSAlgorithm.RS256)<br> .type(JOSEObjectType.JWT)<br> .build();<br> <span class="hljs-comment">// Create JWT</span><br> Instant nowInstant = Instant.now();<br> JWTClaimsSet jwtClaims = <span class="hljs-keyword">new</span> JWTClaimsSet.Builder()<br> .subject(<span class="hljs-string">"user2"</span>)<br> .expirationTime(<span class="hljs-keyword">new</span> Date(nowInstant.plus(<span class="hljs-number">1</span>, ChronoUnit.HOURS).toEpochMilli()))<br> .notBeforeTime(<span class="hljs-keyword">new</span> Date(nowInstant.toEpochMilli()))<br> .issueTime(<span class="hljs-keyword">new</span> Date(nowInstant.toEpochMilli()))<br> .build();<br> Payload payload = jwtClaims.toPayload();<br> Map<String, Object> payloadMap = payload.toJSONObject();<br> payloadMap.put(<span class="hljs-string">"role"</span>, <span class="hljs-string">"Admin"</span>);<br> JWSObject jwsObject = minter.mint(header, <span class="hljs-keyword">new</span> Payload(payloadMap), <span class="hljs-keyword">null</span>);<br> <span class="hljs-keyword">return</span> jwsObject.serialize();<br> }<br> <br>}<br><br></code></pre></td></tr></table></figure><p>生成的token可以在<a href="https://jwt.io/%E8%BF%9B%E8%A1%8C%E9%AA%8C%E8%AF%81%EF%BC%8C%E5%A6%82%E4%B8%8B">https://jwt.io/进行验证，如下</a><br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1667789545940.png"></p><h3 id="4-使用token进行访问"><a href="#4-使用token进行访问" class="headerlink" title="4 使用token进行访问"></a>4 使用token进行访问</h3><ol><li>url-token访问<br><a href="http://grafana:33303/datasources?auth_token=YOUR_TOKEN">http://grafana:33303/datasources?auth_token=YOUR_TOKEN</a></li><li>header访问<br>headers携带GF-JWT的token，内容为：<code>Bearer YOUR_TOKEN</code>，注意中间有空格</li></ol>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> grafana </tag>
</tags>
</entry>
<entry>
<title>docker部署nfs服务器v3v4</title>
<link href="//docker-nfs-server-v3-v4/"/>
<url>//docker-nfs-server-v3-v4/</url>
<content type="html"><![CDATA[<p>虽然nfs-server的docker镜像有很多，但总有这样或那样的问题，容易踩坑。<br>这里分享一下自己的使用记录。</p><span id="more"></span><h3 id="v3"><a href="#v3" class="headerlink" title="v3"></a>v3</h3><p>使用的镜像是：erichough/nfs-server<br>相关链接见：<a href="https://hub.docker.com/r/erichough/nfs-server">https://hub.docker.com/r/erichough/nfs-server</a> 、 <a href="https://github.com/ehough/docker-nfs-server">https://github.com/ehough/docker-nfs-server</a><br>本身说是支持v4的，但截至2020.9月使用v4会有问题，见：<a href="https://github.com/ehough/docker-nfs-server/issues/58">https://github.com/ehough/docker-nfs-server/issues/58</a><br>使用v3是可以的，但由于v3占用了111端口，所以运行该容器的服务器没办法用nfs-client（rpc-bind用111端口）<br>而且除了2049，其他的端口没得修改，所以不推荐</p><blockquote><p>NFSv3使用大量辅助协议，客户访问过程首先需要通过portmap/rpcbind获取rpc.mounted监听端口，<br>然后nfs客户端访问rpc.mounted，nfs服务器根据/etc/exports文件进行客户身份验证，<br>验证通过后nfs客户端才能与rpc.nfsd建立联系并访问共享。<br>客户端与服务器数据交互过程的配额管理，文件锁管理以及nfs协议数据统计过程都由单独rpc进程来完成。<br>所有这些进程除了portmap和nfsd之外都是监听动态随机端口。<br>NFSv4自身集成辅助协议，只需要TCP 2049一个端口即可，这样极大方便NFS在防火墙后环境中部署。</p></blockquote><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /opt/nfs-server/share<br>docker rm -f nfs-server<br>docker run \<br> -d \<br> --name nfs-server \<br> --restart=always \<br> --privileged \<br> -e NFS_VERSION=3 \<br> -p 2049:2049 -p 2049:2049/udp \<br> -p 111:111 -p 111:111/udp \<br> -p 32765:32765 -p 32765:32765/udp \<br> -p 32767:32767 -p 32767:32767/udp \<br> -v /opt/nfs-server/share:/nfs-share \<br> -e NFS_EXPORT_0='/nfs-share *(rw,sync,no_root_squash,all_squash,anonuid=0,anongid=0,no_subtree_check)' \<br> erichough/nfs-server<br>docker logs -f nfs-server<br><br></code></pre></td></tr></table></figure><p>客户端操作：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /data/nfs-share<br><span class="hljs-meta">#</span><span class="bash"> 挂载</span><br>mount -v -t nfs -o port=2049 主机:/nfs-share /data/nfs-share<br><span class="hljs-meta">#</span><span class="bash"> 卸载</span><br>umount /data/nfs-share<br><br></code></pre></td></tr></table></figure><h3 id="v4"><a href="#v4" class="headerlink" title="v4"></a>v4</h3><p>使用的镜像是：<a href="https://github.com/sjiveson/nfs-server-alpine">itsthenetwork/nfs-server-alpine</a> 或 <a href="https://hub.docker.com/r/gists/nfs-server">gists/nfs-server</a></p><p>建议用gists/nfs-server，支持NFS_OPTION，配置更加清晰</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /opt/nfs-server/share<br>docker rm -f nfs-server<br><br>docker run \<br> -d \<br> --name nfs-server \<br> --restart=always \<br> --privileged \<br> -p 32049:2049 \<br> -v /opt/nfs-server/share:/nfsshare \<br> -e NFS_DIR=/nfsshare \<br> -e NFS_DOMAIN=* \<br> -e NFS_OPTION="fsid=0,rw,sync,no_root_squash,all_squash,anonuid=0,anongid=0,no_subtree_check" \<br> gists/nfs-server<br> <br>docker logs -f nfs-server<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> docker run \</span><br><span class="bash"><span class="hljs-comment"># -d \</span></span><br><span class="bash"><span class="hljs-comment"># --name nfs-server \</span></span><br><span class="bash"><span class="hljs-comment"># --restart=always \</span></span><br><span class="bash"><span class="hljs-comment"># --privileged \</span></span><br><span class="bash"><span class="hljs-comment"># -p 32049:2049 \</span></span><br><span class="bash"><span class="hljs-comment"># -v /opt/nfs-server/share:/nfsshare \</span></span><br><span class="bash"><span class="hljs-comment"># -e SHARED_DIRECTORY=/nfsshare \</span></span><br><span class="bash"><span class="hljs-comment"># -e SYNC=true \</span></span><br><span class="bash"><span class="hljs-comment"># itsthenetwork/nfs-server-alpine</span></span><br> <br><br><br><br></code></pre></td></tr></table></figure><p>客户端操作：<br>注意这里用的是 <strong>/</strong> ,而非 <strong>/nfsshare</strong> </p><blockquote><p>NFSv4将所有共享使用一个虚拟文件系统展示给客户端。<br>伪文件系统根目录（/）使用fsid=0标示，只有一个共享可以是fsid=0。<br>客户端需要使用“nfs server ip:/”挂载伪文件系统，<br>伪文件系统一般使用RO方式共享， 其他共享可以通过mount –bind选项在伪文件系统目录下挂载。<br>客户端挂载过程需要通过mount –t nfs4指定NFS版本为4，默认采用nfsv3</p></blockquote><p>现在客户端挂载一般会优先尝试nfsv4,再尝试nfsv3</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /data/nfs-share<br><span class="hljs-meta">#</span><span class="bash"> 挂载</span><br>mount -v -t nfs -o port=32049 主机:/ /data/nfs-share<br><span class="hljs-meta">#</span><span class="bash"> 卸载</span><br>umount /data/nfs-share<br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 运维部署 </category>
</categories>
<tags>
<tag> docker </tag>
<tag> nfs-server </tag>
</tags>
</entry>
<entry>
<title>我的服务器系列：tailscale使用自定义derper服务器（docker部署）</title>
<link href="//tailscale-derper-docker/"/>
<url>//tailscale-derper-docker/</url>
<content type="html"><![CDATA[<p>使用docker部署tailscale私有中继服务器，并使用nginx四层代理与原有服务共用443端口，最后配置使用并检测。<br>derper-docker部署及使用较麻烦，本文分享踩坑记录及经验，建议阅读。</p><p>2023.10更新：随着网站备案注销，derper已切换到ip模式，增加对应示例及说明</p><span id="more"></span><h2 id="一-设计及参考"><a href="#一-设计及参考" class="headerlink" title="一 设计及参考"></a>一 设计及参考</h2><h3 id="设计"><a href="#设计" class="headerlink" title="设计"></a>设计</h3><ol><li>使用DockerHub上面唯一主流的derper镜像进行部署，保证可靠性</li><li>使用nginx四层代理转发443端口请求到derper容器</li><li>在tailscale上面配置使用derper服务器</li><li>测试derper的连通<h3 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h3>推荐阅读：<br>官方说明（ <a href="https://tailscale.com/kb/1118/custom-derp-servers/">https://tailscale.com/kb/1118/custom-derp-servers/</a> ）<br>TAILSCALE 的一些使用心得（ <a href="https://leitalk.com/12245">https://leitalk.com/12245</a> ）<br>Tailscale 基础教程：部署私有 DERP 中继服务器（ <a href="https://icloudnative.io/posts/custom-derp-servers">https://icloudnative.io/posts/custom-derp-servers</a> ）</li></ol><h2 id="二-部署derper容器（已备案域名）"><a href="#二-部署derper容器（已备案域名）" class="headerlink" title="二 部署derper容器（已备案域名）"></a>二 部署derper容器（已备案域名）</h2><p>参考：<a href="https://hub.docker.com/r/fredliang/derper">https://hub.docker.com/r/fredliang/derper</a><br>下面是我个人的配置<br>注意：</p><ol><li>ssl证书<br>ssl证书可外部挂载或由容器自动申请并维护证书更新（基于LetsEncrypt）<br>因为derper使用的ssl证书有格式和命名要求，挂载使用很不优雅，故放弃该方案<br>而自动申请需要使用443端口，因为我443端口已经使用了，所以需要用nginx stream反向代理的形式做443端口4层转发。<br>后文会进行详细说明。</li><li>3478端口不能修改，因为走udp所以也不建议转发。</li><li>如果使用DERP_VERIFY_CLIENTS则需要挂载tailscaled.sock，使容器能访问到外部机器derper进程。<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /data/derper/certs<br>docker rm -f derper<br><br>docker run -d \<br>-p 3443:443 \<br>-p 3478:3478/udp \<br>--name derper \<br>--restart=always \<br>-v /data/derper/certs:/app/certs \<br>-v /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock \<br>-e DERP_ADDR=":443" \<br>-e DERP_VERIFY_CLIENTS=true \<br>-e DERP_DOMAIN="derper.linshenkx.cn" \<br>fredliang/derper<br><br>docker logs -f derper<br><br></code></pre></td></tr></table></figure><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1652431206.png" alt="derper服务器"></li></ol><h2 id="二-部署derper容器（ip）"><a href="#二-部署derper容器（ip）" class="headerlink" title="二 部署derper容器（ip）"></a>二 部署derper容器（ip）</h2><p>参考 <a href="https://github.com/yangchuansheng/ip_derper">https://github.com/yangchuansheng/ip_derper</a></p><p>另外，反正直接走ip了，直接端口也换掉，这里用33380<br>因为不走域名，也不需要nginx来根据域名分流了，直接把端口暴露出来就好</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker rm -f derper<br><br>docker run -d \<br>-p 3478:3478/udp \<br>-p 33380:33380 \<br>--name derper \<br>--restart=always \<br>-v /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock \<br>-e DERP_ADDR=":33380" \<br>-e DERP_VERIFY_CLIENTS=true \<br>ghcr.io/yangchuansheng/ip_derper<br><br>docker logs -f derper<br><br></code></pre></td></tr></table></figure><h2 id="三-nginx代理配置（域名方式需要）"><a href="#三-nginx代理配置（域名方式需要）" class="headerlink" title="三 nginx代理配置（域名方式需要）"></a>三 nginx代理配置（域名方式需要）</h2><p>虽然有些人说derper运行一段时间就会崩溃（<a href="https://github.com/tailscale/tailscale/issues/4082">Derper TLS handshake error: remote error: tls: internal error </a>），<br>但我没有遇到过，按照<a href="https://github.com/tailscale/tailscale/issues/4082">官方issue</a>和<a href="https://leitalk.com/12245">TAILSCALE 的一些使用心得</a>里的说法，可能和墙/备案有关。</p><blockquote><p>顺带一提，如腾讯云这样的国内云服务器提供商，会审查 TLS handshake 里的 SNI 信息， 如果发现 SNI 域名未备案，会阻断 TLS 握手。 所以，我曾经也尝试过用一个境外服务器的域名，然后指向到境内服务器 IP 的形式尝试绕过备案， 然而短暂的用了几分钟后，就遇到了 tls handshake reset 的问题。 目前看来，只要你想用境内服务器，那么备案就是绕不过的问题。<br>Ps. 如果你有备案域名，那么可以用 nginx stream 反向代理的形式做 443 端口 4 层转发。 nginx stream 可以在四层探测 SNI 信息，然后分发到不同的后端，这样你 derper 的 443 和其他域名的 443 就可以共存在同一个服务器上了。</p></blockquote><p>这里主要知识点是：</p><ol><li>TLS handshake 里的 SNI 信息</li><li>nginx四层负载均衡</li></ol><p>具体配置参考如下：<br>在nginx.conf添加如下配置</p><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><code class="hljs nginx"><span class="hljs-section">stream</span> {<br> <span class="hljs-comment"># 这里就是 SNI 识别，将域名映射成一个配置名</span><br> <span class="hljs-attribute">map</span> $ssl_preread_server_name $backend_name {<br> <span class="hljs-comment"># 把derper.linshenkx.cn的流量转到derper的upstream</span><br> derper.linshenkx.<span class="hljs-attribute">cn</span> derper;<br> <span class="hljs-comment"># 域名都不匹配情况下的默认值</span><br> <span class="hljs-attribute">default</span> https_web;<br> }<br> <span class="hljs-comment"># 监听 443 并开启 ssl_preread</span><br> <span class="hljs-section">server</span> {<br> <span class="hljs-attribute">listen</span> <span class="hljs-number">443</span> reuseport;<br> <span class="hljs-attribute">listen</span> [::]:<span class="hljs-number">443</span> reuseport;<br> <span class="hljs-attribute">proxy_pass</span> $backend_name;<br> <span class="hljs-attribute">ssl_preread</span> <span class="hljs-literal">on</span>;<br> }<br> <span class="hljs-attribute">upstream</span> derper {<br> <span class="hljs-attribute">server</span> <span class="hljs-number">127.0.0.1:3443</span>;<br> }<br> <span class="hljs-attribute">upstream</span> https_web {<br> <span class="hljs-attribute">server</span> <span class="hljs-number">127.0.0.1:12443</span>;<br> }<br>}<br></code></pre></td></tr></table></figure><p>然后再把default.conf里server的443换成12443，如下</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><code class="hljs shell">upstream xxx {<br> server 127.0.0.1:12345;<br>}<br><span class="hljs-meta">#</span><span class="bash"> redirect all traffic to https</span><br>server {<br> listen 80 default_server;<br> listen [::]:80 default_server;<br> server_name _;<br> return 301 https://$host$request_uri;<br>}<br>server {<br> listen 12443 ssl http2 ;<br> listen [::]:12443 ssl http2 ;<br><br> server_name xxx.linshenkx.cn;<br><br> include /config/nginx/proxy-confs/*.subfolder.conf;<br><br> include /config/nginx/ssl.conf;<br><br> client_max_body_size 0;<br><br> location / {<br> proxy_pass http://xxx;<br> proxy_set_header HOST $host;<br> proxy_set_header X-Forwarded-Proto $scheme;<br> proxy_set_header X-Real-IP $remote_addr;<br> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br><br> }<br><br>}<br><br></code></pre></td></tr></table></figure><p>这样访问 <a href="https://derper.linshenkx.cn/">https://derper.linshenkx.cn</a> 的流程是 :</p><ol><li>443端口根据nginx.conf转到本地的3443</li><li>由docker映射到derper容器内的443端口</li></ol><p>而访问 <a href="https://xxx.linshenkx.cn/">https://xxx.linshenkx.cn</a> 的流程是 :</p><ol><li>443端口根据nginx.conf转到本地的12443</li><li>12443端口根据nginx的default.conf转发到本地的12345</li><li>12345是应用的暴露端口/docker的映射端口</li></ol><p>对于大部分的http应用，可以通过xxx的方式配置访问，自动套上https，并且避免泄露端口。<br>个别更底层协议的应用，如derper、gitlfs，则需在nginx.conf配置转发 </p><h2 id="四-tailscale配置"><a href="#四-tailscale配置" class="headerlink" title="四 tailscale配置"></a>四 tailscale配置</h2><p>Access Contros配置内容参考如下：<br>主要是添加了derpMap，其他都保持默认。<br>如果有多个derper服务器，建议配置为多个region而非node。<br>因为延迟的比较是以region为单位的，会方便测试。</p><p>一开始不确定derper是否运行正常的时候，建议先把OmitDefaultRegions设置为true，关闭默认的region。</p><p>另外，修改Access Contros后，需要重启tailscale服务才能生效！</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><code class="hljs shell">// Example/default ACLs for unrestricted connections.<br>{<br> // Declare static groups of users beyond those in the identity service.<br> "groups": {<br> "group:example": [ "user1@example.com", "user2@example.com" ],<br> },<br> // Declare convenient hostname aliases to use in place of IP addresses.<br> "hosts": {<br> "example-host-1": "100.100.100.100",<br> },<br> // Access control lists.<br> "acls": [<br> // Match absolutely everything. Comment out this section if you want<br> // to define specific ACL restrictions.<br> { "action": "accept", "users": ["*"], "ports": ["*:*"] },<br> ]<br> ,<br> "derpMap": {<br> "OmitDefaultRegions": false<br> ,<br> "Regions": {<br> "900": {<br> "RegionID": 900,<br> "RegionCode": "lian",<br> "RegionName": "LIAN",<br> "Nodes": [{<br> "Name": "tx",<br> "RegionID": 900,<br> "HostName": "derper.linshenkx.cn",<br> "DERPPort": 443<br> }<br> ]<br> }<br> }<br> }<br>}<br></code></pre></td></tr></table></figure><p>如果是ip方式，则使用如下：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><code class="hljs shell">// Example/default ACLs for unrestricted connections.<br>{<br>// Access control lists.<br>"acls": [<br>{"action": "accept", "src": ["*"], "dst": ["*:*"]},<br>// Match absolutely everything. Comment out this section if you want<br>// to define specific ACL restrictions.<br>],<br>"derpMap": {<br>"OmitDefaultRegions": false,<br>"Regions": {<br>"900": {<br>"RegionID": 900,<br>"RegionCode": "lian",<br>"RegionName": "LIAN",<br>"Nodes": [<br>{<br>"Name": "tx",<br>"RegionID": 900,<br>"HostName": "你的ip",<br>"IPv4": "你的ip",<br>"DERPPort": 33380,<br># 这里要去掉检测<br>"InsecureForTests": true,<br>},<br>],<br>},<br>},<br>},<br>}<br><br><br></code></pre></td></tr></table></figure><h2 id="五-测试使用derper"><a href="#五-测试使用derper" class="headerlink" title="五 测试使用derper"></a>五 测试使用derper</h2><p>相关命令</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 显示集群状态</span><br>tailscale status<br><span class="hljs-meta">#</span><span class="bash"> 显示网络状态</span><br>tailscale netcheck<br>tailscale ping 节点<br><br></code></pre></td></tr></table></figure><p>如下图，自定义的derper比默认的延迟要低不上。<br>（不过只有在极端情况下流量才会走derper，通常derper只要连得上就行，所以延迟影响不大）<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1652430906(1).png" alt="tailscale netcheck"></p><p>需要注意的是，这里显示一切正常，不代表derper就是在正常工作了。<br>只是说能访问到 <a href="https://derper.linshenkx.cn/">https://derper.linshenkx.cn</a> 而已。</p><p>关键还是要看ping。<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1652431460(1).png" alt="tailscale ping"><br>如图，代表了几种情况：</p><ol><li>tx.linshenkx.cn<br>先通过DERP(sfo)连接，然后打洞成功，进化为ip:端口直连</li><li>uc.linshenkx.cn<br>识别到是本机</li><li>lian.linshenkx.cn<br>尝试通过DERP(lian)连接，但失败（因为对方tailscale服务异常）</li><li>nas.linshenkx.cn<br>先通过DERP(lian)连接，然后打洞成功，进化为ip:端口直连</li></ol><p>derper日志类似如下<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1652431977(1).png" alt="docker logs -f derper"></p>]]></content>
<categories>
<category> 我的服务器系列 </category>
</categories>
<tags>
<tag> tailscale </tag>
<tag> derper </tag>
<tag> docker </tag>
<tag> 我的服务器系列 </tag>
</tags>
</entry>
<entry>
<title>ubuntu获取离线安装包</title>
<link href="//ubuntu-offline-deb-package/"/>
<url>//ubuntu-offline-deb-package/</url>
<content type="html"><![CDATA[<p>工作经常会遇到需要离线安装软件的情况，特此记录下来，避免每次都得去翻笔记。</p><span id="more"></span><h2 id="一-构造运行环境"><a href="#一-构造运行环境" class="headerlink" title="一 构造运行环境"></a>一 构造运行环境</h2><h3 id="1-使用docker提供目标操作系统"><a href="#1-使用docker提供目标操作系统" class="headerlink" title="1 使用docker提供目标操作系统"></a>1 使用docker提供目标操作系统</h3><p>在本地创建data目录并挂载到容器里，方便将生成的依赖取出。</p><ol><li>centos7<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /home/lin/ubuntu22/data<br>docker rm -f ubuntu22<br>docker run -d --name ubuntu22 -v /home/lin/ubuntu22/data:/data ubuntu:22.04 tail -f /dev/null<br>docker exec -it ubuntu22 /bin/bash<br><br></code></pre></td></tr></table></figure></li></ol><h3 id="2-更新yum镜像仓库"><a href="#2-更新yum镜像仓库" class="headerlink" title="2 更新yum镜像仓库"></a>2 更新yum镜像仓库</h3><p>参考阿里的开源镜像站：<a href="https://developer.aliyun.com/mirror/">https://developer.aliyun.com/mirror/</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">cp -a /etc/apt/sources.list /etc/apt/sources.list.bak \<br> && sed -i "s@http://.*archive.ubuntu.com@http://repo.huaweicloud.com@g" /etc/apt/sources.list \<br> && sed -i "s@http://.*security.ubuntu.com@http://repo.huaweicloud.com@g" /etc/apt/sources.list \<br> && apt-get update<br><br></code></pre></td></tr></table></figure><h2 id="二-下载依赖包"><a href="#二-下载依赖包" class="headerlink" title="二 下载依赖包"></a>二 下载依赖包</h2><p>以nfs-common为例</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt-get clean<br>mkdir /data/nfs<br>apt-get install --reinstall -d -y nfs-common<br>mv /var/cache/apt/archives/*.deb /data/nfs/<br><br></code></pre></td></tr></table></figure><h2 id="三-安装"><a href="#三-安装" class="headerlink" title="三 安装"></a>三 安装</h2><p>将生成的文件夹拷贝到目标集群上</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">dpkg -i *.deb<br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 运维部署 </category>
</categories>
<tags>
<tag> ubuntu </tag>
</tags>
</entry>
<entry>
<title>PySpark自定义Transformer</title>
<link href="//pyspark-transformer/"/>
<url>//pyspark-transformer/</url>
<content type="html"><![CDATA[<p>使用python实现自定义Transformer以对pyspark的pipeline进行增强</p><span id="more"></span><h3 id="一-示例"><a href="#一-示例" class="headerlink" title="一 示例"></a>一 示例</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> pyspark <span class="hljs-keyword">import</span> keyword_only<br><span class="hljs-keyword">from</span> pyspark.ml <span class="hljs-keyword">import</span> Transformer<br><span class="hljs-keyword">from</span> pyspark.ml.param.shared <span class="hljs-keyword">import</span> HasOutputCols, Param, Params<br><span class="hljs-keyword">from</span> pyspark.ml.util <span class="hljs-keyword">import</span> DefaultParamsReadable, DefaultParamsWritable<br><br><span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">SelectColsTransformer</span>(<span class="hljs-params"></span></span><br><span class="hljs-params"><span class="hljs-class"> Transformer, DefaultParamsReadable, DefaultParamsWritable</span></span><br><span class="hljs-params"><span class="hljs-class"></span>):</span><br> cols = Param(<br> Params._dummy(),<br> <span class="hljs-string">"cols"</span>,<br> <span class="hljs-string">"cols to select"</span><br> )<br><br><span class="hljs-meta"> @keyword_only</span><br> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">__init__</span>(<span class="hljs-params">self, cols:<span class="hljs-built_in">list</span>[<span class="hljs-built_in">float</span>]=[<span class="hljs-string">'*'</span>]</span>):</span><br> <span class="hljs-built_in">super</span>(SelectColsTransformer, self).__init__()<br> self._setDefault(cols=[<span class="hljs-string">'*'</span>])<br> kwargs = self._input_kwargs<br> self.setParams(**kwargs)<br><br><br><span class="hljs-meta"> @keyword_only</span><br> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">setParams</span>(<span class="hljs-params">self, cols:<span class="hljs-built_in">list</span>[<span class="hljs-built_in">float</span>]=[<span class="hljs-string">'*'</span>]</span>):</span><br> kwargs = self._input_kwargs<br> <span class="hljs-keyword">return</span> self._<span class="hljs-built_in">set</span>(**kwargs)<br><br><br> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">_transform</span>(<span class="hljs-params">self, dataset</span>):</span><br> <span class="hljs-keyword">return</span> dataset.select(self.getOrDefault(self.cols));<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 机器学习 </category>
</categories>
<tags>
<tag> pyspark </tag>
<tag> transformer </tag>
</tags>
</entry>
<entry>
<title>PySpark学习笔记</title>
<link href="//pyspark-notes/"/>
<url>//pyspark-notes/</url>
<content type="html"><![CDATA[<p>pyspark使用笔记，含1.jupyter-docker环境搭建 2.dataframe2jdbc,jdbc2dataframe 3.模型训练及保存 4.模型导入及使用 等。</p><span id="more"></span><h3 id="一-环境部署"><a href="#一-环境部署" class="headerlink" title="一 环境部署"></a>一 环境部署</h3><p>执行以下命令，访问 ip:18804即可，密码为：my-password<br>镜像使用参考：<a href="https://jupyter-docker-stacks.readthedocs.io/en/latest">https://jupyter-docker-stacks.readthedocs.io/en/latest</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /data/pyspark && chmod 777 -R /data/pyspark<br><br>docker run -d \<br>--name my-spark-notebook \<br>--restart=always \<br>-e GRANT_SUDO=yes \<br>-v /data/pyspark:/home/jovyan \<br>--network host \<br>jupyter/all-spark-notebook \<br>start.sh jupyter lab \<br>--NotebookApp.password='sha1:7cca89c48283:e3c1f9fbc06d1d2aa59555dfd5beed925e30dd2c'<br><br></code></pre></td></tr></table></figure><h3 id="二-dataframe导出数据库"><a href="#二-dataframe导出数据库" class="headerlink" title="二 dataframe导出数据库"></a>二 dataframe导出数据库</h3><p>参考：</p><ol><li><a href="https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/from_to_dbms.html">https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/from_to_dbms.html</a></li><li><a href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameWriter.jdbc.html?highlight=dataframewriter">https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameWriter.jdbc.html?highlight=dataframewriter</a></li></ol><p>在jupyter打开的终端界面中执行以下命令，下载mysql驱动包</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">curl -O https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar<br></code></pre></td></tr></table></figure><p>如下，</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> pyspark.sql <span class="hljs-keyword">import</span> SparkSession<br><span class="hljs-keyword">import</span> os<br><span class="hljs-keyword">from</span> pyspark <span class="hljs-keyword">import</span> SparkContext<br><span class="hljs-keyword">from</span> pyspark.sql <span class="hljs-keyword">import</span> SQLContext<br><span class="hljs-keyword">from</span> pyspark.ml.feature <span class="hljs-keyword">import</span> VectorAssembler<br><span class="hljs-keyword">from</span> pyspark.ml.regression <span class="hljs-keyword">import</span> LinearRegression<br><span class="hljs-keyword">from</span> sklearn.datasets <span class="hljs-keyword">import</span> load_boston<br><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd <br><span class="hljs-keyword">from</span> pyspark.ml <span class="hljs-keyword">import</span> Pipeline<br><br>spark = SparkSession.builder.master(<span class="hljs-string">'local'</span>) \<br>.appName(<span class="hljs-string">"pyspark-lian"</span>) \<br>.config(<span class="hljs-string">"spark.jars"</span>,<span class="hljs-string">"{}/mysql-connector-java-8.0.28.jar"</span>.<span class="hljs-built_in">format</span>(os.getcwd())) \<br>.config(<span class="hljs-string">"spark.driver.extraClassPath"</span>,<span class="hljs-string">"{}/mysql-connector-java-8.0.28.jar"</span>.<span class="hljs-built_in">format</span>(os.getcwd())) \<br>.getOrCreate()<br>sc = spark.sparkContext<br><br>boston = load_boston()<br>df_boston = pd.DataFrame(boston.data,columns=boston.feature_names)<br>df_boston[<span class="hljs-string">'target'</span>] = pd.Series(boston.target)<br><span class="hljs-built_in">print</span>(df_boston.head())<br><br>sqlContext = SQLContext(sc)<br><br>data = sqlContext.createDataFrame(df_boston)<br><span class="hljs-built_in">print</span>(<span class="hljs-built_in">type</span>(data))<br><span class="hljs-built_in">print</span>(data.printSchema())<br><br>data.write.jdbc(table=<span class="hljs-string">'df_boston'</span>,mode=<span class="hljs-string">'overwrite'</span>,url=<span class="hljs-string">'jdbc:mysql://localhost:3306/test?createDatabaseIfNotExist=true&useSSL=false'</span>,properties={<span class="hljs-string">'user'</span>:<span class="hljs-string">'root'</span>,<span class="hljs-string">'password'</span>:<span class="hljs-string">'123456'</span>})<br><br></code></pre></td></tr></table></figure><h3 id="三-读取mysql数据完成线性回归并导出模型文件"><a href="#三-读取mysql数据完成线性回归并导出模型文件" class="headerlink" title="三 读取mysql数据完成线性回归并导出模型文件"></a>三 读取mysql数据完成线性回归并导出模型文件</h3><p>参考：<a href="https://www.datatechnotes.com/2021/05/mllib-linear-regression-example-with.html">https://www.datatechnotes.com/2021/05/mllib-linear-regression-example-with.html</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br></pre></td><td class="code"><pre><code class="hljs shell">from pyspark.sql import SparkSession<br>import pyspark.pandas as ps<br>import os<br>from pyspark import SparkContext<br>from pyspark.sql import SQLContext<br>from pyspark.ml.feature import VectorAssembler<br>from pyspark.ml.regression import LinearRegression<br>import matplotlib.pyplot as plt<br>from sklearn.datasets import load_boston<br>import pandas as pd <br><br>spark = SparkSession.builder.master('local') \<br>.appName("pyspark-lian") \<br>.config("spark.jars","{}/mysql-connector-java-8.0.28.jar".format(os.getcwd())) \<br>.config("spark.driver.extraClassPath","{}/mysql-connector-java-8.0.28.jar".format(os.getcwd())) \<br>.getOrCreate()<br>sc = spark.sparkContext<br><br>df_boston = ps.read_sql('df_boston', 'jdbc:mysql://localhost:3306/test?createDatabaseIfNotExist=true&useSSL=false&user=root&password=123456')<br>print(type(df_boston))<br>print(df_boston.head())<br>data = df_boston.to_spark()<br>print(type(data))<br>print(data.printSchema()) <br> <br>columns=data.columns<br>print(columns)<br>columns.remove('target')<br>print(columns)<br><br>va = VectorAssembler(inputCols=columns, outputCol='features')<br>print(data.count())<br>va_df = va.transform(data.limit(200))<br>va_df = va_df.select(['features', 'target'])<br>va_df.show(3)<br> <br>lr=LinearRegression(featuresCol='features', labelCol='target',<br> regParam=0.3, elasticNetParam=0.8)<br><br>lr_model = lr.fit(va_df)<br>print(type(lr_model))<br>print("Coefficients: ", lr_model.coefficients)<br>print("Intercept: ", lr_model.intercept)<br><br>print("MSE: ", lr_model.summary.meanSquaredError)<br>print("MAE: ", lr_model.summary.meanAbsoluteError)<br>print("R-squared: ", lr_model.summary.r2) <br> <br>lr_model.save('LinearRegressionModel_model')<br><br>mdata = lr_model.transform(va_df)<br>mdata.show(3) <br> <br>x_ax = range(0, mdata.count())<br>y_pred = mdata.select("prediction").collect()<br>y_orig = mdata.select("target").collect() <br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> Stop session</span> <br>sc.stop() <br><br></code></pre></td></tr></table></figure><h3 id="四-加载模型并进行预测"><a href="#四-加载模型并进行预测" class="headerlink" title="四 加载模型并进行预测"></a>四 加载模型并进行预测</h3><p>注意 LinearRegressionModel 和 LinearRegression 的关系。<br>在上一步 LinearRegression fit之后产生LinearRegressionModel并保存为文件。<br>现在再使用 LinearRegressionModel的load方法从文件重新加载回来进行 transform 。</p><p>参考pipeline的概念的话：<br>LinearRegression 是 Estimator<br>而 LinearRegressionModel 是 Transformer</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> pyspark.sql <span class="hljs-keyword">import</span> SparkSession<br><span class="hljs-keyword">import</span> pyspark.pandas <span class="hljs-keyword">as</span> ps<br><span class="hljs-keyword">import</span> os<br><span class="hljs-keyword">from</span> pyspark <span class="hljs-keyword">import</span> SparkContext<br><span class="hljs-keyword">from</span> pyspark.sql <span class="hljs-keyword">import</span> SQLContext<br><span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt<br><span class="hljs-keyword">from</span> sklearn.datasets <span class="hljs-keyword">import</span> load_boston<br><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd<br><span class="hljs-keyword">import</span> pyspark.ml <span class="hljs-keyword">as</span> ml<br><span class="hljs-keyword">import</span> pyspark <span class="hljs-keyword">as</span> pyspark<br><span class="hljs-keyword">import</span> sys<br><br>spark = SparkSession.builder.master(<span class="hljs-string">'local'</span>) \<br>.appName(<span class="hljs-string">"pyspark-lian"</span>) \<br>.config(<span class="hljs-string">"spark.jars"</span>,<span class="hljs-string">"{}/mysql-connector-java-8.0.28.jar"</span>.<span class="hljs-built_in">format</span>(os.getcwd())) \<br>.config(<span class="hljs-string">"spark.driver.extraClassPath"</span>,<span class="hljs-string">"{}/mysql-connector-java-8.0.28.jar"</span>.<span class="hljs-built_in">format</span>(os.getcwd())) \<br>.getOrCreate()<br>sc = spark.sparkContext<br><br>result_mysqlSource = ps.read_sql(<span class="hljs-string">'df_boston'</span>, <span class="hljs-string">'jdbc:mysql://localhost:3306/test?createDatabaseIfNotExist=true&useSSL=false&user=root&password=123456'</span>).to_spark()<br>result_transition0=result_mysqlSource<br><br>va = pyspark.ml.feature.VectorAssembler()<br>va.setParams(inputCols=[<span class="hljs-string">'CRIM'</span>,<span class="hljs-string">'ZN'</span>,<span class="hljs-string">'INDUS'</span>,<span class="hljs-string">'CHAS'</span>,<span class="hljs-string">'NOX'</span>,<span class="hljs-string">'RM'</span>,<span class="hljs-string">'AGE'</span>,<span class="hljs-string">'DIS'</span>,<span class="hljs-string">'RAD'</span>,<span class="hljs-string">'TAX'</span>,<span class="hljs-string">'PTRATIO'</span>,<span class="hljs-string">'B'</span>,<span class="hljs-string">'LSTAT'</span>])<br>va.setParams(outputCol=<span class="hljs-string">'features'</span>)<br>result_va=va.transform(result_transition0)<br><br>lr_model = pyspark.ml.regression.LinearRegressionModel.load(<span class="hljs-string">'file:///home/jovyan/LinearRegressionModel_model_2'</span>)<br>result_lr_model=lr_model.transform(result_va)<br><br>result_lr_model.select([<span class="hljs-string">'CRIM'</span>,<span class="hljs-string">'ZN'</span>,<span class="hljs-string">'INDUS'</span>,<span class="hljs-string">'CHAS'</span>,<span class="hljs-string">'NOX'</span>,<span class="hljs-string">'RM'</span>,<span class="hljs-string">'AGE'</span>,<span class="hljs-string">'DIS'</span>,<span class="hljs-string">'RAD'</span>,<span class="hljs-string">'TAX'</span>,<span class="hljs-string">'PTRATIO'</span>,<span class="hljs-string">'B'</span>,<span class="hljs-string">'LSTAT'</span>,<span class="hljs-string">'target'</span>,<span class="hljs-string">'prediction'</span>]) \<br>.write.jdbc(table=<span class="hljs-string">'df_boston_result'</span>,mode=<span class="hljs-string">'overwrite'</span>,url=<span class="hljs-string">'jdbc:mysql://localhost:3306/test?createDatabaseIfNotExist=true&useSSL=false'</span>,properties={<span class="hljs-string">'user'</span>:<span class="hljs-string">'root'</span>,<span class="hljs-string">'password'</span>:<span class="hljs-string">'123456'</span>})<br><br>sc.stop()<br><br></code></pre></td></tr></table></figure><h3 id="五-使用-pipeline-改造"><a href="#五-使用-pipeline-改造" class="headerlink" title="五 使用 pipeline 改造"></a>五 使用 pipeline 改造</h3><p>参考：<a href="https://spark.apache.org/docs/latest/ml-pipeline.html">https://spark.apache.org/docs/latest/ml-pipeline.html</a><br>部分中文翻译参考： <a href="https://zhuanlan.zhihu.com/p/33619687">https://zhuanlan.zhihu.com/p/33619687</a></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> pyspark.sql <span class="hljs-keyword">import</span> SparkSession<br><span class="hljs-keyword">import</span> pyspark.pandas <span class="hljs-keyword">as</span> ps<br><span class="hljs-keyword">import</span> os<br><span class="hljs-keyword">from</span> pyspark <span class="hljs-keyword">import</span> SparkContext<br><span class="hljs-keyword">from</span> pyspark.sql <span class="hljs-keyword">import</span> SQLContext<br><span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt<br><span class="hljs-keyword">from</span> sklearn.datasets <span class="hljs-keyword">import</span> load_boston<br><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd<br><span class="hljs-keyword">import</span> pyspark.ml <span class="hljs-keyword">as</span> ml<br><span class="hljs-keyword">import</span> pyspark <span class="hljs-keyword">as</span> pyspark<br><span class="hljs-keyword">from</span> pyspark.ml <span class="hljs-keyword">import</span> Pipeline<br><span class="hljs-keyword">from</span> pyspark.ml <span class="hljs-keyword">import</span> PipelineModel<br><span class="hljs-keyword">import</span> sys<br><br>spark = SparkSession.builder.master(<span class="hljs-string">'local'</span>) \<br>.appName(<span class="hljs-string">"pyspark-lin"</span>) \<br>.config(<span class="hljs-string">"spark.jars"</span>,<span class="hljs-string">"{}/mysql-connector-java-8.0.28.jar"</span>.<span class="hljs-built_in">format</span>(os.getcwd())) \<br>.config(<span class="hljs-string">"spark.driver.extraClassPath"</span>,<span class="hljs-string">"{}/mysql-connector-java-8.0.28.jar"</span>.<span class="hljs-built_in">format</span>(os.getcwd())) \<br>.getOrCreate()<br>sc = spark.sparkContext<br><br>result_mysqlSource = ps.read_sql(<span class="hljs-string">'df_boston'</span>, <span class="hljs-string">'jdbc:mysql://localhost:3306/test?createDatabaseIfNotExist=true&useSSL=false&user=root&password=123456'</span>).to_spark()<br>result_transition0=result_mysqlSource<br><br>va = pyspark.ml.feature.VectorAssembler(inputCols=[<span class="hljs-string">'CRIM'</span>,<span class="hljs-string">'ZN'</span>,<span class="hljs-string">'INDUS'</span>,<span class="hljs-string">'CHAS'</span>,<span class="hljs-string">'NOX'</span>,<span class="hljs-string">'RM'</span>,<span class="hljs-string">'AGE'</span>,<span class="hljs-string">'DIS'</span>,<span class="hljs-string">'RAD'</span>,<span class="hljs-string">'TAX'</span>,<span class="hljs-string">'PTRATIO'</span>,<span class="hljs-string">'B'</span>,<span class="hljs-string">'LSTAT'</span>],outputCol=<span class="hljs-string">'features'</span>)<br>lr_model = pyspark.ml.regression.LinearRegressionModel.load(<span class="hljs-string">'file:///home/jovyan/LinearRegressionModel_model'</span>)<br>pipelineModel = PipelineModel(stages=[va, lr_model])<br><br>result=pipelineModel.transform(result_mysqlSource);<br><br>result.select([<span class="hljs-string">'CRIM'</span>,<span class="hljs-string">'ZN'</span>,<span class="hljs-string">'INDUS'</span>,<span class="hljs-string">'CHAS'</span>,<span class="hljs-string">'NOX'</span>,<span class="hljs-string">'RM'</span>,<span class="hljs-string">'AGE'</span>,<span class="hljs-string">'DIS'</span>,<span class="hljs-string">'RAD'</span>,<span class="hljs-string">'TAX'</span>,<span class="hljs-string">'PTRATIO'</span>,<span class="hljs-string">'B'</span>,<span class="hljs-string">'LSTAT'</span>,<span class="hljs-string">'target'</span>,<span class="hljs-string">'prediction'</span>]) \<br> .write.jdbc(table=<span class="hljs-string">'df_boston_result2'</span>,mode=<span class="hljs-string">'overwrite'</span>,url=<span class="hljs-string">'jdbc:mysql://localhost:3306/test?createDatabaseIfNotExist=true&useSSL=false'</span>,properties={<span class="hljs-string">'user'</span>:<span class="hljs-string">'root'</span>,<span class="hljs-string">'password'</span>:<span class="hljs-string">'123456'</span>})<br><br>sc.stop()<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 机器学习 </category>
</categories>
<tags>
<tag> 笔记 </tag>
<tag> pyspark </tag>
<tag> jupyter </tag>
</tags>
</entry>
<entry>
<title>vbs实现wincc或mysql数据导出成csv</title>
<link href="//vbs-wincc-mysql-csv/"/>
<url>//vbs-wincc-mysql-csv/</url>
<content type="html"><![CDATA[<p>工作要求，需要编写vbs实现将wincc数据导出，估计以后也没机会再接触这门古董语言了，所以就记录一下。<br>因为wincc用得太少了，测试中会使用mysql代替。</p><span id="more"></span><h3 id="业务逻辑"><a href="#业务逻辑" class="headerlink" title="业务逻辑"></a>业务逻辑</h3><ol><li>读取sql.txt文件，文件中每一行的格式为 <strong>id:select语句</strong>。</li><li>连接数据库，按行遍历执行查询，查询到的结果输出到指定文件夹下，得到以id命名的csv文件。</li><li>将执行过程、花费时间记录起来。</li></ol><h3 id="操作过程"><a href="#操作过程" class="headerlink" title="操作过程"></a>操作过程</h3><ol><li>准备sql文本文件</li><li>将代码复制到文本编辑器中，修改数据库连接参数、sql文本文件位置，输出文件夹位置，日志文件位置等</li><li>修改代码文件后缀为vbs</li><li>直接执行该文件即可</li><li>得到csv文件<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1642602647(1).png" alt="示例"></li></ol><h3 id="代码"><a href="#代码" class="headerlink" title="代码"></a>代码</h3><figure class="highlight vbs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br><span class="line">239</span><br><span class="line">240</span><br><span class="line">241</span><br><span class="line">242</span><br><span class="line">243</span><br><span class="line">244</span><br><span class="line">245</span><br><span class="line">246</span><br><span class="line">247</span><br><span class="line">248</span><br><span class="line">249</span><br><span class="line">250</span><br><span class="line">251</span><br><span class="line">252</span><br><span class="line">253</span><br><span class="line">254</span><br><span class="line">255</span><br></pre></td><td class="code"><pre><code class="hljs vbs"><br><span class="hljs-keyword">Sub</span> Main<br><br><span class="hljs-keyword">set</span> fs =<span class="hljs-built_in">createobject</span>(<span class="hljs-string">"scripting.filesystemobject"</span>)<br><span class="hljs-keyword">set</span> sqlTxt=fs.opentextfile(<span class="hljs-string">"C:\Users\15588\Desktop\vscode\sql.txt"</span>,<span class="hljs-number">1</span>,<span class="hljs-literal">false</span>)<br><span class="hljs-keyword">do</span> <span class="hljs-keyword">while</span> sqlTxt.atendofstream<><span class="hljs-literal">true</span><br>StartTime=Timer<br>queryStr=sqlTxt.readline<br>logs(<span class="hljs-string">"开始执行: "</span> & queryStr & <span class="hljs-string">"------------------------------"</span>)<br>queryStrArray=<span class="hljs-built_in">split</span>(queryStr,<span class="hljs-string">":"</span>)<br>id=queryStr<span class="hljs-built_in">Array</span>(<span class="hljs-number">0</span>)<br>sSql=queryStr<span class="hljs-built_in">Array</span>(<span class="hljs-number">1</span>)<br><span class="hljs-keyword">Dim</span> db<br><span class="hljs-keyword">Set</span> db = <span class="hljs-keyword">New</span> MySQLDB<br>db.Connect <span class="hljs-string">"数据库名"</span>, <span class="hljs-string">"root"</span>, <span class="hljs-string">"数据库密码"</span>, <span class="hljs-string">"ip"</span>, <span class="hljs-number">3306</span><br><span class="hljs-comment">''' Set db = New WinccDB</span><br><span class="hljs-comment">''' db.Connect "WinCCOLEDBProvider.1", "CC_wincctes_18_07_11_22_23_06R", "WINCC-PC\WinCC"</span><br><br><span class="hljs-keyword">If</span> Err <span class="hljs-keyword">Then</span><br>MsgBox Err.Description<br>db.UnConnect<br><span class="hljs-keyword">Exit</span> <span class="hljs-keyword">Sub</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br><br><span class="hljs-keyword">Call</span> queryToCsv(id, db.oConn,sSql,<span class="hljs-string">"C:\Users\15588\Desktop\vscode\halodb\"</span> & id & <span class="hljs-string">".csv"</span>)<br><br><span class="hljs-keyword">If</span> Err <span class="hljs-keyword">Then</span><br>MsgBox Err.Description<br>db.UnConnect<br><span class="hljs-keyword">Exit</span> <span class="hljs-keyword">Sub</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br>db.UnConnect<br><span class="hljs-keyword">Set</span> db = <span class="hljs-literal">Nothing</span><br><br>EndTime=Timer<br>Timelt=EndTime-StartTime<br>logs(<span class="hljs-string">"结束执行,耗时（秒）:"</span>& Timelt)<br><br><span class="hljs-keyword">loop</span><br><br><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><span class="hljs-keyword">Call</span> Main<br><br><span class="hljs-keyword">Class</span> WinccDB<br><br><span class="hljs-comment">'''Current Error Code:</span><br><span class="hljs-comment">'''- 1</span><br><span class="hljs-comment">'''- 2</span><br><span class="hljs-comment">'''- 3</span><br><span class="hljs-keyword">Public</span> oConn<br><span class="hljs-keyword">Private</span> oRecSet<br><br><br><span class="hljs-keyword">Private</span> <span class="hljs-keyword">Sub</span> <span class="hljs-keyword">Class_Initialize</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><br><span class="hljs-keyword">Private</span> <span class="hljs-keyword">Sub</span> <span class="hljs-keyword">Class_Terminate</span><br>UnConnect<br><span class="hljs-keyword">Set</span> dictErrDef = <span class="hljs-literal">Nothing</span><br><span class="hljs-comment">'''logs( "Clean works complete.")</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><br><span class="hljs-comment">'''' Public Methods ''''</span><br><span class="hljs-keyword">Public</span> <span class="hljs-keyword">Sub</span> Connect(provider,cataloge,ds)<br><span class="hljs-keyword">On</span> <span class="hljs-keyword">Error</span> <span class="hljs-keyword">Resume</span> <span class="hljs-keyword">Next</span><br>Err.Clear<br><br><span class="hljs-keyword">Dim</span> sConnStr<br><br><span class="hljs-comment">'''Create CONNECTION STRING</span><br>sConnStr =<span class="hljs-string">"Provider="</span> & provider & <span class="hljs-string">";Catalog="</span> & cataloge & <span class="hljs-string">";Data Source="</span> & ds<br><span class="hljs-comment">'''logs( "sConnStr:" & sConnStr)</span><br><span class="hljs-comment">'''Create CONNECTION</span><br><span class="hljs-keyword">Set</span> oConn = <span class="hljs-built_in">CreateObject</span>(<span class="hljs-string">"ADODB.Connection"</span>)<br>oConn.ConnectionString = sConnStr<br>oConn.CursorLocation = <span class="hljs-number">3</span><br>oConn.Open<br><span class="hljs-keyword">If</span> Err <span class="hljs-keyword">Then</span><br>MsgBox <span class="hljs-string">"Connection error"</span><br>MsgBox Err.Source<br>MsgBox Err.Description<br>Err.Raise vbObjectError + <span class="hljs-number">1</span><br><span class="hljs-keyword">Exit</span> <span class="hljs-keyword">Sub</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br><span class="hljs-keyword">If</span> oConn.State = <span class="hljs-number">0</span> <span class="hljs-keyword">Then</span><br>MsgBox <span class="hljs-string">"oConn.State = 0 "</span><br>Err.Raise vbObjectError + <span class="hljs-number">1</span><br><span class="hljs-keyword">Exit</span> <span class="hljs-keyword">Sub</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br><span class="hljs-comment">'''MsgBox "Connection created."</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><span class="hljs-keyword">Public</span> <span class="hljs-keyword">Sub</span> UnConnect()<br><span class="hljs-keyword">If</span> oConn <span class="hljs-keyword">Is</span> <span class="hljs-literal">Nothing</span> <span class="hljs-keyword">Then</span><br><span class="hljs-keyword">Exit</span> <span class="hljs-keyword">Sub</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br>oConn.Close<br><span class="hljs-keyword">Set</span> oConn = <span class="hljs-literal">Nothing</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Class</span><br><br><span class="hljs-keyword">Class</span> MySQLDB<br><br><span class="hljs-comment">'''Current Error Code:</span><br><span class="hljs-comment">'''- 1</span><br><span class="hljs-comment">'''- 2</span><br><span class="hljs-comment">'''- 3</span><br><span class="hljs-keyword">Public</span> oConn<br><span class="hljs-keyword">Private</span> oRecSet<br><br><br><span class="hljs-keyword">Private</span> <span class="hljs-keyword">Sub</span> <span class="hljs-keyword">Class_Initialize</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><br><span class="hljs-keyword">Private</span> <span class="hljs-keyword">Sub</span> <span class="hljs-keyword">Class_Terminate</span><br>UnConnect<br><span class="hljs-keyword">Set</span> dictErrDef = <span class="hljs-literal">Nothing</span><br><span class="hljs-comment">'''logs( "Clean works complete.")</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><br><span class="hljs-comment">'''' Public Methods ''''</span><br><span class="hljs-keyword">Public</span> <span class="hljs-keyword">Sub</span> Connect(db,uid,pwd,host,port)<br><span class="hljs-keyword">On</span> <span class="hljs-keyword">Error</span> <span class="hljs-keyword">Resume</span> <span class="hljs-keyword">Next</span><br>Err.Clear<br><br><span class="hljs-keyword">Dim</span> sConnStr<br><br><span class="hljs-comment">'''Create CONNECTION STRING</span><br>sConnStr = <span class="hljs-string">"DRIVER={MySQL ODBC 8.0 Unicode Driver};"</span><br><span class="hljs-comment">'sConnStr = "DRIVER={MySQL ODBC 5.2 ANSI Driver};"</span><br>sConnStr = sConnStr & <span class="hljs-string">"Database="</span> & db & <span class="hljs-string">";"</span><br>sConnStr = sConnStr & <span class="hljs-string">"User="</span> & uid & <span class="hljs-string">";"</span><br>sConnStr = sConnStr & <span class="hljs-string">"Password="</span> & pwd & <span class="hljs-string">";"</span><br>sConnStr = sConnStr & <span class="hljs-string">"Server="</span> & host & <span class="hljs-string">";"</span><br>sConnStr = sConnStr & <span class="hljs-string">"Port="</span> & port & <span class="hljs-string">";"</span><br>sConnStr = sConnStr & <span class="hljs-string">"Option=3;"</span><br><br><span class="hljs-comment">'''Create CONNECTION</span><br><span class="hljs-keyword">Set</span> oConn = <span class="hljs-built_in">CreateObject</span>(<span class="hljs-string">"ADODB.Connection"</span>)<br>oConn.Open sConnStr<br><span class="hljs-keyword">If</span> Err <span class="hljs-keyword">Or</span> oConn.State = <span class="hljs-number">0</span> <span class="hljs-keyword">Then</span><br>MsgBox <span class="hljs-string">"oConn.State = 0 "</span><br>Err.Raise vbObjectError + <span class="hljs-number">1</span><br><span class="hljs-keyword">Exit</span> <span class="hljs-keyword">Sub</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br><span class="hljs-comment">'''MsgBox "Connection created."</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><span class="hljs-keyword">Public</span> <span class="hljs-keyword">Sub</span> UnConnect()<br><span class="hljs-keyword">If</span> oConn <span class="hljs-keyword">Is</span> <span class="hljs-literal">Nothing</span> <span class="hljs-keyword">Then</span><br><span class="hljs-keyword">Exit</span> <span class="hljs-keyword">Sub</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br>oConn.Close<br><span class="hljs-keyword">Set</span> oConn = <span class="hljs-literal">Nothing</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Sub</span><br><br><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Class</span><br><br><br><span class="hljs-keyword">Public</span> <span class="hljs-keyword">Function</span> queryToCsv(id,oConn,sql,path)<br><span class="hljs-keyword">On</span> <span class="hljs-keyword">Error</span> <span class="hljs-keyword">Resume</span> <span class="hljs-keyword">Next</span><br>Err.Clear<br><br><span class="hljs-comment">'''CREATE RECORDSET</span><br><span class="hljs-keyword">Set</span> oRecSet = <span class="hljs-built_in">CreateObject</span>(<span class="hljs-string">"ADODB.Recordset"</span>)<br>oRecSet.CursorLocation = <span class="hljs-number">3</span><br>oRecSet.Open sql, oConn<br><span class="hljs-keyword">If</span> Err <span class="hljs-keyword">Then</span><br>MsgBox <span class="hljs-string">"query error"</span><br>MsgBox Err.Source<br>MsgBox Err.Description<br><span class="hljs-keyword">Exit</span> <span class="hljs-keyword">Function</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br>logs( <span class="hljs-string">"oRecSet.RecordCount:"</span> & oRecSet.RecordCount)<br><span class="hljs-keyword">Call</span> RecordSet2Csv(id,oRecSet,path)<br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Function</span><br><span class="hljs-keyword">Public</span> <span class="hljs-keyword">Function</span> RecordSet2Csv(id,rs,path)<br><span class="hljs-keyword">Dim</span> field<br><span class="hljs-keyword">Dim</span> tmpStr<br><span class="hljs-keyword">Dim</span> c<br>c = <span class="hljs-number">1</span><br><span class="hljs-keyword">Set</span> outFile = <span class="hljs-built_in">CreateObject</span>(<span class="hljs-string">"Scripting.FileSystemObject"</span>).CreateTextFile(path)<br>tmpStr=CsvValue_(<span class="hljs-string">"id"</span>) & <span class="hljs-string">","</span><br><span class="hljs-keyword">For</span> <span class="hljs-keyword">Each</span> field <span class="hljs-keyword">In</span> rs.Fields<br><span class="hljs-keyword">If</span> field.Name = <span class="hljs-string">""</span> <span class="hljs-keyword">Then</span><br>tmpStr=tmpStr & CsvValue_(<span class="hljs-string">"(computed"</span> & c & <span class="hljs-string">")"</span>) & <span class="hljs-string">","</span><br>c = c + <span class="hljs-number">1</span><br><span class="hljs-keyword">Else</span><br>tmpStr=tmpStr & CsvValue_(field.Name) & <span class="hljs-string">","</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br><span class="hljs-keyword">Next</span><br>outFile.WriteLine RTrimOne_(tmpStr)<br> c = <span class="hljs-number">1</span><br><span class="hljs-keyword">Do</span> <span class="hljs-keyword">While</span> <span class="hljs-keyword">Not</span> rs.EOF<br><span class="hljs-comment">'''If c=65535 Then</span><br><span class="hljs-comment">''' MsgBox "记录数超过65535！请缩小导出时间范围！"</span><br><span class="hljs-comment">''' Exit Do</span><br><span class="hljs-comment">'''End If</span><br>tmpStr = CsvValue_(id) & <span class="hljs-string">","</span><br><span class="hljs-keyword">For</span> <span class="hljs-keyword">Each</span> field <span class="hljs-keyword">In</span> rs.Fields<br><span class="hljs-comment">'''tmpStr=tmpStr & CsvValue_(field.Value) & ","</span><br>tmpStr=tmpStr & field.Value & <span class="hljs-string">","</span><br><span class="hljs-keyword">Next</span><br>outFile.WriteLine RTrimOne_(tmpStr)<br>rs.MoveNext<br> c = c+<span class="hljs-number">1</span><br><span class="hljs-keyword">Loop</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Function</span><br><br><span class="hljs-keyword">Private</span> <span class="hljs-keyword">Function</span> CsvValue_(val)<br><span class="hljs-keyword">If</span> <span class="hljs-built_in">IsNull</span>(val) <span class="hljs-keyword">Then</span><br>CsvValue_ = <span class="hljs-string">""</span><br><span class="hljs-keyword">Else</span><br>CsvValue_ = <span class="hljs-string">""""</span> & <span class="hljs-built_in">Replace</span>(val, <span class="hljs-string">""""</span>, <span class="hljs-string">""""""</span>) & <span class="hljs-string">""""</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Function</span><br><br><span class="hljs-keyword">Private</span> <span class="hljs-keyword">Function</span> RTrimOne_(str)<br>RTrimOne_ = <span class="hljs-built_in">Left</span>(str, <span class="hljs-built_in">Len</span>(str) - <span class="hljs-number">1</span>)<br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Function</span><br><br><span class="hljs-keyword">Private</span> <span class="hljs-keyword">Function</span> logs(logdetail)<br><br>FilePath = <span class="hljs-string">"C:\Users\15588\Desktop\vscode\mysql2csv.log"</span> <span class="hljs-comment">'LOG文件的路径</span><br>LOGDetail = logdetail <span class="hljs-comment">'LOG的内容</span><br><br><span class="hljs-keyword">Set</span> fso = <span class="hljs-built_in">CreateObject</span>(<span class="hljs-string">"scripting.FileSystemObject"</span>)<br><br><span class="hljs-comment">'判断LOG文件是否存在,如不存在,则按指定路径新建</span><br><br><span class="hljs-keyword">If</span> fso.FileExists(FilePath) = <span class="hljs-literal">False</span> <span class="hljs-keyword">Then</span><br><br><span class="hljs-keyword">Set</span> LOGFile = fso.CreateTextFile(FilePath, <span class="hljs-literal">True</span>)<br><br>LOGFile.Close<br><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">If</span><br><br><br><span class="hljs-keyword">Set</span> LOGFile = fso.OpenTextFile(FilePath, <span class="hljs-number">8</span>, <span class="hljs-literal">True</span>) <span class="hljs-comment">'8为追加</span><br><br>LOGFile.WriteLine <span class="hljs-built_in">Cstr</span>(Now) & <span class="hljs-string">" : "</span> & LOGDetail<br><br>LOGFile.Close<br><br><span class="hljs-keyword">End</span> <span class="hljs-keyword">Function</span><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 后端开发 </category>
</categories>
<tags>
<tag> vbs </tag>
<tag> wincc </tag>
<tag> mysql </tag>
<tag> csv </tag>
</tags>
</entry>
<entry>
<title>基于hdfs的ftp服务器的设计与实现</title>
<link href="//hdfs-ftp-server-design-and-implementation/"/>
<url>//hdfs-ftp-server-design-and-implementation/</url>
<content type="html"><![CDATA[<p>本文主要记录hdfs-ftp-server的设计思路、实现过程，使用的话直接看github项目文档即可。<br>项目地址：<a href="https://github.com/linshenkx/hdfs-ftp-server">https://github.com/linshenkx/hdfs-ftp-server</a></p><span id="more"></span><h2 id="第一版"><a href="#第一版" class="headerlink" title="第一版"></a>第一版</h2><p>hdfs-ftp-server这个项目我在20年的时候就开源过一版了，先做一下前提介绍。</p><h3 id="1-需求"><a href="#1-需求" class="headerlink" title="1 需求"></a>1 需求</h3><p>因为我们大数据平台是容器化的，一些简单的操作也需要进入容器里面才能执行，而且传递文件也很繁琐。<br>所以那时候的主要需求就是<strong>实现功能</strong>，即通过ftp的方式实现对hdfs文件系统的操作，方便日常操作。</p><h3 id="2-思路"><a href="#2-思路" class="headerlink" title="2 思路"></a>2 思路</h3><p>那时候主要是参考github上一些老一点的项目进行改造升级，以适配高可用、kerberos认证连接。<br>然后再进行易用性改造，更适合生产环境使用。</p><p>核心思路是对外作为ftp服务器，内部使用hdfs实现接口功能即可，很简单的功能。</p><h3 id="3-存在问题"><a href="#3-存在问题" class="headerlink" title="3 存在问题"></a>3 存在问题</h3><p>那时候用的hadoop是官方的2.9.2，后面hadoop升级到3.3.1了，<br>而且有时候还得对接cdh的hadoop，这个时候这个工程就不能直接拿来用了。</p><p>这种时候常规解决办法就是修改一下依赖，重新编译。</p><p>但是我觉得太麻烦了，这样也不利于代码的管理。<br>所以就一直想着要把这个工程模块化，核心工程独立且不常变，其他的作为模块工程对接各个版本的hadoop客户端。</p><h2 id="第二版"><a href="#第二版" class="headerlink" title="第二版"></a>第二版</h2><p>针对第一版存在的问题，第二版主要是对项目进行模块化改造</p><h3 id="技术调研及选型"><a href="#技术调研及选型" class="headerlink" title="技术调研及选型"></a>技术调研及选型</h3><p>最开始我目标是很明确的，就是用蚂蚁金服的<a href="https://github.com/sofastack/sofa-ark">ark框架</a>来实现类隔离，<br>毕竟不同大版本/发行版本的hdfs依赖多少会有类冲突，直接用是肯定不放心的。</p><p>ark我之前就一直有在留意，也跑过demo，觉得还是可以寄予厚望的。<br>事实上这个工程重构所用的时间一大半都花在ark上。<br>本来是想把各个实现模块作为ark的插件的，发现ark的插件又不支持动态插拔，其核心是类隔离而不是模块化。<br>那就把每个模块作为service，来实现动态引用，这也是可以的，也确实改好代码打包好了。<br>但最终运行的时候还是有bug，类似于：<a href="https://github.com/sofastack/sofa-ark/issues/421%E3%80%82">https://github.com/sofastack/sofa-ark/issues/421。</a></p><p>终于还是折腾不下去，放弃ark了，开源支持太差了，我水平有限，玩不动。</p><p>不用ark，osgi又太重了，没必要，那就还是老老实实用java的spi吧。<br>类似于jdbc的用法，添加对应jar包，指定类名进行加载。</p><p>但spi并不能实现类隔离，所以就预先打包好各个jar包，运行时再根据对接hadoop版本进行加载。</p><p>即每个模块形成一个fatjar，如cdh632.jar，official331.jar。<br>在对接cdh6.3.2的时候将cdh632.jar添加到类路径下即可，如果是对接官方3.3.1则添加official331.jar。</p><p>这里还有一个问题，要不要把hadoop的依赖包也一并打包呢？<br>这个倒没纠结太久，还是打包了。缺点是体积大很多，优点是摆脱依赖。<br>你只需要给我hadoop的连接配置文件就行了，而不需要整个hadoop客户端环境。<br>至于包大点就大点，不差那点流量、空间。</p><p>为了简化、统一子模块的开发，我又把gradle任务统一定义在父工程的build.gradle里面。<br>让子模块更加的简单易懂。</p><h3 id="成果总结"><a href="#成果总结" class="headerlink" title="成果总结"></a>成果总结</h3><p><a href="https://github.com/linshenkx/hdfs-ftp-server">https://github.com/linshenkx/hdfs-ftp-server</a></p><p>应该说还是比较满意的。</p><p>虽然就技术来说很简单，毕竟别人十年前就做过的东西，<br>但还是花了很多时间、心思在上面折腾，把这个小东西做得好用一些。<br>在这个过程也算有点收获吧。</p>]]></content>
<categories>
<category> 后端开发 </category>
</categories>
<tags>
<tag> ftp </tag>
<tag> hdfs </tag>
</tags>
</entry>
<entry>
<title>基于kube-prometheus的大数据平台监控系统设计</title>
<link href="//kube-prometheus-bigdata/"/>
<url>//kube-prometheus-bigdata/</url>
<content type="html"><![CDATA[<p>本文介绍了如何基于kube-prometheus设计一个监控系统， 以灵活简单的方式对kubernetes上的应用进行指标采集，并实现监控报警功能。<br>本文提供了作者的应用示例，另外还记录了作者在学习、使用Prometheus过程中的一些笔记，如arm版镜像获取、一些工具的使用等。</p><span id="more"></span><h2 id="零-前言"><a href="#零-前言" class="headerlink" title="零 前言"></a>零 前言</h2><p>众所周知，大数据产品作为底层平台，其运维监控一直是生产实践的痛点难点，且在稳定运行的基础之上<br>往往还需要对性能进行评估优化，所以其监控系统的建设显得尤为重要。</p><p>Prometheus作为云原生时代最火的监控软件，很多大数据组件或原生或以第三方插件/exporter的形式对Prometheus做了支持。</p><p>我使用的大数据平台是基于kubernetes运行的，有部署灵活管理方便的优点，更容易与Prometheus进行结合。</p><p>下面将对设计思路和技术实现进行阐述探讨。</p><h2 id="一-设计思路"><a href="#一-设计思路" class="headerlink" title="一 设计思路"></a>一 设计思路</h2><p>监控系统的核心任务是将暴露出来的指标数据进行抓取，在此之上进行分析、告警<br>所以有以下几个要明确的问题：</p><ol><li>监控对象是什么</li><li>监控对象如何暴露指标数据</li><li>监控系统如何对指标进行抓取</li><li>如何实现告警规则动态配置、管理</li></ol><h3 id="1-监控对象"><a href="#1-监控对象" class="headerlink" title="1. 监控对象"></a>1. 监控对象</h3><p>以pod（容器）形式运行在kubernetes集群上的各个大数据组件。</p><h3 id="2-指标暴露方式"><a href="#2-指标暴露方式" class="headerlink" title="2. 指标暴露方式"></a>2. 指标暴露方式</h3><p>各组件根据对Prometheus的支持程度，可分为3种类型的指标暴露方式：</p><ol><li>直接暴露Prometheus指标数据(直接，拉)</li><li>主动将指标数据推送到prometheus-pushGateway，由pushGateway暴露数据（间接，推）</li><li>自定义exporter将其他形式的指标数据转换为符合Prometheus标准的格式进行暴露（exporter，直接，拉）</li></ol><p>个别组件同时支持多种方式，如flink支持直接和间接方式，spark支持直接方式而且也有第三方exporter。<br>大部分组件都有官方/第三方的exporter，极少数需要自己开发。</p><p>一般情况下直接方式就可以了<br>需要注意的是，像flink（spark） on yarn模式运行的时候，flink节点是跑在yarn容器里面的。<br>这种情况下Prometheus很难对其直接进行抓取，这种时候就只能用间接方式，主动将数据推送到pushGateway。</p><p>另外那些短暂生命周期的组件也建议用主动push到pushGateway。</p><h3 id="3-指标抓取方式"><a href="#3-指标抓取方式" class="headerlink" title="3. 指标抓取方式"></a>3. 指标抓取方式</h3><p>不管是exporter还是pushGateway，到最后必然是由Prometheus主动对这些目标进行抓取。</p><blockquote><p>Prometheus 主要通过 Pull 的方式来抓取目标服务暴露出来的监控接口，<br>因此需要配置对应的抓取任务来请求监控数据并写入到 Prometheus 提供的存储中，<br>目前 Prometheus 服务提供了如下几个任务的配置：</p><ul><li>原生 Job 配置：提供 Prometheus 原生抓取 Job 的配置。</li><li>Pod Monitor：在 K8S 生态下，基于 Prometheus Operator 来抓取 Pod 上对应的监控数据。</li><li>Service Monitor：在 K8S 生态下，基于 Prometheus Operator 来抓取 Service 对应 Endpoints 上的监控数据。</li></ul><p>参考：<a href="https://cloud.tencent.com/document/product/1416/55995">https://cloud.tencent.com/document/product/1416/55995</a></p></blockquote><p>既然都上了kubernetes环境了，一般当然是推荐直接用 podMonitor。配置更简洁易懂。<br>podMonitorSelector的过滤在prometheus-prometheus.yaml配置。</p><blockquote><p>prometheus-prometheus.yaml是核心配置文件，不宜频繁修改(会导致Prometheus重启)。<br>主要配置项为：serviceMonitorSelector，podMonitorSelector，ruleSelector，alertmanagers。<br>其中service监控选择器和pod监控选择器默认选择所有，这里建议把 ruleSelector 也修改为选择所有</p></blockquote><p>不过一个podMonitor一般只对应一种类型的pod，在已有pod类型较多的情况下，<br>还可以考虑一种更取巧的方法就是Prometheus的kubernetes服务发现功能。即kubernetes_sd_config。<br>这种属于<em>原生Job配置</em>，建议使用<a href="https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md">additional-scrape-config</a><br>进行配置。</p><p><a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config">kubernetes_sd_config</a>赋予了Prometheus通过kubernetes rest api感知kubernetes资源的功能，<br>利用该能力，可以使用<em>原生Job配置</em>自动发现pod，将其作为监控目标。<br>再利用Prometheus的<a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config">Relabel功能</a>可以改写发现的标签，进行前置处理、转换。实现pod筛选，修改抓取配置的效果。<br>而自动发现的pod的标签的来源又可以是pod资源的label/annotation等。<br>最终实现的效果如下，<br>这是一个pushGateway的pod的配置,则Prometheus会通过其19091端口访问/metrics路径获取其指标数据</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">annotations:</span><br> <span class="hljs-attr">prometheus.io/scrape:</span> <span class="hljs-string">"true"</span><br> <span class="hljs-attr">prometheus.io/scheme:</span> <span class="hljs-string">"http"</span><br> <span class="hljs-attr">prometheus.io/path:</span> <span class="hljs-string">"/metrics"</span><br> <span class="hljs-attr">prometheus.io/port:</span> <span class="hljs-string">"19091"</span><br></code></pre></td></tr></table></figure><p>这部分的内容主要参考：</p><ol><li><a href="https://godleon.github.io/blog/Prometheus/Prometheus-Relabel">https://godleon.github.io/blog/Prometheus/Prometheus-Relabel</a></li><li><a href="https://yunlzheng.gitbook.io/prometheus-book/part-iii-prometheus-shi-zhan/readmd/service-discovery-with-kubernetes">https://yunlzheng.gitbook.io/prometheus-book/part-iii-prometheus-shi-zhan/readmd/service-discovery-with-kubernetes</a></li></ol><p>podMonitor是官方支持，简洁易懂。<br>kubernetes_sd_config+relabel的方案较复杂，难度较高，但不用写那么多的podMonitor。<br>自行抉择就行，也可以一起用。</p><h3 id="4-告警设计"><a href="#4-告警设计" class="headerlink" title="4. 告警设计"></a>4. 告警设计</h3><h4 id="告警流程"><a href="#告警流程" class="headerlink" title="告警流程"></a>告警流程</h4><p>prometheus的监控告警基本流程是：</p><ol><li>服务发生异常</li><li>触发prometheus服务器发出告警信息（alert）</li><li>alertmanager收到告警信息</li><li>alertmanager根据预配置的规则对告警信息进行处理，实现业务逻辑，如分组、抑制、触发短信邮箱等</li></ol><p>当然具体的流程没那么简单，有很多细节需要注意，特别是触发告警时机，是个重点。<br>这些属于Prometheus的机制实现，这里就不展开赘述，推荐阅读以下文章：<br><a href="https://my.oschina.net/u/4400455/blog/3442101">Prometheus 一条告警的触发流程、等待时间</a><br><a href="https://www.qikqiak.com/post/alertmanager-when-alert/">AlertManager 何时报警</a></p><p>后边会给出一个本人实际应用测试的例子，可供参考，会直观一些。</p><h4 id="告警的动态配置"><a href="#告警的动态配置" class="headerlink" title="告警的动态配置"></a>告警的动态配置</h4><p>kube-prometheus的告警规则分两部分：</p><ol><li>alertmanager: 即对告警信息的处理策略<br>配置参考：<a href="https://yunlzheng.gitbook.io/prometheus-book/parti-prometheus-ji-chu/alert/alert-manager-config">https://yunlzheng.gitbook.io/prometheus-book/parti-prometheus-ji-chu/alert/alert-manager-config</a><br>核心是alertmanager-secret.yaml配置文件，该文件以 secret 的形式被 Alertmanager 读取。<br>Alertmanager会自动读取secret中的配置进行更新。</li><li>alertRule: 即具体的告警规则<br>配置参考：<a href="https://yunlzheng.gitbook.io/prometheus-book/parti-prometheus-ji-chu/alert/prometheus-alert-rule">https://yunlzheng.gitbook.io/prometheus-book/parti-prometheus-ji-chu/alert/prometheus-alert-rule</a><br>在kubernetes中是以PrometheusRule类型操作，所以管理起来跟pod一样，直接使用kubelet增删改即可</li></ol><h4 id="接入自定义告警平台"><a href="#接入自定义告警平台" class="headerlink" title="接入自定义告警平台"></a>接入自定义告警平台</h4><p>从个人实践的角度来看，AlertManager处理web hook之外的告警接收插件，如短信、邮箱等只适合测着玩。<br>生产使用还是要通过web hook将告警信息发送到自己的告警平台。可以根据业务需要对告警信息做高度定制化处理、记录等。<br>另外可以在告警信息中携带具体告警规则等信息指导告警平台进行处理。</p><p>这里要做个区分，AlertManager是告警信息的前置处理，负责非业务性前置操作，如告警信息分组、平抑等。<br>而自定义告警平台则负责告警信息的业务处理，如记录、去敏、发送到多终端等。</p><p>AlertManager可能收到1w条告警信息，经过处理最终只发了1条到自定义告警平台。<br>而自定义告警平台可以将这1条告警信息记录起来，修改内容，同时使用邮箱、短信通知到多个负责人。</p><h4 id="告警层级标签设计"><a href="#告警层级标签设计" class="headerlink" title="告警层级标签设计"></a>告警层级标签设计</h4><p>监控对象的粒度决定告警的层级，体现在配置上则是告警规则的分组。<br>分组信息决定alertManager的处理方式。<br>alertManager对告警信息的路由策略是树状的，所以可通过多个分组标签实现多层级路由处理。<br>具体设计应结合业务需求，不在这里展开，感兴趣的可以看我下面的实现举例。</p><h2 id="二-技术实现"><a href="#二-技术实现" class="headerlink" title="二 技术实现"></a>二 技术实现</h2><p>技术实现主要分以下几部分：</p><ol><li>kubernetes环境下prometheus的部署(kube-prometheus)</li><li>kube-prometheus的增强配置:即kubernetes_sd_config+relabel方案的实现</li><li>bigdata-exporter的实现</li><li>告警设计实例</li></ol><h3 id="1-kubernetes环境下prometheus的部署"><a href="#1-kubernetes环境下prometheus的部署" class="headerlink" title="1. kubernetes环境下prometheus的部署"></a>1. kubernetes环境下prometheus的部署</h3><h4 id="1-kube-prometheus-vs-prometheus-operator"><a href="#1-kube-prometheus-vs-prometheus-operator" class="headerlink" title="1) kube-prometheus vs prometheus-operator"></a>1) kube-prometheus vs prometheus-operator</h4><p>github 上 coreos 下有两个项目： kube-prometheus 和 prometheus-operator，<br>两者都可以实现 prometheus 的创建及管理。</p><p>需要注意的是，kube-prometheus 上的配置操作也是基于 prometheus-operator 的，<br>并提供了大量的默认配置，故这里使用的是 kube-prometheus 项目的配置。</p><p>另外使用前需注意k8s版本要求，找到对应的kube-prometheus版本，弄清楚对应的prometheus-operator版本<br>如：k8s1.14版本最高可使用 kube-prometheus 0.3，对应的 prometheus-operator 版本是 0.32<br>阅读文档时注意对应版本。</p><h4 id="2-kube-prometheus-使用前说明"><a href="#2-kube-prometheus-使用前说明" class="headerlink" title="2) kube-prometheus 使用前说明"></a>2) kube-prometheus 使用前说明</h4><p>kube-prometheus 使用 jsonnet 编写配置模板文件，生成k8s配置清单<br>已提供默认清单文件，在 manifests 文件夹下<br>如果需要修改默认清单配置，需要在go环境下使用jp编译清单<br>下面都以默认配置为例</p><h4 id="3-安装教程"><a href="#3-安装教程" class="headerlink" title="3) 安装教程"></a>3) 安装教程</h4><p>参考官方说明即可</p><ol><li>git clone 项目 并切换到指定分支</li><li>kubectl create<br>清单文件中各配置已附带namespace信息，故执行时不需要指定namespace，否则可能出错。<br>官方命令如下： <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash">Create the namespace and CRDs, and <span class="hljs-keyword">then</span> <span class="hljs-built_in">wait</span> <span class="hljs-keyword">for</span> them to be availble before creating the remaining resources</span><br><br>kubectl create -f manifests/setup<br>until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done<br>kubectl create -f manifests/<br></code></pre></td></tr></table></figure></li></ol><h3 id="2-kubernetes-sd-config-relabel方案的实现"><a href="#2-kubernetes-sd-config-relabel方案的实现" class="headerlink" title="2. kubernetes_sd_config+relabel方案的实现"></a>2. kubernetes_sd_config+relabel方案的实现</h3><p>见： <a href="https://github.com/linshenkx/kube-prometheus-enhance">https://github.com/linshenkx/kube-prometheus-enhance</a></p><h3 id="3-bigdata-exporter的实现"><a href="#3-bigdata-exporter的实现" class="headerlink" title="3. bigdata-exporter的实现"></a>3. bigdata-exporter的实现</h3><p>hdfs、yarn、hbase、yarn等组件都提供了web获取jmx指标的方式。</p><p>这里的思路是使用一个bigdata-exporter，去采集多个组件多个节点的指标数据，并进行转换，然后以Prometheus规定的格式对外公开。</p><p>指标数据的转换规则可以查看github上的一些项目，要注意版本，也可以像我一样自己写，更可靠。</p><p>bigdata-exporter如何感知到采集目标？</p><p>除了部署ip不同，不同组件不同角色的指标对外端口、路径、内容（解析规则）也都不一样。<br>这里可以参考上面kubernetes_sd_config+relabel的方案，做得优雅一些：</p><ol><li>授予bigdata-exporter调用kubernetes app的能力，</li><li>利用label和annotations进行筛选和信息传递，确定捕捉目标和途径。</li><li>使用role代表解析内容的类型，根据role确定解析规则<figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">labels:</span><br> <span class="hljs-attr">bigData.metrics.object:</span> <span class="hljs-string">pod</span><br><span class="hljs-attr">annotations:</span><br> <span class="hljs-attr">bigData.metrics/scrape:</span> <span class="hljs-string">"true"</span><br> <span class="hljs-attr">bigData.metrics/scheme:</span> <span class="hljs-string">"https"</span><br> <span class="hljs-attr">bigData.metrics/path:</span> <span class="hljs-string">"/jmx"</span><br> <span class="hljs-attr">bigData.metrics/port:</span> <span class="hljs-string">"29871"</span><br> <span class="hljs-attr">bigData.metrics/role:</span> <span class="hljs-string">"hdfs-nn,common"</span><br></code></pre></td></tr></table></figure><h3 id="4-告警设计示例"><a href="#4-告警设计示例" class="headerlink" title="4. 告警设计示例"></a>4. 告警设计示例</h3>这里以<em>组</em>和<em>实例</em>两个维度为例，用groupId和instanceId表示。</li></ol><h4 id="1-alertManager配置示例"><a href="#1-alertManager配置示例" class="headerlink" title="1) alertManager配置示例"></a>1) alertManager配置示例</h4><p>以下是alertmanager的规则配置，有两个接收者，<br>其中 test.web.hook 指向自定义告警平台<br>default 是空白接收者，不做处理。<br>路由策略是根据groupId,instanceId分组，对节点磁盘使用率、kafka队列堆积两个组处理，instanceId还没有展开。</p><p>旧版本是用secret的data字段，需要将配置内容转成base64编码格式。<br>新版本直接用stringData字段。推荐用stringData字段配置。<br>其实只要看一下kube-prometheus的alertmanager-secret.yaml文件就知道怎么回事了。</p><p>使用data字段的配置方法：<br>写好config文件，以 alertmanager.yaml 命名（不能使用其他名称）。<br>执行以下命令,即可更新secret。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl -n monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run -o yaml | kubectl -n=monitoring apply -f -<br><br></code></pre></td></tr></table></figure><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">global:</span><br> <span class="hljs-attr">resolve_timeout:</span> <span class="hljs-string">5m</span><br><span class="hljs-attr">receivers:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">'default'</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">'test.web.hook'</span><br> <span class="hljs-attr">webhook_configs:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">url:</span> <span class="hljs-string">'http://alert-url'</span><br><span class="hljs-attr">route:</span><br> <span class="hljs-attr">receiver:</span> <span class="hljs-string">'default'</span><br> <span class="hljs-attr">group_wait:</span> <span class="hljs-string">30s</span><br> <span class="hljs-attr">group_interval:</span> <span class="hljs-string">5m</span><br> <span class="hljs-attr">repeat_interval:</span> <span class="hljs-string">2h</span><br> <span class="hljs-attr">group_by:</span> [<span class="hljs-string">groupId</span>,<span class="hljs-string">instanceId</span>]<br> <span class="hljs-attr">routes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">receiver:</span> <span class="hljs-string">'test.web.hook'</span><br> <span class="hljs-attr">continue:</span> <span class="hljs-literal">true</span><br> <span class="hljs-attr">match:</span><br> <span class="hljs-attr">groupId:</span> <span class="hljs-string">node-disk-usage</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">receiver:</span> <span class="hljs-string">'test.web.hook'</span><br> <span class="hljs-attr">continue:</span> <span class="hljs-literal">true</span><br> <span class="hljs-attr">match:</span><br> <span class="hljs-attr">groupId:</span> <span class="hljs-string">kafka-topic-highstore</span><br><br></code></pre></td></tr></table></figure><h4 id="2-alertRule配置示例"><a href="#2-alertRule配置示例" class="headerlink" title="2) alertRule配置示例"></a>2) alertRule配置示例</h4><p>组代表一个类型的所有目标：即所有节点<br>实例则代表具体的某个节点</p><p>disk-usage.yaml.ftl磁盘使用率告警配置示例如下：<br>注意：${path}为监控的磁盘路径，${thresholdValue}为使用率阈值，需自行替换<br>labels中的userIds和receivers为传递给自定义告警平台的参数，以指导告警平台如何操作。<br>在这个任务中，我们的目标是组粒度的（所有节点），所以不需要设置instanceId。</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">monitoring.coreos.com/v1</span><br><span class="hljs-attr">kind:</span> <span class="hljs-string">PrometheusRule</span><br><span class="hljs-attr">metadata:</span><br> <span class="hljs-attr">creationTimestamp:</span> <span class="hljs-literal">null</span><br> <span class="hljs-attr">labels:</span><br> <span class="hljs-attr">role:</span> <span class="hljs-string">alert-rules</span><br> <span class="hljs-attr">name:</span> <span class="hljs-string">node-disk-usage</span><br> <span class="hljs-attr">namespace:</span> <span class="hljs-string">monitoring</span><br><span class="hljs-attr">spec:</span><br> <span class="hljs-attr">groups:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">node-disk-usage</span><br> <span class="hljs-attr">rules:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">node-disk-usage</span><br> <span class="hljs-attr">expr:</span> <span class="hljs-number">100</span><span class="hljs-string">*(1-node_filesystem_avail_bytes{mountpoint="${path}"}/node_filesystem_size_bytes{mountpoint="${path}"}</span> <span class="hljs-string">)</span> <span class="hljs-string">></span> <span class="hljs-string">${thresholdValue}</span><br> <span class="hljs-attr">for:</span> <span class="hljs-string">1m</span><br> <span class="hljs-attr">labels:</span><br> <span class="hljs-attr">groupId:</span> <span class="hljs-string">node-disk-usage</span><br> <span class="hljs-attr">userIds:</span> <span class="hljs-string">super</span><br> <span class="hljs-attr">receivers:</span> <span class="hljs-string">SMS</span><br> <span class="hljs-attr">annotations:</span><br> <span class="hljs-attr">title:</span> <span class="hljs-string">"磁盘警告：节点<span class="hljs-template-variable">{{$labels.instance}}</span>的 ${path} 目录使用率已达到<span class="hljs-template-variable">{{$value}}</span>%"</span><br> <span class="hljs-attr">content:</span> <span class="hljs-string">"磁盘警告：节点<span class="hljs-template-variable">{{$labels.instance}}</span>的 ${path} 目录使用率已达到<span class="hljs-template-variable">{{$value}}</span>%"</span><br><br></code></pre></td></tr></table></figure><p>kafka-topic-highstore.yaml.ftl kafka队列消费堆积告警配置示例如下：<br>我们只关心个别队列的消费情况，所以这里的粒度为instance。<br>注意：${uniqueName}为队列名，${consumergroup}为消费组名称，${thresholdValue}为堆积数量阈值</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">monitoring.coreos.com/v1</span><br><span class="hljs-attr">kind:</span> <span class="hljs-string">PrometheusRule</span><br><span class="hljs-attr">metadata:</span><br> <span class="hljs-attr">creationTimestamp:</span> <span class="hljs-literal">null</span><br> <span class="hljs-attr">labels:</span><br> <span class="hljs-attr">role:</span> <span class="hljs-string">alert-rules</span><br> <span class="hljs-attr">name:</span> <span class="hljs-string">kafka-topic-highstore-${uniqueName}</span><br> <span class="hljs-attr">namespace:</span> <span class="hljs-string">monitoring</span><br><span class="hljs-attr">spec:</span><br> <span class="hljs-attr">groups:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">kafka-topic-highstore</span><br> <span class="hljs-attr">rules:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">kafka-topic-highstore-${uniqueName}</span><br> <span class="hljs-attr">expr:</span> <span class="hljs-string">sum(kafka_consumergroup_lag{exporterType="kafka",consumergroup="${consumergroup}"})</span> <span class="hljs-string">></span> <span class="hljs-string">${thresholdValue}</span><br> <span class="hljs-attr">for:</span> <span class="hljs-string">1m</span><br> <span class="hljs-attr">labels:</span><br> <span class="hljs-attr">groupId:</span> <span class="hljs-string">kafka-topic-highstore</span><br> <span class="hljs-attr">instanceId:</span> <span class="hljs-string">${uniqueName}</span><br> <span class="hljs-attr">userIds:</span> <span class="hljs-string">super</span><br> <span class="hljs-attr">receivers:</span> <span class="hljs-string">SMS</span><br> <span class="hljs-attr">annotations:</span><br> <span class="hljs-attr">title:</span> <span class="hljs-string">"KAFKA警告：消费组${consumergroup}的堆积数量达到:<span class="hljs-template-variable">{{$value}}</span>"</span><br> <span class="hljs-attr">content:</span> <span class="hljs-string">"KAFKA警告：消费组${consumergroup}的堆积数量达到:<span class="hljs-template-variable">{{$value}}</span>"</span><br><br></code></pre></td></tr></table></figure><h2 id="三-其他"><a href="#三-其他" class="headerlink" title="三 其他"></a>三 其他</h2><h3 id="告警流程示例"><a href="#告警流程示例" class="headerlink" title="告警流程示例"></a>告警流程示例</h3><p>这里以两个节点node1和node2配置了磁盘空间监控为例，空间使用到达阈值则触发告警。<br>（测试过程中可通过生成、删除指定体积的文件来控制空间占用）</p><p>其中配置项如下：</p><ul><li>告警规则:node-disk-usage <ul><li>for 为 1m</li></ul></li><li>告警中心:alertManager <ul><li>group_wait: 30s</li><li>group_interval: 5m</li><li>repeat_interval: 10m</li></ul></li></ul><p>收到的告警短信内容及时间线如下：</p><ul><li>10:23:14收到第一次警报：<br>node1于 10:22:44 进入异常</li><li>10:28:14收到第二次警报：<br>node1于 10:22:44 进入异常<br>node2于 10:24:44 进入异常</li><li>10:38:29收到第三次警报：<br>node1于 10:22:44 进入异常<br>node2于 10:24:44 进入异常</li><li>10:48:44收到第四次警报：<br>node1于 10:22:44 进入异常<br>node2于 10:24:44 进入异常</li><li>10:58:44收到第五次警报：恢复告警<br>node1于 10:22:44 进入异常，并于10:55:44恢复<br>node2于 10:24:44 进入异常，并于10:49:14恢复</li></ul><p>总共收到5次短信，第1次是node1异常，<br>第2到4次是node1和node2都异常，因为属于同个分组group，所以合并发送。<br>第5次是已经恢复正常了。<br>根据短信内容和时间，整理出告警逻辑时间线如下：</p><ol><li>node1等待for <strong>1分钟</strong> 后警报进入group<br>node1记录异常时间为10:22:44，实际异常状态至少在10:22:44的一分钟前</li><li>group等待group_wait <strong>30s</strong> 后发送第一次告警<br>firing:node1</li><li>node2等待for <strong>1分钟</strong> 后警报进入group<br>node2记录异常时间为10:24:44，实际异常状态至少在10:24:44的一分钟前<br>此时group中有两个异常目标node1和node2</li><li>group等待group_interval <strong>5m</strong> 后发送第二次告警<br>firing:node1,node2<br>注意：因为group发生了变化，所以这里用的是group_interval</li><li>group等待repeat_interval <strong>10m</strong> 后发送第三次告警<br>firing:node1,node2<br>注意：因为group没有变化，属于重复告警，用的是repeat_interval</li><li>group等待repeat_interval <strong>10m</strong> 后发送第四次告警<br>firing:node1,node2<br>同上一次</li><li>第四次告警后的 前5分钟： node2恢复正常</li><li>第四次告警后的 后5分钟： node1恢复正常</li><li>group等待repeat_interval <strong>10m</strong> 后发送第五次告警<br>resolved:node1,node2<br>注意，这里node1,node2都恢复正常用的也是repeat_interval</li></ol><p>综上：</p><ul><li>for<br>是告警规则个体的监控配置，用来衡量服务多久检测不通过才算异常</li><li>group_wait 初次发送告警的等待时间<br>用于group创建后的等待，这个值通常设置较小，在几分钟以内</li><li>group_interval 同一个组其他新发生的告警发送时间间隔<br>是group内容发生变化后的告警间隔</li><li>repeat_interval 重复发送同一个告警的时间间隔<br>group内容没有变化且上一次发生成功时用的发生间隔</li></ul><p>需要注意，恢复正常不属于group变化，用的是repeat_interval。<br>这有点反直觉，且个人认为不是很合理，不知道是不是测试有问题，也没有找到比较好的资料说明。<br>希望知道的可以指教一下。</p><h3 id="exporter的位置"><a href="#exporter的位置" class="headerlink" title="exporter的位置"></a>exporter的位置</h3><p>exporter可以以sidecar的形式和原容器放在同一个pod内（1对1），也可以以独立部署的形式存在（1对1/1对多）。<br>这个视具体情况而定，技术上没什么不同，sidecar可以绑定生命周期，视为对原有组件的补充。独立部署则耦合度更低，更灵活。<br>像单节点的mysql，用sidecar则只需要1个pod，不会太复杂。<br>而如果像多节点的kafka集群，用独立部署则只需要一个exporter就可以实现对多个节点的采集监控。</p><p>这里出于减小耦合、节省资源的目的，我主要使用的是独立部署形式。</p><h3 id="使用promtool检查指标格式是否正确"><a href="#使用promtool检查指标格式是否正确" class="headerlink" title="使用promtool检查指标格式是否正确"></a>使用promtool检查指标格式是否正确</h3><p>promtool 使用方法：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 进入pod</span><br> kubectl -n=monitoring exec -it prometheus-k8s-0 sh<br><span class="hljs-meta">#</span><span class="bash"> 查看帮助</span><br>promtool -h<br><span class="hljs-meta">#</span><span class="bash"> 检查指标格式</span><br>curl -s http://ip:9999/metrics | promtool check metrics<br><br></code></pre></td></tr></table></figure><p>比方说 指标name、labelname不能使用小数点</p><h3 id="使用port-forward临时提供Prometheus外部访问"><a href="#使用port-forward临时提供Prometheus外部访问" class="headerlink" title="使用port-forward临时提供Prometheus外部访问"></a>使用port-forward临时提供Prometheus外部访问</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> prometheus</span><br>nohup kubectl port-forward --address 0.0.0.0 service/prometheus-k8s 19090:9090 -n=monitoring &<br><span class="hljs-meta">#</span><span class="bash"> grafana</span><br>nohup kubectl port-forward --address 0.0.0.0 service/grafana 13000:3000 -n=monitoring &<br><span class="hljs-meta">#</span><span class="bash"> alertmanager</span><br>nohup kubectl port-forward --address 0.0.0.0 service/alertmanager-main 9093:9093 -n=monitoring &<br><br></code></pre></td></tr></table></figure><p>用 <code>jobs -l</code> 可以查看</p><h3 id="kube-prometheus对arm的支持"><a href="#kube-prometheus对arm的支持" class="headerlink" title="kube-prometheus对arm的支持"></a>kube-prometheus对arm的支持</h3><p>目标是找到kube-prometheus用到的镜像的arm版本</p><p>可以使用 <a href="https://quay.io/">https://quay.io/</a> 搜索</p><p>以下是用到的镜像和各自对arm的支持情况。<br>注意对照自己使用的版本，很多镜像高版本都支持arm了</p><p>未标记（不支持）其实也是可以用，但不保证</p><p>quay.io/prometheus/prometheus:v2.11.0 支持arm（v2.10.0开始）<br>quay.io/prometheus/alertmanager:v0.18.0 支持arm（v0.17.0开始）<br>quay.io/coreos/kube-state-metrics:v1.8.0 未标记（不支持）<br>quay.io/coreos/kube-rbac-proxy:v0.4.1 未标记（不支持）<br>quay.io/prometheus/node-exporter:v0.18.1 支持arm（v0.18.0开始）<br>quay.io/coreos/prometheus-operator:v0.34.0 不支持arm（v0.39开始） 修改成使用0.39.0，0.39以后的prometheus要求k8s必须>=1.16<br>quay.io/coreos/configmap-reload:v0.0.1 未标记（不支持）<br>grafana/grafana:6.4.3 官方说支持arm，但其实不支持，有bug，见：<a href="https://bleepcoder.com/cn/grafana/501674494/docker-arm-images-doesn-t-work-since-v6-4-x">https://bleepcoder.com/cn/grafana/501674494/docker-arm-images-doesn-t-work-since-v6-4-x</a></p><p>quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0 未找到arm版本镜像<br>最新的官方版本已经改用 directxman12/k8s-prometheus-adapter:v0.8.4<br>而directxman12/k8s-prometheus-adapter:v0.5.0 开始支持arm<br>见：<a href="https://hub.docker.com/r/directxman12/k8s-prometheus-adapter/tags">https://hub.docker.com/r/directxman12/k8s-prometheus-adapter/tags</a></p>]]></content>
<categories>
<category> Kubernetes </category>
</categories>
<tags>
<tag> kubernetes </tag>
<tag> prometheus </tag>
<tag> bigdata </tag>
</tags>
</entry>
<entry>
<title>centos7配置wifi</title>
<link href="//centos7-wifi/"/>
<url>//centos7-wifi/</url>
<content type="html"><![CDATA[<p>自用笔记：笔记本当服务器时配置连接WiFi和关闭合盖休眠操作</p><span id="more"></span><h3 id="准备"><a href="#准备" class="headerlink" title="准备"></a>准备</h3><p>查看 NetworkManager-wifi 是否安装</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">rpm -q NetworkManager-wifi<br><br></code></pre></td></tr></table></figure><p>未安装则先进行临时wifi连接</p><h3 id="临时连接"><a href="#临时连接" class="headerlink" title="临时连接"></a>临时连接</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 1. 查看联网信息</span><br> ip addr<br><span class="hljs-meta">#</span><span class="bash"> 能看到有个 wlp1s0 的无线网卡（也可能是其他名称，自行替换）</span><br><span class="hljs-meta">#</span><span class="bash"> 2. 打开无线网卡驱动</span><br> ip link set wlp1s0 up<br><span class="hljs-meta">#</span><span class="bash"> 无输出则表示正常</span><br><span class="hljs-meta">#</span><span class="bash"> 3. 查看周围wifi</span><br> iw dev wlp1s0 scan | grep SSID<br><span class="hljs-meta">#</span><span class="bash"> 4. 连接wifi</span><br> wpa_supplicant -B -i wlp1s0 -c <(wpa_passphrase "wifi名称" "wifi密码")<br><span class="hljs-meta">#</span><span class="bash"> 会提示 Successfully initialized wpa_supplicant</span><br><span class="hljs-meta">#</span><span class="bash"> 6. 用dhcp获得IP</span><br> dhclient wlp1s0<br><span class="hljs-meta">#</span><span class="bash"> 7. 查看网络状态</span><br> ip addr<br> <br></code></pre></td></tr></table></figure><h3 id="设置永久自动连接wifi"><a href="#设置永久自动连接wifi" class="headerlink" title="设置永久自动连接wifi"></a>设置永久自动连接wifi</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 安装NetworkManager-wifi</span><br>yum -y install NetworkManager-wifi<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 重启NetworkManager</span><br>systemctl restart NetworkManager<br><span class="hljs-meta">#</span><span class="bash"> 开启wifi</span><br>nmcli r wifi on<br><span class="hljs-meta">#</span><span class="bash"> 查看wifi</span><br>nmcli dev wifi<br><span class="hljs-meta">#</span><span class="bash"> 连接wifi</span><br>nmcli d wifi connect "wifi名称" password "wifi密码"<br><span class="hljs-meta">#</span><span class="bash"> Error: Connection activation failed:(7) Secrets were required,but not provided.</span><br><span class="hljs-meta">#</span><span class="bash"> 这一步可能会报错，reboot一下，再执行即可</span><br><span class="hljs-meta">#</span><span class="bash"> 再查看网络状态，显示已经连上了</span><br>ip addr<br><br></code></pre></td></tr></table></figure><h2 id="设置wifi静态ip"><a href="#设置wifi静态ip" class="headerlink" title="设置wifi静态ip"></a>设置wifi静态ip</h2><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 查看DNS</span><br>cat /etc/resolv.conf<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 查看GATEWAY</span><br>netstat -rn<br><span class="hljs-meta">#</span><span class="bash"> 修改网络连接配置文件</span><br>/etc/sysconfig/network-scripts/ifcfg-WIFI连接名称<br><span class="hljs-meta">#</span><span class="bash"> 修改BOOTPROTO=static</span><br><span class="hljs-meta">#</span><span class="bash"> 添加配置：</span><br><span class="hljs-meta">#</span><span class="bash"> IPADDR=静态ip</span><br><span class="hljs-meta">#</span><span class="bash"> NETMASK=255.255.255.0</span><br><span class="hljs-meta">#</span><span class="bash"> DNS1=上面看到的DNS</span><br><span class="hljs-meta">#</span><span class="bash"> DNS2=上面看到的DNS或8.8.8.8</span><br><span class="hljs-meta">#</span><span class="bash"> GATEWAY=上面看到的GATEWAY</span><br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 重启</span><br>reboot<br><br><br></code></pre></td></tr></table></figure><h3 id="设置笔记本合盖不休眠"><a href="#设置笔记本合盖不休眠" class="headerlink" title="设置笔记本合盖不休眠"></a>设置笔记本合盖不休眠</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell"><br>vim /etc/systemd/logind.conf<br><span class="hljs-meta">#</span><span class="bash"> 设置 HandleLidSwitch=lock</span><br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> 重启服务使配置生效</span><br>systemctl restart systemd-logind <br><br></code></pre></td></tr></table></figure><ul><li>ignore 忽略，跳过</li><li>power off 关机</li><li>eboot 重启</li><li>halt 挂起</li><li>suspend shell内建指令，可暂停目前正在执行的shell。若要恢复，则必须使用SIGCONT信息。所有的进程都会暂停，但不是消失（halt是进程关闭）</li><li>hibernate 让笔记本进入休眠状态</li><li>hybrid-sleep 混合睡眠，主要是为台式机设计的，是睡眠和休眠的结合体，当你选择Hybird时，系统会像休眠一样把内存里的数据从头到尾复制到硬盘里 ，然后进入睡眠状态，即内存和CPU还是活动的，其他设置不活动，这样你想用电脑时就可以快速恢复到之前的状态了，笔记本一般不用这个功能。</li><li>lock 仅锁屏，计算机继续工作。</li></ul>]]></content>
<categories>
<category> 运维部署 </category>
</categories>
</entry>
<entry>
<title>基于k8s的家用大数据集群设计与实现</title>
<link href="//household_k8s_bigdata/"/>
<url>//household_k8s_bigdata/</url>
<content type="html"><![CDATA[<p>使用3台废旧笔记本搭建k8s集群，部署大数据组件，利用路由器进行异地组网，配合wsl作为管理和客户端，<br>实现随时随地，在工作笔记本上以本地访问的体验使用自建家庭大数据平台进行学习、开发、测试。</p><span id="more"></span><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><h3 id="起因"><a href="#起因" class="headerlink" title="起因"></a>起因</h3><p>最近换了电脑，之前的机器闲置着浪费，又刚好看到了<a href="https://github.com/geekyouth/SZT-bigdata">https://github.com/geekyouth/SZT-bigdata</a> 这个项目，不错不错，让我来可以做得更好。</p><p>索性自己在家里搭个大数据集群来玩。而且女朋友的笔记本也闲置着，再搜刮一台来组个集群，就可以为所欲为了。</p><h3 id="最初目标"><a href="#最初目标" class="headerlink" title="最初目标"></a>最初目标</h3><p>在主力机的wsl里可以提交flink任务，在idea的单元测试里可以配置wsl执行hadoop操作。<br>这样基本就可以满足日常的开发测试需求了。</p><h2 id="物理配置及组网"><a href="#物理配置及组网" class="headerlink" title="物理配置及组网"></a>物理配置及组网</h2><h3 id="机器"><a href="#机器" class="headerlink" title="机器"></a>机器</h3><p>服务器：lian1、lian2、lian3 </p><p>蒲公英家用路由器（带内网异地组网功能）</p><p>客户端：工作用笔记本（win11，带wsl）</p><h3 id="网络配置"><a href="#网络配置" class="headerlink" title="网络配置"></a>网络配置</h3><p>3台服务器利用网线连接路由器，组成一个局域网。</p><p>在家的时候笔记本直接通过wifi连接到局域网，在外也还可以通过蒲公英的异地组网功能连接。忽略网速的话体验一致。</p><p>注意：也可以通过一些内网穿透的软件达到相同效果，不推荐专门去买路由器。</p><h2 id="技术选型"><a href="#技术选型" class="headerlink" title="技术选型"></a>技术选型</h2><h3 id="大数据平台选型"><a href="#大数据平台选型" class="headerlink" title="大数据平台选型"></a>大数据平台选型</h3><p>本来只是出于实验目的，想直接用cdh，方便管理一些。</p><p>CDH6.3.3之后没有免费社区版了。我之前还部署了一套6.3.2的。<br>但是从2021年2月开始下载之前的版本也要订阅了，很麻烦。<br>算了，人家就是不想让你用免费的，而且免费的组件版本还老，不用也罢。</p><p>然后我就开始物色其他的，hdp也早被cdh收编了，mapr本来就是主打商用的。</p><p>国内的华为阿里腾讯都是云服务，不会提供这种社区版。</p><p>倒是有看到UCloud出了一个USDP的大数据平台免费版。有点意思。<br>然后看了一遍部署教程和操作文档。嗯。。。再看一眼组件版本，嗯。。。</p><p>算了，还是用我自己的基于k8s的方法部署吧</p><h3 id="k8s部署管理平台选型"><a href="#k8s部署管理平台选型" class="headerlink" title="k8s部署管理平台选型"></a>k8s部署管理平台选型</h3><p>kubernetes相关部署管理工具较多，我是不推荐自己手动一步步部署的</p><p>这里我用的是： kubeoperator</p><p>使用wsl-docker作为kubeoperator的管理端，将k8s部署到3台机器上<br>平时需要用到管理功能的时候再启动win11上的docker就行，日常运行不需要</p><p>按公网教程一步步操作即可，这里有个坑：<br>主机的ip只能填ip不能填hostname，虽然填的时候不会报错，但是在生成k8s证书的时候会失败！</p><h2 id="资源分配"><a href="#资源分配" class="headerlink" title="资源分配"></a>资源分配</h2><p>3台机器本来就是闲置机，唯一配置好一点的前主力机的32g内存也被我拆到现在的主力机上了。<br>机器配置都是4核，但内存加起来才32g，可以说是非常垃圾，但是受限于经济能力，没办法。<br>不过一想到云服务器1c2g也被我跑了那么多服务，12c32g好像也还不错了，至少比在自己笔记本上开3台虚拟机强。</p><p>经常看到群里出于学习要搭大数据的，怕自己配置不够。<br>够不够其实看需求，只是做功能测试、验证的，没多大数据量，耗不了多少资源。<br>再说，就算数据量大，只要控制好，反正最后都是在yarn上面跑，无非就是慢一点，总是能跑完，挂不了。<br>更何况我还是放在kubernetes上的，就算一个挂了也影响不了其他。这点比cdh脚本部署强。</p><p>下面来看各个组件的安排</p><p>物理机上直接启用的有： nfs<br>docker直接启用的有： docker镜像仓库（docker-registry、docker-registry-web-server）、mysql、 ldap（openldap、ldapadmin）<br>kubernetes管理的有：</p><h3 id="1-kerberos"><a href="#1-kerberos" class="headerlink" title="1. kerberos"></a>1. kerberos</h3><p>用于大数据组件的认证，支持1主多从，这里给1主1从，毕竟是最基础的组件，就象征性地高可用吧<br>配置的话1c1g就够了，这里的资源值是指limit而非request，即最高能申请到的内存，实际用到的会低很多</p><h3 id="2-zookeeper"><a href="#2-zookeeper" class="headerlink" title="2. zookeeper"></a>2. zookeeper</h3><p>分布式协调，一般是其他组件实现高可用的基础。至少3台，1c1g。</p><h3 id="3-elasticsearch"><a href="#3-elasticsearch" class="headerlink" title="3. elasticsearch"></a>3. elasticsearch</h3><p>搜索引擎，这个是给ranger存储审计日志用的。支持多个，但1个就够了，1c1g。</p><h3 id="4-ranger"><a href="#4-ranger" class="headerlink" title="4. ranger"></a>4. ranger</h3><p>用于大数据权限管理，主要跑两个程序：<br>ranger-admin提供ranger核心服务和web管理功能<br>ranger-usersync负责从ldap同步用户账户信息</p><h3 id="5-hdfs"><a href="#5-hdfs" class="headerlink" title="5. hdfs"></a>5. hdfs</h3><p>大数据最基本的存储组件，因为是高可用部署，得区分好几种角色（这里用的时hadoop最新版本3.3.1的架构配置）。</p><ul><li>zkfc-format:用于第一次运行时格式化hdfs在zookeeper上的存储，一次性运行，故配置不重要</li><li>jn:journalnode，日志节点，hdfs内部的高可用机制的实现，至少3台，1c1g</li><li>nn-active：active-namenode，默认状态为active的名称节点，即默认主节点，1个，在第一次启动时负责做初始化操作，1c2g</li><li>nn-standby:standby-namenode，默认状态为standby的名称节点，即默认从节点，支持多个，参与主节点选举以实现高可用。这里也给1个，1c2g</li><li>nn-observer:observer-namenode，状态为observer的名称节点，即观察者节点，负责读写分离的读的部分，不参与主节点选举。这里直接不给，省点资源。</li><li>dn:datanode，数据节点，至少3个，1c2g。</li></ul><p>虽然把nn-observer给省略了，但实际上hadoop2的时候namenode也只支持两个，所以其实现在这样从设计上来说也很不错了。<br>一般给1c1g是因为我知道它跑不到1g，1c2g是因为内部进程jvm参数容易配置些。</p><h3 id="6-yarn"><a href="#6-yarn" class="headerlink" title="6. yarn"></a>6. yarn</h3><p>大数据最基本的资源调度组件</p><ul><li>rm:resource-manager,资源管理器，支持任意多个实现高可用，这里给2个，1c2g</li><li>nm：node-manager,节点管理器，支持任意多个横向拓展，这里给1个，3c8g。</li></ul><p>注意，nm的3c8g里面2c6g是给yarn平台调度用的，1g是给nm进程本身的，其他的预留给容器本身的。<br>所以2c6g就是这个大数据平台的计算任务能调度的最多资源了…</p><h3 id="其他"><a href="#其他" class="headerlink" title="其他"></a>其他</h3><p>到yarn就已经把大数据最最基础的部分搭建好了。<br>其他组件按需启动就行，像hive、hbase这些有主从多节点高可用结构的我暂时没用到，又耗资源，就先不管<br>像flink、spark这种支持把任务放到yarn上面跑的，本身留个一个历史日志服务器节点就够了，甚至不需要。<br>另外用得比较多就是kafka，一般也是至少3台，1台也可以，先不管</p><h3 id="汇总"><a href="#汇总" class="headerlink" title="汇总"></a>汇总</h3><table><thead><tr><th>hostname</th><th>cpu核数</th><th>内存（g）</th><th>组件</th><th>limit资源总和</th></tr></thead><tbody><tr><td>lian1</td><td>4</td><td>16</td><td>kerberos-master、zookeeper、es、ranger、hdfs-jn、hdfs-nn-active、hdfs-dn、yarn-rm、yarn-nm</td><td>12c19g</td></tr><tr><td>lian2</td><td>4</td><td>12</td><td>kerberos-slave、zookeeper、hdfs-jn、hdfs-nn-standby、hdfs-dn、yarn-rm</td><td>6c9g</td></tr><tr><td>lian3</td><td>4</td><td>4</td><td>zookeeper、hdfs-jn、hdfs-dn</td><td>3c3g</td></tr></tbody></table><p>可以看到limit的资源总和是超过实际资源的，但是只是我为了方便分配和设置而已<br>大部分进程cpu用得都是很低的，内存不跑任务的时候也不多<br>毕竟条件所限，这样子也不错了<br>接下来就是实际测试了</p>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> 生产力 </tag>
<tag> 大数据 </tag>
<tag> WSL </tag>
</tags>
</entry>
<entry>
<title>个人博客系统设计(支持hexo和halo同步)</title>
<link href="//my-blog-system-design/"/>
<url>//my-blog-system-design/</url>
<content type="html"><![CDATA[<p>本文主要介绍自己的博客系统是如何设计的，并使用<a href="https://github.com/linshenkx/haloSyncServer">Halo博客同步器</a> 将hexo（git pages: <a href="https://linshenkx.github.io/">https://linshenkx.github.io</a> ）文章自动同步到halo( <a href="http://linshenkx.cn/">http://linshenkx.cn</a> )。<br>实现一次编写、两套博客系统并存、多个网址访问的效果。<br>2023 更新：放弃halo，单纯使用hexo<br>2022.11 更新：因其审核问题删除gitee个人主页（无标准无原因，你永远不知道为什么无法发布）</p><span id="more"></span><h2 id="一-总览"><a href="#一-总览" class="headerlink" title="一 总览"></a>一 总览</h2><h3 id="达到效果"><a href="#达到效果" class="headerlink" title="达到效果"></a>达到效果</h3><table><thead><tr><th>个人博客网址</th><th>介绍</th><th>对应git仓库/管理界面</th></tr></thead><tbody><tr><td><a href="https://linshenkx.github.io/">https://linshenkx.github.io</a></td><td>hexo next github pages</td><td><a href="https://github.com/linshenkx/linshenkx.github.io">https://github.com/linshenkx/linshenkx.github.io</a></td></tr><tr><td><a href="https://linshen.netlify.app/">https://linshen.netlify.app</a></td><td>netlify加速，文章同步自blog源码仓库</td><td><a href="https://app.netlify.com/teams/linshenkx">https://app.netlify.com/teams/linshenkx</a></td></tr><tr><td><a href="https://linshenkx.cn/">https://linshenkx.cn</a></td><td>halo个人网站，文章同步自blog源码仓库</td><td><a href="https://linshenkx.cn/admin/index.html#/dashboard">https://linshenkx.cn/admin/index.html#/dashboard</a></td></tr></tbody></table><p>blog博客源码仓库（核心，私有）：<a href="https://github.com/linshenkx/blog">https://github.com/linshenkx/blog</a></p><h3 id="博客发布流程"><a href="#博客发布流程" class="headerlink" title="博客发布流程"></a>博客发布流程</h3><ol><li>编写博客<br>在blog工程下写博客，工程为标准hexo，博客为markdown文件放在source/_posts目录下，使用多层级分类存放</li><li>发布到git pages<br>完成博客的增删改后，在工程目录下执行<code>hexo clean && hexo d -g</code>部署到git pages。<br>这里我配置了同时发布到github和gitee，需要注意的是，gitee的git pages需要手动去触发更新才能生效。</li><li>提交并推送工程<br>提交并推送blog工程的修改。<br>netlify将自动获取blog工程，并执行hexo部署脚本（效果和git pages一样，只是用netlify访问据说会快一点）<br>自己开发的<a href="https://github.com/linshenkx/haloSyncServer">Halo博客同步器</a>也会检测到blog工程更新，根据更新情况将变化同步到halo博客系统中。</li></ol><h2 id="二-设计思路"><a href="#二-设计思路" class="headerlink" title="二 设计思路"></a>二 设计思路</h2><h3 id="1-起因"><a href="#1-起因" class="headerlink" title="1 起因"></a>1 起因</h3><p>本来我一直是在使用csdn的，但是网页端写作确实不方便，而且还可能受网络情况限制。<br>所以我后面一般都是用印象笔记做记录，在印象笔记写好再看心情整理到csdn上去。<br>但是悄不注意的，在21年初csdn改版，同时也改变了排名和引流规则。<br>之前一个星期2500到3000的访问量现在只剩1500到2000了。</p><p>嗯，不可忍。换。</p><h3 id="2-调研"><a href="#2-调研" class="headerlink" title="2 调研"></a>2 调研</h3><p>市面上的博客系统可根据对Git Pages的支持（即是否支持生成静态网站）分为两大类：</p><p>一是以hexo为代表的静态网站生成器：如hexo、hugo、jekyll，较成熟，有较多第三方主题和插件，可与git pages搭配使用，也可自行部署。</p><p>二是以halo为代表的五花八门的个人博客系统，功能更加强大，自由度更高，通常带后台管理，但不支持git pages，需自行部署。</p><h3 id="3-分析"><a href="#3-分析" class="headerlink" title="3 分析"></a>3 分析</h3><p>个人博客的话使用git pages比较稳定，网址固定，可以永久使用，而且可以通过搭配不同的git服务商来保证访问速度。<br>但是git pages的缺点也很明显，是静态网站，虽然可以搭配第三方插件增强，但说到底还是个静态网站。</p><p>而如果自己买服务器，买域名，用第三方个人博客系统，就可以玩得比较花里胡哨了，但谁知道会用多久呢。<br>服务器、域名都要自己负责，三五年之后还能不能访问就比较难说了。<br>但是年轻人嘛，总还是花里胡哨点才香。</p><p>那我就全都要。</p><p>git pages作为专业性较强的个人网站可以永久访问，<br>然后再弄个服务器放个博客系统自己玩。</p><h3 id="4-选型"><a href="#4-选型" class="headerlink" title="4 选型"></a>4 选型</h3><p>静态网站生成器选的是hexo，传统一点，支持的插件和主题比较多。<br>hugo虽然也不错，但似乎国内用的不多，支持可能还不够完善。</p><p>然后hexo的主题用的最经典的next，比较成熟，功能也很完善<br>虽然整体比较严肃压抑，但可以自己加个live2d增添点活力，<br>作为一个展示专业性的博客网站这样也就够了</p><p>自定义博客系统的话我选的是halo，最主要原因是它是java写的，利于二次开发（事实上后面用着也确实有问题，还提交了一个issue）<br>而且功能比较强大，生态比较完善，虽然第三方主题少且基本都没更新，但是…实在是找不出其他一个能打的了<br>另外halo支持导入markdown，且功能基本都通过rest接口放开，适合开发者使用</p><h2 id="三-设计实现"><a href="#三-设计实现" class="headerlink" title="三 设计实现"></a>三 设计实现</h2><h3 id="1-hexo"><a href="#1-hexo" class="headerlink" title="1 hexo"></a>1 hexo</h3><p>hexo本身只是静态网站生成器，你可以把hexo项目本身发布成为git pages项目，<br>像github、gitee这些会识别出这是一个hexo项目，然后进行编译，得到静态资源供外部访问。<br>这也是最简单的用法。</p><p>但是不推荐。</p><p>因为git pages项目一般都要求是public的（且名称固定，一个git账号只有一个git pages仓库），<br>hexo项目包含你的博客markdown源文件和其他的个人信息。<br>我们只是想把必要的生成后的静态网页放出去而已，至于项目的配置信息和markdown源文件应该藏起来。</p><p>所以需要使用 hexo-deployer-git 插件进行git pages的部署。<br>即放到git公开的文件只有生成后的网页文件而已，git只是把你生成后的index.html进行直接展示，不会再去编译了<br>（需要在source目录下添加.nojekyll文件表明为静态网页，无须编译）</p><p>而项目本身为了更好地进行管理和记录，还是要发布到git上面的，作为一个普通的私有仓库，名称可以任意（如 blog）</p><p>这样，每次要增删改完文章只需要执行<code>hexo clean && hexo d -g</code>即可发布到git仓库上<br>注意，不同git服务商git pages规则不一样。<br>比方说我gitee和github的用户名都是linshenkx<br>但是gitee要求的仓库名是linshenkx，而github的仓库名就必须是linshenkx.github.io了<br>而github的git pages仓库在接收到推送后就自动（编译）部署<br>gitee则需要到仓库web界面手动触发更新</p><p>截至到这一步是大多数人的做法，即git上两个仓库并存，一（或多）个git pages公有仓库做展示，一个blog仓库存放博客源码<br>注意：如果git pages仓库允许私有，则可以使用一个仓库多个分支来实现相同效果。<br>但还是推荐使用两个仓库，因为这样更通用，设计上也更合理。</p><p>工程总体结构如下，为普通hexo工程：<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1631718173(1).png"><br>博客源码目录结构如下，为多层级结构：<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1631718305(1).png"></p><h3 id="2-halo"><a href="#2-halo" class="headerlink" title="2 halo"></a>2 halo</h3><p>halo的使用看官方文档一般就够了，这里需要补充的是其代理配置。<br>因为halo的在线下载更新主题功能通常需要连接到github，我习惯通过代理访问<br>这里提供一下配置方法<br>即在容器启动时添加JVM参数即可</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker run -it -d --name halo --network host -e JVM_OPTS="-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=7890 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=7890" -v /opt/halo/workspace:/root/.halo --restart=always halohub/halo<br><br></code></pre></td></tr></table></figure><h3 id="3-markdown图片"><a href="#3-markdown图片" class="headerlink" title="3 markdown图片"></a>3 markdown图片</h3><p>markdown图片的存放一直是个麻烦的问题。<br>最害怕遇到就是图链的失效，而且往往自己还不能发现。<br>理想状态下就是markdown一张图片支持配置多个图床链接，第一个图床链接超时就使用下一个。<br>这种服务端的处理思想很明显不适合放到客户端。<br>退而求其次，配置一个链接，访问这个链接会触发对多个图床的访问，然后那个快用那个。<br>这个效果技术上不难实现，也有个商业产品（聚合图床）是这样的，缺点是收费。<br>然后我又在github、gitee上找了各个图床软件，都不怎么样（这个时间成本都够给聚合图床开几年会员了）<br>最终还是妥协，用云存储吧，选了阿里<br>七牛、腾讯也都试了，其实都差不多，看个人爱好，没有太特别的理由</p><p>如果你用typora写markdown的话很方便，它支持picgo插件</p><p>但我习惯在idea里面编写，idea也有一些markdown-image插件，基本都不好用<br>所以我还是安装了picgo，开了快捷键，复制图片直接快捷键粘贴体验也还是比较舒服的<br>picgo的特点是插件多，不过插件质量一般，有很多bug</p><p>花了两天时间纠结、测试，最后的方案是：idea编辑+阿里云存储+picgo上传</p><h3 id="4-同步"><a href="#4-同步" class="headerlink" title="4 同步"></a>4 同步</h3><p>这才是重点</p><h4 id="1-同步的方向"><a href="#1-同步的方向" class="headerlink" title="1 同步的方向"></a>1 同步的方向</h4><p>即在哪里写文章，同步到哪里</p><p>我还是习惯用idea写markdown文档而不是在网页上。<br>所以确定是流向为 hexo->halo</p><h4 id="2-技术支撑"><a href="#2-技术支撑" class="headerlink" title="2 技术支撑"></a>2 技术支撑</h4><p>halo支持导入markdown文件，所以主要问题为hexo的markdown博客源码文件的获取<br>hexo文章存储路径为 source/_posts ，有多层级文件夹，可以简单地理解成文件IO操作获取文章内容。<br>但关键是存储在git上，这里可以用JGit进行操作。<br>同时，JGit支持获取两次commit之间的文件变化情况。<br>即可以捕获到文章的增删改操作，而不用每次都全量地同步。</p><h4 id="3-成果"><a href="#3-成果" class="headerlink" title="3 成果"></a>3 成果</h4><p>又处理了一些细节问题，最终还是自己做了个haloSyncServer同步程序，<br>封装成docker，放服务器上跑，实现同步。<br>待整理后开源。</p><p>2021年11月更新<br>开源地址为：<a href="https://github.com/linshenkx/haloSyncServer">https://github.com/linshenkx/haloSyncServer</a><br>效果<br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/1637394238(1).jpg"></p>]]></content>
<categories>
<category> 程序员杂记 </category>
</categories>
<tags>
<tag> 生产力 </tag>
<tag> 博客 </tag>
<tag> hexo </tag>
<tag> halo </tag>
</tags>
</entry>
<entry>
<title>ubuntu制作离线源</title>
<link href="//ubuntu-offline-sources/"/>
<url>//ubuntu-offline-sources/</url>
<content type="html"><![CDATA[<p>自用笔记：ubuntu（18.04）制作离线源</p><span id="more"></span><h2 id="0-前言"><a href="#0-前言" class="headerlink" title="0 前言"></a>0 前言</h2><p>参考：<a href="https://blog.csdn.net/yruilin/article/details/85124870">https://blog.csdn.net/yruilin/article/details/85124870</a></p><h2 id="1-在外网机器"><a href="#1-在外网机器" class="headerlink" title="1 在外网机器"></a>1 在外网机器</h2><h3 id="1-安装gpg软件和相关软件"><a href="#1-安装gpg软件和相关软件" class="headerlink" title="1 安装gpg软件和相关软件"></a>1 安装gpg软件和相关软件</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt-get install gnupg<br>apt-get install rng-tools<br><br></code></pre></td></tr></table></figure><p>密钥创建过程中，需要使用到足够的随机数(random)，可先行安装rng-tools, 该工具可以常驻后台的方式, 生成随机数，避免gpg密钥创建过程中的长时间等待问题 </p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">rngd -r /dev/urandom<br></code></pre></td></tr></table></figure><p>生成公钥和私钥：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">gpg --gen-key<br></code></pre></td></tr></table></figure><p>执行gpg会进入一些对话，其中要新建一个用户名username和相应的密码。<br>如 linshen <a href="mailto:linshen@qq.com">linshen@qq.com</a> 12345678</p><p>结束之后，输入命令，可以查看key：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">gpg --list-key<br></code></pre></td></tr></table></figure><p>导出gpg公钥和私钥：私钥，供Server端，对release文件签名使用，好像不做也能签名</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">gpg -a --export-secret-keys linshen > Ubuntu_Local_Archive_Automatic_Signing_Key_2017.sec<br></code></pre></td></tr></table></figure><p>公钥，需在Ubuntu client 导入，供apt-get使用</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">gpg -a --export linshen> linshen.pub<br></code></pre></td></tr></table></figure><h3 id="2-准备安装包源"><a href="#2-准备安装包源" class="headerlink" title="2 准备安装包源"></a>2 准备安装包源</h3><p>清空缓存目录，这一步也可以不做</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">rm -rf /var/cache/apt/archives/*<br></code></pre></td></tr></table></figure><p>-d只是下载安装包，并不安装。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt-get -d install -y <包名><br></code></pre></td></tr></table></figure><p>如 mysql-server ntp nfs-kernel-server portmap nfs-common</p><p>在本地建一个目录，将下载下来的安装包拷贝到此目录：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir /var/debs<br>cp -r /var/cache/apt/archives/*.deb /var/debs/<br></code></pre></td></tr></table></figure><p>在debs这个目录创建Packages.gz，注意生成的路径带debs，否则内网安装时会说找不到文件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell"><br>cd /var<br>apt-ftparchive packages debs > debs/Packages<br>cd debs<br>gzip -c Packages > Packages.gz<br><br></code></pre></td></tr></table></figure><p>在debs这个目录下创建release file</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt-ftparchive release ./ > Release<br></code></pre></td></tr></table></figure><p>ubuntu apt-get 对软件包索引，首先要求InRelease文件，其次才去找Release、Release.gpg文件； 这情况下， 其实只需要创建InRelease文件(包含Release文件和明文签名)即可:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">gpg --clearsign -o InRelease Release <br>gpg -abs -o Release.gpg Release <br></code></pre></td></tr></table></figure><h2 id="2-拷贝文件"><a href="#2-拷贝文件" class="headerlink" title="2 拷贝文件"></a>2 拷贝文件</h2><p>将外网机器的debs目录和公钥文件 linshen.pub拷贝到内网机器上</p><h2 id="3-内网机器"><a href="#3-内网机器" class="headerlink" title="3 内网机器"></a>3 内网机器</h2><p>导入公钥</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt-key add linshen.pub<br></code></pre></td></tr></table></figure><p>将debs目录拷贝到/home/lin下面</p><p>备份apt源文件/etc/apt/source.list,并修改源</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">cp /etc/apt/sources.list /etc/apt/sources.list.bak<br>echo "deb [trusted=yes] file:/home/lin debs/ " >/etc/apt/sources.list<br></code></pre></td></tr></table></figure><p>更新索引</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt-get update<br></code></pre></td></tr></table></figure><p>至此，可以在内网机器上执行apt-get install 命令了<br>如</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt-get install mysql-server<br>apt-get install nfs-kernel-server portmap<br>apt-get install nfs-common<br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 运维部署 </category>
</categories>
</entry>
<entry>
<title>Ranger2.1编译笔记</title>
<link href="//ranger-compile/"/>
<url>//ranger-compile/</url>
<content type="html"><![CDATA[<p>本文记录了 ranger2.1版本的编译过程，与遇到的一些bug的解决方法。</p><span id="more"></span><h2 id="准备"><a href="#准备" class="headerlink" title="准备"></a>准备</h2><h3 id="1-外部工具依赖"><a href="#1-外部工具依赖" class="headerlink" title="1 外部工具依赖"></a>1 外部工具依赖</h3><p>需要linux环境，保证python命令可用，如果只装了python3，推荐</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt install -y python-is-python3<br><br></code></pre></td></tr></table></figure><p>另外还需要安装gcc</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">apt install gcc -y<br><br></code></pre></td></tr></table></figure><h3 id="2-排除kylin模块"><a href="#2-排除kylin模块" class="headerlink" title="2 排除kylin模块"></a>2 排除kylin模块</h3><p>kylin2.6.4依赖calcite-linq4j-1.16.0-kylin-r2<br>这个jar包在外部maven仓库是找不到的，导致build失败，参考：<br><a href="https://issues.apache.org/jira/browse/RANGER-2994">https://issues.apache.org/jira/browse/RANGER-2994</a><br><a href="https://issues.apache.org/jira/browse/RANGER-2999">https://issues.apache.org/jira/browse/RANGER-2999</a></p><p>因为是kylin本身的依赖问题，不是ranger的锅，这里谴责一下kylin然后对其选择性放弃<br>放弃方法：在pom.xml将其所有的module注释掉，如下</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs xml"><span class="hljs-comment"><!-- <module>plugin-kylin</module>--></span><br><span class="hljs-comment"><!-- <module>ranger-kylin-plugin-shim</module>--></span><br></code></pre></td></tr></table></figure><h3 id="3-ranger-examples-distro错误"><a href="#3-ranger-examples-distro错误" class="headerlink" title="3 ranger-examples-distro错误"></a>3 ranger-examples-distro错误</h3><p>参考：<a href="https://github.com/apache/ranger/pull/71">https://github.com/apache/ranger/pull/71</a><br>根据issue，修改 sampleapp.xml 和 plugin-sampleapp.xml</p><h3 id="4-ranger-client"><a href="#4-ranger-client" class="headerlink" title="4 ranger client"></a>4 ranger client</h3><p>官方的rangerClient有bug，新版本已修复，但还没发布，需要自行编译<br>步骤：</p><ol><li>参考官方仓库 ranger/intg/src/main/java 工程，可直接搬运过来，原来的目录要先删掉</li><li>将 ranger/intg/pom.xml 中 ranger-plugins-common 的版本从${project.version} 改为2.1.0（即公网仓库能找到的最新版本）</li></ol><h2 id="执行编译"><a href="#执行编译" class="headerlink" title="执行编译"></a>执行编译</h2><h3 id="1-编译各个组件-tar-gz包"><a href="#1-编译各个组件-tar-gz包" class="headerlink" title="1 编译各个组件 tar.gz包"></a>1 编译各个组件 tar.gz包</h3><p>在ranger工程下执行，执行成功可以在target下看到包</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">mvn clean compile package assembly:assembly -DskipTests -DskipJSTests -Dpmd.skip=true<br><br></code></pre></td></tr></table></figure><p>日志参考：</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br></pre></td><td class="code"><pre><code class="hljs log">[INFO] ------------------------------------------------------------------------<br>[INFO] Reactor Summary for ranger 2.1.1-SNAPSHOT:<br>[INFO] <br>[INFO] ranger ............................................. SUCCESS [02:31 min]<br>[INFO] Jdbc SQL Connector ................................. SUCCESS [ 12.635 s]<br>[INFO] Credential Support ................................. SUCCESS [ 14.241 s]<br>[INFO] Audit Component .................................... SUCCESS [ 40.494 s]<br>[INFO] ranger-plugin-classloader .......................... SUCCESS [ 8.902 s]<br>[INFO] Common library for Plugins ......................... SUCCESS [02:43 min]<br>[INFO] ranger-intg ........................................ SUCCESS [ 15.121 s]<br>[INFO] Installer Support Component ........................ SUCCESS [ 4.590 s]<br>[INFO] Credential Builder ................................. SUCCESS [ 7.686 s]<br>[INFO] Embedded Web Server Invoker ........................ SUCCESS [ 12.830 s]<br>[INFO] Key Management Service ............................. SUCCESS [ 52.264 s]<br>[INFO] HBase Security Plugin Shim ......................... SUCCESS [ 15.598 s]<br>[INFO] HBase Security Plugin .............................. SUCCESS [ 32.827 s]<br>[INFO] Hdfs Security Plugin ............................... SUCCESS [ 34.502 s]<br>[INFO] Hive Security Plugin ............................... SUCCESS [ 35.109 s]<br>[INFO] Knox Security Plugin Shim .......................... SUCCESS [ 11.906 s]<br>[INFO] Knox Security Plugin ............................... SUCCESS [ 49.254 s]<br>[INFO] Storm Security Plugin .............................. SUCCESS [ 23.171 s]<br>[INFO] YARN Security Plugin ............................... SUCCESS [ 16.011 s]<br>[INFO] Ozone Security Plugin .............................. SUCCESS [ 16.418 s]<br>[INFO] Ranger Util ........................................ SUCCESS [01:43 min]<br>[INFO] Unix Authentication Client ......................... SUCCESS [ 8.165 s]<br>[INFO] Security Admin Web Application ..................... SUCCESS [21:33 min]<br>[INFO] KAFKA Security Plugin .............................. SUCCESS [ 21.139 s]<br>[INFO] SOLR Security Plugin ............................... SUCCESS [ 16.487 s]<br>[INFO] NiFi Security Plugin ............................... SUCCESS [ 18.495 s]<br>[INFO] NiFi Registry Security Plugin ...................... SUCCESS [ 15.196 s]<br>[INFO] Kudu Security Plugin ............................... SUCCESS [ 9.105 s]<br>[INFO] Unix User Group Synchronizer ....................... SUCCESS [ 47.248 s]<br>[INFO] Ldap Config Check Tool ............................. SUCCESS [ 8.631 s]<br>[INFO] Unix Authentication Service ........................ SUCCESS [ 9.789 s]<br>[INFO] Unix Native Authenticator .......................... SUCCESS [ 4.242 s]<br>[INFO] KMS Security Plugin ................................ SUCCESS [ 34.582 s]<br>[INFO] Tag Synchronizer ................................... SUCCESS [ 28.172 s]<br>[INFO] Hdfs Security Plugin Shim .......................... SUCCESS [ 7.847 s]<br>[INFO] Hive Security Plugin Shim .......................... SUCCESS [ 13.040 s]<br>[INFO] YARN Security Plugin Shim .......................... SUCCESS [ 10.582 s]<br>[INFO] OZONE Security Plugin Shim ......................... SUCCESS [ 16.250 s]<br>[INFO] Storm Security Plugin shim ......................... SUCCESS [ 10.807 s]<br>[INFO] KAFKA Security Plugin Shim ......................... SUCCESS [ 8.435 s]<br>[INFO] SOLR Security Plugin Shim .......................... SUCCESS [ 10.037 s]<br>[INFO] Atlas Security Plugin Shim ......................... SUCCESS [ 9.938 s]<br>[INFO] KMS Security Plugin Shim ........................... SUCCESS [ 13.735 s]<br>[INFO] ranger-examples .................................... SUCCESS [ 2.577 s]<br>[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [ 13.963 s]<br>[INFO] Ranger Examples - SampleApp ........................ SUCCESS [ 14.751 s]<br>[INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [ 9.067 s]<br>[INFO] sample-client ...................................... SUCCESS [ 21.965 s]<br>[INFO] Apache Ranger Examples Distribution ................ SUCCESS [ 17.195 s]<br>[INFO] Ranger Tools ....................................... SUCCESS [ 32.274 s]<br>[INFO] Atlas Security Plugin .............................. SUCCESS [ 15.610 s]<br>[INFO] SchemaRegistry Security Plugin ..................... SUCCESS [ 33.643 s]<br>[INFO] Sqoop Security Plugin .............................. SUCCESS [ 32.979 s]<br>[INFO] Sqoop Security Plugin Shim ......................... SUCCESS [ 8.851 s]<br>[INFO] Presto Security Plugin ............................. SUCCESS [ 36.558 s]<br>[INFO] Presto Security Plugin Shim ........................ SUCCESS [01:06 min]<br>[INFO] Elasticsearch Security Plugin Shim ................. SUCCESS [ 14.989 s]<br>[INFO] Elasticsearch Security Plugin ...................... SUCCESS [ 13.180 s]<br>[INFO] Apache Ranger Distribution ......................... SUCCESS [38:46 min]<br>[INFO] ------------------------------------------------------------------------<br>[INFO] BUILD SUCCESS<br>[INFO] ------------------------------------------------------------------------<br>[INFO] Total time: 01:24 h<br>[INFO] Finished at: 2021-05-27T15:47:58+08:00<br>[INFO] ------------------------------------------------------------------------<br></code></pre></td></tr></table></figure><h3 id="2-编译-ranger-client-jar包"><a href="#2-编译-ranger-client-jar包" class="headerlink" title="2 编译 ranger_client jar包"></a>2 编译 ranger_client jar包</h3><p>在ranger/intg下执行</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">mvn clean compile package install -DskipTests<br></code></pre></td></tr></table></figure><p>可以在 ranger/intg/target 看到生成后的jar包</p>]]></content>
<categories>
<category> ranger </category>
</categories>
</entry>
<entry>
<title>WSL开发系列-idea篇(WSL2配置与结合IDEA2021使用体验（及wsl-gui踩坑）)</title>
<link href="//wsl2_idea2021/"/>
<url>//wsl2_idea2021/</url>
<content type="html"><![CDATA[<p>2023.2更新：因更新较多，将大部分内容迁移至 WSL开发系列。<br>随着版本的迭代完善，加上本人丰富的踩坑经验，本人已放弃使用wslg版idea。<br>更推荐直接使用windows-idea的wsl功能。</p><p>本文给出相关使用建议。</p><span id="more"></span><h2 id="导航"><a href="#导航" class="headerlink" title="导航"></a>导航</h2><ul><li><a href="https://linshenkx.github.io/wsl-dev-base/">基础篇</a></li><li><a href="https://linshenkx.github.io/wsl-dev-static-ip/">网络篇</a></li><li><a href="https://linshenkx.github.io/wsl-dev-wslg/">gui篇</a></li><li><a href="https://linshenkx.github.io/wsl2_idea2021/">idea篇</a></li></ul><h2 id="一-idea-wsl开发总结"><a href="#一-idea-wsl开发总结" class="headerlink" title="一 idea-wsl开发总结"></a>一 idea-wsl开发总结</h2><p>首先需要明确，绝大部分开发工作都是可以在windows环境下完成的。<br>不推荐任何为了linux而linux的行为。</p><p>对我来说，WSL的开发应用场景主要在于：</p><ul><li>大数据功能开发<br>之前简单的hdfs操作都需要在windows中配置hadoop相关的环境变量，而且存在各种版本或兼容性问题，很麻烦，以后这些直接丢wsl里面</li><li>大数据组件编译<br>很多组件并不是纯Java代码，可能还需要一些脚本，这些有平台依赖性的，很可能因为windows而导致编译失败</li><li>linux脚本测试</li></ul><p>以下为之前的记录，可参考，部分已过期。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><code class="hljs awk">此部分过期：<br><br>经过测试，常规SpringBoot工程，在Windows目录下的，可以使用WSL环境进行debug、部署<br>对于纯maven工程，支持也还好<br>但对于纯gradle工程（非spring），则无法完成gradle初始化，无论工程是否是在WSL中都不行<br>设置JDK是WSL，但将Gradle路径指定为WSL却一直不行，说路径有问题，只能使用默认的Gradle。<br>根据issue的说法，建议将Windows的gradle path设置为WSL的路径，不过我没有试，我觉得这单纯就是官方的bug。<br>官方issue：https:<span class="hljs-regexp">//y</span>outrack.jetbrains.com<span class="hljs-regexp">/issues/</span>IDEA?q=wsl<br>另外，对于WSL下工程的git，idea似乎也无法识别<br>总之，IDEA（<span class="hljs-number">2021.1</span>）的WSL2支持尚不成熟，但是也已经初步支持，还可以自动下载、选择WSL环境的JDK<br>可以作为日常开发的辅助支持，但目前仍不建议全面迁移过去<br><br><span class="hljs-number">2021</span>年<span class="hljs-number">11</span>月更新：<br>IDEA2021.<span class="hljs-number">2.3</span>的wsl工程的bug已经较少，git、gradle支持相对正常，可以满足一般场景下的开发需求。<br>但仍不能在工程里读取到wsl的环境变量（能读取到windows的），所以我选择放弃。<br>直接在WSLg里装jetbrains toolbox，然后启动idea。<br>除去一些小bug，已经和原生linux体验差不多了，用来做大数据开发很舒服。<br>配置好环境就可以直接本地进行大数据操作，不再需要远程开发了，之前被这东西折磨得不行。<br>性能的话比windows idea的wsl开发要快很多，前提是你内存够。<br>wsl比较吃内存，当然开发的标配是<span class="hljs-number">32</span>g，一般还是够用的。<br><br>目前遇到的问题有：<br><span class="hljs-number">1</span>. 全屏偏移<br> 在全屏状态下，无法进行部分拖拽操作，可能与这个issue有关<br> https:<span class="hljs-regexp">//gi</span>thub.com<span class="hljs-regexp">/microsoft/</span>wslg<span class="hljs-regexp">/issues/</span><span class="hljs-number">502</span><br><span class="hljs-number">2</span>. 卡死<br> 通常是在调出新的窗口的时候<br> 需要手动wsl --shutdown，再启动，但可能造成idea文件未保存，修改丢失的情况<br> 在windows上不管怎么造基本都不会出现idea文件修改丢失的情况<br><br></code></pre></td></tr></table></figure><p>总结一下，idea的wsl开发有两种方案：idea自带的wsl支持，以及，利用wslg在Linux系统里面装idea</p><ol><li>idea自带的wsl支持<br>前期存在较多bug，现使用上没有明显问题，<br>缺点是无法读取到wsl环境变量、性能相对较差</li><li>wslg的idea<br>优点是原生linux应用、速度更快<br>缺点是wslg应用和windows切换并不顺畅，如输入法、快捷方式等<br>另外存在应用冻结（卡死）的bug，需强制关闭wsl重启，甚至会丢修改，这是不能忍受的</li></ol><p>wslg的缺点是无法忍受的，所以最终我还是决定使用idea自带的wsl支持<br>下面会分享我个人的一些使用体验和处理方法</p><h2 id="二-idea-wsl开发经验"><a href="#二-idea-wsl开发经验" class="headerlink" title="二 idea-wsl开发经验"></a>二 idea-wsl开发经验</h2><ol><li>尽量使用基于wsl的环境<br>如：项目放在wsl文件系统下，运行目标选择wsl，项目jdk选择wsl的，gradle jvm也选择wsl的</li><li>处理环境变量加载问题<br>运行目标选择wsl的工程，无法在运行时获取到wsl系统变量（/etc/profile），<br>这里建议在运行配置添加环境变量<br>如下，注意：</li></ol><ul><li>不要加载windows的环境变量</li><li>环境变量除了键值也可以像linux系统一样，使用命令的形式<br>如：<code>hadoop classpath</code><br><img src="https://lian-gallery.oss-cn-guangzhou.aliyuncs.com/img/202302201035656.png" alt="wsl-环境变量配置"></li></ul>]]></content>
<categories>
<category> 程序员杂记 </category>
</categories>
<tags>
<tag> 生产力 </tag>
<tag> WSL </tag>
</tags>
</entry>
<entry>
<title>大数据通用计算平台(支持flink、spark、storm)-系统调研及设计</title>
<link href="//bigdata_compute_platform/"/>
<url>//bigdata_compute_platform/</url>
<content type="html"><![CDATA[<p>项目源于对flink_sql流计算任务的实际使用需求，最初目标是设计一个系统可以在线提交sql生成flink流式计算任务，并进行监控监测。 后延申至支持在线jar包提交的方式，同时支持批式计算任务。并以模块化开发的思路，引入对spark的支持。</p><span id="more"></span><h3 id="一-简介"><a href="#一-简介" class="headerlink" title="一 简介"></a>一 简介</h3><h4 id="1-系统介绍"><a href="#1-系统介绍" class="headerlink" title="1 系统介绍"></a>1 系统介绍</h4><p>本系统基于多种不同的底层计算框架，如flink、spark、storm，提供可视化web界面，实现大数据任务的在线发布、管理、监控等功能。<br>支持多种任务类型，如sql提交、jar包提交等，屏蔽底层部署细节与参数配置，使开发人员可以专注于业务本身，提高工作效率<br>平台本身依托于现有大数据计算框架，不会引入其他依赖，不会对现有开发造成入侵，可放心使用。</p><h4 id="2-设计目标"><a href="#2-设计目标" class="headerlink" title="2 设计目标"></a>2 设计目标</h4><ol><li>支持多种计算框架：flink、spark、storm等</li><li>支持多种计算任务：批、流</li><li>支持多种任务类型：sql、jar包、dag图等</li><li>支持功能：发布、停止、删除、监控等</li><li>零入侵：纯客户端工具，不需要调整业务代码，不需要修改平台配置</li></ol><h4 id="3-RoadMap"><a href="#3-RoadMap" class="headerlink" title="3 RoadMap"></a>3 RoadMap</h4><ol><li>支持更多的提交方式，如脚本提交</li><li>支持更多的底层运行平台，如kubernetes、standalone等</li></ol><h3 id="二-设计调研"><a href="#二-设计调研" class="headerlink" title="二 设计调研"></a>二 设计调研</h3><h4 id="0-基础方向"><a href="#0-基础方向" class="headerlink" title="0 基础方向"></a>0 基础方向</h4><h5 id="1-底层运行平台的选择"><a href="#1-底层运行平台的选择" class="headerlink" title="1. 底层运行平台的选择"></a>1. 底层运行平台的选择</h5><p>不管是flink还是spark，都只是计算框架，需要真正的运行平台作为支持，主流的有yarn、kubernetes、standalone等</p><ol><li>local模式基本只适用于测试，这里不作考虑 </li><li>其中standalone即独立集群模式建设维护成本高，且重复建设易造成资源浪费（如flink一套、spark一套）。<br>不过standalone往往可以实现更多的特性，如spark的spark.master.rest.enabled配置项就只支持standalone模式，<br>而且大部分时候开发者只会用其中一个平台，并不存在重复建设的情况。<br>这里出于开发成本考虑，先不作实现，后续考虑支持。</li><li>各个计算框架基本都出了对k8s的支持，但文档来看还并不成熟，实际生产环境使用还较少，先观望，后续考虑支持。</li><li>Mesos我不熟<br>根据排除法，就只剩下yarn作为底层计算框架了。<br>优点是成熟、稳定，经过大规模检验，基本生产环境都是用yarn，而且不同的计算框架都可以共用一套yarn计算平台。</li></ol><h5 id="2-提交部署方式的选择"><a href="#2-提交部署方式的选择" class="headerlink" title="2. 提交部署方式的选择"></a>2. 提交部署方式的选择</h5><ol><li>在确定底层运行平台使用yarn之后，提交部署方式主要就剩下cluster、client两种了（此处使用spark的说法）<br>两者的区别主要在于启动程序的解析运行在客户端还是在集群上进行。<br>作为平台型工具，自然是选择cluster，减小客户端压力。client更适合个人调试用，可以直接看到输出。</li><li>对应flink的说法即为 per-job方式（客户端）和application方式（集群），即使用application方式。<br>另外需注意，application方式得在flink高版本（1.11+）才有。</li><li>另外，flink还有个session模式，简单理解为上面的是一任务一集群，这个是多任务一集群，适合轻量级使用，自带web端jar包提交，方便快捷。<br>但不建议生产环境使用，很容易一个任务崩就导致集群崩溃。</li></ol><h5 id="3-sql任务的实现"><a href="#3-sql任务的实现" class="headerlink" title="3. sql任务的实现"></a>3. sql任务的实现</h5><p>将sql保存为文件，使用约定好的jar程序进行解析，即将sql任务转化为jar任务。其他类型的任务同理。</p><h4 id="1-flink"><a href="#1-flink" class="headerlink" title="1 flink"></a>1 flink</h4><p>flink计算平台网上有很多，其中质量比较好且更新时间比较近的主要有：Flink-SQL流计算平台管理系统（ <a href="https://github.com/zhp8341/flink-streaming-platform-web">https://github.com/zhp8341/flink-streaming-platform-web</a> ）<br>本人最开始就是受了这个系统启发，最初的代码和思路也有部分参考，比较推荐。<br>优点是功能完整，且更新快。不过sql方面定制化程度有点高，不够通用，存在一定程度的入侵，且功能对于我来说有点花哨，支持flink本地模式、yarn-per模式、STANDALONE模式，最新的application模式倒没有。</p><h5 id="代码提交or接口提交or命令行提交？"><a href="#代码提交or接口提交or命令行提交？" class="headerlink" title="代码提交or接口提交or命令行提交？"></a>代码提交or接口提交or命令行提交？</h5><ol><li>代码提交？<br>yarn-application方式官方不支持，网上有的一般是session模式的，还有一些自己看源码挖出来提交代码的。<br>这里不推荐的去挖代码，官方没有放出来的东西可能换个版本就不一样了，如非必要，这种开发思路并不可取。</li><li>接口提交？<br>flink的web界面确实有jar包提交功能，但首先要把flink跑起来，即session模式，这里不使用。</li><li>命令行提交<br>默认的提交方法，万能的。</li></ol><h5 id="监控实现？"><a href="#监控实现？" class="headerlink" title="监控实现？"></a>监控实现？</h5><p>api+metrics</p><ul><li>api调用流程：<ol><li>提交yarn的时候指定tag，根据tag进行搜索，获取任务的appId，进行任务追踪</li><li>yarn层根据appId和yarn的rest接口跟踪任务在yarn上的状态</li><li>flink层根据appId可以组成出flink的web监控页面，同时flink提供了web界面所需的各项数据的rest接口，可以二次开发</li></ol></li><li>metrics获取：<br>flink 支持主动将指标数据推送到 Prometheus 的pushgateway。<h5 id="任务管理？"><a href="#任务管理？" class="headerlink" title="任务管理？"></a>任务管理？</h5>flink任务的停止可以直接调用其rest接口<br>savepoint信息也可以提供rest接口获取，savepoint的执行通过rest执行好像有问题，这里还是先用命令行操作<h4 id="2-spark"><a href="#2-spark" class="headerlink" title="2 spark"></a>2 spark</h4>spark作为一个<del>过时</del>成熟稳重的计算框架，虽然不支持流批一体，但是相关的第三方框架更加成熟。<h5 id="代码提交or接口提交or命令行提交？-1"><a href="#代码提交or接口提交or命令行提交？-1" class="headerlink" title="代码提交or接口提交or命令行提交？"></a>代码提交or接口提交or命令行提交？</h5>我们还是先从官网看起。大体情况和flink类似。<br>没有官方的代码提交接口，都是各自挖掘出来的，而且写法还都不一样，这里不推荐。<br>接口提交有个spark.master.rest.enabled配置项，但只支持standalone模式<br>所以还是走命令行提交<h5 id="监控实现？-1"><a href="#监控实现？-1" class="headerlink" title="监控实现？"></a>监控实现？</h5>api+metrics</li><li>api调用流程：<br>类似flink</li><li>metrics获取：<br>spark最新3.2.0版本支持显示Prometheus格式的指标数据，但不会主动推送<br>不过社区有支持pushgateway的插件：<a href="https://github.com/banzaicloud/spark-metrics">https://github.com/banzaicloud/spark-metrics</a><h5 id="任务管理？-1"><a href="#任务管理？-1" class="headerlink" title="任务管理？"></a>任务管理？</h5>主要就是想找个spark原生的rest api来kill它。<br>网上有一些spark隐藏rest api合集之类的东西，我本身是不想用的，毕竟你都知道是隐藏了，说明官方还不想给你用，你又不知道什么时间哪个版本就会变了<br>不过我也很好奇为什么官方不提供，很多人跟我有同样的想法，还跑去问了。<a href="https://issues.apache.org/jira/browse/SPARK-12528">https://issues.apache.org/jira/browse/SPARK-12528</a><br>standalone已经可以通过spark.master.rest.enabled来用了，但yarn还不行。讨论的人有提到可以用Livy或spark-jobserver。</li></ul><ol><li>spark-jobserver的社区比较活跃，更新比较快，功能也很强大，但是需要继承指定类，造成极强的入侵性，这是无法容忍的，这里不做考虑。<br>按我的理解，你都能继承自己的类了，那你做出什么增强的功能都不稀奇，因为你本质上已经不是工具型产品，而是一个基于spark二次开发的平台，还不支持spark原生jar包。</li><li>Livy毕竟是Apache孵化项目，虽然还没毕业，但使用还是没什么问题的。<br>接口很简单，使用也很方便。主要就是可以通过livy的rest接口实现任务的发布、停止、状态获取、日志获取。<br>其实livy的存在已经和本项目对spark的支持重复了…但是本着轮子要自己造的思想，我们还是来<del>挑挑刺</del>研究一下livy的优缺点<br>嗯…文档不清晰：虽然参数都是对官方的封装，但是缺少必要的描述说明，比方说没有说jar包的路径默认是hdfs的（当然这不是主要问题…）<br>其实没什么大问题，还是推荐使用的，任务提交我们已经通过脚本方式实现了，这里就没必要再引入livy了，毕竟是个单独的web服务器，也挺重的。</li></ol><p>既然如此就还是走yarn的api去kill吧。</p><h3 id="三-设计思路"><a href="#三-设计思路" class="headerlink" title="三 设计思路"></a>三 设计思路</h3><p>项目以分为三个类型工程</p><ol><li>core ：核心处理工程。实现核心逻辑，对外提供统一的计算任务管理接口，不同计算平台任务依赖不同模块工程进行实现</li><li>module : 计算平台处理工程。对接不同的底层计算平台，如flink和spark</li><li>plugin : 独立插件工程。本质为自定义功能jar包，封装特定任务，按约定进行解析处理。如flink-sql的jar包封装。</li></ol><h4 id="core工程"><a href="#core工程" class="headerlink" title="core工程"></a>core工程</h4><p>core是计算平台的核心。对不同类型的任务进行封装，对外提供统一的api。并在数据库中记录、跟踪任务信息。<br>从功能设计来看主要类有：</p><ul><li>controller：对外接口类<br>对外提供start、stop、delete、getLog、getStatus等api<br>start需传入任务配置json，根据任务类型的不同交由不同的module工程的JobServer去解析处理。<br>start、stop、delete 也需调用不同的JobServer进行实现。<br>getLog、getStatus本质上只是读取数据库，而数据库的信息维护和同步也依赖各个module工程。</li><li>jobInfo: 任务信息类<br>对不同任务的统一封装，拥有以下属性<br>deployMode 任务类型：FLINK_SQL、FLINK_JAR、SPARK_SQL、SPARK_JAR等<br>runConfig 运行时配置，json，不同类型任务不同<br>runInfo 运行时信息，json，不同类型任务不同，如flink任务包含yarn运行信息和flink的jobId、savepoint信息等<br>runLog 任务运行日志,text，用以记录任务的基本操作记录以及状态更改记录等（不包含业务日志，不属于监控范围）<br>status 任务运行状态，基于yarn的job status进行抽象。各个类型任务可以有自身的状态标识，但必须实现特定接口以转化为顶层标准status。</li><li>JobServer: 任务处理核心抽象接口<br>定义了 start、stop、delete、checkAllJobStatus、checkJobStatus(String jobInfoId)等接口并由各个模块自行实现</li><li>SchedulerTask：定时任务类<br>定时调用各个模块的checkStatus接口，用以检测任务真实运行状态，维护数据库任务信息真实性有效性，防止已停任务仍在运行</li></ul><p>另外我还把命令行执行模块、yarn功能管理模块放到了这里。<br>命令行执行要包含日志记录、异步处理、异常处理、多命令同会话执行，还是比较麻烦的，踩了一些坑，以后再开一篇分享一下。<br>本来的设想就是把yarn作为基础运行平台，所以对yarn的操作也统一到了这里。</p><h4 id="module工程"><a href="#module工程" class="headerlink" title="module工程"></a>module工程</h4><p>一个deployMode对应一个JobServer实现类，相同类型的JobServer可共用一个module工程，如FLINK_SQL和FLINK_JAR<br>各个module工程以其JobServer实现类为核心，拥有自己的runConfig、runInfo、status实体类<br>除此之外，根据需求，还可以有自身的定时任务类，如Flink可定时执行savepoint操作</p><h4 id="plugin工程"><a href="#plugin工程" class="headerlink" title="plugin工程"></a>plugin工程</h4><p>plugin工程本质上为特殊任务类型的支持实现。<br>这里以FLINK_SQL工程为例。</p><p>Flink原生的sql只支持shell执行，且需在session模式中运行，<br>使用上也不算十分方便，需要先在配置文件中写好各种连接配置，下载好各种connector的jar包。<br>作为测试用倒还可以，语句可以直接修改，不用重新打包，但性能、可靠性都存疑，本质上难以在生产环境中应用。<br>而Jar包执行是绝对通用的，sql的功能基本都可以通过flink的executeSql来执行，理论上对sql脚本进行解析，一句句转成用executeSql来调用即可。<br>不过也有一些坑，比分说hive-catalog的相关配置，我觉得flink关于hive的设计是比较别扭的，hive一方面是作为元数据仓库使用，一方面又可以作为数据源连接，另外flink还支持hive作为sql方言使用。<br>而hive的连接也相对复杂一些，需要先准备hive-conf配置文件，其jar包的位置也应该是放到flink的classpath下，而不是用户的fat-jar。注意，这一点和connector是不一样的。<br>对于元数据仓库catalog、方言dialect这些特殊sql语句，都需要识别出来调用专门代码实现，而不能直接使用executeSql</p><p>sql脚本应该通过文件作为参数传递给Jar包进行读取，其他类型的任务同理，无非是把sql脚本换成scala脚本、shell脚本之类的<br>具体做法是将sql写为本地临时文件，通过flink的yarn.ship-files参数分发到Jar包运行容器里面，sql文件路径作为Jar包启动参数即可。</p>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> flink </tag>
<tag> spark </tag>
<tag> storm </tag>
</tags>
</entry>
<entry>
<title>hive使用bulkLoad批量导入数据到hbase</title>
<link href="//hive_bulkLoad_hbase/"/>
<url>//hive_bulkLoad_hbase/</url>
<content type="html"><![CDATA[<p>本文主要参考了hbase和hive官方文档的说明，并结合cdh和hdp的一些教程以及个人在生产中的实践进行记录。主要内容有hbase bulkload的原理以及对应hive的操作步骤，最后基于cdh进行完整实验提供参考实例。<br>不过整个操作确实很复杂繁琐，不是很建议使用。现在有挺多使用Spark Bulkload，下次有机会尝试一下。<br>之前是遇到一个需求，源表在hbase上，需要重新生成rowkey并提取部分字段形成新表。功能很简单，就是行对行映射过去，但是效率太低耗时太长，用bulkload确实解决了当时的麻烦。<br>写这篇文章是对自己操作的复盘，同时也梳理一下知识。<br>实验环境为：CDH6.3.2，对应的各个组件版本为：hadoop3.0.0，hbase2.1.0，hive2.1.1</p><span id="more"></span><h2 id="一-hbase-bulk-loading批量加载-原理"><a href="#一-hbase-bulk-loading批量加载-原理" class="headerlink" title="一 hbase bulk loading批量加载 原理"></a>一 hbase bulk loading批量加载 原理</h2><p>这里推荐阅读hbase 最新官方文档：<a href="http://hbase.apache.org/2.3/book.html#arch.bulk.load">http://hbase.apache.org/2.3/book.html#arch.bulk.load</a><br>注意官方文档提供了2.2版本和2.3版本，这里原理讲解使用的是最新的，具体操作需结合根据实际使用版本来，有一些细节可能存在差异，例如2.2和2.3版本最终bulkload使用的jar包名称不一样。<br>这里简单地机翻一下：</p><h3 id="1-概述"><a href="#1-概述" class="headerlink" title="1 概述"></a>1 概述</h3><p>HBase包括几种将数据加载到表中的方法。 最直接的方法是使用MapReduce任务中的TableOutputFormat类，或使用常规客户端API。 但是，这些方法并不总是最有效的方法。<br>批量加载功能使用MapReduce任务以HBase的内部数据格式输出表数据，然后将生成的StoreFiles直接加载到正在运行的集群中。 与通过HBase API进行加载相比，使用批量加载将占用更少的CPU和网络资源。</p><h3 id="2-批量加载架构"><a href="#2-批量加载架构" class="headerlink" title="2 批量加载架构"></a>2 批量加载架构</h3><p>HBase批量加载过程包括两个主要步骤。</p><h4 id="1-通过MapReduce作业准备数据"><a href="#1-通过MapReduce作业准备数据" class="headerlink" title="1. 通过MapReduce作业准备数据"></a>1. 通过MapReduce作业准备数据</h4><p>批量加载的第一步是使用HFileOutputFormat2从MapReduce任务生成HBase数据文件(存储文件)。这种输出格式以HBase的内部存储格式写出数据，以便稍后可以有效地将它们加载到集群中。<br>为了有效地运行，必须配置HFileOutputFormat2，使每个输出的HFile都适用于单个region。为此，那些 输出结果将被批量加载到HBase中 的MapReduce任务，使用 Hadoop 的 TotalOrderPartitioner 类将map输出划分为 key 空间的不相交范围，该范围与表中 region 的 key 范围相对应。</p><p>HFileOutputFormat2包括一个便捷函数configureIncrementalLoad()，该函数根据表的当前区域边界自动设置TotalOrderPartitioner。</p><h4 id="2-完成数据加载"><a href="#2-完成数据加载" class="headerlink" title="2. 完成数据加载"></a>2. 完成数据加载</h4><p>在准备好要导入的数据之后，通过使用具有 “importtsv.bulk.output” 选项的 importtsv 工具，或通过使用 HFileOutputFormat 的 MapReduce 作业，可以使用 completebulkload 工具将数据导入到正在运行的集群中。 此命令行工具遍历准备好的数据文件，并为每个文件确定文件所属的region。 然后，它将连接上 HFile 对应的合适的 RegionServer ，将其移入其存储目录并将数据提供给客户端。</p><p>如果region边界在批量加载准备过程中或在准备和完成步骤之间发生了更改，则completebulkload程序将自动将数据文件分割为与新边界对应的部分。这个过程效率不高，所以用户应该尽量减少准备批量加载和将其导入集群之间的延迟，特别是当其他客户机通过其他方式同时加载数据时。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell"><br>hadoop jar hbase-mapreduce-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /user/todd/myoutput mytable<br><br></code></pre></td></tr></table></figure><p>如果类路径中没有提供相应的hbase参数，那么-c config-file选项可以用来指定包含相应hbase参数的文件(例如hbase-site.xml)(另外，如果zookeeper不是由hbase管理的，那么类路径必须包含包含zookeeper配置文件的目录)。</p><h2 id="二-hive-操作步骤"><a href="#二-hive-操作步骤" class="headerlink" title="二 hive 操作步骤"></a>二 hive 操作步骤</h2><p>这里主要参考hive的文档：<a href="https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad">https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad</a></p><h3 id="1-确定目标hbase表的结构"><a href="#1-确定目标hbase表的结构" class="headerlink" title="1. 确定目标hbase表的结构"></a>1. 确定目标hbase表的结构</h3><p>这里需要明确最终导入的hbase表的几个限制条件：</p><ol><li>目标表必须是新建的（即不能导入到已经存在的表）</li><li>目标表只能有一个列族</li><li>目标表不能是稀疏的（即每一行数据的结构必须一致）</li></ol><h3 id="2-添加必要的jar包"><a href="#2-添加必要的jar包" class="headerlink" title="2. 添加必要的jar包"></a>2. 添加必要的jar包</h3><blockquote><p>实测这一步可跳过，不如自己在hive shell上 add jar 。个人感觉hive.aux.jars.path 这个配置比较鸡肋。又不支持直接指定文件夹，发现少了jar包后还得再去编辑文件，需要的jar包一多xml看起来就很恶心。</p></blockquote><ol><li>将hive操作所需的jar包上传到hdfs<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">hadoop dfs -put /usr/lib/hive/lib/hbase-VERSION.jar /user/hive/hbase-VERSION.jar<br>hadoop dfs -put /usr/lib/hive/lib/hive-hbase-handler-VERSION.jar /user/hive/hive-hbase-handler-VERSION.jar<br><br></code></pre></td></tr></table></figure></li><li>将其路径添加进 hive-site.xml<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell"><property><br> <name>hive.aux.jars.path</name><br> <value>/user/hive/hbase-VERSION.jar,/user/hive/hive-hbase-handler-VERSION.jar</value><br></property><br><br><br></code></pre></td></tr></table></figure></li></ol><h3 id="3-确定分区范围"><a href="#3-确定分区范围" class="headerlink" title="3. 确定分区范围"></a>3. 确定分区范围</h3><p>为了最后bulk load的时候可以使用多个reducer加载到多个region，我们需要根据rowkey预先划分好数据的分区范围。这个范围将以文件的形式提供给下一步的中间hive表。这一步的目的就是要得到这个分区文件。</p><h4 id="1-创建分区信息表"><a href="#1-创建分区信息表" class="headerlink" title="1. 创建分区信息表"></a>1. 创建分区信息表</h4><p>分区文件以 org.apache.hadoop.hive.ql.io.HiveNullValueSequenceFileOutputFormat outputformat格式存在于hdfs上。我们可以创建一个分区信息表，指定好该表的存储位置，这样该位置上的文件即为我们要的分区信息文件。<br>分区信息表的创建如下：</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs hql">create external table hb_range_keys(transaction_id_range_start string)<br>row format serde<br>'org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe'<br>stored as<br>inputformat<br>'org.apache.hadoop.mapred.TextInputFormat'<br>outputformat<br>'org.apache.hadoop.hive.ql.io.HiveNullValueSequenceFileOutputFormat'<br>location '/tmp/hb_range_keys';<br><br></code></pre></td></tr></table></figure><h4 id="2-插入分区边界"><a href="#2-插入分区边界" class="headerlink" title="2. 插入分区边界"></a>2. 插入分区边界</h4><p>这一步需要插入rowkey的边界。例如，插入数据为 005，则最终生成的hbase的region有两个：下边界<del>005、005</del>上边界。<br>边界的选择很重要，良好的预分区设计可以减少hbase再平衡的压力，在大数据量下很有用。<br>因为我们的源表已经存在于hive中了，而目标hbase表的rowkey又是符合字典排序的。所以我们完全可以利用hive的抽样功能先生成hbase的rowkey序列并排好序，再以一定的间隔进行取样，这样就可以得到分布比较合理的边界值了。<br>抽样的实现参考如下：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><code class="hljs shell"><br>-- 添加必要的 jar 包 <br><br>add jar lib/hive-contrib-0.7.0.jar;<br><br>--设置reduce任务数为1<br><br>set mapred.reduce.tasks=1;<br><br>--创建临时函数 row_sequence，用于让结果带上排序编号<br><br>create temporary function row_sequence as<br>'org.apache.hadoop.hive.contrib.udf.UDFRowSequence';<br><br>--插入抽样结果到 分区信息表<br><br>insert overwrite table hb_range_keys -- 步骤（3）<br> select transaction_id from -- 步骤（2）<br> (select transaction_id -- 步骤（1）<br> from transactions<br> tablesample(bucket 1 out of 10000 on transaction_id) s<br> order by transaction_id<br> limit 10000000<br> ) x<br> where (row_sequence() % 910000)=0<br> order by transaction_id<br> limit 11;<br> <br>-- region数量=count(1)+1<br> <br>select count(1) from hb_range_keys ;<br><br></code></pre></td></tr></table></figure><p>这里 transactions 是源表，transaction_id 是导入到 hbase 中的 rowkey （实际使用可能需要生成，这里暂用 transaction_id 字段代替）<br>步骤（1）：根据transaction_id字段将数据分到 1万 个桶里，取里面的第一个桶。理论上约等于按 万分之一 进行抽样，抽样后的结果限制最多为 1千万 条。<br>步骤（2）：对步骤（1）的结果按照 rowkey 排序，每 91万 条 取 一条 作为最终结果，最多有 11 条（1000/91<=11）。<br>步骤（3）：将步骤（2）的结果插入到分区信息表</p><p>综上，假设源表的数据量在1千亿以上，经过两次抽样后最终最多被划分成了12个region。</p><blockquote><p>注意：桶抽样得到的数量不可靠，在极端情况下步骤（1）抽样的结果可能较少，所以需要自己确定分区信息表的记录行数，后面会用到。</p></blockquote><h4 id="3-得到分区文件"><a href="#3-得到分区文件" class="headerlink" title="3. 得到分区文件"></a>3. 得到分区文件</h4><p>在hive shell下执行改命令，即可得到分区文件 /tmp/hb_range_key_list</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">dfs -cp /tmp/hb_range_keys/* /tmp/hb_range_key_list;<br><br></code></pre></td></tr></table></figure><h3 id="4-生成中间hive表"><a href="#4-生成中间hive表" class="headerlink" title="4. 生成中间hive表"></a>4. 生成中间hive表</h3><p>该表用于生成符合 bulk load 格式要求的hfile文件。最终该表的数据都将移动到hbase中。</p><h4 id="1-准备中间表存储位置"><a href="#1-准备中间表存储位置" class="headerlink" title="1. 准备中间表存储位置"></a>1. 准备中间表存储位置</h4><p>因为中间表的占用的空间一般要比hbase的还大（数据是相同的，但hbase一般会压缩得比较好）<br>所以中间表的存储位置要特别注意。<br>如果存储位置不存在，hive会自动创建。如果存储位置已存在，需确保其内容为空。</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs hql">dfs -rmr /tmp/hbsort;<br>dfs -mkdir /tmp/hbsort;<br><br></code></pre></td></tr></table></figure><h4 id="2-创建中间表"><a href="#2-创建中间表" class="headerlink" title="2. 创建中间表"></a>2. 创建中间表</h4><p>mapred.reduce.tasks=region数量=分区信息表记录数+1<br>total.order.partitioner.path即为分区文件路径<br>注意：hfile.family.path=中间表存储路径/列族名</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs hql">set hive.execution.engine=mr;<br>set mapred.reduce.tasks=12;<br>set hive.mapred.partitioner=org.apache.hadoop.mapred.lib.TotalOrderPartitioner;<br>set total.order.partitioner.path=/tmp/hb_range_key_list;<br>set hfile.compression=gz;<br> <br>create table hbsort(transaction_id string, user_name string, amount double, ...)<br>stored as<br>INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'<br>OUTPUTFORMAT 'org.apache.hadoop.hive.hbase.HiveHFileOutputFormat'<br>TBLPROPERTIES ('hfile.family.path' = '/tmp/hbsort/cf');<br> <br></code></pre></td></tr></table></figure><h4 id="3-导入数据"><a href="#3-导入数据" class="headerlink" title="3. 导入数据"></a>3. 导入数据</h4><p>注意这里使用 cluster by 后加 排序字段（rowkey）</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">insert</span> overwrite <span class="hljs-keyword">table</span> hbsort<br><span class="hljs-keyword">select</span> transaction_id, user_name, amount, ...<br><span class="hljs-keyword">from</span> transactions<br>cluster <span class="hljs-keyword">by</span> transaction_id;<br><br></code></pre></td></tr></table></figure><h3 id="5-完成bulk-load"><a href="#5-完成bulk-load" class="headerlink" title="5. 完成bulk load"></a>5. 完成bulk load</h3><p>这一步已经和hive没什么关系了，以上面 bulk load 为准。hive的文档有些过时了：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">hadoop jar hbase-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /tmp/hbsort transactions<br><br></code></pre></td></tr></table></figure><h2 id="三-实验"><a href="#三-实验" class="headerlink" title="三 实验"></a>三 实验</h2><h3 id="0-测试环境及数据源准备"><a href="#0-测试环境及数据源准备" class="headerlink" title="0 测试环境及数据源准备"></a>0 测试环境及数据源准备</h3><h4 id="1-环境"><a href="#1-环境" class="headerlink" title="1. 环境"></a>1. 环境</h4><p>实验环境为：CDH6.3.2 parcel安装版本，对应的各个组件版本为：hadoop3.0.0，hbase2.1.0，hive2.1.1</p><p>注意，不同方式安装的组件位置不同，jar包位置也不一样<br>parcels 安装的 默认在 /opt/cloudera/parcels/CDH/lib/组件名<br>package 安装的 默认在 /usr/lib/组件名<br>开源hadoop的就随意了<br>下面的以 parcels 为准</p><h4 id="2-权限相关"><a href="#2-权限相关" class="headerlink" title="2. 权限相关"></a>2. 权限相关</h4><p>请自行解决。<br>我这里直接把dfs的权限认证给关闭了。生产环境我用的是kerberos，主要需要确保使用的用户对hive、hbase目录有相应的读写权限。</p><h4 id="3-实验用例"><a href="#3-实验用例" class="headerlink" title="3. 实验用例"></a>3. 实验用例</h4><p>参考 hdp的文档： <a href="https://docs.cloudera.com/HDPDocuments/HDP1/HDP-1.3.10/bk_user-guide/content/user-guide-hbase-import-1.html">https://docs.cloudera.com/HDPDocuments/HDP1/HDP-1.3.10/bk_user-guide/content/user-guide-hbase-import-1.html</a><br>我下载的数据是：<br><a href="https://dumps.wikimedia.org/other/pagecounts-raw/2008/2008-08/pagecounts-20080801-010000.gz">https://dumps.wikimedia.org/other/pagecounts-raw/2008/2008-08/pagecounts-20080801-010000.gz</a></p><h3 id="1-创建源表"><a href="#1-创建源表" class="headerlink" title="1 创建源表"></a>1 创建源表</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 下载测试数据</span><br>wget https://dumps.wikimedia.org/other/pagecounts-raw/2008/2008-08/pagecounts-20080801-010000.gz<br><span class="hljs-meta">#</span><span class="bash"> 解压</span><br>gzip pagecounts-20080801-010000.gz -d <br><span class="hljs-meta">#</span><span class="bash"> 创建源表存储路径</span><br>hadoop fs -mkdir /tmp/wikistats<br><span class="hljs-meta">#</span><span class="bash"> 上传测试数据</span><br>hadoop fs -put pagecounts-20080801-010000 /tmp/wikistats/<br><br></code></pre></td></tr></table></figure><p>进入 hive shell 创建表<br>这里我们创建了 pagecounts 表作为源表</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> pagecounts (projectcode STRING, pagename STRING, pageviews STRING, bytes STRING)<br><span class="hljs-type">ROW</span> FORMAT<br> DELIMITED FIELDS TERMINATED <span class="hljs-keyword">BY</span> <span class="hljs-string">' '</span><br> LINES TERMINATED <span class="hljs-keyword">BY</span> <span class="hljs-string">'\n'</span><br>STORED <span class="hljs-keyword">AS</span> TEXTFILE<br>LOCATION <span class="hljs-string">'/tmp/wikistats'</span>;<br></code></pre></td></tr></table></figure><p>我们可以select一下看一下数据的样子<br><img src="https://img-blog.csdnimg.cn/20201120005936525.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"><br>最好再count一下看一下数据量大小，这里可以看到是 3039029<br><img src="https://img-blog.csdnimg.cn/20201120010036509.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><h3 id="2-创建rowkey视图"><a href="#2-创建rowkey视图" class="headerlink" title="2 创建rowkey视图"></a>2 创建rowkey视图</h3><p>创建视图，用于获取自定义的rowkey，以后对源表的操作的转为对此视图的操作<br>注意必须保证生成rowkey没有重复，不然在最后一步写hfile的时候会报错：Added a key not lexically larger than previous。hbase的api写rowkey重复倒没什么问题，相当于覆盖，但这里是直接写hfile，必须保证rowkey不重复。<br>实测这里给的rowkey生成方法是会导致重复的。</p><p>hql如下：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">VIEW</span> IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> pgc (rowkey, pageviews, bytes) <span class="hljs-keyword">AS</span><br><span class="hljs-keyword">SELECT</span> concat_ws(<span class="hljs-string">'/'</span>,<br> projectcode,<br> concat_ws(<span class="hljs-string">'/'</span>,<br> pagename,<br> regexp_extract(INPUT__FILE__NAME, <span class="hljs-string">'pagecounts-(\\d{8}-\\d{6})\\..*$'</span>, <span class="hljs-number">1</span>))),<br> pageviews, bytes<br><span class="hljs-keyword">FROM</span> pagecounts;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/2020112001012265.png#pic_center" alt="在这里插入图片描述"><br>检测重复：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">select</span> rowkey,<span class="hljs-built_in">count</span>(<span class="hljs-number">1</span>) <span class="hljs-keyword">from</span> pgc <span class="hljs-keyword">group</span> <span class="hljs-keyword">by</span> rowkey <span class="hljs-keyword">having</span> <span class="hljs-built_in">count</span>(<span class="hljs-number">1</span>)<span class="hljs-operator">></span><span class="hljs-number">1</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20201123011902531.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"><br>既然如此，我们还是直接创建一个rowkey视图表出来，然后把重复的和源表都删掉吧。</p><blockquote><p>注意hive的distinct是无法识别字段级的重复的，而group by也没办法用在这里，所以我用的是 not in。</p></blockquote><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-comment">-- 创建 pgct表 取代 源表pagecounts和视图pgc</span><br><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> pgct <br><span class="hljs-type">ROW</span> FORMAT<br> DELIMITED FIELDS TERMINATED <span class="hljs-keyword">BY</span> <span class="hljs-string">' '</span><br> LINES TERMINATED <span class="hljs-keyword">BY</span> <span class="hljs-string">'\n'</span><br>STORED <span class="hljs-keyword">AS</span> TEXTFILE<br>LOCATION <span class="hljs-string">'/tmp/wikistats_pgct'</span><br> <span class="hljs-keyword">AS</span><br><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span><br><span class="hljs-keyword">FROM</span> pgc <span class="hljs-keyword">where</span> rowkey <span class="hljs-keyword">not</span> <span class="hljs-keyword">in</span><br>( <span class="hljs-keyword">select</span> rowkey <span class="hljs-keyword">from</span> pgc <span class="hljs-keyword">group</span> <span class="hljs-keyword">by</span> rowkey <span class="hljs-keyword">having</span> <span class="hljs-built_in">count</span>(<span class="hljs-number">1</span>)<span class="hljs-operator">></span><span class="hljs-number">1</span>)<br>;<br><span class="hljs-comment">-- 查看内容格式是否正确</span><br><span class="hljs-keyword">select</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">from</span> pgct limit <span class="hljs-number">1</span>;<br><span class="hljs-comment">-- 再次确认是否有重复的rowkey</span><br><span class="hljs-keyword">select</span> rowkey,<span class="hljs-built_in">count</span>(<span class="hljs-number">1</span>) <span class="hljs-keyword">from</span> pgct <span class="hljs-keyword">group</span> <span class="hljs-keyword">by</span> rowkey <span class="hljs-keyword">having</span> <span class="hljs-built_in">count</span>(<span class="hljs-number">1</span>)<span class="hljs-operator">></span><span class="hljs-number">1</span>;<br><span class="hljs-comment">-- 删除源表</span><br><span class="hljs-keyword">drop</span> <span class="hljs-keyword">table</span> pagecounts;<br><span class="hljs-comment">-- 删除视图</span><br><span class="hljs-keyword">drop</span> <span class="hljs-keyword">view</span> pgc;<br></code></pre></td></tr></table></figure><p>如果视图pgc没有重复的rowkey就不用那么麻烦再生成pgct表了</p><h3 id="3-创建分区信息表"><a href="#3-创建分区信息表" class="headerlink" title="3 创建分区信息表"></a>3 创建分区信息表</h3><p>创建的分区信息表名为：hbase_splits</p><blockquote><p>原教程的列名为 partition ，会导致异常，我这里改为了 key。异常信息为：FAILED: ParseException line 1:49 cannot recognize input near ‘partition’ ‘STRING’ ‘)’ in column name or primary key or foreign key</p></blockquote><blockquote><p>hbase_splits_tmp的作用：<br>用于获取分区信息的行数<br>其他的教程都没有写到需要count获取分区信息表的行数，但如果没获取就没办法正确设置下一步的mapred.reduce.tasks。<br>hbase_splits表由于存储格式和inputformat的关系，存在一些bug，没办法select出内容，甚至连count都不行。所以我这里用了hbase_splits_tmp进行记录，内容先写到hbase_splits_tmp，count完再把hbase_splits_tmp的内容写到hbase_splits上。</p></blockquote><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">EXTERNAL</span> <span class="hljs-keyword">TABLE</span> IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> hbase_splits(key STRING) <br><span class="hljs-type">ROW</span> FORMAT<br> SERDE <span class="hljs-string">'org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe'</span><br>STORED <span class="hljs-keyword">AS</span><br> INPUTFORMAT <span class="hljs-string">'org.apache.hadoop.mapred.TextInputFormat'</span><br> OUTPUTFORMAT <span class="hljs-string">'org.apache.hadoop.hive.ql.io.HiveNullValueSequenceFileOutputFormat'</span><br>LOCATION <span class="hljs-string">'/tmp/hbase_splits_out'</span>;<br><br><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> hbase_splits_tmp(key STRING) <br><span class="hljs-type">ROW</span> FORMAT<br> DELIMITED FIELDS TERMINATED <span class="hljs-keyword">BY</span> <span class="hljs-string">' '</span><br> LINES TERMINATED <span class="hljs-keyword">BY</span> <span class="hljs-string">'\n'</span><br>STORED <span class="hljs-keyword">AS</span> TEXTFILE<br>LOCATION <span class="hljs-string">'/tmp/hbase_splits_tmp'</span>;<br><br></code></pre></td></tr></table></figure><h3 id="4-获取分区信息"><a href="#4-获取分区信息" class="headerlink" title="4 获取分区信息"></a>4 获取分区信息</h3><p>我们源表的数据在300万条以上（教程里的数据是在400w条左右，这里为了偷懒，就不改了）<br>这里设计为每1w条抽出一条，得到400条以内的结果，再每100条取1，最终不超过4条rowkey作为边界。</p><p>这里为了使用 row_seq 需要添加 hive-contrib 包</p><p>这里是对视图进行操作，视图里已经规定了rowkey的设计方法了<br>最后，要根据存储位置把文件cp成一个文件</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">ADD</span> JAR <span class="hljs-operator">/</span>opt<span class="hljs-operator">/</span>cloudera<span class="hljs-operator">/</span>parcels<span class="hljs-operator">/</span>CDH<span class="hljs-operator">/</span>lib<span class="hljs-operator">/</span>hive<span class="hljs-operator">/</span>contrib<span class="hljs-operator">/</span>hive<span class="hljs-operator">-</span>contrib<span class="hljs-number">-2.1</span><span class="hljs-number">.1</span><span class="hljs-operator">-</span>cdh6<span class="hljs-number">.3</span><span class="hljs-number">.2</span>.jar;<br><span class="hljs-keyword">SET</span> mapred.reduce.tasks<span class="hljs-operator">=</span><span class="hljs-number">1</span>;<br><br><span class="hljs-keyword">CREATE</span> TEMPORARY <span class="hljs-keyword">FUNCTION</span> row_seq <span class="hljs-keyword">AS</span> <span class="hljs-string">'org.apache.hadoop.hive.contrib.udf.UDFRowSequence'</span>;<br><br><span class="hljs-comment">-- 获取分区信息</span><br><span class="hljs-keyword">INSERT</span> OVERWRITE <span class="hljs-keyword">TABLE</span> hbase_splits_tmp<br><span class="hljs-keyword">SELECT</span> rowkey <span class="hljs-keyword">FROM</span><br> (<span class="hljs-keyword">SELECT</span> rowkey, row_seq() <span class="hljs-keyword">AS</span> seq <span class="hljs-keyword">FROM</span> pgct<br> <span class="hljs-keyword">TABLESAMPLE</span>(BUCKET <span class="hljs-number">1</span> <span class="hljs-keyword">OUT</span> <span class="hljs-keyword">OF</span> <span class="hljs-number">10000</span> <span class="hljs-keyword">ON</span> rowkey) s<br> <span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> rowkey<br> LIMIT <span class="hljs-number">400</span>) x<br><span class="hljs-keyword">WHERE</span> (seq <span class="hljs-operator">%</span> <span class="hljs-number">100</span>) <span class="hljs-operator">=</span> <span class="hljs-number">0</span><br><span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> rowkey<br>LIMIT <span class="hljs-number">4</span>;<br><br><span class="hljs-comment">-- 查看内容 及 获取count数</span><br><span class="hljs-keyword">select</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">from</span> hbase_splits_tmp;<br><span class="hljs-keyword">select</span> <span class="hljs-built_in">count</span>(<span class="hljs-number">1</span>) <span class="hljs-keyword">from</span> hbase_splits_tmp;<br><br><span class="hljs-comment">-- 内容写到hbase_splits_tmp</span><br><span class="hljs-keyword">INSERT</span> OVERWRITE <span class="hljs-keyword">TABLE</span> hbase_splits_tmp<br><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">FROM</span> hbase_splits_tmp;<br><br><span class="hljs-comment">-- 删除临时工具表</span><br><span class="hljs-keyword">drop</span> <span class="hljs-keyword">table</span> hbase_splits_tmp;<br><br><span class="hljs-comment">-- 得到分区信息文件</span><br>dfs <span class="hljs-operator">-</span>cp <span class="hljs-operator">/</span>tmp<span class="hljs-operator">/</span>hbase_splits_out<span class="hljs-comment">/* /tmp/hbase_splits;</span><br><span class="hljs-comment"></span><br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20201123142921465.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><h3 id="5-创建hfile表并插入数据"><a href="#5-创建hfile表并插入数据" class="headerlink" title="5 创建hfile表并插入数据"></a>5 创建hfile表并插入数据</h3><h4 id="1-创建hfile表"><a href="#1-创建hfile表" class="headerlink" title="1 创建hfile表"></a>1 创建hfile表</h4><p>这个表用来生成hfile，这里我们使用的列族为 w</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs hql">CREATE TABLE hbase_hfiles(rowkey STRING, pageviews STRING, bytes STRING)<br>STORED AS<br> INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'<br> OUTPUTFORMAT 'org.apache.hadoop.hive.hbase.HiveHFileOutputFormat'<br>TBLPROPERTIES('hfile.family.path' = '/tmp/hbase_hfiles/w');<br><br></code></pre></td></tr></table></figure><h4 id="2-插入数据"><a href="#2-插入数据" class="headerlink" title="2 插入数据"></a>2 插入数据</h4><p>total.order.partitioner.path 用于指定使用的分区排序文件的位置，已经过时，使用 mapreduce.totalorderpartitioner.path 代替。<br>注意，我这里把hive 的lib下所有带有hbase的包都添加进去了。理论上应该是完整的了，不过实测还是会提示缺少类：.ClassNotFoundException: org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos$FileInfoProto。<br>这个类是在hbase-protocol-shaded.jar包里的，需要到hbase目录里才能找到，也给添加进去，就可以了。<br>mapred.reduce.tasks数目是前方预估的regions数。</p><p>最后一步select出来的数据必须保证无重复的rowkey，实测加distinct 也没用。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><code class="hljs shell">ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-client.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-common.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-hadoop2-compat.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-hadoop-compat.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-http.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-mapreduce.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-metrics-api.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-metrics.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-procedure.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-protocol.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-replication.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-server.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-shaded-miscellaneous.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-shaded-netty.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-shaded-protobuf.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hbase-zookeeper.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/tephra-hbase-compat-1.0-0.6.0.jar;<br>ADD JAR /opt/cloudera/parcels/CDH/lib/hive/lib/hive-hbase-handler.jar;<br><br>ADD JAR /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol-shaded.jar;<br><br>SET mapred.reduce.tasks=3;<br>SET hive.mapred.partitioner=org.apache.hadoop.mapred.lib.TotalOrderPartitioner;<br>SET mapreduce.totalorderpartitioner.path=/tmp/hbase_splits;<br><br>-- generate hfiles using the splits ranges<br>INSERT OVERWRITE TABLE hbase_hfiles<br>SELECT distinct rowkey, pageviews, bytes FROM pgct<br>CLUSTER BY rowkey;<br><br></code></pre></td></tr></table></figure><h3 id="6-完成bulk-load"><a href="#6-完成bulk-load" class="headerlink" title="6 完成bulk load"></a>6 完成bulk load</h3><h4 id="1-创建目标hbase表"><a href="#1-创建目标hbase表" class="headerlink" title="1. 创建目标hbase表"></a>1. 创建目标hbase表</h4><p>在hbase shell下执行</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">create 'page_count', { NAME =>'w',COMPRESSION => 'SNAPPY' }<br><br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20201123142510273.png#pic_center" alt="在这里插入图片描述"></p><h4 id="2-bulkload"><a href="#2-bulkload" class="headerlink" title="2. bulkload"></a>2. bulkload</h4><p>bulkload有两种写法，这里推荐使用下面的第二种，不需要设置classpath，如果是使用 hadoop jar需要将 hbase classpath 添加到 hadoop classpath</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> hadoop jar hbase-mapreduce.jar completebulkload /tmp/hbase_hfiles page_count</span><br>hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /tmp/hbase_hfiles page_count<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20201123142428299.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><h3 id="7-总结"><a href="#7-总结" class="headerlink" title="7 总结"></a>7 总结</h3><p>不同的版本会有不同的问题。我自己使用的官方开源版本的hive获取分区信息的时候可以直接count，这里用的比较老，似乎就有问题。<br>另外需要注意可能会各种缺少class，所以建议直接add所有hbase相关的包。</p><h2 id="四-参考文章"><a href="#四-参考文章" class="headerlink" title="四 参考文章"></a>四 参考文章</h2><ul><li>hbase官方教程：<br><a href="http://hbase.apache.org/2.2/book.html#arch.bulk.load">http://hbase.apache.org/2.2/book.html#arch.bulk.load</a></li><li>hive 官方教程：<br><a href="https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad">https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad</a></li><li>hortonworks教程：<br><a href="https://docs.cloudera.com/HDPDocuments/HDP1/HDP-1.3.10/bk_user-guide/content/user-guide-hbase-import-1.html">https://docs.cloudera.com/HDPDocuments/HDP1/HDP-1.3.10/bk_user-guide/content/user-guide-hbase-import-1.html</a></li><li>有赞HBase Bulkload 实践探讨：<a href="https://tech.youzan.com/hbase-bulkloadshi-practice/">https://tech.youzan.com/hbase-bulkloadshi-practice/</a></li></ul>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> hbase </tag>
<tag> hive </tag>
<tag> bulkload </tag>
</tags>
</entry>
<entry>
<title>hive编写udf实践记录</title>
<link href="//hive_udf/"/>
<url>//hive_udf/</url>
<content type="html"><![CDATA[<p>官方教程：<a href="https://cwiki.apache.org/confluence/display/Hive/HivePlugins">https://cwiki.apache.org/confluence/display/Hive/HivePlugins</a><br>简单使用查看上面官方的文档即可。这里记录一下我使用的实践和一点注意事项。</p><span id="more"></span><h2 id="一-编写udf"><a href="#一-编写udf" class="headerlink" title="一 编写udf"></a>一 编写udf</h2><p>这里的需求是写一个udf，用于将经纬度转换成geohash。参数有 经纬度和geohash的精度。</p><h3 id="gradle配置"><a href="#gradle配置" class="headerlink" title="gradle配置"></a>gradle配置</h3><p>gradle 部分配置如下：</p><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs gradle"><span class="hljs-keyword">dependencies</span> {<br> <span class="hljs-keyword">compile</span> <span class="hljs-keyword">group</span>: <span class="hljs-string">'ch.hsr'</span>, name: <span class="hljs-string">'geohash'</span>, version: <span class="hljs-string">'1.3.0'</span><br> compileOnly <span class="hljs-keyword">group</span>: <span class="hljs-string">'org.apache.hive'</span>, name: <span class="hljs-string">'hive-exec'</span>, version: <span class="hljs-string">'2.3.6'</span><br>}<br><br><span class="hljs-keyword">task</span> fatJar(type: Jar) {<br> <span class="hljs-comment">//baseName = project.name</span><br> baseName = <span class="hljs-string">'hiveFunction'</span><br> <span class="hljs-keyword">project</span>.<span class="hljs-keyword">fileTree</span>(<span class="hljs-string">"$buildDir/fatjar/libs"</span>).forEach { jarFile -><br> <span class="hljs-keyword">from</span> zipTree(jarFile )<br> }<br> with jar<br> <span class="hljs-keyword">destinationDir</span> = <span class="hljs-keyword">file</span>(<span class="hljs-string">"$buildDir/fatjar"</span>)<br>}<br><br><span class="hljs-keyword">task</span> copyToLib(type: <span class="hljs-keyword">Copy</span>) {<br> <span class="hljs-keyword">into</span> <span class="hljs-string">"$buildDir/fatjar/libs"</span><br> <span class="hljs-keyword">from</span> <span class="hljs-keyword">configurations</span>.<span class="hljs-keyword">runtime</span><br>}<br></code></pre></td></tr></table></figure><h3 id="UDF类"><a href="#UDF类" class="headerlink" title="UDF类"></a>UDF类</h3><p>GeoHashUDF.java 代码如下：<br>这里需要注意的就是注释的写法，以及异常的处理。<br>因为在hive执行的过程中，如果udf抛一个异常出来，有可能会导致整个hive sql执行的失败。所以，对于业务异常，应该避免向外抛出。<br>以本geohash的udf为例，如果给定的经纬度不合法，则可返回用0填充的默认字符串。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-keyword">import</span> ch.hsr.geohash.GeoHash;<br><span class="hljs-keyword">import</span> org.apache.hadoop.hive.ql.exec.Description;<br><span class="hljs-keyword">import</span> org.apache.hadoop.hive.ql.exec.UDF;<br><br><span class="hljs-keyword">import</span> java.util.HashMap;<br><span class="hljs-keyword">import</span> java.util.Map;<br><br><span class="hljs-meta">@Description(</span><br><span class="hljs-meta"> name = "geohash",</span><br><span class="hljs-meta"> value = "_FUNC_(double lat,double lon,int precison) - Returns geohash",</span><br><span class="hljs-meta"> extended = "Example:\n > SELECT _FUNC_(\'20.1,111.2,6\') FROM src LIMIT 1;\n \'hello:Facebook\'"</span><br><span class="hljs-meta">)</span><br><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">GeoHashUDF</span> <span class="hljs-keyword">extends</span> <span class="hljs-title">UDF</span> </span>{<br><br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> Map<Integer,String> defaultValueMap = <span class="hljs-keyword">new</span> HashMap<> ();<br><br> <span class="hljs-keyword">static</span> {<br> StringBuilder sb = <span class="hljs-keyword">new</span> StringBuilder(<span class="hljs-string">""</span>);<br> <span class="hljs-keyword">for</span>(<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < <span class="hljs-number">20</span>; i++){<br> defaultValueMap.put(i,sb.toString());<br> sb.append(<span class="hljs-number">0</span>);<br> }<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> String <span class="hljs-title">evaluate</span> <span class="hljs-params">(Double lat,Double lon,Integer precision)</span> </span>{<br> <span class="hljs-keyword">try</span> {<br> <span class="hljs-keyword">return</span> GeoHash.geoHashStringWithCharacterPrecision(lat, lon, precision);<br> }<span class="hljs-keyword">catch</span> (Exception e) {<br> <span class="hljs-keyword">return</span> defaultValueMap.get(precision);<br> }<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">main</span><span class="hljs-params">(String[] args)</span> </span>{<br> System.out.println(<span class="hljs-keyword">new</span> GeoHashUDF().evaluate(<span class="hljs-number">20.1</span>,<span class="hljs-number">111.2</span>,<span class="hljs-number">6</span>));<br> }<br><br>}<br><br><br></code></pre></td></tr></table></figure><h3 id="打jar包"><a href="#打jar包" class="headerlink" title="打jar包"></a>打jar包</h3><p>先执行gradle copyToLib，再执行 gradle fatJar<br>得到 hiveFunction.jar</p><h2 id="二-创建function"><a href="#二-创建function" class="headerlink" title="二 创建function"></a>二 创建function</h2><h3 id="1-创建临时function"><a href="#1-创建临时function" class="headerlink" title="1 创建临时function"></a>1 创建临时function</h3><ol><li>先将生成的jar包添加到hive的lib目录下<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">\cp -f hiveFunction.jar $HIVE_HOME/lib/<br><br></code></pre></td></tr></table></figure></li><li>再在hive shell里面执行以下命令即可，注意要使用全称类名<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">create temporary function geohash as 'GeoHashUDF';<br><br></code></pre></td></tr></table></figure></li><li>删除<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">drop temporary function if exists geohash;<br><br></code></pre></td></tr></table></figure><h3 id="2-创建永久function"><a href="#2-创建永久function" class="headerlink" title="2 创建永久function"></a>2 创建永久function</h3></li><li>上传jar包到hdfs上<figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">hadoop fs -rm /my/hive/libs/hiveFunction.jar<br>hadoop fs -put hiveFunction.jar /my/hive/libs/hiveFunction.jar<br></code></pre></td></tr></table></figure></li><li>创建永久函数<figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">create function geohash as 'com.test.hive.function.GeoHashUDF' using jar 'hdfs:///my/hive/libs/hiveFunction.jar';<br><br></code></pre></td></tr></table></figure></li><li>删除<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">drop function if exists geohash;<br><br></code></pre></td></tr></table></figure></li></ol>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> hive </tag>
</tags>
</entry>
<entry>
<title>CDH客户端环境搭建</title>
<link href="//cdh_client_env_deploy/"/>
<url>//cdh_client_env_deploy/</url>
<content type="html"><![CDATA[<p>最近遇到一个需求：要使用azkaban对接客户的CDH集群，CDH用的是oozie，azkaban只能部署在我们客户端的机器上，所以需要在客户机上手动搭建CDH的hadoop环境。操作很简单，过程比较麻烦，这里记录一下。</p><span id="more"></span><h2 id="一-获取所需CDH-rpm包"><a href="#一-获取所需CDH-rpm包" class="headerlink" title="一 获取所需CDH rpm包"></a>一 获取所需CDH rpm包</h2><h3 id="1-搭建本地CDH-package仓库"><a href="#1-搭建本地CDH-package仓库" class="headerlink" title="1. 搭建本地CDH package仓库"></a>1. 搭建本地CDH package仓库</h3><p>说明：CDH有两种本地仓库配置方式，package和parcel。官方是推荐使用package，我推荐两种都配置好，CDH还是用parcel。package方式可用于获取rpm包。<br><a href="https://docs.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_create_local_package_repo.html">https://docs.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_create_local_package_repo.html</a><br>yum包的下载建议使用idm的站点抓取功能，不要用wget下载</p><h3 id="2-使用-yumdownloader-获取rpm包"><a href="#2-使用-yumdownloader-获取rpm包" class="headerlink" title="2. 使用 yumdownloader 获取rpm包"></a>2. 使用 yumdownloader 获取rpm包</h3><p>这一步需要在docker容器环境下进行，这样环境很纯粹，获取到到rpm包更完整，参考：<a href="https://blog.csdn.net/alinyua/article/details/108071365">https://blog.csdn.net/alinyua/article/details/108071365</a></p><figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> yum install yum-utils -y</span><br>mkdir cdh-packages && cd cdh-packages <br>yumdownloader --resolve hadoop-client hbase hive-hbase spark-core<br>cd .. && tar -zcf cdh-packages.tar.gz cdh-packages<br></code></pre></td></tr></table></figure><h2 id="二-安装CDH-rpm包"><a href="#二-安装CDH-rpm包" class="headerlink" title="二 安装CDH rpm包"></a>二 安装CDH rpm包</h2><p>安装时虽然不会有依赖缺少，但可能由于系统版本关系部分系统自带软件版本不一致，我是参考错误提示直接忽略掉，后面也没有遇到问题。</p><figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">tar -zxf cdh-packages.tar.gz<br>yum -y localinstall cdh-packages/*.rpm<br></code></pre></td></tr></table></figure><h2 id="三-配置CDH-环境"><a href="#三-配置CDH-环境" class="headerlink" title="三 配置CDH 环境"></a>三 配置CDH 环境</h2><h3 id="1-配置环境变量"><a href="#1-配置环境变量" class="headerlink" title="1. 配置环境变量"></a>1. 配置环境变量</h3><figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell">echo "export HADOOP_HOME=/usr/lib/hadoop">> /etc/profile<br>echo "export HBASE_HOME=/usr/lib/hbase">> /etc/profile<br>echo "export HIVE_HOME=/usr/lib/hive">> /etc/profile<br>echo "export SPARK_HOME=/usr/lib/spark">> /etc/profile<br>echo "export HADOOP_CONF_DIR=/etc/hadoop/conf/">> /etc/profile<br><br>source /etc/profile<br></code></pre></td></tr></table></figure><h3 id="2-添加配置文件"><a href="#2-添加配置文件" class="headerlink" title="2. 添加配置文件"></a>2. 添加配置文件</h3><p>在CDH管理界面下载各组件的客户端配置，添加到对应文件夹下<br>各组件配置文件位置如下：<br>hadoop/yarn: /etc/hadoop/conf/<br>hbase: /etc/hbase/conf<br>hive: /etc/hive/conf</p><h2 id="四-验证"><a href="#四-验证" class="headerlink" title="四 验证"></a>四 验证</h2><p>输入以下命令对各个组件进行验证</p><h3 id="1-hdfs"><a href="#1-hdfs" class="headerlink" title="1. hdfs"></a>1. hdfs</h3><figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">hadoop fs -ls /<br></code></pre></td></tr></table></figure><h3 id="2-yarn"><a href="#2-yarn" class="headerlink" title="2. yarn"></a>2. yarn</h3><figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">yarn node -list<br></code></pre></td></tr></table></figure><h3 id="3-hbase"><a href="#3-hbase" class="headerlink" title="3. hbase"></a>3. hbase</h3><figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">hbase shell<br><br>status<br></code></pre></td></tr></table></figure><h3 id="4-hive"><a href="#4-hive" class="headerlink" title="4. hive"></a>4. hive</h3><figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">hive shell<br><br>show tables;<br></code></pre></td></tr></table></figure><h3 id="5-spark"><a href="#5-spark" class="headerlink" title="5. spark"></a>5. spark</h3><figure class="highlight shell"><figcaption><span>script</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 能正确进入即可</span><br>spark-shell<br><br></code></pre></td></tr></table></figure><blockquote><p>如果报错：java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V<br>根据：<a href="https://issues.apache.org/jira/browse/SPARK-25783">https://issues.apache.org/jira/browse/SPARK-25783</a><br>可下载 jline-2.14.3.jar 放到 $SPARK_HOME/jars 下<br>下载地址：<a href="https://repo1.maven.org/maven2/jline/jline/2.14.3/jline-2.14.3.jar">https://repo1.maven.org/maven2/jline/jline/2.14.3/jline-2.14.3.jar</a></p></blockquote>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> CDH </tag>
</tags>
</entry>
<entry>
<title>CDH部署笔记</title>
<link href="//cdh_server_env_deploy/"/>
<url>//cdh_server_env_deploy/</url>
<content type="html"><![CDATA[<p>本文为个人安装CDH时记录，不具普适性，仅供参考。建议对比官方文档阅读。</p><span id="more"></span><h2 id="一-依赖检查"><a href="#一-依赖检查" class="headerlink" title="一 依赖检查"></a>一 依赖检查</h2><p>以centos7为例</p><h3 id="1-软件"><a href="#1-软件" class="headerlink" title="1. 软件"></a>1. 软件</h3><p><a href="https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_os_requirements.html">https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_os_requirements.html</a></p><h4 id="0-常用软件安装"><a href="#0-常用软件安装" class="headerlink" title="0. 常用软件安装"></a>0. 常用软件安装</h4><p>此步骤非必须</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum install -y bind-utils<br></code></pre></td></tr></table></figure><h4 id="1-python"><a href="#1-python" class="headerlink" title="1. python"></a>1. python</h4><p>使用2.7版本或以上版本，但不支持3<br>centos7 中默认已包含，如果有多版本需设置PYSPARK_PYTHON和<br>PYSPARK_DRIVER_PYTHON环境变量</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 查看版本</span><br>python --version<br></code></pre></td></tr></table></figure><h4 id="2-Perl"><a href="#2-Perl" class="headerlink" title="2. Perl"></a>2. Perl</h4><p>一般已安装</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 查看版本</span><br>perl -v<br></code></pre></td></tr></table></figure><h4 id="3-python-psycopg2"><a href="#3-python-psycopg2" class="headerlink" title="3. python-psycopg2"></a>3. python-psycopg2</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 要先安装epel-release，，这个包包含了 EPEL 源的 gpg 密钥和软件源信息，该软件包会自动配置yum的软件仓库</span><br><span class="hljs-meta">#</span><span class="bash"> 否则下一步会提示：没有可用软件包 pip-python</span><br>yum -y install epel-release<br><span class="hljs-meta">#</span><span class="bash"> 安装pip</span><br>yum -y install python-pip<br><span class="hljs-meta">#</span><span class="bash"> 使用pip 安装 psycopg2</span><br>pip install psycopg2==2.7.5 --ignore-installed<br></code></pre></td></tr></table></figure><h3 id="2-网络"><a href="#2-网络" class="headerlink" title="2. 网络"></a>2. 网络</h3><h4 id="禁用ipv6"><a href="#禁用ipv6" class="headerlink" title="禁用ipv6"></a>禁用ipv6</h4><p><a href="https://www.jianshu.com/p/225d040d0b66">https://www.jianshu.com/p/225d040d0b66</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysctl -w net.ipv6.conf.all.disable_ipv6=1<br>sysctl -w net.ipv6.conf.default.disable_ipv6=1<br></code></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">reboot<br>netstat -lnpt<br></code></pre></td></tr></table></figure><h4 id="配置hostname"><a href="#配置hostname" class="headerlink" title="配置hostname"></a>配置hostname</h4><p><a href="https://docs.cloudera.com/documentation/enterprise/6/latest/topics/configure_network_names.html">https://docs.cloudera.com/documentation/enterprise/6/latest/topics/configure_network_names.html</a><br>用全称域名，如：paas-201.adp.com 而不是 paas-201</p><ol><li><p>配置hostname</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo hostnamectl set-hostname foo-1.example.com<br><br></code></pre></td></tr></table></figure></li><li><p>编辑/ets/hosts(集群统一)</p><figure class="highlight accesslog"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs accesslog"><span class="hljs-number">1.1.1.1</span> foo-<span class="hljs-number">1</span>.example.com foo-<span class="hljs-number">1</span><br><span class="hljs-number">2.2.2.2</span> foo-<span class="hljs-number">2</span>.example.com foo-<span class="hljs-number">2</span><br><span class="hljs-number">3.3.3.3</span> foo-<span class="hljs-number">3</span>.example.com foo-<span class="hljs-number">3</span><br><span class="hljs-number">4.4.4.4</span> foo-<span class="hljs-number">4</span>.example.com foo-<span class="hljs-number">4</span><br></code></pre></td></tr></table></figure></li><li><p>编辑/etc/sysconfig/network</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">echo "HOSTNAME=$HOSTNAME" >>/etc/sysconfig/network<br><br></code></pre></td></tr></table></figure><h4 id="关闭防火墙"><a href="#关闭防火墙" class="headerlink" title="关闭防火墙"></a>关闭防火墙</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 保存规则</span><br>sudo iptables-save > ~/firewall.rules<br><span class="hljs-meta">#</span><span class="bash"> 关闭</span><br>sudo systemctl disable firewalld<br>sudo systemctl stop firewalld<br></code></pre></td></tr></table></figure><h3 id="3-关闭SELinux"><a href="#3-关闭SELinux" class="headerlink" title="3. 关闭SELinux"></a>3. 关闭SELinux</h3><p>查看是否已经关闭</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">getenforce<br></code></pre></td></tr></table></figure><p>关闭方法：<br>修改/etc/selinux/config<br>SELINUX=permissive<br>并执行 setenforce 0 立即生效</p><h3 id="4-启用ntp"><a href="#4-启用ntp" class="headerlink" title="4. 启用ntp"></a>4. 启用ntp</h3><p>安装后用以下命令进行验证</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">systemctl status chronyd.service <br>chronyc sources -v <br>chronyc sourcestats -v<br></code></pre></td></tr></table></figure><h2 id="二-安装"><a href="#二-安装" class="headerlink" title="二 安装"></a>二 安装</h2><h3 id="一-配置本地仓库"><a href="#一-配置本地仓库" class="headerlink" title="一. 配置本地仓库"></a>一. 配置本地仓库</h3><p><a href="https://docs.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_create_local_package_repo.html">https://docs.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_create_local_package_repo.html</a></p><h4 id="1-配置web服务器"><a href="#1-配置web服务器" class="headerlink" title="1. 配置web服务器"></a>1. 配置web服务器</h4></li><li><p>安装</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo yum install httpd<br><br></code></pre></td></tr></table></figure><p>需保证/var/www磁盘空间足够，建议挂载，这里假设 /home/www 有足够的空间</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">mv /var/www /var/www2<br>mkdir /home/www<br>ln -s /home/www /var/www<br>mv /var/www2/* /var/www/<br>rm -rf /var/www2<br>ls /home/www<br></code></pre></td></tr></table></figure></li><li><p>启动</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo systemctl start httpd<br><br></code></pre></td></tr></table></figure><h4 id="2-下载文件-clouderaManager和cdh）"><a href="#2-下载文件-clouderaManager和cdh）" class="headerlink" title="2. 下载文件(clouderaManager和cdh）"></a>2. 下载文件(clouderaManager和cdh）</h4></li><li><p>clouderaManager<br>下载：<a href="https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/cm6.3.1-redhat7.tar.gz">https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/cm6.3.1-redhat7.tar.gz</a></p></li></ol><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo mkdir -p /var/www/html/cloudera-repos/cm6<br>tar xvfz cm6.3.1-redhat7.tar.gz -C /var/www/html/cloudera-repos/cm6 --strip-components=1<br>sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cm6<br></code></pre></td></tr></table></figure><ol start="2"><li>cdh</li></ol><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo mkdir -p /var/www/html/cloudera-repos<br>sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cdh6/6.3.2/redhat7/ -P /var/www/html/cloudera-repos<br><br>sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/6.3.2/redhat7/ -P /var/www/html/cloudera-repos<br><br>sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6<br>sudo chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6<br></code></pre></td></tr></table></figure><p>访问 http://<host>/cloudera-repos/ ，应该可以看到下载的文件</p><h4 id="3-配置使用本地存储库"><a href="#3-配置使用本地存储库" class="headerlink" title="3. 配置使用本地存储库"></a>3. 配置使用本地存储库</h4><p>创建/etc/yum.repos.d/cloudera-repo.repo<br>内容如下：</p><p>[cloudera-repo]<br>name=cloudera-repo<br>baseurl=<a href="http://paas-241/cloudera-repos/cm6">http://paas-241/cloudera-repos/cm6</a><br>enabled=1<br>gpgcheck=0</p><h3 id="三-安装CDH"><a href="#三-安装CDH" class="headerlink" title="三. 安装CDH"></a>三. 安装CDH</h3><h4 id="1-安装java"><a href="#1-安装java" class="headerlink" title="1. 安装java"></a>1. 安装java</h4> <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum install -y oracle-j2sdk1.8<br></code></pre></td></tr></table></figure><h4 id="2-安装CM"><a href="#2-安装CM" class="headerlink" title="2. 安装CM"></a>2. 安装CM</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server<br></code></pre></td></tr></table></figure><h4 id="3-安装数据库"><a href="#3-安装数据库" class="headerlink" title="3. 安装数据库"></a>3. 安装数据库</h4><p>假设使用mysql且已安装</p><p>在cm服务器上：</p><h5 id="1-安装jdbc驱动"><a href="#1-安装jdbc驱动" class="headerlink" title="1. 安装jdbc驱动"></a>1. 安装jdbc驱动</h5><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs shell">wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz<br><br>tar zxvf mysql-connector-java-5.1.46.tar.gz<br><br>sudo mkdir -p /usr/share/java/<br><br>cd mysql-connector-java-5.1.46<br><br>sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar<br><br></code></pre></td></tr></table></figure><h5 id="2-创建数据库并授权"><a href="#2-创建数据库并授权" class="headerlink" title="2. 创建数据库并授权"></a>2. 创建数据库并授权</h5><p>登录mysql进行配置：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">mysql -u root -h 192.168.10.200 -p<br><br></code></pre></td></tr></table></figure><p>执行以下命令：假设数据库为cdh 授权用户为root，密码为123456<br> <figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">CREATE</span> DATABASE cdh <span class="hljs-keyword">DEFAULT</span> <span class="hljs-type">CHARACTER</span> <span class="hljs-keyword">SET</span> utf8 <span class="hljs-keyword">DEFAULT</span> <span class="hljs-keyword">COLLATE</span> utf8_general_ci;<br><span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">ALL</span> <span class="hljs-keyword">ON</span> cdh.<span class="hljs-operator">*</span> <span class="hljs-keyword">TO</span> <span class="hljs-string">'root'</span>@<span class="hljs-string">'%'</span> IDENTIFIED <span class="hljs-keyword">BY</span> <span class="hljs-string">'123456'</span>;<br><span class="hljs-keyword">SHOW</span> DATABASES;<br><span class="hljs-keyword">SHOW</span> GRANTS <span class="hljs-keyword">FOR</span> <span class="hljs-string">'root'</span>@<span class="hljs-string">'%'</span>;<br><br></code></pre></td></tr></table></figure></p><h5 id="3-执行创建脚本"><a href="#3-执行创建脚本" class="headerlink" title="3. 执行创建脚本"></a>3. 执行创建脚本</h5><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell"><br>/opt/cloudera/cm/schema/scm_prepare_database.sh -h 192.168.10.200 mysql cdh root 123456<br><br></code></pre></td></tr></table></figure><h4 id="四-启动"><a href="#四-启动" class="headerlink" title="四 启动"></a>四 启动</h4> <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo systemctl enable cloudera-scm-server<br>sudo systemctl start cloudera-scm-server<br>sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log<br><br></code></pre></td></tr></table></figure><p>直到看到：<br>INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.<br>则说明启动完成，打开<br><a href="http://paas-241:7180/">http://paas-241:7180</a> 即可 admin/admin</p><p>注意配置自定义仓库：<br><a href="http://192.168.10.241/cloudera-repos/cdh6/6.3.2/redhat7/yum/">http://192.168.10.241/cloudera-repos/cdh6/6.3.2/redhat7/yum/</a><br>方法是数据表</p>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> CDH </tag>
</tags>
</entry>
<entry>
<title>oracle_logminer学习和实践笔记</title>
<link href="//oracle_logminer/"/>
<url>//oracle_logminer/</url>
<content type="html"><![CDATA[<p>Oracle LogMiner是Oracle公司从产品8i以后提供的一个实际非常有用的分析工具，使用该工具可以轻松获得Oracle 在线/归档日志文件中的具体内容，特别是该工具可以分析出所有对于数据库操作的DML和DDL语句。该工具特别适用于调试、审计或者回退某个特定的事务。</p><span id="more"></span><h2 id="优先参考资料"><a href="#优先参考资料" class="headerlink" title="优先参考资料"></a>优先参考资料</h2><p>华为相关配置： <a href="https://support.huaweicloud.com/usermanual-roma/fdi-ug-190624017.html">https://support.huaweicloud.com/usermanual-roma/fdi-ug-190624017.html</a><br>flink-cdc相关配置： <a href="https://ververica.github.io/flink-cdc-connectors/master/content/connectors/oracle-cdc.html#setup-oracle">https://ververica.github.io/flink-cdc-connectors/master/content/connectors/oracle-cdc.html#setup-oracle</a></p><h2 id="一-搭建环境"><a href="#一-搭建环境" class="headerlink" title="一 搭建环境"></a>一 搭建环境</h2><h3 id="1-使用docker提供oracle运行环境"><a href="#1-使用docker提供oracle运行环境" class="headerlink" title="1 使用docker提供oracle运行环境"></a>1 使用docker提供oracle运行环境</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker run -d --name my-oracle --restart=always -p 49161:1521 -e ORACLE_ALLOW_REMOTE=true wnameless/oracle-xe-11g-r2<br></code></pre></td></tr></table></figure><p>连接配置如下：</p><blockquote><p>hostname: localhost<br>port: 49161<br>sid: xe<br>username: system 管理员：sys as sysdba<br>password: oracle</p></blockquote><p>以下操作需在sql plus环境下执行</p><blockquote><p>navicat 配置sql plus方法如下：<br>到 <a href="https://www.oracle.com/database/technologies/instant-client/winx64-64-downloads.html">https://www.oracle.com/database/technologies/instant-client/winx64-64-downloads.html</a> 下载对应版本的 Instant Client Package - Basic 基础安装包 和 Instant Client Package - SQL*Plus，解压到同一目录，在navicat - 选项 - 环境 里配置该目录即可</p></blockquote><p>不过还是建议直接到容器里面执行</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 进入容器</span><br>docker exec -it my-oracle bash<br><span class="hljs-meta">#</span><span class="bash"> 以系统管理员登录sqlplus</span><br>sqlplus sys/oracle as sysdba<br></code></pre></td></tr></table></figure><p>也可以创建一个系统管理员，如下，其中”CONNECT”, “DBA”, “RESOURCE”是角色，SYSDBA 是服务器权限，不能同时grant</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">USER</span> "LOGMINER01" IDENTIFIED <span class="hljs-keyword">BY</span> "123456";<br><br><span class="hljs-keyword">GRANT</span> "CONNECT", "DBA", "RESOURCE" <span class="hljs-keyword">TO</span> "LOGMINER01";<br><br><span class="hljs-keyword">GRANT</span> SYSDBA <span class="hljs-keyword">TO</span> "LOGMINER01";<br></code></pre></td></tr></table></figure><h3 id="2-加入功能包"><a href="#2-加入功能包" class="headerlink" title="2 加入功能包"></a>2 加入功能包</h3><p>通常在安装数据库后就已经安装了Logminer，要查看数据库是否安装了LogMiner，只需查看数据库中是否已经有了dbms_logmnr和dbms_logmnr_d这2个package，如果有了，则已经安装，如果没有，执行下面两个脚本即可</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">@/u01/app/oracle/product/11.2.0/xe/rdbms/admin/dbmslm.sql<br>@/u01/app/oracle/product/11.2.0/xe/rdbms/admin/dbmslmd.sql<br><br></code></pre></td></tr></table></figure><p>如果环境不一样，不知道对应的文件位置，建议直接 find / -name ‘dbmslm.sql’<br><img src="https://img-blog.csdnimg.cn/20200917224107575.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="加入功能包"></p><h3 id="3-启用补充日志"><a href="#3-启用补充日志" class="headerlink" title="3 启用补充日志"></a>3 启用补充日志</h3><p>补充日志有几种类型，SUPPLEMENTAL LOG DATA是最小补充日志，这里推荐使用SUPPLEMENTAL LOG DATA (ALL) COLUMNS，当数据行更新时会记录完整的字段数据（而非只有更新的字段数据）</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-comment">-- 最小补充日志</span><br><span class="hljs-comment">-- ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;</span><br><span class="hljs-keyword">ALTER</span> DATABASE <span class="hljs-keyword">ADD</span> SUPPLEMENTAL LOG DATA (<span class="hljs-keyword">ALL</span>) COLUMNS;<br><span class="hljs-keyword">SELECT</span> SUPPLEMENTAL_LOG_DATA_MIN <span class="hljs-keyword">FROM</span> V$DATABASE;<br><br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20200917225120914.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="启用补充日志"></p><h3 id="4-开启archivelog-mode"><a href="#4-开启archivelog-mode" class="headerlink" title="4 开启archivelog mode"></a>4 开启archivelog mode</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">SHUTDOWN IMMEDIATE<br>STARTUP MOUNT<br>ALTER DATABASE ARCHIVELOG;<br>ALTER DATABASE OPEN;<br>archive log list;<br><br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20200917225424282.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="开启archivelog mode"></p><h3 id="5-创建日常维护账号"><a href="#5-创建日常维护账号" class="headerlink" title="5 创建日常维护账号"></a>5 创建日常维护账号</h3><p>需要注意，最后的 ALTER system 权限过大，如果不需要使用alter system switch logfile语句的，可以不授权。<br>alter system switch logfile 的具体说明参考：<a href="https://blog.csdn.net/fred_yang2013/article/details/45171283">https://blog.csdn.net/fred_yang2013/article/details/45171283</a></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-comment">-- 创建user1用户</span><br><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">USER</span> "USER1" IDENTIFIED <span class="hljs-keyword">BY</span> "123456";<br><span class="hljs-comment">-- 授予 连接 权限</span><br><span class="hljs-keyword">GRANT</span> "CONNECT","RESOURCE" <span class="hljs-keyword">TO</span> "USER1";<br><span class="hljs-comment">-- 授予 使用LogMiner PL / SQL程序包 权限</span><br><span class="hljs-keyword">GRANT</span> EXECUTE_CATALOG_ROLE <span class="hljs-keyword">TO</span> "USER1";<br><span class="hljs-comment">-- 授予 查询V$ARCHIVED_LOG视图 权限</span><br><span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">SELECT</span> <span class="hljs-keyword">ON</span>V_$ARCHIVED_LOG <span class="hljs-keyword">TO</span> "USER1";<br><span class="hljs-comment">-- 授予 使用DBMS_LOGMNR_D程序 权限（用于生成字典文件）</span><br><span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">execute</span> <span class="hljs-keyword">ON</span> DBMS_LOGMNR_D <span class="hljs-keyword">TO</span> "USER1";<br><span class="hljs-comment">-- 授予 使用DBMS_LOGMNR程序 权限 （用于生成分析会话）</span><br><span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">execute</span> <span class="hljs-keyword">ON</span> DBMS_LOGMNR <span class="hljs-keyword">TO</span> "USER1";<br><span class="hljs-comment">-- 授予 查询V$LOGMNR_CONTENTS TO视图 权限（必须拥有SELECT ANY TRANSACTION权限才能查询此视图）</span><br><span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">SELECT</span> <span class="hljs-keyword">ANY</span> TRANSACTION <span class="hljs-keyword">TO</span> USER1;<br><span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">SELECT</span> <span class="hljs-keyword">ON</span>V_$LOGMNR_CONTENTS <span class="hljs-keyword">TO</span> "USER1";<br><span class="hljs-comment">-- 授予 执行alter system switch logfile生成日志文件 权限</span><br><span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">ALTER</span> <span class="hljs-keyword">system</span> <span class="hljs-keyword">TO</span> "USER1";<br><br><br></code></pre></td></tr></table></figure><h2 id="二-常用命令"><a href="#二-常用命令" class="headerlink" title="二 常用命令"></a>二 常用命令</h2><h3 id="1-查看日志文件"><a href="#1-查看日志文件" class="headerlink" title="1 查看日志文件"></a>1 查看日志文件</h3><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">FROM</span> V$ARCHIVED_LOG ;<br></code></pre></td></tr></table></figure><h3 id="2-创建字典文件"><a href="#2-创建字典文件" class="headerlink" title="2 创建字典文件"></a>2 创建字典文件</h3><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">BEGIN</span><br> DBMS_LOGMNR_D.BUILD(OPTIONS<span class="hljs-operator">=</span><span class="hljs-operator">></span> DBMS_LOGMNR_D.STORE_IN_REDO_LOGS);<br><span class="hljs-keyword">end</span>;<br><br></code></pre></td></tr></table></figure><h3 id="3-创建归档日志"><a href="#3-创建归档日志" class="headerlink" title="3 创建归档日志"></a>3 创建归档日志</h3><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">alter</span> <span class="hljs-keyword">system</span> switch logfile<br></code></pre></td></tr></table></figure><h2 id="三-使用示例"><a href="#三-使用示例" class="headerlink" title="三 使用示例"></a>三 使用示例</h2><pre><code class="hljs"> 0. 先执行归档操作 1. 根据给定范围，获取要分析的文件列表 2. 先生成字典，再根据1，选定最近的字典.字典可能有多文件，需找出最后一个 2.0 生成字典文件 2.1 根据1的文件找出1之前最近的一个字典结束文件 2.2 根据2.1文件找出2.1之前最近的一个字典开始文件 3. 全部加入要分析的文件列表中 4. 执行分析</code></pre><h3 id="0-归档"><a href="#0-归档" class="headerlink" title="0 归档"></a>0 归档</h3><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">alter</span> <span class="hljs-keyword">system</span> switch logfile<br></code></pre></td></tr></table></figure><h3 id="1-获取要分析的文件列表"><a href="#1-获取要分析的文件列表" class="headerlink" title="1 获取要分析的文件列表"></a>1 获取要分析的文件列表</h3><p>如下，可以根据scn序号范围或者时间范围获取归档文件列表。<br>这里可以限制待会要分析的文件数量，减少服务器工作量。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> recid , stamp , name , sequence# <span class="hljs-keyword">as</span> sequence, next_change# <span class="hljs-keyword">as</span> nextChange,next_time <span class="hljs-keyword">as</span> nextTime,completion_time <span class="hljs-keyword">as</span> completionTime ,dictionary_begin <span class="hljs-keyword">as</span> dictionaryBegin,dictionary_end <span class="hljs-keyword">as</span> dictionaryEnd<br><span class="hljs-keyword">FROM</span> V$ARCHIVED_LOG <br><span class="hljs-keyword">WHERE</span> NEXT_CHANGE# <span class="hljs-operator">>=</span> ${beginScn?c} <span class="hljs-keyword">AND</span><br>FIRST_CHANGE# <span class="hljs-operator"><=</span> ${endScn?c} <span class="hljs-keyword">AND</span><br>NEXT_TIME <span class="hljs-operator">>=</span> to_date(<span class="hljs-string">'${beginTime}'</span>,<span class="hljs-string">'YYYYMMDDHH24MISS'</span>) <span class="hljs-keyword">AND</span><br>FIRST_TIME <span class="hljs-operator"><=</span> to_date(<span class="hljs-string">'${endTime}'</span>,<span class="hljs-string">'YYYYMMDDHH24MISS'</span>) <span class="hljs-keyword">AND</span><br><span class="hljs-number">1</span> <span class="hljs-operator">=</span> <span class="hljs-number">1</span><br></code></pre></td></tr></table></figure><h3 id="2-选择字典文件"><a href="#2-选择字典文件" class="headerlink" title="2 选择字典文件"></a>2 选择字典文件</h3><p>这一步参考：<a href="https://docs.oracle.com/en/database/oracle/oracle-database/12.2/sutil/oracle-logminer-utility.html#GUID-90944343-46BB-4BD5-A0C6-7A4B79D9BEF0">https://docs.oracle.com/en/database/oracle/oracle-database/12.2/sutil/oracle-logminer-utility.html#GUID-90944343-46BB-4BD5-A0C6-7A4B79D9BEF0</a></p><h4 id="2-0-生成字典文件"><a href="#2-0-生成字典文件" class="headerlink" title="2.0. 生成字典文件"></a>2.0. 生成字典文件</h4><p>这一步非必要，个人习惯，每次多生成一份字典文件</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">BEGIN</span><br> DBMS_LOGMNR_D.BUILD(OPTIONS<span class="hljs-operator">=</span><span class="hljs-operator">></span> DBMS_LOGMNR_D.STORE_IN_REDO_LOGS);<br><span class="hljs-keyword">end</span>;<br></code></pre></td></tr></table></figure><h4 id="2-1-根据1的文件找出1之前最近的一个字典结束文件"><a href="#2-1-根据1的文件找出1之前最近的一个字典结束文件" class="headerlink" title="2.1 根据1的文件找出1之前最近的一个字典结束文件"></a>2.1 根据1的文件找出1之前最近的一个字典结束文件</h4><p>将${minSequence?c}替换成第1步获取到的分析文件列表中最小的的sequence</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> recid , stamp , name , sequence# sequence, next_change# <span class="hljs-keyword">as</span> nextChange,next_time <span class="hljs-keyword">as</span> nextTime,completion_time <span class="hljs-keyword">as</span> completionTime ,dictionary_begin <span class="hljs-keyword">as</span> dictionaryBegin,dictionary_end <span class="hljs-keyword">as</span> dictionaryEnd<br><span class="hljs-keyword">FROM</span> V$ARCHIVED_LOG<br><span class="hljs-keyword">WHERE</span> SEQUENCE# <span class="hljs-operator">=</span> (<br><span class="hljs-keyword">SELECT</span> <span class="hljs-built_in">MAX</span> (SEQUENCE#) <span class="hljs-keyword">FROM</span> V$ARCHIVED_LOG<br><span class="hljs-keyword">WHERE</span> DICTIONARY_END <span class="hljs-operator">=</span> <span class="hljs-string">'YES'</span><br><span class="hljs-keyword">and</span> SEQUENCE# <span class="hljs-operator"><=</span> ${minSequence?c} )<br></code></pre></td></tr></table></figure><p>注意，如果这一步没有符合条件的数据，说明分析的范围之前没有字典文件生成，这里我的处理方法是直接把子查询改成查最远的一个字典文件：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> <span class="hljs-built_in">MIN</span> (SEQUENCE#) <span class="hljs-keyword">FROM</span> V$ARCHIVED_LOG <br><span class="hljs-keyword">WHERE</span> DICTIONARY_END <span class="hljs-operator">=</span> <span class="hljs-string">'YES'</span><br></code></pre></td></tr></table></figure><h4 id="2-2-根据2-1文件找出2-1之前最近的一个字典开始文件"><a href="#2-2-根据2-1文件找出2-1之前最近的一个字典开始文件" class="headerlink" title="2.2 根据2.1文件找出2.1之前最近的一个字典开始文件"></a>2.2 根据2.1文件找出2.1之前最近的一个字典开始文件</h4><p>将${minSequence?c}替换成2.1找出的那个字典结束文件的序号</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> recid , stamp , name , sequence# sequence, next_change# <span class="hljs-keyword">as</span> nextChange,next_time <span class="hljs-keyword">as</span> nextTime,completion_time <span class="hljs-keyword">as</span> completionTime ,dictionary_begin <span class="hljs-keyword">as</span> dictionaryBegin,dictionary_end <span class="hljs-keyword">as</span> dictionaryEnd<br><span class="hljs-keyword">FROM</span> V$ARCHIVED_LOG<br><span class="hljs-keyword">WHERE</span> SEQUENCE# <span class="hljs-operator">=</span> (<br><span class="hljs-keyword">SELECT</span> <span class="hljs-built_in">MAX</span> (SEQUENCE#) <span class="hljs-keyword">FROM</span> V$ARCHIVED_LOG<br><span class="hljs-keyword">WHERE</span> DICTIONARY_BEGIN <span class="hljs-operator">=</span> <span class="hljs-string">'YES'</span><br><span class="hljs-keyword">and</span> SEQUENCE# <span class="hljs-operator"><=</span> ${minSequence?c} )<br></code></pre></td></tr></table></figure><h3 id="3-全部加入要分析的文件列表中"><a href="#3-全部加入要分析的文件列表中" class="headerlink" title="3. 全部加入要分析的文件列表中"></a>3. 全部加入要分析的文件列表中</h3><p>把1、2找到的文件都丢进去分析列表中<br>这里用freemarker语法表达如下</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">BEGIN</span><br><span class="hljs-operator"><</span>#list logFileNameSet <span class="hljs-keyword">as</span> logFile<span class="hljs-operator">></span><br> DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME <span class="hljs-operator">=</span><span class="hljs-operator">></span> <span class="hljs-string">'${logFile}'</span><span class="hljs-operator"><</span>#if logFile_index<span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-number">0</span><span class="hljs-operator">></span>,OPTIONS <span class="hljs-operator">=</span><span class="hljs-operator">></span> DBMS_LOGMNR.NEW<span class="hljs-operator"><</span><span class="hljs-operator">/</span>#if<span class="hljs-operator">></span>);<br><span class="hljs-operator"><</span><span class="hljs-operator">/</span>#list<span class="hljs-operator">></span><br><span class="hljs-keyword">END</span>;<br></code></pre></td></tr></table></figure><h3 id="4-启动logminer"><a href="#4-启动logminer" class="headerlink" title="4. 启动logminer"></a>4. 启动logminer</h3><p>直接开始即可</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">BEGIN</span><br>DBMS_LOGMNR.START_LOGMNR ( OPTIONS <span class="hljs-operator">=</span><span class="hljs-operator">></span> DBMS_LOGMNR.DICT_FROM_REDO_LOGS );<br><span class="hljs-keyword">END</span>;<br><br></code></pre></td></tr></table></figure><h3 id="5-从视图中查询"><a href="#5-从视图中查询" class="headerlink" title="5. 从视图中查询"></a>5. 从视图中查询</h3><p>如下，从v$logmnr_contents 中查询</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">select</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">from</span> v$logmnr_contents <span class="hljs-keyword">where</span> TABLE_NAME<span class="hljs-operator">=</span>? <span class="hljs-keyword">order</span> <span class="hljs-keyword">by</span> SCN <span class="hljs-keyword">asc</span><br></code></pre></td></tr></table></figure><h3 id="6-退出logminer"><a href="#6-退出logminer" class="headerlink" title="6. 退出logminer"></a>6. 退出logminer</h3><p>每次分析只对当前会话有效，会话关闭后自动退出logminer。建议手动关闭</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">BEGIN</span><br>DBMS_LOGMNR.END_LOGMNR();<br><span class="hljs-keyword">END</span>;<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 后端开发 </category>
</categories>
<tags>
<tag> oracle </tag>
<tag> logminer </tag>
</tags>
</entry>
<entry>
<title>Kubernetes应用中使用TLS(SSL)证书的两种方法及实践</title>
<link href="//kubernetes-tls-ssl-certificates/"/>
<url>//kubernetes-tls-ssl-certificates/</url>
<content type="html"><![CDATA[<p>在k8s应用注入自签发的TLS/SSL证书有两种思路：1.使用certificates.k8s.io API 进行签发。2. 直接利用自己的CA证书进行签发。一般推荐第二种方法，本文记录了两种方法的完整实践并最后将其转换为JAVA的使用格式。</p><span id="more"></span><h2 id="零-前言"><a href="#零-前言" class="headerlink" title="零 前言"></a>零 前言</h2><h3 id="流程图"><a href="#流程图" class="headerlink" title="流程图"></a>流程图</h3><p>阅读顺序参照下图</p><pre><code class=" mermaid">graph LRA[ peer-config.json] --> C[ ca.pem]CC --> D( k8s API)C --> E( 直接签发)</code></pre><h3 id="比较"><a href="#比较" class="headerlink" title="比较"></a>比较</h3><p>直接签发方式是传统方式，只不过是使用了secret技术进行传递而已（换成nfs也可以），本身与k8s没有关系<br>certificates.k8s.io API 方式一般用于kubernetes内部组件，一般不建议与业务系统耦合。</p><h2 id="一-创建证书签名请求配置（peer-config-json）"><a href="#一-创建证书签名请求配置（peer-config-json）" class="headerlink" title="一 创建证书签名请求配置（peer-config.json）"></a>一 创建证书签名请求配置（peer-config.json）</h2><p>这个东西用于确定最终签发的目标证书的内容，示例如下：<br>这个配置生成的证书可用于linshenkx.com做https用（建议hosts把可能用到的都写上，特别是一个证书给多个节点用的情况，否则以后用的时候会有证书hostname不匹配的警告）</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs yaml">{<br> <span class="hljs-attr">"hosts":</span> [<br> <span class="hljs-string">"linshenkx.com"</span><br> ],<br> <span class="hljs-attr">"CN":</span> <span class="hljs-string">"linshenkx"</span>,<br> <span class="hljs-attr">"key":</span> {<br> <span class="hljs-attr">"algo":</span> <span class="hljs-string">"ecdsa"</span>,<br> <span class="hljs-attr">"size":</span> <span class="hljs-number">256</span><br> }<br>}<br><br></code></pre></td></tr></table></figure><h2 id="二-获取CA证书（ca-pem）"><a href="#二-获取CA证书（ca-pem）" class="headerlink" title="二 获取CA证书（ca.pem）"></a>二 获取CA证书（ca.pem）</h2><h3 id="1-准备好openssl工具"><a href="#1-准备好openssl工具" class="headerlink" title="1. 准备好openssl工具"></a>1. 准备好openssl工具</h3><p>下载及使用参考：<a href="https://kubernetes.io/zh/docs/concepts/cluster-administration/certificates/#cfssl">https://kubernetes.io/zh/docs/concepts/cluster-administration/certificates/#cfssl</a><br>如下：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o cfssl<br>curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o cfssljson<br>curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o cfssl-certinfo<br>mv cfssl* /usr/local/bin/<br>chmod +x /usr/local/bin/cfssl*<br><br></code></pre></td></tr></table></figure><h3 id="2-创建-CA证书签名申请-配置文件（ca-csr-config-json）"><a href="#2-创建-CA证书签名申请-配置文件（ca-csr-config-json）" class="headerlink" title="2. 创建 CA证书签名申请 配置文件（ca-csr-config.json）"></a>2. 创建 CA证书签名申请 配置文件（ca-csr-config.json）</h3><p>这个东西即代表CA本身的概念，可用于生成CA的公钥密钥。<br>使用 <code>cfssl print-defaults csr > ca-csr-config.json</code> 可获取默认配置如下</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs yaml">{<br> <span class="hljs-attr">"CN":</span> <span class="hljs-string">"example.net"</span>,<br> <span class="hljs-attr">"hosts":</span> [<br> <span class="hljs-string">"example.net"</span>,<br> <span class="hljs-string">"www.example.net"</span><br> ],<br> <span class="hljs-attr">"key":</span> {<br> <span class="hljs-attr">"algo":</span> <span class="hljs-string">"ecdsa"</span>,<br> <span class="hljs-attr">"size":</span> <span class="hljs-number">256</span><br> },<br> <span class="hljs-attr">"names":</span> [<br> {<br> <span class="hljs-attr">"C":</span> <span class="hljs-string">"US"</span>,<br> <span class="hljs-attr">"L":</span> <span class="hljs-string">"CA"</span>,<br> <span class="hljs-attr">"ST":</span> <span class="hljs-string">"San Francisco"</span><br> }<br> ]<br>}<br></code></pre></td></tr></table></figure><p>根据需求去配置，参考如下</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs yaml">{<br> <span class="hljs-attr">"CN":</span> <span class="hljs-string">"LinShen Self Signed Ca"</span>,<br> <span class="hljs-attr">"key":</span> {<br> <span class="hljs-attr">"algo":</span> <span class="hljs-string">"rsa"</span>,<br> <span class="hljs-attr">"size":</span> <span class="hljs-number">2048</span><br> },<br> <span class="hljs-attr">"names":</span> [{<br> <span class="hljs-attr">"C":</span> <span class="hljs-string">"CN"</span>,<br> <span class="hljs-attr">"ST":</span> <span class="hljs-string">"GD"</span>,<br> <span class="hljs-attr">"L":</span> <span class="hljs-string">"GZ"</span>,<br> <span class="hljs-attr">"O":</span> <span class="hljs-string">"LinShen"</span>,<br> <span class="hljs-attr">"OU":</span> <span class="hljs-string">"LS"</span><br> }]<br>}<br></code></pre></td></tr></table></figure><h3 id="3-生成-CA密钥及证书"><a href="#3-生成-CA密钥及证书" class="headerlink" title="3. 生成 CA密钥及证书"></a>3. 生成 CA密钥及证书</h3><p>有了CA的CSR配置就可以生成CA密钥及证书了<br>生成ca.pem(CA证书，含公钥)、ca.csr（步骤2对应的csr，没什么用了）、ca-key.pem(CA私钥,需妥善保管)</p><figure class="highlight stata"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs stata">cfssl gencert -initca <span class="hljs-keyword">ca</span>-csr-config.json | cfssljson -bare <span class="hljs-keyword">ca</span> -<br><br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20200826234715284.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="生成 CA密钥及证书"></p><p>使用思路1转一，思路2转二</p><h2 id="三-certificates-k8s-io-API"><a href="#三-certificates-k8s-io-API" class="headerlink" title="三 certificates.k8s.io API"></a>三 certificates.k8s.io API</h2><h3 id="签发"><a href="#签发" class="headerlink" title="签发"></a>签发</h3><p>主要参考：<a href="https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster">https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster</a></p><h4 id="1-配置kubernetes使用自定义的CA根证书"><a href="#1-配置kubernetes使用自定义的CA根证书" class="headerlink" title="1 配置kubernetes使用自定义的CA根证书"></a>1 配置kubernetes使用自定义的CA根证书</h4><p>kube-controller-manager已经有了一个默认的实现了，如果想使用上文中我们创建的CA证书作为集群的根证书，则可如下：<br>Kubernetes 提供了一个 certificates.k8s.io API，可以使用配置的 CA 根证书来签发用户证书。该 API 由 kube-controller-manager 实现，其签发证书使用的根证书在下面的命令行中进行配置。我们希望 Kubernetes 采用集群根 CA 来签发用户证书，因此在 kube-controller-manager 的命令行参数中将相关参数配置为了集群根 CA。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">/usr/local/bin/kube-controller-manager \\<br>--cluster-signing-cert-file=ca.pem # 用于签发证书的 CA 根证书<br>--cluster-signing-key-file=ca-key.pem # 用于签发证书的 CA 根证书的私钥 <br>--experimental-cluster-signing-duration=438000h # 所签署的证书的有效期时长。(默认值：默认值：8760h0m0s 即一年)<br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 更多参数参考：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-controller-manager/</span></span><br></code></pre></td></tr></table></figure><h4 id="2-创建证书签名请求"><a href="#2-创建证书签名请求" class="headerlink" title="2 创建证书签名请求"></a>2 创建证书签名请求</h4><p>根据流程一的配置，运行以下命令生成私钥和证书签名请求（或CSR）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash">cfssl genkey ./peer-config.json | cfssljson -bare peer<br><br><br></code></pre></td></tr></table></figure><p>生成文件有 peer-key.pem（私钥，自己先留着）及peer.csr（证书签名请求，用于下一步申请证书）<br><img src="https://img-blog.csdnimg.cn/20200827210639951.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="创建证书签名请求"></p><h4 id="3-创建证书签名请求对象并发送到-Kubernetes-API"><a href="#3-创建证书签名请求对象并发送到-Kubernetes-API" class="headerlink" title="3 创建证书签名请求对象并发送到 Kubernetes API"></a>3 创建证书签名请求对象并发送到 Kubernetes API</h4><p>使用以下命令创建 CSR yaml 文件，并发送到 API server：<br>下面命令的作用为 将上一步peer.csr的内容用base64封装在一个CertificateSigningRequest对象里面。<br>其中usages详细选项参照：<a href="https://godoc.org/k8s.io/api/certificates/v1beta1#KeyUsage">https://godoc.org/k8s.io/api/certificates/v1beta1#KeyUsage</a></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs bash">cat <<<span class="hljs-string">EOF | kubectl create -f -</span><br><span class="hljs-string">apiVersion: certificates.k8s.io/v1beta1</span><br><span class="hljs-string">kind: CertificateSigningRequest</span><br><span class="hljs-string">metadata:</span><br><span class="hljs-string"> name: peer</span><br><span class="hljs-string">spec:</span><br><span class="hljs-string"> groups:</span><br><span class="hljs-string"> - system:authenticated</span><br><span class="hljs-string"> request: $(cat peer.csr | base64 | tr -d '\n')</span><br><span class="hljs-string"> usages:</span><br><span class="hljs-string"> - digital signature</span><br><span class="hljs-string"> - key encipherment</span><br><span class="hljs-string"> - server auth</span><br><span class="hljs-string"> - client auth</span><br><span class="hljs-string">EOF</span><br><br></code></pre></td></tr></table></figure><p>执行完毕会提醒：<code> certificatesigningrequest.certificates.k8s.io/peer created </code></p><h4 id="4-同意证书签名请求"><a href="#4-同意证书签名请求" class="headerlink" title="4. 同意证书签名请求"></a>4. 同意证书签名请求</h4><p>这一步应该等待集群管理员来做的</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash">kubectl certificate approve peer<br><br><br></code></pre></td></tr></table></figure><p>执行完毕会提醒： <code>certificatesigningrequest.certificates.k8s.io/peer approved</code></p><h4 id="5-导出证书"><a href="#5-导出证书" class="headerlink" title="5. 导出证书"></a>5. 导出证书</h4><p>通过后就可以导出成pem证书了</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash">kubectl get csr peer -o jsonpath=<span class="hljs-string">'{.status.certificate}'</span> \<br>| base64 -d > peer-cert.pem<br><br></code></pre></td></tr></table></figure><p>其实来到这一步就行了，证书已经签发完毕并且拿到手了，可以看到peer相关的有4个文件：<br>peer-config.json 、peer.csr 用于请求证书签名，可删除<br>peer-key.pem为私钥，peer-cert.pem为被签发的证书（含公钥）<br><img src="https://img-blog.csdnimg.cn/20200827212050426.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="导出证书"></p><h3 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h3><p>在kubernetes内使用需将证书文件peer-cert.pem和私钥peer-key.pem封装成secret传递进去</p><h4 id="1-创建secret"><a href="#1-创建secret" class="headerlink" title="1 创建secret"></a>1 创建secret</h4><p>命令参考自：<a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands">https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands</a><br><code>$ kubectl create tls NAME --cert=path/to/cert/file --key=path/to/key/file [--dry-run=server|client|none]</code></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl create secret tls peer --cert peer-cert.pem --key peer-key.pem<br><br><br></code></pre></td></tr></table></figure><p>可以看到，secret里面的私钥及证书为tls.key，tls.crt<br><img src="https://img-blog.csdnimg.cn/20200827220952946.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="创建secret"></p><h4 id="2-挂载secret"><a href="#2-挂载secret" class="headerlink" title="2 挂载secret"></a>2 挂载secret</h4><p>以下命令将secret peer挂载到/tmp/tls_secret下，而CA证书会自动加载到pod的/var/run/secrets/kubernetes.io/serviceaccount/ca.crt</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs yaml"> <span class="hljs-attr">volumeMounts:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">tls-secret</span><br> <span class="hljs-attr">mountPath:</span> <span class="hljs-string">/tmp/tls_secret</span><br> <span class="hljs-attr">readOnly:</span> <span class="hljs-literal">true</span><br> <br><span class="hljs-attr">volumes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">tls-secret</span><br> <span class="hljs-attr">secret:</span><br> <span class="hljs-attr">secretName:</span> <span class="hljs-string">peer</span><br></code></pre></td></tr></table></figure><p>综上，pod里有签发证书（tls.crt）及其私钥(tls.key)，和CA根证书(ca.crt)，已可用于业务系统的tls认证。</p><h2 id="四-直接签发"><a href="#四-直接签发" class="headerlink" title="四 直接签发"></a>四 直接签发</h2><p>测试的时候记得把方法一产生的peer相关文件移除掉，留下peer-config.json就好</p><h3 id="签发-1"><a href="#签发-1" class="headerlink" title="签发"></a>签发</h3><h4 id="1-创建-CA证书-配置文件（ca-config-json）"><a href="#1-创建-CA证书-配置文件（ca-config-json）" class="headerlink" title="1 创建 CA证书 配置文件（ca-config.json）"></a>1 创建 CA证书 配置文件（ca-config.json）</h4><p>这个东西用于以后签发CA证书时的配置（比分说证书有效期/用途 之类）<br>同样，使用<code>cfssl print-defaults config > ca-config.json</code>可获取默认配置，如下<br>如果签发证书的时候不指定profile，则证书有效期只有一周</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs yaml">{<br> <span class="hljs-attr">"signing":</span> {<br> <span class="hljs-attr">"default":</span> {<br> <span class="hljs-attr">"expiry":</span> <span class="hljs-string">"168h"</span><br> },<br> <span class="hljs-attr">"profiles":</span> {<br> <span class="hljs-attr">"www":</span> {<br> <span class="hljs-attr">"expiry":</span> <span class="hljs-string">"8760h"</span>,<br> <span class="hljs-attr">"usages":</span> [<br> <span class="hljs-string">"signing"</span>,<br> <span class="hljs-string">"key encipherment"</span>,<br> <span class="hljs-string">"server auth"</span><br> ]<br> },<br> <span class="hljs-attr">"client":</span> {<br> <span class="hljs-attr">"expiry":</span> <span class="hljs-string">"8760h"</span>,<br> <span class="hljs-attr">"usages":</span> [<br> <span class="hljs-string">"signing"</span>,<br> <span class="hljs-string">"key encipherment"</span>,<br> <span class="hljs-string">"client auth"</span><br> ]<br> }<br> }<br> }<br>}<br></code></pre></td></tr></table></figure><p>这里我的配置参考如下：<br>按服务端、客户端及双向认证进行分类，且有效期为50年（按365天算）</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><code class="hljs yaml">{<br> <span class="hljs-attr">"signing":</span> {<br> <span class="hljs-attr">"default":</span> {<br> <span class="hljs-attr">"expiry":</span> <span class="hljs-string">"438000h"</span><br> },<br> <span class="hljs-attr">"profiles":</span> {<br> <span class="hljs-attr">"server":</span> {<br> <span class="hljs-attr">"expiry":</span> <span class="hljs-string">"438000h"</span>,<br> <span class="hljs-attr">"usages":</span> [<br> <span class="hljs-string">"signing"</span>,<br> <span class="hljs-string">"key encipherment"</span>,<br> <span class="hljs-string">"server auth"</span><br> ]<br> },<br> <span class="hljs-attr">"client":</span> {<br> <span class="hljs-attr">"expiry":</span> <span class="hljs-string">"438000h"</span>,<br> <span class="hljs-attr">"usages":</span> [<br> <span class="hljs-string">"signing"</span>,<br> <span class="hljs-string">"key encipherment"</span>,<br> <span class="hljs-string">"client auth"</span><br> ]<br> },<br> <span class="hljs-attr">"peer":</span> {<br> <span class="hljs-attr">"expiry":</span> <span class="hljs-string">"438000h"</span>,<br> <span class="hljs-attr">"usages":</span> [<br> <span class="hljs-string">"signing"</span>,<br> <span class="hljs-string">"key encipherment"</span>,<br> <span class="hljs-string">"server auth"</span>,<br> <span class="hljs-string">"client auth"</span><br> ]<br> }<br> }<br> }<br>}<br></code></pre></td></tr></table></figure><h4 id="2-生成证书和私钥"><a href="#2-生成证书和私钥" class="headerlink" title="2 生成证书和私钥"></a>2 生成证书和私钥</h4><p>这里总共用到4个文件，意思为<br>使用ca公密钥（即代表ca本身）根据ca-config.json(即证书签名的配置)以其中peer的profile配置为peer-config.json代表的对象签发名为peer的证书</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash">cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer peer-config.json | cfssljson -bare peer<br><br></code></pre></td></tr></table></figure><p>可以看到多出了3个文件：<br>peer.csr : 证书签名请求，没用了<br>peer.pem : 证书（公钥）<br>peer-key.pem : 私钥<br>至此证书即签发完毕了<br><img src="https://img-blog.csdnimg.cn/20200827223420525.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="生成证书及私钥"></p><h3 id="使用-1"><a href="#使用-1" class="headerlink" title="使用"></a>使用</h3><h4 id="1-创建secret-1"><a href="#1-创建secret-1" class="headerlink" title="1 创建secret"></a>1 创建secret</h4><p>由于是直接使用我们的ca.pem、ca-key.pem进行签发的，所以我们是已经拥有ca.pem、peer-key.pem、peer.pem三件套了，这里讲一下将这三个文件传递到pod里面。<br>这里可以视为普通的将文件传递到pod里面，参考 <a href="https://kubernetes.io/zh/docs/concepts/configuration/secret/">https://kubernetes.io/zh/docs/concepts/configuration/secret/</a> 从文件生成secret<br>kubectl apply -k 使用方法参考： <a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands">https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands</a></p><p>脚本如下：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir kustomization && cp peer.pem peer-key.pem ca.pem kustomization <br>cd kustomization<br>cat <<EOF >./kustomization.yaml<br>secretGenerator:<br>- name: peer-ca<br> files:<br> - peer.pem<br> - peer-key.pem<br> - ca.pem<br>EOF<br>cd ..<br>kubectl apply -k kustomization <br><br></code></pre></td></tr></table></figure><p>需要注意的是，这里的secret名字是随机的，需要自己记下来<br><img src="https://img-blog.csdnimg.cn/20200827224758858.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="创建secret"></p><h4 id="2-挂载secret-1"><a href="#2-挂载secret-1" class="headerlink" title="2 挂载secret"></a>2 挂载secret</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs yaml"> <span class="hljs-attr">volumeMounts:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">tls-secret</span><br> <span class="hljs-attr">mountPath:</span> <span class="hljs-string">/tmp/tls_secret</span><br> <span class="hljs-attr">readOnly:</span> <span class="hljs-literal">true</span><br> <br><span class="hljs-attr">volumes:</span><br> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">tls-secret</span><br> <span class="hljs-attr">secret:</span><br> <span class="hljs-attr">secretName:</span> <span class="hljs-string">peer-ca-dg97cgfkgg</span><br></code></pre></td></tr></table></figure><p>综上，pod里有签发证书（peer.pem）及其私钥(peer-key.pem)，和CA根证书(ca.pem)，已可用于业务系统的tls认证。</p><h2 id="五-业务系统使用（HADOOP-JAVA为例）"><a href="#五-业务系统使用（HADOOP-JAVA为例）" class="headerlink" title="五 业务系统使用（HADOOP-JAVA为例）"></a>五 业务系统使用（HADOOP-JAVA为例）</h2><p>该脚本将根据TLS_KEY、TLS_CRT、CA_CRT生成Java用的keystore证书到TARGET_DIR目录下，另外还将<br>脚本解释：</p><ol><li>使用openssl 将pem格式的证书先转成PKCS12再转成JKS 格式<br>TLS_KEY、TLS_CRT => DEST_KEYSTORE<br>参考自：<a href="https://docs.cloudera.com/documentation/enterprise/5-10-x/topics/cm_sg_openssl_jks.html">https://docs.cloudera.com/documentation/enterprise/5-10-x/topics/cm_sg_openssl_jks.html</a></li><li>CA_CRT => TRUST_KEYSTORE<br>使用 keytool -import 将CA证书导入自定义JKS 文件和JDK的JKS文件。</li></ol><p>另外需要注意，如果不同JDK版本的 cacerts 位置是不一样的：<br>1.8及以下：<code>$JAVA_HOME/jre/lib/security/cacerts</code><br>9及以上：<code>$JAVA_HOME/lib/security/cacerts </code><br>有空再改一下脚本写成通用的</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><code class="hljs shell"><br><span class="hljs-meta"> #</span><span class="bash">!/bin/bash</span><br> set -e<br> <br><span class="hljs-meta"> #</span><span class="bash"> 使用certificates.k8s.io API方式</span><br> TLS_KEY=/tmp/tls_secret/tls.key<br> TLS_CRT=/tmp/tls_secret/tls.crt<br> CA_CRT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt<br><br><span class="hljs-meta"> #</span><span class="bash"> 使用直接签发方式</span><br><span class="hljs-meta"> #</span><span class="bash"> TLS_KEY=/tmp/tls_secret/peer.pem</span><br><span class="hljs-meta"> #</span><span class="bash"> TLS_CRT=/tmp/tls_secret/peer-key.pem</span><br><span class="hljs-meta"> #</span><span class="bash"> CA_CRT=/tmp/tls_secret/ca.pem</span><br><br> PASSWORD=1qaz@WSX<br> TARGET_DIR=/keystore<br> PKCS12_OUTPUT=$TARGET_DIR/keystore.pkcs12<br> DEST_KEYSTORE=$TARGET_DIR/my.jks<br> TRUST_KEYSTORE=$TARGET_DIR/my-truststore.jks<br> <br> ALIAS_NAME="service"<br><br>mkdir -p $TARGET_DIR<br> openssl "pkcs12" -export -inkey "${TLS_KEY}" -in "${TLS_CRT}" -out "${PKCS12_OUTPUT}" -password "pass:${PASSWORD}"<br> keytool -importkeystore -noprompt -srckeystore "${PKCS12_OUTPUT}" -srcstoretype "pkcs12" -destkeystore "${DEST_KEYSTORE}" -storepass "${PASSWORD}" -srcstorepass "${PASSWORD}"<br><br> csplit -z -f crt- ${CA_CRT} '/-----BEGIN CERTIFICATE-----/' '{*}'<br><br> for file in crt-*; do<br> keytool -import -noprompt -keystore "${TRUST_KEYSTORE}" -file "${file}" -storepass "${PASSWORD}" -alias ${ALIAS_NAME}-$file;<br> done<br> for file in crt-*; do<br> keytool -import -noprompt -keystore $JAVA_HOME/jre/lib/security/cacerts -file "${file}" -storepass "changeit" -alias ${ALIAS_NAME}-$file;<br> <br> done<br> rm -f crt-*<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> Kubernetes </category>
</categories>
</entry>
<entry>
<title>Xloggc实践（JVM1.8及之前）</title>
<link href="//Xloggc/"/>
<url>//Xloggc/</url>
<content type="html"><![CDATA[<p>Java服务器调优免不了要对gc日志进行分析，我一般是上传gc日志文件到GCEasy进行处理的，上传的文件有大小限制。另外默认的gc日志打印还会存在重启后丢失的问题。综上，我们希望gc日志文件在不能丢失（但允许超过一定时间或大小被清理掉）的情况下控制gc日志的大小或者按时间切割，即像Java日志框架那样的效果。Java9对jvm的日志系统进行了比较大的升级，可以比较好的实现这些需求，但大部分服务端的Java软件还只支持Jdk8，本文记录作者自己的相关配置。</p><span id="more"></span><h3 id="分析"><a href="#分析" class="headerlink" title="分析"></a>分析</h3><p>首先推荐看两篇gceasy的博客文章，网上对UseGCLogFileRotation的相关讨论很多都来自于这里：<br><a href="https://blog.gceasy.io/2016/11/15/rotating-gc-log-files/">https://blog.gceasy.io/2016/11/15/rotating-gc-log-files/</a><br><a href="https://blog.gceasy.io/2019/01/29/try-to-avoid-xxusegclogfilerotation/">https://blog.gceasy.io/2019/01/29/try-to-avoid-xxusegclogfilerotation/</a></p><p>简单来说，UseGCLogFileRotation 可以控制gc日志文件大小，且按日期分片切割<br>缺点是：</p><ol><li>记录丢失<br>个人认为不是问题，超过指定文件数量及大小的日志被丢弃是预期操作。</li><li>循环打印<br> 若限制日志文件数共5个，分别为0、1、2、3、4，在文件4达到文件最大值后系统将从1开始覆写，最终的结果就是顺序错乱，不能直接使用（需要人为地修改文件名以修正顺序）</li><li>重启后从编号0开始写入，而非上次的写入位置。结合第2点你就会发现你的日志顺序已经完全不可信了。</li></ol><h3 id="应对方法"><a href="#应对方法" class="headerlink" title="应对方法"></a>应对方法</h3><ol start="0"><li>直接使用-Xloggc:gc-%t.log（<code>推荐</code>）<br>简单粗暴，缺点是文件依然太大，需要用自己切割</li><li>结合-Xloggc:gc-%t.log使用<br>使用 -Xloggc:gc-%t.log可以解决问题3，但仍存在循环打印。<ol><li>设置较大的文件数量和大小限制，尽量使其不产生循环打印<br>上面文章讨论区的一个小伙伴提出的，其实也是个办法<br>以下是我的日志文件输出测试，不同一个批次的文件数量限制可能不同，用蓝框隔离。</li><li>手动调整顺序（其实也不麻烦）。</li></ol></li><li>如果是业务系统的话直接用高版本的jdk吧，不用看这篇文章了</li></ol><h3 id="示例"><a href="#示例" class="headerlink" title="示例"></a>示例</h3><p>（使用两个gc日志文件，每个文件最大为1k）：<br>每一个批次都有一个current后缀的日志文件标识当前写入，同一批次的日志文件用从0开始的序号后缀隔开，通常current应该在序号最大的位置，如果不是则说明存在循环打印的情况。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">-Xmx60M -Xms60M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=1K<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20200820003502171.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="日志示例"></p>]]></content>
<categories>
<category> 后端开发 </category>
</categories>
<tags>
<tag> jvm </tag>
</tags>
</entry>
<entry>
<title>kafka动态调整副本因子replication.factor及json生成脚本</title>
<link href="//kafka_replication_factor/"/>
<url>//kafka_replication_factor/</url>
<content type="html"><![CDATA[<p>kafka默认的副本因子default.replication.factor是1，即无额外副本，如果在创建topic时没有指定副本数，则无高可用性。</p><span id="more"></span><h3 id="1-说明"><a href="#1-说明" class="headerlink" title="1. 说明"></a>1. 说明</h3><p>kafka默认的副本因子default.replication.factor是1，即无额外副本，如果在创建topic时没有指定副本数，则无高可用性。该参数在topic创建时生效，topic创建后无法直接对topic级别的副本数进行修改，但官方提供了在partition级别增加副本数的功能，用于集群添加节点的情况。<br>详情参考官方文档：<a href="https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor">https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor</a></p><p>简单来说就是使用json文件描述该topic每一个partition的情况，每一个partition包含副本分布的描述。然后使用 kafka-reassign-partitions.sh 执行安装json文件完成再分配任务即可。</p><h3 id="2-json生成脚本"><a href="#2-json生成脚本" class="headerlink" title="2. json生成脚本"></a>2. json生成脚本</h3><p>这里提供json的生成脚本，参考自：<a href="https://github.com/dkurzaj/generate-kafka-replication-factor-json/blob/master/generate-kafka-replication-factor-json.sh">https://github.com/dkurzaj/generate-kafka-replication-factor-json/blob/master/generate-kafka-replication-factor-json.sh</a></p><p>其中BROKER_IDS为要分配的brokerId,<br>NUMBER_OF_PARTITIONS为topic分区数</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><code class="hljs shell">BROKER_IDS=(1 2 3)<br>NUMBER_OF_PARTITIONS=5<br>TOPIC_NAME=__consumer_offsets<br><br>output_file="increase-rf-json/increase-replication-factor-"${TOPIC_NAME}".json"<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> Beginning of the file</span><br>echo '{"version":1,' > $output_file<br>echo ' "partitions":[' >> $output_file<br><br>current_broker_id_index=0<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> Responsible <span class="hljs-keyword">for</span> the circular array over the brokers IDs</span><br>set_next_broker(){<br> current_broker_id_index=$1<br> current_broker_id_index=$(($current_broker_id_index + 1))<br> current_broker_id_index=$(($current_broker_id_index % ${#BROKER_IDS[@]}))<br> return $current_broker_id_index<br>}<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> Forges the string containing the replicas brokers of a partition</span><br>get_brokers_string(){<br> current_broker_id_index=$1<br> brokers_string="${BROKER_IDS[$current_broker_id_index]}"<br> set_next_broker $current_broker_id_index<br> current_broker_id_index=$?<br> brokers_string="$brokers_string,${BROKER_IDS[$current_broker_id_index]}"<br> set_next_broker $current_broker_id_index<br> current_broker_id_index=$?<br> brokers_string="$brokers_string,${BROKER_IDS[$current_broker_id_index]}"<br> set_next_broker $current_broker_id_index<br> current_broker_id_index=$?<br> echo $brokers_string<br> return $current_broker_id_index<br>}<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> Create all the lines</span><br>partition_number=0<br>while (("$partition_number" < "$NUMBER_OF_PARTITIONS-1")); do<br> brokers_string=$(get_brokers_string $current_broker_id_index)<br> current_broker_id_index=$?<br> echo " {\"topic\":\"$TOPIC_NAME\",\"partition\":$partition_number,\"replicas\":[$brokers_string]}," >> $output_file<br> partition_number=$(($partition_number + 1))<br>done<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> Last line without trailing coma</span><br>brokers_string=$(get_brokers_string $current_broker_id_index)<br>echo " {\"topic\":\"$TOPIC_NAME\",\"partition\":$partition_number,\"replicas\":[$brokers_string]}" >> $output_file<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"> End of the file</span><br>echo ']}' >> $output_file<br><br>exit 0<br><br></code></pre></td></tr></table></figure><p>生成json内容如下：<br>可以看到这个文件本质上是对partition而非topic的描述</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs json">{<span class="hljs-attr">"version"</span>:<span class="hljs-number">1</span>,<br> <span class="hljs-attr">"partitions"</span>:[<br> {<span class="hljs-attr">"topic"</span>:<span class="hljs-string">"__consumer_offsets"</span>,<span class="hljs-attr">"partition"</span>:<span class="hljs-number">0</span>,<span class="hljs-attr">"replicas"</span>:[<span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>]},<br> {<span class="hljs-attr">"topic"</span>:<span class="hljs-string">"__consumer_offsets"</span>,<span class="hljs-attr">"partition"</span>:<span class="hljs-number">1</span>,<span class="hljs-attr">"replicas"</span>:[<span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>]},<br> {<span class="hljs-attr">"topic"</span>:<span class="hljs-string">"__consumer_offsets"</span>,<span class="hljs-attr">"partition"</span>:<span class="hljs-number">2</span>,<span class="hljs-attr">"replicas"</span>:[<span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>]},<br> {<span class="hljs-attr">"topic"</span>:<span class="hljs-string">"__consumer_offsets"</span>,<span class="hljs-attr">"partition"</span>:<span class="hljs-number">3</span>,<span class="hljs-attr">"replicas"</span>:[<span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>]},<br> {<span class="hljs-attr">"topic"</span>:<span class="hljs-string">"__consumer_offsets"</span>,<span class="hljs-attr">"partition"</span>:<span class="hljs-number">4</span>,<span class="hljs-attr">"replicas"</span>:[<span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>]}<br>]}<br><br></code></pre></td></tr></table></figure><h3 id="3-例子"><a href="#3-例子" class="headerlink" title="3. 例子"></a>3. 例子</h3><p>生成后执行命令格式如下（也可使用zookeeper代替bootstrap-server）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase-replication-factor.json --execute<br></code></pre></td></tr></table></figure><p>实例（有50个partition）：<br><img src="https://img-blog.csdnimg.cn/20200819094107721.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="execute"></p><p>使用verify代替execute即可查看执行进度<br><img src="https://img-blog.csdnimg.cn/20200819094400579.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70#pic_center" alt="verify"></p><h3 id="4-改进方向"><a href="#4-改进方向" class="headerlink" title="4. 改进方向"></a>4. 改进方向</h3><p>目前的shell脚本生成的partition副本分布固定为 BROKER_IDS ，适用于节点数和副本数相同的情况，如果有10节点而只要3副本就不行，生成的json会使副本集中的3个节点。<br>不过一般还是建议修改默认副本数或者创建topic时执行副本数而非使用这个脚本，如果是集群添加节点的情况，建议还是使用专业的带负载平衡功能的kafka管理系统。</p><h3 id="5-建议设置参数"><a href="#5-建议设置参数" class="headerlink" title="5. 建议设置参数"></a>5. 建议设置参数</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">offsets.topic.replication.factor=3<br>transaction.state.log.replication.factor=3<br>transaction.state.log.min.isr=3<br>default.replication.factor=3<br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 大数据 </category>
</categories>
<tags>
<tag> kafka </tag>
</tags>
</entry>
<entry>
<title>centos利用yumdownloader获取含依赖的rpm安装包</title>
<link href="//centos-yumdownloader-rpm-package/"/>
<url>//centos-yumdownloader-rpm-package/</url>
<content type="html"><![CDATA[<p>工作经常会遇到需要离线安装软件的情况，特此记录下来，避免每次都得去翻笔记。</p><span id="more"></span><h2 id="零-思路"><a href="#零-思路" class="headerlink" title="零 思路"></a>零 思路</h2><ol><li>使用docker提供目标包洁净隔离环境（如centos7、centos8）。</li><li>使用<code>yumdownloader -resolve</code>下载完整依赖。</li><li>在目标环境使用<code>yum -y localinstall</code> 一键安装</li></ol><h2 id="一-构造运行环境"><a href="#一-构造运行环境" class="headerlink" title="一 构造运行环境"></a>一 构造运行环境</h2><h3 id="1-使用docker提供目标操作系统"><a href="#1-使用docker提供目标操作系统" class="headerlink" title="1 使用docker提供目标操作系统"></a>1 使用docker提供目标操作系统</h3><p>在本地创建data目录并挂载到容器里，方便将生成的依赖取出。<br>如果想将容器当linux远程主机使用，将22端口映射出来即可： -p 5022:22 </p><ol><li>centos7<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /home/lin/centos7/data<br>docker run -d --name centos7 -v /home/lin/centos7/data:/data centos:7 /usr/sbin/init<br>docker exec -it centos7 /bin/bash<br><br></code></pre></td></tr></table></figure></li><li>centos8<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /home/lin/centos8/data<br>docker run -d --name centos8 -v /home/lin/centos8/data:/data centos:8 /usr/sbin/init<br>docker exec -it centos8 /bin/bash<br><br></code></pre></td></tr></table></figure><h3 id="2-更新yum镜像仓库"><a href="#2-更新yum镜像仓库" class="headerlink" title="2 更新yum镜像仓库"></a>2 更新yum镜像仓库</h3>参考阿里的开源镜像站：<a href="https://developer.aliyun.com/mirror/">https://developer.aliyun.com/mirror/</a><br>以centos7为例：<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup<br>curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo<br>sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo<br>yum makecache<br><br></code></pre></td></tr></table></figure><h2 id="二-使用yumdownloader"><a href="#二-使用yumdownloader" class="headerlink" title="二 使用yumdownloader"></a>二 使用yumdownloader</h2><h3 id="1-安装yumdownloader"><a href="#1-安装yumdownloader" class="headerlink" title="1. 安装yumdownloader"></a>1. 安装yumdownloader</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum install yum-utils -y<br><br></code></pre></td></tr></table></figure><h3 id="2-下载依赖包"><a href="#2-下载依赖包" class="headerlink" title="2. 下载依赖包"></a>2. 下载依赖包</h3>命令为格式为 yumdownloader –resolve 需要的包<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> 进入挂载文件夹</span><br>cd /data<br><span class="hljs-meta">#</span><span class="bash"> 根据需要先添加对应的软件存储库源，如</span><br>rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm<br><span class="hljs-meta">#</span><span class="bash"> 创建子目录</span><br>mkdir packages && cd packages <br><span class="hljs-meta">#</span><span class="bash"> 再使用yumdownloader --resolve取代yum install，这里以tar和netstat为例</span><br>yumdownloader --resolve tar netstat <br><br></code></pre></td></tr></table></figure>一般下载的安装rpm包都是带有x86_64的，某些旧一点的软件会支持32位，为了后面可以直接一键安装，这里推荐把不需要的32位版本的依赖删掉<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">rm -f *.i386*<br>rm -f *.i586*<br>rm -f *.i686*<br>rm -f *.noarch*<br><br></code></pre></td></tr></table></figure></li></ol><h3 id="3-localinstall-安装"><a href="#3-localinstall-安装" class="headerlink" title="3. localinstall 安装"></a>3. localinstall 安装</h3><p>将生成的文件夹拷贝到目标集群上，使用localinstall 进行安装</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum -y localinstall *.rpm<br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> 运维部署 </category>
</categories>
<tags>
<tag> docker </tag>
<tag> yumdownloader </tag>
</tags>
</entry>
<entry>
<title>空间索引 S2 学习指南及Java工具类实践</title>
<link href="//google_spatial_search_s2/"/>
<url>//google_spatial_search_s2/</url>
<content type="html"><![CDATA[<p>geohash对于大区域查询表现极不良好，经调研测试，改用google的s2。因为涉及的资料、工具较多，特此记录，以备后用。</p><span id="more"></span><h2 id="一-学习指南"><a href="#一-学习指南" class="headerlink" title="一 学习指南"></a>一 学习指南</h2><h3 id="0-介绍说明"><a href="#0-介绍说明" class="headerlink" title="0 介绍说明"></a>0 介绍说明</h3><p>班门不弄斧，这里推荐 halfrost 大神的空间搜索系列文章，推荐先浏览一遍。<br>这一篇是对S2的概念介绍：<a href="https://github.com/halfrost/Halfrost-Field/blob/master/contents/Go/go_spatial_search.md">高效的多维空间点索引算法 — Geohash 和 Google S2</a><br>这一篇是对S2里面的各个组件的介绍：<a href="https://github.com/halfrost/Halfrost-Field/blob/master/contents/Go/go_s2_regionCoverer.md">Google S2 是如何解决空间覆盖最优解问题的?</a></p><h3 id="1-s2-对比-geohash-的优点"><a href="#1-s2-对比-geohash-的优点" class="headerlink" title="1 s2 对比 geohash 的优点"></a>1 s2 对比 geohash 的优点</h3><ol><li>s2有30级，geohash只有12级。s2的层级变化较平缓，方便选择。</li><li>s2功能强大，解决了向量计算，面积计算，多边形覆盖，距离计算等问题，减少了开发工作量。</li><li><em><strong>s2解决了多边形覆盖问题</strong></em>。个人认为这是其与geohash功能上最本质的不同。给定不规则范围，s2可以计算出一个多边形近似覆盖这个范围。其覆盖用的格子数量根据精确度可控。geohash在这方面十分不友好，划定一个大一点的区域，其格子数可能达到数千，若减少格子数则丢失精度，查询区域过大。<br>如下，在min level和max level不变的情况下，只需设置可接受的max cells数值，即可控制覆盖精度。而且其cell的region大小自动适配。geohash要在如此大范围实现高精度覆盖则会产生极为庞大的网格数。<br>另外需要注意的是,在minLevel,maxLevel,maxCells这3个参数中,不一定能完全满足.一般而言是优先满足maxLevel即最精细的网格大小,再尽可能控制cell数量在maxCells里面.而minLevel由于会合并网格,所以很难满足(在查询大区域的时候可能会出现一个大网格和很多个小网格,导致木桶效应.这个时候可能将大网格划分为指定等级的小网格,即最终效果为,严格遵循minLevel和maxLevel,为此牺牲maxCells,后面有代码)<br><img src="https://img-blog.csdnimg.cn/20200429231333130.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="max cells 为10"><br><img src="https://img-blog.csdnimg.cn/20200429231612572.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="max cells 为45"><h3 id="2-精度表"><a href="#2-精度表" class="headerlink" title="2 精度表"></a>2 精度表</h3></li></ol><table><thead><tr><th align="center">level</th><th align="center">min area</th><th align="center">max area</th><th align="center">average area</th><th align="center">units</th><th align="center">Random cell 1 (UK) min edge length</th><th align="center">Random cell 1 (UK) max edge length</th><th align="center">Random cell 2 (US) min edge length</th><th align="center">Random cell 2 (US) max edge length</th><th align="center">Number of cells</th></tr></thead><tbody><tr><td align="center">00</td><td align="center">85011012.19</td><td align="center">85011012.19</td><td align="center">85011012.19</td><td align="center">km2</td><td align="center">7842 km</td><td align="center">7842 km</td><td align="center">7842 km</td><td align="center">7842 km</td><td align="center">6</td></tr><tr><td align="center">01</td><td align="center">21252753.05</td><td align="center">21252753.05</td><td align="center">21252753.05</td><td align="center">km2</td><td align="center">3921 km</td><td align="center">5004 km</td><td align="center">3921 km</td><td align="center">5004 km</td><td align="center">24</td></tr><tr><td align="center">02</td><td align="center">4919708.23</td><td align="center">6026521.16</td><td align="center">5313188.26</td><td align="center">km2</td><td align="center">1825 km</td><td align="center">2489 km</td><td align="center">1825 km</td><td align="center">2489 km</td><td align="center">96</td></tr><tr><td align="center">03</td><td align="center">1055377.48</td><td align="center">1646455.50</td><td align="center">1328297.07</td><td align="center">km2</td><td align="center">840 km</td><td align="center">1167 km</td><td align="center">1130 km</td><td align="center">1310 km</td><td align="center">384</td></tr><tr><td align="center">04</td><td align="center">231564.06</td><td align="center">413918.15</td><td align="center">332074.27</td><td align="center">km2</td><td align="center">432 km</td><td align="center">609 km</td><td align="center">579 km</td><td align="center">636 km</td><td align="center">1536</td></tr><tr><td align="center">05</td><td align="center">53798.67</td><td align="center">104297.91</td><td align="center">83018.57</td><td align="center">km2</td><td align="center">210 km</td><td align="center">298 km</td><td align="center">287 km</td><td align="center">315 km</td><td align="center">6K</td></tr><tr><td align="center">06</td><td align="center">12948.81</td><td align="center">26113.30</td><td align="center">20754.64</td><td align="center">km2</td><td align="center">108 km</td><td align="center">151 km</td><td align="center">143 km</td><td align="center">156 km</td><td align="center">24K</td></tr><tr><td align="center">07</td><td align="center">3175.44</td><td align="center">6529.09</td><td align="center">5188.66</td><td align="center">km2</td><td align="center">54 km</td><td align="center">76 km</td><td align="center">72 km</td><td align="center">78 km</td><td align="center">98K</td></tr><tr><td align="center">08</td><td align="center">786.20</td><td align="center">1632.45</td><td align="center">1297.17</td><td align="center">km2</td><td align="center">27 km</td><td align="center">38 km</td><td align="center">36 km</td><td align="center">39 km</td><td align="center">393K</td></tr><tr><td align="center">09</td><td align="center">195.59</td><td align="center">408.12</td><td align="center">324.29</td><td align="center">km2</td><td align="center">14 km</td><td align="center">19 km</td><td align="center">18 km</td><td align="center">20 km</td><td align="center">1573K</td></tr><tr><td align="center">10</td><td align="center">48.78</td><td align="center">102.03</td><td align="center">81.07</td><td align="center">km2</td><td align="center">7 km</td><td align="center">9 km</td><td align="center">9 km</td><td align="center">10 km</td><td align="center">6M</td></tr><tr><td align="center">11</td><td align="center">12.18</td><td align="center">25.51</td><td align="center">20.27</td><td align="center">km2</td><td align="center">3 km</td><td align="center">5 km</td><td align="center">4 km</td><td align="center">5 km</td><td align="center">25M</td></tr><tr><td align="center">12</td><td align="center">3.04</td><td align="center">6.38</td><td align="center">5.07</td><td align="center">km2</td><td align="center">1699 m</td><td align="center">2 km</td><td align="center">2 km</td><td align="center">2 km</td><td align="center">100M</td></tr><tr><td align="center">13</td><td align="center">0.76</td><td align="center">1.59</td><td align="center">1.27</td><td align="center">km2</td><td align="center">850 m</td><td align="center">1185 m</td><td align="center">1123 m</td><td align="center">1225 m</td><td align="center">402M</td></tr><tr><td align="center">14</td><td align="center">0.19</td><td align="center">0.40</td><td align="center">0.32</td><td align="center">km2</td><td align="center">425 m</td><td align="center">593 m</td><td align="center">562 m</td><td align="center">613 m</td><td align="center">1610M</td></tr><tr><td align="center">15</td><td align="center">47520.30</td><td align="center">99638.93</td><td align="center">79172.67</td><td align="center">m2</td><td align="center">212 m</td><td align="center">296 m</td><td align="center">281 m</td><td align="center">306 m</td><td align="center">6B</td></tr><tr><td align="center">16</td><td align="center">11880.08</td><td align="center">24909.73</td><td align="center">19793.17</td><td align="center">m2</td><td align="center">106 m</td><td align="center">148 m</td><td align="center">140 m</td><td align="center">153 m</td><td align="center">25B</td></tr><tr><td align="center">17</td><td align="center">2970.02</td><td align="center">6227.43</td><td align="center">4948.29</td><td align="center">m2</td><td align="center">53 m</td><td align="center">74 m</td><td align="center">70 m</td><td align="center">77 m</td><td align="center">103B</td></tr><tr><td align="center">18</td><td align="center">742.50</td><td align="center">1556.86</td><td align="center">1237.07</td><td align="center">m2</td><td align="center">27 m</td><td align="center">37 m</td><td align="center">35 m</td><td align="center">38 m</td><td align="center">412B</td></tr><tr><td align="center">19</td><td align="center">185.63</td><td align="center">389.21</td><td align="center">309.27</td><td align="center">m2</td><td align="center">13 m</td><td align="center">19 m</td><td align="center">18 m</td><td align="center">19 m</td><td align="center">1649B</td></tr><tr><td align="center">20</td><td align="center">46.41</td><td align="center">97.30</td><td align="center">77.32</td><td align="center">m2</td><td align="center">7 m</td><td align="center">9 m</td><td align="center">9 m</td><td align="center">10 m</td><td align="center">7T</td></tr><tr><td align="center">21</td><td align="center">11.60</td><td align="center">24.33</td><td align="center">19.33</td><td align="center">m2</td><td align="center">3 m</td><td align="center">5 m</td><td align="center">4 m</td><td align="center">5 m</td><td align="center">26T</td></tr><tr><td align="center">22</td><td align="center">2.90</td><td align="center">6.08</td><td align="center">4.83</td><td align="center">m2</td><td align="center">166 cm</td><td align="center">2 m</td><td align="center">2 m</td><td align="center">2 m</td><td align="center">105T</td></tr><tr><td align="center">23</td><td align="center">0.73</td><td align="center">1.52</td><td align="center">1.21</td><td align="center">m2</td><td align="center">83 cm</td><td align="center">116 cm</td><td align="center">110 cm</td><td align="center">120 cm</td><td align="center">422T</td></tr><tr><td align="center">24</td><td align="center">0.18</td><td align="center">0.38</td><td align="center">0.30</td><td align="center">m2</td><td align="center">41 cm</td><td align="center">58 cm</td><td align="center">55 cm</td><td align="center">60 cm</td><td align="center">1689T</td></tr><tr><td align="center">25</td><td align="center">453.19</td><td align="center">950.23</td><td align="center">755.05</td><td align="center">cm2</td><td align="center">21 cm</td><td align="center">29 cm</td><td align="center">27 cm</td><td align="center">30 cm</td><td align="center">7e15</td></tr><tr><td align="center">26</td><td align="center">113.30</td><td align="center">237.56</td><td align="center">188.76</td><td align="center">cm2</td><td align="center">10 cm</td><td align="center">14 cm</td><td align="center">14 cm</td><td align="center">15 cm</td><td align="center">27e15</td></tr><tr><td align="center">27</td><td align="center">28.32</td><td align="center">59.39</td><td align="center">47.19</td><td align="center">cm2</td><td align="center">5 cm</td><td align="center">7 cm</td><td align="center">7 cm</td><td align="center">7 cm</td><td align="center">108e15</td></tr><tr><td align="center">28</td><td align="center">7.08</td><td align="center">14.85</td><td align="center">11.80</td><td align="center">cm2</td><td align="center">2 cm</td><td align="center">4 cm</td><td align="center">3 cm</td><td align="center">4 cm</td><td align="center">432e15</td></tr><tr><td align="center">29</td><td align="center">1.77</td><td align="center">3.71</td><td align="center">2.95</td><td align="center">cm2</td><td align="center">12 mm</td><td align="center">18 mm</td><td align="center">17 mm</td><td align="center">18 mm</td><td align="center">1729e15</td></tr><tr><td align="center">30</td><td align="center">0.44</td><td align="center">0.93</td><td align="center">0.74</td><td align="center">cm2</td><td align="center">6 mm</td><td align="center">9 mm</td><td align="center">8 mm</td><td align="center">9 mm</td><td align="center">7e18</td></tr></tbody></table><h3 id="3-相关资料"><a href="#3-相关资料" class="headerlink" title="3 相关资料"></a>3 相关资料</h3><ol><li>halfrost 的 git 仓库，包含空间搜索系列文章：<a href="https://github.com/halfrost/Halfrost-Field">https://github.com/halfrost/Halfrost-Field</a></li><li>s2 官网：<a href="https://s2geometry.io/">https://s2geometry.io</a></li><li>s2 地图/可视化工具（功能强大，强烈推荐）： <a href="http://s2.sidewalklabs.com/regioncoverer/">http://s2.sidewalklabs.com/regioncoverer/</a></li><li>经纬度画圆/画矩形 地图/可视化工具 ：<a href="https://www.mapdevelopers.com/draw-circle-tool.php">https://www.mapdevelopers.com/draw-circle-tool.php</a></li><li>经纬度画多边形 地图/可视化工具 ：<a href="http://apps.headwallphotonics.com/">http://apps.headwallphotonics.com</a></li><li>csdn参考文章：Google S2 常用操作 ：<a href="https://blog.csdn.net/deng0515001/article/details/88031153">https://blog.csdn.net/deng0515001/article/details/88031153</a></li></ol><h2 id="二-工具类及测试"><a href="#二-工具类及测试" class="headerlink" title="二 工具类及测试"></a>二 工具类及测试</h2><h3 id="工具类"><a href="#工具类" class="headerlink" title="工具类"></a>工具类</h3><h4 id="说明"><a href="#说明" class="headerlink" title="说明"></a>说明</h4><p>以下是个人使用的Java工具类，持有对象S2RegionCoverer（用于获取给定区域的cellId），用于操作3种常见区域类型（圆，矩形，多边形）。支持多种传参（ch.hsr.geohash的WGS84Point传递经纬度，或者Tuple工具类传递经纬度）<br>主要包含3类方法：</p><ol><li>getS2RegionByXXX<br>获取给定经纬度坐标对应的S2Region,该region可用于获取cellId,或用于判断包含关系</li><li>getCellIdList<br>获取给定region的cellId,并通过childrenCellId方法控制其严格遵守minLevel</li><li>contains<br>对于指定S2Region,判断经纬度或CellToken是否在其范围内</li></ol><p>注意事项：</p><ol><li>该S2RegionCoverer不确定是否线程安全，待测试，不建议动态修改其配置参数</li><li>原生的矩形Rect在某些参数下表现不正常，待确认，这里将其转为多边形对待。<h4 id="代码"><a href="#代码" class="headerlink" title="代码"></a>代码</h4><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">enum</span> <span class="hljs-title">S2Util</span> </span>{<br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 实例</span><br><span class="hljs-comment"> */</span><br> INSTANCE;<br><br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">int</span> minLevel = <span class="hljs-number">11</span>;<br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">int</span> maxLevel = <span class="hljs-number">16</span>;<br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">int</span> maxCells = <span class="hljs-number">100</span>;<br><br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">final</span> S2RegionCoverer COVERER = <span class="hljs-keyword">new</span> S2RegionCoverer();<br><br> <span class="hljs-keyword">static</span> {<br> COVERER.setMinLevel(minLevel);<br> COVERER.setMaxLevel(maxLevel);<br> COVERER.setMaxCells(maxCells);<br> }<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 将单个cellId转换为多个指定level的cellId</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> s2CellId</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> desLevel</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span></span><br><span class="hljs-comment"> */</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> List<S2CellId> <span class="hljs-title">childrenCellId</span><span class="hljs-params">(S2CellId s2CellId, Integer desLevel)</span> </span>{<br> <span class="hljs-keyword">return</span> childrenCellId(s2CellId, s2CellId.level(), desLevel);<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> List<S2CellId> <span class="hljs-title">childrenCellId</span><span class="hljs-params">(S2CellId s2CellId, Integer curLevel, Integer desLevel)</span> </span>{<br> <span class="hljs-keyword">if</span> (curLevel < desLevel) {<br> <span class="hljs-keyword">long</span> interval = (s2CellId.childEnd().id() - s2CellId.childBegin().id()) / <span class="hljs-number">4</span>;<br> List<S2CellId> s2CellIds = Lists.newArrayList();<br> <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < <span class="hljs-number">4</span>; i++) {<br> <span class="hljs-keyword">long</span> id = s2CellId.childBegin().id() + interval * i;<br> s2CellIds.addAll(childrenCellId(<span class="hljs-keyword">new</span> S2CellId(id), curLevel + <span class="hljs-number">1</span>, desLevel));<br> }<br> <span class="hljs-keyword">return</span> s2CellIds;<br> } <span class="hljs-keyword">else</span> {<br> <span class="hljs-keyword">return</span> Lists.newArrayList(s2CellId);<br> }<br> }<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 将cellToken转换为经纬度</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> token</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span></span><br><span class="hljs-comment"> */</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> Tuple2<Double, Double> <span class="hljs-title">toLatLon</span><span class="hljs-params">(String token)</span> </span>{<br> S2LatLng latLng = <span class="hljs-keyword">new</span> S2LatLng(S2CellId.fromToken(token).toPoint());<br> <span class="hljs-keyword">return</span> Tuple2.tuple(latLng.latDegrees(), latLng.lngDegrees());<br> }<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 将经纬度转换为cellId</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> lat</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> lon</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span></span><br><span class="hljs-comment"> */</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2CellId <span class="hljs-title">toCellId</span><span class="hljs-params">(<span class="hljs-keyword">double</span> lat, <span class="hljs-keyword">double</span> lon)</span> </span>{<br> <span class="hljs-keyword">return</span> S2CellId.fromLatLng(S2LatLng.fromDegrees(lat, lon));<br> }<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 判断region是否包含指定cellToken</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> region</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> cellToken</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span></span><br><span class="hljs-comment"> */</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">contains</span><span class="hljs-params">(S2Region region, String cellToken)</span> </span>{<br> <span class="hljs-keyword">return</span> region.contains(<span class="hljs-keyword">new</span> S2Cell(S2CellId.fromToken(cellToken)));<br> }<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 判断region是否包含指定经纬度坐标</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> region</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> lat</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> lon</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span></span><br><span class="hljs-comment"> */</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">contains</span><span class="hljs-params">(S2Region region, <span class="hljs-keyword">double</span> lat, <span class="hljs-keyword">double</span> lon)</span> </span>{<br> S2LatLng s2LatLng = S2LatLng.fromDegrees(lat, lon);<br> <span class="hljs-keyword">try</span> {<br> <span class="hljs-keyword">boolean</span> contains = region.contains(<span class="hljs-keyword">new</span> S2Cell(s2LatLng));<br> <span class="hljs-keyword">return</span> contains;<br> } <span class="hljs-keyword">catch</span> (NullPointerException e) {<br> e.printStackTrace();<br> <span class="hljs-keyword">return</span> <span class="hljs-keyword">false</span>;<br> }<br> }<br><br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 根据region获取cellId列表</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> region</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span></span><br><span class="hljs-comment"> */</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> List<S2CellId> <span class="hljs-title">getCellIdList</span><span class="hljs-params">(S2Region region)</span> </span>{<br> List<S2CellId> primeS2CellIdList = COVERER.getCovering(region).cellIds();<br> <span class="hljs-keyword">return</span> primeS2CellIdList.stream().flatMap(s2CellId -> S2Util.childrenCellId(s2CellId, S2Util.minLevel).stream()).collect(Collectors.toList());<br> }<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 根据region获取合并后的cellId列表</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> region</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span></span><br><span class="hljs-comment"> */</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> List<S2CellId> <span class="hljs-title">getCompactCellIdList</span><span class="hljs-params">(S2Region region)</span> </span>{<br> List<S2CellId> primeS2CellIdList = COVERER.getCovering(region).cellIds();<br> <span class="hljs-keyword">return</span> primeS2CellIdList;<br> }<br><br> <span class="hljs-comment">///////////// 获取圆形region ///////////////</span><br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2Region <span class="hljs-title">getS2RegionByCircle</span><span class="hljs-params">(<span class="hljs-keyword">double</span> lat, <span class="hljs-keyword">double</span> lon, <span class="hljs-keyword">double</span> radius)</span> </span>{<br> <span class="hljs-keyword">double</span> capHeight = (<span class="hljs-number">2</span> * S2.M_PI) * (radius / <span class="hljs-number">40075017</span>);<br> S2Cap cap = S2Cap.fromAxisHeight(S2LatLng.fromDegrees(lat, lon).toPoint(), capHeight * capHeight / <span class="hljs-number">2</span>);<br> S2CellUnion s2CellUnion = COVERER.getCovering(cap);<br> <span class="hljs-keyword">return</span> cap;<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2Region <span class="hljs-title">getS2RegionByCircle</span><span class="hljs-params">(WGS84Point point, <span class="hljs-keyword">double</span> radius)</span> </span>{<br> <span class="hljs-keyword">return</span> getS2RegionByCircle(point.getLatitude(), point.getLongitude(), radius);<br> }<br><br> <span class="hljs-comment">///////////// 获取矩形region ///////////////</span><br><br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2Region <span class="hljs-title">geS2RegionByRect</span><span class="hljs-params">(WGS84Point point1, WGS84Point point2)</span> </span>{<br> <span class="hljs-keyword">return</span> getS2RegionByRect(point1.getLatitude(), point1.getLongitude(), point2.getLatitude(), point2.getLongitude());<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2Region <span class="hljs-title">getS2RegionByRect</span><span class="hljs-params">(Tuple2<Double, Double> point1, Tuple2<Double, Double> point2)</span> </span>{<br> <span class="hljs-keyword">return</span> getS2RegionByRect(point1.getVal1(), point1.getVal2(), point2.getVal1(), point2.getVal2());<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2Region <span class="hljs-title">getS2RegionByRect</span><span class="hljs-params">(<span class="hljs-keyword">double</span> lat1, <span class="hljs-keyword">double</span> lon1, <span class="hljs-keyword">double</span> lat2, <span class="hljs-keyword">double</span> lon2)</span> </span>{<br> List<Tuple2<Double, Double>> latLonTuple2List = Lists.newArrayList(Tuple2.tuple(lat1, lon1), Tuple2.tuple(lat1, lon2), Tuple2.tuple(lat2, lon2), Tuple2.tuple(lat2, lon1));<br> <span class="hljs-keyword">return</span> getS2RegionByPolygon(latLonTuple2List);<br> }<br><br> <span class="hljs-comment">///////////// 获取多边形region ///////////////</span><br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2Region <span class="hljs-title">getS2RegionByPolygon</span><span class="hljs-params">(WGS84Point[] pointArray)</span> </span>{<br> List<Tuple2<Double, Double>> latLonTuple2List = Lists.newArrayListWithExpectedSize(pointArray.length);<br> <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < pointArray.length; ++i) {<br> latLonTuple2List.add(Tuple2.tuple(pointArray[i].getLatitude(), pointArray[i].getLongitude()));<br> }<br> <span class="hljs-keyword">return</span> getS2RegionByPolygon(latLonTuple2List);<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2Region <span class="hljs-title">getS2RegionByPolygon</span><span class="hljs-params">(Tuple2<Double, Double>[] tuple2Array)</span> </span>{<br> <span class="hljs-keyword">return</span> getS2RegionByPolygon(Lists.newArrayList(tuple2Array));<br> }<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 注意需要以逆时针方向添加坐标点</span><br><span class="hljs-comment"> */</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> S2Region <span class="hljs-title">getS2RegionByPolygon</span><span class="hljs-params">(List<Tuple2<Double, Double>> latLonTuple2List)</span> </span>{<br> List<S2Point> pointList = Lists.newArrayList();<br> <span class="hljs-keyword">for</span> (Tuple2<Double, Double> latlonTuple2 : latLonTuple2List) {<br> pointList.add(S2LatLng.fromDegrees(latlonTuple2.getVal1(), latlonTuple2.getVal2()).toPoint());<br><br> }<br> S2Loop s2Loop = <span class="hljs-keyword">new</span> S2Loop(pointList);<br> S2PolygonBuilder builder = <span class="hljs-keyword">new</span> S2PolygonBuilder(S2PolygonBuilder.Options.DIRECTED_XOR);<br> builder.addLoop(s2Loop);<br> <span class="hljs-keyword">return</span> builder.assemblePolygon();<br> }<br><br><br> <span class="hljs-comment">///////////// 配置coverer参数 ///////////////</span><br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">int</span> <span class="hljs-title">getMinLevel</span><span class="hljs-params">()</span> </span>{<br> <span class="hljs-keyword">return</span> minLevel;<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">setMinLevel</span><span class="hljs-params">(<span class="hljs-keyword">int</span> minLevel)</span> </span>{<br> S2Util.minLevel = minLevel;<br> COVERER.setMinLevel(minLevel);<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">int</span> <span class="hljs-title">getMaxLevel</span><span class="hljs-params">()</span> </span>{<br> <span class="hljs-keyword">return</span> maxLevel;<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">setMaxLevel</span><span class="hljs-params">(<span class="hljs-keyword">int</span> maxLevel)</span> </span>{<br> S2Util.maxLevel = maxLevel;<br> COVERER.setMaxLevel(maxLevel);<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">int</span> <span class="hljs-title">getMaxCells</span><span class="hljs-params">()</span> </span>{<br> <span class="hljs-keyword">return</span> maxCells;<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">setMaxCells</span><span class="hljs-params">(<span class="hljs-keyword">int</span> maxCells)</span> </span>{<br> S2Util.maxCells = maxCells;<br> COVERER.setMaxCells(maxCells);<br> }<br>}<br><br></code></pre></td></tr></table></figure></li></ol><h4 id="测试"><a href="#测试" class="headerlink" title="测试"></a>测试</h4><h5 id="1-不规则-多边形"><a href="#1-不规则-多边形" class="headerlink" title="1. (不规则)多边形"></a>1. (不规则)多边形</h5><ol><li>去<a href="http://apps.headwallphotonics.com/%E7%94%BB%E4%B8%AA%E5%A4%9A%E8%BE%B9%E5%BD%A2,%E8%BF%99%E9%87%8C%E6%88%91%E7%94%BB%E4%BA%86%E4%B8%AA%E6%9C%89%E6%A3%B1%E6%9C%89%E8%A7%92%E7%9A%84%E7%88%B1%E5%BF%83%E6%A0%87%E5%BF%97,%E5%A6%82%E4%B8%8B">http://apps.headwallphotonics.com/画个多边形,这里我画了个有棱有角的爱心标志,如下</a>.<br><img src="https://img-blog.csdnimg.cn/20200502173431384.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="爱心标志"></li><li>将左下角的坐标信息作为参数,调用,测试代码如下<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-meta">@Test</span><br><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">getCellIdListByPolygon</span><span class="hljs-params">()</span> </span>{<br> Map<Integer,Integer> sizeCountMap= Maps.newHashMap();<br> StringBuilder sb3=<span class="hljs-keyword">new</span> StringBuilder();<br> S2Region s2Region = S2Util.getS2RegionByPolygon(Lists.newArrayList(Tuple2.tuple(<span class="hljs-number">23.851458634747043</span>, <span class="hljs-number">113.66432546548037</span>), Tuple2.tuple(<span class="hljs-number">21.60205563594303</span>, <span class="hljs-number">114.82887624673037</span>),Tuple2.tuple(<span class="hljs-number">23.771049234941454</span>, <span class="hljs-number">116.18019460610537</span>),Tuple2.tuple(<span class="hljs-number">23.16640234327511</span>, <span class="hljs-number">114.94423269204286</span>)));<br> List<S2CellId> cellIdListByPolygon = S2Util.getCellIdList(s2Region);<br> cellIdListByPolygon.forEach(s2CellId -> {<br> System.out.println(<span class="hljs-string">"Level:"</span> + s2CellId.level() + <span class="hljs-string">",ID:"</span> + s2CellId.toToken() + <span class="hljs-string">",Min:"</span> + s2CellId.rangeMin().toToken() + <span class="hljs-string">",Max:"</span> + s2CellId.rangeMax().toToken());<br> sb3.append(<span class="hljs-string">","</span>).append(s2CellId.toToken());<br> sizeCountMap.put(s2CellId.level(),sizeCountMap.getOrDefault(s2CellId.level(),<span class="hljs-number">0</span>)+<span class="hljs-number">1</span>);<br> });<br> System.out.println(sb3.substring(<span class="hljs-number">1</span>));<br> System.out.println(<span class="hljs-string">"totalSize:"</span>+cellIdListByPolygon.size());<br> sizeCountMap.entrySet().forEach(integerIntegerEntry -> {<br> System.out.printf(<span class="hljs-string">"level:%d,size:%d\n"</span>,integerIntegerEntry.getKey(),integerIntegerEntry.getValue());<br> });<br>}<br></code></pre></td></tr></table></figure></li><li>执行结果如下<br><img src="https://img-blog.csdnimg.cn/20200502173708151.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><br>可以看到网格数远远超出了设定的100,不过网格大小严格控制在了11到16之间.</li><li>将cellToken列表(逗号分隔)复制到 <a href="http://s2.sidewalklabs.com/regioncoverer/">http://s2.sidewalklabs.com/regioncoverer/</a> ,点击那个网格(data)标志即可,如下<br><img src="https://img-blog.csdnimg.cn/20200502174203269.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>如果调用的是getCompactCellIdList,则结果如下,其cell数从1000多压缩到200多.<br><img src="https://img-blog.csdnimg.cn/20200502174644676.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><h5 id="2-圆形"><a href="#2-圆形" class="headerlink" title="2 圆形"></a>2 圆形</h5></li><li>这次我们再用 <a href="https://www.mapdevelopers.com/draw-circle-tool.php">https://www.mapdevelopers.com/draw-circle-tool.php</a> 到台湾省上面画个圈圈,如下<br><img src="https://img-blog.csdnimg.cn/20200502175137239.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="圈圈"></li><li>然后将其经纬度和坐标信息作为参数,调用,测试方法如下,使用的是getCompactCellIdList<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">getCellIdListByCircle</span><span class="hljs-params">()</span> </span>{<br> Map<Integer,Integer> sizeCountMap= Maps.newHashMap();<br> StringBuilder sb3=<span class="hljs-keyword">new</span> StringBuilder();<br> S2Region s2Region = S2Util.getS2RegionByCircle(<span class="hljs-number">23.753954</span>,<span class="hljs-number">120.749615</span>,<span class="hljs-number">193511.10</span>);<br> List<S2CellId> cellIdListByPolygon = S2Util.getCompactCellIdList(s2Region);<br> cellIdListByPolygon.forEach(s2CellId -> {<br> System.out.println(<span class="hljs-string">"Level:"</span> + s2CellId.level() + <span class="hljs-string">",ID:"</span> + s2CellId.toToken() + <span class="hljs-string">",Min:"</span> + s2CellId.rangeMin().toToken() + <span class="hljs-string">",Max:"</span> + s2CellId.rangeMax().toToken());<br> sb3.append(<span class="hljs-string">","</span>).append(s2CellId.toToken());<br> sizeCountMap.put(s2CellId.level(),sizeCountMap.getOrDefault(s2CellId.level(),<span class="hljs-number">0</span>)+<span class="hljs-number">1</span>);<br> });<br> System.out.println(sb3.substring(<span class="hljs-number">1</span>));<br> System.out.println(<span class="hljs-string">"totalSize:"</span>+cellIdListByPolygon.size());<br> sizeCountMap.entrySet().forEach(integerIntegerEntry -> {<br> System.out.printf(<span class="hljs-string">"level:%d,size:%d\n"</span>,integerIntegerEntry.getKey(),integerIntegerEntry.getValue());<br> });<br>}<br></code></pre></td></tr></table></figure></li><li>结果如下<br><img src="https://img-blog.csdnimg.cn/20200502175555629.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li></ol>]]></content>
<categories>
<category> 后端开发 </category>
</categories>
<tags>
<tag> s2 </tag>
</tags>
</entry>
<entry>
<title>HBCK2修复RIT实践笔记</title>
<link href="//hbck2_rit/"/>
<url>//hbck2_rit/</url>
<content type="html"><![CDATA[<p>本文记录了作者使用HBCK2工具对线上HBase发生RIT状态的处理，仅供参考，若有疵漏，还望指正。<br>网络上关于HBCK2的文章很少，基本都是复制粘贴自田竞云(小米)的这一篇：<a href="https://mp.weixin.qq.com/s/GVMWwB1WsKcdvZGfvX1lcA?spm=a2c4e.11153940.blogcont683107.11.49d762a815MegW">HBase指南 | HBase 2.0之修复工具HBCK2运维指南</a><br>事实上这一篇文章介绍得也已经很详细了。这里只是做一些实践上的补充说明。</p><span id="more"></span><h3 id="1-下载"><a href="#1-下载" class="headerlink" title="1. 下载"></a>1. 下载</h3><p>直接去<a href="https://hbase.apache.org/downloads.html">hbase的官网下载地址</a>里就可以找到。这里直接给最新版本的下载链接（截止至2020年4月）：<a href="https://downloads.apache.org/hbase/hbase-operator-tools-1.0.0/hbase-operator-tools-1.0.0-bin.tar.gz">https://downloads.apache.org/hbase/hbase-operator-tools-1.0.0/hbase-operator-tools-1.0.0-bin.tar.gz</a><br>但还是推荐自己去git clone编译，因为官网提供的编译版本有滞后性。通常来说，使用最新版本的hbase再搭配使用最新编译的HBCK2,可以解决绝大部分莫名其妙的问题。（fixMeta+restart暴力流）<br>新版本的HBCK2有更多更方便的功能，不过一般只能在新版本的hbase中使用。</p><h3 id="2-使用命令"><a href="#2-使用命令" class="headerlink" title="2. 使用命令"></a>2. 使用命令</h3><p>将其解压后得到 hbase-hbck2-1.0.0.jar，再cp到$HBASE_HOME/lib下，执行 <em><strong>hbase org.apache.hbase.HBCK2 <命令></strong></em> 即可，第一次使用推荐 <em><strong>hbase org.apache.hbase.HBCK2 -h</strong></em> 查看详细介绍</p><h2 id="使用方法"><a href="#使用方法" class="headerlink" title="使用方法"></a>使用方法</h2><h3 id="1-查找问题"><a href="#1-查找问题" class="headerlink" title="1. 查找问题"></a>1. 查找问题</h3><p>参考HBCK2运维指南的思路：</p><pre><code class=" mermaid">graph LRA(canary tool)--> B(Procedures & Locks 页面状态)B --> C(RIT队列)C --> D(Master日志)</code></pre><h3 id="2-实践例子"><a href="#2-实践例子" class="headerlink" title="2. 实践例子"></a>2. 实践例子</h3><ol><li><p>处理Procedures & Locks<br>在 Procedures & Locks 页面查找waiting状态的procedure，按顺序进行bypass。按顺序是因为有一些waiting的发生是procedure存在依赖关系，将其bypass后后面的procedure会进入success状态。如果bypass返回false就使用bypass -r，还是不行再使用bypass -or</p></li><li><p>处理RIT队列<br>参考HBCK2运维指南可以以txt格式拿到RIT队列的所有procedure的id，将其复制到任意文件（如pid.txt），再执行以下命令即可</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat pid.txt | xargs hbase org.apache.hbase.HBCK2 bypass -or <br></code></pre></td></tr></table></figure><p>然后再以txt格式拿到RIT队列的所有region的encodedName，将其复制到任意文件（如region.txt），再执行以下命令即可</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat region.txt | xargs hbase org.apache.hbase.HBCK2 assigns<br></code></pre></td></tr></table></figure></li><li><p>assign各个表中offline的region<br>检查一下各个表中是否有region的StorefileSize为0，当然也可能是本身没有存储多少数据，要注意辨别。这种一般对其assigns就可以了。<br><img src="https://img-blog.csdnimg.cn/20200416012024639.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><p>对于一些大表可能有上千个region的，一个个甄选未免太浪费时间，可以直接在web界面将其region区域全部拷贝下来，复制到txt，使用下述命令进行筛选</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat regions.txt |grep -v 'GB' |grep -v 'MB' > region2.txt<br></code></pre></td></tr></table></figure></li><li><p>高版本的hbase有hbck记录页，去页面查看，根据提醒操作就行，更方便<br><img src="https://img-blog.csdnimg.cn/20201214142116129.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><p>使用以下Java代码即可提取出encodedName</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-meta">@Test</span><br><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">printRegionEncodedName</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> IOException </span>{<br> List<String> lines = FileUtil.readUtf8Lines(<span class="hljs-string">"C:\\tmp\\region2.txt"</span>);<br> List<String> regionEncodedNameList= Lists.newArrayListWithExpectedSize(lines.size());<br> <span class="hljs-keyword">for</span> (String line : lines) {<br> <span class="hljs-keyword">if</span>(StringUtils.isBlank(line)){<br> <span class="hljs-keyword">continue</span>;<br> }<br> <span class="hljs-keyword">int</span> point1=line.indexOf(<span class="hljs-string">"."</span>) ;<br> <span class="hljs-keyword">int</span> point2=line.indexOf(<span class="hljs-string">"."</span>, line.indexOf(<span class="hljs-string">"."</span>)+<span class="hljs-number">1</span>);<br> String s = line.substring(point1+<span class="hljs-number">1</span>, point2);<br> System.out.println(s);<br> regionEncodedNameList.add(s);<br> }<br> System.out.println(<span class="hljs-string">"size:"</span>+regionEncodedNameList.size());<br> <span class="hljs-keyword">try</span> (PrintWriter pw = <span class="hljs-keyword">new</span> PrintWriter(<span class="hljs-keyword">new</span> FileWriter(<span class="hljs-string">"C:\\tmp\\encodedName.txt"</span>));){<br> regionEncodedNameList.forEach(regionEncodedName->pw.println(regionEncodedName+<span class="hljs-string">" "</span>));<br> }<br>}<br></code></pre></td></tr></table></figure><p>然后又是 <em><strong>cat encodedName.txt | xargs hbase org.apache.hbase.HBCK2 assigns</strong></em> 就可以了<br>如下，一次性对大量region进行操作<br><img src="https://img-blog.csdnimg.cn/20201214141729730.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p></li></ol><h3 id="3-注意事项"><a href="#3-注意事项" class="headerlink" title="3. 注意事项"></a>3. 注意事项</h3><p>对procedure执行bypass后其状态会由waiting转换为success(bypassed)，但不会立即移除出rit队列。可通过重启master解决。 </p>]]></content>
<categories>
<category> 大数据 </category>
<category> HBASE </category>
<category> bug </category>
</categories>
<tags>
<tag> hbase </tag>
<tag> debug </tag>
</tags>
</entry>
<entry>
<title>k8s常用命令笔记</title>
<link href="//kubernetes-commands/"/>
<url>//kubernetes-commands/</url>
<content type="html"><![CDATA[<p>一些自用的kubectl命令</p><span id="more"></span><h3 id="重启pod"><a href="#重启pod" class="headerlink" title="重启pod"></a>重启pod</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">NAME_SPACE=default<br>kubectl get pod podname -n=${NAME_SPACE} -o yaml | kubectl replace --force -f - <br><br></code></pre></td></tr></table></figure><h3 id="强制删除pod"><a href="#强制删除pod" class="headerlink" title="强制删除pod"></a>强制删除pod</h3><p>解决：加参数<code> --force --grace-period=0</code><br><strong>grace-period</strong>表示过渡存活期，默认30s，在删除POD之前允许POD慢慢终止其上的容器进程，从而优雅退出<br>0表示立即终止POD</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl delete pod <your-pod-name> -n=<name-space> --force --grace-period=0<br><br></code></pre></td></tr></table></figure><h3 id="根据状态过滤批量操作"><a href="#根据状态过滤批量操作" class="headerlink" title="根据状态过滤批量操作"></a>根据状态过滤批量操作</h3><p>过滤条件： ImagePullBackOff|CrashLoopBackOff|Evicted|Terminating</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs shell">NAME_SPACE=default<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 删除所有ImagePullBackOff或CrashLoopBackOff的pod</span></span><br>kubectl get pods -n=${NAME_SPACE} | grep -E 'ImagePullBackOff|CrashLoopBackOff' | awk '{print $1}' | xargs kubectl delete pod -n=${NAME_SPACE}<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash"><span class="hljs-comment"># 重启所有evicted或Terminating的pod</span></span><br>kubectl get pods -n=${NAME_SPACE} | grep -E 'Evicted|Terminating' | awk '{print $1}' | xargs kubectl get pod -n=${NAME_SPACE} -o yaml | kubectl replace --force -f - <br><br></code></pre></td></tr></table></figure><h3 id="pod-forward"><a href="#pod-forward" class="headerlink" title="pod-forward"></a>pod-forward</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell"><br>kubectl port-forward --address 0.0.0.0 pod/pod名称 暴露端口:内部端口<br>kubectl port-forward --address 0.0.0.0 service/service名称 暴露端口:内部端口<br><br></code></pre></td></tr></table></figure><h3 id="kubectl-expose"><a href="#kubectl-expose" class="headerlink" title="kubectl expose"></a>kubectl expose</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl -n=default delete svc log-np<br><br>kubectl -n=default expose pod log-0 --name=log-np --type=NodePort --overrides \<br>'{ "apiVersion": "v1","spec":{"ports": [{"port":9200,"protocol":"TCP","targetPort":9200,"nodePort":30792}]}}'<br><br>kubectl -n=default expose pod manager-0 --name=manager-np --type=NodePort --overrides \<br>'{ "apiVersion": "v1","spec":{"ports": [{"protocol":"TCP","port":5000,"targetPort":5000,"nodePort":30750}]}}'<br><br><br></code></pre></td></tr></table></figure><h3 id="xargs查看pod日志"><a href="#xargs查看pod日志" class="headerlink" title="xargs查看pod日志"></a>xargs查看pod日志</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">NAME_SPACE=bigdata<br>POD_NAME="kafka-clean"<br><br>kubectl -n=${NAME_SPACE} get pods | grep $POD_NAME | awk '{print $1}' | xargs kubectl -n=${NAME_SPACE} logs<br><br></code></pre></td></tr></table></figure><h3 id="生成kubernetes集群最高权限admin用户的token"><a href="#生成kubernetes集群最高权限admin用户的token" class="headerlink" title="生成kubernetes集群最高权限admin用户的token"></a>生成kubernetes集群最高权限admin用户的token</h3><p>参考：<a href="https://jimmysong.io/kubernetes-handbook/guide/auth-with-kubeconfig-or-token.html">https://jimmysong.io/kubernetes-handbook/guide/auth-with-kubeconfig-or-token.html</a></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><code class="hljs bash">cat <<<span class="hljs-string">EOF >./my-admin-role.yaml</span><br><span class="hljs-string">kind: ClusterRoleBinding</span><br><span class="hljs-string">apiVersion: rbac.authorization.k8s.io/v1beta1</span><br><span class="hljs-string">metadata:</span><br><span class="hljs-string"> name: my-admin</span><br><span class="hljs-string"> annotations:</span><br><span class="hljs-string"> rbac.authorization.kubernetes.io/autoupdate: "true"</span><br><span class="hljs-string">roleRef:</span><br><span class="hljs-string"> kind: ClusterRole</span><br><span class="hljs-string"> name: cluster-admin</span><br><span class="hljs-string"> apiGroup: rbac.authorization.k8s.io</span><br><span class="hljs-string">subjects:</span><br><span class="hljs-string">- kind: ServiceAccount</span><br><span class="hljs-string"> name: my-admin</span><br><span class="hljs-string"> namespace: kube-system</span><br><span class="hljs-string">---</span><br><span class="hljs-string">apiVersion: v1</span><br><span class="hljs-string">kind: ServiceAccount</span><br><span class="hljs-string">metadata:</span><br><span class="hljs-string"> name: my-admin</span><br><span class="hljs-string"> namespace: kube-system</span><br><span class="hljs-string"> labels:</span><br><span class="hljs-string"> kubernetes.io/cluster-service: "true"</span><br><span class="hljs-string"> addonmanager.kubernetes.io/mode: Reconcile</span><br><span class="hljs-string"></span><br><span class="hljs-string">EOF</span><br><br><span class="hljs-comment"># 创建</span><br>kubectl create -f my-admin-role.yaml<br><span class="hljs-comment"># 获取token的值</span><br>kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep <span class="hljs-string">'my-admin-token'</span> | awk <span class="hljs-string">'{print $1}'</span>)<br><br></code></pre></td></tr></table></figure><h3 id="生成kubernetes集群最高权限admin用户的token-高版本"><a href="#生成kubernetes集群最高权限admin用户的token-高版本" class="headerlink" title="生成kubernetes集群最高权限admin用户的token(高版本)"></a>生成kubernetes集群最高权限admin用户的token(高版本)</h3><p>参考：<a href="https://blog.csdn.net/wuchenlhy/article/details/128578633">https://blog.csdn.net/wuchenlhy/article/details/128578633</a></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><code class="hljs bash">cat <<<span class="hljs-string">EOF >./my-admin-role.yaml</span><br><span class="hljs-string">kind: ClusterRoleBinding</span><br><span class="hljs-string">apiVersion: rbac.authorization.k8s.io/v1</span><br><span class="hljs-string">metadata:</span><br><span class="hljs-string"> name: my-admin</span><br><span class="hljs-string"> annotations:</span><br><span class="hljs-string"> rbac.authorization.kubernetes.io/autoupdate: "true"</span><br><span class="hljs-string">roleRef:</span><br><span class="hljs-string"> kind: ClusterRole</span><br><span class="hljs-string"> name: cluster-admin</span><br><span class="hljs-string"> apiGroup: rbac.authorization.k8s.io</span><br><span class="hljs-string">subjects:</span><br><span class="hljs-string">- kind: ServiceAccount</span><br><span class="hljs-string"> name: my-admin</span><br><span class="hljs-string"> namespace: kube-system</span><br><span class="hljs-string">---</span><br><span class="hljs-string">apiVersion: v1</span><br><span class="hljs-string">kind: ServiceAccount</span><br><span class="hljs-string">metadata:</span><br><span class="hljs-string"> name: my-admin</span><br><span class="hljs-string"> namespace: kube-system</span><br><span class="hljs-string"> labels:</span><br><span class="hljs-string"> kubernetes.io/cluster-service: "true"</span><br><span class="hljs-string"> addonmanager.kubernetes.io/mode: Reconcile</span><br><span class="hljs-string">---</span><br><span class="hljs-string">apiVersion: v1</span><br><span class="hljs-string">kind: Secret</span><br><span class="hljs-string">metadata:</span><br><span class="hljs-string"> name: my-admin-secret</span><br><span class="hljs-string"> namespace: kube-system</span><br><span class="hljs-string"> annotations:</span><br><span class="hljs-string"> kubernetes.io/service-account.name: my-admin</span><br><span class="hljs-string">type: kubernetes.io/service-account-token</span><br><span class="hljs-string"></span><br><span class="hljs-string">EOF</span><br><br>kubectl create -f my-admin-role.yaml<br>kubectl -n kube-system describe secret my-admin-secret<br><br></code></pre></td></tr></table></figure><h3 id="启用kubectl-proxy"><a href="#启用kubectl-proxy" class="headerlink" title="启用kubectl proxy"></a>启用kubectl proxy</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">nohup kubectl proxy --address='0.0.0.0' --accept-hosts='^*$' --reject-paths=' ' --port=18880 & <br></code></pre></td></tr></table></figure><h3 id="获取当前kubeconfig"><a href="#获取当前kubeconfig" class="headerlink" title="获取当前kubeconfig"></a>获取当前kubeconfig</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl config view --minify --raw<br><br></code></pre></td></tr></table></figure><h3 id="根据pv自动创建pvc"><a href="#根据pv自动创建pvc" class="headerlink" title="根据pv自动创建pvc"></a>根据pv自动创建pvc</h3><p>创建create_pvc.sh脚本，使用类似<code>./create_pvc.sh pv001 default</code>调用即可<br>也可直接修改参数，执行以下语句</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat << 'EEOFF' >./create_pvc.sh<br><span class="hljs-meta"></span><br><span class="hljs-meta">#</span><span class="bash">!/bin/bash</span><br><br>pv_name=$1<br>namespace=$2<br><br>storage=$(kubectl get pv ${pv_name} -o yaml | grep storage: | awk '{print $2}')<br>storageClassName=$(kubectl get pv ${pv_name} -o yaml | grep storageClassName: | awk '{print $2}')<br><br>cat <<EOF | kubectl create -f -<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br> namespace: ${namespace}<br> name: ${pv_name}<br>spec:<br> accessModes:<br> - ReadWriteMany<br> resources: <br> requests:<br> storage: ${storage}<br> storageClassName: ${storageClassName}<br> volumeName: ${pv_name}<br>EOF<br><br>EEOFF<br>chmod a+x create_pvc.sh<br><br>./create_pvc.sh pv001 default<br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category> Kubernetes </category>
</categories>
<tags>
<tag> kubernetes </tag>
<tag> 笔记 </tag>
</tags>
</entry>
<entry>
<title>深入解析Spring使用枚举接收参数和返回值机制并提供自定义最佳实践</title>
<link href="//spring_enum/"/>
<url>//spring_enum/</url>
<content type="html"><![CDATA[<p>Spring对应枚举传参/返回值默认是用字面量实现的（实际情况更复杂），而《阿里巴巴Java开发手册》规定接口返回值不可以使用枚举类型（包括含枚举类型的POJO对象），为此，本文探究了Spring内部对枚举参数的传递和处理机制，并提供了一套自定义方案。</p><span id="more"></span><h2 id="一-目标与思路"><a href="#一-目标与思路" class="headerlink" title="一 目标与思路"></a>一 目标与思路</h2><h3 id="0-起因"><a href="#0-起因" class="headerlink" title="0 起因"></a>0 起因</h3><p>《阿里巴巴Java开发手册》将接口中枚举的使用分为两类，即 接口参数和接口返回值，并规定：<br><strong>接口参数可以使用枚举类型，但接口返回值不可以使用枚举类型（包括含枚举类型的POJO对象）</strong>。</p><p>知乎有相关讨论和作者亲答，详情可见：<a href="https://www.zhihu.com/question/52760637">Java枚举什么不好，《阿里巴巴JAVA开发手册》对于枚举规定的考量是什么？</a></p><p>现摘录一部分作者回答如下：</p><blockquote><blockquote><p><strong>由于升级原因，导致双方的枚举类不尽相同，在接口解析，类反序列化时出现异常</strong>。</p></blockquote><blockquote><p>Java中出现的任何元素，在Gosling的角度都会有背后的思考和逻辑（尽管并非绝对完美，但Java的顶层抽象已经是天才级了），比如：接口、抽象类、注解、和本文提到的枚举。枚举有好处，类型安全，清晰直接，还可以使用等号来判断，也可以用在switch中。<code>它的劣势也是明显的，就是不要扩展</code>。可是为什么在返回值和参数进行了区分呢，如果不兼容，那么两个都有问题，怎么允许参数可以有枚举。当时的考虑，如果参数也不能用，那么枚举几乎无用武之地了。参数输出，毕竟是本地决定的，你本地有的，传送过去，向前兼容是不会有问题的。但如果是接口返回，就比较恶心了，因为解析回来的这个枚举值，可能本地还没有，这时就会抛出序列化异常。</p></blockquote><blockquote><p>比如：你的本地枚举类，有一个天气Enum：SUNNY, RAINY, CLOUDY，如果根据天气计算心情的方法：guess(WeatcherEnum xx)，传入这三个值都是可以的。返回值：Weather guess(参数)，那么对方运算后，返回一个SNOWY，本地枚举里没有这个值，傻眼了。</p></blockquote></blockquote><p>当然，使用 code 照样不能处理，对此，开发手册作者的回答如下</p><blockquote><p>主要是从防止这种序列化异常角度来考虑，使用code至少不会出大乱子。而catch序列化异常，有点像catch(NullPointerException e)一样代码过度，因为它是可预检异常。</p></blockquote><h3 id="1-统一称谓"><a href="#1-统一称谓" class="headerlink" title="1 统一称谓"></a>1 统一称谓</h3><p>假如有一枚举类如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">enum</span> <span class="hljs-title">ReturnCodeEnum</span> </span>{<br> OK(<span class="hljs-number">200</span>),<br> ERROR(<span class="hljs-number">500</span>)<br> ;<br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">final</span> <span class="hljs-keyword">int</span> code;<br> ReturnCodeEnum(<span class="hljs-keyword">int</span> code){<br> <span class="hljs-keyword">this</span>.code=code;<br> }<br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">int</span> <span class="hljs-title">getCode</span><span class="hljs-params">()</span> </span>{<br> <span class="hljs-keyword">return</span> code;<br> }<br>}<br></code></pre></td></tr></table></figure><blockquote><p>枚举实例有两个默认属性，<code>name</code> 和 <code>ordinal</code>，可通过 name()和ordinal()方法分别获得。其中 name 为枚举字面量（如 OK），ordinal 为枚举实例默认次序（从0开始）<br>需要注意的是，不建议使用枚举的 ordinal，因为枚举实例应该是无序的，ordinal 提供的顺序是不可靠的，所以我们应该使用自定义的枚举字段 code。</p></blockquote><p>后文为方便阐述，以 字面量（name）、默认次序（ordinal）和 code来展开阐述。如 OK 的 字面量为 OK，ordinal 为 0 ，code为 200。</p><h3 id="2-目标"><a href="#2-目标" class="headerlink" title="2 目标"></a>2 目标</h3><h4 id="目标"><a href="#目标" class="headerlink" title="目标"></a>目标</h4><ol><li>直接使用 枚举类型 <strong>接收参数</strong>和<strong>返回值</strong></li><li>系统自动将 参数中的 code 转换为 枚举类型，自动将 返回值中的枚举类型转换为 code</li></ol><h4 id="实现效果"><a href="#实现效果" class="headerlink" title="实现效果"></a>实现效果</h4><p>对于实现通用code枚举接口的枚举类型，有如下效果：</p><ol><li>使用 bean（<strong>application/x-www-form-urlencoded</strong>）接收时，支持 code 自动转换为 枚举类型，同时兼容 字面量转换为枚举类型。注意：表单接收的参数都视为 String，即是将String转为 枚举类型</li><li>使用 @RequestBody （<strong>application/json</strong>）接收时，默认只支持 code 自动转换为枚举类型。如果需要同时支持 code 和 字面量（或者只支持字面量），可以在具体的枚举类里添加@JsonCreator注解的方法，下文会给出参考实现。</li><li>可以使用 @RequestParam 和 @PathVariable 接收枚举类型参数</li><li>使用 @ResponseBody / @RestController（返回 Json）时，默认将 枚举类型转换为 code。</li><li>在接收参数/返回值都不允许使用 ordinal ，这只会导致混乱。</li></ol><h3 id="3-SpringMVC-对-枚举参数的处理"><a href="#3-SpringMVC-对-枚举参数的处理" class="headerlink" title="3 SpringMVC 对 枚举参数的处理"></a>3 SpringMVC 对 枚举参数的处理</h3><p>此处只对 restful 接口进行讨论。对于 restful 接口，Spring MVC 的返回值是使用 @ResponseBody 进行处理的。<br>而参数的接收方式则较多，对于非简单类型，如 Enum ，一般的接收方法为 Bean 接收或 @ResponseBody 接收。</p><h4 id="Spring使用Bean接收枚举参数"><a href="#Spring使用Bean接收枚举参数" class="headerlink" title="Spring使用Bean接收枚举参数"></a>Spring使用Bean接收枚举参数</h4><p>简单来说 Spring 默认使用Bean接收枚举参数时支持 <code>字面量</code>，这也是我们常见的做法。</p><blockquote><p>参考自：<a href="http://note4code.com/2018/03/12/spring%E4%B8%8E%E6%9E%9A%E4%B8%BE%E5%8F%82%E6%95%B0/">Spring与枚举参数</a></p><blockquote><p>GET 请求和 POST Form 请求中的字符串到枚举的转化是通过 org.springframework.core.convert.support.StringToEnumConverterFactory 来实现的.<br>该类实现了接口 ConverterFactory ，通过调用 Enum.valueOf(Class, String) 实现了这个功能。<br>向下追溯源码可以发现该方法实际上是从一个 Map<String, Enum> 的字典中获取了转换后的实际值，着这个 String 类型的 Key 的获取方式就是 Enum.name() 返回的结果，即<code>枚举的字面值</code>。</p></blockquote></blockquote><h4 id="Spring使用-RequestBody-接收枚举参数"><a href="#Spring使用-RequestBody-接收枚举参数" class="headerlink" title="Spring使用@RequestBody 接收枚举参数"></a>Spring使用@RequestBody 接收枚举参数</h4><p>简单来说 Spring使用@RequeseBody 接收枚举参数时支持 <code>字面量和 ordinal</code></p><p>对于@RequestBody，Spring会将其内容视为一段 Json，所做工作为使用 Jackson 完成反序列化。其实现会经过Jackson的EnumDeserializer的deserialize方法。感兴趣的可以去看看源码，这里不贴出来，讲一下思路：</p><ol><li><p>使用字面量（String）进行反序列化</p></li><li><p>判断是否是 int 类型，如果是使用 ordinal 进行反序列化，如果数字不在 ordinal 里面，则抛异常</p></li><li><p>判断是否是数组，是的话交由数组处理，否则抛异常</p><h4 id="Spring使用-ResponseBody-返回值"><a href="#Spring使用-ResponseBody-返回值" class="headerlink" title="Spring使用@ResponseBody 返回值"></a>Spring使用@ResponseBody 返回值</h4><p>如我们平常使用所见，返回的是字面量</p><h3 id="4-思路"><a href="#4-思路" class="headerlink" title="4 思路"></a>4 思路</h3><p>参照Spring对枚举参数的处理，我们可以提供覆盖/替换Spring的处理来达到我们的效果，<br>经本人测试，比较好的实现方案有（不考虑反射）：</p></li><li><p>自定义Bean接收枚举参数规则：</p><ol><li>可行方案<br>通过Spring MVC注入特定类型自定义转换器实现从code到 枚举的自动转换</li><li>做法<br>使用 WebMvcConfigurer的<code>addFormatters</code>注入自定义ConverterFactory，该工厂负责生成 通用code枚举接口的实现类对应的转换器<br>详见第二部分–代码实现。</li><li>参考资料<br><a href="https://blog.csdn.net/u014527058/article/details/62883573">Spring Boot绑定枚举类型参数</a></li></ol></li><li><p>自定义@RequestBody 和@ResponseBody处理枚举参数</p><ol><li><p>可行方案<br>使用<code>@JsonValue</code>自定义特定枚举类的Jackson序列化/反序列化方式</p><ol><li>具体做法<br>使用 <code>@JsonValue</code>注解标记 获取code值的枚举实例方法。</li><li>注意事项<br>该code值是使用jackson<code>序列化</code>/<code>反序列化</code>时枚举对应的值，会覆盖原来从字面量反序列化回枚举的默认实现。<br>如果想要保留原来从字面量反序列化回枚举类的功能，需要自定义一个<code> @JsonCreator</code> 的构造/静态工厂方法。</li><li>相关代码<br>代码如下：<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-meta">@JsonValue</span><br><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">int</span> <span class="hljs-title">getCode</span><span class="hljs-params">()</span> </span>{<br> <span class="hljs-keyword">return</span> code;<br>}<br><br><span class="hljs-meta">@JsonCreator</span><br><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> ReturnCodeEnum <span class="hljs-title">create</span><span class="hljs-params">(String name)</span></span>{<br> <span class="hljs-keyword">try</span>{<br> <span class="hljs-keyword">return</span> ReturnCodeEnum.valueOf(name);<br> }<span class="hljs-keyword">catch</span> (IllegalArgumentException e){<br> <span class="hljs-keyword">int</span> code=Integer.parseInt(name);<br> <span class="hljs-keyword">for</span> (ReturnCodeEnum value : ReturnCodeEnum.values()) {<br> <span class="hljs-keyword">if</span>(value.code==code){<br> <span class="hljs-keyword">return</span> value;<br> }<br> }<br> }<br> <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> IllegalArgumentException(<span class="hljs-string">"No element matches "</span>+name);<br>}<br></code></pre></td></tr></table></figure></li></ol></li><li><p>不可行方案</p><ol><li><p>替换@RequestBody和@ResponseBody或相关处理器 / 自定义HttpMessageConverter</p><ul><li><p>例子<br>如使用自定义的 @ResponseBody 注解及对应 HandlerMethodReturnValueHandler<br>使用自定义 HttpMessageConverter 实现对 json 返回资源的完全控制</p></li><li><p>不可行原因<br>我们平时使用 @ResponseBody 是交给 <code>RequestResponseBodyMethodProcessor </code>这个类处理，所以我们也可以弃用@ResponseBody并自己写一个注解和对负责处理该注解的 HandlerMethodReturnValueHandler。这样我们就可以完全控制返回值的处理了。这样也就相对于放弃了 @ResponseBody。<br>自己实现 HttpMessageConverter 则是在更高的层次进行处理，代价太大。</p></li><li><p>相关资料<br><a href="https://www.baeldung.com/spring-httpmessageconverter-rest">baeldung：Http Message Converters with the Spring Framework</a><br><a href="https://www.baeldung.com/spring-type-conversions">baeldung：Guide to Spring Type Conversions</a><br>自定义注解和HandlerMethodReturnValueHandler可以参考：<br><a href="https://www.ctolib.com/topics-109580.html">Spring MVC 更灵活的控制 json 返回（自定义过滤字段）</a><br><a href="https://www.jianshu.com/p/4fa3006c066f">深入Spring:自定义ResponseBody</a><br><a href="http://www.xiaojiezhu.com/wordpress/?p=17#i">spring自定义返回值解析器</a><br><a href="http://www.cnblogs.com/fangjian0423/p/springMVC-request-param-analysis.html#interface_demo">详解SpringMVC中Controller的方法中参数的工作原理[附带源码分析]</a><br><a href="https://blog.csdn.net/a67474506/article/details/46364159">SpringMVC 学习笔记(七) JSON返回:HttpMessageConverter作用</a><br><a href="https://www.scienjus.com/custom-http-message-converter/">使用自定义HttpMessageConverter对返回内容进行加密</a><br><a href="https://www.jianshu.com/p/e25ecc0c5762">自定义枚举 — Gson转换</a></p></li></ul></li><li><p>使用@JsonCreator在接口层面定义反序列化规则</p><ul><li>不可行原因</li></ul><p> <strong>@JsonCreator只适用于枚举类不适用于接口。</strong><br> @JsonCreator本质上是要在没有类实例的时候使用的，所以只能标记在 构造方法或者静态工厂方法上，接口的话不可行，传统的接口方法属实例方法，新增的 default 方法也属实例方法，另外的 static 方法又不可继承。所以这个思路只限于具体类型，不适用于接口。</p><ul><li>相关资料<br>相关Jackson资料参考：<br><a href="https://github.com/FasterXML/jackson-annotations/wiki/Jackson-Annotations">Github：Jackson注解官方文档</a><br><a href="https://cloud.tencent.com/developer/article/1147127">Jackson常用注解详解1 初级2 中级</a></li></ul></li><li><p>适用@JsonDeserialize在接口层面定义反序列化规则</p><ul><li>不可行原因<br><strong>注解自定义从 json字符串 转换为 实体类的方法也只适用于枚举类不适用于接口。</strong><br>使用@JsonDeserialize(using = 自定义反序列化类.class)，在自定义Jackson反序列化类实现deserialize(JsonParser p, DeserializationContext ctxt)方法。<br>可以获取 json字符串（即 code），但没办法通过接口使用code获取枚举对象，理由同上，接口没有可用的同时可继承的方法。</li><li>相关资料<br>自定义Jackson序列化/反序列化类参考：<a href="https://www.ibm.com/developerworks/cn/java/jackson-advanced-application/index.html">IBM:Jackson 框架的高阶应用</a></li></ul></li></ol></li></ol></li></ol><h2 id="二-代码实现"><a href="#二-代码实现" class="headerlink" title="二 代码实现"></a>二 代码实现</h2><h3 id="1-通用code枚举接口"><a href="#1-通用code枚举接口" class="headerlink" title="1 通用code枚举接口"></a>1 通用code枚举接口</h3><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-comment">/**</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@version</span> V1.0</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@author</span>: linshenkx</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@date</span>: 2019/1/12</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@description</span>: 带编号的枚举接口</span><br><span class="hljs-comment"> */</span><br><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">interface</span> <span class="hljs-title">CodedEnum</span> </span>{<br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 使用jackson序列化/反序列化时枚举对应的值</span><br><span class="hljs-comment"> * 如果想要保留原来从字面量反序列化回枚举类的功能，</span><br><span class="hljs-comment"> * 需要自定义一个 <span class="hljs-doctag">@JsonCreator</span> 的构造/静态工厂方法</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span> 自定义枚举code</span><br><span class="hljs-comment"> */</span><br> <span class="hljs-meta">@JsonValue</span><br> <span class="hljs-function"><span class="hljs-keyword">int</span> <span class="hljs-title">getCode</span><span class="hljs-params">()</span></span>;<br><br>}<br></code></pre></td></tr></table></figure><h3 id="2-转换器工厂类"><a href="#2-转换器工厂类" class="headerlink" title="2 转换器工厂类"></a>2 转换器工厂类</h3><p>代码实现</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-comment">/**</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@version</span> V1.0</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@author</span>: linshenkx</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@date</span>: 2019/1/12</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@description</span>: 带编号的枚举转换器 工厂</span><br><span class="hljs-comment"> */</span><br><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">CodedEnumConverterFactory</span> <span class="hljs-keyword">implements</span> <span class="hljs-title">ConverterFactory</span><<span class="hljs-title">String</span>, <span class="hljs-title">CodedEnum</span>> </span>{<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 目标类型与对应转换器的Map</span><br><span class="hljs-comment"> */</span><br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">final</span> Map<Class,Converter> CONVERTER_MAP=<span class="hljs-keyword">new</span> HashMap<>();<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 根据目标类型获取相应的转换器</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> targetType 目标类型</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> <T> CodedEnum的实现类</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@return</span></span><br><span class="hljs-comment"> */</span><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-keyword">public</span> <T extends CodedEnum> <span class="hljs-function">Converter<String, T> <span class="hljs-title">getConverter</span><span class="hljs-params">(Class<T> targetType)</span> </span>{<br> Converter converter=CONVERTER_MAP.get(targetType);<br> <span class="hljs-keyword">if</span>(converter==<span class="hljs-keyword">null</span>){<br> converter=<span class="hljs-keyword">new</span> IntegerStrToEnumConverter<>(targetType);<br> CONVERTER_MAP.put(targetType,converter);<br> }<br> <span class="hljs-keyword">return</span> converter;<br> }<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 将int对应的字符串转换为目标类型的转换器</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> <T> 目标类型（CodedEnum的实现类）</span><br><span class="hljs-comment"> */</span><br> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">IntegerStrToEnumConverter</span><<span class="hljs-title">T</span> <span class="hljs-keyword">extends</span> <span class="hljs-title">CodedEnum</span>> <span class="hljs-keyword">implements</span> <span class="hljs-title">Converter</span><<span class="hljs-title">String</span>,<span class="hljs-title">T</span>></span>{<br> <span class="hljs-keyword">private</span> Map<String,T> enumMap=<span class="hljs-keyword">new</span> HashMap<>();<br><br> <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-title">IntegerStrToEnumConverter</span><span class="hljs-params">(Class<T> enumType)</span></span>{<br> T[] enums=enumType.getEnumConstants();<br> <span class="hljs-keyword">for</span> (T e:enums){<br> <span class="hljs-comment">//从 code 反序列化回枚举</span><br> enumMap.put(e.getCode()+<span class="hljs-string">""</span>,e);<br> <span class="hljs-comment">//从枚举字面量反序列回枚</span><br> <span class="hljs-comment">//是Spring默认的方案</span><br> <span class="hljs-comment">//此处添加可避免下面convert方法抛出IllegalArgumentException异常后被系统捕获再次调用默认方案</span><br> enumMap.put(((Enum)e).name()+<span class="hljs-string">""</span>,e);<br> }<br> }<br><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> T <span class="hljs-title">convert</span><span class="hljs-params">(String source)</span> </span>{<br> T result=enumMap.get(source);<br> <span class="hljs-keyword">if</span>(result==<span class="hljs-keyword">null</span>){<br> <span class="hljs-comment">//抛出该异常后，会调用 spring 的默认转换方案，即使用 枚举字面量进行映射</span><br> <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> IllegalArgumentException(<span class="hljs-string">"No element matches "</span>+source);<br> }<br> <span class="hljs-keyword">return</span> result;<br> }<br> }<br><br>}<br></code></pre></td></tr></table></figure><h3 id="3-Spring-MVC-配置类"><a href="#3-Spring-MVC-配置类" class="headerlink" title="3 Spring MVC 配置类"></a>3 Spring MVC 配置类</h3><h4 id="1-相关知识"><a href="#1-相关知识" class="headerlink" title="1 相关知识"></a>1 相关知识</h4><ol><li>Spring Boot 默认提供Spring MVC 自动配置，不需要使用@EnableWebMvc注解</li><li>如果需要配置MVC（拦截器、格式化、视图等） 请使用添加@Configuration并实现WebMvcConfigurer接口.不要添加@EnableWebMvc注解。</li><li>@EnableWebMvc 只能添加到一个@Configuration配置类上，用于导入Spring Web MVC configuration</li><li>如果Spring Boot在classpath里看到有 spring webmvc 也会自动添加@EnableWebMvc</li></ol><p>简单来说就是在SpringBoot中不要使用@EnableWebMvc，使用@Configuration标记自定义@WebMvcConfigurer类就行，而且该类允许多个同时存在。</p><p>相关资料：<br><a href="https://blog.csdn.net/zxc123e/article/details/84636521">Spring注解@EnableWebMvc使用坑点解析</a><br><a href="https://blog.csdn.net/pinebud55/article/details/53420481">解析@EnableWebMvc 、WebMvcConfigurationSupport和WebMvcConfigurationAdapter</a><br><a href="https://www.jianshu.com/p/d47a09532de7">WebMvcConfigurationSupport与WebMvcConfigurer的关系</a><br><a href="https://blog.csdn.net/lqadam/article/details/80637335">@EnableWebMvc如何禁止@EnableAutoConfiguration</a></p><h4 id="2-代码实现"><a href="#2-代码实现" class="headerlink" title="2 代码实现"></a>2 代码实现</h4><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-comment">/**</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@version</span> V1.0</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@author</span>: linshenkx</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@date</span>: 2019/1/12</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@description</span>: 将转换器工厂添加到Spring</span><br><span class="hljs-comment"> */</span><br><span class="hljs-meta">@Configuration</span><br><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">CodedEnumWebAppConfigurer</span> <span class="hljs-keyword">implements</span> <span class="hljs-title">WebMvcConfigurer</span> </span>{<br><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">addFormatters</span><span class="hljs-params">(FormatterRegistry registry)</span> </span>{<br> registry.addConverterFactory(<span class="hljs-keyword">new</span> CodedEnumConverterFactory());<br> }<br><br>}<br><br></code></pre></td></tr></table></figure><h2 id="三-测试及分析"><a href="#三-测试及分析" class="headerlink" title="三 测试及分析"></a>三 测试及分析</h2><h3 id="1-枚举类"><a href="#1-枚举类" class="headerlink" title="1 枚举类"></a>1 枚举类</h3><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-comment">/**</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@version</span> V1.0</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@author</span>: linshenkx</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@date</span>: 2019/1/13</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@description</span>: 枚举类</span><br><span class="hljs-comment"> */</span><br><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">enum</span> <span class="hljs-title">ReturnCodeEnum</span> <span class="hljs-keyword">implements</span> <span class="hljs-title">CodedEnum</span> </span>{<br><br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 正常</span><br><span class="hljs-comment"> */</span><br> OK(<span class="hljs-number">200</span>),<br> <span class="hljs-comment">/**</span><br><span class="hljs-comment"> * 出错</span><br><span class="hljs-comment"> */</span><br> ERROR(<span class="hljs-number">500</span>)<br> ;<br><br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">final</span> <span class="hljs-keyword">int</span> code;<br><br> ReturnCodeEnum(<span class="hljs-keyword">int</span> code){<br> <span class="hljs-keyword">this</span>.code=code;<br> }<br><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">int</span> <span class="hljs-title">getCode</span><span class="hljs-params">()</span> </span>{<br> <span class="hljs-keyword">return</span> code;<br> }<br><br> <span class="hljs-meta">@JsonCreator</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> ReturnCodeEnum <span class="hljs-title">create</span><span class="hljs-params">(String name)</span></span>{<br> <span class="hljs-keyword">try</span>{<br> <span class="hljs-keyword">return</span> ReturnCodeEnum.valueOf(name);<br> }<span class="hljs-keyword">catch</span> (IllegalArgumentException e){<br> <span class="hljs-keyword">int</span> code=Integer.parseInt(name);<br> <span class="hljs-keyword">for</span> (ReturnCodeEnum value : ReturnCodeEnum.values()) {<br> <span class="hljs-keyword">if</span>(value.code==code){<br> <span class="hljs-keyword">return</span> value;<br> }<br> }<br> }<br> <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> IllegalArgumentException(<span class="hljs-string">"No element matches "</span>+name);<br> }<br><br><br>}<br></code></pre></td></tr></table></figure><h3 id="2-包含枚举类的POJO"><a href="#2-包含枚举类的POJO" class="headerlink" title="2 包含枚举类的POJO"></a>2 包含枚举类的POJO</h3><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-comment">/**</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@version</span> V1.0</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@author</span>: linshenkx</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@date</span>: 2019/1/12</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@description</span>: 枚举包装类</span><br><span class="hljs-comment"> */</span><br><span class="hljs-meta">@Data</span><br><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">MyResult</span></span>{<br> <span class="hljs-keyword">private</span> ReturnCodeEnum returnCode;<br> <span class="hljs-keyword">private</span> String message;<br>}<br><br></code></pre></td></tr></table></figure><h3 id="3-测试类"><a href="#3-测试类" class="headerlink" title="3 测试类"></a>3 测试类</h3><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><code class="hljs java"><span class="hljs-comment">/**</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@version</span> V1.0</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@author</span>: linshenkx</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@date</span>: 2019/1/12</span><br><span class="hljs-comment"> * <span class="hljs-doctag">@description</span>: 测试类</span><br><span class="hljs-comment"> */</span><br><span class="hljs-meta">@RestController</span><br><span class="hljs-meta">@RequestMapping("/test")</span><br><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">TestController</span> </span>{<br><br> <span class="hljs-meta">@PostMapping(value = "/enumForm")</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> MyResult <span class="hljs-title">testEnumForm</span><span class="hljs-params">(</span></span><br><span class="hljs-params"><span class="hljs-function"> <span class="hljs-meta">@RequestBody</span> MyResult myResult)</span> </span>{<br> ReturnCodeEnum status = myResult.getReturnCode();<br> System.out.println(<span class="hljs-string">"name():"</span>+status.name());<br> System.out.println(<span class="hljs-string">"ordinal():"</span>+status.ordinal());<br> System.out.println(<span class="hljs-string">"getCode():"</span>+status.getCode());<br> <span class="hljs-keyword">return</span> myResult;<br> }<br><br> <span class="hljs-meta">@PostMapping(value = "/enumJson")</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> MyResult <span class="hljs-title">testEnumJson</span><span class="hljs-params">(</span></span><br><span class="hljs-params"><span class="hljs-function"> <span class="hljs-meta">@RequestBody</span> MyResult myResult)</span> </span>{<br> ReturnCodeEnum status = myResult.getReturnCode();<br> System.out.println(<span class="hljs-string">"name():"</span>+status.name());<br> System.out.println(<span class="hljs-string">"ordinal():"</span>+status.ordinal());<br> System.out.println(<span class="hljs-string">"getCode():"</span>+status.getCode());<br> <span class="hljs-keyword">return</span> myResult;<br> }<br> <br> <span class="hljs-meta">@PostMapping(value = "/enumPath/{status}")</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> ReturnCodeEnum <span class="hljs-title">testEnumPath</span><span class="hljs-params">(</span></span><br><span class="hljs-params"><span class="hljs-function"> <span class="hljs-meta">@PathVariable</span> ReturnCodeEnum status)</span> </span>{<br> <span class="hljs-keyword">return</span> status;<br> }<br><br> <span class="hljs-meta">@PostMapping(value = "/enumParam")</span><br> <span class="hljs-function"><span class="hljs-keyword">public</span> ReturnCodeEnum <span class="hljs-title">testEnumParam</span><span class="hljs-params">(</span></span><br><span class="hljs-params"><span class="hljs-function"> <span class="hljs-meta">@RequestParam</span> ReturnCodeEnum status)</span> </span>{<br> <span class="hljs-keyword">return</span> status;<br> }<br>}<br></code></pre></td></tr></table></figure><p>另外还需注入上面的转换器工厂，这里不再重复贴出。</p><h3 id="4-测试结果"><a href="#4-测试结果" class="headerlink" title="4 测试结果"></a>4 测试结果</h3><h4 id="预测分析"><a href="#预测分析" class="headerlink" title="预测分析"></a>预测分析</h4><p>如上，因为ReturnCodeEnum 实现了 CodedEnum 接口，并注入对应转换器工厂，所以可以在 表单提交的时候适用code和字面量接收枚举参数。<br>ReturnCodeEnum还写了@JsonValue注解的方法，所以使用Json传参/返回值时使用@JsonValue对应的返回值。<br>因为我们还想实现Json传参的时候支持字面量，所以我们在@JsonCreator注解的方法里写了支持 code 和字面量，该方法会使@JsonValue 对反序列化的支持失效，所以写的时候不仅要支持字面量还要支持原本的目的—–code。<br>由于我们已经覆盖了原来的序列化/反序列化方式，所以 ordinal 的支持已经失效。<br>另外，由于我们可以将参数中的String转化为枚举，所以我们也可以直接使用 @PathVariable 和 @RequestParam（Content-Type: multipart/form-data）来传递枚举参数(相关资料：<a href="https://www.baeldung.com/spring-type-conversions">Baeldung：Guide to Spring Type Conversions</a>)，但是注意这个时候不能使用 包含枚举类型的POJO类，除非你再定义一个从简单类型到复合类型的转换器。</p><h4 id="1-bean接收（application-x-www-form-urlencoded）"><a href="#1-bean接收（application-x-www-form-urlencoded）" class="headerlink" title="1 bean接收（application/x-www-form-urlencoded）"></a>1 bean接收（application/x-www-form-urlencoded）</h4><ol><li><p>字面量<br><img src="https://img-blog.csdnimg.cn/2019011409281053.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="OK"></p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs java">name():<span class="hljs-function">OK</span><br><span class="hljs-function"><span class="hljs-title">ordinal</span><span class="hljs-params">()</span>:0</span><br><span class="hljs-function"><span class="hljs-title">getCode</span><span class="hljs-params">()</span>:200</span><br></code></pre></td></tr></table></figure></li><li><p>code<br><img src="https://img-blog.csdnimg.cn/20190114092919865.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="code"></p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs java">name():<span class="hljs-function">OK</span><br><span class="hljs-function"><span class="hljs-title">ordinal</span><span class="hljs-params">()</span>:0</span><br><span class="hljs-function"><span class="hljs-title">getCode</span><span class="hljs-params">()</span>:200</span><br></code></pre></td></tr></table></figure></li><li><p>ordinal<br><img src="https://img-blog.csdnimg.cn/20190114093114217.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="ordinal"><br>抛出异常</p><blockquote><p>Field error in object ‘myResult’ on field ‘returnCode’: rejected value [0]; codes [typeMismatch.myResult.returnCode,typeMismatch.returnCode,typeMismatch.com.dx.hbdt.system.manager.hongbao.controller.ReturnCodeEnum,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [myResult.returnCode,returnCode]; arguments []; default message [returnCode]]; default message [Failed to convert property value of type ‘java.lang.String’ to required type ‘com.dx.hbdt.system.manager.hongbao.controller.ReturnCodeEnum’ for property ‘returnCode’; nested exception is org.springframework.core.convert.ConversionFailedException: Failed to convert from type [java.lang.String] to type [com.dx.hbdt.system.manager.hongbao.controller.ReturnCodeEnum] for value ‘0’; nested exception is java.lang.IllegalArgumentException: No element matches 0]]</p></blockquote></li></ol><h4 id="2-RequestBody-接收（application-json）"><a href="#2-RequestBody-接收（application-json）" class="headerlink" title="2 @RequestBody 接收（application/json）"></a>2 @RequestBody 接收（application/json）</h4><ol><li><p>字面量<br><img src="https://img-blog.csdnimg.cn/20190114093238352.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="OK"></p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs java">name():<span class="hljs-function">OK</span><br><span class="hljs-function"><span class="hljs-title">ordinal</span><span class="hljs-params">()</span>:0</span><br><span class="hljs-function"><span class="hljs-title">getCode</span><span class="hljs-params">()</span>:200</span><br></code></pre></td></tr></table></figure></li><li><p>code<br><img src="https://img-blog.csdnimg.cn/20190114093333135.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="200"></p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs java">name():<span class="hljs-function">OK</span><br><span class="hljs-function"><span class="hljs-title">ordinal</span><span class="hljs-params">()</span>:0</span><br><span class="hljs-function"><span class="hljs-title">getCode</span><span class="hljs-params">()</span>:200</span><br></code></pre></td></tr></table></figure></li><li><p>ordinal<br><img src="https://img-blog.csdnimg.cn/2019011409350597.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="异常"><br>调用@JsonCreator的create方法的时候抛出 IllegalArgumentException 异常</p></li><li><p>其他<br>字符串是要携带引号的，如果不携带引号会解析错误，如{ “returnCode”: OK }<br>数字可不携带引号，在反序列化成枚举时仍会看成 String 对待，并获得与上面相同的效果。</p></li></ol><h4 id="3-PathVariable"><a href="#3-PathVariable" class="headerlink" title="3 @PathVariable"></a>3 @PathVariable</h4><p><img src="https://img-blog.csdnimg.cn/20190114101616935.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="路径"></p><h4 id="4-RequestParam（multipart-form-data）"><a href="#4-RequestParam（multipart-form-data）" class="headerlink" title="4 @RequestParam（multipart/form-data）"></a>4 @RequestParam（multipart/form-data）</h4><p><img src="https://img-blog.csdnimg.cn/20190114101654469.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsaW55dWE=,size_16,color_FFFFFF,t_70" alt="参数"></p>]]></content>
<categories>
<category> 后端开发 </category>
</categories>
<tags>
<tag> Spring </tag>