-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
312 lines (176 loc) · 560 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>YoungLiang</title>
<icon>https://www.gravatar.com/avatar/efe431aec0b30bd89e12506b00e35758</icon>
<subtitle>生活充满乐趣与童真</subtitle>
<link href="/blogs/atom.xml" rel="self"/>
<link href="https://yongliangzhang.github.io/blogs/"/>
<updated>2018-02-05T13:03:33.000Z</updated>
<id>https://yongliangzhang.github.io/blogs/</id>
<author>
<name>YoungLiang</name>
<email>zhangyongliang1231@163.com</email>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>Flume构建日志采集系统</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/03/Flume%E6%9E%84%E5%BB%BA%E6%97%A5%E5%BF%97%E9%87%87%E9%9B%86%E7%B3%BB%E7%BB%9F/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/03/Flume构建日志采集系统/</id>
<published>2018-02-03T11:45:00.000Z</published>
<updated>2018-02-05T13:03:33.000Z</updated>
<content type="html"><![CDATA[<h1 id="一、Flume介绍"><a href="#一、Flume介绍" class="headerlink" title="一、Flume介绍"></a>一、Flume介绍</h1><h5 id="1-Flume特点"><a href="#1-Flume特点" class="headerlink" title="1.Flume特点"></a>1.Flume特点</h5><ul><li>Flume是一个分布式的、可靠的、高可用的海量日志采集<br>、聚合和传输的系统</li><li>数据流模型：Source-Channel-Sink</li><li>事务机制保证消息传递的可靠性 </li><li>内置丰富插件，轻松与其他系统集成</li><li>Java实现，优秀的系统框架设计，模块分明，易于开发<a id="more"></a><h4 id="2-Flume原型图"><a href="#2-Flume原型图" class="headerlink" title="2.Flume原型图"></a>2.Flume原型图</h4><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-4015c20dd50bef6f.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Flume原型图.png" title=""> </div> <div class="image-caption">Flume原型图.png</div> </figure><h4 id="3-Flume基本组件"><a href="#3-Flume基本组件" class="headerlink" title="3.Flume基本组件"></a>3.Flume基本组件</h4></li><li>Event：消息的基本单位，有header和body组成</li><li>Agent：JVM进程，负责将一端外部来源产生的消息转 发到另一端外部的目的地<br><ul><li>Source：从外部来源读入event，并写入channel</li><li>Channel：event暂存组件，source写入后，event将会 一直保存,</li><li>Sink：从channel读入event，并写入目的地<h4 id="3-Flume事件流"><a href="#3-Flume事件流" class="headerlink" title="3.Flume事件流"></a>3.Flume事件流</h4><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-1aa8fbe82b1057cd.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Flume事件流.png" title=""> </div> <div class="image-caption">Flume事件流.png</div> </figure><h4 id="4-Flumes数据流"><a href="#4-Flumes数据流" class="headerlink" title="4.Flumes数据流"></a>4.Flumes数据流</h4><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-4245413ce4b0ead7.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Flume数据流.png" title=""> </div> <div class="image-caption">Flume数据流.png</div> </figure><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-f3c3e28c92699588.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Flume数据流2.png" title=""> </div> <div class="image-caption">Flume数据流2.png</div> </figure><h1 id="二、Flume搭建"><a href="#二、Flume搭建" class="headerlink" title="二、Flume搭建"></a>二、Flume搭建</h1><h4 id="1-下载二进制安装包"><a href="#1-下载二进制安装包" class="headerlink" title="1.下载二进制安装包"></a>1.下载二进制安装包</h4>下载地址：<a href="http://flume.apache.org/download.html" target="_blank" rel="noopener">http://flume.apache.org/download.html</a><h4 id="2-安装Flume"><a href="#2-安装Flume" class="headerlink" title="2.安装Flume"></a>2.安装Flume</h4>解压缩安装包文件</li></ul></li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 apps]$ tar -zxvf apache-flume-1.8.0-bin.tar.gz </span><br><span class="line">[hadoop@hadoop01 apps]$ cd apache-flume-1.8.0-bin/</span><br><span class="line">[hadoop@hadoop01 apache-flume-1.8.0-bin]$ ll</span><br><span class="line">总用量 148</span><br><span class="line">drwxr-xr-x. 2 hadoop hadoop 62 1月 21 14:31 bin</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 81264 9月 15 20:26 CHANGELOG</span><br><span class="line">drwxr-xr-x. 2 hadoop hadoop 127 1月 21 14:31 conf</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 5681 9月 15 20:26 DEVNOTES</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 2873 9月 15 20:26 doap_Flume.rdf</span><br><span class="line">drwxr-xr-x. 10 hadoop hadoop 4096 9月 15 20:48 docs</span><br><span class="line">drwxr-xr-x. 2 hadoop hadoop 8192 1月 21 14:31 lib</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 27663 9月 15 20:26 LICENSE</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 249 9月 15 20:26 NOTICE</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 2483 9月 15 20:26 README.md</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 1588 9月 15 20:26 RELEASE-NOTES</span><br><span class="line">drwxr-xr-x. 2 hadoop hadoop 68 1月 21 14:31 tools</span><br><span class="line">[hadoop@hadoop01 apache-flume-1.8.0-bin]$</span><br></pre></td></tr></table></figure><h4 id="3-创建软连接【此步骤可省略】"><a href="#3-创建软连接【此步骤可省略】" class="headerlink" title="3.创建软连接【此步骤可省略】"></a>3.创建软连接【此步骤可省略】</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 bin]# ln -s /home/hadoop/apps/apache-flume-1.8.0-bin /usr/local/flume</span><br></pre></td></tr></table></figure><h4 id="4-配置环境变量"><a href="#4-配置环境变量" class="headerlink" title="4.配置环境变量"></a>4.配置环境变量</h4><p>编辑 /etc/profile文件，增加以下内容：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">export FLUME_HOME=/usr/local/flume</span><br><span class="line">export PATH=$PATH:${JAVA_HOME}/bin:${ZOOKEEPER_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${HIVE_HOME}/bin:${FLUME_HOME}/bin</span><br></pre></td></tr></table></figure><h4 id="4-启动flume"><a href="#4-启动flume" class="headerlink" title="4.启动flume"></a>4.启动flume</h4><p>使用example.conf 配置文件启动一个实例<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">a1.sources = r1</span><br><span class="line">a1.channels = c1</span><br><span class="line">a1.sinks = k1</span><br><span class="line">a1.sources.r1.type = netcat</span><br><span class="line">a1.sources.r1.bind = localhost</span><br><span class="line">a1.sources.r1.port = 44444</span><br><span class="line">a1.sources.r1.channels = c1</span><br><span class="line">a1.channels.c1.type = memory</span><br><span class="line">a1.channels.c1.capacity = 1000</span><br><span class="line">a1.channels.c1.transactionCapacity = 100</span><br><span class="line">a1.sinks.k1.type = logger</span><br><span class="line">a1.sinks.k1.channel = c1</span><br></pre></td></tr></table></figure></p><p>启动命令如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 conf]# pwd</span><br><span class="line">/home/hadoop/apps/apache-flume-1.8.0-bin/conf</span><br><span class="line">[root@hadoop01 conf]# flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console</span><br></pre></td></tr></table></figure><p>启动成功后如下图所示：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">........略</span><br><span class="line">18/01/27 18:17:25 INFO node.AbstractConfigurationProvider: Channel c1 connected to [r1, k1]</span><br><span class="line">18/01/27 18:17:25 INFO node.Application: Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@20470f counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }</span><br><span class="line">18/01/27 18:17:25 INFO node.Application: Starting Channel c1</span><br><span class="line">18/01/27 18:17:25 INFO node.Application: Waiting for channel: c1 to start. Sleeping for 500 ms</span><br><span class="line">18/01/27 18:17:25 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.</span><br><span class="line">18/01/27 18:17:25 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started</span><br><span class="line">18/01/27 18:17:26 INFO node.Application: Starting Sink k1</span><br><span class="line">18/01/27 18:17:26 INFO node.Application: Starting Source r1</span><br><span class="line">18/01/27 18:17:26 INFO source.NetcatSource: Source starting</span><br><span class="line">18/01/27 18:17:26 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]</span><br></pre></td></tr></table></figure><p>使用telnet发送数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 apps]# telnet localhost 44444</span><br><span class="line">Trying ::1...</span><br><span class="line">telnet: connect to address ::1: Connection refused</span><br><span class="line">Trying 127.0.0.1...</span><br><span class="line">Connected to localhost.</span><br><span class="line">Escape character is '^]'.</span><br><span class="line">Are you OK ?</span><br><span class="line">OK</span><br></pre></td></tr></table></figure><p>控制台打印如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Impl[/127.0.0.1:44444]</span><br><span class="line">18/01/27 18:21:00 INFO sink.LoggerSink: Event: { headers:{} body: 41 72 65 20 79 6F 75 20 4F 4B 20 3F 0D Are you OK ?. }</span><br></pre></td></tr></table></figure><p><strong>如无法使用telnet，请先安装telnet工具</strong></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 apps]# yum -y install telnet</span><br></pre></td></tr></table></figure><h1 id="三、Flume实践"><a href="#三、Flume实践" class="headerlink" title="三、Flume实践"></a>三、Flume实践</h1><h4 id="1-Source组件清单"><a href="#1-Source组件清单" class="headerlink" title="1.Source组件清单"></a>1.Source组件清单</h4><ul><li>Source：对接各种外部数据源，将收集到的事件发送到Channel中，一个source可以向多个channel发送event，Flume内置非常丰富的Source，同时用户可以自定义Source</li></ul><table><thead><tr><th style="text-align:center">Source类型</th><th style="text-align:center">Type</th><th style="text-align:center">用途</th></tr></thead><tbody><tr><td style="text-align:center">Avro Source</td><td style="text-align:center">avro</td><td style="text-align:center">启动一个Avro Server，可与上一级Agent连接</td></tr><tr><td style="text-align:center">HTTP Source</td><td style="text-align:center">http</td><td style="text-align:center">启动一个HttpServer</td></tr><tr><td style="text-align:center">Exec Source</td><td style="text-align:center">exec</td><td style="text-align:center">执行unix command，获取标准输出，如tail -f</td></tr><tr><td style="text-align:center">Taildir Source</td><td style="text-align:center">TAILDIR</td><td style="text-align:center">监听目录或文件</td></tr><tr><td style="text-align:center">Spooling Directory Source</td><td style="text-align:center">spooldir</td><td style="text-align:center">监听目录下的新增文件</td></tr><tr><td style="text-align:center">Kafka Source</td><td style="text-align:center">org.apache.flume.sourc e.kafka.KafkaSource</td><td style="text-align:center">读取Kafka数据</td></tr><tr><td style="text-align:center">JMS Source</td><td style="text-align:center">jms</td><td style="text-align:center">从JMS源读取数据</td></tr></tbody></table><h4 id="2-avro-Source-Agent-和Exec-Source-Agent"><a href="#2-avro-Source-Agent-和Exec-Source-Agent" class="headerlink" title="2.avro Source Agent 和Exec Source Agent"></a>2.avro Source Agent 和Exec Source Agent</h4><ul><li>配置一个avroagent，avrosource.conf 配置文件如下：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">//avrosource.conf</span><br><span class="line">avroagent.sources = r1</span><br><span class="line">avroagent.channels = c1</span><br><span class="line">avroagent.sinks = k1 </span><br><span class="line">avroagent.sources.r1.type = avro</span><br><span class="line">avroagent.sources.r1.bind = 192.168.43.20</span><br><span class="line">avroagent.sources.r1.port = 8888</span><br><span class="line">avroagent.sources.r1.threads= 3</span><br><span class="line">avroagent.sources.r1.channels = c1</span><br><span class="line">avroagent.channels.c1.type = memory</span><br><span class="line">avroagent.channels.c1.capacity = 10000 </span><br><span class="line">avroagent.channels.c1.transactionCapacity = 1000</span><br><span class="line">avroagent.sinks.k1.type = logger</span><br><span class="line">avroagent.sinks.k1.channel = c1</span><br></pre></td></tr></table></figure><ul><li>启动一个avrosource的agent<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 conf]# flume-ng agent --conf conf --conf-file avrosource.conf --name avroagent -Dflume.root.logger=INFO,console</span><br></pre></td></tr></table></figure></li></ul><p>启动成功入下图所示：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">...略</span><br><span class="line">18/01/27 18:46:36 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.</span><br><span class="line">18/01/27 18:46:36 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started</span><br><span class="line">18/01/27 18:46:36 INFO node.Application: Starting Sink k1</span><br><span class="line">18/01/27 18:46:36 INFO node.Application: Starting Source r1</span><br><span class="line">18/01/27 18:46:36 INFO source.AvroSource: Starting Avro source r1: { bindAddress: 192.168.43.20, port: 8888 }...</span><br><span class="line">18/01/27 18:46:37 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.</span><br><span class="line">18/01/27 18:46:37 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r1 started</span><br><span class="line">18/01/27 18:46:37 INFO source.AvroSource: Avro source r1 started</span><br></pre></td></tr></table></figure><ul><li>配置一个execAgent，实现与sourceAgent实现串联，execsource.conf 配置文件如下：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">execagent.sources = r1 </span><br><span class="line">execagent.channels = c1</span><br><span class="line">execagent.sinks = k1</span><br><span class="line">execagent.sources.r1.type = exec </span><br><span class="line">execagent.sources.r1.command = tail -F /home/hadoop/apps/flume/execsource/exectest.log</span><br><span class="line">execagent.sources.r1.channels = c1</span><br><span class="line">execagent.channels.c1.type = memory</span><br><span class="line">execagent.channels.c1.capacity = 10000 </span><br><span class="line">execagent.channels.c1.transactionCapacity = 1000</span><br><span class="line">execagent.sinks.k1.type = avro</span><br><span class="line">execagent.sinks.k1.channel = c1</span><br><span class="line">execagent.sinks.k1.hostname = 192.168.43.20</span><br><span class="line">execagent.sinks.k1.port = 8888</span><br></pre></td></tr></table></figure><ul><li>启动一个execAgent,并实现execagent监控文件变化，sourceAgent接收变化内容</li></ul><p>启动 execAgent<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 conf]# flume-ng agent --conf conf --conf-file execsource.conf --name execagent</span><br></pre></td></tr></table></figure></p><p>启动成功如下下图所示：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">18/01/27 18:58:43 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started</span><br><span class="line">18/01/27 18:58:43 INFO sink.AbstractRpcSink: Rpc sink k1: Building RpcClient with hostname: 192.168.43.20, port: 8888</span><br><span class="line">18/01/27 18:58:43 INFO sink.AvroSink: Attempting to create Avro Rpc client.</span><br><span class="line">18/01/27 18:58:43 WARN api.NettyAvroRpcClient: Using default maxIOWorkers</span><br><span class="line">18/01/27 18:58:44 INFO sink.AbstractRpcSink: Rpc sink k1 started.</span><br></pre></td></tr></table></figure><p>在execAgent监控的文件下写入内容，观察sourceagent是否接收到变化内容</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 execsource]# echo 222 > exectest.log </span><br><span class="line">[root@hadoop01 execsource]# echo 5555 >> exectest.log </span><br><span class="line">[root@hadoop01 execsource]# cat exectest.log </span><br><span class="line">222</span><br><span class="line">5555</span><br></pre></td></tr></table></figure><p>在sourceagent控制打印台下查看监控消息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">18/01/27 18:58:50 INFO sink.LoggerSink: Event: { headers:{} body: 31 32 33 123 }</span><br><span class="line">18/01/27 18:59:55 INFO sink.LoggerSink: Event: { headers:{} body: 35 35 35 35 5555 }</span><br></pre></td></tr></table></figure><p>则说明2个串联agent传递信息成功。<br><br><strong>说明:</strong><br>avroagent 配置文件配置项起始名称需要与服务启动 -name 名称相一致。</p><h4 id="3-Source组件-Spooling-Directory-Source"><a href="#3-Source组件-Spooling-Directory-Source" class="headerlink" title="3.Source组件- Spooling Directory Source"></a>3.Source组件- Spooling Directory Source</h4><ul><li>配置一个Spooling Directory Source ,spooldirsource.conf 配置文件内容如下：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">a1.sources = r1</span><br><span class="line">a1.channels = c1</span><br><span class="line">a1.sinks = k1</span><br><span class="line">a1.sources.r1.type = spooldir</span><br><span class="line">a1.sources.r1.channels = c1</span><br><span class="line">a1.sources.r1.spoolDir = /home/hadoop/apps/flume/spoolDir</span><br><span class="line">a1.sources.r1.fileHeader = true</span><br><span class="line">a1.channels.c1.type = memory</span><br><span class="line">a1.channels.c1.capacity = 10000</span><br><span class="line">a1.channels.c1.transactionCapacity = 1000</span><br><span class="line">a1.sinks.k1.type = logger</span><br><span class="line">a1.sinks.k1.channel = c1</span><br></pre></td></tr></table></figure><p>/home/hadoop/apps/flume/spoolDir 必须已经创建且具有用户读写权限。</p><p>启动 SpoolDirsourceAgent</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 conf]$ flume-ng agent --conf conf --conf-file spooldirsource.conf --name a1 -Dflume.root.logger=INFO,console</span><br></pre></td></tr></table></figure><p>在spoolDir文件夹下创建文件并写入文件内容，观察控制台消息：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">18/01/28 17:06:54 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /home/hadoop/apps/flume/spoolDir/test to /home/hadoop/apps/flume/spoolDir/test.COMPLETED</span><br><span class="line">18/01/28 17:06:55 INFO sink.LoggerSink: Event: { headers:{file=/home/hadoop/apps/flume/spoolDir/test} body: 32 32 32 222 }</span><br></pre></td></tr></table></figure><p>此时监测到SpoolDirSourceAgent 可以监控到文件变化。<br><br>值得说明的是：<strong>Spooling Directory Source Agent 并不能监听子级文件夹的文件变化,也不支持已存在的文件更新数据变化</strong>.</p><h4 id="4-Source组件-Kafka-Source"><a href="#4-Source组件-Kafka-Source" class="headerlink" title="4.Source组件- Kafka Source"></a>4.Source组件- Kafka Source</h4><ul><li>配置一个Kafa Source , kafasource.conf 配置文件内容如下：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">kafkasourceagent.sources = r1</span><br><span class="line">kafkasourceagent.channels = c1</span><br><span class="line">kafkasourceagent.sinks = k1</span><br><span class="line">kafkasourceagent.sources.r1.type = org.apache.flume.source.kafka.KafkaSource </span><br><span class="line">kafkasourceagent.sources.r1.channels = c1 </span><br><span class="line">kafkasourceagent.sources.r1.batchSize = 100</span><br><span class="line">kafkasourceagent.sources.r1.batchDurationMillis = 1000</span><br><span class="line">kafkasourceagent.sources.r1.kafka.bootstrap.servers = 192.168.43.22:9092,192.168.43.23:9092,192.168.43.24:9092</span><br><span class="line">kafkasourceagent.sources.r1.kafka.topics = flumetopictest1</span><br><span class="line">kafkasourceagent.sources.r1.kafka.consumer.group.id = flumekafkagroupid</span><br><span class="line">kafkasourceagent.channels.c1.type = memory</span><br><span class="line">kafkasourceagent.channels.c1.capacity = 10000 </span><br><span class="line">kafkasourceagent.channels.c1.transactionCapacity = 1000</span><br><span class="line">kafkasourceagent.sinks.k1.type = logger</span><br><span class="line">kafkasourceagent.sinks.k1.channel = c1</span><br></pre></td></tr></table></figure><p>首先启动3个节点的kafka节点服务，在每个kafka节点执行，以后台方式运行</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-server-start.sh -daemon ../config/server.properties</span><br></pre></td></tr></table></figure><p>在kafka节点上创建一个配置好的Topic flumetoptest1,命令如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --create --zookeeper 192.168.43.20:2181 --replication-factor 1 --partitions 3 --topic flumetopictest1</span><br><span class="line">Created topic "flumetopictest1".</span><br></pre></td></tr></table></figure><p>创建成功后，启动一个kafka Source Agent，命令如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 conf]# flume-ng agent --conf conf --conf-file kafkasource.conf --name kafkasourceagent -Dflume.root.logger=INFO,console</span><br></pre></td></tr></table></figure><p>创建一个Kafka 生产者，进行消息发送<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">root@hadoop03 bin]# ./kafka-console-producer.sh --broker-list 192.168.43.22:9092,192.168.43.23:9092 --topic flumetopictest1</span><br></pre></td></tr></table></figure></p><p>发送消息，此时kafka 就可以接收到消息：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">18/02/03 20:36:57 INFO sink.LoggerSink: Event: { headers:{topic=flumetopictest1, partition=2, timestamp=1517661413068} body: 31 32 33 31 33 32 32 31 12313221 }</span><br><span class="line">18/02/03 20:37:09 INFO sink.LoggerSink: Event: { headers:{topic=flumetopictest1, partition=1, timestamp=1517661428930} body: 77 69 20 61 69 79 6F 75 08 08 08 wi aiyou... }</span><br></pre></td></tr></table></figure><h4 id="5-Source-组件-Taildir-source"><a href="#5-Source-组件-Taildir-source" class="headerlink" title="5.Source 组件 -Taildir source"></a>5.Source 组件 -Taildir source</h4><p> 监听一个文件夹或者文件，通过正则表达式匹配需要监听的 数据源文件，Taildir Source通过将监听的文件位置写入到文件中来实现断点续传，并且能够保证没有重复数据的读取.</p><ul><li>重要参数<br><br>type：source类型TAILDIR<br><br>positionFile：保存监听文件读取位置的文件路径<br><br>idleTimeout：关闭空闲文件延迟时间，如果有新的记录添加到已关闭的空闲文件<br><br>taildir srouce将继续打开该空闲文件，默认值120000毫秒<br><br>writePosInterval：向保存读取位置文件中写入读取文件位置的时间间隔，默认值<br>3000毫秒<br><br>batchSize：批量写入channel最大event数，默认值100<br><br>maxBackoffSleep：每次最后一次尝试没有获取到监听文件最新数据的最大延迟时 间，默认值5000毫秒<br><br>cachePatternMatching：对于监听的文件夹下通过正则表达式匹配的文件可能数量 会很多，将匹配成功的监听文件列表和读取文件列表的顺序都添加到缓存中，可以提高性能，默认值true<br><br>fileHeader ：是否添加文件的绝对路径到event的header中，默认值false<br><br>fileHeaderKey：添加到event header中文件绝对路径的键值，默认值file<br><br>filegroups：监听的文件组列表，taildirsource通过文件组监听多个目录或文件<br><br>filegroups.<filegroupname>：文件正则表达式路径或者监听指定文件路径<br><br>channels：Source对接的Channel名称<br></filegroupname></li><li>配置一个taildir Source,具体taildirsource.conf 配置文件内容如下：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">taildiragent.sources=r1</span><br><span class="line">taildiragent.channels=c1</span><br><span class="line">taildiragent.sinks=k1</span><br><span class="line">taildiragent.sources.r1.type=TAILDIR</span><br><span class="line">taildiragent.sources.r1.positionFile=/home/hadoop/apps/flume/taildir/position/taildir_position.json</span><br><span class="line">taildiragent.sources.r1.filegroups=f1 f2</span><br><span class="line">taildiragent.sources.r1.filegroups.f1=/home/hadoop/apps/flume/taildir/test1/test.log</span><br><span class="line">taildiragent.sources.r1.filegroups.f2=/home/hadoop/apps/flume/taildir/test2/.*log.*</span><br><span class="line">taildiragent.sources.r1.channels=c1</span><br><span class="line">taildiragent.channels.c1.type=memory</span><br><span class="line">taildiragent.channels.c1.transcationCapacity=1000</span><br><span class="line">taildiragent.sinks.k1.type=logger</span><br><span class="line">taildiragent.sinks.k1.channel=c1</span><br></pre></td></tr></table></figure><p>启动一个taildirSource agent ,代码如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 conf]# flume-ng agent --conf conf --conf-file taildirsource.conf --name taildiragent -Dflume.root.logger=INFO,console</span><br></pre></td></tr></table></figure><p>开始在test1和test2文件夹写入文件，观察agent消息接收。</p><h4 id="6-Channel组件"><a href="#6-Channel组件" class="headerlink" title="6.Channel组件"></a>6.Channel组件</h4><ul><li>Channel：Channel被设计为event中转暂存区，存储Source 收集并且没有被Sink消费的event ，为了平衡Source收集 和Sink读取数据的速度，可视为Flume内部的消息队列。</li><li>Channel是线程安全的并且具有事务性，支持source写失 败重复写和sink读失败重复读等操作</li><li>常用的Channel类型有：Memory Channel、File Channel、<br>Kafka Channel、JDBC Channel等<h4 id="7-Channel组件-Memory-Channel"><a href="#7-Channel组件-Memory-Channel" class="headerlink" title="7.Channel组件- Memory Channel"></a>7.Channel组件- Memory Channel</h4></li><li>Memory Channel：使用内存作为Channel，Memory Channel读写速度 快，但是存储数据量小，Flume进程挂掉、服务器停机或者重启都会 导致数据丢失。部署Flume Agent的线上服务器内存资源充足、不关 心数据丢失的场景下可以使用<br>关键参数：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">type ：channel类型memory</span><br><span class="line">capacity ：channel中存储的最大event数，默认值100</span><br><span class="line">transactionCapacity ：一次事务中写入和读取的event最大数，默认值100。</span><br><span class="line">keep-alive：在Channel中写入或读取event等待完成的超时时间，默认值3秒</span><br><span class="line">byteCapacityBufferPercentage：缓冲空间占Channel容量（byteCapacity）的百分比，为event中的头信息保留了空间，默认值20（单位百分比）</span><br><span class="line">byteCapacity ：Channel占用内存的最大容量，默认值为Flume堆内存的80%</span><br></pre></td></tr></table></figure><h4 id="8-Channel组件-File-Channel"><a href="#8-Channel组件-File-Channel" class="headerlink" title="8. Channel组件- File Channel"></a>8. Channel组件- File Channel</h4><ul><li>File Channel：将event写入到磁盘文件中，与Memory Channel相比存 储容量大，无数据丢失风险。</li><li>File Channle数据存储路径可以配置多磁盘文件路径，提高写入文件性能</li><li>Flume将Event顺序写入到File Channel文件的末尾，在配置文件中通<br>过设置maxFileSize参数设置数据文件大小上限</li><li>当一个已关闭的只读数据文件中的Event被完全读取完成，并且Sink已经提交读取完成的事务，则Flume将删除存储该数据文件</li><li>通过设置检查点和备份检查点在Agent重启之后能够快速将File Channle中的数据按顺序回放到内存中<br>关键参数如下：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">type：channel类型为file </span><br><span class="line">checkpointDir：检查点目录，默认在启动flume用户目录下创建，建 议单独配置磁盘路径 </span><br><span class="line">useDualCheckpoints：是否开启备份检查点，默认false，建议设置为true开启备份检查点，备份检查点的作用是当Agent意外出错导致写 入检查点文件异常，在重新启动File Channel时通过备份检查点将数据回放到内存中，如果不开启备份检查点，在数据回放的过程中发现检查点文件异常会对所数据进行全回放，全回放的过程相当耗时 </span><br><span class="line">backupCheckpointDir：备份检查点目录，最好不要和检查点目录在同 一块磁盘上 </span><br><span class="line">checkpointInterval：每次写检查点的时间间隔，默认值30000毫秒 </span><br><span class="line">dataDirs：数据文件磁盘存储路径，建议配置多块盘的多个路径，通过磁盘的并行写入来提高file channel性能，多个磁盘路径用逗号隔开</span><br><span class="line">transactionCapacity：一次事务中写入和读取的event最大数，默认值 10000</span><br><span class="line">maxFileSize：每个数据文件的最大大小，默认值：2146435071字节</span><br><span class="line">minimumRequiredSpace：磁盘路径最小剩余空间，如果磁盘剩余空 间小于设置值，则不再写入数据</span><br><span class="line">capacity：file channel可容纳的最大event数</span><br><span class="line">keep-alive：在Channel中写入或读取event等待完成的超时时间，默认值3秒</span><br></pre></td></tr></table></figure><p>配置一个FileChannel,filechannel.conf 的配置内容如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">a1.sources = r1</span><br><span class="line">a1.channels = c1</span><br><span class="line">a1.sinks = k1</span><br><span class="line">a1.sources.r1.type = netcat</span><br><span class="line">a1.sources.r1.bind = localhost</span><br><span class="line">a1.sources.r1.port = 44444</span><br><span class="line">a1.sources.r1.channels = c1</span><br><span class="line">a1.channels.c1.type = file</span><br><span class="line">a1.channels.c1.dataDirs = /home/hadoop/apps/flume/filechannel/data</span><br><span class="line">a1.channels.c1.checkpointDir = /home/hadoop/apps/flume/filechannel/checkpoint </span><br><span class="line">a1.channels.c1.useDualCheckpoints = true</span><br><span class="line">a1.channels.c1.backupCheckpointDir = /home/hadoop/apps/flume/filechannel/backup</span><br><span class="line">a1.sinks.k1.type = logger</span><br><span class="line">a1.sinks.k1.channel = c1</span><br></pre></td></tr></table></figure><p>启动一个FileChannel,启动命令如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 bin]# flume-ng agent --conf conf --conf-file filechannle.conf --name a1 -Dflume.root.logger=INFO,console</span><br></pre></td></tr></table></figure><p>向配置文件端口44444发送数据，观察Channel记录情况</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">telnet localhost asdfasd</span><br></pre></td></tr></table></figure><p>此时可以观察到控制台打印监控结果</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">18/02/04 21:15:44 INFO sink.LoggerSink: Event: { headers:{} body: 61 64 66 61 64 66 61 64 66 61 73 66 0D adfadfadfasf. }</span><br><span class="line">18/02/04 21:15:48 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/hadoop/apps/flume/filechannel/checkpoint/checkpoint, elements to sync = 1</span><br><span class="line">18/02/04 21:15:48 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1517749968978, queueSize: 0, queueHead: 0</span><br><span class="line">18/02/04 21:15:48 INFO file.EventQueueBackingStoreFile: Attempting to back up checkpoint.</span><br><span class="line">18/02/04 21:15:48 INFO file.Serialization: Skipping in_use.lock because it is in excludes set</span><br><span class="line">18/02/04 21:15:48 INFO file.Serialization: Deleted the following files: , checkpoint, checkpoint.meta, inflightputs, inflighttakes.</span><br><span class="line">18/02/04 21:15:48 INFO file.Log: Updated checkpoint for file: /home/hadoop/apps/flume/filechannel/data/log-2 position: 170 logWriteOrderID: 1517749968978</span><br><span class="line">18/02/04 21:15:49 INFO file.EventQueueBackingStoreFile: Checkpoint backup completed.</span><br></pre></td></tr></table></figure><h4 id="9-Channel组件-Kafka-Channel"><a href="#9-Channel组件-Kafka-Channel" class="headerlink" title="9.Channel组件- Kafka Channel"></a>9.Channel组件- Kafka Channel</h4><p>Kafka Channel：将分布式消息队列kafka作为channel相对于Memory Channel和File Channel存储容量更大、 容错能力更强，弥补了其他两种Channel的短板，如果合理利用Kafka的性能，能够达到事半功倍的效果。<br>关键参数如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">type：Kafka Channel类型org.apache.flume.channel.kafka.KafkaChannel</span><br><span class="line">kafka.bootstrap.servers：Kafka broker列表，格式为ip1:port1, ip2:port2…，建 议配置多个值提高容错能力，多个值之间用逗号隔开</span><br><span class="line">kafka.topic：topic名称，默认值“flume-channel”</span><br><span class="line">kafka.consumer.group.id：Consumer Group Id，全局唯一</span><br><span class="line">parseAsFlumeEvent：是否以Avro FlumeEvent模式写入到Kafka Channel中， 默认值true，event的header信息与event body都写入到kafka中</span><br><span class="line">pollTimeout：轮询超时时间，默认值500毫秒</span><br><span class="line">kafka.consumer.auto.offset.reset：earliest表示从最早的偏移量开始拉取，latest表示从最新的偏移量开始拉取，none表示如果没有发现该Consumer组之前拉 取的偏移量则抛异常</span><br></pre></td></tr></table></figure><p>配置一个KafakChannel， kafkachannel.conf 配置内容如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">a1.sources = r1</span><br><span class="line">a1.channels = c1</span><br><span class="line">a1.sinks = k1</span><br><span class="line">a1.sources.r1.type = netcat</span><br><span class="line">a1.sources.r1.bind = localhost</span><br><span class="line">a1.sources.r1.port = 44444</span><br><span class="line">a1.sources.r1.channels = c1</span><br><span class="line">a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel</span><br><span class="line">a1.channels.c1.kafka.bootstrap.servers = 192.168.43.22:9092,192.168.43.23:9092</span><br><span class="line">a1.channels.c1.kafka.topic = flumechannel2</span><br><span class="line">a1.channels.c1.kafka.consumer.group.id = flumecgtest1</span><br><span class="line">a1.sinks.k1.type = logger</span><br><span class="line">a1.sinks.k1.channel = c1</span><br></pre></td></tr></table></figure><p>启动kafak服务，创建一个kafka主题，命令如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-server-start.sh -daemon ../config/server.properties</span><br><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --create --zookeeper 192.168.43.20:2181 --replication-factor 1 --partitions 3 --topic flumechannel2</span><br></pre></td></tr></table></figure><p>查看创建的主题信息<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --list --zookeeper 192.168.43.20:2181</span><br><span class="line">__consumer_offsets</span><br><span class="line">flumechannel2</span><br><span class="line">topicnewtest1</span><br></pre></td></tr></table></figure></p><p>启动kafka agent,使用telnet发送数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 conf]# flume-ng agent --conf conf --conf-file kafkachannel.conf --name a1 -Dflume.root.logger=INFO,console</span><br><span class="line">[root@hadoop01 flume]# clear</span><br><span class="line">[root@hadoop01 flume]# telnet localhost 44444 </span><br><span class="line">Trying ::1...</span><br><span class="line">telnet: connect to address ::1: Connection refused</span><br><span class="line">Trying 127.0.0.1...</span><br><span class="line">Connected to localhost.</span><br><span class="line">Escape character is '^]'.</span><br><span class="line">abc</span><br><span class="line">OK</span><br></pre></td></tr></table></figure><p>监听信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">18/02/04 21:39:33 INFO sink.LoggerSink: Event: { headers:{} body: 61 62 63 0D abc. }</span><br></pre></td></tr></table></figure><h4 id="10-Sink组件"><a href="#10-Sink组件" class="headerlink" title="10.Sink组件"></a>10.Sink组件</h4><ul><li>Sink：从Channel消费event，输出到外部存储，或者输出到下一个阶段的agent</li><li>一个Sink只能从一个Channel中消费event</li><li>当Sink写出event成功后，就会向Channel提交事务。Sink 事务提交成功，处理完成的event将会被Channel删除。否 则Channel会等待Sink重新消费处理失败的event</li><li>Flume提供了丰富的Sink组件，如Avro Sink、HDFS Sink、Kafka Sink、File Roll Sink、HTTP Sink等<h4 id="11-Sink组件-Avro-Sink"><a href="#11-Sink组件-Avro-Sink" class="headerlink" title="11.Sink组件- Avro Sink"></a>11.Sink组件- Avro Sink</h4></li><li>Avro Sink常用于对接下一层的Avro Source，通过发送RPC请求将Event发送到下一层的Avro Source</li><li>为了减少Event传输占用大量的网络资源， Avro Sink提供了端到端的批量压缩数据传输</li></ul><p>关键参数说明<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">type：Sink类型为avro。</span><br><span class="line">hostname：绑定的目标Avro Souce主机名称或者IP</span><br><span class="line">port：绑定的目标Avro Souce端口号</span><br><span class="line">batch-size：批量发送Event数，默认值100</span><br><span class="line">compression-type：是否使用压缩，如果使用压缩设则值为</span><br><span class="line">“deflate”， Avro Sink设置了压缩那么Avro Source也应设置相同的 压缩格式，目前支持zlib压缩，默认值none</span><br><span class="line">compression-level：压缩级别，0表示不压缩，从1到9数字越大压缩</span><br><span class="line">效果越好，默认值6</span><br></pre></td></tr></table></figure></p><h4 id="12-Sink组件-HDFS-Sink"><a href="#12-Sink组件-HDFS-Sink" class="headerlink" title="12.Sink组件- HDFS Sink"></a>12.Sink组件- HDFS Sink</h4><ul><li>HDFS Sink将Event写入到HDFS中持久化存储</li><li>HDFS Sink提供了强大的时间戳转义功能，根据Event头信息中的</li><li>timestamp时间戳信息转义成日期格式，在HDFS中以日期目录分层存储</li></ul><p>关键参数信息说明如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">type：Sink类型为hdfs。</span><br><span class="line">hdfs.path：HDFS存储路径，支持按日期时间分区。</span><br><span class="line">hdfs.filePrefix：Event输出到HDFS的文件名前缀，默认前缀FlumeData</span><br><span class="line">hdfs.fileSuffix：Event输出到HDFS的文件名后缀</span><br><span class="line">hdfs.inUsePrefix：临时文件名前缀</span><br><span class="line">hdfs.inUseSuffix：临时文件名后缀，默认值.tmp</span><br><span class="line">hdfs.rollInterval：HDFS文件滚动生成时间间隔，默认值30秒，该值设置 为0表示文件不根据时间滚动生成</span><br></pre></td></tr></table></figure></p><p>配置一个hdfsink.conf文件，配置内容如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">a1.sources = r1</span><br><span class="line">a1.channels = c1</span><br><span class="line">a1.sinks = k1</span><br><span class="line">a1.sources.r1.type = netcat</span><br><span class="line">a1.sources.r1.bind = localhost</span><br><span class="line">a1.sources.r1.port = 44444</span><br><span class="line">a1.sources.r1.interceptors = i1</span><br><span class="line">a1.sources.r1.interceptors.i1.type = timestamp</span><br><span class="line">a1.sources.r1.interceptors.i1.preserveExisting = false</span><br><span class="line">a1.sources.r1.channels = c1</span><br><span class="line">a1.channels.c1.type = memory</span><br><span class="line">a1.channels.c1.capacity = 10000 </span><br><span class="line">a1.channels.c1.transactionCapacity = 1000</span><br><span class="line">a1.sinks.k1.type = hdfs</span><br><span class="line">a1.sinks.k1.channel = c1</span><br><span class="line">a1.sinks.k1.hdfs.path = /data/flume/%Y%m%d</span><br><span class="line">a1.sinks.k1.hdfs.filePrefix = hdfssink</span><br><span class="line">a1.sinks.k1.hdfs.fileType = DataStream</span><br><span class="line">a1.sinks.k1.hdfs.writeFormat = Text</span><br><span class="line">a1.sinks.k1.hdfs.round = true</span><br><span class="line">a1.sinks.k1.hdfs.roundValue = 1</span><br><span class="line">a1.sinks.k1.hdfs.roundUnit = minute</span><br><span class="line">a1.sinks.k1.hdfs.callTimeout = 60000</span><br></pre></td></tr></table></figure><p>启动一个hdfssink agent，命令如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 conf]# flume-ng agent --conf conf --conf-file hdfssink.conf --name a1 -Dflume.root.logger=INFO,console</span><br></pre></td></tr></table></figure><p>使用telnet 向44444发送数据，观察数据写入结果</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ telnet localhost 44444</span><br><span class="line">Trying ::1...</span><br><span class="line">telnet: connect to address ::1: Connection refused</span><br><span class="line">Trying 127.0.0.1...</span><br><span class="line">Connected to localhost.</span><br><span class="line">Escape character is '^]'.</span><br><span class="line">abc</span><br><span class="line">OK</span><br><span class="line">2323444</span><br><span class="line">OK</span><br></pre></td></tr></table></figure><p>此时控制台打印，在HDFS文件系统生成一个临时文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">8/02/04 22:41:52 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false</span><br><span class="line">18/02/04 22:41:52 INFO hdfs.BucketWriter: Creating /data/flume/20180204/hdfssink.1517755312242.tmp</span><br><span class="line">18/02/04 22:42:24 INFO hdfs.BucketWriter: Closing /data/flume/20180204/hdfssink.1517755312242.tmp</span><br><span class="line">18/02/04 22:42:24 INFO hdfs.BucketWriter: Renaming /data/flume/20180204/hdfssink.1517755312242.tmp to /data/flume/20180204/hdfssink.1517755312242</span><br><span class="line">18/02/04 22:42:24 INFO hdfs.HDFSEventSink: Writer callback called.</span><br></pre></td></tr></table></figure><p>值得注意的是：<strong>请使用hadoop用户来执行agent的创建和消息的发送，避免因权限导致HDFS文件无法写入</strong></p><h4 id="13-Sink组件-Kafka-Sink"><a href="#13-Sink组件-Kafka-Sink" class="headerlink" title="13.Sink组件- Kafka Sink"></a>13.Sink组件- Kafka Sink</h4><p>Flume通过KafkaSink将Event写入到Kafka指定的主题中<br>主要参数说明如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">type：Sink类型，值为KafkaSink类路径 org.apache.flume.sink.kafka.KafkaSink。</span><br><span class="line">kafka.bootstrap.servers：Broker列表，定义格式host:port，多个Broker之间用逗号隔开，可以配置一个也可以配置多个，用于Producer发现集群中的Broker，建议配置多个，防止当个Broker出现问题连接 失败。</span><br><span class="line">kafka.topic：Kafka中Topic主题名称，默认值flume-topic。</span><br><span class="line">flumeBatchSize：Producer端单次批量发送的消息条数，该值应该根据实际环境适当调整，增大批量发送消息的条数能够在一定程度上提高性能，但是同时也增加了延迟和Producer端数据丢失的风险。 默认值100。</span><br><span class="line">kafka.producer.acks：设置Producer端发送消息到Borker是否等待接收Broker返回成功送达信号。0表示Producer发送消息到Broker之后不需要等待Broker返回成功送达的信号，这种方式吞吐量高，但是存 在数据丢失的风险。1表示Broker接收到消息成功写入本地log文件后向Producer返回成功接收的信号，不需要等待所有的Follower全部同步完消息后再做回应，这种方式在数据丢失风险和吞吐量之间做了平衡。all（或者-1）表示Broker接收到Producer的消息成功写入本 地log并且等待所有的Follower成功写入本地log后向Producer返回成功接收的信号，这种方式能够保证消息不丢失，但是性能最差。默 认值1。</span><br><span class="line">useFlumeEventFormat：默认值false，Kafka Sink只会将Event body内 容发送到Kafka Topic中。如果设置为true，Producer发送到KafkaTopic中的Event将能够保留Producer端头信息</span><br></pre></td></tr></table></figure><p>配置一个kafkasink.conf,具体配置内容如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">a1.sources = r1</span><br><span class="line">a1.channels = c1</span><br><span class="line">a1.sinks = k1</span><br><span class="line">a1.sources.r1.type = netcat</span><br><span class="line">a1.sources.r1.bind = localhost</span><br><span class="line">a1.sources.r1.port = 44444</span><br><span class="line">a1.sources.r1.channels = c1</span><br><span class="line">a1.channels.c1.type = memory</span><br><span class="line">a1.channels.c1.capacity = 10000 </span><br><span class="line">a1.channels.c1.transactionCapacity = 1000</span><br><span class="line">a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink</span><br><span class="line">a1.sinks.k1.channel = c1</span><br><span class="line">a1.sinks.k1.kafka.topic = FlumeKafkaSinkTopic1</span><br><span class="line">a1.sinks.k1.kafka.bootstrap.servers = 192.168.43.22:9092,192.168.43.23:9092</span><br><span class="line">a1.sinks.k1.kafka.flumeBatchSize = 100</span><br><span class="line">a1.sinks.k1.kafka.producer.acks = 1</span><br></pre></td></tr></table></figure><p>启动kafka Broker节点22和Broker节点23</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-server-start.sh -daemon ../config/server.properties</span><br></pre></td></tr></table></figure><p>按配置文件创建主题信息<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --create --zookeeper 192.168.43.20:2181 --replication-factor 1 --partitions 3 --topic FlumeKafkaSinkTopic1</span><br><span class="line">Created topic "FlumeKafkaSinkTopic1".</span><br></pre></td></tr></table></figure></p><p>启动一个kafkasink agent，启动命令如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 conf]# flume-ng agent --conf conf --conf-file kafkasink.conf --name a1 >/dev/null 2>&1 &</span><br></pre></td></tr></table></figure></p><h4 id="14-Interceptor拦截器"><a href="#14-Interceptor拦截器" class="headerlink" title="14.Interceptor拦截器"></a>14.Interceptor拦截器</h4><ul><li>Source将event写入到Channel之前调用拦截器</li><li>Source和Channel之间可以有多个拦截器，不同的拦截器使用不同的 规则处理Event</li><li>可选、轻量级、可插拔的插件</li><li>通过实现Interceptor接口实现自定义的拦截器</li><li>内置拦截器：Timestamp Interceptor、Host Interceptor、UUID Interceptor、Static Interceptor、Regex Filtering Interceptor等<h4 id="15-Timestamp-Interceptor"><a href="#15-Timestamp-Interceptor" class="headerlink" title="15.Timestamp Interceptor"></a>15.Timestamp Interceptor</h4></li><li>Flume使用时间戳拦截器在event头信息中添加时间戳信息， Key为timestamp，Value为拦截器拦截Event时的时间戳</li><li>头信息时间戳的作用，比如HDFS存储的数据采用时间分区存储，Sink可以根据Event头信息中的时间戳将Event按照时间分区写入到 HDFS</li><li>关键参数说明：<ul><li>type:拦截器类型为timestamp<ul><li>preserveExisting：如果头信息中存在timestamp时间戳信息是否保留原来的时间戳信息，true保留，false使用新的时间戳替换已经存在的时间戳，默认值为false<h4 id="16-Host-Interceptor"><a href="#16-Host-Interceptor" class="headerlink" title="16.Host Interceptor"></a>16.Host Interceptor</h4></li></ul></li></ul></li><li>Flume使用主机戳拦截器在Event头信息中添加主机名称或者IP</li><li>主机拦截器的作用：比如Source将Event按照主机名称写入到不同的Channel中便于后续的Sink对不同Channnel中的数据分开处理</li><li>关键参数说明：<ul><li>type:拦截器类型为host</li><li>preserveExisting：如果头信息中存在timestamp时间戳信息是否保留原来的时间戳信息，true保留，false使用新的时间戳替换已经存在的时间戳，默认值为false</li><li>useIP：是否使用IP作为主机信息写入都信息，默认值为false</li><li>hostHeader：设置头信息中主机信息的Key，默认值为host<br><h4 id="17-Host-InterceptorStatic-Interceptor"><a href="#17-Host-InterceptorStatic-Interceptor" class="headerlink" title="17.Host InterceptorStatic Interceptor"></a>17.Host InterceptorStatic Interceptor</h4></li></ul></li><li>Flume使用static interceptor静态拦截器在evetn头信息添加静态信息</li><li>关键参数说明：</li><li>type:拦截器类型为static<ul><li>preserveExisting：如果头信息中存在timestamp时间戳信息是否保留原来的时间戳信息，true保留，false使用新的时间戳替换已经 存在的时间戳，默认值为false</li><li>key：头信息中的键</li><li>value：头信息中键对应的值<h4 id="18-Selector选择器"><a href="#18-Selector选择器" class="headerlink" title="18.Selector选择器"></a>18.Selector选择器</h4></li></ul></li><li>Source将event写入到Channel之前调用拦截器，如果配置了Interceptor拦截器，则Selector在拦截器全部处理完之后调用。通过<br>selector决定event写入Channel的方式</li><li>内置Replicating Channel Selector复制Channel选择器、 Multiplexing Channel Selector复用Channel选择器<h4 id="19-Replicating-Channel-Selector"><a href="#19-Replicating-Channel-Selector" class="headerlink" title="19.Replicating Channel Selector"></a>19.Replicating Channel Selector</h4></li><li>如果Channel选择器没有指定，默认是Replicating Channel Selector。即一个Source以复制的方式将一个event同时写入到多个Channel中，不同的Sink可以从不同的Channel中获取相同的event。</li><li>关键参数说明：<ul><li>selector.type：Channel选择器类型为replicating</li><li>selector.optional：定义可选Channel，当写入event到可选Channel失败时，不会向Source抛出异常，继续执行。多个可选Channel之 间用空格隔开</li></ul></li></ul><p>一个source将一个event拷贝到多个channel，通过不同的sink消费不同的channel，将相同的event输出到不同的地方<br>配置文件：replicating_selector.conf</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">a1.sources = r1</span><br><span class="line">a1.channels = c1 c2</span><br><span class="line">a1.sinks = k1 k2</span><br><span class="line">#定义source</span><br><span class="line">a1.sources.r1.type = netcat</span><br><span class="line">a1.sources.r1.bind = localhost</span><br><span class="line">a1.sources.r1.port = 44444</span><br><span class="line">#设置复制选择器</span><br><span class="line">a1.sources.r1.selector.type = replicating</span><br><span class="line">#设置required channel</span><br><span class="line">a1.sources.r1.channels = c1 c2</span><br><span class="line">#设置channel c1</span><br><span class="line">a1.channels.c1.type = memory </span><br><span class="line">a1.channels.c1.capacity = 1000</span><br><span class="line">a1.channels.c1.transactionCapacity = 1000</span><br><span class="line">#设置channel c2</span><br><span class="line">a1.channels.c2.type = memory </span><br><span class="line">a1.channels.c2.capacity = 1000</span><br><span class="line">a1.channels.c2.transactionCapacity = 1000</span><br><span class="line">#设置kafka sink</span><br><span class="line">a1.sinks.k1.channel = c1</span><br><span class="line">a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink</span><br><span class="line">a1.sinks.k1.kafka.topic = FlumeSelectorTopic1</span><br><span class="line">a1.sinks.k1.kafka.bootstrap.servers = 192.168.43.22:9092,192.168.23.103:9092</span><br><span class="line">a1.sinks.k1.kafka.flumeBatchSize = 5</span><br><span class="line">a1.sinks.k1.kafka.producer.acks = 1</span><br><span class="line">#设置file sink</span><br><span class="line">a1.sinks.k2.channel = c2</span><br><span class="line">a1.sinks.k2.type = file_roll</span><br><span class="line">a1.sinks.k2.sink.directory = /home/hadoop/apps/flume/selector</span><br><span class="line">a1.sinks.k2.sink.rollInterval = 60</span><br></pre></td></tr></table></figure><p>分别写入到kafka和文件中</p><p>创建主题FlumeKafkaSinkTopic1</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-topics.sh --create --zookeeper 192.168.183.100:2181 --replication-factor 1 --partitions 3 --topic FlumeSelectorTopic1</span><br></pre></td></tr></table></figure><p>启动flume agent</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/flume-ng agent --conf conf --conf-file conf/replicating_selector.conf --name a1</span><br></pre></td></tr></table></figure><p>使用telnet发送数据<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">telnet localhost 44444</span><br></pre></td></tr></table></figure></p><p>查看/home/hadoop/apps/flume/selector路径下的数据</p><p>查看kafka FlumeSelectorTopic1主题数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-console-consumer.sh --zookeeper 192.168.183.100:2181 --from-beginning --topic FlumeSelectorTopic1</span><br></pre></td></tr></table></figure><h4 id="20-Multiplexing-Channel-Selector"><a href="#20-Multiplexing-Channel-Selector" class="headerlink" title="20.Multiplexing Channel Selector"></a>20.Multiplexing Channel Selector</h4><p>-Multiplexing Channel Selector多路复用选择器根据event的头信息中不<br>同键值数据来判断Event应该被写入到哪个Channel中</p><ul><li>三种级别的Channel，分别是必选channle、可选channel、默认channel</li><li>关键参数说明：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">selector.type：Channel选择器类型为multiplexing</span><br><span class="line">selector.header：设置头信息中用于检测的headerName</span><br><span class="line">selector.default：默认写入的Channel列表</span><br><span class="line">selector.mapping.*：headerName对应的不同值映射的不同Channel列表</span><br><span class="line">selector.optional：可选写入的Channel列表</span><br></pre></td></tr></table></figure><p>配置文件multiplexing_selector.conf、avro_sink1.conf、avro_sink2.conf、avro_sink3.conf<br>向不同的avro_sink对应的配置文件的agent发送数据，不同的avro_sink配置文件通过static interceptor在event头信息中写入不同的静态数据<br>multiplexing_selector根据event头信息中不同的静态数据类型分别发送到不同的目的地 <br><br>multiplexing_selector.conf<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">a3.sources = r1</span><br><span class="line">a3.channels = c1 c2 c3</span><br><span class="line">a3.sinks = k1 k2 k3</span><br><span class="line">a3.sources.r1.type = avro</span><br><span class="line">a3.sources.r1.bind = 192.168.183.100</span><br><span class="line">a3.sources.r1.port = 8888</span><br><span class="line">a3.sources.r1.threads= 3</span><br><span class="line">#设置multiplexing selector</span><br><span class="line">a3.sources.r1.selector.type = multiplexing</span><br><span class="line">a3.sources.r1.selector.header = logtype</span><br><span class="line">#通过header中logtype键对应的值来选择不同的sink</span><br><span class="line">a3.sources.r1.selector.mapping.ad = c1</span><br><span class="line">a3.sources.r1.selector.mapping.search = c2</span><br><span class="line">a3.sources.r1.selector.default = c3</span><br><span class="line">a3.sources.r1.channels = c1 c2 c3</span><br><span class="line">a3.channels.c1.type = memory</span><br><span class="line">a3.channels.c1.capacity = 10000</span><br><span class="line">a3.channels.c1.transactionCapacity = 1000</span><br><span class="line">a3.channels.c2.type = memory</span><br><span class="line">a3.channels.c2.capacity = 10000</span><br><span class="line">a3.channels.c2.transactionCapacity = 1000</span><br><span class="line">a3.channels.c3.type = memory</span><br><span class="line">a3.channels.c3.capacity = 10000</span><br><span class="line">a3.channels.c3.transactionCapacity = 1000</span><br><span class="line">#分别设置三个sink的不同输出</span><br><span class="line">a3.sinks.k1.type = file_roll</span><br><span class="line">a3.sinks.k1.channel = c1</span><br><span class="line">a3.sinks.k1.sink.directory = /home/hadoop/apps/flume/multiplexing/k11</span><br><span class="line">a3.sinks.k1.sink.rollInterval = 60</span><br><span class="line">a3.sinks.k2.channel = c2</span><br><span class="line">a3.sinks.k2.type = file_roll</span><br><span class="line">a3.sinks.k2.sink.directory = /home/hadoop/apps/flume/multiplexing/k12</span><br><span class="line">a3.sinks.k2.sink.rollInterval = 60</span><br><span class="line">a3.sinks.k3.channel = c3</span><br><span class="line">a3.sinks.k3.type = file_roll</span><br><span class="line">a3.sinks.k3.sink.directory = /home/hadoop/apps/flume/multiplexing/k13</span><br><span class="line">a3.sinks.k3.sink.rollInterval = 60</span><br></pre></td></tr></table></figure></p><p>avro_sink1.conf</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">agent1.sources = r1</span><br><span class="line">agent1.channels = c1</span><br><span class="line">agent1.sinks = k1</span><br><span class="line">agent1.sources.r1.type = netcat</span><br><span class="line">agent1.sources.r1.bind = localhost</span><br><span class="line">agent1.sources.r1.port = 44444</span><br><span class="line">agent1.sources.r1.interceptors = i1</span><br><span class="line">agent1.sources.r1.interceptors.i1.type = static</span><br><span class="line">agent1.sources.r1.interceptors.i1.key = logtype</span><br><span class="line">agent1.sources.r1.interceptors.i1.value = ad</span><br><span class="line">agent1.sources.r1.interceptors.i1.preserveExisting = false</span><br><span class="line">agent1.sources.r1.channels = c1</span><br><span class="line">agent1.channels.c1.type = memory</span><br><span class="line">agent1.channels.c1.capacity = 10000 </span><br><span class="line">agent1.channels.c1.transactionCapacity = 1000</span><br><span class="line">agent1.sinks.k1.type = avro</span><br><span class="line">agent1.sinks.k1.channel = c1</span><br><span class="line">agent1.sinks.k1.hostname = 192.168.183.100</span><br><span class="line">agent1.sinks.k1.port = 8888</span><br></pre></td></tr></table></figure><p>avro_sink2.conf</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">agent2.sources = r1</span><br><span class="line">agent2.channels = c1</span><br><span class="line">agent2.sinks = k1</span><br><span class="line">agent2.sources.r1.type = netcat</span><br><span class="line">agent2.sources.r1.bind = localhost</span><br><span class="line">agent2.sources.r1.port = 44445</span><br><span class="line">agent2.sources.r1.interceptors = i1</span><br><span class="line">agent2.sources.r1.interceptors.i1.type = static</span><br><span class="line">agent2.sources.r1.interceptors.i1.key = logtype</span><br><span class="line">agent2.sources.r1.interceptors.i1.value = search</span><br><span class="line">agent2.sources.r1.interceptors.i1.preserveExisting = false</span><br><span class="line">agent2.sources.r1.channels = c1</span><br><span class="line">agent2.channels.c1.type = memory</span><br><span class="line">agent2.channels.c1.capacity = 10000 </span><br><span class="line">agent2.channels.c1.transactionCapacity = 1000</span><br><span class="line">agent2.sinks.k1.type = avro</span><br><span class="line">agent2.sinks.k1.channel = c1</span><br><span class="line">agent2.sinks.k1.hostname = 192.168.183.100</span><br><span class="line">agent2.sinks.k1.port = 8888</span><br></pre></td></tr></table></figure><p>avro_sink3.conf</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">agent3.sources = r1</span><br><span class="line">agent3.channels = c1</span><br><span class="line">agent3.sinks = k1</span><br><span class="line">agent3.sources.r1.type = netcat</span><br><span class="line">agent3.sources.r1.bind = localhost</span><br><span class="line">agent3.sources.r1.port = 44446</span><br><span class="line">agent3.sources.r1.interceptors = i1</span><br><span class="line">agent3.sources.r1.interceptors.i1.type = static</span><br><span class="line">agent3.sources.r1.interceptors.i1.key = logtype</span><br><span class="line">agent3.sources.r1.interceptors.i1.value = other</span><br><span class="line">agent3.sources.r1.interceptors.i1.preserveExisting = false</span><br><span class="line">agent3.sources.r1.channels = c1</span><br><span class="line">agent3.channels.c1.type = memory</span><br><span class="line">agent3.channels.c1.capacity = 10000 </span><br><span class="line">agent3.channels.c1.transactionCapacity = 1000</span><br><span class="line">agent3.sinks.k1.type = avro</span><br><span class="line">agent3.sinks.k1.channel = c1</span><br><span class="line">agent3.sinks.k1.hostname = 192.168.183.100</span><br><span class="line">agent3.sinks.k1.port = 8888</span><br></pre></td></tr></table></figure><p>在/home/hadoop/apps/flume/multiplexing目录下分别创建看k1 k2 k3目录</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">bin/flume-ng agent --conf conf --conf-file conf/multiplexing_selector.conf --name a3 -Dflume.root.logger=INFO,console</span><br><span class="line">bin/flume-ng agent --conf conf --conf-file conf/avro_sink1.conf --name agent1 >/dev/null 2>&1 &</span><br><span class="line">bin/flume-ng agent --conf conf --conf-file conf/avro_sink2.conf --name agent2 >/dev/null 2>&1 &</span><br><span class="line">bin/flume-ng agent --conf conf --conf-file conf/avro_sink3.conf --name agent3 >/dev/null 2>&1 &</span><br></pre></td></tr></table></figure><p>使用telnet发送数据<br>telnet localhost 44444</p><h4 id="21-Sink-Processor"><a href="#21-Sink-Processor" class="headerlink" title="21.Sink Processor"></a>21.Sink Processor</h4><ul><li>Sink Processor协调多个sink间进行load balance和fail over</li><li>Default Sink Processor只有一个sink，无需创建Sink Processor</li><li>Sink Group：将多个sink放到一个组内，要求组内一个sink消费channel</li><li>Load-Balancing Sink Processor（负载均衡处理器）round_robin(默认)或 random</li><li>Failover Sink Processor（容错处理器）可定义一个sink优先级列表，根据优先级选择使用的sink<h4 id="22-Load-Balancing-Sink-Processor"><a href="#22-Load-Balancing-Sink-Processor" class="headerlink" title="22.Load-Balancing Sink Processor"></a>22.Load-Balancing Sink Processor</h4>关键参数说明：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">sinks：sink组内的子Sink，多个子sink之间用空格隔开</span><br><span class="line">processor.type：设置负载均衡类型load_balance</span><br><span class="line">processor.backoff：设置为true时，如果在系统运行过程中执行的Sink失败，会将失败的Sink放进一个冷却池中。默认值false</span><br><span class="line">processor.selector.maxTimeOut：失败sink在冷却池中最大驻留时间，默认值30000ms</span><br><span class="line">processor.selector：负载均衡选择算法，可以使用轮询“round_robin”、随机“random”或者是继承AbstractSinkSelector类的自定义负载均衡实现类</span><br></pre></td></tr></table></figure><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-2e86bae3b2af0531.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="示例" title=""> </div> <div class="image-caption">示例</div> </figure><h4 id="23-Failover-Sink-Processor"><a href="#23-Failover-Sink-Processor" class="headerlink" title="23.Failover Sink Processor"></a>23.Failover Sink Processor</h4><p>关键参数说明：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">sinks：sink组内的子Sink，多个子sink之间用空格隔开</span><br><span class="line">processor.type：设置故障转移类型“failover”</span><br><span class="line">processor.priority.<sinkName>：指定Sink组内各子Sink的优先级别，优先级从高到低，数值越大优先级越高</span><br><span class="line">processor.maxpenalty：等待失败的Sink恢复的最长时间，默认值30000毫秒</span><br></pre></td></tr></table></figure></p><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-01c63fe5f0586cb0.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="示例" title=""> </div> <div class="image-caption">示例</div> </figure><h4 id="24-Failover应用场景"><a href="#24-Failover应用场景" class="headerlink" title="24.Failover应用场景"></a>24.Failover应用场景</h4><ul><li>分布式日志收集场景<ul><li>多个agent收集不同机器上相同类型的日志数据，为了保障高可用，采用分层部署，日志收集层Collector部署两个甚至多个，Agent通过Failover SinkProcessor实现其中任何一个collector挂掉不影响系统的日志收集服务<figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-782d0cca4d4aa7a7.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="示例" title=""> </div> <div class="image-caption">示例</div> </figure><h1 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h1><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-9b701d7d5da1facc.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="总结" title=""> </div> <div class="image-caption">总结</div> </figure></li></ul></li></ul>]]></content>
<summary type="html">
<h1 id="一、Flume介绍"><a href="#一、Flume介绍" class="headerlink" title="一、Flume介绍"></a>一、Flume介绍</h1><h5 id="1-Flume特点"><a href="#1-Flume特点" class="headerlink" title="1.Flume特点"></a>1.Flume特点</h5><ul>
<li>Flume是一个分布式的、可靠的、高可用的海量日志采集<br>、聚合和传输的系统</li>
<li>数据流模型：Source-Channel-Sink</li>
<li>事务机制保证消息传递的可靠性 </li>
<li>内置丰富插件，轻松与其他系统集成</li>
<li>Java实现，优秀的系统框架设计，模块分明，易于开发
</summary>
<category term="flume" scheme="https://yongliangzhang.github.io/blogs/tags/flume/"/>
<category term="kafka" scheme="https://yongliangzhang.github.io/blogs/tags/kafka/"/>
</entry>
<entry>
<title>Mysql 操作笔记</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/02/Mysql_%E6%93%8D%E4%BD%9C%E7%AC%94%E8%AE%B0/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/02/Mysql_操作笔记/</id>
<published>2018-02-02T09:03:00.000Z</published>
<updated>2018-02-02T09:04:43.000Z</updated>
<content type="html"><![CDATA[<h3 id="1-登录"><a href="#1-登录" class="headerlink" title="1.登录"></a>1.登录</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cd usr/local/mysql/bin --->./myql -uroot</span><br></pre></td></tr></table></figure><h3 id="2-创建数据库"><a href="#2-创建数据库" class="headerlink" title="2.创建数据库"></a>2.创建数据库</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">CREATE DATABASE IF NOT EXISTS jxytest DEFAULT CHARSET utf8 COLLATE utf8_general_ci;</span><br></pre></td></tr></table></figure><h3 id="3-为用户授权操作数据库"><a href="#3-为用户授权操作数据库" class="headerlink" title="3.为用户授权操作数据库"></a>3.为用户授权操作数据库</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">grant all on jxytest.* to 'yxy_jx'@'%' identified by 'lzyljd_yxyjx_2017' with grant option;</span><br></pre></td></tr></table></figure><h3 id="4-刷出权限"><a href="#4-刷出权限" class="headerlink" title="4.刷出权限"></a>4.刷出权限</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">flush privileges;</span><br></pre></td></tr></table></figure><p>查看Mysql最大文件导入大小<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">show VARIABLES like '%max_allowed_packet%';</span><br></pre></td></tr></table></figure></p><p>设置最大文件导入大小：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">set global max_allowed_packet = 9*1024*1024*100; // 900M</span><br></pre></td></tr></table></figure></p><h3 id="5-慢查询"><a href="#5-慢查询" class="headerlink" title="5.慢查询"></a>5.慢查询</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">show status 显示系统状态参数</span><br></pre></td></tr></table></figure><p>1.查询慢查询是否开启：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">show global variables like '%slow_query%';</span><br><span class="line">slow_query_log 为OFF 表示没有开启，mysql默认是没有开启慢查询日志记录功能的</span><br><span class="line">set slow_query_log=1 开启慢查询日志记录</span><br></pre></td></tr></table></figure></p><p>2.查看慢查询时间<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">show variables like 'long_query_time'; 默认为10秒钟，意识是大于10s才算慢查询</span><br></pre></td></tr></table></figure></p><p>3.修改慢查询记录的时间<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">set long_query_time=1; 设置时间为1s</span><br></pre></td></tr></table></figure></p><p>4.记录慢查询日志的次数<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">show status like 'slow_queries';</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<h3 id="1-登录"><a href="#1-登录" class="headerlink" title="1.登录"></a>1.登录</h3><figure class="highlight plain"><table><tr><td class="gutter"><pr
</summary>
<category term="MySQL" scheme="https://yongliangzhang.github.io/blogs/tags/MySQL/"/>
</entry>
<entry>
<title>Hadoop MapReduce优化和资源调度器</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/Hadoop_MapReduce%E4%BC%98%E5%8C%96%E5%92%8C%E8%B5%84%E6%BA%90%E8%B0%83%E5%BA%A6%E5%99%A8/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/Hadoop_MapReduce优化和资源调度器/</id>
<published>2018-02-01T06:06:18.374Z</published>
<updated>2018-02-01T06:05:34.000Z</updated>
<content type="html"><![CDATA[<figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-a9deb9418a339af6.jpg?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Hadoop Shuffle过程" title=""> </div> <div class="image-caption">Hadoop Shuffle过程</div> </figure><h1 id="1-Hadoop-MapReduce-Shuffle过程"><a href="#1-Hadoop-MapReduce-Shuffle过程" class="headerlink" title="1.Hadoop MapReduce Shuffle过程"></a>1.Hadoop MapReduce Shuffle过程</h1><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-a9deb9418a339af6.jpg?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Hadoop Shuffle过程" title=""> </div> <div class="image-caption">Hadoop Shuffle过程</div> </figure><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-0526447f1e1ba015.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Map Shuffle过程图2" title=""> </div> <div class="image-caption">Map Shuffle过程图2</div> </figure><h1 id="2-Shuffle过程要点记录"><a href="#2-Shuffle过程要点记录" class="headerlink" title="2.Shuffle过程要点记录"></a>2.Shuffle过程要点记录</h1><ol><li>每个Map Task把输出结果写到内存中的环形缓冲区。</li><li>当内存环形缓冲区写入的数据量达到一定阈值时，后台线程会把 数据溢写到磁盘。<ul><li>根据Partitioner，把数据写入到不同的partition</li><li>对于每个partition的数据进行排序</li></ul></li><li>随着Map Task的不断运行，磁盘上的溢出文件越来越多<ul><li>将这些溢出文件合并<ul><li>对于一个partition下的不同分片，使用归并排序，同一分区内数据有序</li></ul></li></ul></li><li>Reduce Task通过网络远程拷贝MapTask的结果文件中的属于它的分区数据<ul><li>合并所有已拷贝过来的数据文件</li><li>采用归并排序算法，对文件数据内容整理排序，将相同key的数据分 为一组，不同key之间有序</li><li>最终生成一个key对应一组值的数据集，一个key对应的一组数据会调用一次reduce方法<h1 id="3-Combinery优化总结"><a href="#3-Combinery优化总结" class="headerlink" title="3. Combinery优化总结"></a>3. Combinery优化总结</h1><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-2b84e712726ccd27.jpg?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Combiner优化" title=""> </div> <div class="image-caption">Combiner优化</div> </figure></li></ul></li><li>Combiner调用的地方<ul><li>MapTask的环形缓冲区向磁盘溢写文件之前调用Combiner</li><li>Map阶段在合并本地多个文件写入一个大文件之前调用Combiner</li></ul></li><li>使用Combiner的好处<ul><li>减少Map Task输出数据量，由于临时结果写入到本地磁盘，所以能 够减少磁盘IO</li><li>减少Reduce-Map网络传输数据量，由于reduce需要远程通过网络从 Map拷贝数据，提高拷贝速度</li></ul></li><li>应用场景<ul><li>针对结果可以叠加的场景</li><li>SUM(YES) Average（NO）</li></ul></li><li>设置方法（local reducer）<ul><li>job.setCombinerClass(WordCountReducer.class)<h1 id="4-YARN-资源调度器"><a href="#4-YARN-资源调度器" class="headerlink" title="4.YARN 资源调度器"></a>4.YARN 资源调度器</h1><h4 id="1-YARN-FIFO-Scheduler"><a href="#1-YARN-FIFO-Scheduler" class="headerlink" title="1. YARN-FIFO Scheduler"></a>1. YARN-FIFO Scheduler</h4><strong>将所有应用程序放入到一个队列中</strong></li></ul></li></ol><ul><li>先进入队里排在前面的程序先获得资源<br><strong>局限性</strong></li><li>资源利用率低，无法交叉运行作业</li><li>不够灵活，比如紧急的作业无法插队，耗时长作业拖慢耗时短作业<h4 id="2-YARN-多队列分开调度器"><a href="#2-YARN-多队列分开调度器" class="headerlink" title="2. YARN-多队列分开调度器"></a>2. YARN-多队列分开调度器</h4><strong>所有资源按照比例划分到不同的队列</strong><br></li></ul><p><strong>每个队列可以实现单独的调度策略</strong><br></p><p><strong>优点</strong><br></p><ul><li>按照不同的资源使用情况将资源划分到不同队列</li><li>能够让更多的应用程序获得资源</li><li>使用灵活，资源利用率高<br></li><li><p><strong>调度器</strong></p></li><li><p>CapacityScheduler调度器</p></li><li>FairScheduler调度器</li></ul><p><strong>CapacityScheduler</strong></p><ul><li>由Yahoo开源，共享集群调度器</li><li>以队列方式组织作业</li><li>每个队列内部采用FIFO调度策略</li><li>每个队列分配一定比例资源</li><li>可限制每个用户使用资源量<figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-21603d1cdbfffd40.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="CapacityScheduler.png" title=""> </div> <div class="image-caption">CapacityScheduler.png</div> </figure><strong>CapacityScheduler 配置方法</strong></li></ul><p>在yarn-site.xml 设置使用CapacityScheduler调度器</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><property></span><br><span class="line"> <name>yarn.resourcemanager.scheduler.class</name></span><br><span class="line"> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value></span><br><span class="line"></property></span><br></pre></td></tr></table></figure><p>在Hadoop配置文件目录下/usr/local/hadoop/etc/hadoop创建capacity-scheduler.xml,添加信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><configuration></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.scheduler.capacity.root.queues</name></span><br><span class="line"> <value>default,data-bi</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.scheduler.capacity.root.default.capacity</name> </span><br><span class="line"> <value>60</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.scheduler.capacity.root.default.maximum-capacity</name></span><br><span class="line"> <value>80</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.scheduler.capacity.root.bi.capacity</name></span><br><span class="line"> <value>40</vaule></span><br><span class="line"> </property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><p><strong>配置说明</strong><br></p><ul><li>capacity-scheduler.xml参数说明</li><li>capacity：队列占用的集群资源容量百分比，所有队列的容量 之和应小于100</li><li>maximum-capacity：由于存在资源共享，因此一个队列使用 的资源量可能超过其容量，而最多使用资源量可通过该参数 限制</li><li>配置完成无需重启YARN，使用管理命令刷新调度配置 bin/yarn rmadmin -refreshQueues</li></ul><p><strong>FairScheduler</strong><br><br> 公平调度器的目的:</p><ul><li>允许多用户共享集群资源。</li><li>允许短时的临时作业与长时作业共享集群资源</li><li>根据比例来管理集群资源，确保集群资源的有效利用’</li></ul><p><strong>FairScheduler配置方法</strong><br>在Hadoop配置目录下/usr/local/hadoop/etc/hadoop yarn-site.xml 增加如下信息：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><property></span><br><span class="line"> <name>yarn.resourcemanager.scheduler.class</name></span><br><span class="line"><value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value></span><br><span class="line"></property></span><br><span class="line"><property></span><br><span class="line"><name>yarn.scheduler.fair.user-as-default-queue</name></span><br><span class="line"><value>true</value></span><br><span class="line"></property></span><br><span class="line"><property></span><br><span class="line"><name>yarn.scheduler.fair.allocation.file</name></span><br><span class="line"><value>/usr/local/hadoop/etc/hadoop/fair-scheduler.xml</value></span><br><span class="line"></property></span><br><span class="line"><property></span><br><span class="line"><name>yarn.scheduler.fair.preemption</name></span><br><span class="line"><value>true</value></span><br><span class="line"></property></span><br></pre></td></tr></table></figure></p><p>新建一个公平调度配置文件fair-scheduler.xml ，信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><allocations></span><br><span class="line"><queue name="data_bi"></span><br><span class="line"><minResources>8000 mb,4 vcores</minResources></span><br><span class="line"><maxResources>10000 mb, 6 vcores</maxResources></span><br><span class="line"><maxRunningApps>2</maxRunningApps></span><br><span class="line"><weight>1.0</weight></span><br><span class="line"></queue></span><br><span class="line"></allocations></span><br></pre></td></tr></table></figure><p>上述配置以 data_bi 用户名作为公平调度的队列名称。</p><p><strong>yarn-site.xml参数说明</strong></p><ul><li>yarn.resourcemanager.scheduler.class配置yarn使用的调度器类型</li><li>yarn.scheduler.fair.allocation.file配置公平调度器自定义配置文件路径，该文件每隔10秒就会被加载一次，这样就可以在集群运行过程中改变队列的配置</li><li>yarn.scheduler.fair.user-as-default-queue当应用程序未指定队列名时，是否指定用户名作为应用程序所在的队列名。如果设置为false或者未设置，所有 未知队列的应用程序将被提交到default队列中，默认值为true</li><li>yarn.scheduler.fair.preemption如果一个队列占用的资源量少于最小资源量限制，是否启用资源抢占，默认false。抢占机制可以使其他队列的作业容器终止，从而使占用的资源让出，将资源分配给占用资源量少于最小资源量限制的队列<br></li></ul><p><strong>fair-scheduler.xml参数说明</strong></p><ul><li>queue name：配置队列名</li><li>minResources ：分配给该队列的最小资源量，设置格式为“X mb, Y vcores”，当调度策略属性schedulingPolicy的属性值是fair时，其cores值会被忽略，仅按照申请的内存大小来调度。</li><li>maxResources：分配给该队列的最大资源量。设置格式为“X mb, Y vcores”，当调度策略属性schedulingPolicy的属性值是fair时，其cores值会被忽略，仅按照申请的内存大小来调度。</li><li>maxRunningApps：最多同时运行的应用程序数目。通过限制该数目，可防止超量MapTask同时运行时产生的中间输出结果撑爆磁盘。</li><li>weight：标记了资源池的权重，当资源池中有任务等待，并且集群中有空闲资源时候，每个资源池可以根据权重获得不同比例的集群空闲资源，默认值是1</li></ul>]]></content>
<summary type="html">
<figure class="image-bubble">
<div class="img-lightbox">
<div class="overlay"></div>
</summary>
</entry>
<entry>
<title>CentOS磁盘空间爆满的解决办法</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/CentOS%E7%A3%81%E7%9B%98%E7%A9%BA%E9%97%B4%E7%88%86%E6%BB%A1%E7%9A%84%E8%A7%A3%E5%86%B3%E5%8A%9E%E6%B3%95/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/CentOS磁盘空间爆满的解决办法/</id>
<published>2018-02-01T06:04:28.743Z</published>
<updated>2018-02-01T06:03:48.000Z</updated>
<content type="html"><![CDATA[<h1 id="1-查看磁盘具体占用情况"><a href="#1-查看磁盘具体占用情况" class="headerlink" title="1.查看磁盘具体占用情况"></a>1.查看磁盘具体占用情况</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost gsidc]# df -hT</span><br><span class="line">Filesystem Type Size Used Avail Use% Mounted on</span><br><span class="line">/dev/sda3 xfs 92G 21G 72G 23% /</span><br><span class="line">devtmpfs devtmpfs 7.8G 0 7.8G 0% /dev</span><br><span class="line">tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm</span><br><span class="line">tmpfs tmpfs 7.8G 129M 7.7G 2% /run</span><br><span class="line">tmpfs tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup</span><br><span class="line">/dev/sda1 xfs 197M 192M 5.0M 98% /boot</span><br><span class="line">tmpfs tmpfs 1.6G 12K 1.6G 1% /run/user/42</span><br><span class="line">tmpfs tmpfs 1.6G 0 1.6G 0% /run/user/1000</span><br></pre></td></tr></table></figure><h1 id="2-删除具体位置磁盘占用无用文件"><a href="#2-删除具体位置磁盘占用无用文件" class="headerlink" title="2.删除具体位置磁盘占用无用文件"></a>2.删除具体位置磁盘占用无用文件</h1><h1 id="3-查看磁盘删除情况"><a href="#3-查看磁盘删除情况" class="headerlink" title="3. 查看磁盘删除情况"></a>3. 查看磁盘删除情况</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost gsidc]# lsof |grep delete</span><br></pre></td></tr></table></figure><h1 id="4-彻底释放磁盘空间"><a href="#4-彻底释放磁盘空间" class="headerlink" title="4. 彻底释放磁盘空间"></a>4. 彻底释放磁盘空间</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">mysqld_sa 11317 root cwd DIR 8,6 0 102107 /usr/local/mysql (deleted)</span><br><span class="line">mysqld_sa 11317 root 255r REG 8,6 13620 102182 /usr/local/mysql/bin/mysqld_safe (deleted)</span><br><span class="line">mysqld 11353 mysql txt REG 8,6 29382763 102138 /usr/local/mysql/bin/mysqld (deleted)</span><br><span class="line">mysqld 11353 mysql 5u REG 8,9 0 58761 /tmp/ibpfbHsa (deleted)</span><br><span class="line">mysqld 11353 mysql 6u REG 8,9 0 58771 /tmp/ibqoMnag (deleted)</span><br><span class="line">mysqld 11353 mysql 7u REG 8,9 0 58772 /tmp/ibN4C4Rl (deleted)</span><br><span class="line">mysqld 11353 mysql 8u REG 8,9 0 58781 /tmp/ibkWP8zr (deleted)</span><br><span class="line">mysqld 11353 mysql 12u REG 8,9 0 58782 /tmp/ibnqrZsx (deleted)</span><br><span class="line">java 16961 root 1w REG 8,3 5822263296 457562 /opt/tomcat/logs/catalina.out (deleted)</span><br><span class="line">java 16961 root 2w REG 8,3 5822263296 457562 /opt/tomcat/logs/catalina.out (deleted)</span><br><span class="line">java 16961 root 12w REG 8,3 5469261824 197191 /opt/tomcat/logs/catalina.2013-01-02.log (deleted)</span><br><span class="line">java 16961 root 13w REG 8,3 2166784 197192 /opt/tomcat/logs/localhost.2013-01-02.log (deleted)</span><br><span class="line">java 16961 root 14w REG 8,3 0 391681 /opt/tomcat/logs/manager.2013-01-02.log (deleted)</span><br><span class="line">java 16961 root 15w REG 8,3 0 391682 /opt/tomcat/logs/host-manager.2013-01-02.log (deleted)</span><br></pre></td></tr></table></figure><p>关闭删除文件进程<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost /]# kill -9 16961</span><br></pre></td></tr></table></figure></p><p>磁盘容量恢复</p>]]></content>
<summary type="html">
<h1 id="1-查看磁盘具体占用情况"><a href="#1-查看磁盘具体占用情况" class="headerlink" title="1.查看磁盘具体占用情况"></a>1.查看磁盘具体占用情况</h1><figure class="highlight plain"><
</summary>
</entry>
<entry>
<title>Keepalived+Nginx+Tomcat 实现高可用Web集群</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/Keepalived+Nginx+Tomcat_%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8Web%E9%9B%86%E7%BE%A4/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/Keepalived+Nginx+Tomcat_实现高可用Web集群/</id>
<published>2018-02-01T06:03:33.047Z</published>
<updated>2018-02-01T06:02:45.000Z</updated>
<content type="html"><![CDATA[<figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://static.blog.mozhu.org/images/nginx/Keepalived+Nginx+Tomcat2.png" alt="集群规划图片" title=""> </div> <div class="image-caption">集群规划图片</div> </figure><h1 id="一、Nginx的安装过程"><a href="#一、Nginx的安装过程" class="headerlink" title="一、Nginx的安装过程"></a>一、Nginx的安装过程</h1><h3 id="1-下载Nginx安装包，安装依赖环境包"><a href="#1-下载Nginx安装包，安装依赖环境包" class="headerlink" title="1.下载Nginx安装包，安装依赖环境包"></a>1.下载Nginx安装包，安装依赖环境包</h3><h4 id="1-安装-C-编译环境"><a href="#1-安装-C-编译环境" class="headerlink" title="(1)安装 C++编译环境"></a>(1)安装 C++编译环境</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum -y install gcc #C++</span><br></pre></td></tr></table></figure><h4 id="2-安装pcre"><a href="#2-安装pcre" class="headerlink" title="(2)安装pcre"></a>(2)安装pcre</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum -y install pcre-devel</span><br></pre></td></tr></table></figure><h4 id="3-安装zlib"><a href="#3-安装zlib" class="headerlink" title="(3)安装zlib"></a>(3)安装zlib</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum -y install zlib-devel</span><br></pre></td></tr></table></figure><h4 id="4-安装Nginx"><a href="#4-安装Nginx" class="headerlink" title="(4)安装Nginx"></a>(4)安装Nginx</h4><p>定位到nginx 解压文件位置，执行编译安装命令<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost nginx-1.12.2]# pwd</span><br><span class="line">/usr/local/nginx/nginx-1.12.2</span><br><span class="line">[root@localhost nginx-1.12.2]# ./configure && make && make install</span><br></pre></td></tr></table></figure></p><h4 id="5-启动Nginx"><a href="#5-启动Nginx" class="headerlink" title="(5)启动Nginx"></a>(5)启动Nginx</h4><p>安装完成后先寻找那安装完成的目录位置<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost nginx-1.12.2]# whereis nginx</span><br><span class="line">nginx: /usr/local/nginx</span><br><span class="line">[root@localhost nginx-1.12.2]#</span><br></pre></td></tr></table></figure></p><p>进入Nginx子目录sbin启动Nginx<br><figure class="highlight plain"><figcaption><span>nginx]# cd sbin/</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost sbin]# ls</span><br><span class="line">nginx</span><br><span class="line">[root@localhost sbin]# ./nginx &</span><br><span class="line">[1] 5768</span><br><span class="line">[root@localhost sbin]#</span><br></pre></td></tr></table></figure></p><p>查看Nginx是否启动<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-21709243b7d20285.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Niginx启动成功截图" title=""> </div> <div class="image-caption">Niginx启动成功截图</div> </figure><br>或通过进程查看Nginx启动情况<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost sbin]# ps -aux|grep nginx</span><br><span class="line">root 5769 0.0 0.0 20484 608 ? Ss 14:03 0:00 nginx: master process ./nginx</span><br><span class="line">nobody 5770 0.0 0.0 23012 1620 ? S 14:03 0:00 nginx: worker process</span><br><span class="line">root 5796 0.0 0.0 112668 972 pts/0 R+ 14:07 0:00 grep --color=auto nginx</span><br><span class="line">[1]+ 完成 ./nginx</span><br><span class="line">[root@localhost sbin]#</span><br></pre></td></tr></table></figure></p><p>到此Nginx安装完成并启动成功。</p><h4 id="6-Nginx快捷启动和开机启动配置"><a href="#6-Nginx快捷启动和开机启动配置" class="headerlink" title="(6)Nginx快捷启动和开机启动配置"></a>(6)Nginx快捷启动和开机启动配置</h4><p>编辑Nginx快捷启动脚本【<strong><font color="red">注意Nginx安装路径</font></strong>，<strong>需要根据自己的NGINX路径进行改动</strong>】<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost init.d]# vim /etc/rc.d/init.d/nginx</span><br></pre></td></tr></table></figure></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br></pre></td><td class="code"><pre><span class="line">#!/bin/sh</span><br><span class="line">#</span><br><span class="line"># nginx - this script starts and stops the nginx daemon</span><br><span class="line">#</span><br><span class="line"># chkconfig: - 85 15</span><br><span class="line"># description: Nginx is an HTTP(S) server, HTTP(S) reverse \</span><br><span class="line"># proxy and IMAP/POP3 proxy server</span><br><span class="line"># processname: nginx</span><br><span class="line"># config: /etc/nginx/nginx.conf</span><br><span class="line"># config: /usr/local/nginx/conf/nginx.conf</span><br><span class="line"># pidfile: /usr/local/nginx/logs/nginx.pid</span><br><span class="line"> </span><br><span class="line"># Source function library.</span><br><span class="line">. /etc/rc.d/init.d/functions</span><br><span class="line"> </span><br><span class="line"># Source networking configuration.</span><br><span class="line">. /etc/sysconfig/network</span><br><span class="line"> </span><br><span class="line"># Check that networking is up.</span><br><span class="line">[ "$NETWORKING" = "no" ] && exit 0</span><br><span class="line">nginx="/usr/local/nginx/sbin/nginx"</span><br><span class="line">prog=$(basename $nginx)</span><br><span class="line">NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"</span><br><span class="line">[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx</span><br><span class="line">lockfile=/var/lock/subsys/nginx</span><br><span class="line"> </span><br><span class="line">make_dirs() {</span><br><span class="line"> # make required directories</span><br><span class="line"> user=`$nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`</span><br><span class="line"> if [ -z "`grep $user /etc/passwd`" ]; then</span><br><span class="line"> useradd -M -s /bin/nologin $user</span><br><span class="line"> fi</span><br><span class="line"> options=`$nginx -V 2>&1 | grep 'configure arguments:'`</span><br><span class="line"> for opt in $options; do</span><br><span class="line"> if [ `echo $opt | grep '.*-temp-path'` ]; then</span><br><span class="line"> value=`echo $opt | cut -d "=" -f 2`</span><br><span class="line"> if [ ! -d "$value" ]; then</span><br><span class="line"> # echo "creating" $value</span><br><span class="line"> mkdir -p $value && chown -R $user $value</span><br><span class="line"> fi</span><br><span class="line"> fi</span><br><span class="line"> done</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">start() {</span><br><span class="line"> [ -x $nginx ] || exit 5</span><br><span class="line"> [ -f $NGINX_CONF_FILE ] || exit 6</span><br><span class="line"> make_dirs</span><br><span class="line"> echo -n $"Starting $prog: "</span><br><span class="line"> daemon $nginx -c $NGINX_CONF_FILE</span><br><span class="line"> retval=$?</span><br><span class="line"> echo</span><br><span class="line"> [ $retval -eq 0 ] && touch $lockfile</span><br><span class="line"> return $retval</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">stop() {</span><br><span class="line"> echo -n $"Stopping $prog: "</span><br><span class="line"> killproc $prog -QUIT</span><br><span class="line"> retval=$?</span><br><span class="line"> echo</span><br><span class="line"> [ $retval -eq 0 ] && rm -f $lockfile</span><br><span class="line"> return $retval</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">restart() {</span><br><span class="line"> #configtest || return $?</span><br><span class="line"> stop</span><br><span class="line"> sleep 1</span><br><span class="line"> start</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">reload() {</span><br><span class="line"> #configtest || return $?</span><br><span class="line"> echo -n $"Reloading $prog: "</span><br><span class="line"> killproc $nginx -HUP</span><br><span class="line"> RETVAL=$?</span><br><span class="line"> echo</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">force_reload() {</span><br><span class="line"> restart</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">configtest() {</span><br><span class="line"> $nginx -t -c $NGINX_CONF_FILE</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">rh_status() {</span><br><span class="line"> status $prog</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">rh_status_q() {</span><br><span class="line"> rh_status >/dev/null 2>&1</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">case "$1" in</span><br><span class="line">start)</span><br><span class="line">rh_status_q && exit 0</span><br><span class="line">$1</span><br><span class="line">;;</span><br><span class="line">stop)</span><br><span class="line"> </span><br><span class="line">rh_status_q || exit 0</span><br><span class="line">$1</span><br><span class="line">;;</span><br><span class="line">restart|configtest)</span><br><span class="line">$1</span><br><span class="line">;;</span><br><span class="line">reload)</span><br><span class="line">rh_status_q || exit 7</span><br><span class="line">$1</span><br><span class="line">;;</span><br><span class="line">force-reload)</span><br><span class="line">force_reload</span><br><span class="line">;;</span><br><span class="line">status)</span><br><span class="line">rh_status</span><br><span class="line">;;</span><br><span class="line">condrestart|try-restart)</span><br><span class="line">rh_status_q || exit 0</span><br><span class="line">;;</span><br><span class="line">*)</span><br><span class="line">echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"</span><br><span class="line">exit 2</span><br><span class="line">esac</span><br></pre></td></tr></table></figure><p>为启动脚本授权 并加入开机启动<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost init.d]# chmod -R 777 /etc/rc.d/init.d/nginx </span><br><span class="line">[root@localhost init.d]# chkconfig nginx</span><br></pre></td></tr></table></figure></p><p>启动Nginx<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost init.d]# ./nginx start</span><br></pre></td></tr></table></figure></p><p>将Nginx加入系统环境变量<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost init.d]# echo 'export PATH=$PATH:/usr/local/nginx/sbin'>>/etc/profile && source /etc/profile</span><br></pre></td></tr></table></figure></p><p>Nginx命令 [ service nginx (start|stop|restart) ]<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost init.d]# service nginx start</span><br><span class="line">Starting nginx (via systemctl): [ 确定 ]</span><br></pre></td></tr></table></figure></p><p><strong>Tips</strong>:<font color="green"><strong>快捷命令</strong></font><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">service nginx (start|stop|restart)</span><br></pre></td></tr></table></figure></p><h1 id="二、KeepAlived安装和配置"><a href="#二、KeepAlived安装和配置" class="headerlink" title="二、KeepAlived安装和配置"></a>二、KeepAlived安装和配置</h1><h3 id="1-安装Keepalived依赖环境"><a href="#1-安装Keepalived依赖环境" class="headerlink" title="1.安装Keepalived依赖环境"></a>1.安装Keepalived依赖环境</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">yum install -y popt-devel </span><br><span class="line">yum install -y ipvsadm</span><br><span class="line">yum install -y libnl*</span><br><span class="line">yum install -y libnf*</span><br><span class="line">yum install -y openssl-devel</span><br></pre></td></tr></table></figure><h3 id="2-编译Keepalived并安装"><a href="#2-编译Keepalived并安装" class="headerlink" title="2.编译Keepalived并安装"></a>2.编译Keepalived并安装</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived-1.3.9]# ./configure</span><br><span class="line">[root@localhost keepalived-1.3.9]# make && make install</span><br></pre></td></tr></table></figure><h3 id="3-将Keepalive-安装成系统服务"><a href="#3-将Keepalive-安装成系统服务" class="headerlink" title="3.将Keepalive 安装成系统服务"></a>3.将Keepalive 安装成系统服务</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost etc]# mkdir /etc/keepalived</span><br><span class="line">[root@localhost etc]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/</span><br></pre></td></tr></table></figure><p>手动复制默认的配置文件到默认路径<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost etc]# mkdir /etc/keepalived</span><br><span class="line">[root@localhost etc]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/</span><br><span class="line">[root@localhost etc]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/</span><br></pre></td></tr></table></figure></p><p>为keepalived 创建软链接<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost sysconfig]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/</span><br></pre></td></tr></table></figure></p><p>设置Keepalived开机自启动<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost sysconfig]# chkconfig keepalived on</span><br><span class="line">注意：正在将请求转发到“systemctl enable keepalived.service”。</span><br><span class="line">Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service</span><br></pre></td></tr></table></figure></p><p>启动Keepalived服务<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# keepalived -D -f /etc/keepalived/keepalived.conf</span><br></pre></td></tr></table></figure></p><p>关闭Keepalived服务<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# killall keepalived</span><br></pre></td></tr></table></figure></p><h1 id="三、集群规划和搭建"><a href="#三、集群规划和搭建" class="headerlink" title="三、集群规划和搭建"></a>三、集群规划和搭建</h1><h2 id=""><a href="#" class="headerlink" title=""></a><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://static.blog.mozhu.org/images/nginx/Keepalived+Nginx+Tomcat2.png" alt="集群规划图片" title=""> </div> <div class="image-caption">集群规划图片</div> </figure><br></h2><p>环境准备：<br><br>CentOS 7.2 <br><br>Keepalived Version 1.4.0 - December 29, 2017 <br><br>Nginx Version: nginx/1.12.2 <br></p><h2 id="Tomcat-nbsp-nbsp-nbsp-nbsp-Version-8"><a href="#Tomcat-nbsp-nbsp-nbsp-nbsp-Version-8" class="headerlink" title="Tomcat Version:8"></a>Tomcat Version:8</h2><p>集群规划清单<br>虚拟机 | IP |说明<br>—|—|——-<br>Keepalived+Nginx1[Master] | 192.168.43.101 | Nginx Server 01<br>Keeepalived+Nginx[Backup] | 192.168.43.102 | Nginx Server 02<br>Tomcat01|192.168.43.103|Tomcat Web Server01<br>Tomcat02|192.168.43.104|Tomcat Web Server02<br>VIP|192.168.43.150|虚拟漂移IP</p><h2 id="1-更改Tomcat默认欢迎页面，用于标识切换Web"><a href="#1-更改Tomcat默认欢迎页面，用于标识切换Web" class="headerlink" title="1.更改Tomcat默认欢迎页面，用于标识切换Web"></a>1.更改Tomcat默认欢迎页面，用于标识切换Web</h2><p> 更改TomcatServer01 节点ROOT/index.jsp 信息，加入TomcatIP地址，并加入Nginx值，即修改节点192.168.43.103信息如下：<br><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><div id="asf-box"></span><br><span class="line"> <h1>${pageContext.servletContext.serverInfo}(192.168.224.103)<%=request.getHeader("X-NGINX")%></h1></span><br><span class="line"></div></span><br></pre></td></tr></table></figure></p><p>更改TomcatServer02 节点ROOT/index.jsp信息，加入TomcatIP地址，并加入Nginx值，即修改节点192.168.43.104信息如下：<br><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><div id="asf-box"></span><br><span class="line"> <h1>${pageContext.servletContext.serverInfo}(192.168.224.104)<%=request.getHeader("X-NGINX")%></h1></span><br><span class="line"></div></span><br></pre></td></tr></table></figure></p><h2 id="2-启动Tomcat服务，查看Tomcat服务IP信息，此时Nginx未启动，因此request-header没有Nginx信息。"><a href="#2-启动Tomcat服务，查看Tomcat服务IP信息，此时Nginx未启动，因此request-header没有Nginx信息。" class="headerlink" title="2.启动Tomcat服务，查看Tomcat服务IP信息，此时Nginx未启动，因此request-header没有Nginx信息。"></a>2.启动Tomcat服务，查看Tomcat服务IP信息，此时Nginx未启动，因此request-header没有Nginx信息。</h2><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://static.blog.mozhu.org/images/nginx/1-1.png" alt="Tomcat启动信息" title=""> </div> <div class="image-caption">Tomcat启动信息</div> </figure><h2 id="3-配置Nginx代理信息"><a href="#3-配置Nginx代理信息" class="headerlink" title="3.配置Nginx代理信息"></a>3.配置Nginx代理信息</h2><h4 id="1-配置Master节点-192-168-43-101-代理信息"><a href="#1-配置Master节点-192-168-43-101-代理信息" class="headerlink" title="1.配置Master节点[192.168.43.101]代理信息"></a>1.配置Master节点[192.168.43.101]代理信息</h4> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"> upstream tomcat {</span><br><span class="line"> server 192.168.43.103:8080 weight=1;</span><br><span class="line"> server 192.168.43.104:8080 weight=1;</span><br><span class="line">}</span><br><span class="line">server{</span><br><span class="line"> location / {</span><br><span class="line"> proxy_pass http://tomcat;</span><br><span class="line"> proxy_set_header X-NGINX "NGINX-1";</span><br><span class="line"> }</span><br><span class="line"> #......其他省略</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="2-配置Backup节点-192-168-43-102-代理信息"><a href="#2-配置Backup节点-192-168-43-102-代理信息" class="headerlink" title="2.配置Backup节点[192.168.43.102]代理信息"></a>2.配置Backup节点[192.168.43.102]代理信息</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">upstream tomcat {</span><br><span class="line"> server 192.168.43.103:8080 weight=1;</span><br><span class="line"> server 192.168.43.104:8080 weight=1;</span><br><span class="line">}</span><br><span class="line">server{</span><br><span class="line"> location / {</span><br><span class="line"> proxy_pass http://tomcat;</span><br><span class="line"> proxy_set_header X-NGINX "NGINX-2";</span><br><span class="line"> }</span><br><span class="line"> #......其他省略</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="3-启动Master-节点Nginx服务"><a href="#3-启动Master-节点Nginx服务" class="headerlink" title="3.启动Master 节点Nginx服务"></a>3.启动Master 节点Nginx服务</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost init.d]# service nginx start</span><br><span class="line">Starting nginx (via systemctl): [ 确定 ]</span><br></pre></td></tr></table></figure><p>此时访问 192.168.43.101 可以看到103和104节点Tcomat交替显示，说明Nginx服务已经将请求负载到了2台tomcat上。<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://static.blog.mozhu.org/images/nginx/1-3.png" alt="Nginx 负载效果" title=""> </div> <div class="image-caption">Nginx 负载效果</div> </figure></p><h4 id="4-同理配置Backup-192-168-43-102-Nginx信息，启动Nginx后，访问192-168-43-102后可以看到Backup节点已起到负载的效果。"><a href="#4-同理配置Backup-192-168-43-102-Nginx信息，启动Nginx后，访问192-168-43-102后可以看到Backup节点已起到负载的效果。" class="headerlink" title="4.同理配置Backup[192.168.43.102] Nginx信息，启动Nginx后，访问192.168.43.102后可以看到Backup节点已起到负载的效果。"></a>4.同理配置Backup[192.168.43.102] Nginx信息，启动Nginx后，访问192.168.43.102后可以看到Backup节点已起到负载的效果。</h4> <figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://static.blog.mozhu.org/images/nginx/1-4.png" alt="Backup负载效果" title=""> </div> <div class="image-caption">Backup负载效果</div> </figure><h2 id="4-配置Keepalived-脚本信息"><a href="#4-配置Keepalived-脚本信息" class="headerlink" title="4.配置Keepalived 脚本信息"></a>4.配置Keepalived 脚本信息</h2><h4 id="1-在Master节点和Slave节点-etc-keepalived目录下添加check-nginx-sh-文件，用于检测Nginx的存货状况，添加keepalived-conf文件"><a href="#1-在Master节点和Slave节点-etc-keepalived目录下添加check-nginx-sh-文件，用于检测Nginx的存货状况，添加keepalived-conf文件" class="headerlink" title="1.在Master节点和Slave节点 /etc/keepalived目录下添加check_nginx.sh 文件，用于检测Nginx的存货状况，添加keepalived.conf文件"></a>1.<strong><font color="red">在Master节点和Slave节点</font></strong> /etc/keepalived目录下添加check_nginx.sh 文件，用于检测Nginx的存货状况，添加keepalived.conf文件<br></h4><p> check_nginx.sh文件信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">#!/bin/bash</span><br><span class="line">#时间变量，用于记录日志</span><br><span class="line">d=`date --date today +%Y%m%d_%H:%M:%S`</span><br><span class="line">#计算nginx进程数量</span><br><span class="line">n=`ps -C nginx --no-heading|wc -l`</span><br><span class="line">#如果进程为0，则启动nginx，并且再次检测nginx进程数量，</span><br><span class="line">#如果还为0，说明nginx无法启动，此时需要关闭keepalived</span><br><span class="line">if [ $n -eq "0" ]; then</span><br><span class="line"> /etc/rc.d/init.d/nginx start</span><br><span class="line"> n2=`ps -C nginx --no-heading|wc -l`</span><br><span class="line"> if [ $n2 -eq "0" ]; then</span><br><span class="line"> echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log</span><br><span class="line"> systemctl stop keepalived</span><br><span class="line"> fi</span><br><span class="line">fi</span><br></pre></td></tr></table></figure><p>添加完成后，为check_nginx.sh 文件授权，便于脚本获得执行权限。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# chmod -R 777 /etc/keepalived/check_nginx.sh</span><br></pre></td></tr></table></figure><h4 id="2-在Master-节点-etc-keepalived目录下，添加keepalived-conf-文件，具体信息如下："><a href="#2-在Master-节点-etc-keepalived目录下，添加keepalived-conf-文件，具体信息如下：" class="headerlink" title="2.在Master 节点 /etc/keepalived目录下，添加keepalived.conf 文件，具体信息如下："></a>2.在Master 节点 /etc/keepalived目录下，添加keepalived.conf 文件，具体信息如下：</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">vrrp_script chk_nginx { </span><br><span class="line"> script "/etc/keepalived/check_nginx.sh" //检测nginx进程的脚本 </span><br><span class="line"> interval 2 </span><br><span class="line"> weight -20 </span><br><span class="line">} </span><br><span class="line"></span><br><span class="line">global_defs { </span><br><span class="line"> notification_email { </span><br><span class="line"> //可以添加邮件提醒 </span><br><span class="line"> } </span><br><span class="line">} </span><br><span class="line">vrrp_instance VI_1 { </span><br><span class="line"> state MASTER #标示状态为MASTER 备份机为BACKUP</span><br><span class="line"> interface ens33 #设置实例绑定的网卡(ip addr查看，需要根据个人网卡绑定)</span><br><span class="line"> virtual_router_id 51 #同一实例下virtual_router_id必须相同 </span><br><span class="line"> mcast_src_ip 192.168.43.101 </span><br><span class="line"> priority 250 #MASTER权重要高于BACKUP 比如BACKUP为240 </span><br><span class="line"> advert_int 1 #MASTER与BACKUP负载均衡器之间同步检查的时间间隔，单位是秒</span><br><span class="line"> nopreempt #非抢占模式</span><br><span class="line"> authentication { #设置认证</span><br><span class="line"> auth_type PASS #主从服务器验证方式</span><br><span class="line"> auth_pass 123456 </span><br><span class="line"> } </span><br><span class="line"> track_script { </span><br><span class="line"> check_nginx </span><br><span class="line"> } </span><br><span class="line"> virtual_ipaddress { #设置vip</span><br><span class="line"> 192.168.43.150 #可以多个虚拟IP，换行即可</span><br><span class="line"> } </span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="3-在Backup节点-etc-keepalived目录下添加-keepalived-conf-配置文件"><a href="#3-在Backup节点-etc-keepalived目录下添加-keepalived-conf-配置文件" class="headerlink" title="3.在Backup节点 etc/keepalived目录下添加 keepalived.conf 配置文件"></a>3.在Backup节点 etc/keepalived目录下添加 keepalived.conf 配置文件</h4><p>信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">vrrp_script chk_nginx { </span><br><span class="line"> script "/etc/keepalived/check_nginx.sh" //检测nginx进程的脚本 </span><br><span class="line"> interval 2 </span><br><span class="line"> weight -20 </span><br><span class="line">} </span><br><span class="line"></span><br><span class="line">global_defs { </span><br><span class="line"> notification_email { </span><br><span class="line"> //可以添加邮件提醒 </span><br><span class="line"> } </span><br><span class="line">} </span><br><span class="line">vrrp_instance VI_1 { </span><br><span class="line"> state BACKUP #标示状态为MASTER 备份机为BACKUP</span><br><span class="line"> interface ens33 #设置实例绑定的网卡(ip addr查看)</span><br><span class="line"> virtual_router_id 51 #同一实例下virtual_router_id必须相同 </span><br><span class="line"> mcast_src_ip 192.168.43.102 </span><br><span class="line"> priority 240 #MASTER权重要高于BACKUP 比如BACKUP为240 </span><br><span class="line"> advert_int 1 #MASTER与BACKUP负载均衡器之间同步检查的时间间隔，单位是秒</span><br><span class="line"> nopreempt #非抢占模式</span><br><span class="line"> authentication { #设置认证</span><br><span class="line"> auth_type PASS #主从服务器验证方式</span><br><span class="line"> auth_pass 123456 </span><br><span class="line"> } </span><br><span class="line"> track_script { </span><br><span class="line"> check_nginx </span><br><span class="line"> } </span><br><span class="line"> virtual_ipaddress { #设置vip</span><br><span class="line"> 192.168.43.150 #可以多个虚拟IP，换行即可</span><br><span class="line"> } </span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><strong>Tips</strong>:<font color="green"><strong>关于配置信息的几点说明</strong></font></p><ul><li>state - 主服务器需配成MASTER，从服务器需配成BACKUP </li><li>interface - 这个是网卡名，我使用的是VM12.0的版本，所以这里网卡名为ens33 </li><li>mcast_src_ip - 配置各自的实际IP地址 </li><li>priority - 主服务器的优先级必须比从服务器的高，这里主服务器配置成250，从服务器配置成240 </li><li>virtual_ipaddress - 配置虚拟IP（192.168.43.150） </li><li>authentication - auth_pass主从服务器必须一致，keepalived靠这个来通信 </li><li>virtual_router_id - 主从服务器必须保持一致<h2 id="5-集群高可用-HA-验证"><a href="#5-集群高可用-HA-验证" class="headerlink" title="5.集群高可用(HA)验证"></a>5.集群高可用(HA)验证</h2></li><li>Step1 启动Master机器的Keepalived和 Nginx服务</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# keepalived -D -f /etc/keepalived/keepalived.conf</span><br><span class="line">[root@localhost keepalived]# service nginx start</span><br></pre></td></tr></table></figure><p>查看服务启动进程<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# ps -aux|grep nginx</span><br><span class="line">root 6390 0.0 0.0 20484 612 ? Ss 19:13 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf</span><br><span class="line">nobody 6392 0.0 0.0 23008 1628 ? S 19:13 0:00 nginx: worker process</span><br><span class="line">root 6978 0.0 0.0 112672 968 pts/0 S+ 20:08 0:00 grep --color=auto nginx</span><br></pre></td></tr></table></figure></p><p>查看Keepalived启动进程</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# ps -aux|grep keepalived</span><br><span class="line">root 6402 0.0 0.0 45920 1016 ? Ss 19:13 0:00 keepalived -D -f /etc/keepalived/keepalived.conf</span><br><span class="line">root 6403 0.0 0.0 48044 1468 ? S 19:13 0:00 keepalived -D -f /etc/keepalived/keepalived.conf</span><br><span class="line">root 6404 0.0 0.0 50128 1780 ? S 19:13 0:00 keepalived -D -f /etc/keepalived/keepalived.conf</span><br><span class="line">root 7004 0.0 0.0 112672 976 pts/0 S+ 20:10 0:00 grep --color=auto keepalived</span><br></pre></td></tr></table></figure><p>使用 ip add 查看虚拟IP绑定情况，如出现192.168.43.150 节点信息则绑定到Master节点</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# ip add</span><br><span class="line">1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1</span><br><span class="line"> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</span><br><span class="line"> inet 127.0.0.1/8 scope host lo</span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line"> inet6 ::1/128 scope host </span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line">2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</span><br><span class="line"> link/ether 00:0c:29:91:bf:59 brd ff:ff:ff:ff:ff:ff</span><br><span class="line"> inet 192.168.43.101/24 brd 192.168.43.255 scope global ens33</span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line"> inet 192.168.43.150/32 scope global ens33</span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line"> inet6 fe80::9abb:4544:f6db:8255/64 scope link </span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line"> inet6 fe80::b0b3:d0ca:7382:2779/64 scope link tentative dadfailed </span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line"> inet6 fe80::314f:5fe7:4e4b:64ed/64 scope link tentative dadfailed </span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line">3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000</span><br><span class="line"> link/ether 52:54:00:2b:74:aa brd ff:ff:ff:ff:ff:ff</span><br><span class="line"> inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0</span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line">4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000</span><br><span class="line"> link/ether 52:54:00:2b:74:aa brd ff:ff:ff:ff:ff:ff</span><br></pre></td></tr></table></figure><ul><li>Step 2 启动Backup节点Nginx服务和Keepalived服务，查看服务启动情况，如Backup节点出现了虚拟IP，则Keepalvied配置文件有问题，此情况称为脑裂。</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# clear</span><br><span class="line">[root@localhost keepalived]# ip add</span><br><span class="line">1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1</span><br><span class="line"> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</span><br><span class="line"> inet 127.0.0.1/8 scope host lo</span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line"> inet6 ::1/128 scope host </span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line">2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</span><br><span class="line"> link/ether 00:0c:29:14:df:79 brd ff:ff:ff:ff:ff:ff</span><br><span class="line"> inet 192.168.43.102/24 brd 192.168.43.255 scope global ens33</span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line"> inet6 fe80::314f:5fe7:4e4b:64ed/64 scope link </span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line">3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000</span><br><span class="line"> link/ether 52:54:00:2b:74:aa brd ff:ff:ff:ff:ff:ff</span><br><span class="line"> inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0</span><br><span class="line"> valid_lft forever preferred_lft forever</span><br><span class="line">4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000</span><br><span class="line"> link/ether 52:54:00:2b:74:aa brd ff:ff:ff:ff:ff:ff</span><br></pre></td></tr></table></figure><ul><li>Step 3 验证服务<br><br>浏览并多次强制刷新地址： <a href="http://192.168.43.150" target="_blank" rel="noopener">http://192.168.43.150</a> ,可以看到103和104多次交替显示，并显示Nginx-1,则表明 Master节点在进行web服务转发。</li><li>Step 4 关闭Master keepalived服务和Nginx服务，访问Web服务观察服务转移情况</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost keepalived]# killall keepalived</span><br><span class="line">[root@localhost keepalived]# service nginx stop</span><br></pre></td></tr></table></figure><p>此时强制刷新192.168.43.150发现 页面交替显示103和104并显示Nginx-2 ，VIP已转移到192.168.43.102上，已证明服务自动切换到备份节点上。</p><ul><li>Step 5 启动Master Keepalived 服务和Nginx服务<br><br>此时再次验证发现，VIP已被Master重新夺回，并页面交替显示 103和104，此时显示Nginx-1<h1 id="四、Keepalived抢占模式和非抢占模式"><a href="#四、Keepalived抢占模式和非抢占模式" class="headerlink" title="四、Keepalived抢占模式和非抢占模式"></a>四、Keepalived抢占模式和非抢占模式</h1>keepalived的HA分为抢占模式和非抢占模式，抢占模式即MASTER从故障中恢复后，会将VIP从BACKUP节点中抢占过来。非抢占模式即MASTER恢复后不抢占BACKUP升级为MASTER后的VIP。 <br><br>非抢占模式配置：<br><ul><li>1> 在vrrp_instance块下两个节点各增加了nopreempt指令，表示不争抢vip </li><li>2> 节点的state都为BACKUP<br>两个keepalived节点都启动后，默认都是BACKUP状态，双方在发送组播信息后，会根据优先级来选举一个MASTER出来。由于两者都配置了nopreempt，所以MASTER从故障中恢复后，不会抢占vip。这样会避免VIP切换可能造成的服务延迟。</li></ul></li></ul>]]></content>
<summary type="html">
<figure class="image-bubble">
<div class="img-lightbox">
<div class="overlay"></div>
</summary>
</entry>
<entry>
<title>CentOS 7.2 安装Hive和Hive学习使用札记</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/CentOS_7.2_%E5%AE%89%E8%A3%85Hive%E5%92%8CHive%E5%AD%A6%E4%B9%A0%E4%BD%BF%E7%94%A8%E6%9C%AD%E8%AE%B0/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/CentOS_7.2_安装Hive和Hive学习使用札记/</id>
<published>2018-02-01T06:02:28.037Z</published>
<updated>2018-02-01T06:01:43.000Z</updated>
<content type="html"><![CDATA[<p>Hive由Facebook开源，是一个构建在Hadoop之上的数据仓库将结构化的数据映射成表支持类SQL查询，Hive中称为HQL无法实时更新，只支持向现有表中追加数据。<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-6ac187cbbbf83261.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Hive原理图.png" title=""> </div> <div class="image-caption">Hive原理图.png</div> </figure></p><h1 id="Hive常用文件格式类型"><a href="#Hive常用文件格式类型" class="headerlink" title="Hive常用文件格式类型"></a>Hive常用文件格式类型</h1><p><strong>TEXTFILE</strong></p><ul><li>默认文件格式，建表时用户需要显示指定分隔符</li><li>存储方式：行存储</li></ul><p><strong>SequenceFile</strong></p><ul><li>二进制键值对序列化文件格式</li><li>存储方式：行存储</li></ul><p><strong>列式存储格式</strong></p><ul><li>RCFILE/ORC</li><li>存储方式：列存储<h1 id="常用数据类型"><a href="#常用数据类型" class="headerlink" title="常用数据类型"></a>常用数据类型</h1><h4 id="1-整数类型"><a href="#1-整数类型" class="headerlink" title="1.整数类型"></a>1.整数类型</h4></li><li>SMALLINT、INT、BIGINT</li><li>空间占用分别是1字节、2字节、4字节、8字节<h4 id="2-浮点类型"><a href="#2-浮点类型" class="headerlink" title="2.浮点类型"></a>2.浮点类型</h4></li><li>DOUBLE</li><li>空间占用分别是32位和64位浮点数<h4 id="3-布尔类型BOOLEAN"><a href="#3-布尔类型BOOLEAN" class="headerlink" title="3. 布尔类型BOOLEAN"></a>3. 布尔类型BOOLEAN</h4></li><li>用于存储true和false<h4 id="4-字符串文本类型STRING"><a href="#4-字符串文本类型STRING" class="headerlink" title="4.字符串文本类型STRING"></a>4.字符串文本类型STRING</h4></li><li>存储变长字符串，对类型长度没有限制<h4 id="5-时间戳类型TIMESTAMP"><a href="#5-时间戳类型TIMESTAMP" class="headerlink" title="5.时间戳类型TIMESTAMP"></a>5.时间戳类型TIMESTAMP</h4></li><li>存储精度为纳秒的时间戳<h1 id="复杂数据类型"><a href="#复杂数据类型" class="headerlink" title="复杂数据类型"></a>复杂数据类型</h1><h4 id="1-ARRAY"><a href="#1-ARRAY" class="headerlink" title="1.ARRAY"></a>1.ARRAY</h4></li><li>存储相同类型的数据，可以通过下标获取数据</li><li>定义：ARRAY<string></string></li><li>查询：array[index]<h4 id="2-MAP"><a href="#2-MAP" class="headerlink" title="2.MAP"></a>2.MAP</h4></li><li>存储键值对数据，键或者值的类型必须相同，通过键获取值。</li><li>定义：MAP<string,int></string,int></li><li>查询：map[‘key’]<h4 id="3-STRUCT"><a href="#3-STRUCT" class="headerlink" title="3.STRUCT"></a>3.STRUCT</h4></li><li>可以存储多种不同的数据类型，一旦声明好结构，各字段的位置不能够改变。</li><li>定义：STRUCT<city:string, address="" :string,door_num:string=""></city:string,></li><li>查询：struct.fieldname<h1 id="一、Hive的安装"><a href="#一、Hive的安装" class="headerlink" title="一、Hive的安装"></a>一、Hive的安装</h1><h4 id="1-下载Hive安装包并解压"><a href="#1-下载Hive安装包并解压" class="headerlink" title="1.下载Hive安装包并解压"></a>1.下载Hive安装包并解压</h4></li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 apps]$ tar -zxvf apache-hive-1.2.2-bin.tar.gz</span><br></pre></td></tr></table></figure><h4 id="2-使用Root用户创建软链接"><a href="#2-使用Root用户创建软链接" class="headerlink" title="2.使用Root用户创建软链接"></a>2.使用Root用户创建软链接</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 apps]# ln -s /home/hadoop/apps/apache-hive-1.2.2-bin /usr/local/hive</span><br></pre></td></tr></table></figure><h4 id="3-为Hive指定用户组"><a href="#3-为Hive指定用户组" class="headerlink" title="3.为Hive指定用户组"></a>3.为Hive指定用户组</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 apps]# chown -R hadoop:hadoop /usr/local/hive</span><br></pre></td></tr></table></figure><h4 id="4-添加Hive到系统环境变量并生效"><a href="#4-添加Hive到系统环境变量并生效" class="headerlink" title="4. 添加Hive到系统环境变量并生效"></a>4. 添加Hive到系统环境变量并生效</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 apps]# vim /etc/profile</span><br></pre></td></tr></table></figure><p>添加环境变量内容为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">export HIVE_HOME=/usr/local/hive</span><br><span class="line">export PATH=$PATH:${JAVA_HOME}/bin:${ZOOKEEPER_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin</span><br><span class="line">:${HIVE_HOME}/bin</span><br></pre></td></tr></table></figure><p>生效环境变量</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 apps]# source /etc/profile</span><br></pre></td></tr></table></figure><h4 id="5-配置Hive的默认metastore"><a href="#5-配置Hive的默认metastore" class="headerlink" title="5.配置Hive的默认metastore"></a>5.配置Hive的默认metastore</h4><p>修改Hive配置目录下的hive-site.xml配置文件，编辑内容如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><?xml version="1.0" encoding="UTF-8" standalone="no"?></span><br><span class="line"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span><br><span class="line"><!--</span><br><span class="line"> Licensed to the Apache Software Foundation (ASF) under one or more</span><br><span class="line"> contributor license agreements. See the NOTICE file distributed with</span><br><span class="line"> this work for additional information regarding copyright ownership.</span><br><span class="line"> The ASF licenses this file to You under the Apache License, Version 2.0</span><br><span class="line"> (the "License"); you may not use this file except in compliance with</span><br><span class="line"> the License. You may obtain a copy of the License at</span><br><span class="line"></span><br><span class="line"> http://www.apache.org/licenses/LICENSE-2.0</span><br><span class="line"></span><br><span class="line"> Unless required by applicable law or agreed to in writing, software</span><br><span class="line"> distributed under the License is distributed on an "AS IS" BASIS,</span><br><span class="line"> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span><br><span class="line"> See the License for the specific language governing permissions and</span><br><span class="line"> limitations under the License.</span><br><span class="line">--></span><br><span class="line"><configuration></span><br><span class="line"><property></span><br><span class="line"><name>javax.jdo.option.ConnectionURL</name></span><br><span class="line"><value>jdbc:mysql://192.168.43.50:3306/hive?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8</value></span><br><span class="line"><description>JDBC connect string for a JDBC metastore</description></span><br><span class="line"></property></span><br><span class="line"><property></span><br><span class="line"><name>javax.jdo.option.ConnectionDriverName</name></span><br><span class="line"><value>com.mysql.jdbc.Driver</value></span><br><span class="line"><description>Driver class name for a JDBC metastore</description></span><br><span class="line"></property></span><br><span class="line"><property></span><br><span class="line"><name>javax.jdo.option.ConnectionUserName</name></span><br><span class="line"><value>hadoop</value></span><br><span class="line"><description>username to use against metastore database</description></span><br><span class="line"></property></span><br><span class="line"><property></span><br><span class="line"><name>javax.jdo.option.ConnectionPassword</name></span><br><span class="line"><value>xxxx</value></span><br><span class="line"><description>password to use against metastore database</description></span><br><span class="line"></property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><p><strong>说明：数据库连接地址为Mysql地址，且所配置用户具有外网访问数据库权限,ConnectionPassword配置成个人Mysql数据库用户密码</strong></p><h1 id="二、Hive的使用"><a href="#二、Hive的使用" class="headerlink" title="二、Hive的使用"></a>二、Hive的使用</h1><h4 id="1-运行hive"><a href="#1-运行hive" class="headerlink" title="1.运行hive"></a>1.运行hive</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 ~]$ hive</span><br><span class="line"></span><br><span class="line">Logging initialized using configuration in jar:file:/home/hadoop/apps/apache-hive-1.2.2-bin/lib/hive-common-1.2.2.jar!/hive-log4j.properties</span><br></pre></td></tr></table></figure><h4 id="2-查看数据库"><a href="#2-查看数据库" class="headerlink" title="2.查看数据库"></a>2.查看数据库</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hive> show databases;</span><br><span class="line">OK</span><br><span class="line">default</span><br><span class="line">Time taken: 0.99 seconds, Fetched: 1 row(s)</span><br></pre></td></tr></table></figure><h4 id="3-创建用户表：-user-info"><a href="#3-创建用户表：-user-info" class="headerlink" title="3.创建用户表： user_info"></a>3.创建用户表： user_info</h4><p>字段信息 </p><table><thead><tr><th>字段名称</th><th>字段类型</th></tr></thead><tbody><tr><td>用户id</td><td>string</td></tr><tr><td>地域id</td><td>string</td></tr><tr><td>年龄</td><td>int</td></tr><tr><td>职业</td><td>string</td></tr></tbody></table><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">create table user_info(</span><br><span class="line">user_id string,</span><br><span class="line">area_id string,</span><br><span class="line">age int,</span><br><span class="line">occupation string</span><br><span class="line">)</span><br><span class="line">row format delimited fields terminated by '\t' </span><br><span class="line">lines terminated by '\n'</span><br><span class="line">stored as textfile;</span><br></pre></td></tr></table></figure><h4 id="4-查看表"><a href="#4-查看表" class="headerlink" title="4.查看表"></a>4.查看表</h4><p>查看default库中的表，发现新建的user_info表在default库中</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hive> show tables;</span><br><span class="line">OK</span><br><span class="line">user_info</span><br><span class="line">Time taken: 0.04 seconds, Fetched: 1 row(s)</span><br></pre></td></tr></table></figure><p>查看对应文件目录信息</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ hadoop fs -ls /user/hive/warehouse</span><br><span class="line">Found 1 items</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-14 19:48 /user/hive/warehouse/user_info</span><br></pre></td></tr></table></figure><h4 id="5-hive删除表"><a href="#5-hive删除表" class="headerlink" title="5.hive删除表"></a>5.hive删除表</h4><p>删除user_info表，user_info表在hdfs的目录也会被同时删除</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">hive> drop table user_info;</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.935 seconds</span><br><span class="line">hive> show tables;</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.041 seconds</span><br></pre></td></tr></table></figure><p>查看文件目录位置</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ hadoop fs -ls /user/hive/warehouse</span><br><span class="line">[hadoop@hadoop01 root]$</span><br></pre></td></tr></table></figure><h4 id="6-创建数据库，用于存储维度"><a href="#6-创建数据库，用于存储维度" class="headerlink" title="6.创建数据库，用于存储维度"></a>6.创建数据库，用于存储维度</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">hive> create database rel;</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.098 seconds</span><br><span class="line">hive> show databases;</span><br><span class="line">OK</span><br><span class="line">default</span><br><span class="line">rel</span><br><span class="line">Time taken: 0.025 seconds, Fetched: 2 row(s)</span><br></pre></td></tr></table></figure><p>查看对应文件目录信息：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ hadoop fs -ls /user/hive/warehouse</span><br><span class="line">Found 1 items</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-14 19:55 /user/hive/warehouse/rel.db</span><br></pre></td></tr></table></figure><h4 id="7-创建内部管理表"><a href="#7-创建内部管理表" class="headerlink" title="7.创建内部管理表"></a>7.创建内部管理表</h4><p>在数据库rel中创建学生信息表，字段信息：学号、姓名、年龄、地域。切换使用rel数据库：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">use rel;</span><br><span class="line">create table student_info(</span><br><span class="line">student_id string comment '学号',</span><br><span class="line">name string comment '姓名',</span><br><span class="line">age int comment '年龄',</span><br><span class="line">origin string comment '地域'</span><br><span class="line">)</span><br><span class="line">comment '学生信息表'</span><br><span class="line">row format delimited </span><br><span class="line">fields terminated by '\t' </span><br><span class="line">lines terminated by '\n'</span><br><span class="line">stored as textfile;</span><br></pre></td></tr></table></figure><p>查看对应目录信息</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ hadoop fs -ls /user/hive/warehouse/rel.db</span><br><span class="line">Found 1 items</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-14 19:59 /user/hive/warehouse/rel.db/student_info</span><br></pre></td></tr></table></figure><h4 id="8-使用load从本地加载数据到表"><a href="#8-使用load从本地加载数据到表" class="headerlink" title="8.使用load从本地加载数据到表"></a>8.使用load从本地加载数据到表</h4><p>使用load从本地加载数据到表student_info</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">hive> load data local inpath '/home/hadoop/apps/hive_test_data/student_info_data.txt' into table student_info;</span><br><span class="line">Loading data to table rel.student_info</span><br><span class="line">Table rel.student_info stats: [numFiles=1, totalSize=341]</span><br><span class="line">OK</span><br><span class="line">Time taken: 1.144 seconds</span><br></pre></td></tr></table></figure><p>查看student_info 表信息和对应文件路径</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">hive> select * from student_info;</span><br><span class="line">OK</span><br><span class="line">1xiaoming2011</span><br><span class="line">2xiaobai2131</span><br><span class="line">3zhangfei2244</span><br><span class="line">4likui1944</span><br><span class="line">5zhaoyun2113</span><br><span class="line">6zhangsan2011</span><br><span class="line">7lisi1911</span><br><span class="line">8wangwu2331</span><br><span class="line">9zhaofei1921</span><br><span class="line">10zhangyan2021</span><br><span class="line">11lihe2022</span><br><span class="line">12caoyang1732</span><br><span class="line">13lihao1932</span><br><span class="line">14zhaoming2150</span><br><span class="line">15zhouhong1851</span><br><span class="line">16yangshuo2333</span><br><span class="line">17xiaofei2413</span><br><span class="line">18liman2313</span><br><span class="line">19qianbao2013</span><br><span class="line">20sunce2141</span><br><span class="line">Time taken: 0.767 seconds, Fetched: 20 row(s)</span><br></pre></td></tr></table></figure><p>查看对应文件夹路径信息</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 hive_test_data]$ hadoop fs -ls /user/hive/warehouse/rel.db/student_info</span><br><span class="line">Found 1 items</span><br><span class="line">-rwxr-xr-x 3 hadoop supergroup 341 2018-01-14 20:09 /user/hive/warehouse/rel.db/student_info/student_info_data.txt</span><br></pre></td></tr></table></figure><h4 id="9-使用load从HDFS上加载数据到表student-info"><a href="#9-使用load从HDFS上加载数据到表student-info" class="headerlink" title="9.使用load从HDFS上加载数据到表student_info"></a>9.使用load从HDFS上加载数据到表student_info</h4><p>先删除原有数据文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 hive_test_data]$ hadoop fs -rm -f /user/hive/warehouse/rel.db/student_info/student_info_data.txt</span><br><span class="line">18/01/14 20:15:31 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.</span><br><span class="line">Deleted /user/hive/warehouse/rel.db/student_info/student_info_data.txt</span><br></pre></td></tr></table></figure><p>将本地文件上传到HDFS根目录下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 hive_test_data]$ hadoop fs -put /home/hadoop/apps/hive_test_data/student_info_data.txt /</span><br><span class="line">[hadoop@hadoop01 hive_test_data]$ hadoop fs -ls /</span><br><span class="line">Found 6 items</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-14 16:23 /addata</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2017-12-23 20:20 /data</span><br><span class="line">-rw-r--r-- 3 hadoop supergroup 341 2018-01-14 20:16 /student_info_data.txt</span><br><span class="line">drwxrwx--- - hadoop supergroup 0 2018-01-14 17:26 /tmp</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-14 19:48 /user</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-13 16:26 /wordcount</span><br></pre></td></tr></table></figure><p>使用load将HDFS文件加载到student_info 表中</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line">hive> load data inpath '/student_info_data.txt' into table student_info;</span><br><span class="line">Loading data to table rel.student_info</span><br><span class="line">Table rel.student_info stats: [numFiles=1, totalSize=341]</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.602 seconds</span><br><span class="line">hive> select * from student_info;</span><br><span class="line">OK</span><br><span class="line">1xiaoming2011</span><br><span class="line">2xiaobai2131</span><br><span class="line">3zhangfei2244</span><br><span class="line">4likui1944</span><br><span class="line">5zhaoyun2113</span><br><span class="line">6zhangsan2011</span><br><span class="line">7lisi1911</span><br><span class="line">8wangwu2331</span><br><span class="line">9zhaofei1921</span><br><span class="line">10zhangyan2021</span><br><span class="line">11lihe2022</span><br><span class="line">12caoyang1732</span><br><span class="line">13lihao1932</span><br><span class="line">14zhaoming2150</span><br><span class="line">15zhouhong1851</span><br><span class="line">16yangshuo2333</span><br><span class="line">17xiaofei2413</span><br><span class="line">18liman2313</span><br><span class="line">19qianbao2013</span><br><span class="line">20sunce2141</span><br><span class="line">Time taken: 0.143 seconds, Fetched: 20 row(s)</span><br></pre></td></tr></table></figure><p>采用覆盖重写方式加载文件到student_info 表中</p><ul><li>原hdfs根目录下的student_info_data.txt已经被剪切到student_info表的hdfs路径下/user/hive/warehouse/rel.db/student_info<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">hive> load data inpath '/student_info_data.txt' overwrite into table student_info;</span><br><span class="line">Loading data to table rel.student_info</span><br><span class="line">Table rel.student_info stats: [numFiles=1, numRows=0, totalSize=341, rawDataSize=0]</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.41 seconds</span><br></pre></td></tr></table></figure></li></ul><h4 id="10-Hive的数据类型"><a href="#10-Hive的数据类型" class="headerlink" title="10.Hive的数据类型"></a>10.Hive的数据类型</h4><table><thead><tr><th style="text-align:left">字段名</th><th style="text-align:left">类型</th><th style="text-align:left">注释</th></tr></thead><tbody><tr><td style="text-align:left">user_id</td><td style="text-align:left">string</td><td style="text-align:left">用户ID</td></tr><tr><td style="text-align:left">salary</td><td style="text-align:left">int</td><td style="text-align:left">工资</td></tr><tr><td style="text-align:left">worked_citys</td><td style="text-align:left">array<string></string></td><td style="text-align:left">工作过的城市</td></tr><tr><td style="text-align:left">social_security</td><td style="text-align:left">map<string,float></string,float></td><td style="text-align:left">社保缴费情况(养老，医保)</td></tr><tr><td style="text-align:left">wealfare</td><td style="text-align:left">struct<meal_allowance:float,if_regular:boolean,commerical_insurance:float></meal_allowance:float,if_regular:boolean,commerical_insurance:float></td><td style="text-align:left">福利(吃饭补助(float),是否转正(boolean),商业保险(float)</td></tr></tbody></table><p>创建员工表</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">hive> create table rel.employee(</span><br><span class="line"> > user_id string,</span><br><span class="line"> > salary int,</span><br><span class="line"> > worked_citys array<string>,</span><br><span class="line"> > social_security map<string,float>,</span><br><span class="line"> > welfare struct<meal_allowance:float,if_regular:boolean,commercial_insurance:float></span><br><span class="line"> > )</span><br><span class="line"> > row format delimited fields terminated by '\t' </span><br><span class="line"> > collection items terminated by ','</span><br><span class="line"> > map keys terminated by ':'</span><br><span class="line"> > lines terminated by '\n'</span><br><span class="line"> > stored as textfile;</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.212 seconds</span><br><span class="line">hive> show tables;</span><br><span class="line">OK</span><br><span class="line">employee</span><br><span class="line">student_info</span><br><span class="line">Time taken: 0.057 seconds, Fetched: 2 row(s)</span><br></pre></td></tr></table></figure><p>从本地加载数据到表employee</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">hive> load data local inpath '/home/hadoop/apps/hive_test_data/employee_data.txt' into table employee;</span><br><span class="line">Loading data to table rel.employee</span><br><span class="line">Table rel.employee stats: [numFiles=1, totalSize=206]</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.388 seconds</span><br><span class="line">hive> select * from employee;</span><br><span class="line">OK</span><br><span class="line">zhangsan10800["beijing","shanghai"]{"养老":1000.0,"医疗":600.0}{"meal_allowance":2000.0,"if_regular":true,"commercial_insurance":500.0}</span><br><span class="line">lisi20000["beijing","nanjing"]{"养老":2000.0,"医疗":1200.0}{"meal_allowance":2000.0,"if_regular":false,"commercial_insurance":500.0}</span><br><span class="line">wangwu17000["shanghai","nanjing"]{"养老":1800.0,"医疗":1100.0}{"meal_allowance":2000.0,"if_regular":true,"commercial_insurance":500.0}</span><br><span class="line">Time taken: 0.127 seconds, Fetched: 3 row(s)</span><br></pre></td></tr></table></figure><p>查询已转正的员工编号，工资，工作过的第一个城市，社保养老缴费情况，福利餐补金额</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">select user_id,</span><br><span class="line">salary,</span><br><span class="line">worked_citys[0],</span><br><span class="line">social_security['养老'],</span><br><span class="line">welfare.meal_allowance </span><br><span class="line">from rel.employee</span><br><span class="line">where welfare.if_regular=true;</span><br></pre></td></tr></table></figure><h4 id="11-创建外部表-【常用】"><a href="#11-创建外部表-【常用】" class="headerlink" title="11.创建外部表 【常用】"></a>11.创建外部表 【常用】</h4><p>可以提前创建好hdfs路径<br>hadoop mkdir -p /user/hive/warehouse/data/student_school_info<br>如果没有提前创建好，在创建外部表的时候会根据指定路径自动创建<br>|字段名|字段类型|字段注释|<br>|:—-:|:—-:|:—-:|<br>|student_id|string|学生ID|<br>|name|string|学生姓名|<br>|institute_id|string|学院ID|<br>|major_id|string|专业ID|<br>|school_year|string|入学年份|</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">create external table rel.student_school_info(</span><br><span class="line">student_id string,</span><br><span class="line">name string,</span><br><span class="line">institute_id string,</span><br><span class="line">major_id string,</span><br><span class="line">school_year string</span><br><span class="line">)</span><br><span class="line">row format delimited </span><br><span class="line">fields terminated by '\t' </span><br><span class="line">lines terminated by '\n'</span><br><span class="line">stored as textfile</span><br><span class="line">location '/user/hive/warehouse/data/student_school_info';</span><br></pre></td></tr></table></figure><p>查看对应文件目录</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ hadoop fs -ls /user/hive/warehouse/data/</span><br><span class="line">Found 1 items</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-15 14:08 /user/hive/warehouse/data/student_school_info</span><br></pre></td></tr></table></figure><p>上传本地数据文件到hdfs</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ hadoop fs -put /home/hadoop/apps/hive_test_data/student_school_info_external_data.txt /user/hive/warehouse/data/student_school_info/</span><br></pre></td></tr></table></figure><h4 id="12-创建内部分区表"><a href="#12-创建内部分区表" class="headerlink" title="12.创建内部分区表"></a>12.创建内部分区表</h4><table><thead><tr><th style="text-align:center">字段名称</th><th style="text-align:center">类型</th><th style="text-align:center">注释</th></tr></thead><tbody><tr><td style="text-align:center">studnet_id</td><td style="text-align:center">string</td><td style="text-align:center">学号</td></tr><tr><td style="text-align:center">name</td><td style="text-align:center">string</td><td style="text-align:center">姓名</td></tr><tr><td style="text-align:center">institute_id</td><td style="text-align:center">string</td><td style="text-align:center">学院ID</td></tr><tr><td style="text-align:center">major_id</td><td style="text-align:center">string</td><td style="text-align:center">专业ID</td></tr></tbody></table><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">create table student_school_info_partition(</span><br><span class="line">student_id string,</span><br><span class="line">name string,</span><br><span class="line">institute_id string,</span><br><span class="line">major_id string</span><br><span class="line">)</span><br><span class="line">partitioned by(school_year string) </span><br><span class="line">row format delimited</span><br><span class="line">fields terminated by '\t' </span><br><span class="line">lines terminated by '\n'</span><br><span class="line">stored as textfile;</span><br></pre></td></tr></table></figure><p> 使用insert into从student_school_info表将2017年入学的学籍信息导入到student_school_info_partition分区表中</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">insert into table student_school_info_partition partition(school_year='2017')</span><br><span class="line">select t1.student_id,t1.name,t1.institute_id,t1.major_id</span><br><span class="line">from student_school_info t1</span><br><span class="line">where t1.school_year=2017;</span><br></pre></td></tr></table></figure><h4 id="13-查看分区"><a href="#13-查看分区" class="headerlink" title="13.查看分区"></a>13.查看分区</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hive> show partitions student_school_info_partition;</span><br><span class="line">OK</span><br><span class="line">school_year=2017</span><br><span class="line">Time taken: 0.191 seconds, Fetched: 1 row(s)</span><br></pre></td></tr></table></figure><p>查看hdfs路径</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ hadoop fs -ls /user/hive/warehouse/rel.db/student_school_info_partition/</span><br><span class="line">Found 1 items</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-16 13:20 /user/hive/warehouse/rel.db/student_school_info_partition/school_year=2017</span><br></pre></td></tr></table></figure><p>查询student_school_info_partition</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">hive> select * from student_school_info_partition where school_year='2017';</span><br><span class="line">OK</span><br><span class="line">1xiaominginformationsoftware2017</span><br><span class="line">2xiaobaiinformationcomputer2017</span><br><span class="line">3zhangfeiinformationcomputer2017</span><br><span class="line">4likuiinformationbigdata2017</span><br><span class="line">5zhaoyuninformationbigdata2017</span><br><span class="line">6zhangsaninformationsoftware2017</span><br><span class="line">7lisiinformationbigdata2017</span><br><span class="line">8wangwuinformationcomputer2017</span><br><span class="line">Time taken: 0.226 seconds, Fetched: 8 row(s)</span><br></pre></td></tr></table></figure><h4 id="14-删除分区"><a href="#14-删除分区" class="headerlink" title="14.删除分区"></a>14.删除分区</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hive> alter table student_school_info_partition drop partition (school_year='2017');</span><br><span class="line">Dropped the partition school_year=2017</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.71 seconds</span><br></pre></td></tr></table></figure><h4 id="15-使用动态分区添加数据"><a href="#15-使用动态分区添加数据" class="headerlink" title="15.使用动态分区添加数据"></a>15.使用动态分区添加数据</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">set hive.exec.dynamic.partition=true;</span><br><span class="line">set hive.exec.dynamic.partition.mode=nonstrict;</span><br><span class="line">insert overwrite table student_school_info_partition partition(school_year)</span><br><span class="line">select t1.student_id,t1.name,t1.institute_id,t1.major_id,t1.school_year</span><br><span class="line">from student_school_info t1</span><br></pre></td></tr></table></figure><p>查看分区</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hive> show partitions student_school_info_partition;</span><br><span class="line">OK</span><br><span class="line">school_year=2017</span><br><span class="line">Time taken: 0.12 seconds, Fetched: 1 row(s)</span><br></pre></td></tr></table></figure><p>查看hdfs路径</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 root]$ hadoop fs -ls /user/hive/warehouse/rel.db/student_school_info_partition/</span><br><span class="line">Found 1 items</span><br><span class="line">drwxr-xr-x - hadoop supergroup 0 2018-01-16 13:27 /user/hive/warehouse/rel.db/student_school_info_partition/school_year=2017</span><br></pre></td></tr></table></figure><h4 id="15-创建外部分区表【常用】"><a href="#15-创建外部分区表【常用】" class="headerlink" title="15.创建外部分区表【常用】"></a>15.创建外部分区表【常用】</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">create external table rel.student_school_info_external_partition(</span><br><span class="line">student_id string,</span><br><span class="line">name string,</span><br><span class="line">institute_id string,</span><br><span class="line">major_id string</span><br><span class="line">)</span><br><span class="line">partitioned by(school_year string) </span><br><span class="line">row format delimited </span><br><span class="line">fields terminated by '\t' </span><br><span class="line">lines terminated by '\n'</span><br><span class="line">stored as textfile</span><br><span class="line">location '/user/hive/warehouse/data/student_school_info_external_partition';</span><br></pre></td></tr></table></figure><p>在分区表的hdfs路径中添加school_year=2017目录</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hadoop fs -mkdir /user/hive/warehouse/data/student_school_info_external_partition/school_year=2017</span><br></pre></td></tr></table></figure><p>将student_school_external_partition_data.txt文件上传到school_year=2017文件夹下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hadoop fs -put student_school_external_partition_data.txt /user/hive/warehouse/data/student_school_info_external_partition/school_year=2017</span><br></pre></td></tr></table></figure><p>虽然数据已经添加到了分区对应的hdfs路径，但是表还没有添加分区，所以查询的时候没有数据</p><p>手动添加分区</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">hive> alter table student_school_info_external_partition add partition(school_year='2017');</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.111 seconds</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">hive> select * from student_school_info_external_partition;</span><br><span class="line">OK</span><br><span class="line">1xiaominginformationsoftware2017</span><br><span class="line">2xiaobaiinformationcomputer2017</span><br><span class="line">3zhangfeiinformationcomputer2017</span><br><span class="line">4likuiinformationbigdata2017</span><br><span class="line">5zhaoyuninformationbigdata2017</span><br><span class="line">6zhangsaninformationsoftware2017</span><br><span class="line">7lisiinformationbigdata2017</span><br><span class="line">8wangwuinformationcomputer2017</span><br><span class="line">Time taken: 0.127 seconds, Fetched: 8 row(s)</span><br></pre></td></tr></table></figure><p>删除分区</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hive> alter table student_school_info_external_partition drop partition(school_year='2017');</span><br><span class="line">Dropped the partition school_year=2017</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.19 seconds</span><br></pre></td></tr></table></figure><p>查看分区，分区已经被删除</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">hive> show partitions student_school_info_external_partition;</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.168 seconds</span><br></pre></td></tr></table></figure><p>查看hdfs分区数据，分区数据还在</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 hive_test_data]$ hadoop fs -ls /user/hive/warehouse/data/student_school_info_external_partition/school_year=2017</span><br><span class="line">Found 1 items</span><br><span class="line">-rw-r--r-- 3 hadoop supergroup 250 2018-01-16 13:33 /user/hive/warehouse/data/student_school_info_external_partition/school_year=2017/student_school_external_partition_data.txt</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p>Hive由Facebook开源，是一个构建在Hadoop之上的数据仓库将结构化的数据映射成表支持类SQL查询，Hive中称为HQL无法实时更新，只支持向现有表中追加数据。<br><figure class="image-bubble">
<d
</summary>
</entry>
<entry>
<title>Centos7.2安装MariaDB</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/Centos7.2%E5%AE%89%E8%A3%85MariaDB/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/Centos7.2安装MariaDB/</id>
<published>2018-02-01T06:01:37.203Z</published>
<updated>2018-02-01T06:00:59.000Z</updated>
<content type="html"><![CDATA[<h1 id="1-检查是否已经具有MariaDB相关安装，并删除已有安装"><a href="#1-检查是否已经具有MariaDB相关安装，并删除已有安装" class="headerlink" title="1.检查是否已经具有MariaDB相关安装，并删除已有安装"></a>1.检查是否已经具有MariaDB相关安装，并删除已有安装</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]# rpm -qa|grep mariadb #查询已安装包</span><br><span class="line">mariadb-libs-5.5.52-1.el7.x86_64</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]# rpm -e --nodeps mariadb-* # 移除已安装包</span><br><span class="line">错误：未安装软件包 mariadb-*</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]# yum remove mysql mysql-server mysql-libs compat-mysql51 # 删除Mysql服务</span><br><span class="line">已加载插件：fastestmirror, langpacks</span><br><span class="line">参数 mysql 没有匹配</span><br><span class="line">参数 mysql-server 没有匹配</span><br><span class="line">参数 compat-mysql51 没有匹配</span><br><span class="line">正在解决依赖关系</span><br><span class="line">--> 正在检查事务</span><br><span class="line">---> 软件包 mariadb-libs.x86_64.1.5.5.52-1.el7 将被 删除</span><br><span class="line">--> 正在处理依赖关系 libmysqlclient.so.18()(64bit)，它被软件包 2:postfix-2.10.1-6.el7.x86_64 需要</span><br><span class="line">--> 正在处理依赖关系 libmysqlclient.so.18(libmysqlclient_18)(64bit)，它被软件包 2:postfix-2.10.1-6.el7.x86_64 需要</span><br><span class="line">--> 正在检查事务</span><br><span class="line">---> 软件包 postfix.x86_64.2.2.10.1-6.el7 将被 删除</span><br><span class="line"> > 解决依赖关系完成</span><br><span class="line">▽ase/7/x86_64 | 3.6 kB 00:00:00 </span><br><span class="line">extras/7/x86_64 | 3.4 kB 00:00:00 </span><br><span class="line">updates/7/x86_64 | 3.4 kB 00:00:00 </span><br><span class="line"></span><br><span class="line">依赖关系解决</span><br><span class="line"></span><br><span class="line">============================================================================================================================================================================================================================================</span><br><span class="line"> Package 架构 版本 源 大小</span><br><span class="line">============================================================================================================================================================================================================================================</span><br><span class="line">正在删除:</span><br><span class="line"> mariadb-libs x86_64 1:5.5.52-1.el7 @anaconda 4.4 M</span><br><span class="line">为依赖而移除:</span><br><span class="line"> postfix x86_64 2:2.10.1-6.el7 @anaconda 12 M</span><br><span class="line"></span><br><span class="line">事务概要</span><br><span class="line">============================================================================================================================================================================================================================================</span><br><span class="line">移除 1 软件包 (+1 依赖软件包)</span><br><span class="line"></span><br><span class="line">安装大小：17 M</span><br><span class="line">是否继续？[y/N]：y</span><br><span class="line">Downloading packages:</span><br><span class="line">Running transaction check</span><br><span class="line">Running transaction test</span><br><span class="line">Transaction test succeeded</span><br><span class="line">Running transaction</span><br><span class="line"> 正在删除 : 2:postfix-2.10.1-6.el7.x86_64 1/2 </span><br><span class="line"> 正在删除 : 1:mariadb-libs-5.5.52-1.el7.x86_64 2/2 </span><br><span class="line"> 验证中 : 2:postfix-2.10.1-6.el7.x86_64 1/2 </span><br><span class="line"> 验证中 : 1:mariadb-libs-5.5.52-1.el7.x86_64 2/2 </span><br><span class="line"></span><br><span class="line">删除:</span><br><span class="line"> mariadb-libs.x86_64 1:5.5.52-1.el7 </span><br><span class="line"></span><br><span class="line">作为依赖被删除:</span><br><span class="line"> postfix.x86_64 2:2.10.1-6.el7 </span><br><span class="line"></span><br><span class="line">完毕！</span><br></pre></td></tr></table></figure><h1 id="2-增加MariaDB的仓库源"><a href="#2-增加MariaDB的仓库源" class="headerlink" title="2.增加MariaDB的仓库源"></a>2.增加MariaDB的仓库源</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]#vi /etc/yum.repos.d/MariaDB.repo 增加MariaDB的数据库镜像信息</span><br><span class="line"># MariaDB 10.2 CentOS repository list - created 2017-12-26 06:46 UTC</span><br><span class="line"># http://downloads.mariadb.org/mariadb/repositories/</span><br><span class="line">[mariadb]</span><br><span class="line">name = MariaDB</span><br><span class="line">baseurl = http://yum.mariadb.org/10.2/centos7-amd64</span><br><span class="line">gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB</span><br><span class="line">gpgcheck=1</span><br></pre></td></tr></table></figure><p>文件信息请参考[<a href="https://downloads.mariadb.org/mariadb/repositories/#mirror=tuna" target="_blank" rel="noopener">https://downloads.mariadb.org/mariadb/repositories/#mirror=tuna</a>] 设置具体需要安装的版本，本文下载MariaDB10.2稳定版。</p><h1 id="3-安装MariaDB"><a href="#3-安装MariaDB" class="headerlink" title="3.安装MariaDB"></a>3.安装MariaDB</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost ~]# yum -y install MariaDB-server MariaDB-client</span><br><span class="line">已加载插件：fastestmirror, langpacks</span><br><span class="line">mariadb | 2.9 kB 00:00:00 </span><br><span class="line">mariadb/primary_db | 21 kB 00:00:01 </span><br><span class="line">Loading mirror speeds from cached hostfile</span><br><span class="line"> * base: mirrors.aliyun.com</span><br><span class="line"> * extras: mirrors.sohu.com</span><br><span class="line"> * updates: mirrors.aliyun.com</span><br><span class="line">正在解决依赖关系</span><br><span class="line">--> 正在检查事务</span><br><span class="line">---> 软件包 MariaDB-client.x86_64.0.10.2.11-1.el7.centos 将被 安装</span><br><span class="line">--> 正在处理依赖关系 MariaDB-common，它被软件包 MariaDB-client-10.2.11-1.el7.centos.x86_64 需要</span><br><span class="line">---> 软件包 MariaDB-server.x86_64.0.10.2.11-1.el7.centos 将被 安装</span><br><span class="line">--> 正在处理依赖关系 perl(DBI)，它被软件包 MariaDB-server-10.2.11-1.el7.centos.x86_64 需要</span><br><span class="line">--> 正在处理依赖关系 galera，它被软件包 MariaDB-server-10.2.11-1.el7.centos.x86_64 需要</span><br><span class="line">--> 正在检查事务</span><br><span class="line">---> 软件包 MariaDB-common.x86_64.0.10.2.11-1.el7.centos 将被 安装</span><br><span class="line">--> 正在处理依赖关系 MariaDB-compat，它被软件包 MariaDB-common-10.2.11-1.el7.centos.x86_64 需要</span><br><span class="line">---> 软件包 galera.x86_64.0.25.3.22-1.rhel7.el7.centos 将被 安装</span><br><span class="line">--> 正在处理依赖关系 libboost_program_options.so.1.53.0()(64bit)，它被软件包 galera-25.3.22-1.rhel7.el7.centos.x86_64 需要</span><br><span class="line">---> 软件包 perl-DBI.x86_64.0.1.627-4.el7 将被 安装</span><br><span class="line">--> 正在处理依赖关系 perl(RPC::PlServer) >= 0.2001，它被软件包 perl-DBI-1.627-4.el7.x86_64 需要</span><br><span class="line">--> 正在处理依赖关系 perl(RPC::PlClient) >= 0.2000，它被软件包 perl-DBI-1.627-4.el7.x86_64 需要</span><br><span class="line">--> 正在检查事务</span><br><span class="line">---> 软件包 MariaDB-compat.x86_64.0.10.2.11-1.el7.centos 将被 安装</span><br><span class="line">---> 软件包 boost-program-options.x86_64.0.1.53.0-27.el7 将被 安装</span><br><span class="line">---> 软件包 perl-PlRPC.noarch.0.0.2020-14.el7 将被 安装</span><br><span class="line">--> 正在处理依赖关系 perl(Net::Daemon) >= 0.13，它被软件包 perl-PlRPC-0.2020-14.el7.noarch 需要</span><br><span class="line">--> 正在处理依赖关系 perl(Net::Daemon::Test)，它被软件包 perl-PlRPC-0.2020-14.el7.noarch 需要</span><br><span class="line">--> 正在处理依赖关系 perl(Net::Daemon::Log)，它被软件包 perl-PlRPC-0.2020-14.el7.noarch 需要</span><br><span class="line">--> 正在处理依赖关系 perl(Compress::Zlib)，它被软件包 perl-PlRPC-0.2020-14.el7.noarch 需要</span><br><span class="line">--> 正在检查事务</span><br><span class="line">---> 软件包 perl-IO-Compress.noarch.0.2.061-2.el7 将被 安装</span><br><span class="line">--> 正在处理依赖关系 perl(Compress::Raw::Zlib) >= 2.061，它被软件包 perl-IO-Compress-2.061-2.el7.noarch 需要</span><br><span class="line">--> 正在处理依赖关系 perl(Compress::Raw::Bzip2) >= 2.061，它被软件包 perl-IO-Compress-2.061-2.el7.noarch 需要</span><br><span class="line">---> 软件包 perl-Net-Daemon.noarch.0.0.48-5.el7 将被 安装</span><br><span class="line">--> 正在检查事务</span><br><span class="line">---> 软件包 perl-Compress-Raw-Bzip2.x86_64.0.2.061-3.el7 将被 安装</span><br><span class="line">---> 软件包 perl-Compress-Raw-Zlib.x86_64.1.2.061-4.el7 将被 安装</span><br><span class="line">--> 解决依赖关系完成</span><br><span class="line"></span><br><span class="line">依赖关系解决</span><br><span class="line"></span><br><span class="line">============================================================================================================================================================================================================================================</span><br><span class="line"> Package 架构 版本 源 大小</span><br><span class="line">============================================================================================================================================================================================================================================</span><br><span class="line">正在安装:</span><br><span class="line"> MariaDB-client x86_64 10.2.11-1.el7.centos mariadb 48 M</span><br><span class="line"> MariaDB-server x86_64 10.2.11-1.el7.centos mariadb 110 M</span><br><span class="line">为依赖而安装:</span><br><span class="line"> MariaDB-common x86_64 10.2.11-1.el7.centos mariadb 154 k</span><br><span class="line"> MariaDB-compat x86_64 10.2.11-1.el7.centos mariadb 2.8 M</span><br><span class="line"> boost-program-options x86_64 1.53.0-27.el7 base 156 k</span><br><span class="line"> galera x86_64 25.3.22-1.rhel7.el7.centos mariadb 8.0 M</span><br><span class="line"> perl-Compress-Raw-Bzip2 x86_64 2.061-3.el7 base 32 k</span><br><span class="line"> perl-Compress-Raw-Zlib x86_64 1:2.061-4.el7 base 57 k</span><br><span class="line"> perl-DBI x86_64 1.627-4.el7 base 802 k</span><br><span class="line"> perl-IO-Compress noarch 2.061-2.el7 base 260 k</span><br><span class="line"> perl-Net-Daemon noarch 0.48-5.el7 base 51 k</span><br><span class="line"> perl-PlRPC noarch 0.2020-14.el7 base 36 k</span><br><span class="line"></span><br><span class="line">事务概要</span><br><span class="line">============================================================================================================================================================================================================================================</span><br></pre></td></tr></table></figure><p>等待下载完成,下载完成后会自动安装。</p><h1 id="4-MariaDB服务管理"><a href="#4-MariaDB服务管理" class="headerlink" title="4.MariaDB服务管理"></a>4.MariaDB服务管理</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]# systemctl start mariadb # 开启数据库服务</span><br><span class="line">[root@hadoop01 home]# systemctl enable mariadb # 开机自启动</span><br><span class="line">[root@hadoop01 home]# systemctl restart mariadb # 重启服务 </span><br><span class="line">[root@hadoop01 home]# systemctl status mariadb #查看数据库状态</span><br><span class="line">[root@hadoop01 home]# systemctl stop mariadb.service # 停止数据库服务</span><br></pre></td></tr></table></figure><h1 id="5-数据库登录"><a href="#5-数据库登录" class="headerlink" title="5.数据库登录"></a>5.数据库登录</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]# mysql -uroot</span><br><span class="line">Welcome to the MariaDB monitor. Commands end with ; or \g.</span><br><span class="line">Your MariaDB connection id is 8</span><br><span class="line">Server version: 10.2.11-MariaDB MariaDB Server</span><br><span class="line"></span><br><span class="line">Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.</span><br><span class="line"></span><br><span class="line">Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.</span><br><span class="line"></span><br><span class="line">MariaDB [(none)]></span><br></pre></td></tr></table></figure><h1 id="6-设置数据库密码"><a href="#6-设置数据库密码" class="headerlink" title="6.设置数据库密码"></a>6.设置数据库密码</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]# mysql_secure_installation #初始化密码</span><br><span class="line"></span><br><span class="line">NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB</span><br><span class="line"> SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!</span><br><span class="line"></span><br><span class="line">In order to log into MariaDB to secure it, we'll need the current</span><br><span class="line">password for the root user. If you've just installed MariaDB, and</span><br><span class="line">you haven't set the root password yet, the password will be blank,</span><br><span class="line">so you should just press enter here.</span><br><span class="line"></span><br><span class="line">Enter current password for root (enter for none): #输入当前密码，一般没设置直接回车</span><br><span class="line">ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)</span><br><span class="line">Enter current password for root (enter for none): </span><br><span class="line">OK, successfully used password, moving on...</span><br><span class="line"></span><br><span class="line">Setting the root password ensures that nobody can log into the MariaDB</span><br><span class="line">root user without the proper authorisation.</span><br><span class="line"></span><br><span class="line">Set root password? [Y/n] y #是否设置root密码，输入y</span><br><span class="line">New password: #输入新密码</span><br><span class="line">Re-enter new password: #重复密码 </span><br><span class="line">Password updated successfully!</span><br><span class="line">Reloading privilege tables..</span><br><span class="line"> ... Success!</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">By default, a MariaDB installation has an anonymous user, allowing anyone</span><br><span class="line">to log into MariaDB without having to have a user account created for</span><br><span class="line">them. This is intended only for testing, and to make the installation</span><br><span class="line">go a bit smoother. You should remove them before moving into a</span><br><span class="line">production environment.</span><br><span class="line"></span><br><span class="line">Remove anonymous users? [Y/n] #删除匿名用户 y</span><br><span class="line"> ... Success!</span><br><span class="line"></span><br><span class="line">Normally, root should only be allowed to connect from 'localhost'. This</span><br><span class="line">ensures that someone cannot guess at the root password from the network.</span><br><span class="line"></span><br><span class="line">Disallow root login remotely? [Y/n] #禁止root远程登录</span><br><span class="line"> ... Success!</span><br><span class="line"></span><br><span class="line">By default, MariaDB comes with a database named 'test' that anyone can</span><br><span class="line">access. This is also intended only for testing, and should be removed</span><br><span class="line">before moving into a production environment.</span><br><span class="line"></span><br><span class="line">Remove test database and access to it? [Y/n] #是否删除test数据库</span><br><span class="line"> - Dropping test database...</span><br><span class="line"> ... Success!</span><br><span class="line"> - Removing privileges on test database...</span><br><span class="line"> ... Success!</span><br><span class="line"></span><br><span class="line">Reloading the privilege tables will ensure that all changes made so far</span><br><span class="line">will take effect immediately.</span><br><span class="line"></span><br><span class="line">Reload privilege tables now? [Y/n] #是否重新加载权限表</span><br><span class="line"> ... Success!</span><br><span class="line"></span><br><span class="line">Cleaning up...</span><br><span class="line"></span><br><span class="line">All done! If you've completed all of the above steps, your MariaDB</span><br><span class="line">installation should now be secure.</span><br><span class="line"></span><br><span class="line">Thanks for using MariaDB!</span><br></pre></td></tr></table></figure><h2 id="测试登录"><a href="#测试登录" class="headerlink" title="测试登录"></a>测试登录</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]# mysql -uroot -p</span><br><span class="line">Enter password: </span><br><span class="line">Welcome to the MariaDB monitor. Commands end with ; or \g.</span><br><span class="line">Your MariaDB connection id is 22</span><br><span class="line">Server version: 10.2.11-MariaDB MariaDB Server</span><br><span class="line"></span><br><span class="line">Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.</span><br><span class="line"></span><br><span class="line">Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.</span><br><span class="line"></span><br><span class="line">MariaDB [(none)]></span><br></pre></td></tr></table></figure><h1 id="7-配置MariaDB数据库字符集"><a href="#7-配置MariaDB数据库字符集" class="headerlink" title="7.配置MariaDB数据库字符集"></a>7.配置MariaDB数据库字符集</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 home]# cd /etc/my.cnf.d/</span><br><span class="line">[root@hadoop01 my.cnf.d]# ls</span><br><span class="line">enable_encryption.preset mysql-clients.cnf server.cnf</span><br><span class="line">[root@hadoop01 my.cnf.d]# vim server.cnf</span><br></pre></td></tr></table></figure><h3 id="1-在server-cnf-文件在-mysqld-标签下增加以下信息："><a href="#1-在server-cnf-文件在-mysqld-标签下增加以下信息：" class="headerlink" title="(1).在server.cnf 文件在[mysqld]标签下增加以下信息："></a>(1).在server.cnf 文件在[mysqld]标签下增加以下信息：</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">init_connect='SET collation_connection = utf8_unicode_ci' </span><br><span class="line">init_connect='SET NAMES utf8' </span><br><span class="line">character-set-server=utf8 </span><br><span class="line">collation-server=utf8_unicode_ci </span><br><span class="line">skip-character-set-client-handshake</span><br></pre></td></tr></table></figure><h3 id="2-在mysql-clients-cnf-文件-mysql-标签下增加如下信息："><a href="#2-在mysql-clients-cnf-文件-mysql-标签下增加如下信息：" class="headerlink" title="(2).在mysql-clients.cnf 文件[mysql]标签下增加如下信息："></a>(2).在mysql-clients.cnf 文件[mysql]标签下增加如下信息：</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">default-character-set=utf8</span><br></pre></td></tr></table></figure><p>全部配置完成后重启数据库服务<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 my.cnf.d]# systemctl restart mariadb</span><br></pre></td></tr></table></figure></p><p>之后进入MariaDB查看字符集<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 my.cnf.d]# systemctl restart mariadb</span><br><span class="line">[root@hadoop01 my.cnf.d]# mysql -uroot -p</span><br><span class="line">Enter password: </span><br><span class="line">Welcome to the MariaDB monitor. Commands end with ; or \g.</span><br><span class="line">Your MariaDB connection id is 9</span><br><span class="line">Server version: 10.2.11-MariaDB MariaDB Server</span><br><span class="line"></span><br><span class="line">Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.</span><br><span class="line"></span><br><span class="line">Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.</span><br><span class="line"></span><br><span class="line">MariaDB [(none)]> show variables like "%character%";show variables like "%collation%";</span><br><span class="line">+--------------------------+----------------------------+</span><br><span class="line">| Variable_name | Value |</span><br><span class="line">+--------------------------+----------------------------+</span><br><span class="line">| character_set_client | utf8 |</span><br><span class="line">| character_set_connection | utf8 |</span><br><span class="line">| character_set_database | utf8 |</span><br><span class="line">| character_set_filesystem | binary |</span><br><span class="line">| character_set_results | utf8 |</span><br><span class="line">| character_set_server | utf8 |</span><br><span class="line">| character_set_system | utf8 |</span><br><span class="line">| character_sets_dir | /usr/share/mysql/charsets/ |</span><br><span class="line">+--------------------------+----------------------------+</span><br><span class="line">8 rows in set (0.00 sec)</span><br><span class="line"></span><br><span class="line">+----------------------+-----------------+</span><br><span class="line">| Variable_name | Value |</span><br><span class="line">+----------------------+-----------------+</span><br><span class="line">| collation_connection | utf8_unicode_ci |</span><br><span class="line">| collation_database | utf8_unicode_ci |</span><br><span class="line">| collation_server | utf8_unicode_ci |</span><br><span class="line">+----------------------+-----------------+</span><br><span class="line">3 rows in set (0.00 sec)</span><br><span class="line"></span><br><span class="line">MariaDB [(none)]></span><br></pre></td></tr></table></figure></p><p>字符集配置完成</p><h1 id="8-创建用户、添加授权"><a href="#8-创建用户、添加授权" class="headerlink" title="8.创建用户、添加授权"></a>8.创建用户、添加授权</h1><h3 id="1-创建用户"><a href="#1-创建用户" class="headerlink" title="(1).创建用户"></a>(1).创建用户</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">MariaDB [(none)]> create user hadoop@localhost identified by 'xxxx'; #请替换xx为密码</span><br><span class="line">Query OK, 0 rows affected (0.01 sec)</span><br></pre></td></tr></table></figure><h3 id="2-为用户进行操作授权"><a href="#2-为用户进行操作授权" class="headerlink" title="(2).为用户进行操作授权"></a>(2).为用户进行操作授权</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">MariaDB [(none)]> grant all on *.* to hadoop@locahost identified by 'xxxx'; #请替换xx为密码</span><br><span class="line">Query OK, 0 rows affected (0.00 sec)</span><br></pre></td></tr></table></figure><h3 id="3-授权外网登录权限"><a href="#3-授权外网登录权限" class="headerlink" title="(3).授权外网登录权限"></a>(3).授权外网登录权限</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">MariaDB [(none)]> grant all privileges on *.* to hadoop@'%' identified by 'xxx'; #请替换xx为密码</span><br><span class="line">Query OK, 0 rows affected (0.00 sec)</span><br></pre></td></tr></table></figure><p>查询用户授权结果<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">MariaDB [mysql]> select host,user,password from user;</span><br><span class="line">+-----------+--------+-------------------------------------------+</span><br><span class="line">| host | user | password |</span><br><span class="line">+-----------+--------+-------------------------------------------+</span><br><span class="line">| localhost | root | *7D8990305DAAE2A688433D400E6559EBDF439529 |</span><br><span class="line">| 127.0.0.1 | root | *7D8990305DAAE2A688433D400E6559EBDF439529 |</span><br><span class="line">| ::1 | root | *7D8990305DAAE2A688433D400E6559EBDF439529 |</span><br><span class="line">| locahost | hadoop | *7D8990305DAAE2A688433D400E6559EBDF439529 |</span><br><span class="line">| localhost | hadoop | *AB7E9F716159ED905A3E5DA78DA0DFD516C429E1 |</span><br><span class="line">| % | hadoop | *7D8990305DAAE2A688433D400E6559EBDF439529 |</span><br><span class="line">+-----------+--------+-------------------------------------------+</span><br><span class="line">6 rows in set (0.00 sec)</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<h1 id="1-检查是否已经具有MariaDB相关安装，并删除已有安装"><a href="#1-检查是否已经具有MariaDB相关安装，并删除已有安装" class="headerlink" title="1.检查是否已经具有MariaDB相关安装，并删除已有安装"></a
</summary>
</entry>
<entry>
<title>java 实现FastDFS文件操作</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/java_%E5%AE%9E%E7%8E%B0FastDFS%E6%96%87%E4%BB%B6%E6%93%8D%E4%BD%9C/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/java_实现FastDFS文件操作/</id>
<published>2018-02-01T05:59:44.984Z</published>
<updated>2018-02-01T05:59:05.000Z</updated>
<content type="html"><![CDATA[<h1 id="java-实现FastDFS-文件操作"><a href="#java-实现FastDFS-文件操作" class="headerlink" title="java 实现FastDFS 文件操作"></a>java 实现FastDFS 文件操作</h1><h4 id="1-下载FastClient并实现依赖Jar安装"><a href="#1-下载FastClient并实现依赖Jar安装" class="headerlink" title="1.下载FastClient并实现依赖Jar安装"></a>1.下载FastClient并实现依赖Jar安装</h4><p>访问余大github项目 地址为<a href="https://github.com/happyfish100/fastdfs-client-java" target="_blank" rel="noopener">fastdfs-client-java</a>下载后导入Idea，修改项目Java编译版本，如图所示：<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-112894627c30e1a9.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="fastDFS java项目截图" title=""> </div> <div class="image-caption">fastDFS java项目截图</div> </figure><br>使用Maven进行 编译和构建，dos窗口定位到该项目路径下，进行编译和构建</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">E:\fastdfs-client-java-master>mvn clean install</span><br></pre></td></tr></table></figure><p>构建成功后，会在maven 本地仓库出现相关Jar包，如图所示：<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-d0364a9b4e4ef646.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="依赖Jar构建成功" title=""> </div> <div class="image-caption">依赖Jar构建成功</div> </figure><br>在需要进行文件操作的项目模块增加Pom文件依赖配置，配置内容如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><!-- fastdfs上传下载图片 路径和上面的pom中对应 --></span><br><span class="line"><dependency></span><br><span class="line"><groupId>org.csource.fastdfs-client-java</groupId></span><br><span class="line"><artifactId>fastdfs-client-java</artifactId></span><br><span class="line"><version>1.27</version></span><br><span class="line"></dependency></span><br></pre></td></tr></table></figure><h4 id="2-增加FastDFS连接配置文件"><a href="#2-增加FastDFS连接配置文件" class="headerlink" title="2.增加FastDFS连接配置文件"></a>2.增加FastDFS连接配置文件</h4><p>在需要的项目模块资源配置文件夹下 src/resource 目录下新增配置文件 fdfs_client.properties<br>配置内容具体如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">connect_timeout = 2</span><br><span class="line">network_timeout = 30</span><br><span class="line">charset = UTF-8</span><br><span class="line">http.tracker_http_port = 8088 # tracker Http端口</span><br><span class="line">http.anti_steal_token = no # 暂无作用</span><br><span class="line">http.secret_key = FastDFS1234567890 # 暂无作用</span><br><span class="line">tracker_server = 192.168.43.60:22122 # tracker Server地址信息</span><br></pre></td></tr></table></figure><h4 id="3-编写FastClient-工具类，用于封装文件操作"><a href="#3-编写FastClient-工具类，用于封装文件操作" class="headerlink" title="3.编写FastClient 工具类，用于封装文件操作"></a>3.编写FastClient 工具类，用于封装文件操作</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br></pre></td><td class="code"><pre><span class="line">package com.gsww.ctyxy.util;</span><br><span class="line">import org.csource.common.MyException;</span><br><span class="line">import org.csource.common.NameValuePair;</span><br><span class="line">import org.csource.fastdfs.*;</span><br><span class="line">import java.io.BufferedOutputStream;</span><br><span class="line">import java.io.IOException;</span><br><span class="line">import java.net.URLDecoder;</span><br><span class="line"></span><br><span class="line">/**</span><br><span class="line"> * FastDFS工具类【实现文件上传、下载、删除、查询】</span><br><span class="line"> * @author Zhangyongliang</span><br><span class="line"> */</span><br><span class="line">public class FastDFSClient {</span><br><span class="line"></span><br><span class="line"> private TrackerClient trackerClient = null;</span><br><span class="line"> private TrackerServer trackerServer = null;</span><br><span class="line"> private StorageServer storageServer = null;</span><br><span class="line"> private StorageClient1 storageClient = null;</span><br><span class="line"></span><br><span class="line"> public FastDFSClient(String conf) throws Exception {</span><br><span class="line"> if (conf.contains("classpath:")) {</span><br><span class="line"> String path = URLDecoder.decode(getClass().getProtectionDomain().getCodeSource().getLocation().toString(),"UTF-8");</span><br><span class="line"> path=path.substring(6);</span><br><span class="line"> conf = conf.replace("classpath:",URLDecoder.decode(path,"UTF-8"));</span><br><span class="line"> }</span><br><span class="line"> ClientGlobal.init(conf);</span><br><span class="line"> trackerClient = new TrackerClient();</span><br><span class="line"> trackerServer = trackerClient.getConnection();</span><br><span class="line"> storageServer = null;</span><br><span class="line"> storageClient = new StorageClient1(trackerServer, storageServer);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /**</span><br><span class="line"> * 上传文件方法</span><br><span class="line"> * <p>Title: uploadFile</p></span><br><span class="line"> * <p>Description: </p></span><br><span class="line"> * @param fileName 文件全路径</span><br><span class="line"> * @param extName 文件扩展名，不包含（.）</span><br><span class="line"> * @param metas 文件扩展信息</span><br><span class="line"> * @return</span><br><span class="line"> * @throws Exception</span><br><span class="line"> */</span><br><span class="line"> public String uploadFile(String fileName, String extName, NameValuePair[] metas) {</span><br><span class="line"> String result=null;</span><br><span class="line"> try {</span><br><span class="line"> result = storageClient.upload_file1(fileName, extName, metas);</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } catch (MyException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> return result;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /**</span><br><span class="line"> * 上传文件,传fileName</span><br><span class="line"> * @param fileName 文件的磁盘路径名称 如：D:/image/aaa.jpg</span><br><span class="line"> * @return null为失败</span><br><span class="line"> */</span><br><span class="line"> public String uploadFile(String fileName) {</span><br><span class="line"> return uploadFile(fileName, null, null);</span><br><span class="line"> }</span><br><span class="line"> /**</span><br><span class="line"> *</span><br><span class="line"> * @param fileName 文件的磁盘路径名称 如：D:/image/aaa.jpg</span><br><span class="line"> * @param extName 文件的扩展名 如 txt jpg等</span><br><span class="line"> * @return null为失败</span><br><span class="line"> */</span><br><span class="line"> public String uploadFile(String fileName, String extName) {</span><br><span class="line"> return uploadFile(fileName, extName, null);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /**</span><br><span class="line"> * 上传文件方法</span><br><span class="line"> * <p>Title: uploadFile</p></span><br><span class="line"> * <p>Description: </p></span><br><span class="line"> * @param fileContent 文件的内容，字节数组</span><br><span class="line"> * @param extName 文件扩展名</span><br><span class="line"> * @param metas 文件扩展信息</span><br><span class="line"> * @return</span><br><span class="line"> * @throws Exception</span><br><span class="line"> */</span><br><span class="line"> public String uploadFile(byte[] fileContent, String extName, NameValuePair[] metas) {</span><br><span class="line"> String result=null;</span><br><span class="line"> try {</span><br><span class="line"> result = storageClient.upload_file1(fileContent, extName, metas);</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } catch (MyException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> return result;</span><br><span class="line"> }</span><br><span class="line"> /**</span><br><span class="line"> * 上传文件</span><br><span class="line"> * @param fileContent 文件的字节数组</span><br><span class="line"> * @return null为失败</span><br><span class="line"> * @throws Exception</span><br><span class="line"> */</span><br><span class="line"> public String uploadFile(byte[] fileContent) throws Exception {</span><br><span class="line"> return uploadFile(fileContent, null, null);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /**</span><br><span class="line"> * 上传文件</span><br><span class="line"> * @param fileContent 文件的字节数组</span><br><span class="line"> * @param extName 文件的扩展名 如 txt jpg png 等</span><br><span class="line"> * @return null为失败</span><br><span class="line"> */</span><br><span class="line"> public String uploadFile(byte[] fileContent, String extName) {</span><br><span class="line"> return uploadFile(fileContent, extName, null);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /**</span><br><span class="line"> * 文件下载到磁盘</span><br><span class="line"> * @param path 图片路径</span><br><span class="line"> * @param output 输出流 中包含要输出到磁盘的路径</span><br><span class="line"> * @return -1失败,0成功</span><br><span class="line"> */</span><br><span class="line"> public int download_file(String path,BufferedOutputStream output) {</span><br><span class="line"> int result=-1;</span><br><span class="line"> try {</span><br><span class="line"> byte[] b = storageClient.download_file1(path);</span><br><span class="line"> try{</span><br><span class="line"> if(b != null){</span><br><span class="line"> output.write(b);</span><br><span class="line"> result=0;</span><br><span class="line"> }</span><br><span class="line"> }catch (Exception e){} //用户可能取消了下载</span><br><span class="line"> finally {</span><br><span class="line"> if (output != null){</span><br><span class="line"> try {</span><br><span class="line"> output.close();</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> } catch (Exception e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> return result;</span><br><span class="line"> }</span><br><span class="line"> /**</span><br><span class="line"> * 获取文件数组</span><br><span class="line"> * @param path 文件的路径 如group1/M00/00/00/wKgRsVjtwpSAXGwkAAAweEAzRjw471.jpg</span><br><span class="line"> * @return</span><br><span class="line"> */</span><br><span class="line"> public byte[] download_bytes(String path) {</span><br><span class="line"> byte[] b=null;</span><br><span class="line"> try {</span><br><span class="line"> b = storageClient.download_file1(path);</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } catch (MyException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> return b;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /**</span><br><span class="line"> * 删除文件</span><br><span class="line"> * @param group 组名 如：group1</span><br><span class="line"> * @param storagePath 不带组名的路径名称 如：M00/00/00/wKgRsVjtwpSAXGwkAAAweEAzRjw471.jpg</span><br><span class="line"> * @return -1失败,0成功</span><br><span class="line"> */</span><br><span class="line"> public Integer delete_file(String group ,String storagePath){</span><br><span class="line"> int result=-1;</span><br><span class="line"> try {</span><br><span class="line"> result = storageClient.delete_file(group, storagePath);</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } catch (MyException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> return result;</span><br><span class="line"> }</span><br><span class="line"> /**</span><br><span class="line"> *</span><br><span class="line"> * @param storagePath 文件的全部路径 如：group1/M00/00/00/wKgRsVjtwpSAXGwkAAAweEAzRjw471.jpg</span><br><span class="line"> * @return -1失败,0成功</span><br><span class="line"> * @throws IOException</span><br><span class="line"> * @throws Exception</span><br><span class="line"> */</span><br><span class="line"> public Integer delete_file(String storagePath){</span><br><span class="line"> int result=-1;</span><br><span class="line"> try {</span><br><span class="line"> result = storageClient.delete_file1(storagePath);</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } catch (MyException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> return result;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /**</span><br><span class="line"> * 获取远程服务器文件资源信息</span><br><span class="line"> * @param groupName 文件组名 如：group1</span><br><span class="line"> * @param remoteFileName M00/00/00/wKgRsVjtwpSAXGwkAAAweEAzRjw471.jpg</span><br><span class="line"> * @return</span><br><span class="line"> */</span><br><span class="line"> public FileInfo getFile(String groupName,String remoteFileName){</span><br><span class="line"> try {</span><br><span class="line"> return storageClient.get_file_info(groupName, remoteFileName);</span><br><span class="line"> } catch (Exception e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> return null;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="4-文件操作"><a href="#4-文件操作" class="headerlink" title="4.文件操作"></a>4.文件操作</h4><p>在web 项目Controller层进行文件的操作</p><p>上传文件<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">@RequestMapping(value = "file/uploadFast",method = RequestMethod.GET)</span><br><span class="line">public void uploadFast(HttpServletRequest request)throws Exception{</span><br><span class="line"> // 1、把FastDFS提供的jar包添加到工程中</span><br><span class="line"> // 2、初始化全局配置。加载一个配置文件。</span><br><span class="line"> String confUrl=this.getClass().getClassLoader().getResource("/fdfs_client.properties").getPath();</span><br><span class="line"> FastDFSClient fastDFSClient=new FastDFSClient(confUrl);</span><br><span class="line"> //上传文件</span><br><span class="line"> String filePath= fastDFSClient.uploadFile("F:\\Photos\\P70602-192547.jpg");</span><br><span class="line"> System.out.println("返回路径："+filePath);</span><br><span class="line"> //省略其他</span><br></pre></td></tr></table></figure></p><p>删除文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">//删除文件</span><br><span class="line"> int flag=fastDFSClient.delete_file("group1/M00/00/00/wKgrPFpf94KASn3vAAsC7gailiI018.jpg");</span><br><span class="line"> System.out.println("删除结果：" +(flag==0?"删除成功":"删除失败"));</span><br></pre></td></tr></table></figure><p>下载文件到桌面</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">//下载文件到用户桌面位置</span><br><span class="line"> FileSystemView fsv = FileSystemView.getFileSystemView();</span><br><span class="line"> File com=fsv.getHomeDirectory(); //读取桌面路径</span><br><span class="line"> int downFlag=fastDFSClient.download_file("group1/M00/00/00/wKgrPFpe9OqAWsHxAAH5yvc2jn8251.jpg",new BufferedOutputStream(new FileOutputStream(com.getPath()+"\\aa.jpg")));</span><br><span class="line"> System.out.println("下载结果为：" +(downFlag==0?"下载文件成功":"下载文件失败"));</span><br></pre></td></tr></table></figure><p>查询文件信息</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">//获取文件信息</span><br><span class="line"> FileInfo file=fastDFSClient.getFile("group1","M00/00/00/wKgrPFpe9OqAWsHxAAH5yvc2jn8251.jpg");</span><br><span class="line"> System.out.println("获取文件信息成功："+file.getFileSize());</span><br></pre></td></tr></table></figure><p><strong>说明：在FastDFS工具类中集合了支持字节数组的上传入参。</strong></p>]]></content>
<summary type="html">
<h1 id="java-实现FastDFS-文件操作"><a href="#java-实现FastDFS-文件操作" class="headerlink" title="java 实现FastDFS 文件操作"></a>java 实现FastDFS 文件操作</h1><h4 i
</summary>
</entry>
<entry>
<title>Kafka集群的安装部署和实践应用</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/Kafka%E9%9B%86%E7%BE%A4%E7%9A%84%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2%E5%92%8C%E5%AE%9E%E8%B7%B5%E5%BA%94%E7%94%A8/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/Kafka集群的安装部署和实践应用/</id>
<published>2018-02-01T05:58:55.363Z</published>
<updated>2018-02-01T05:58:13.000Z</updated>
<content type="html"><![CDATA[<h1 id="Kafka介绍"><a href="#Kafka介绍" class="headerlink" title="Kafka介绍"></a>Kafka介绍</h1><p>Kafka是一种高吞吐量的分布式发布订阅消息系统，有如下特性：</p><ul><li>通过O(1)的磁盘数据结构提供消息的持久化，这种结构对于即使数以TB的消息存储也能够保持长时间的稳定性能。</li><li>高吞吐量：即使是非常普通的硬件Kafka也可以支持每秒数百万的消息。</li><li>支持通过Kafka服务器和消费机集群来分区消息。</li><li>支持Hadoop并行数据加载。<h3 id="消息队列的作用"><a href="#消息队列的作用" class="headerlink" title="消息队列的作用"></a>消息队列的作用</h3></li><li>应用程序解耦并行处理</li><li>顺序保证</li><li>高吞吐率</li><li>高容错、高可用</li><li>可扩展</li><li>峰值处理<figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-1c7a43b5a712f24e.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Kafka集群.png" title=""> </div> <div class="image-caption">Kafka集群.png</div> </figure><h2 id="kafka原理"><a href="#kafka原理" class="headerlink" title="kafka原理"></a>kafka原理</h2>Kafka集群由多个实例组成，每个节点称为Broker，对消息保存时根据Topic进行归类<br>一个Topic可以被划分为多个Partition每个Partition可以有多个副本。<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-ac62aa43ce99e97e.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Kafka原理图01.png" title=""> </div> <div class="image-caption">Kafka原理图01.png</div> </figure></li></ul><p>Partition内顺序存储，写入新消息采用追加的方式，消费消息采用FIFO的方式顺序拉取消息<br>一个Topic可以有多个分区，Kafka只保证同一个分区内有序，不保证Topic整体（多个分区之间）有序<br><br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-1be60752cebd5d73.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="kafka原理图02.png" title=""> </div> <div class="image-caption">kafka原理图02.png</div> </figure></p><p>Consumer Group（CG），为了加快读取速度，多个consumer可以划分为一个组，并行消费一个Toic,一个Topic可以由多个CG订阅，多个CG之间是平等的，同一个CG内可以有一个或多个consumer，同一个CG内的consumer之间是竞争 关系，一个消息在一个CG内的只能被一个consumer消费<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-e591cf134ad4f728.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="kafka原理图03.png" title=""> </div> <div class="image-caption">kafka原理图03.png</div> </figure></p><h1 id="一、Kafka集群部署"><a href="#一、Kafka集群部署" class="headerlink" title="一、Kafka集群部署"></a>一、Kafka集群部署</h1><p>集群规划清单</p><table><thead><tr><th style="text-align:center">名称</th><th style="text-align:center">节点</th><th style="text-align:left">说明</th><th style="text-align:center">节点名</th></tr></thead><tbody><tr><td style="text-align:center">Broker01</td><td style="text-align:center">192.168.43.22</td><td style="text-align:left">kafka节点01</td><td style="text-align:center">hadoop03</td></tr><tr><td style="text-align:center">Broker02</td><td style="text-align:center">192.168.43.23</td><td style="text-align:left">kafka节点02</td><td style="text-align:center">hadoop04</td></tr><tr><td style="text-align:center">Broker03</td><td style="text-align:center">192.168.43.24</td><td style="text-align:left">kafka节点03</td><td style="text-align:center">hadoop05</td></tr><tr><td style="text-align:center">Zookeeper</td><td style="text-align:center">192.168.43.20/21/22</td><td style="text-align:left">Zookeeper集群节点</td><td style="text-align:center">hadoop01/hadoop02/hadoop03</td></tr></tbody></table><h4 id="1-下载Kafka安装包-并解压安装"><a href="#1-下载Kafka安装包-并解压安装" class="headerlink" title="1.下载Kafka安装包,并解压安装"></a>1.下载Kafka安装包,并解压安装</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 kafka_2.11-0.10.2.1]# ll</span><br><span class="line">总用量 52</span><br><span class="line">drwxr-xr-x. 3 hadoop hadoop 4096 4月 22 2017 bin</span><br><span class="line">drwxr-xr-x. 2 hadoop hadoop 4096 4月 22 2017 config</span><br><span class="line">drwxr-xr-x. 2 root root 152 1月 20 18:57 kafka-logs</span><br><span class="line">drwxr-xr-x. 2 hadoop hadoop 4096 1月 20 18:43 libs</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 28824 4月 22 2017 LICENSE</span><br><span class="line">drwxr-xr-x. 2 root root 4096 1月 20 23:07 logs</span><br><span class="line">-rw-r--r--. 1 hadoop hadoop 336 4月 22 2017 NOTICE</span><br><span class="line">drwxr-xr-x. 2 hadoop hadoop 47 4月 22 2017 site-docs</span><br></pre></td></tr></table></figure><h4 id="2-创建软链接"><a href="#2-创建软链接" class="headerlink" title="2.创建软链接"></a>2.创建软链接</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 kafka_2.11-0.10.2.1]# ln -s /home/hadoop/apps/kafka_2.11-0.10.2.1 /usr/local/kafka</span><br></pre></td></tr></table></figure><h4 id="3-创建日志文件夹"><a href="#3-创建日志文件夹" class="headerlink" title="3.创建日志文件夹"></a>3.创建日志文件夹</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 kafka]# pwd</span><br><span class="line">/usr/local/kafka</span><br><span class="line">[root@hadoop03 kafka]# mkdir kafka-logs/</span><br></pre></td></tr></table></figure><h4 id="4-配置服务启动信息"><a href="#4-配置服务启动信息" class="headerlink" title="4.配置服务启动信息"></a>4.配置服务启动信息</h4><p>在/usr/local/kafka/config目录下修改server.properties文件，具体内容如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br></pre></td><td class="code"><pre><span class="line"># Licensed to the Apache Software Foundation (ASF) under one or more</span><br><span class="line"># contributor license agreements. See the NOTICE file distributed with</span><br><span class="line"># this work for additional information regarding copyright ownership.</span><br><span class="line"># The ASF licenses this file to You under the Apache License, Version 2.0</span><br><span class="line"># (the "License"); you may not use this file except in compliance with</span><br><span class="line"># the License. You may obtain a copy of the License at</span><br><span class="line">#</span><br><span class="line"># http://www.apache.org/licenses/LICENSE-2.0</span><br><span class="line">#</span><br><span class="line"># Unless required by applicable law or agreed to in writing, software</span><br><span class="line"># distributed under the License is distributed on an "AS IS" BASIS,</span><br><span class="line"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span><br><span class="line"># See the License for the specific language governing permissions and</span><br><span class="line"># limitations under the License.</span><br><span class="line"></span><br><span class="line"># see kafka.server.KafkaConfig for additional details and defaults</span><br><span class="line"></span><br><span class="line">############################# Server Basics #############################</span><br><span class="line"></span><br><span class="line">#每个borker的id是唯一的，多个broker要设置不同的id</span><br><span class="line">broker.id=0</span><br><span class="line"></span><br><span class="line">#访问端口号</span><br><span class="line">port=9092</span><br><span class="line"></span><br><span class="line">#访问地址</span><br><span class="line">host.name=192.168.43.22</span><br><span class="line"></span><br><span class="line">#允许删除topic</span><br><span class="line">delete.topic.enable=true</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># The number of threads handling network requests</span><br><span class="line">num.network.threads=3</span><br><span class="line"></span><br><span class="line"># The number of threads doing disk I/O</span><br><span class="line">num.io.threads=8</span><br><span class="line"></span><br><span class="line"># The send buffer (SO_SNDBUF) used by the socket server</span><br><span class="line">socket.send.buffer.bytes=102400</span><br><span class="line"></span><br><span class="line"># The receive buffer (SO_RCVBUF) used by the socket server</span><br><span class="line">socket.receive.buffer.bytes=102400</span><br><span class="line"></span><br><span class="line"># The maximum size of a request that the socket server will accept (protection against OOM)</span><br><span class="line">socket.request.max.bytes=104857600</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">############################# Log Basics #############################</span><br><span class="line"></span><br><span class="line">#存储数据路径，默认是在/tmp目录下，需要修改</span><br><span class="line">log.dirs=/usr/local/kafka/kafka-logs</span><br><span class="line"></span><br><span class="line">#创建topic默认分区数</span><br><span class="line">num.partitions=1</span><br><span class="line"></span><br><span class="line"># The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.</span><br><span class="line"># This value is recommended to be increased for installations with data dirs located in RAID array.</span><br><span class="line">num.recovery.threads.per.data.dir=1</span><br><span class="line"></span><br><span class="line">############################# Log Flush Policy #############################</span><br><span class="line"></span><br><span class="line"># Messages are immediately written to the filesystem but by default we only fsync() to sync</span><br><span class="line"># the OS cache lazily. The following configurations control the flush of data to disk.</span><br><span class="line"># There are a few important trade-offs here:</span><br><span class="line"># 1. Durability: Unflushed data may be lost if you are not using replication.</span><br><span class="line"># 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.</span><br><span class="line"># 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks.</span><br><span class="line"># The settings below allow one to configure the flush policy to flush data after a period of time or</span><br><span class="line"># every N messages (or both). This can be done globally and overridden on a per-topic basis.</span><br><span class="line"></span><br><span class="line"># The number of messages to accept before forcing a flush of data to disk</span><br><span class="line">#log.flush.interval.messages=10000</span><br><span class="line"></span><br><span class="line"># The maximum amount of time a message can sit in a log before we force a flush</span><br><span class="line">#log.flush.interval.ms=1000</span><br><span class="line"></span><br><span class="line">############################# Log Retention Policy #############################</span><br><span class="line"></span><br><span class="line"># The following configurations control the disposal of log segments. The policy can</span><br><span class="line"># be set to delete segments after a period of time, or after a given size has accumulated.</span><br><span class="line"># A segment will be deleted whenever *either* of these criteria are met. Deletion always happens</span><br><span class="line"># from the end of the log.</span><br><span class="line"></span><br><span class="line">#数据保存时间，默认7天，单位小时</span><br><span class="line">log.retention.hours=168</span><br><span class="line"></span><br><span class="line"># A size-based retention policy for logs. Segments are pruned from the log as long as the remaining</span><br><span class="line"># segments don't drop below log.retention.bytes. Functions independently of log.retention.hours.</span><br><span class="line">#log.retention.bytes=1073741824</span><br><span class="line"></span><br><span class="line"># The maximum size of a log segment file. When this size is reached a new log segment will be created.</span><br><span class="line">log.segment.bytes=1073741824</span><br><span class="line"></span><br><span class="line"># The interval at which log segments are checked to see if they can be deleted according</span><br><span class="line"># to the retention policies</span><br><span class="line">log.retention.check.interval.ms=300000</span><br><span class="line"></span><br><span class="line">############################# Zookeeper #############################</span><br><span class="line"></span><br><span class="line">#zookeeper地址，多个地址用逗号隔开</span><br><span class="line">zookeeper.connect=192.168.43.20:2181,192.168.43.21:2181,192.168.43.22:2181</span><br><span class="line"></span><br><span class="line"># Timeout in ms for connecting to zookeeper</span><br><span class="line">zookeeper.connection.timeout.ms=6000</span><br></pre></td></tr></table></figure><h4 id="5-拷贝文件信息到Broker02-Broker03节点上"><a href="#5-拷贝文件信息到Broker02-Broker03节点上" class="headerlink" title="5.拷贝文件信息到Broker02/Broker03节点上"></a>5.拷贝文件信息到Broker02/Broker03节点上</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">scp -r /home/hadoop/apps/kafka_2.11-0.10.2.1 hadoop@node04:/home/hadoop/apps/</span><br><span class="line">scp -r /home/hadoop/apps/kafka_2.11-0.10.2.1 hadoop@node04:/home/hadoop/apps/</span><br></pre></td></tr></table></figure><h4 id="6-修改Broker02和Broker03信息"><a href="#6-修改Broker02和Broker03信息" class="headerlink" title="6.修改Broker02和Broker03信息"></a>6.修改Broker02和Broker03信息</h4><p>创建软连接</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 kafka_2.11-0.10.2.1]# ln -s /home/hadoop/apps/kafka_2.11-0.10.2.1 /usr/local/kafka</span><br></pre></td></tr></table></figure><p>修改配置文件server.properties信息<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">broker.id=1</span><br><span class="line">host.name=192.168.43.23</span><br></pre></td></tr></table></figure></p><p>修改Broker03节点server.properties信息</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">broker.id=2</span><br><span class="line">host.name=192.168.43.24</span><br></pre></td></tr></table></figure><h4 id="7-分别启动Broker01-Broker02-Broker03"><a href="#7-分别启动Broker01-Broker02-Broker03" class="headerlink" title="7.分别启动Broker01/Broker02/Broker03"></a>7.分别启动Broker01/Broker02/Broker03</h4><p>以后台进程的方式启动Kafka<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]#./kafka-server-start.sh -daemon config/server.properties</span><br></pre></td></tr></table></figure></p><h1 id="二、Kafka应用实践"><a href="#二、Kafka应用实践" class="headerlink" title="二、Kafka应用实践"></a>二、Kafka应用实践</h1><h4 id="1-创建主题"><a href="#1-创建主题" class="headerlink" title="1.创建主题"></a>1.创建主题</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# pwd</span><br><span class="line">/usr/local/kafka/bin</span><br><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --create --zookeeper 192.168.43.20:2181 --replication-factor 2 --partitions 3 --topic topicnewtest1</span><br><span class="line">Created topic "topicnewtest1".</span><br></pre></td></tr></table></figure><h4 id="2-查看主题"><a href="#2-查看主题" class="headerlink" title="2.查看主题"></a>2.查看主题</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --list --zookeeper 192.168.43.20:2181</span><br><span class="line">topicnewtest1</span><br></pre></td></tr></table></figure><h4 id="3-查看主题信息"><a href="#3-查看主题信息" class="headerlink" title="3.查看主题信息"></a>3.查看主题信息</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --describe --zookeeper 192.168.43.20:2181 --topic topicnewtest1</span><br><span class="line">Topic:topicnewtest1PartitionCount:3ReplicationFactor:2Configs:</span><br><span class="line">Topic: topicnewtest1Partition: 0Leader: 2Replicas: 2,0Isr: 2,0</span><br><span class="line">Topic: topicnewtest1Partition: 1Leader: 0Replicas: 0,1Isr: 0,1</span><br><span class="line">Topic: topicnewtest1Partition: 2Leader: 1Replicas: 1,2Isr: 1,2</span><br></pre></td></tr></table></figure><h4 id="4-删除主题"><a href="#4-删除主题" class="headerlink" title="4.删除主题"></a>4.删除主题</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --delete --zookeeper 192.168.43.20:2181 --topic topicnewtest1</span><br><span class="line">Topic topicnewtest1 is marked for deletion.</span><br><span class="line">Note: This will have no impact if delete.topic.enable is not set to true.</span><br></pre></td></tr></table></figure><h4 id="5-增加分区"><a href="#5-增加分区" class="headerlink" title="5.增加分区"></a>5.增加分区</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --alter --zookeeper 192.168.43.20:2181 --topic topicnewtest1 --partitions 5</span><br><span class="line">WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected</span><br><span class="line">Adding partitions succeeded!</span><br><span class="line">[root@hadoop03 bin]# ./kafka-topics.sh --describe --zookeeper 192.168.43.20:2181 --topic topicnewtest1</span><br><span class="line">Topic:topicnewtest1PartitionCount:5ReplicationFactor:2Configs:</span><br><span class="line">Topic: topicnewtest1Partition: 0Leader: 1Replicas: 1,0Isr: 1,0</span><br><span class="line">Topic: topicnewtest1Partition: 1Leader: 2Replicas: 2,1Isr: 2,1</span><br><span class="line">Topic: topicnewtest1Partition: 2Leader: 0Replicas: 0,2Isr: 0,2</span><br><span class="line">Topic: topicnewtest1Partition: 3Leader: 1Replicas: 1,2Isr: 1,2</span><br><span class="line">Topic: topicnewtest1Partition: 4Leader: 2Replicas: 2,0Isr: 2,0</span><br></pre></td></tr></table></figure><h4 id="6-使用kafka自带的生产者客户端脚本和消费端脚本"><a href="#6-使用kafka自带的生产者客户端脚本和消费端脚本" class="headerlink" title="6.使用kafka自带的生产者客户端脚本和消费端脚本"></a>6.使用kafka自带的生产者客户端脚本和消费端脚本</h4><p>使用kafka自带的生产者客户端脚本</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop03 bin]# ./kafka-console-producer.sh --broker-list 192.168.43.22:9092,192.168.43.23:9092 --topic topicnewtest1</span><br></pre></td></tr></table></figure><p>使用kafka自带的消费者客户端脚本</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop04 bin]# ./kafka-console-consumer.sh --zookeeper 192.168.43.20:2181 --from-beginning --topic topicnewtest1</span><br></pre></td></tr></table></figure><p>在生成端发送消息，可以在消费看到消息</p><h4 id="7-使用Java访问Kafka产生消息和消费消息"><a href="#7-使用Java访问Kafka产生消息和消费消息" class="headerlink" title="7.使用Java访问Kafka产生消息和消费消息"></a>7.使用Java访问Kafka产生消息和消费消息</h4><ul><li><p>Producer</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br></pre></td><td class="code"><pre><span class="line">package cn.chinahadoop.client;</span><br><span class="line"></span><br><span class="line">import org.apache.kafka.clients.producer.KafkaProducer;</span><br><span class="line">import org.apache.kafka.clients.producer.Producer;</span><br><span class="line">import org.apache.kafka.clients.producer.ProducerRecord;</span><br><span class="line"></span><br><span class="line">import java.util.Date;</span><br><span class="line">import java.util.Properties;</span><br><span class="line">import java.util.Random;</span><br><span class="line"></span><br><span class="line">/**</span><br><span class="line"> * Kafka生产端</span><br><span class="line"> * @author Zhangyongliang</span><br><span class="line"> */</span><br><span class="line">public class ProducerClient {</span><br><span class="line"> public static void main(String[] args){</span><br><span class="line"> Properties props = new Properties();</span><br><span class="line"> //kafka broker列表</span><br><span class="line"> props.put("bootstrap.servers", "192.168.43.22:9092,192.168.43.23:9092,192.168.43.24:9092");</span><br><span class="line"> //acks=1表示Broker接收到消息成功写入本地log文件后向Producer返回成功接收的信号，不需要等待所有的Follower全部同步完消息后再做回应</span><br><span class="line"> props.put("acks", "1");</span><br><span class="line"> //key和value的字符串序列化类</span><br><span class="line"> props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");</span><br><span class="line"> props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");</span><br><span class="line"></span><br><span class="line"> Producer<String, String> producer = new KafkaProducer<String, String>(props);</span><br><span class="line"> //用户产生随机数，模拟消息生成</span><br><span class="line"> Random rand = new Random();</span><br><span class="line"> for(int i = 0; i < 20; i++) {</span><br><span class="line"> //通过随机数产生一个ip地址作为key发送出去</span><br><span class="line"> String ip = "192.168.1." + rand.nextInt(255);</span><br><span class="line"> long runtime = new Date().getTime();</span><br><span class="line"> //组装一条消息内容</span><br><span class="line"> String msg = runtime + "---" + ip;</span><br><span class="line"> try {</span><br><span class="line"> Thread.sleep(1000);</span><br><span class="line"> } catch (InterruptedException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> System.out.println("send to kafka->key:" + ip + " value:" + msg);</span><br><span class="line"> //向kafka topictest1主题发送消息</span><br><span class="line"> producer.send(new ProducerRecord<String, String>("topicnewtest1", ip, msg));</span><br><span class="line"> }</span><br><span class="line"> producer.close();</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></li><li><p>ConSumer</p></li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br></pre></td><td class="code"><pre><span class="line">package com.yongliang.kafka;</span><br><span class="line"></span><br><span class="line">import org.apache.kafka.clients.consumer.ConsumerRecord;</span><br><span class="line">import org.apache.kafka.clients.consumer.ConsumerRecords;</span><br><span class="line">import org.apache.kafka.clients.consumer.KafkaConsumer;</span><br><span class="line"></span><br><span class="line">import java.util.ArrayList;</span><br><span class="line">import java.util.Arrays;</span><br><span class="line">import java.util.List;</span><br><span class="line">import java.util.Properties;</span><br><span class="line"></span><br><span class="line">/**</span><br><span class="line"> * Kafka消费端</span><br><span class="line"> * @author Zhangyongliang</span><br><span class="line"> */</span><br><span class="line">public class ConsumerClient {</span><br><span class="line"> /**</span><br><span class="line"> * 手动提交偏移量</span><br><span class="line"> */</span><br><span class="line"> public static void manualCommintClient(){</span><br><span class="line"> Properties props = new Properties();</span><br><span class="line"> //kafka broker列表</span><br><span class="line"> props.put("bootstrap.servers", "192.168.43.22:9092,192.168.43.23:9092,192.168.43.24:9092");</span><br><span class="line"> //consumer group id</span><br><span class="line"> props.put("group.id", "yongliang");</span><br><span class="line"> //手动提交offset</span><br><span class="line"> props.put("enable.auto.commit", "false");</span><br><span class="line"> //earliest表示从最早的偏移量开始拉取，latest表示从最新的偏移量开始拉取，none表示如果没有发现该Consumer组之前拉取的偏移量则抛异常。默认值latest。</span><br><span class="line"> props.put("auto.offset.reset", "earliest");</span><br><span class="line"> //key和value的字符串反序列化类</span><br><span class="line"> props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");</span><br><span class="line"> props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");</span><br><span class="line"> KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);</span><br><span class="line"> //consumer订阅topictest1主题，同时消费多个主题用逗号隔开</span><br><span class="line"> consumer.subscribe(Arrays.asList("topicnewtest1"));</span><br><span class="line"> //每次最少处理10条消息后才提交</span><br><span class="line"> final int minBatchSize = 10;</span><br><span class="line"> //用于保存消息的list</span><br><span class="line"> List<ConsumerRecord<String, String>> bufferList = new ArrayList<ConsumerRecord<String, String>>();</span><br><span class="line"> while (true) {</span><br><span class="line"> System.out.println("--------------start pull message---------------" );</span><br><span class="line"> long starttime = System.currentTimeMillis();</span><br><span class="line"> //poll方法需要传入一个超时时间，当没有可以拉取的消息时先等待，</span><br><span class="line"> //如果已到超时时间还没有可以拉取的消息则进行下一轮拉取，单位毫秒</span><br><span class="line"> ConsumerRecords<String, String> records = consumer.poll(1000);</span><br><span class="line"> long endtime = System.currentTimeMillis();</span><br><span class="line"> long tm = (endtime - starttime) / 1000;</span><br><span class="line"> System.out.println("--------------end pull message and times=" + tm + "s -------------");</span><br><span class="line"></span><br><span class="line"> for (ConsumerRecord<String, String> record : records) {</span><br><span class="line"> System.out.printf("partition = %d, offset = %d, key = %s, value = %s%n", record.partition(), record.offset(), record.key(), record.value());</span><br><span class="line"> bufferList.add(record);</span><br><span class="line"> }</span><br><span class="line"> System.out.println("--------------buffer size->" + bufferList.size());</span><br><span class="line"> //如果读取到的消息满了10条, 就进行处理</span><br><span class="line"> if (bufferList.size() >= minBatchSize) {</span><br><span class="line"> System.out.println("******start deal message******");</span><br><span class="line"> try {</span><br><span class="line"> //当前线程睡眠1秒钟，模拟消息处理过程</span><br><span class="line"> Thread.sleep(1000);</span><br><span class="line"> } catch (InterruptedException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> System.out.println("manual commint offset start...");</span><br><span class="line"> //处理完之后进行提交</span><br><span class="line"> consumer.commitSync();</span><br><span class="line"> //清除list, 继续接收</span><br><span class="line"> bufferList.clear();</span><br><span class="line"> System.out.println("manual commint offset end...");</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /**</span><br><span class="line"> * 自动提交偏移量</span><br><span class="line"> */</span><br><span class="line"> public static void autoCommintClient(){</span><br><span class="line"> Properties props = new Properties();</span><br><span class="line"> //kafka broker列表</span><br><span class="line"> props.put("bootstrap.servers", "192.168.43.22:9092,192.168.43.23:9092,192.168.43.24:9092");</span><br><span class="line"> props.put("group.id", "newConsumerGroup");</span><br><span class="line"> //自动提交</span><br><span class="line"> props.put("enable.auto.commit", "true");</span><br><span class="line"> //自动提交时间间隔1000毫秒</span><br><span class="line"> props.put("auto.commit.interval.ms", "1000");</span><br><span class="line"> //earliest表示从最早的偏移量开始拉取，latest表示从最新的偏移量开始拉取，none表示如果没有发现该Consumer组之前拉取的偏移量则抛异常。默认值latest。</span><br><span class="line"> props.put("auto.offset.reset", "earliest");</span><br><span class="line"> //key和value的字符串反序列化类</span><br><span class="line"> props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");</span><br><span class="line"> props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");</span><br><span class="line"> KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);</span><br><span class="line"> //consumer订阅topictest1主题，同时消费多个主题用逗号隔开</span><br><span class="line"> consumer.subscribe(Arrays.asList("topicnewtest1"));</span><br><span class="line"> while (true) {</span><br><span class="line"> //poll方法需要传入一个超时时间，当没有可以拉取的消息时先等待，</span><br><span class="line"> //如果已到超时时间还没有可以拉取的消息则进行下一轮拉取，单位毫秒</span><br><span class="line"> ConsumerRecords<String, String> records = consumer.poll(1000);</span><br><span class="line"> //处理拉取过来的消息</span><br><span class="line"> for (ConsumerRecord<String, String> record : records){</span><br><span class="line"> System.out.printf("partition = %d, offset = %d, key = %s, value = %s%n", record.partition(), record.offset(), record.key(), record.value());</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> public static void main(String[] args){</span><br><span class="line"> //自动提交offset</span><br><span class="line">// autoCommintClient();</span><br><span class="line"> //手动提交offset</span><br><span class="line"> manualCommintClient();</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h1 id="Kafka介绍"><a href="#Kafka介绍" class="headerlink" title="Kafka介绍"></a>Kafka介绍</h1><p>Kafka是一种高吞吐量的分布式发布订阅消息系统，有如下特性：</p>
<ul>
<li>通过O(1
</summary>
</entry>
<entry>
<title>CentOS 单机搭建FastDFS文件系统</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/CentOS_%E5%8D%95%E6%9C%BA%E6%90%AD%E5%BB%BAFastDFS%E6%96%87%E4%BB%B6%E7%B3%BB%E7%BB%9F/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/CentOS_单机搭建FastDFS文件系统/</id>
<published>2018-02-01T05:58:09.645Z</published>
<updated>2018-02-01T05:57:28.000Z</updated>
<content type="html"><![CDATA[<h1 id="此例为在CentOS7-2-单机上搭建一个FastDFS-文件管理系统"><a href="#此例为在CentOS7-2-单机上搭建一个FastDFS-文件管理系统" class="headerlink" title="此例为在CentOS7.2 单机上搭建一个FastDFS 文件管理系统"></a>此例为在CentOS7.2 单机上搭建一个FastDFS 文件管理系统</h1><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-8768c3c3d2313c24.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="FastDFS架构图" title=""> </div> <div class="image-caption">FastDFS架构图</div> </figure><p><strong>安装所需文件均上传到百度云盘，位置<a href="https://pan.baidu.com/s/1htBiHoS" target="_blank" rel="noopener">：FastDFS百度云盘</a></strong><br>安装清单如下：<br>|软件名称|版本|百度云盘存放名称|<br>|:—-:|:—-:|:—–:|<br>|FastDFS|5.11|fastdfs-5.11.zip|<br>|FastDFS-Nginx-module|无|fastdfs-nginx-module-master.zip|<br>|LibFastCommon|1.0.36|libfastcommon-1.0.36.zip|<br>|nginx|1.10.3|nginx-1.10.3.tar.gz|</p><h1 id="一、安装FastDFS"><a href="#一、安装FastDFS" class="headerlink" title="一、安装FastDFS"></a>一、安装FastDFS</h1><h4 id="1-安装libfastcommon"><a href="#1-安装libfastcommon" class="headerlink" title="1.安装libfastcommon"></a>1.安装libfastcommon</h4><p>先解压安装包到目录</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastDFS]# unzip libfastcommon-1.0.36.zip</span><br></pre></td></tr></table></figure><p>解压后目录如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# ll</span><br><span class="line">[root@localhost libfastcommon-1.0.36]# ll</span><br><span class="line">总用量 32</span><br><span class="line">drwxr-xr-x. 2 root root 117 4月 5 2017 doc</span><br><span class="line">-rw-r--r--. 1 root root 8005 4月 5 2017 HISTORY</span><br><span class="line">-rw-r--r--. 1 root root 566 4月 5 2017 INSTALL</span><br><span class="line">-rw-r--r--. 1 root root 1606 4月 5 2017 libfastcommon.spec</span><br><span class="line">-rwxr-xr-x. 1 root root 3099 4月 5 2017 make.sh</span><br><span class="line">drwxr-xr-x. 2 root root 191 4月 5 2017 php-fastcommon</span><br><span class="line">-rw-r--r--. 1 root root 2763 4月 5 2017 README</span><br><span class="line">drwxr-xr-x. 3 root root 4096 1月 17 11:21 src</span><br></pre></td></tr></table></figure></p><p>安装C编译工具 gcc</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# yum -y install gcc-c++</span><br></pre></td></tr></table></figure><p>编译libfastcommon软件并安装</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# ./make.sh && ./make.sh install</span><br></pre></td></tr></table></figure><p>为libcommon 创建软链接到/usr/local/lib目录下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so</span><br><span class="line">[root@localhost fastdfs-5.11]# ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so</span><br><span class="line">[root@localhost fastdfs-5.11]# ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so</span><br><span class="line">[root@localhost fastdfs-5.11]# ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so</span><br></pre></td></tr></table></figure><h4 id="2-安装FastDFS"><a href="#2-安装FastDFS" class="headerlink" title="2.安装FastDFS"></a>2.安装FastDFS</h4><p>解压安装包</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastDFS]# unzip fastdfs-5.11.zip</span><br></pre></td></tr></table></figure><p>解压后目录如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">总用量 128</span><br><span class="line">drwxr-xr-x. 3 root root 4096 1月 17 11:25 client</span><br><span class="line">drwxr-xr-x. 2 root root 4096 1月 17 11:25 common</span><br><span class="line">drwxr-xr-x. 2 root root 146 6月 3 2017 conf</span><br><span class="line">-rw-r--r--. 1 root root 35067 6月 3 2017 COPYING-3_0.txt</span><br><span class="line">-rw-r--r--. 1 root root 3171 6月 3 2017 fastdfs.spec</span><br><span class="line">-rw-r--r--. 1 root root 33100 6月 3 2017 HISTORY</span><br><span class="line">drwxr-xr-x. 2 root root 48 6月 3 2017 init.d</span><br><span class="line">-rw-r--r--. 1 root root 7755 6月 3 2017 INSTALL</span><br><span class="line">-rwxr-xr-x. 1 root root 5548 6月 3 2017 make.sh</span><br><span class="line">drwxr-xr-x. 2 root root 4096 6月 3 2017 php_client</span><br><span class="line">-rw-r--r--. 1 root root 2380 6月 3 2017 README.md</span><br><span class="line">-rwxr-xr-x. 1 root root 1768 6月 3 2017 restart.sh</span><br><span class="line">-rwxr-xr-x. 1 root root 1680 6月 3 2017 stop.sh</span><br><span class="line">drwxr-xr-x. 4 root root 4096 1月 17 11:25 storage</span><br><span class="line">drwxr-xr-x. 2 root root 4096 6月 3 2017 test</span><br><span class="line">drwxr-xr-x. 2 root root 4096 1月 17 11:25 tracker</span><br></pre></td></tr></table></figure></p><p>进入解压目录并进行编译和安装</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastDFS]# cd fastdfs-5.11/</span><br><span class="line">[root@localhost fastdfs-5.11]# ./make.sh && ./make.sh install</span><br></pre></td></tr></table></figure><p>安装成功后，FastDFS会安装在/etc/fdfs目录下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# ll /etc/fdfs/</span><br><span class="line">总用量 76</span><br><span class="line">-rw-r--r--. 1 root root 316 1月 17 11:47 client.conf</span><br><span class="line">-rw-r--r--. 1 root root 1461 1月 17 11:25 client.conf.sample</span><br><span class="line">-rw-r--r--. 1 root root 955 1月 17 13:20 http.conf</span><br><span class="line">-rw-r--r--. 1 root root 31172 1月 17 13:21 mime.types</span><br><span class="line">-rw-r--r--. 1 root root 3716 1月 17 12:57 mod_fastdfs.conf</span><br><span class="line">-rw-r--r--. 1 root root 1278 1月 17 11:40 storage.conf</span><br><span class="line">-rw-r--r--. 1 root root 7927 1月 17 11:25 storage.conf.sample</span><br><span class="line">-rw-r--r--. 1 root root 105 1月 17 11:25 storage_ids.conf.sample</span><br><span class="line">-rw-r--r--. 1 root root 1356 1月 17 11:34 tracker.conf</span><br><span class="line">-rw-r--r--. 1 root root 7389 1月 17 11:25 tracker.conf.sample</span><br></pre></td></tr></table></figure><p>我们需要把这三个示例文件复制一份，去掉.sample</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# cp client.conf.sample client.conf</span><br><span class="line">[root@localhost fdfs]# cp storage.conf.sample storage.conf</span><br><span class="line">[root@localhost fdfs]# cp tracker.conf.sample tracker.conf</span><br></pre></td></tr></table></figure><p>FastDFS安装结束</p><h1 id="二、安装Tracker"><a href="#二、安装Tracker" class="headerlink" title="二、安装Tracker"></a>二、安装Tracker</h1><h4 id="1-创建tracker工作目录"><a href="#1-创建tracker工作目录" class="headerlink" title="1.创建tracker工作目录"></a>1.创建tracker工作目录</h4><p>此目录用于保存tracker 的data和log</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# mkdir /opt/fastdfs_tracker</span><br></pre></td></tr></table></figure><h4 id="2-配置tracker"><a href="#2-配置tracker" class="headerlink" title="2.配置tracker"></a>2.配置tracker</h4><p>配置 /etc/fdfs目录下tracker.conf<br>主要实现以下4个配置内容：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">1.disabled=false </span><br><span class="line">2.port=22122 #默认端口号 </span><br><span class="line">3.base_path=/opt/fastdfs_tracker #我刚刚创建的目录 </span><br><span class="line">4.http.server_port=8080 #默认端口是8080</span><br><span class="line">5.bind_addr= 0.0.0.0 监听地址</span><br></pre></td></tr></table></figure><p>完整tracker.conf 文件信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br></pre></td><td class="code"><pre><span class="line">disabled=false</span><br><span class="line"></span><br><span class="line">bind_addr= 0.0.0.0</span><br><span class="line"></span><br><span class="line">port=22122</span><br><span class="line"></span><br><span class="line">connect_timeout=30</span><br><span class="line"></span><br><span class="line">network_timeout=60</span><br><span class="line"></span><br><span class="line">base_path=/opt/fastdfs_tracker</span><br><span class="line"></span><br><span class="line">max_connections=512</span><br><span class="line"></span><br><span class="line">accept_threads=1</span><br><span class="line"></span><br><span class="line">work_threads=4</span><br><span class="line"></span><br><span class="line">min_buff_size = 8KB</span><br><span class="line"></span><br><span class="line">max_buff_size = 128KB</span><br><span class="line"></span><br><span class="line">store_lookup=2</span><br><span class="line"></span><br><span class="line">store_group=group2</span><br><span class="line"></span><br><span class="line">store_server=0</span><br><span class="line"></span><br><span class="line">store_path=0</span><br><span class="line"></span><br><span class="line">download_server=0</span><br><span class="line"></span><br><span class="line">reserved_storage_space = 10%</span><br><span class="line"></span><br><span class="line">log_level=info</span><br><span class="line"></span><br><span class="line">run_by_group=</span><br><span class="line"></span><br><span class="line">run_by_user=</span><br><span class="line"></span><br><span class="line">allow_hosts=*</span><br><span class="line"></span><br><span class="line">sync_log_buff_interval = 10</span><br><span class="line"></span><br><span class="line">check_active_interval = 120</span><br><span class="line"></span><br><span class="line">thread_stack_size = 64KB</span><br><span class="line"></span><br><span class="line">storage_ip_changed_auto_adjust = true</span><br><span class="line"></span><br><span class="line">storage_sync_file_max_delay = 86400</span><br><span class="line"></span><br><span class="line">storage_sync_file_max_time = 300</span><br><span class="line"></span><br><span class="line">use_trunk_file = false </span><br><span class="line"></span><br><span class="line">slot_min_size = 256</span><br><span class="line"></span><br><span class="line">slot_max_size = 16MB</span><br><span class="line"></span><br><span class="line">trunk_file_size = 64MB</span><br><span class="line"></span><br><span class="line">trunk_create_file_advance = false</span><br><span class="line"></span><br><span class="line">trunk_create_file_time_base = 02:00</span><br><span class="line"></span><br><span class="line">trunk_create_file_interval = 86400</span><br><span class="line"></span><br><span class="line">trunk_create_file_space_threshold = 20G</span><br><span class="line"></span><br><span class="line">trunk_init_check_occupying = false</span><br><span class="line"></span><br><span class="line">trunk_init_reload_from_binlog = false</span><br><span class="line"></span><br><span class="line">trunk_compress_binlog_min_interval = 0</span><br><span class="line"></span><br><span class="line">use_storage_id = false</span><br><span class="line"></span><br><span class="line">storage_ids_filename = storage_ids.conf</span><br><span class="line"></span><br><span class="line">id_type_in_filename = ip</span><br><span class="line"></span><br><span class="line">store_slave_file_use_link = false</span><br><span class="line"></span><br><span class="line">rotate_error_log = false</span><br><span class="line"></span><br><span class="line">error_log_rotate_time=00:00</span><br><span class="line"></span><br><span class="line">rotate_error_log_size = 0</span><br><span class="line"></span><br><span class="line">log_file_keep_days = 0</span><br><span class="line"></span><br><span class="line">use_connection_pool = false</span><br><span class="line"></span><br><span class="line">connection_pool_max_idle_time = 3600</span><br><span class="line"></span><br><span class="line">http.server_port=8080</span><br><span class="line"></span><br><span class="line">http.check_alive_interval=30</span><br><span class="line"></span><br><span class="line">http.check_alive_type=tcp</span><br><span class="line"></span><br><span class="line">http.check_alive_uri=/status.html</span><br></pre></td></tr></table></figure><p>修改保存后创建软引用</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# ln -s /usr/bin/fdfs_storaged /usr/local/bin</span><br></pre></td></tr></table></figure><h4 id="4-启动tracker，并加入开机启动项"><a href="#4-启动tracker，并加入开机启动项" class="headerlink" title="4.启动tracker，并加入开机启动项"></a>4.启动tracker，并加入开机启动项</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# service fdfs_trackerd start</span><br></pre></td></tr></table></figure><p>进行刚刚创建的tracker目录，发现目录中多了data和log两个目录</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# ll /opt/fastdfs_tracker/</span><br><span class="line">总用量 0</span><br><span class="line">drwxr-xr-x. 2 root root 178 1月 17 13:48 data</span><br><span class="line">drwxr-xr-x. 2 root root 26 1月 17 11:35 logs</span><br></pre></td></tr></table></figure><p>将tracker加入开机启动项</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# echo "service fdfs_trackerd start" |tee -a /etc/rc.d/rc.local</span><br></pre></td></tr></table></figure><p>查看一下tracker的端口监听情况</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# netstat -unltp|grep fdfs</span><br><span class="line">tcp 0 0 0.0.0.0:22122 0.0.0.0:* LISTEN 3088/fdfs_trackerd</span><br></pre></td></tr></table></figure><h1 id="三、安装Storage"><a href="#三、安装Storage" class="headerlink" title="三、安装Storage"></a>三、安装Storage</h1><h4 id="1-配置storage工作目录"><a href="#1-配置storage工作目录" class="headerlink" title="1.配置storage工作目录"></a>1.配置storage工作目录</h4><p>由于storage还需要一个目录用来存储数据，因此多建了两个目录fastdfs_storage_data,fastdfs_storage<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost opt]# mkdir fastdfs_storage</span><br><span class="line">[root@localhost opt]# mkdir fastdfs_storage_data</span><br><span class="line">[root@localhost opt]# ll</span><br><span class="line">总用量 0</span><br><span class="line">drwxr-xr-x. 4 root root 30 1月 17 11:45 fastdfs_storage</span><br><span class="line">drwxr-xr-x. 3 root root 18 1月 17 11:45 fastdfs_storage_data</span><br><span class="line">drwxr-xr-x. 4 root root 30 1月 17 11:35 fastdfs_tracker</span><br></pre></td></tr></table></figure></p><h4 id="2-配置storage文件"><a href="#2-配置storage文件" class="headerlink" title="2.配置storage文件"></a>2.配置storage文件</h4><p>修改 /etc/fdfs 目录下 storage.conf 文件<br>修改要点如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">1.disabled=false </span><br><span class="line">2.group_name=group1 #组名，根据实际情况修改 </span><br><span class="line">3.port=23000 #设置storage的端口号，默认是23000，同一个组的storage端口号必须一致 </span><br><span class="line">4.base_path=/opt/fastdfs_storage #设置storage数据文件和日志目录 </span><br><span class="line">5.store_path_count=1 #存储路径个数，需要和store_path个数匹配 </span><br><span class="line">6.store_path0=/opt/fastdfs_storage_data #实际文件存储路径 </span><br><span class="line">7.tracker_server=192.168.43.60:22122 #我CentOS7的ip地址 </span><br><span class="line">8.http.server_port=8888 #设置 http 端口号</span><br></pre></td></tr></table></figure><p>完整信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line">disabled=false</span><br><span class="line">group_name=group1</span><br><span class="line">bind_addr= 0.0.0.0</span><br><span class="line">client_bind=true</span><br><span class="line">port=23000</span><br><span class="line">connect_timeout=30</span><br><span class="line">network_timeout=60</span><br><span class="line">heart_beat_interval=30</span><br><span class="line">stat_report_interval=60</span><br><span class="line">base_path=/opt/fastdfs_storage</span><br><span class="line">max_connections=256</span><br><span class="line">buff_size = 256KB</span><br><span class="line">accept_threads=1</span><br><span class="line">work_threads=4</span><br><span class="line">disk_rw_separated = true</span><br><span class="line">disk_reader_threads = 1</span><br><span class="line">disk_writer_threads = 1</span><br><span class="line">sync_wait_msec=50</span><br><span class="line">sync_interval=0</span><br><span class="line">sync_start_time=00:00</span><br><span class="line">sync_end_time=23:59</span><br><span class="line">write_mark_file_freq=500</span><br><span class="line">store_path_count=1</span><br><span class="line">store_path0=/opt/fastdfs_storage_data</span><br><span class="line">subdir_count_per_path=256</span><br><span class="line">tracker_server=192.168.43.60:22122</span><br><span class="line">log_level=info</span><br><span class="line">run_by_group=</span><br><span class="line">run_by_user=</span><br><span class="line">allow_hosts=*</span><br><span class="line">file_distribute_path_mode=0</span><br><span class="line">file_distribute_rotate_count=100</span><br><span class="line">fsync_after_written_bytes=0</span><br><span class="line">sync_log_buff_interval=10</span><br><span class="line">sync_binlog_buff_interval=10</span><br><span class="line">sync_stat_file_interval=300</span><br><span class="line">thread_stack_size=512KB</span><br><span class="line">upload_priority=10</span><br><span class="line">if_alias_prefix=</span><br><span class="line">check_file_duplicate=0</span><br><span class="line">file_signature_method=hash</span><br><span class="line">key_namespace=FastDFS</span><br><span class="line">keep_alive=0</span><br><span class="line">use_access_log = false</span><br><span class="line">rotate_access_log = false</span><br><span class="line">access_log_rotate_time=00:00</span><br><span class="line">rotate_error_log = false</span><br><span class="line">error_log_rotate_time=00:00</span><br><span class="line">rotate_access_log_size = 0</span><br><span class="line">rotate_error_log_size = 0</span><br><span class="line">log_file_keep_days = 0</span><br><span class="line">file_sync_skip_invalid_record=false</span><br><span class="line">use_connection_pool = false</span><br><span class="line">connection_pool_max_idle_time = 3600</span><br><span class="line">http.domain_name=</span><br><span class="line">http.server_port=8888</span><br></pre></td></tr></table></figure><p>修改保存后创建软引用<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# ln -s /usr/bin/fdfs_storaged /usr/local/bin</span><br></pre></td></tr></table></figure></p><h4 id="3-启动Storage"><a href="#3-启动Storage" class="headerlink" title="3.启动Storage"></a>3.启动Storage</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# service fdfs_storaged start</span><br></pre></td></tr></table></figure><p>设置开机启动：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# echo "service fdfs_storaged start" |tee -a /etc/rc.d/rc.local</span><br></pre></td></tr></table></figure><p>查看一下服务是否启动</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# netstat -unltp | grep fdfs </span><br><span class="line">tcp 0 0 0.0.0.0:22122 0.0.0.0:* LISTEN 3088/fdfs_trackerd </span><br><span class="line">tcp 0 0 0.0.0.0:23000 0.0.0.0:* LISTEN 3139/fdfs_storaged</span><br></pre></td></tr></table></figure><h4 id="4-校验整合"><a href="#4-校验整合" class="headerlink" title="4.校验整合"></a>4.校验整合</h4><p>到这里，fastdfs的东西都已安装完成，最后我们还要确定一下，storage是否注册到了tracker中去。<br>查看命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# /usr/bin/fdfs_monitor /etc/fdfs/storage.conf</span><br></pre></td></tr></table></figure><p>成功后可以看到：<br>ip_addr = 192.168.43.60 (localhost.localdomain) ACTIVE</p><h1 id="四、测试"><a href="#四、测试" class="headerlink" title="四、测试"></a>四、测试</h1><h4 id="1-配置客户端"><a href="#1-配置客户端" class="headerlink" title="1.配置客户端"></a>1.配置客户端</h4><p>修改 /etc/fdfs/目录下的client.conf 文件<br>修改要点为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">base_path=/opt/fastdfs_tracker #tracker服务器文件路径</span><br><span class="line">tracker_server=192.168.43.60:22122 #tracker服务器IP地址和端口号</span><br><span class="line">http.tracker_server_port=8080 # tracker 服务器的 http端口号，必须和tracker的设置对应起来</span><br></pre></td></tr></table></figure><p>完整client.conf 文件信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">connect_timeout=30</span><br><span class="line">network_timeout=60</span><br><span class="line">base_path=/opt/fastdfs_tracker</span><br><span class="line">tracker_server=192.168.43.60:22122</span><br><span class="line">log_level=info</span><br><span class="line">use_connection_pool = false</span><br><span class="line">connection_pool_max_idle_time = 3600</span><br><span class="line">load_fdfs_parameters_from_tracker=false</span><br><span class="line">use_storage_id = false</span><br><span class="line">storage_ids_filename = storage_ids.conf</span><br><span class="line">http.tracker_server_port=8080</span><br></pre></td></tr></table></figure><p>模拟上传<br>从个人用户目录上传一个图片，进行测试</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# fdfs_upload_file /etc/fdfs/client.conf /home/zhangyongliang/9408.jpg #这后面放的是图片的位置</span><br></pre></td></tr></table></figure><p>成功后会返回图片存储路径</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# fdfs_upload_file /etc/fdfs/client.conf /home/zhangyongliang/9408.jpg </span><br><span class="line">group1/M00/00/00/wKgrPFpe9OqAWsHxAAH5yvc2jn8251.jpg</span><br></pre></td></tr></table></figure><p>组名：group1<br>磁盘：M00<br>目录：00/00<br>文件名称：wKgrPFpe9OqAWsHxAAH5yvc2jn8251.jpg<br>定位上传的文件位置如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# ll /opt/fastdfs_storage_data/data/00/00/</span><br><span class="line">总用量 256</span><br><span class="line">-rw-r--r--. 1 root root 129482 1月 17 15:02 wKgrPFpe9OqAWsHxAAH5yvc2jn8251.jpg</span><br><span class="line">-rw-r--r--. 1 root root 129482 1月 17 11:53 wKgrPFpeyM2ATkGUAAH5yvc2jn8013.jpg</span><br></pre></td></tr></table></figure><p>实际文件存储路径下有创建好的多级目录。data下有256个1级目录，每级目录下又有256个2级子目录，总共65536个文件，新写的文件会以hash的方式被路由到其中某个子目录下，然后将文件数据直接作为一个本地文件存储到该目录中。<br><br>如果要访问刚上传的图片，我们得需要结合nginx来实现</p><h1 id="五、安装Nginx并实现配置"><a href="#五、安装Nginx并实现配置" class="headerlink" title="五、安装Nginx并实现配置"></a>五、安装Nginx并实现配置</h1><h4 id="1-安装Nginx依赖环境"><a href="#1-安装Nginx依赖环境" class="headerlink" title="1.安装Nginx依赖环境"></a>1.安装Nginx依赖环境</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# yum -y install pcre pcre-devel </span><br><span class="line">[root@localhost fdfs]# yum -y install zlib zlib-devel </span><br><span class="line">[root@localhost fdfs]# yum -y install openssl openssl-devel</span><br></pre></td></tr></table></figure><h4 id="2-安装nginx并添加fastdfs-nginx-module"><a href="#2-安装nginx并添加fastdfs-nginx-module" class="headerlink" title="2.安装nginx并添加fastdfs-nginx-module"></a>2.安装nginx并添加fastdfs-nginx-module</h4><p>解压nginx和fastdfs-nginx-module</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# tar -zxvf nginx-1.10.3.tar.gz</span><br><span class="line">[root@localhost fdfs]# unzip fastdfs-nginx-module-master.zip</span><br></pre></td></tr></table></figure><p>解压后进入nginx目录编译安装nginx,并添加fastdfs-nginx-module</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost nginx-1.10.3]# ./configure --prefix=/usr/local/nginx --add-module=/home/zhangyongliang/apps/fastdfs-nginx-module-master/src #解压后fastdfs-nginx-module所在的位置</span><br></pre></td></tr></table></figure><p>之后进行编译和安装</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost nginx-1.10.3]# make && make isntall</span><br></pre></td></tr></table></figure><p>安装成功后，nginx会安装在/usr/local/nginx,安装后查看</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# ll /usr/local/nginx/</span><br><span class="line">总用量 8</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 client_body_temp</span><br><span class="line">drwxr-xr-x. 2 root root 4096 1月 17 13:17 conf</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 fastcgi_temp</span><br><span class="line">drwxr-xr-x. 2 root root 40 1月 17 13:17 html</span><br><span class="line">drwxr-xr-x. 2 root root 58 1月 17 13:49 logs</span><br><span class="line">-rw-r--r--. 1 root root 1156 1月 17 13:29 nginx.conf</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 proxy_temp</span><br><span class="line">drwxr-xr-x. 2 root root 19 1月 17 13:17 sbin</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 scgi_temp</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 uwsgi_temp</span><br></pre></td></tr></table></figure><p>安装成功后，nginx尚未运行时，nginx文件夹没有临时文件夹，例如fastcgi_temp这些文件。</p><h4 id="3-配置Storage-Nginx"><a href="#3-配置Storage-Nginx" class="headerlink" title="3.配置Storage Nginx"></a>3.配置Storage Nginx</h4><p>修改Nginx 目录下 conf 的配置文件nginx.conf,新增location信息，具体如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">server {</span><br><span class="line"> listen 9991;</span><br><span class="line"> server_name localhost;</span><br><span class="line"></span><br><span class="line"> location / {</span><br><span class="line"> root html;</span><br><span class="line"> index index.html index.htm;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> location ~/group1/M00 {</span><br><span class="line"> root /opt/fastdfs_storage/data;</span><br><span class="line"> ngx_fastdfs_module;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> location = /50x.html {</span><br><span class="line"> root html;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>然后进入FastDFS安装时的解压过的目录，将http.conf和mime.types拷贝到/etc/fdfs目录下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# cp http.conf mime.types /etc/fdfs/</span><br><span class="line">[root@localhost src]# cp mime.types /etc/fdfs/</span><br></pre></td></tr></table></figure></p><p>另外还需要把fastdfs-nginx-module安装目录中src目录下的mod_fastdfs.conf也拷贝到/etc/fdfs目录下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# cp mod_fastdfs.conf /etc/fdfs/</span><br></pre></td></tr></table></figure><p>对刚刚拷贝的mod_fastdfs.conf文件进行修改：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">base_path=/opt/fastdfs_storage #保存日志目录</span><br><span class="line">tracker_server=192.168.43.60:22122 #tracker服务器的IP地址以及端口号</span><br><span class="line">storage_server_port=23000 #storage服务器的端口号</span><br><span class="line">url_have_group_name = true #文件 url 中是否有 group 名</span><br><span class="line">store_path0=/opt/fastdfs_storage_data #存储路径</span><br><span class="line">group_count = 1 #设置组的个数</span><br></pre></td></tr></table></figure><p>在文件的最后，设置group<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[group1]</span><br><span class="line">group_name=group1</span><br><span class="line">storage_server_port=23000</span><br><span class="line">store_path_count=1</span><br><span class="line">store_path0=/opt/fastdfs_storage_data</span><br></pre></td></tr></table></figure></p><p>创建M00至storage存储目录的符号连接：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ln -s /opt/fastdfs_storage_data/data/ /opt/fastdfs_storage_data/data/M00</span><br></pre></td></tr></table></figure><p>启动Nginx：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# /usr/local/nginx/sbin/nginx</span><br></pre></td></tr></table></figure><p>访问Nginx是否启动</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# curl localhost:9991</span><br><span class="line"><!DOCTYPE html></span><br><span class="line"><html></span><br><span class="line"><head></span><br><span class="line"><title>Welcome to nginx!</title></span><br><span class="line"><style></span><br><span class="line"> body {</span><br><span class="line"> width: 35em;</span><br><span class="line"> margin: 0 auto;</span><br><span class="line"> font-family: Tahoma, Verdana, Arial, sans-serif;</span><br><span class="line"> }</span><br><span class="line"></style></span><br><span class="line"></head></span><br><span class="line"><body></span><br><span class="line"><h1>Welcome to nginx!</h1></span><br><span class="line"><p>If you see this page, the nginx web server is successfully installed and</span><br><span class="line">working. Further configuration is required.</p></span><br><span class="line"></span><br><span class="line"><p>For online documentation and support please refer to</span><br><span class="line"><a href="http://nginx.org/">nginx.org</a>.<br/></span><br><span class="line">Commercial support is available at</span><br><span class="line"><a href="http://nginx.com/">nginx.com</a>.</p></span><br><span class="line"></span><br><span class="line"><p><em>Thank you for using nginx.</em></p></span><br><span class="line"></body></span><br><span class="line"></html></span><br></pre></td></tr></table></figure><h4 id="4-配置Tracker-Nginx"><a href="#4-配置Tracker-Nginx" class="headerlink" title="4.配置Tracker Nginx"></a>4.配置Tracker Nginx</h4><p>在nginx.conf 文件添加一个虚拟机</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">upstream fdfs_group1 {</span><br><span class="line"> server 127.0.0.1:9991;</span><br><span class="line"> }</span><br><span class="line"> server {</span><br><span class="line"> listen 80;</span><br><span class="line"> server_name localhost;</span><br><span class="line"> location /group1/M00 {</span><br><span class="line"> proxy_pass http://fdfs_group1;</span><br><span class="line"> }</span><br><span class="line"> error_page 500 502 503 504 /50x.html;</span><br><span class="line"> location = /50x.html {</span><br><span class="line"> root html;</span><br><span class="line"> }</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><p>完整nginx.conf配置文件如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">#user nobody;</span><br><span class="line">worker_processes 1;</span><br><span class="line"></span><br><span class="line">#error_log logs/error.log;</span><br><span class="line">#error_log logs/error.log notice;</span><br><span class="line">#error_log logs/error.log info;</span><br><span class="line"></span><br><span class="line">#pid logs/nginx.pid;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">events {</span><br><span class="line"> worker_connections 1024;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">http {</span><br><span class="line"> include mime.types;</span><br><span class="line"> default_type application/octet-stream;</span><br><span class="line"></span><br><span class="line"> #log_format main '$remote_addr - $remote_user [$time_local] "$request" '</span><br><span class="line"> # '$status $body_bytes_sent "$http_referer" '</span><br><span class="line"> # '"$http_user_agent" "$http_x_forwarded_for"';</span><br><span class="line"></span><br><span class="line"> #access_log logs/access.log main;</span><br><span class="line"></span><br><span class="line"> sendfile on;</span><br><span class="line"> #tcp_nopush on;</span><br><span class="line"></span><br><span class="line"> #keepalive_timeout 0;</span><br><span class="line"> keepalive_timeout 65;</span><br><span class="line"></span><br><span class="line"> #gzip on;</span><br><span class="line"></span><br><span class="line"> server {</span><br><span class="line"> listen 9991;</span><br><span class="line"> server_name localhost;</span><br><span class="line"></span><br><span class="line"> #charset koi8-r;</span><br><span class="line"></span><br><span class="line"> #access_log logs/host.access.log main;</span><br><span class="line"></span><br><span class="line"> location / {</span><br><span class="line"> root html;</span><br><span class="line"> index index.html index.htm;</span><br><span class="line"> }</span><br><span class="line"> location ~/group1/M00 {</span><br><span class="line"> root /opt/fastdfs_storage/data;</span><br><span class="line"> ngx_fastdfs_module;</span><br><span class="line"> }</span><br><span class="line"> #error_page 404 /404.html;</span><br><span class="line"></span><br><span class="line"> # redirect server error pages to the static page /50x.html</span><br><span class="line"> #</span><br><span class="line"> error_page 500 502 503 504 /50x.html;</span><br><span class="line"> location = /50x.html {</span><br><span class="line"> root html;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> # proxy the PHP scripts to Apache listening on 127.0.0.1:80</span><br><span class="line"> #</span><br><span class="line"> #location ~ \.php$ {</span><br><span class="line"> # proxy_pass http://127.0.0.1;</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"> # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000</span><br><span class="line"> #</span><br><span class="line"> #location ~ \.php$ {</span><br><span class="line"> # root html;</span><br><span class="line"> # fastcgi_pass 127.0.0.1:9000;</span><br><span class="line"> # fastcgi_index index.php;</span><br><span class="line"> # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;</span><br><span class="line"> # include fastcgi_params;</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"> # deny access to .htaccess files, if Apache's document root</span><br><span class="line"> # concurs with nginx's one</span><br><span class="line"> #</span><br><span class="line"> #location ~ /\.ht {</span><br><span class="line"> # deny all;</span><br><span class="line"> #}</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> upstream fdfs_group1 {</span><br><span class="line"> server 127.0.0.1:9991;</span><br><span class="line"> }</span><br><span class="line"> server {</span><br><span class="line"> listen 80;</span><br><span class="line"> server_name localhost;</span><br><span class="line"> location /group1/M00 {</span><br><span class="line"> proxy_pass http://fdfs_group1;</span><br><span class="line"> }</span><br><span class="line"> error_page 500 502 503 504 /50x.html;</span><br><span class="line"> location = /50x.html {</span><br><span class="line"> root html;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> # another virtual host using mix of IP-, name-, and port-based configuration</span><br><span class="line"> #</span><br><span class="line"> #server {</span><br><span class="line"> # listen 8000;</span><br><span class="line"> # listen somename:8080;</span><br><span class="line"> # server_name somename alias another.alias;</span><br><span class="line"></span><br><span class="line"> # location / {</span><br><span class="line"> # root html;</span><br><span class="line"> # index index.html index.htm;</span><br><span class="line"> # }</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> # HTTPS server</span><br><span class="line"> #</span><br><span class="line"> #server {</span><br><span class="line"> # listen 443 ssl;</span><br><span class="line"> # server_name localhost;</span><br><span class="line"></span><br><span class="line"> # ssl_certificate cert.pem;</span><br><span class="line"> # ssl_certificate_key cert.key;</span><br><span class="line"></span><br><span class="line"> # ssl_session_cache shared:SSL:1m;</span><br><span class="line"> # ssl_session_timeout 5m;</span><br><span class="line"></span><br><span class="line"> # ssl_ciphers HIGH:!aNULL:!MD5;</span><br><span class="line"> # ssl_prefer_server_ciphers on;</span><br><span class="line"></span><br><span class="line"> # location / {</span><br><span class="line"> # root html;</span><br><span class="line"> # index index.html index.htm;</span><br><span class="line"> # }</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>将Nginx重新启动</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# /usr/local/nginx/sbin/nginx -s reload</span><br></pre></td></tr></table></figure><p>访问Nginx是否已经启动</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# curl localhost:9991</span><br><span class="line"><!DOCTYPE html></span><br><span class="line"><html></span><br><span class="line"><head></span><br><span class="line"><title>Welcome to nginx!</title></span><br><span class="line"><style></span><br><span class="line"> body {</span><br><span class="line"> width: 35em;</span><br><span class="line"> margin: 0 auto;</span><br><span class="line"> font-family: Tahoma, Verdana, Arial, sans-serif;</span><br><span class="line"> }</span><br><span class="line"></style></span><br><span class="line"></head></span><br><span class="line"><body></span><br><span class="line"><h1>Welcome to nginx!</h1></span><br><span class="line"><p>If you see this page, the nginx web server is successfully installed and</span><br><span class="line">working. Further configuration is required.</p></span><br><span class="line"></span><br><span class="line"><p>For online documentation and support please refer to</span><br><span class="line"><a href="http://nginx.org/">nginx.org</a>.<br/></span><br><span class="line">Commercial support is available at</span><br><span class="line"><a href="http://nginx.com/">nginx.com</a>.</p></span><br><span class="line"></span><br><span class="line"><p><em>Thank you for using nginx.</em></p></span><br><span class="line"></body></span><br><span class="line"></html></span><br><span class="line">[root@localhost src]# curl localhost</span><br><span class="line"><!DOCTYPE html></span><br><span class="line"><html></span><br><span class="line"><head></span><br><span class="line"><title>Welcome to nginx!</title></span><br><span class="line"><style></span><br><span class="line"> body {</span><br><span class="line"> width: 35em;</span><br><span class="line"> margin: 0 auto;</span><br><span class="line"> font-family: Tahoma, Verdana, Arial, sans-serif;</span><br><span class="line"> }</span><br><span class="line"></style></span><br><span class="line"></head></span><br><span class="line"><body></span><br><span class="line"><h1>Welcome to nginx!</h1></span><br><span class="line"><p>If you see this page, the nginx web server is successfully installed and</span><br><span class="line">working. Further configuration is required.</p></span><br><span class="line"></span><br><span class="line"><p>For online documentation and support please refer to</span><br><span class="line"><a href="http://nginx.org/">nginx.org</a>.<br/></span><br><span class="line">Commercial support is available at</span><br><span class="line"><a href="http://nginx.com/">nginx.com</a>.</p></span><br><span class="line"></span><br><span class="line"><p><em>Thank you for using nginx.</em></p></span><br><span class="line"></body></span><br><span class="line"></html></span><br></pre></td></tr></table></figure><h4 id="5-HTTP访问图片资源"><a href="#5-HTTP访问图片资源" class="headerlink" title="5.HTTP访问图片资源"></a>5.HTTP访问图片资源</h4><p> 访问图片资源路径<a href="http://192.168.43.60/group1/M00/00/00/wKgrPFpeyM2ATkGUAAH5yvc2jn8013.jpg" target="_blank" rel="noopener">http://192.168.43.60/group1/M00/00/00/wKgrPFpeyM2ATkGUAAH5yvc2jn8013.jpg</a><br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-563976cb823c2804.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="FastDFS资源访问截图" title=""> </div> <div class="image-caption">FastDFS资源访问截图</div> </figure><br>可以看到已上传的图片，说明你已成功实现了FastDFS单机的文件系统搭建。</p><h3 id="补充说明："><a href="#补充说明：" class="headerlink" title="补充说明："></a>补充说明：</h3><p>如果Tracker 服务、Storage服务、Nginx服务开机后没有自启动，请执行一下操作并进行重启</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost ~]# chkconfig --add fdfs_trackerd</span><br><span class="line">[root@localhost ~]# chkconfig --add fdfs_storaged</span><br></pre></td></tr></table></figure><p>编辑目录下的/etc/rc.d/rc.local,内容如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">#!/bin/bash</span><br><span class="line"># THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES</span><br><span class="line">#</span><br><span class="line"># It is highly advisable to create own systemd services or udev rules</span><br><span class="line"># to run scripts during boot instead of using this file.</span><br><span class="line">#</span><br><span class="line"># In contrast to previous versions due to parallel execution during boot</span><br><span class="line"># this script will NOT be run after all other services.</span><br><span class="line">#</span><br><span class="line"># Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure</span><br><span class="line"># that this script will be executed during boot.</span><br><span class="line"></span><br><span class="line">touch /var/lock/subsys/local</span><br><span class="line">/usr/local/nginx/sbin/nginx</span><br></pre></td></tr></table></figure></p><p>主要增加了Nginx的启动，之后进行文件生效，重新启动系统</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost ~]# chmod +x /etc/rc.d/rc.local</span><br><span class="line">[root@localhost ~]# source /etc/rc.d/rc.local </span><br><span class="line">[root@localhost ~]# reboot</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h1 id="此例为在CentOS7-2-单机上搭建一个FastDFS-文件管理系统"><a href="#此例为在CentOS7-2-单机上搭建一个FastDFS-文件管理系统" class="headerlink" title="此例为在CentOS7.2 单机上搭建一个F
</summary>
</entry>
<entry>
<title>CentOS 7.2搭建FastDFS 分布式文件系统，实现高可用集群</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/CentOS_7.2%E6%90%AD%E5%BB%BAFastDFS_%E5%88%86%E5%B8%83%E5%BC%8F%E6%96%87%E4%BB%B6%E7%B3%BB%E7%BB%9F%EF%BC%8C%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8%E9%9B%86%E7%BE%A4/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/CentOS_7.2搭建FastDFS_分布式文件系统，实现高可用集群/</id>
<published>2018-02-01T05:57:24.762Z</published>
<updated>2018-02-01T05:56:27.000Z</updated>
<content type="html"><![CDATA[<h3 id="分布式集群搭建结构"><a href="#分布式集群搭建结构" class="headerlink" title="分布式集群搭建结构"></a>分布式集群搭建结构</h3><ul><li>双Tracker</li><li>2组Group</li><li>轮询存储策略</li><li>Keepalived+Nginx高可用</li><li>Nginx缓存</li><li>4个存储节点</li></ul><h1 id="一、-集群规划清单"><a href="#一、-集群规划清单" class="headerlink" title="一、 集群规划清单"></a>一、 集群规划清单</h1><h4 id="1-安装清单"><a href="#1-安装清单" class="headerlink" title="1.安装清单"></a>1.安装清单</h4><table><thead><tr><th style="text-align:center">软件名称</th><th style="text-align:center">版本</th><th style="text-align:center">百度云盘存放名称</th></tr></thead><tbody><tr><td style="text-align:center">FastDFS</td><td style="text-align:center">5.11</td><td style="text-align:center">fastdfs-5.11.zip</td></tr><tr><td style="text-align:center">FastDFS-Nginx-module</td><td style="text-align:center">无</td><td style="text-align:center">fastdfs-nginx-module-master.zip</td></tr><tr><td style="text-align:center">LibFastCommon</td><td style="text-align:center">1.0.36</td><td style="text-align:center">libfastcommon-1.0.36.zip</td></tr><tr><td style="text-align:center">nginx</td><td style="text-align:center">1.10.3</td><td style="text-align:center">nginx-1.10.3.tar.gz</td></tr><tr><td style="text-align:center">nginx-pure-cache</td><td style="text-align:center">2.3</td><td style="text-align:center">ngx_cache_purge-2.3.tar.gz</td></tr></tbody></table><p><strong>安装所需文件均上传到百度云盘，位置<a href="https://pan.baidu.com/s/1htBiHoS" target="_blank" rel="noopener">：FastDFS百度云盘</a></strong></p><h4 id="2-集群规划"><a href="#2-集群规划" class="headerlink" title="2.集群规划"></a>2.集群规划</h4><table><thead><tr><th style="text-align:center">虚拟机</th><th style="text-align:center">IP</th><th style="text-align:center">说明</th></tr></thead><tbody><tr><td style="text-align:center">Keepalived+Nginx1[Master]</td><td style="text-align:center">192.168.43.101</td><td style="text-align:center">Nginx Server 01</td></tr><tr><td style="text-align:center">Keeepalived+Nginx[Backup]</td><td style="text-align:center">192.168.43.102</td><td style="text-align:center">Nginx Server 02</td></tr><tr><td style="text-align:center">VIP</td><td style="text-align:center">192.168.43.150</td><td style="text-align:center">虚拟漂移IP</td></tr><tr><td style="text-align:center">Tracker01</td><td style="text-align:center">192.168.43.70</td><td style="text-align:center">Tracker01服务器</td></tr><tr><td style="text-align:center">Tracker02</td><td style="text-align:center">192.168.43.71</td><td style="text-align:center">Tracker02服务器</td></tr><tr><td style="text-align:center">Storage01</td><td style="text-align:center">192.168.43.72</td><td style="text-align:center">Storage01服务器【group1】</td></tr><tr><td style="text-align:center">Storage02</td><td style="text-align:center">192.168.43.73</td><td style="text-align:center">Storage02服务器【group1】</td></tr><tr><td style="text-align:center">Storage03</td><td style="text-align:center">192.168.43.74</td><td style="text-align:center">Storage03服务器【group2】</td></tr><tr><td style="text-align:center">Storage04</td><td style="text-align:center">192.168.43.75</td><td style="text-align:center">Storage04服务器【group2】</td></tr></tbody></table><p>整体架构图如下图所示：<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-ed325e68032fbca3?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="图片来源：CSDN作者 liuyazhuang" title=""> </div> <div class="image-caption">图片来源：CSDN作者 liuyazhuang</div> </figure></p><h1 id="二、集群安装"><a href="#二、集群安装" class="headerlink" title="二、集群安装"></a>二、集群安装</h1><p><strong>以下操作均在关闭所有节点防火墙进行的，请根据个人情况开启相关端口或关闭防火墙</strong></p><h4 id="1-安装LibFastCommon-FastDFS模块"><a href="#1-安装LibFastCommon-FastDFS模块" class="headerlink" title="1.安装LibFastCommon/FastDFS模块"></a>1.安装LibFastCommon/FastDFS模块</h4><p><strong>执行节点 Tracker01、Tracker02、Storage01、Storage03、Storage04</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastDFS]# unzip libfastcommon-1.0.36.zip</span><br></pre></td></tr></table></figure></p><p>解压后目录如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# ll</span><br><span class="line">[root@localhost libfastcommon-1.0.36]# ll</span><br><span class="line">总用量 32</span><br><span class="line">drwxr-xr-x. 2 root root 117 4月 5 2017 doc</span><br><span class="line">-rw-r--r--. 1 root root 8005 4月 5 2017 HISTORY</span><br><span class="line">-rw-r--r--. 1 root root 566 4月 5 2017 INSTALL</span><br><span class="line">-rw-r--r--. 1 root root 1606 4月 5 2017 libfastcommon.spec</span><br><span class="line">-rwxr-xr-x. 1 root root 3099 4月 5 2017 make.sh</span><br><span class="line">drwxr-xr-x. 2 root root 191 4月 5 2017 php-fastcommon</span><br><span class="line">-rw-r--r--. 1 root root 2763 4月 5 2017 README</span><br><span class="line">drwxr-xr-x. 3 root root 4096 1月 17 11:21 src</span><br></pre></td></tr></table></figure></p><p>安装C编译工具 gcc</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# yum -y install gcc-c++</span><br></pre></td></tr></table></figure><ul><li>安装装LibFastCommon</li></ul><p>编译libfastcommon软件并安装</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# ./make.sh && ./make.sh install</span><br></pre></td></tr></table></figure><p>为libcommon 创建软链接到/usr/local/lib目录下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so</span><br><span class="line">[root@localhost fastdfs-5.11]# ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so</span><br><span class="line">[root@localhost fastdfs-5.11]# ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so</span><br><span class="line">[root@localhost fastdfs-5.11]# ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so</span><br></pre></td></tr></table></figure><ul><li>安装FastDFS<br>解压安装包<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastDFS]# unzip fastdfs-5.11.zip</span><br></pre></td></tr></table></figure></li></ul><p>进入解压目录并进行编译和安装</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastDFS]# cd fastdfs-5.11/</span><br><span class="line">[root@localhost fastdfs-5.11]# ./make.sh && ./make.sh install</span><br></pre></td></tr></table></figure><p>安装成功后，FastDFS会安装在/etc/fdfs目录下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs-5.11]# ll /etc/fdfs/</span><br><span class="line">总用量 76</span><br><span class="line">-rw-r--r--. 1 root root 316 1月 17 11:47 client.conf</span><br><span class="line">-rw-r--r--. 1 root root 1461 1月 17 11:25 client.conf.sample</span><br><span class="line">-rw-r--r--. 1 root root 955 1月 17 13:20 http.conf</span><br><span class="line">-rw-r--r--. 1 root root 31172 1月 17 13:21 mime.types</span><br><span class="line">-rw-r--r--. 1 root root 3716 1月 17 12:57 mod_fastdfs.conf</span><br><span class="line">-rw-r--r--. 1 root root 1278 1月 17 11:40 storage.conf</span><br><span class="line">-rw-r--r--. 1 root root 7927 1月 17 11:25 storage.conf.sample</span><br><span class="line">-rw-r--r--. 1 root root 105 1月 17 11:25 storage_ids.conf.sample</span><br><span class="line">-rw-r--r--. 1 root root 1356 1月 17 11:34 tracker.conf</span><br><span class="line">-rw-r--r--. 1 root root 7389 1月 17 11:25 tracker.conf.sample</span><br></pre></td></tr></table></figure><p>我们需要把这三个示例文件复制一份，去掉.sample</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# cp client.conf.sample client.conf</span><br><span class="line">[root@localhost fdfs]# cp storage.conf.sample storage.conf</span><br><span class="line">[root@localhost fdfs]# cp tracker.conf.sample tracker.conf</span><br></pre></td></tr></table></figure><p>FastDFS安装结束</p><h3 id="2-安装Tracker并实现节点信息配置"><a href="#2-安装Tracker并实现节点信息配置" class="headerlink" title="2.安装Tracker并实现节点信息配置"></a>2.安装Tracker并实现节点信息配置</h3><p><strong>执行节点 Tracker01、Tracker02</strong></p><ul><li>创建tracker工作目录</li></ul><p>此目录用于保存tracker 的data和log</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# mkdir /opt/fastdfs_tracker</span><br></pre></td></tr></table></figure><ul><li>配置tracker</li></ul><p>配置 /etc/fdfs目录下tracker.conf<br>主要实现以下5个配置内容：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">1.disabled=false </span><br><span class="line">2.port=22122 #默认端口号 </span><br><span class="line">3.base_path=/opt/fastdfs_tracker #我刚刚创建的目录 </span><br><span class="line">4.http.server_port=8080 #默认端口是8080</span><br><span class="line">5.store_lookup=0 #采用轮询策略进行存储，0 轮询 1：始终定向到某个group 2：负载进行存储文件</span><br></pre></td></tr></table></figure><p>完整tracker.conf 文件信息如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br></pre></td><td class="code"><pre><span class="line">disabled=false</span><br><span class="line"></span><br><span class="line">bind_addr= 0.0.0.0</span><br><span class="line"></span><br><span class="line">port=22122</span><br><span class="line"></span><br><span class="line">connect_timeout=30</span><br><span class="line"></span><br><span class="line">network_timeout=60</span><br><span class="line"></span><br><span class="line">base_path=/opt/fastdfs_tracker</span><br><span class="line"></span><br><span class="line">max_connections=512</span><br><span class="line"></span><br><span class="line">accept_threads=1</span><br><span class="line"></span><br><span class="line">work_threads=4</span><br><span class="line"></span><br><span class="line">min_buff_size = 8KB</span><br><span class="line"></span><br><span class="line">max_buff_size = 128KB</span><br><span class="line"></span><br><span class="line">store_lookup=0</span><br><span class="line"></span><br><span class="line">store_group=group2</span><br><span class="line"></span><br><span class="line">store_server=0</span><br><span class="line"></span><br><span class="line">store_path=0</span><br><span class="line"></span><br><span class="line">download_server=0</span><br><span class="line"></span><br><span class="line">reserved_storage_space = 10%</span><br><span class="line"></span><br><span class="line">log_level=info</span><br><span class="line"></span><br><span class="line">run_by_group=</span><br><span class="line"></span><br><span class="line">run_by_user=</span><br><span class="line"></span><br><span class="line">allow_hosts=*</span><br><span class="line"></span><br><span class="line">sync_log_buff_interval = 10</span><br><span class="line"></span><br><span class="line">check_active_interval = 120</span><br><span class="line"></span><br><span class="line">thread_stack_size = 64KB</span><br><span class="line"></span><br><span class="line">storage_ip_changed_auto_adjust = true</span><br><span class="line"></span><br><span class="line">storage_sync_file_max_delay = 86400</span><br><span class="line"></span><br><span class="line">storage_sync_file_max_time = 300</span><br><span class="line"></span><br><span class="line">use_trunk_file = false </span><br><span class="line"></span><br><span class="line">slot_min_size = 256</span><br><span class="line"></span><br><span class="line">slot_max_size = 16MB</span><br><span class="line"></span><br><span class="line">trunk_file_size = 64MB</span><br><span class="line"></span><br><span class="line">trunk_create_file_advance = false</span><br><span class="line"></span><br><span class="line">trunk_create_file_time_base = 02:00</span><br><span class="line"></span><br><span class="line">trunk_create_file_interval = 86400</span><br><span class="line"></span><br><span class="line">trunk_create_file_space_threshold = 20G</span><br><span class="line"></span><br><span class="line">trunk_init_check_occupying = false</span><br><span class="line"></span><br><span class="line">trunk_init_reload_from_binlog = false</span><br><span class="line"></span><br><span class="line">trunk_compress_binlog_min_interval = 0</span><br><span class="line"></span><br><span class="line">use_storage_id = false</span><br><span class="line"></span><br><span class="line">storage_ids_filename = storage_ids.conf</span><br><span class="line"></span><br><span class="line">id_type_in_filename = ip</span><br><span class="line"></span><br><span class="line">store_slave_file_use_link = false</span><br><span class="line"></span><br><span class="line">rotate_error_log = false</span><br><span class="line"></span><br><span class="line">error_log_rotate_time=00:00</span><br><span class="line"></span><br><span class="line">rotate_error_log_size = 0</span><br><span class="line"></span><br><span class="line">log_file_keep_days = 0</span><br><span class="line"></span><br><span class="line">use_connection_pool = false</span><br><span class="line"></span><br><span class="line">connection_pool_max_idle_time = 3600</span><br><span class="line"></span><br><span class="line">http.server_port=8080</span><br><span class="line"></span><br><span class="line">http.check_alive_interval=30</span><br><span class="line"></span><br><span class="line">http.check_alive_type=tcp</span><br><span class="line"></span><br><span class="line">http.check_alive_uri=/status.html</span><br></pre></td></tr></table></figure></p><p>修改保存后创建软引用</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# ln -s /usr/bin/fdfs_storaged /usr/local/bin</span><br></pre></td></tr></table></figure><ul><li>启动tracker，并加入开机启动项</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# service fdfs_trackerd start</span><br></pre></td></tr></table></figure><p>将tracker加入开机启动项</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# echo "service fdfs_trackerd start" |tee -a /etc/rc.d/rc.local</span><br></pre></td></tr></table></figure><h4 id="3-安装Storage模块并实现配置"><a href="#3-安装Storage模块并实现配置" class="headerlink" title="3.安装Storage模块并实现配置"></a>3.安装Storage模块并实现配置</h4><p><strong>执行节点 Storage01、Storage02、Storage03、Storage04</strong></p><ul><li>建立存储目录</li></ul><p>在存储各节点建了两个目录fastdfs_storage_data,fastdfs_storage</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost opt]# mkdir fastdfs_storage</span><br><span class="line">[root@localhost opt]# mkdir fastdfs_storage_data</span><br><span class="line">[root@localhost opt]# ll</span><br><span class="line">总用量 0</span><br><span class="line">drwxr-xr-x. 4 root root 30 1月 17 11:45 fastdfs_storage</span><br><span class="line">drwxr-xr-x. 3 root root 18 1月 17 11:45 fastdfs_storage_data</span><br><span class="line">drwxr-xr-x. 4 root root 30 1月 17 11:35 fastdfs_tracker</span><br></pre></td></tr></table></figure><ul><li>修改存储节点目录下/etc/fdfs/storage.conf配置信息，具体如下：</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">disabled=false #启用配置文件 </span><br><span class="line">group_name=group1 #组名（第一组为 group1， 第二组为 group2） </span><br><span class="line">port=23000 #storage 的端口号,同一个组的 storage 端口号必须相同 </span><br><span class="line">base_path=/opt/fastdfs_storage #设置storage数据文件和日志目录 </span><br><span class="line">store_path0=/opt/fastdfs_storage_data #实际文件存储路径 </span><br><span class="line">store_path_count=1 #存储路径个数，需要和 store_path 个数匹配 </span><br><span class="line">tracker_server=192.168.43.70:22122 #tracker 服务器的 IP 地址和端口 </span><br><span class="line">tracker_server=192.168.43.70:22122 #多个 tracker 直接添加多条配置 </span><br><span class="line">http.server_port=8888 #设置 http 端口号</span><br></pre></td></tr></table></figure><p>完整配置信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><span class="line">disabled=false</span><br><span class="line">group_name=group1</span><br><span class="line">bind_addr=</span><br><span class="line">client_bind=true</span><br><span class="line">port=23000</span><br><span class="line">connect_timeout=30</span><br><span class="line">network_timeout=60</span><br><span class="line">heart_beat_interval=30</span><br><span class="line">stat_report_interval=60</span><br><span class="line">base_path=/opt/fastdfs_storage</span><br><span class="line">max_connections=256</span><br><span class="line">buff_size = 256KB</span><br><span class="line">accept_threads=1</span><br><span class="line">work_threads=4</span><br><span class="line">disk_rw_separated = true</span><br><span class="line">disk_reader_threads = 1</span><br><span class="line">disk_writer_threads = 1</span><br><span class="line">sync_wait_msec=50</span><br><span class="line">sync_interval=0</span><br><span class="line">sync_start_time=00:00</span><br><span class="line">sync_end_time=23:59</span><br><span class="line">write_mark_file_freq=500</span><br><span class="line">store_path_count=1</span><br><span class="line">store_path0=/opt/fastdfs_storage_data</span><br><span class="line">subdir_count_per_path=256</span><br><span class="line">tracker_server=192.168.43.70:22122</span><br><span class="line">tracker_server=192.168.43.71:22122</span><br><span class="line">log_level=info</span><br><span class="line">run_by_group=</span><br><span class="line">run_by_user=</span><br><span class="line">allow_hosts=*</span><br><span class="line">file_distribute_path_mode=0</span><br><span class="line">file_distribute_rotate_count=100</span><br><span class="line">fsync_after_written_bytes=0</span><br><span class="line">sync_log_buff_interval=10</span><br><span class="line">sync_binlog_buff_interval=10</span><br><span class="line">sync_stat_file_interval=300</span><br><span class="line">thread_stack_size=512KB</span><br><span class="line">upload_priority=10</span><br><span class="line">if_alias_prefix=</span><br><span class="line">check_file_duplicate=0</span><br><span class="line">file_signature_method=hash</span><br><span class="line">key_namespace=FastDFS</span><br><span class="line">keep_alive=0</span><br><span class="line">use_access_log = false</span><br><span class="line">rotate_access_log = false</span><br><span class="line">access_log_rotate_time=00:00</span><br><span class="line">rotate_error_log = false</span><br><span class="line">error_log_rotate_time=00:00</span><br><span class="line">rotate_access_log_size = 0</span><br><span class="line">rotate_error_log_size = 0</span><br><span class="line">log_file_keep_days = 0</span><br><span class="line">file_sync_skip_invalid_record=false</span><br><span class="line">use_connection_pool = false</span><br><span class="line">connection_pool_max_idle_time = 3600</span><br><span class="line">http.domain_name=</span><br><span class="line">http.server_port=8888</span><br></pre></td></tr></table></figure><ul><li>启动Storage<br>各节点配置好信息好之后，启动Storage</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# service fdfs_storaged start</span><br></pre></td></tr></table></figure><p>启动后查看日志情况</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br><span class="line">239</span><br><span class="line">240</span><br><span class="line">241</span><br><span class="line">242</span><br><span class="line">243</span><br><span class="line">244</span><br><span class="line">245</span><br><span class="line">246</span><br><span class="line">247</span><br><span class="line">248</span><br><span class="line">249</span><br><span class="line">250</span><br><span class="line">251</span><br><span class="line">252</span><br><span class="line">253</span><br><span class="line">254</span><br><span class="line">255</span><br><span class="line">256</span><br><span class="line">257</span><br><span class="line">258</span><br><span class="line">259</span><br><span class="line">260</span><br><span class="line">261</span><br><span class="line">262</span><br><span class="line">263</span><br><span class="line">264</span><br><span class="line">265</span><br><span class="line">266</span><br><span class="line">267</span><br><span class="line">268</span><br><span class="line">269</span><br><span class="line">270</span><br><span class="line">271</span><br><span class="line">272</span><br><span class="line">273</span><br><span class="line">274</span><br><span class="line">275</span><br><span class="line">276</span><br><span class="line">277</span><br><span class="line">278</span><br><span class="line">279</span><br><span class="line">280</span><br><span class="line">281</span><br><span class="line">282</span><br><span class="line">283</span><br><span class="line">284</span><br><span class="line">285</span><br><span class="line">286</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# fdfs_monitor /etc/fdfs/storage.conf</span><br><span class="line">[2018-01-20 16:56:48] DEBUG - base_path=/opt/fastdfs_storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0</span><br><span class="line"></span><br><span class="line">server_count=2, server_index=0</span><br><span class="line"></span><br><span class="line">tracker server is 192.168.43.70:22122</span><br><span class="line"></span><br><span class="line">group count: 2</span><br><span class="line"></span><br><span class="line">Group 1:</span><br><span class="line">group name = group1</span><br><span class="line">disk total space = 47073 MB</span><br><span class="line">disk free space = 35162 MB</span><br><span class="line">trunk free space = 0 MB</span><br><span class="line">storage server count = 2</span><br><span class="line">active server count = 2</span><br><span class="line">storage server port = 23000</span><br><span class="line">storage HTTP port = 8888</span><br><span class="line">store path count = 1</span><br><span class="line">subdir count per path = 256</span><br><span class="line">current write server index = 0</span><br><span class="line">current trunk file id = 0</span><br><span class="line"></span><br><span class="line">Storage 1:</span><br><span class="line">id = 192.168.43.72</span><br><span class="line">ip_addr = 192.168.43.72 (localhost.localdomain) ACTIVE</span><br><span class="line">http domain = </span><br><span class="line">version = 5.11</span><br><span class="line">join time = 2018-01-19 13:59:30</span><br><span class="line">up time = 2018-01-20 12:37:18</span><br><span class="line">total storage = 47073 MB</span><br><span class="line">free storage = 35162 MB</span><br><span class="line">upload priority = 10</span><br><span class="line">store_path_count = 1</span><br><span class="line">subdir_count_per_path = 256</span><br><span class="line">storage_port = 23000</span><br><span class="line">storage_http_port = 8888</span><br><span class="line">current_write_path = 0</span><br><span class="line">source storage id = </span><br><span class="line">if_trunk_server = 0</span><br><span class="line">connection.alloc_count = 256</span><br><span class="line">connection.current_count = 1</span><br><span class="line">connection.max_count = 1</span><br><span class="line">total_upload_count = 3</span><br><span class="line">success_upload_count = 3</span><br><span class="line">total_append_count = 0</span><br><span class="line">success_append_count = 0</span><br><span class="line">total_modify_count = 0</span><br><span class="line">success_modify_count = 0</span><br><span class="line">total_truncate_count = 0</span><br><span class="line">success_truncate_count = 0</span><br><span class="line">total_set_meta_count = 0</span><br><span class="line">success_set_meta_count = 0</span><br><span class="line">total_delete_count = 0</span><br><span class="line">success_delete_count = 0</span><br><span class="line">total_download_count = 0</span><br><span class="line">success_download_count = 0</span><br><span class="line">total_get_meta_count = 0</span><br><span class="line">success_get_meta_count = 0</span><br><span class="line">total_create_link_count = 0</span><br><span class="line">success_create_link_count = 0</span><br><span class="line">total_delete_link_count = 0</span><br><span class="line">success_delete_link_count = 0</span><br><span class="line">total_upload_bytes = 791904</span><br><span class="line">success_upload_bytes = 791904</span><br><span class="line">total_append_bytes = 0</span><br><span class="line">success_append_bytes = 0</span><br><span class="line">total_modify_bytes = 0</span><br><span class="line">success_modify_bytes = 0</span><br><span class="line">stotal_download_bytes = 0</span><br><span class="line">success_download_bytes = 0</span><br><span class="line">total_sync_in_bytes = 775234</span><br><span class="line">success_sync_in_bytes = 775234</span><br><span class="line">total_sync_out_bytes = 0</span><br><span class="line">success_sync_out_bytes = 0</span><br><span class="line">total_file_open_count = 4</span><br><span class="line">success_file_open_count = 4</span><br><span class="line">total_file_read_count = 0</span><br><span class="line">success_file_read_count = 0</span><br><span class="line">total_file_write_count = 8</span><br><span class="line">success_file_write_count = 8</span><br><span class="line">last_heart_beat_time = 2018-01-20 16:56:18</span><br><span class="line">last_source_update = 2018-01-19 19:34:55</span><br><span class="line">last_sync_update = 2018-01-19 15:28:56</span><br><span class="line">last_synced_timestamp = 2018-01-19 15:28:48 (0s delay)</span><br><span class="line">Storage 2:</span><br><span class="line">id = 192.168.43.73</span><br><span class="line">ip_addr = 192.168.43.73 ACTIVE</span><br><span class="line">http domain = </span><br><span class="line">version = 5.11</span><br><span class="line">join time = 2018-01-19 14:00:21</span><br><span class="line">up time = 2018-01-20 12:37:42</span><br><span class="line">total storage = 47073 MB</span><br><span class="line">free storage = 35166 MB</span><br><span class="line">upload priority = 10</span><br><span class="line">store_path_count = 1</span><br><span class="line">subdir_count_per_path = 256</span><br><span class="line">storage_port = 23000</span><br><span class="line">storage_http_port = 8888</span><br><span class="line">current_write_path = 0</span><br><span class="line">source storage id = 192.168.43.72</span><br><span class="line">if_trunk_server = 0</span><br><span class="line">connection.alloc_count = 256</span><br><span class="line">connection.current_count = 1</span><br><span class="line">connection.max_count = 1</span><br><span class="line">total_upload_count = 1</span><br><span class="line">success_upload_count = 1</span><br><span class="line">total_append_count = 0</span><br><span class="line">success_append_count = 0</span><br><span class="line">total_modify_count = 0</span><br><span class="line">success_modify_count = 0</span><br><span class="line">total_truncate_count = 0</span><br><span class="line">success_truncate_count = 0</span><br><span class="line">total_set_meta_count = 0</span><br><span class="line">success_set_meta_count = 0</span><br><span class="line">total_delete_count = 0</span><br><span class="line">success_delete_count = 0</span><br><span class="line">total_download_count = 0</span><br><span class="line">success_download_count = 0</span><br><span class="line">total_get_meta_count = 0</span><br><span class="line">success_get_meta_count = 0</span><br><span class="line">total_create_link_count = 0</span><br><span class="line">success_create_link_count = 0</span><br><span class="line">total_delete_link_count = 0</span><br><span class="line">success_delete_link_count = 0</span><br><span class="line">total_upload_bytes = 775234</span><br><span class="line">success_upload_bytes = 775234</span><br><span class="line">total_append_bytes = 0</span><br><span class="line">success_append_bytes = 0</span><br><span class="line">total_modify_bytes = 0</span><br><span class="line">success_modify_bytes = 0</span><br><span class="line">stotal_download_bytes = 0</span><br><span class="line">success_download_bytes = 0</span><br><span class="line">total_sync_in_bytes = 791904</span><br><span class="line">success_sync_in_bytes = 791904</span><br><span class="line">total_sync_out_bytes = 0</span><br><span class="line">success_sync_out_bytes = 0</span><br><span class="line">total_file_open_count = 4</span><br><span class="line">success_file_open_count = 4</span><br><span class="line">total_file_read_count = 0</span><br><span class="line">success_file_read_count = 0</span><br><span class="line">total_file_write_count = 8</span><br><span class="line">success_file_write_count = 8</span><br><span class="line">last_heart_beat_time = 2018-01-20 16:56:42</span><br><span class="line">last_source_update = 2018-01-19 15:28:48</span><br><span class="line">last_sync_update = 2018-01-19 19:34:59</span><br><span class="line">last_synced_timestamp = 2018-01-19 19:34:55 (0s delay)</span><br><span class="line"></span><br><span class="line">Group 2:</span><br><span class="line">group name = group2</span><br><span class="line">disk total space = 47073 MB</span><br><span class="line">disk free space = 35165 MB</span><br><span class="line">trunk free space = 0 MB</span><br><span class="line">storage server count = 2</span><br><span class="line">active server count = 2</span><br><span class="line">storage server port = 23000</span><br><span class="line">storage HTTP port = 8888</span><br><span class="line">store path count = 1</span><br><span class="line">subdir count per path = 256</span><br><span class="line">current write server index = 0</span><br><span class="line">current trunk file id = 0</span><br><span class="line"></span><br><span class="line">Storage 1:</span><br><span class="line">id = 192.168.43.74</span><br><span class="line">ip_addr = 192.168.43.74 ACTIVE</span><br><span class="line">http domain = </span><br><span class="line">version = 5.11</span><br><span class="line">join time = 2018-01-19 14:01:05</span><br><span class="line">up time = 2018-01-20 12:38:00</span><br><span class="line">total storage = 47073 MB</span><br><span class="line">free storage = 35165 MB</span><br><span class="line">upload priority = 10</span><br><span class="line">store_path_count = 1</span><br><span class="line">subdir_count_per_path = 256</span><br><span class="line">storage_port = 23000</span><br><span class="line">storage_http_port = 8888</span><br><span class="line">current_write_path = 0</span><br><span class="line">source storage id = </span><br><span class="line">if_trunk_server = 0</span><br><span class="line">connection.alloc_count = 256</span><br><span class="line">connection.current_count = 1</span><br><span class="line">connection.max_count = 1</span><br><span class="line">total_upload_count = 4</span><br><span class="line">success_upload_count = 4</span><br><span class="line">total_append_count = 0</span><br><span class="line">success_append_count = 0</span><br><span class="line">total_modify_count = 0</span><br><span class="line">success_modify_count = 0</span><br><span class="line">total_truncate_count = 0</span><br><span class="line">success_truncate_count = 0</span><br><span class="line">total_set_meta_count = 0</span><br><span class="line">success_set_meta_count = 0</span><br><span class="line">total_delete_count = 0</span><br><span class="line">success_delete_count = 0</span><br><span class="line">total_download_count = 0</span><br><span class="line">success_download_count = 0</span><br><span class="line">total_get_meta_count = 0</span><br><span class="line">success_get_meta_count = 0</span><br><span class="line">total_create_link_count = 0</span><br><span class="line">success_create_link_count = 0</span><br><span class="line">total_delete_link_count = 0</span><br><span class="line">success_delete_link_count = 0</span><br><span class="line">total_upload_bytes = 2107770</span><br><span class="line">success_upload_bytes = 2107770</span><br><span class="line">total_append_bytes = 0</span><br><span class="line">success_append_bytes = 0</span><br><span class="line">total_modify_bytes = 0</span><br><span class="line">success_modify_bytes = 0</span><br><span class="line">stotal_download_bytes = 0</span><br><span class="line">success_download_bytes = 0</span><br><span class="line">total_sync_in_bytes = 1550468</span><br><span class="line">success_sync_in_bytes = 1550468</span><br><span class="line">total_sync_out_bytes = 0</span><br><span class="line">success_sync_out_bytes = 0</span><br><span class="line">total_file_open_count = 6</span><br><span class="line">success_file_open_count = 6</span><br><span class="line">total_file_read_count = 0</span><br><span class="line">success_file_read_count = 0</span><br><span class="line">total_file_write_count = 15</span><br><span class="line">success_file_write_count = 15</span><br><span class="line">last_heart_beat_time = 2018-01-20 16:56:38</span><br><span class="line">last_source_update = 2018-01-19 19:35:40</span><br><span class="line">last_sync_update = 2018-01-19 15:28:53</span><br><span class="line">last_synced_timestamp = 2018-01-19 15:28:50 (-1s delay)</span><br><span class="line">Storage 2:</span><br><span class="line">id = 192.168.43.75</span><br><span class="line">ip_addr = 192.168.43.75 ACTIVE</span><br><span class="line">http domain = </span><br><span class="line">version = 5.11</span><br><span class="line">join time = 2018-01-19 14:01:27</span><br><span class="line">up time = 2018-01-20 12:38:20</span><br><span class="line">total storage = 47073 MB</span><br><span class="line">free storage = 35165 MB</span><br><span class="line">upload priority = 10</span><br><span class="line">store_path_count = 1</span><br><span class="line">subdir_count_per_path = 256</span><br><span class="line">storage_port = 23000</span><br><span class="line">storage_http_port = 8888</span><br><span class="line">current_write_path = 0</span><br><span class="line">source storage id = 192.168.43.74</span><br><span class="line">if_trunk_server = 0</span><br><span class="line">connection.alloc_count = 256</span><br><span class="line">connection.current_count = 1</span><br><span class="line">connection.max_count = 1</span><br><span class="line">total_upload_count = 2</span><br><span class="line">success_upload_count = 2</span><br><span class="line">total_append_count = 0</span><br><span class="line">success_append_count = 0</span><br><span class="line">total_modify_count = 0</span><br><span class="line">success_modify_count = 0</span><br><span class="line">total_truncate_count = 0</span><br><span class="line">success_truncate_count = 0</span><br><span class="line">total_set_meta_count = 0</span><br><span class="line">success_set_meta_count = 0</span><br><span class="line">total_delete_count = 0</span><br><span class="line">success_delete_count = 0</span><br><span class="line">total_download_count = 0</span><br><span class="line">success_download_count = 0</span><br><span class="line">total_get_meta_count = 0</span><br><span class="line">success_get_meta_count = 0</span><br><span class="line">total_create_link_count = 0</span><br><span class="line">success_create_link_count = 0</span><br><span class="line">total_delete_link_count = 0</span><br><span class="line">success_delete_link_count = 0</span><br><span class="line">total_upload_bytes = 1550468</span><br><span class="line">success_upload_bytes = 1550468</span><br><span class="line">total_append_bytes = 0</span><br><span class="line">success_append_bytes = 0</span><br><span class="line">total_modify_bytes = 0</span><br><span class="line">success_modify_bytes = 0</span><br><span class="line">stotal_download_bytes = 0</span><br><span class="line">success_download_bytes = 0</span><br><span class="line">total_sync_in_bytes = 2107770</span><br><span class="line">success_sync_in_bytes = 2107770</span><br><span class="line">total_sync_out_bytes = 0</span><br><span class="line">success_sync_out_bytes = 0</span><br><span class="line">total_file_open_count = 6</span><br><span class="line">success_file_open_count = 6</span><br><span class="line">total_file_read_count = 0</span><br><span class="line">success_file_read_count = 0</span><br><span class="line">total_file_write_count = 15</span><br><span class="line">success_file_write_count = 15</span><br><span class="line">last_heart_beat_time = 2018-01-20 16:56:23</span><br><span class="line">last_source_update = 2018-01-19 15:28:49</span><br><span class="line">last_sync_update = 2018-01-19 19:35:46</span><br><span class="line">last_synced_timestamp = 2018-01-19 19:35:40 (0s delay)</span><br></pre></td></tr></table></figure><p>如果看到有2组Storage信息，则表示配置信息配置成功，并注册到Tracker中，查看日志启动情况</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# tail -f /opt/fastdfs_storage/logs/storaged.log </span><br><span class="line"></span><br><span class="line">[2018-01-20 12:37:18] INFO - FastDFS v5.11, base_path=/opt/fastdfs_storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s</span><br><span class="line">[2018-01-20 12:37:18] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0</span><br><span class="line">[2018-01-20 12:37:18] INFO - file: storage_func.c, line: 257, tracker_client_ip: 192.168.43.72, my_server_id_str: 192.168.43.72, g_server_id_in_filename: 1210820800</span><br><span class="line">[2018-01-20 12:37:18] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.43.71:22122, as a tracker client, my ip is 192.168.43.72</span><br><span class="line">[2018-01-20 12:37:18] INFO - file: tracker_client_thread.c, line: 1947, tracker server: #0. 192.168.43.70:22122, my_report_status: -1</span><br><span class="line">[2018-01-20 12:37:18] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.43.70:22122, as a tracker client, my ip is 192.168.43.72</span><br><span class="line">[2018-01-20 12:37:18] INFO - file: tracker_client_thread.c, line: 1947, tracker server: #0. 192.168.43.70:22122, my_report_status: -1</span><br><span class="line">[2018-01-20 12:37:48] INFO - file: tracker_client_thread.c, line: 1263, tracker server 192.168.43.71:22122, set tracker leader: 192.168.43.71:22122</span><br><span class="line">[2018-01-20 12:37:48] INFO - file: storage_sync.c, line: 2732, successfully connect to storage server 192.168.43.73:23000</span><br></pre></td></tr></table></figure><p>发现此时192.168.43.71作为Tracker的Leader。</p><ul><li>设置Storage开机自启动</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# echo "service fdfs_storaged start" |tee -a /etc/rc.d/rc.local</span><br></pre></td></tr></table></figure><ul><li>安装fastdfs-nginx-module、Nginx模块</li></ul><p>安装Nginx模块所需的依赖环境</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# yum -y install pcre pcre-devel </span><br><span class="line">[root@localhost fdfs]# yum -y install zlib zlib-devel </span><br><span class="line">[root@localhost fdfs]# yum -y install openssl openssl-devel</span><br></pre></td></tr></table></figure><p>解压nginx和fastdfs-nginx-module<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fdfs]# tar -zxvf nginx-1.10.3.tar.gz</span><br><span class="line">[root@localhost fdfs]# unzip fastdfs-nginx-module-master.zip</span><br></pre></td></tr></table></figure></p><p>进入Nginx解压目录进行编译安装<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost nginx-1.10.3]# ./configure --prefix=/usr/local/nginx --add-module=/home/zhangyongliang/apps/fastdfs-nginx-module-master/src #解压后fastdfs-nginx-module所在的位置</span><br></pre></td></tr></table></figure></p><p>安装成功后，nginx会安装在/usr/local/nginx,安装后查看</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# ll /usr/local/nginx/</span><br><span class="line">总用量 8</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 client_body_temp</span><br><span class="line">drwxr-xr-x. 2 root root 4096 1月 17 13:17 conf</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 fastcgi_temp</span><br><span class="line">drwxr-xr-x. 2 root root 40 1月 17 13:17 html</span><br><span class="line">drwxr-xr-x. 2 root root 58 1月 17 13:49 logs</span><br><span class="line">-rw-r--r--. 1 root root 1156 1月 17 13:29 nginx.conf</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 proxy_temp</span><br><span class="line">drwxr-xr-x. 2 root root 19 1月 17 13:17 sbin</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 scgi_temp</span><br><span class="line">drwx------. 2 nobody root 6 1月 17 13:23 uwsgi_temp</span><br></pre></td></tr></table></figure><p>安装成功后，nginx尚未运行时，nginx文件夹没有临时文件夹，例如fastcgi_temp这些文件。<br><br><br>复制 fastdfs-nginx-module 源码中的配置文件到/etc/fdfs 目录， 并修改</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost src]# cp /usr/local/src/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/ </span><br><span class="line">[root@localhost src]# vi /etc/fdfs/mod_fastdfs.conf</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">(1)第1组 Storage 的 mod_fastdfs.conf 配置如下： </span><br><span class="line">connect_timeout=10 </span><br><span class="line">base_path=/opt/fastdfs_storage</span><br><span class="line">tracker_server=192.168.1.131:22122 </span><br><span class="line">tracker_server=192.168.1.132:22122 </span><br><span class="line">storage_server_port=23000 </span><br><span class="line">group_name=group1 </span><br><span class="line">url_have_group_name = true </span><br><span class="line">store_path0=/opt/fastdfs_storage_data</span><br><span class="line">group_count = 2 </span><br><span class="line">[group1] </span><br><span class="line">group_name=group1 </span><br><span class="line">storage_server_port=23000 </span><br><span class="line">store_path_count=1 </span><br><span class="line">store_path0=/opt/fastdfs_storage_data</span><br><span class="line">[group2] </span><br><span class="line">group_name=group2 </span><br><span class="line">storage_server_port=23000 </span><br><span class="line">store_path_count=1 </span><br><span class="line">store_path0=/opt/fastdfs_storage_data </span><br><span class="line">(2)第2组 Storage 的 mod_fastdfs.conf 配置与第一组配置只有 group_name 不同： </span><br><span class="line">group_name=group2</span><br></pre></td></tr></table></figure><p>完整信息如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br></pre></td><td class="code"><pre><span class="line"># connect timeout in seconds</span><br><span class="line"># default value is 30s</span><br><span class="line">connect_timeout=2</span><br><span class="line"></span><br><span class="line"># network recv and send timeout in seconds</span><br><span class="line"># default value is 30s</span><br><span class="line">network_timeout=30</span><br><span class="line"></span><br><span class="line"># the base path to store log files</span><br><span class="line">base_path=/opt/fastdfs_storage</span><br><span class="line"></span><br><span class="line"># if load FastDFS parameters from tracker server</span><br><span class="line"># since V1.12</span><br><span class="line"># default value is false</span><br><span class="line">load_fdfs_parameters_from_tracker=true</span><br><span class="line"></span><br><span class="line"># storage sync file max delay seconds</span><br><span class="line"># same as tracker.conf</span><br><span class="line"># valid only when load_fdfs_parameters_from_tracker is false</span><br><span class="line"># since V1.12</span><br><span class="line"># default value is 86400 seconds (one day)</span><br><span class="line">storage_sync_file_max_delay = 86400</span><br><span class="line"></span><br><span class="line"># if use storage ID instead of IP address</span><br><span class="line"># same as tracker.conf</span><br><span class="line"># valid only when load_fdfs_parameters_from_tracker is false</span><br><span class="line"># default value is false</span><br><span class="line"># since V1.13</span><br><span class="line">use_storage_id = false</span><br><span class="line"></span><br><span class="line"># specify storage ids filename, can use relative or absolute path</span><br><span class="line"># same as tracker.conf</span><br><span class="line"># valid only when load_fdfs_parameters_from_tracker is false</span><br><span class="line"># since V1.13</span><br><span class="line">storage_ids_filename = storage_ids.conf</span><br><span class="line"></span><br><span class="line"># FastDFS tracker_server can ocur more than once, and tracker_server format is</span><br><span class="line"># "host:port", host can be hostname or ip address</span><br><span class="line"># valid only when load_fdfs_parameters_from_tracker is true</span><br><span class="line">tracker_server=192.168.43.70:22122</span><br><span class="line">tracker_server=192.168.43.71:22122</span><br><span class="line"></span><br><span class="line"># the port of the local storage server</span><br><span class="line"># the default value is 23000</span><br><span class="line">storage_server_port=23000</span><br><span class="line"></span><br><span class="line"># the group name of the local storage server</span><br><span class="line">group_name=group1</span><br><span class="line"></span><br><span class="line"># if the url / uri including the group name</span><br><span class="line"># set to false when uri like /M00/00/00/xxx</span><br><span class="line"># set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx</span><br><span class="line"># default value is false</span><br><span class="line">url_have_group_name = true</span><br><span class="line"></span><br><span class="line"># path(disk or mount point) count, default value is 1</span><br><span class="line"># must same as storage.conf</span><br><span class="line">store_path_count=1</span><br><span class="line"></span><br><span class="line"># store_path#, based 0, if store_path0 not exists, it's value is base_path</span><br><span class="line"># the paths must be exist</span><br><span class="line"># must same as storage.conf</span><br><span class="line">store_path0=/opt/fastdfs_storage_data</span><br><span class="line">#store_path1=/home/yuqing/fastdfs1</span><br><span class="line"></span><br><span class="line"># standard log level as syslog, case insensitive, value list:</span><br><span class="line">### emerg for emergency</span><br><span class="line">### alert</span><br><span class="line">### crit for critical</span><br><span class="line">### error</span><br><span class="line">### warn for warning</span><br><span class="line">### notice</span><br><span class="line">### info</span><br><span class="line">### debug</span><br><span class="line">log_level=info</span><br><span class="line"></span><br><span class="line"># set the log filename, such as /usr/local/apache2/logs/mod_fastdfs.log</span><br><span class="line"># empty for output to stderr (apache and nginx error_log file)</span><br><span class="line">log_filename=</span><br><span class="line"></span><br><span class="line"># response mode when the file not exist in the local file system</span><br><span class="line">## proxy: get the content from other storage server, then send to client</span><br><span class="line">## redirect: redirect to the original storage server (HTTP Header is Location)</span><br><span class="line">response_mode=proxy</span><br><span class="line"></span><br><span class="line"># the NIC alias prefix, such as eth in Linux, you can see it by ifconfig -a</span><br><span class="line"># multi aliases split by comma. empty value means auto set by OS type</span><br><span class="line"># this paramter used to get all ip address of the local host</span><br><span class="line"># default values is empty</span><br><span class="line">if_alias_prefix=</span><br><span class="line"></span><br><span class="line"># use "#include" directive to include HTTP config file</span><br><span class="line"># NOTE: #include is an include directive, do NOT remove the # before include</span><br><span class="line">#include http.conf</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># if support flv</span><br><span class="line"># default value is false</span><br><span class="line"># since v1.15</span><br><span class="line">flv_support = true</span><br><span class="line"></span><br><span class="line"># flv file extension name</span><br><span class="line"># default value is flv</span><br><span class="line"># since v1.15</span><br><span class="line">flv_extension = flv</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># set the group count</span><br><span class="line"># set to none zero to support multi-group on this storage server</span><br><span class="line"># set to 0 for single group only</span><br><span class="line"># groups settings section as [group1], [group2], ..., [groupN]</span><br><span class="line"># default value is 0</span><br><span class="line"># since v1.14</span><br><span class="line">group_count = 2</span><br><span class="line"></span><br><span class="line"># group settings for group #1</span><br><span class="line"># since v1.14</span><br><span class="line"># when support multi-group on this storage server, uncomment following section</span><br><span class="line">[group1]</span><br><span class="line">group_name=group1</span><br><span class="line">storage_server_port=23000</span><br><span class="line">store_path_count=1</span><br><span class="line">store_path0=/opt/fastdfs_storage_data</span><br><span class="line">[group2]</span><br><span class="line">group_name=group2</span><br><span class="line">storage_server_port=23000</span><br><span class="line">store_path_count=1</span><br><span class="line">store_path0=/opt/fastdfs_storage_data</span><br><span class="line"># group settings for group #2</span><br><span class="line"># since v1.14</span><br><span class="line"># when support multi-group, uncomment following section as neccessary</span><br><span class="line">#[group2]</span><br><span class="line">#group_name=group2</span><br><span class="line">#storage_server_port=23000</span><br><span class="line">#store_path_count=1</span><br><span class="line">#store_path0=/home/yuqing/fastdfs</span><br></pre></td></tr></table></figure></p><p> 复制 FastDFS 安装目录的部分配置文件到/etc/fdfs 目录</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost conf]# pwd</span><br><span class="line">/home/zhangyongliang/apps/fastdfs-5.11/conf</span><br><span class="line">[root@localhost conf]# cp http.conf mime.types /etc/fdfs/</span><br></pre></td></tr></table></figure><p>创建M00至storage存储目录的符号连接：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ln -s /opt/fastdfs_storage_data/data/ /opt/fastdfs_storage_data/data/M00</span><br></pre></td></tr></table></figure><p>配置 Nginx， 简洁版 nginx 配置样例</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"># vi /usr/local/nginx/conf/nginx.conf </span><br><span class="line">user root; </span><br><span class="line">worker_processes 1; </span><br><span class="line">events { </span><br><span class="line"> worker_connections 1024; </span><br><span class="line">} </span><br><span class="line">http { </span><br><span class="line"> include mime.types; </span><br><span class="line"> default_type application/octet-stream; </span><br><span class="line"> sendfile on; </span><br><span class="line"> keepalive_timeout 65; </span><br><span class="line"> server { </span><br><span class="line"> listen 8888; </span><br><span class="line"> server_name localhost; </span><br><span class="line"> location ~/group([0-9])/M00 { </span><br><span class="line"> #alias /fastdfs/storage/data; </span><br><span class="line"> ngx_fastdfs_module; </span><br><span class="line"> } </span><br><span class="line"> error_page 500 502 503 504 /50x.html; </span><br><span class="line"> location = /50x.html { </span><br><span class="line"> root html; </span><br><span class="line"> } </span><br><span class="line"> } </span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>注意、 说明：<br><br> 8888 端口值是要与/etc/fdfs/storage.conf 中的 http.server_port=8888 相对应，因为 http.server_port 默认为 8888,如果想改成 80，则要对应修改过来。<br><br>重新启动各节点的Nginx服务<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost conf]#/usr/local/nginx/sbin/nginx -s reload</span><br></pre></td></tr></table></figure></p><h4 id="4-文件上传测试"><a href="#4-文件上传测试" class="headerlink" title="4.文件上传测试"></a>4.文件上传测试</h4><p><strong>执行节点Tracker01、Tracker02</strong><br><br>修改 Tracker 服务器中的客户端配置文件<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"># vi /etc/fdfs/client.conf </span><br><span class="line">base_path=/fastdfs/tracker </span><br><span class="line">tracker_server=192.168.43.70:22122 </span><br><span class="line">tracker_server=192.168.43.71:22122</span><br></pre></td></tr></table></figure></p><p>执行如下文件上传命令</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost zhangyongliang]# fdfs_upload_file /etc/fdfs/client.conf P71022-205803.jpg </span><br><span class="line">group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg</span><br><span class="line">[root@localhost zhangyongliang]# fdfs_upload_file /etc/fdfs/client.conf P71022-205803.jpg </span><br><span class="line">group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg</span><br></pre></td></tr></table></figure><p>进行2次上传后，发现文件被均衡分到2个group。</p><h4 id="5-Tracker安装Nginx、-ngx-cache-purge-模块"><a href="#5-Tracker安装Nginx、-ngx-cache-purge-模块" class="headerlink" title="5.Tracker安装Nginx、 ngx_cache_purge 模块"></a>5.Tracker安装Nginx、 ngx_cache_purge 模块</h4><ul><li>安装编译 Nginx 所需的依赖包<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost zhangyongliang]# yum install gcc gcc-c++ make automake autoconf libtool pcre pcre-devel zlib zlib-devel openssl openssl-devel</span><br></pre></td></tr></table></figure></li></ul><p>解压Nginx和ngx_cache_pure模块</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost apps]# tar ngx_cache_purge-2.3.tar.gz</span><br><span class="line">[root@localhost apps]# tar nginx-1.10.3.tar.gz</span><br></pre></td></tr></table></figure><p>编译安装 Nginx（添加 ngx_cache_purge 模块）</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost apps]# cd nginx-1.13.0 </span><br><span class="line">[root@localhost nginx-1.13.0# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/src/ngx_cache_purge-2.3 </span><br><span class="line">[root@localhost nginx-1.13.0]# make && make install</span><br></pre></td></tr></table></figure><p>配置 Nginx， 设置负载均衡以及缓存</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br></pre></td><td class="code"><pre><span class="line"># vi /usr/local/nginx/conf/nginx.conf </span><br><span class="line"></span><br><span class="line">#user nobody;</span><br><span class="line">worker_processes 1;</span><br><span class="line"></span><br><span class="line">#error_log logs/error.log;</span><br><span class="line">#error_log logs/error.log notice;</span><br><span class="line">#error_log logs/error.log info;</span><br><span class="line"></span><br><span class="line">#pid logs/nginx.pid;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">events {</span><br><span class="line"> worker_connections 1024;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">http {</span><br><span class="line"> include mime.types;</span><br><span class="line"> default_type application/octet-stream;</span><br><span class="line"></span><br><span class="line"> #log_format main '$remote_addr - $remote_user [$time_local] "$request" '</span><br><span class="line"> # '$status $body_bytes_sent "$http_referer" '</span><br><span class="line"> # '"$http_user_agent" "$http_x_forwarded_for"';</span><br><span class="line"></span><br><span class="line"> #access_log logs/access.log main;</span><br><span class="line"> sendfile on; </span><br><span class="line"> tcp_nopush on; </span><br><span class="line"> #tcp_nopush on;</span><br><span class="line"> #keepalive_timeout 0;</span><br><span class="line"> keepalive_timeout 65;</span><br><span class="line"></span><br><span class="line"> #gzip on;</span><br><span class="line">#设置缓存 </span><br><span class="line"> server_names_hash_bucket_size 128; </span><br><span class="line"> client_header_buffer_size 32k; </span><br><span class="line"> large_client_header_buffers 4 32k; </span><br><span class="line"> client_max_body_size 300m; </span><br><span class="line"> </span><br><span class="line"> proxy_redirect off; </span><br><span class="line"> proxy_set_header Host $http_host; </span><br><span class="line"> proxy_set_header X-Real-IP $remote_addr; </span><br><span class="line"> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; </span><br><span class="line"> proxy_connect_timeout 90; </span><br><span class="line"> proxy_send_timeout 90; </span><br><span class="line"> proxy_read_timeout 90; </span><br><span class="line"> proxy_buffer_size 16k; </span><br><span class="line"> proxy_buffers 4 64k; </span><br><span class="line"> proxy_busy_buffers_size 128k; </span><br><span class="line"> proxy_temp_file_write_size 128k;</span><br><span class="line">#设置缓存存储路径，存储方式，分别内存大小，磁盘最大空间，缓存期限 </span><br><span class="line"> proxy_cache_path /opt/fastdfs_tracker/proxy_cache levels=1:2 </span><br><span class="line">keys_zone=http-cache:200m max_size=1g inactive=30d;</span><br><span class="line"> proxy_temp_path /opt/fastdfs_tracker/tmp;</span><br><span class="line"> #group1的服务设置 </span><br><span class="line"> upstream fdfs_group1 { </span><br><span class="line"> server 192.168.43.72:8888 weight=1 max_fails=2 fail_timeout=30s; </span><br><span class="line"> server 192.168.43.73:8888 weight=1 max_fails=2 fail_timeout=30s; </span><br><span class="line"> }</span><br><span class="line"> #group2的服务设置 </span><br><span class="line"> upstream fdfs_group2 { </span><br><span class="line"> server 192.168.43.74:8888 weight=1 max_fails=2 fail_timeout=30s; </span><br><span class="line"> server 192.168.43.75:8888 weight=1 max_fails=2 fail_timeout=30s; </span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> server {</span><br><span class="line"> listen 8000;</span><br><span class="line"> server_name localhost;</span><br><span class="line"></span><br><span class="line"> #charset koi8-r;</span><br><span class="line"></span><br><span class="line"> #access_log logs/host.access.log main;</span><br><span class="line">#group1的负载均衡配置 </span><br><span class="line"> location /group1/M00 { </span><br><span class="line"> proxy_next_upstream http_502 http_504 error timeout invalid_header; </span><br><span class="line"> proxy_cache http-cache; </span><br><span class="line"> proxy_cache_valid 200 304 12h; </span><br><span class="line"> proxy_cache_key $uri$is_args$args; </span><br><span class="line"> #对应group1的服务设置 </span><br><span class="line"> proxy_pass http://fdfs_group1; </span><br><span class="line"> expires 30d; </span><br><span class="line"> }</span><br><span class="line"> location /group2/M00 { </span><br><span class="line"> proxy_next_upstream http_502 http_504 error timeout invalid_header; </span><br><span class="line"> proxy_cache http-cache; </span><br><span class="line"> proxy_cache_valid 200 304 12h; </span><br><span class="line"> proxy_cache_key $uri$is_args$args; </span><br><span class="line"> #对应group2的服务设置 </span><br><span class="line"> proxy_pass http://fdfs_group2; </span><br><span class="line"> expires 30d; </span><br><span class="line"> }</span><br><span class="line"> location ~/purge(/.*) { </span><br><span class="line"> allow 127.0.0.1; </span><br><span class="line"> allow 192.168.43.0/24; </span><br><span class="line"> deny all; </span><br><span class="line"> proxy_cache_purge http-cache $1$is_args$args; </span><br><span class="line"> } </span><br><span class="line"> location / {</span><br><span class="line"> root html;</span><br><span class="line"> index index.html index.htm;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> #error_page 404 /404.html;</span><br><span class="line"></span><br><span class="line"> # redirect server error pages to the static page /50x.html</span><br><span class="line"> #</span><br><span class="line"> error_page 500 502 503 504 /50x.html;</span><br><span class="line"> location = /50x.html {</span><br><span class="line"> root html;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> # proxy the PHP scripts to Apache listening on 127.0.0.1:80</span><br><span class="line"> #</span><br><span class="line"> #location ~ \.php$ {</span><br><span class="line"> # proxy_pass http://127.0.0.1;</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"> # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000</span><br><span class="line"> #</span><br><span class="line"> #location ~ \.php$ {</span><br><span class="line"> # root html;</span><br><span class="line"> # fastcgi_pass 127.0.0.1:9000;</span><br><span class="line"> # fastcgi_index index.php;</span><br><span class="line"> # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;</span><br><span class="line"> # include fastcgi_params;</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"> # deny access to .htaccess files, if Apache's document root</span><br><span class="line"> # concurs with nginx's one</span><br><span class="line"> #</span><br><span class="line"> #location ~ /\.ht {</span><br><span class="line"> # deny all;</span><br><span class="line"> #}</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> # another virtual host using mix of IP-, name-, and port-based configuration</span><br><span class="line"> #</span><br><span class="line"> #server {</span><br><span class="line"> # listen 8000;</span><br><span class="line"> # listen somename:8080;</span><br><span class="line"> # server_name somename alias another.alias;</span><br><span class="line"></span><br><span class="line"> # location / {</span><br><span class="line"> # root html;</span><br><span class="line"> # index index.html index.htm;</span><br><span class="line"> # }</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> # HTTPS server</span><br><span class="line"> #</span><br><span class="line"> #server {</span><br><span class="line"> # listen 443 ssl;</span><br><span class="line"> # server_name localhost;</span><br><span class="line"></span><br><span class="line"> # ssl_certificate cert.pem;</span><br><span class="line"> # ssl_certificate_key cert.key;</span><br><span class="line"></span><br><span class="line"> # ssl_session_cache shared:SSL:1m;</span><br><span class="line"> # ssl_session_timeout 5m;</span><br><span class="line"></span><br><span class="line"> # ssl_ciphers HIGH:!aNULL:!MD5;</span><br><span class="line"> # ssl_prefer_server_ciphers on;</span><br><span class="line"></span><br><span class="line"> # location / {</span><br><span class="line"> # root html;</span><br><span class="line"> # index index.html index.htm;</span><br><span class="line"> # }</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>根据Nginx配置，创建对应目录下的文件夹</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost fastdfs_tracker]# mkdir proxy_cache tmp</span><br><span class="line">[root@localhost fastdfs_tracker]# ll</span><br><span class="line">总用量 0</span><br><span class="line">drwxr-xr-x. 2 root root 178 1月 20 12:37 data</span><br><span class="line">drwxr-xr-x. 2 root root 26 1月 19 12:01 logs</span><br><span class="line">drwxr-xr-x. 7 nobody root 51 1月 19 19:35 proxy_cache</span><br><span class="line">drwxr-xr-x. 2 nobody root 6 1月 19 19:35 tmp</span><br></pre></td></tr></table></figure><p>重启Nginx进行访问测试</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">重启 Nginx </span><br><span class="line">[root@localhost fastdfs_tracker]# /usr/local/nginx/sbin/nginx -s reload</span><br></pre></td></tr></table></figure><p>前面直接通过访问 Storage 节点中的 Nginx 的文件<br><br><a href="http://192.168.43.72:8888/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg" target="_blank" rel="noopener">http://192.168.43.72:8888/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg</a>]<br><br><a href="http://192.168.43.74:8888/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg" target="_blank" rel="noopener">http://192.168.43.74:8888/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg</a><br><br>现在可以通过 Tracker 中的 Nginx 来进行访问<br><br>(1)通过 Tracker1 中的 Nginx 来访问<br><br><a href="http://192.168.43.70:8000/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg" target="_blank" rel="noopener">http://192.168.43.70:8000/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg</a><br><br><a href="http://192.168.43.70:8000/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg" target="_blank" rel="noopener">http://192.168.43.70:8000/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg</a><br><br>(2)通过 Tracker2 中的 Nginx 来访问<br><br><a href="http://192.168.43.71:8000/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg" target="_blank" rel="noopener">http://192.168.43.71:8000/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg</a><br><br><a href="http://192.168.50.71:8000/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg" target="_blank" rel="noopener">http://192.168.50.71:8000/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg</a><br></p><h4 id="6-构建Keepalive-Nginx-实现虚拟IP的代理"><a href="#6-构建Keepalive-Nginx-实现虚拟IP的代理" class="headerlink" title="6.构建Keepalive+Nginx 实现虚拟IP的代理"></a>6.构建Keepalive+Nginx 实现虚拟IP的代理</h4><p>关于使用Keepalive+Nginx进行代理的环境安装，请参考本人简书此文<a href="https://www.jianshu.com/p/bc34f9101c5e" target="_blank" rel="noopener">：Keepalived+Nginx+Tomcat 实现高可用Web集群</a><br>本文不再做赘述说明<br></p><ul><li>启动Keepalvie+nginx Master主节点【192.168.43.101】</li><li>启动Keepalvie+nginx BackUp备节点【192.168.43.102】<br><br><br>修改2个节点Nginx下目录的nginx.conf的配置文件信息，添加如下内容<br><br>主要内容为2个Tracker加入到Keepalive+nginx代理当中<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">upstream fastdfs_tracker { </span><br><span class="line"> server 192.168.43.70:8000 weight=1 max_fails=2 fail_timeout=30s; </span><br><span class="line"> server 192.168.43.71:8000 weight=1 max_fails=2 fail_timeout=30s; </span><br><span class="line">}</span><br></pre></td></tr></table></figure></li></ul><p>第二处修改是添加了一个location并且匹配规则是路径当中有fastdfs</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">location /fastdfs { </span><br><span class="line"> root html; </span><br><span class="line"> index index.html index.htm; </span><br><span class="line"> proxy_pass http://fastdfs_tracker/; </span><br><span class="line"> proxy_set_header Host $http_host; </span><br><span class="line"> proxy_set_header Cookie $http_cookie; </span><br><span class="line"> proxy_set_header X-Real-IP $remote_addr; </span><br><span class="line"> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; </span><br><span class="line"> proxy_set_header X-Forwarded-Proto $scheme; </span><br><span class="line"> client_max_body_size 300m; </span><br><span class="line"> }</span><br></pre></td></tr></table></figure><p>完整配置代码如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">#user nobody;</span><br><span class="line">worker_processes 1;</span><br><span class="line"></span><br><span class="line">#error_log logs/error.log;</span><br><span class="line">#error_log logs/error.log notice;</span><br><span class="line">#error_log logs/error.log info;</span><br><span class="line"></span><br><span class="line">#pid logs/nginx.pid;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">events {</span><br><span class="line"> worker_connections 1024;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">http {</span><br><span class="line"> include mime.types;</span><br><span class="line"> default_type application/octet-stream;</span><br><span class="line"></span><br><span class="line"> #log_format main '$remote_addr - $remote_user [$time_local] "$request" '</span><br><span class="line"> # '$status $body_bytes_sent "$http_referer" '</span><br><span class="line"> # '"$http_user_agent" "$http_x_forwarded_for"';</span><br><span class="line"></span><br><span class="line"> #access_log logs/access.log main;</span><br><span class="line"></span><br><span class="line"> sendfile on;</span><br><span class="line"> #tcp_nopush on;</span><br><span class="line"></span><br><span class="line"> #keepalive_timeout 0;</span><br><span class="line"> keepalive_timeout 65;</span><br><span class="line"></span><br><span class="line"> #gzip on;</span><br><span class="line">upstream fastdfs_tracker { </span><br><span class="line"> server 192.168.43.70:8000 weight=1 max_fails=2 fail_timeout=30s; </span><br><span class="line"> server 192.168.43.71:8000 weight=1 max_fails=2 fail_timeout=30s; </span><br><span class="line"> } </span><br><span class="line"> upstream tomcat{</span><br><span class="line"> server 192.168.43.103:8080 weight=1;</span><br><span class="line"> server 192.168.43.104:8080 weight=1;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"> server {</span><br><span class="line"> listen 80;</span><br><span class="line"> server_name localhost;</span><br><span class="line"></span><br><span class="line"> #charset koi8-r;</span><br><span class="line"></span><br><span class="line"> #access_log logs/host.access.log main;</span><br><span class="line"></span><br><span class="line"> location / {</span><br><span class="line"> proxy_pass http://tomcat;</span><br><span class="line">proxy_set_header X-NGINX "NGINX-1";</span><br><span class="line"> root html;</span><br><span class="line"> index index.html index.htm;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> #error_page 404 /404.html;</span><br><span class="line"></span><br><span class="line"> # redirect server error pages to the static page /50x.html</span><br><span class="line"> #</span><br><span class="line"> error_page 500 502 503 504 /50x.html;</span><br><span class="line"> location = /50x.html {</span><br><span class="line"> root html;</span><br><span class="line"> }</span><br><span class="line">location /fastdfs { </span><br><span class="line"> root html; </span><br><span class="line"> index index.html index.htm; </span><br><span class="line"> proxy_pass http://fastdfs_tracker/; </span><br><span class="line"> proxy_set_header Host $http_host; </span><br><span class="line"> proxy_set_header Cookie $http_cookie; </span><br><span class="line"> proxy_set_header X-Real-IP $remote_addr; </span><br><span class="line"> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; </span><br><span class="line"> proxy_set_header X-Forwarded-Proto $scheme; </span><br><span class="line"> client_max_body_size 300m; </span><br><span class="line"> } </span><br><span class="line"></span><br><span class="line"> # proxy the PHP scripts to Apache listening on 127.0.0.1:80</span><br><span class="line"> #</span><br><span class="line"> #location ~ \.php$ {</span><br><span class="line"> # proxy_pass http://127.0.0.1;</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"> # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000</span><br><span class="line"> #</span><br><span class="line"> #location ~ \.php$ {</span><br><span class="line"> # root html;</span><br><span class="line"> # fastcgi_pass 127.0.0.1:9000;</span><br><span class="line"> # fastcgi_index index.php;</span><br><span class="line"> # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;</span><br><span class="line"> # include fastcgi_params;</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"> # deny access to .htaccess files, if Apache's document root</span><br><span class="line"> # concurs with nginx's one</span><br><span class="line"> #</span><br><span class="line"> #location ~ /\.ht {</span><br><span class="line"> # deny all;</span><br><span class="line"> #}</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> # another virtual host using mix of IP-, name-, and port-based configuration</span><br><span class="line"> #</span><br><span class="line"> #server {</span><br><span class="line"> # listen 8000;</span><br><span class="line"> # listen somename:8080;</span><br><span class="line"> # server_name somename alias another.alias;</span><br><span class="line"></span><br><span class="line"> # location / {</span><br><span class="line"> # root html;</span><br><span class="line"> # index index.html index.htm;</span><br><span class="line"> # }</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> # HTTPS server</span><br><span class="line"> #</span><br><span class="line"> #server {</span><br><span class="line"> # listen 443 ssl;</span><br><span class="line"> # server_name localhost;</span><br><span class="line"></span><br><span class="line"> # ssl_certificate cert.pem;</span><br><span class="line"> # ssl_certificate_key cert.key;</span><br><span class="line"></span><br><span class="line"> # ssl_session_cache shared:SSL:1m;</span><br><span class="line"> # ssl_session_timeout 5m;</span><br><span class="line"></span><br><span class="line"> # ssl_ciphers HIGH:!aNULL:!MD5;</span><br><span class="line"> # ssl_prefer_server_ciphers on;</span><br><span class="line"></span><br><span class="line"> # location / {</span><br><span class="line"> # root html;</span><br><span class="line"> # index index.html index.htm;</span><br><span class="line"> # }</span><br><span class="line"> #}</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>修改之后，重新启动Keepalive+Nginx2台主备节点。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@nginx1 conf]# /usr/local/nginx/sbin/nginx -s reload</span><br></pre></td></tr></table></figure><p>我们现在就用虚拟IP192.168.43.150来访问我们刚才上传的图片，只是注意在地址栏中要记得输入fastdfs（这是我们nginx.conf文件中location /fastdfs{}规则规定的）。如下图所示，发现，我们通过虚拟IP便可以访问我们上传的图片了。这样的好处是，对用户来说，只需要访问这个虚拟IP就可以了，不用关心FastDFS集群内部的转发机制。<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-1e5312e9fa770c87.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="集群VIP访问截图.png" title=""> </div> <div class="image-caption">集群VIP访问截图.png</div> </figure><br>至此，分布式文件系统就搭建完成了，在通过Java访问时，只要在配置文件配置所有的Tracker节点IP信息就可以啦！</p><h3 id="补充说明："><a href="#补充说明：" class="headerlink" title="补充说明："></a>补充说明：</h3><p>如果Tracker 服务、Storage服务、Nginx服务开机后没有自启动，请执行一下操作并进行重启</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost ~]# chkconfig --add fdfs_trackerd</span><br><span class="line">[root@localhost ~]# chkconfig --add fdfs_storaged</span><br></pre></td></tr></table></figure><p>编辑目录下的/etc/rc.d/rc.local,内容如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">#!/bin/bash</span><br><span class="line"># THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES</span><br><span class="line">#</span><br><span class="line"># It is highly advisable to create own systemd services or udev rules</span><br><span class="line"># to run scripts during boot instead of using this file.</span><br><span class="line">#</span><br><span class="line"># In contrast to previous versions due to parallel execution during boot</span><br><span class="line"># this script will NOT be run after all other services.</span><br><span class="line">#</span><br><span class="line"># Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure</span><br><span class="line"># that this script will be executed during boot.</span><br><span class="line"></span><br><span class="line">touch /var/lock/subsys/local</span><br><span class="line">/usr/local/nginx/sbin/nginx</span><br></pre></td></tr></table></figure></p><p>主要增加了Nginx的启动，之后进行文件生效，重新启动系统</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost ~]# chmod +x /etc/rc.d/rc.local</span><br><span class="line">[root@localhost ~]# source /etc/rc.d/rc.local </span><br><span class="line">[root@localhost ~]# reboot</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="分布式集群搭建结构"><a href="#分布式集群搭建结构" class="headerlink" title="分布式集群搭建结构"></a>分布式集群搭建结构</h3><ul>
<li>双Tracker</li>
<li>2组Group</li>
<li>轮
</summary>
</entry>
<entry>
<title>Hive高级操作</title>
<link href="https://yongliangzhang.github.io/blogs/2018/02/01/Hive%E9%AB%98%E7%BA%A7%E6%9F%A5%E8%AF%A2%E6%93%8D%E4%BD%9C/"/>
<id>https://yongliangzhang.github.io/blogs/2018/02/01/Hive高级查询操作/</id>
<published>2018-02-01T05:51:58.981Z</published>
<updated>2018-02-01T05:51:13.000Z</updated>
<content type="html"><![CDATA[<h1 id="Hive高级操作"><a href="#Hive高级操作" class="headerlink" title="Hive高级操作"></a>Hive高级操作</h1><h5 id="1-使用LIKE、AS创建表-表重命名，添加、修改、删除列"><a href="#1-使用LIKE、AS创建表-表重命名，添加、修改、删除列" class="headerlink" title="1.使用LIKE、AS创建表,表重命名，添加、修改、删除列"></a>1.使用LIKE、AS创建表,表重命名，添加、修改、删除列</h5><ul><li>表结构数据复制<br><br>根据已存在的表结构，使用like关键字，复制一个表结构一模一样的新表</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">hive> create table student_info2 like student_info;</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.73 seconds</span><br><span class="line">hive> show tables;</span><br><span class="line">OK</span><br><span class="line">employee</span><br><span class="line">student_info</span><br><span class="line">student_info2</span><br><span class="line">student_school_info</span><br><span class="line">student_school_info_external_partition</span><br><span class="line">student_school_info_partition</span><br><span class="line">Time taken: 0.15 seconds, Fetched: 6 row(s)</span><br></pre></td></tr></table></figure><p>根据已经存在的表，使用as关键字，创建一个与查询结果字段一致的表，同时将查询结果数据插入到新表</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">create table student_info3 as select * from student_info;</span><br></pre></td></tr></table></figure><p>只有student_id,name两个字段的表</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">create table student_info4 as select student_id,name from student_info;</span><br></pre></td></tr></table></figure><ul><li>表重命名<br><br>student_info4表重命名为student_id_name</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">alter table student_info4 rename to student_id_name;</span><br></pre></td></tr></table></figure><ul><li>添加列<br><br>给student_info3表添加性别列,新添加的字段会在所有列最后，分区列之前，在添加新列之前已经存在的数据文件中。<br><br>如果没有新添加列对应的数据，在查询的时候显示为空。添加多个列用逗号隔开。</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">hive> alter table student_info3 add columns(gender string comment '性别');</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.185 seconds</span><br></pre></td></tr></table></figure><ul><li>删除列或修改列<br><br>修改列，将继续存在的列再定义一遍，需要替换的列重新定义</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">hive> alter table student_info3 replace columns(student_id string,name string,age int,origin string,gender2 int);</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.422 seconds</span><br></pre></td></tr></table></figure><p>删除列,将继续存在的列再定义一遍，需要删除的列不再定义</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">hive> alter table student_info3 replace columns(student_id string,name string,age int,origin string);</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.529 seconds</span><br></pre></td></tr></table></figure><h4 id="2-分桶表使用"><a href="#2-分桶表使用" class="headerlink" title="2.分桶表使用"></a>2.分桶表使用</h4><ul><li>创建分桶表<br><br>按照指定字段取它的hash散列值分桶,创建学生入学信息分桶表</li></ul><table><thead><tr><th style="text-align:center">字段名称</th><th style="text-align:center">类型</th><th style="text-align:center">注释</th><th style="text-align:center">分桶字段</th></tr></thead><tbody><tr><td style="text-align:center">student_id</td><td style="text-align:center">string</td><td style="text-align:center">学生ID</td><td style="text-align:center">是</td></tr><tr><td style="text-align:center">name</td><td style="text-align:center">string</td><td style="text-align:center">姓名</td><td style="text-align:center">否</td></tr><tr><td style="text-align:center">age</td><td style="text-align:center">int</td><td style="text-align:center">年龄</td><td style="text-align:center">否</td></tr><tr><td style="text-align:center">origin</td><td style="text-align:center">string</td><td style="text-align:center">学院ID</td><td style="text-align:center">否</td></tr></tbody></table><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">create table rel.student_info_bucket(</span><br><span class="line">student_id string,</span><br><span class="line">name string,</span><br><span class="line">age int,</span><br><span class="line">origin string</span><br><span class="line">)</span><br><span class="line">clustered by (student_id) sorted by (student_id asc) into 4 buckets </span><br><span class="line">row format delimited </span><br><span class="line">fields terminated by '\t' </span><br><span class="line">lines terminated by '\n' </span><br><span class="line">stored as textfile;</span><br></pre></td></tr></table></figure><ul><li>分桶表插入数据<br><br>向student_info_bucket分桶表插入数据</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">set hive.enforce.bucketing = true;</span><br><span class="line">set mapreduce.job.reduces=4;</span><br><span class="line">insert overwrite table student_info_bucket </span><br><span class="line">select student_id,name,age,origin </span><br><span class="line">from student_info </span><br><span class="line">cluster by(student_id);</span><br></pre></td></tr></table></figure><p>查看hdfs分桶文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 ~]# hadoop fs -ls /user/hive/warehouse/rel.db/student_info_bucket</span><br><span class="line">Found 4 items</span><br><span class="line">-rwxr-xr-x 3 hadoop supergroup 78 2018-01-24 19:33 /user/hive/warehouse/rel.db/student_info_bucket/000000_0</span><br><span class="line">-rwxr-xr-x 3 hadoop supergroup 84 2018-01-24 19:33 /user/hive/warehouse/rel.db/student_info_bucket/000001_0</span><br><span class="line">-rwxr-xr-x 3 hadoop supergroup 80 2018-01-24 19:33 /user/hive/warehouse/rel.db/student_info_bucket/000002_0</span><br><span class="line">-rwxr-xr-x 3 hadoop supergroup 81 2018-01-24 19:33 /user/hive/warehouse/rel.db/student_info_bucket/000003_0</span><br></pre></td></tr></table></figure><p><strong>说明：</strong></p><p>分桶表一般不使用load向分桶表中导入数据，因为load导入数据只是将数据复制到表的数据存储目录下，hive并不会在load的时候对数据进行分析然后按照分桶字段分桶，load只会将一个文件全部导入到分桶表中，并没有分桶。一般采用insert从其他表向分桶表插入数据。<br>分桶表在创建表的时候只是定义表的模型，插入的时候需要做如下操作：<br>在每次执行分桶插入的时候在当前执行的session会话中要设置</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hive.enforce.bucketing=true;</span><br></pre></td></tr></table></figure><p>声明本次执行的是一次分桶操作。需要指定reduce个数与分桶的数量相同</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">set mapreduce.job.reduces=4，</span><br></pre></td></tr></table></figure><p>这样才能保证有多少桶就生成多少个文件。<br>如果定义了按照分桶字段排序，需要在从其他表查询数据过程中将数据按照分区字段排序之后插入各个桶中，分桶表并不会将各分桶中的数据排序。排序和分桶的字段相同的时候使用Cluster by(字段),cluster by 默认按照分桶字段在桶内升序排列，如果需要在桶内降序排列，使用distribute by (col) sort by (col desc)组合实现。</p><h4 id="3-导出数据"><a href="#3-导出数据" class="headerlink" title="3.导出数据"></a>3.导出数据</h4><ul><li>使用insert将student_info表数据导出到本地指定路径<br></li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">insert overwrite local directory '/home/hadoop/apps/hive_test_data/export_data' </span><br><span class="line">row format delimited fields terminated by '\t' select * from student_info;</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop01 export_data]# cat 000000_0 </span><br><span class="line">1xiaoming2011</span><br><span class="line">2xiaobai2131</span><br><span class="line">3zhangfei2244</span><br><span class="line">4likui1944</span><br><span class="line">5zhaoyun2113</span><br><span class="line">6zhangsan2011</span><br><span class="line">7lisi1911</span><br><span class="line">8wangwu2331</span><br><span class="line">9zhaofei1921</span><br><span class="line">10zhangyan2021</span><br><span class="line">11lihe2022</span><br><span class="line">12caoyang1732</span><br><span class="line">13lihao1932</span><br><span class="line">14zhaoming2150</span><br><span class="line">15zhouhong1851</span><br><span class="line">16yangshuo2333</span><br><span class="line">17xiaofei2413</span><br><span class="line">18liman2313</span><br><span class="line">19qianbao2013</span><br><span class="line">20sunce2141</span><br></pre></td></tr></table></figure><p>导出数据到本地的常用方法<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 export_data]$ hive -e"select * from rel.student_info"> /home/hadoop/student_info_data.txt</span><br></pre></td></tr></table></figure></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@hadoop01 ~]$ cat student_info_data.txt </span><br><span class="line">1xiaoming2011</span><br><span class="line">2xiaobai2131</span><br><span class="line">3zhangfei2244</span><br><span class="line">4likui1944</span><br><span class="line">5zhaoyun2113</span><br><span class="line">6zhangsan2011</span><br><span class="line">7lisi1911</span><br><span class="line">8wangwu2331</span><br><span class="line">9zhaofei1921</span><br><span class="line">10zhangyan2021</span><br><span class="line">11lihe2022</span><br><span class="line">12caoyang1732</span><br><span class="line">13lihao1932</span><br><span class="line">14zhaoming2150</span><br><span class="line">15zhouhong1851</span><br><span class="line">16yangshuo2333</span><br><span class="line">17xiaofei2413</span><br><span class="line">18liman2313</span><br><span class="line">19qianbao2013</span><br><span class="line">20sunce2141</span><br></pre></td></tr></table></figure><p>默认结果分隔符：’\t’</p><h4 id="4-关联查询"><a href="#4-关联查询" class="headerlink" title="4.关联查询"></a>4.关联查询</h4><p> 创建2张表</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">create table rel.a(</span><br><span class="line">id int,</span><br><span class="line">name string</span><br><span class="line">)</span><br><span class="line">row format delimited </span><br><span class="line">fields terminated by '\t' </span><br><span class="line">lines terminated by '\n' </span><br><span class="line">stored as textfile;</span><br><span class="line"></span><br><span class="line">create table rel.b(</span><br><span class="line">id int,</span><br><span class="line">name string</span><br><span class="line">)</span><br><span class="line">row format delimited </span><br><span class="line">fields terminated by '\t' </span><br><span class="line">lines terminated by '\n' </span><br><span class="line">stored as textfile;</span><br></pre></td></tr></table></figure><p>导入数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">hive> load data local inpath '/home/hadoop/apps/hive_test_data/a_join_data' into table a;</span><br><span class="line">Loading data to table rel.a</span><br><span class="line">Table rel.a stats: [numFiles=1, totalSize=61]</span><br><span class="line">OK</span><br><span class="line">Time taken: 1.79 seconds</span><br><span class="line">hive> load data local inpath '/home/hadoop/apps/hive_test_data/b_join_data' into table b;</span><br><span class="line">Loading data to table rel.b</span><br><span class="line">Table rel.b stats: [numFiles=1, totalSize=38]</span><br><span class="line">OK</span><br><span class="line">Time taken: 0.562 seconds</span><br></pre></td></tr></table></figure><ul><li>inner或inner join<br><br><br>两个表通过id关联，只把id值相等的数据查询出来。join的查询结果与inner join的查询结果相同。</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select * from a join b on a.id=b.id;</span><br></pre></td></tr></table></figure><p>等同于</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">select * from a inner join b on a.id=b.id;</span><br><span class="line">.....</span><br><span class="line">OK</span><br><span class="line">1a1AA</span><br><span class="line">2b2BB</span><br><span class="line">3c3CC</span><br><span class="line">6f6FF</span><br><span class="line">Time taken: 44.337 seconds, Fetched: 4 row(s)</span><br></pre></td></tr></table></figure><ul><li>full outer join或full join</li></ul><p>两个表通过id关联，把两个表的数据全部查询出来<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">OK</span><br><span class="line">1a1AA</span><br><span class="line">2b2BB</span><br><span class="line">3c3CC</span><br><span class="line">4dNULLNULL</span><br><span class="line">5eNULLNULL</span><br><span class="line">6f6FF</span><br><span class="line">7gNULLNULL</span><br><span class="line">8hNULLNULL</span><br><span class="line">9iNULLNULL</span><br><span class="line">10jNULLNULL</span><br><span class="line">11kNULLNULL</span><br><span class="line">12lNULLNULL</span><br><span class="line">13mNULLNULL</span><br><span class="line">14nNULLNULL</span><br><span class="line">NULLNULL20TT</span><br><span class="line">NULLNULL21UU</span><br><span class="line">NULLNULL22vv</span><br></pre></td></tr></table></figure></p><ul><li>left join</li></ul><p>左连接时，左表中出现的join字段都保留，右表没有连接上的都为空</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">OK</span><br><span class="line">1a1AA</span><br><span class="line">2b2BB</span><br><span class="line">3c3CC</span><br><span class="line">4dNULLNULL</span><br><span class="line">5eNULLNULL</span><br><span class="line">6f6FF</span><br><span class="line">7gNULLNULL</span><br><span class="line">8hNULLNULL</span><br><span class="line">9iNULLNULL</span><br><span class="line">10jNULLNULL</span><br><span class="line">11kNULLNULL</span><br><span class="line">12lNULLNULL</span><br><span class="line">13mNULLNULL</span><br><span class="line">14nNULLNULL</span><br></pre></td></tr></table></figure><ul><li>right join</li></ul><p>右连接时，右表中出现的join字段都保留，左表没有连接上的都是空</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select * from a right join b on a.id=b.id;</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">OK</span><br><span class="line">1a1AA</span><br><span class="line">2b2BB</span><br><span class="line">3c3CC</span><br><span class="line">6f6FF</span><br><span class="line">NULLNULL20TT</span><br><span class="line">NULLNULL21UU</span><br><span class="line">NULLNULL22vv</span><br><span class="line">Time taken: 25.188 seconds, Fetched: 7 row(s)</span><br></pre></td></tr></table></figure><ul><li>left semi join</li></ul><p>左半连接实现了类似IN/EXISTS的查询语义，输出符合条件的左表内容。<br>hive不支持in …exists这种关系型数据库中的子查询结构，hive暂时不支持右半连接。<br>例如：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select a.id, a.name from a where a.id in (select b.id from b);</span><br></pre></td></tr></table></figure><p>使用Hive对应于如下语句：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select a.id,a.name from a left semi join b on a.id = b.id;</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">OK</span><br><span class="line">1a</span><br><span class="line">2b</span><br><span class="line">3c</span><br><span class="line">6f</span><br><span class="line">Time taken: 27.42 seconds, Fetched: 4 row(s)</span><br></pre></td></tr></table></figure><ul><li>map side join</li></ul><p>使用分布式缓存将小表数据加载都各个map任务中，在map端完成join，map任务输出后，不需要将数据拷贝到reducer阶段再进行join，<br>降低的数据在网络节点之间传输的开销。多表关联数据倾斜优化的一种手段。多表连接，如果只有一个表比较大，其他表都很小，<br>则join操作会转换成一个只包含map的Job。运行日志中会出现Number of reduce tasks is set to 0 since there’s no reduce operator<br>没有reduce的提示。<br>例如：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select /*+ mapjoin(b) */ a.id, a.name from a join b on a.id = b.id</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">Total MapReduce CPU Time Spent: 1 seconds 320 msec</span><br><span class="line">OK</span><br><span class="line">1a</span><br><span class="line">2b</span><br><span class="line">3c</span><br><span class="line">6f</span><br><span class="line">Time taken: 25.538 seconds, Fetched: 4 row(s)</span><br></pre></td></tr></table></figure><h4 id="5-Hive内置函数"><a href="#5-Hive内置函数" class="headerlink" title="5.Hive内置函数"></a>5.Hive内置函数</h4><p>创建用户评分表</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">create table rel.user_core_info(</span><br><span class="line">user_id string,</span><br><span class="line">age int,</span><br><span class="line">gender string,</span><br><span class="line">core int</span><br><span class="line">)</span><br><span class="line">row format delimited fields terminated by '\t' </span><br><span class="line">lines terminated by '\n'</span><br><span class="line">stored as textfile;</span><br></pre></td></tr></table></figure><p>导入数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">load data local inpath '/home/hadoop/apps/hive_test_data/user_core.txt' into table rel.user_core_info;</span><br></pre></td></tr></table></figure><ul><li><ol><li>条件函数 case when <br><br>语法1：CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END <br><br>说明：如果a等于b，那么返回c；如果a等于d，那么返回e；否则返回f<br>例如：</li></ol></li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hive> select case 1 when 2 then 'two' when 1 then 'one' else 'zero' end;</span><br><span class="line">OK</span><br><span class="line">one</span><br><span class="line">Time taken: 0.152 seconds, Fetched: 1 row(s)</span><br></pre></td></tr></table></figure><p>语法2：CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END<br><br><br>说明：如果a为TRUE，则返回b；如果c为TRUE，则返回d；否则返回e<br>例如：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hive> select case when 1=2 then 'two' when 1=1 then 'one' else 'zero' end;</span><br><span class="line">OK</span><br><span class="line">one</span><br><span class="line">Time taken: 0.33 seconds, Fetched: 1 row(s)</span><br></pre></td></tr></table></figure><p>查询用户评分表，每个年龄段的最大评分值</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">select gender,</span><br><span class="line">case when age<=20 then 'p0' when age>20 and age<=50 then 'p1' when age>=50 then 'p3' else 'p0' end,</span><br><span class="line">max(core) max_core</span><br><span class="line">from rel.user_core_info </span><br><span class="line">group by gender,</span><br><span class="line">case when age<=20 then 'p0' when age>20 and age<=50 then 'p1' when age>=50 then 'p3' else 'p0' end;</span><br></pre></td></tr></table></figure><p>结果为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">OK</span><br><span class="line">femalep090</span><br><span class="line">femalep195</span><br><span class="line">femalep390</span><br><span class="line">malep080</span><br><span class="line">malep180</span><br><span class="line">malep380</span><br><span class="line">Time taken: 28.461 seconds, Fetched: 6 row(s)</span><br></pre></td></tr></table></figure><ul><li><ol><li>自定义UDF函数</li></ol></li></ul><p>当Hive提供的内置函数无法满足你的业务处理需要时，此时就可以考虑使用用户自定义函数（UDF：user-defined function）。<br>UDF 作用于单个数据行，产生一个数据行作为输出。<br>步骤：</p><ol><li>先开发一个java类，继承UDF，并重载evaluate方法</li><li>打成jar包上传到服务器</li><li><p>在使用的时候将jar包添加到hive的classpath</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hive>add jar /home/hadoop/apps/hive_test_data/HiveUdfPro-1.0-SNAPSHOT.jar;</span><br></pre></td></tr></table></figure></li><li><p>创建临时函数与开发好的java class关联</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hive>create temporary function age_partition as 'cn.chinahadoop.udf.AgePartitionFunction';</span><br></pre></td></tr></table></figure></li><li><p>即可在hql中使用自定义的函数<br>新建Maven 项目<br>Pom 信息如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"</span><br><span class="line"> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"></span><br><span class="line"> <modelVersion>4.0.0</modelVersion></span><br><span class="line"></span><br><span class="line"> <groupId>com.yongliang.udf</groupId></span><br><span class="line"> <artifactId>HiveUdfPro</artifactId></span><br><span class="line"> <version>1.0-SNAPSHOT</version></span><br><span class="line"> <packaging>jar</packaging></span><br><span class="line"></span><br><span class="line"> <name>HiveUdfPro</name></span><br><span class="line"> <url>http://maven.apache.org</url></span><br><span class="line"> <properties></span><br><span class="line"> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding></span><br><span class="line"> </properties></span><br><span class="line"> <dependencies></span><br><span class="line"> <dependency></span><br><span class="line"> <groupId>org.apache.hive</groupId></span><br><span class="line"> <artifactId>hive-exec</artifactId></span><br><span class="line"> <version>1.2.2</version></span><br><span class="line"> </dependency></span><br><span class="line"> <dependency></span><br><span class="line"> <groupId>org.apache.hadoop</groupId></span><br><span class="line"> <artifactId>hadoop-common</artifactId></span><br><span class="line"> <version>2.7.0</version></span><br><span class="line"> </dependency></span><br><span class="line"> </dependencies></span><br><span class="line"> <build></span><br><span class="line"> <plugins></span><br><span class="line"> <plugin></span><br><span class="line"> <groupId>org.apache.maven.plugins</groupId></span><br><span class="line"> <artifactId>maven-compiler-plugin</artifactId></span><br><span class="line"> <version>2.3.2</version></span><br><span class="line"> <executions></span><br><span class="line"> <execution></span><br><span class="line"> <id>default-compile</id></span><br><span class="line"> <phase>compile</phase></span><br><span class="line"> <goals></span><br><span class="line"> <goal>compile</goal></span><br><span class="line"> </goals></span><br><span class="line"> <configuration></span><br><span class="line"> <encoding>UTF-8</encoding></span><br><span class="line"> </configuration></span><br><span class="line"> </execution></span><br><span class="line"> </executions></span><br><span class="line"> </plugin></span><br><span class="line"> </plugins></span><br><span class="line"> </build></span><br><span class="line"></project></span><br></pre></td></tr></table></figure></li></ol><p>新建类继承UDF</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">package com.yongliang.udf;</span><br><span class="line"></span><br><span class="line">import org.apache.hadoop.hive.ql.exec.UDF;</span><br><span class="line">/**</span><br><span class="line"> * 创建时间 : 2018/1/27 15:35</span><br><span class="line"> * 类描述 : Hive UDF自定义函数，作用于单个数据行，产生一个数据行作为输出</span><br><span class="line"> * @author zhangyonglaing</span><br><span class="line"> */</span><br><span class="line">public class AgePartitionFunction extends UDF {</span><br><span class="line"> public String evaluate(int age) {</span><br><span class="line"> String partition = "p0";</span><br><span class="line"> if(age <=20){</span><br><span class="line"> partition = "p0";</span><br><span class="line"> }else if(age > 20 && age <=50){</span><br><span class="line"> partition = "p1";</span><br><span class="line"> }else if(age > 50){</span><br><span class="line"> partition = "p2";</span><br><span class="line"> }</span><br><span class="line"> return partition;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>将项目进行打包<figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-a9280913d5aa3b96.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="Hive UDF 自定义函数" title=""> </div> <div class="image-caption">Hive UDF 自定义函数</div> </figure></p><p> <strong>说明:</strong></p><p> <strong>如出现以下异常信息：<br>Failed to execute goal on project hive-exec:<br>Could not resolve dependencies for project org.apache.hive:hive-exec:jar:2.3.0:<br>Could not find artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in alimaven (<a href="http://maven.aliyun.com/nexus/content/groups/public/" target="_blank" rel="noopener">http://maven.aliyun.com/nexus/content/groups/public/</a>) -> [Help 1]</strong><br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="http://upload-images.jianshu.io/upload_images/1784853-67071943803d5760?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="错误异常信息" title=""> </div> <div class="image-caption">错误异常信息</div> </figure> <br><br> 请手动下载Jar包pentaho-aggdesigner-algorithm/5.1.5-jhyde.jar 下载地址：<a href="https://public.nexus.pentaho.org/content/groups/omni/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/" target="_blank" rel="noopener">https://public.nexus.pentaho.org/content/groups/omni/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/</a><br> 将Jar包放置在本地Maven仓库org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde路径下，之后进行重新打包。</p><ul><li>将jar包添加到hive的classpath</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">hive> add jar /home/hadoop/apps/HiveUdfPro-1.0-SNAPSHOT.jar;</span><br><span class="line">Added [/home/hadoop/apps/HiveUdfPro-1.0-SNAPSHOT.jar] to class path</span><br><span class="line">Added resources: [/home/hadoop/apps/HiveUdfPro-1.0-SNAPSHOT.jar]</span><br></pre></td></tr></table></figure><p>创建临时函数与开发好的java class关联</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hive> create temporary function age_partition as 'com.yongliang.udf.AgePartitionFunction';</span><br></pre></td></tr></table></figure><p>在hql中使用自定义的函数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">select gender,</span><br><span class="line">age_partition(age),</span><br><span class="line">max(core) max_core</span><br><span class="line">from rel.user_core_info </span><br><span class="line">group by gender,</span><br><span class="line">age_partition(age);</span><br></pre></td></tr></table></figure><p>结果为：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">OK</span><br><span class="line">femalep090</span><br><span class="line">femalep195</span><br><span class="line">femalep290</span><br><span class="line">malep080</span><br><span class="line">malep180</span><br><span class="line">malep280</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<h1 id="Hive高级操作"><a href="#Hive高级操作" class="headerlink" title="Hive高级操作"></a>Hive高级操作</h1><h5 id="1-使用LIKE、AS创建表-表重命名，添加、修改、删除列"><a href="#
</summary>
</entry>
<entry>
<title>Hello World</title>
<link href="https://yongliangzhang.github.io/blogs/2018/01/30/hello-world/"/>
<id>https://yongliangzhang.github.io/blogs/2018/01/30/hello-world/</id>
<published>2018-01-30T14:51:46.487Z</published>
<updated>2018-01-30T14:51:46.487Z</updated>
<content type="html"><![CDATA[<p>Welcome to <a href="https://hexo.io/" target="_blank" rel="noopener">Hexo</a>! This is your very first post. Check <a href="https://hexo.io/docs/" target="_blank" rel="noopener">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="https://hexo.io/docs/troubleshooting.html" target="_blank" rel="noopener">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues" target="_blank" rel="noopener">GitHub</a>.</p><h2 id="Quick-Start"><a href="#Quick-Start" class="headerlink" title="Quick Start"></a>Quick Start</h2><h3 id="Create-a-new-post"><a href="#Create-a-new-post" class="headerlink" title="Create a new post"></a>Create a new post</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo new <span class="string">"My New Post"</span></span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/writing.html" target="_blank" rel="noopener">Writing</a></p><h3 id="Run-server"><a href="#Run-server" class="headerlink" title="Run server"></a>Run server</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo server</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/server.html" target="_blank" rel="noopener">Server</a></p><h3 id="Generate-static-files"><a href="#Generate-static-files" class="headerlink" title="Generate static files"></a>Generate static files</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo generate</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/generating.html" target="_blank" rel="noopener">Generating</a></p><h3 id="Deploy-to-remote-sites"><a href="#Deploy-to-remote-sites" class="headerlink" title="Deploy to remote sites"></a>Deploy to remote sites</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo deploy</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/deployment.html" target="_blank" rel="noopener">Deployment</a></p>]]></content>
<summary type="html">
<p>Welcome to <a href="https://hexo.io/" target="_blank" rel="noopener">Hexo</a>! This is your very first post. Check <a href="https://hexo.
</summary>
</entry>
</feed>