/
atom.xml
514 lines (319 loc) · 250 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Kay Wu's blog</title>
<link href="/atom.xml" rel="self"/>
<link href="http://kaywu.xyz/"/>
<updated>2020-03-23T09:41:54.444Z</updated>
<id>http://kaywu.xyz/</id>
<author>
<name>Kay Wu</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>MIT 6.824 分布式系统笔记 — Raft</title>
<link href="http://kaywu.xyz/2020/03/23/distributed-system-raft/"/>
<id>http://kaywu.xyz/2020/03/23/distributed-system-raft/</id>
<published>2020-03-23T03:03:57.000Z</published>
<updated>2020-03-23T09:41:54.444Z</updated>
<content type="html"><![CDATA[<p>先简单介绍下 Raft 是什么。Raft 是一种共识算法，解决的是分布式系统对某个提案，大部分节点达成一致意见的过程。<br>上面这样讲感觉很抽象，我们举一个具体的例子。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"> server</span><br><span class="line"> _________</span><br><span class="line">c1 -> | (k, v1) |</span><br><span class="line">c2 -> | (k, v2) |</span><br><span class="line"> |_________|</span><br></pre></td></tr></table></figure><p>假如我们有一个分布式的 KV 系统，有两个客户端 c1, c2，有两个服务端 s1, s2。客户端向服务端发请求修改 k 的值。如上所示，c1 发出将 k 修改为 v1 的请求，c2 发出将 k 修改为 v2 的请求。那么请问这个 KV 系统 k 的值最后为多少呢？</p><p>你肯定会说这依赖于 v1 和 v2 的顺序，v1 先 v2 后则最终值为 v2，反之则为 v1。但这要求 v1 和 v2 之间是有顺序关系的。一种做法是给 v1、v2 附上一个顺序，比如 vector clock。</p><p>还有一种更简单的做法，就是我们让其中一个 server 作为 leader 处理请求，而在单一 server 上，命令执行是自然有序的，之后我们再把这种次序转发给其他 server 就可以了。</p><p>但这又带来了其他的问题，怎么确定这个 leader 呢？简单点的可以直接固定一个，但这就会导致 leader 挂掉后群龙无首。我们也可以让服务器选一个 leader 出来，leader 挂掉后剩下的服务器再次选举就可以了。而这时，就轮到 Raft 出场了，Raft 会选举出一个 leader 出来，leader 执行的命令会以 log 的形式传给剩下的 server，使 server 的状态保持一致。</p><a id="more"></a><h2 id="流程"><a href="#流程" class="headerlink" title="流程"></a>流程</h2><p>可以先看 <a href="http://thesecretlivesofdata.com/raft/" target="_blank" rel="noopener">Raft 动画演示</a> 来直观地了解这一过程。</p><p>Raft 将时间分阶段，用 term 来表示。我们先说同一个 term 中发生的事，再讲不同 term 之间发生的事。</p><p>Raft 把 server 分为三种状态，candidate（候选人），follower（追随者）, leader（领导者）。</p><p>在 term 1 开始的时候，所有的 server 都是 follower 的状态。follower 会记录接收 leader 消息的最近时间，如果间隔超过 timeout，则会从 follower 变为 candidate 开始新一轮选举。</p><p>什么是选举呢？其实就是给其他 server 发消息（RequestVote）让它们投自己当 leader。假设 3 个 server，s1 开始选举，发消息给 s2、s3 让它们投 s1 为 leader。假设 s2 之前没有投过其他人，且 s1 符合条件，s2 就回复给 s1，好的，我投你。s1 一看选我的人（包括自己）超过半数，s1 就将状态改为 leader。</p><p>此时 s1 作为 leader 接收客户端的请求。当接收到客户端的请求 cmd1 时，s1 首先在自己的 log 上加上 <code>Entry(cmd1)</code>，然后发送包含这条 Entry 的 RPC（AppendEntries RPC，以下简称为 AE）给 s2、s3。</p><p>s2、s3 收到 <code>AE(cmd1)</code>，发现符合条件，将状态从 candidate 变为 follower，并在自己的 log 上加上 <code>Entry(cmd1)</code>，然后回复给 s1 表示收到了。当 s1 收到超过半数（包括自己）的回复时，就可以 commit 这条 <code>Entry(cmd1)</code>，之后就可以 apply（执行）了。</p><p>当 s1 收到 cmd2 后，会做同样的事情，在 log 上添加 <code>Entry(cmd2)</code>，并发送 AE 给 s2、s3。<br>s2、s3 接收到 AE 之后，除了保存 log 并回复，它们还发现前一条 Entry 已经可以 commit 了。于是他们就追随 leader 的脚步，对 <code>Entry(cmd1)</code> 进行 commit，然后 apply。</p><p>上面讲的就是同一个 term 中发生的事了。leader 接收请求，通过 log 发送给 follower，如果半数以上的 follower 收到，就 commit 该条 log，之后 apply。follower 会在之后的请求中了解 leader 的 commit 情况，进行 commit 之后 apply。<br>如果某个 follower 在 timeout 之内没有接收到 leader 的消息，就会将 term + 1，开始新的选举。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">s1 start election</span><br><span class="line">s1 -> s2 RequestVote RPC(term 1, latest log info)</span><br><span class="line">s1 -> s3 RequestVote RPC(term 1, latest log info)</span><br><span class="line">s1 <- s2 say yes</span><br><span class="line">s1 <- s3 network loss</span><br><span class="line">s1 become leader</span><br><span class="line">c -> s1 cmd1</span><br><span class="line">s1 add Entry(index 1, cmd1, term 1) in its log</span><br><span class="line">s1 -> s2 AE(cmd1, term 1, leaderCommit 0) RPC</span><br><span class="line">s1 -> s3 AE(cmd1, term 1, leaderCommit 0) RPC</span><br><span class="line">s1 <- s2 received, add Entry(index 1, cmd1, term 1) in its log, s2 change to follower</span><br><span class="line">s1 <- s3 received, add Entry(index 1, cmd1, term 1) in its log, s3 change to follower</span><br><span class="line">s1 commit Entry(index 1, cmd1, term 1), later apply</span><br><span class="line">c -> s1 cmd2</span><br><span class="line">s1 add Entry(index 2, cmd2, term 1) in its log</span><br><span class="line">s1 -> s2 AE(cmd2, term 1, leaderCommit 1) RPC</span><br><span class="line">s1 -> s3 AE(cmd2, term 1, leaderCommit 1) RPC</span><br><span class="line">s1 <- s2 received, add Entry(index 2, cmd2, term 1) in its log, commit Entry(index 1, cmd1, term 1), later apply</span><br><span class="line">s1 <- s3 received, add Entry(index 2, cmd2, term 1) in its log, commit Entry(index 1, cmd1, term 1), later apply</span><br></pre></td></tr></table></figure><h2 id="细节"><a href="#细节" class="headerlink" title="细节"></a>细节</h2><p>了解了流程之后，我发现 Raft 还是挺容易理解的。但在写了 Lab 2 Raft 之后，我才发现里面隐藏了那么多的细节。<br>下面使用 FAQ 来举例说明 Raft 的细节。</p><ol><li>选举一定会成功吗？不成功会怎样？</li></ol><p>选举不一定会成功。比如有 5 个 server, 同一时刻 s1、s2、s3 都开始新的选举，s1 获得 s4 的支持，s2 获得 s5 的支持，在这时候没有一个 server 获得的票数超过半数，选举没有成功。server 会在 timeout 后重新开始新一轮的选举。同时为了减少同时开始选举造成的选票分散的情况，server 会在每一次判断是否需要选举时设置一个随机的 timeout 而不是一直使用固定的 timeout。</p><ol start="2"><li>server 接收到其他 server 的选举请求时，会拒绝吗？</li></ol><p>Raft 的选举不像现实中的选举，它采取的是先到先得的策略，你先告诉我让我投你，符合条件的话我就投你，不再投其他人。<br>这边详细说明会拒绝的情况, RequestVote 表示选举请求：</p><ul><li>1) RequestVote.term < server.currentTerm。选举请求的 term 落后了，server 会拒绝，并返回 server.currentTerm 以供其他 server 更新。</li><li>2) server 暂未投票给其他人或者投的正是 RequestVote 中的 candidate，并且 candidate 的 log 不能落后于 server 的 log，即 <code>lastCandidateLog.term > lastserverLog.term || (lastCandidateLog.term == lastserverLog.term && lastCandidateLog.index >= lastserverLog.index)</code>，返回 yes。<br>这里我们说明下第 2 点对于 log 的要求，它保证了不会有不符合条件的 server 选举成为 leader。</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"># log 下面的 1 表示该 log entry 的 term 为 1</span><br><span class="line">(1)</span><br><span class="line">index 1 2</span><br><span class="line">server log</span><br><span class="line">------------------------</span><br><span class="line">s1 1</span><br><span class="line">s2 1</span><br><span class="line">s3 1</span><br><span class="line"></span><br><span class="line">(2)</span><br><span class="line">index 1 2</span><br><span class="line">server log</span><br><span class="line">------------------------</span><br><span class="line">s1 1 1</span><br><span class="line">s2 1 1</span><br><span class="line">s3 1</span><br><span class="line"></span><br><span class="line">(3)</span><br><span class="line">index 1 2</span><br><span class="line">server log</span><br><span class="line">------------------------</span><br><span class="line">s1 1 1</span><br><span class="line">s2 1</span><br><span class="line">s3 1</span><br></pre></td></tr></table></figure><p>假设 s1 为 leader，<code>Entry(index 1, term 1)</code> 已经全部复制到 3 个 server 上了，如 (1) 所示。</p><p>s1 又添加了 <code>Entry(index 2, term 1)</code>，该 Entry 只被复制到 s2 上，s1 就挂了，示例 (2)。在这种情况下，s1 有可能已经 apply 过 <code>Entry(index 2, term 1)</code>。所以 <code>Entry(index 2, term 1)</code> 不能被覆盖。<br>而第 2 点对 log 的要求，使得 s3 不可能成为 leader。即使 s3 开始了新一轮选举，s2 由于 s3 的 log 落后于自己，不会投票给 s3。在 s1 挂了的情况下只有 s2 有可能成为新的 Leader，从而使得 <code>Entry(index 2, term 1)</code> 被保留。</p><p>如果 <code>Entry(index 2, term 1)</code> 如 (3) 所示，在这种情况下，s1 不可能 apply 过 <code>Entry(index 2, term1)</code>。s2、s3 都有可能成为新的 leader。</p><ol start="3"><li>如果一条 log entry 被复制到超过半数以上的机子上，是否可以认为这个 log entry 是可以被 commit 的？</li></ol><p>绝大部分情况是的。但存在一个例外，也就是 <a href="http://nil.csail.mit.edu/6.824/2020/papers/raft-extended.pdf" target="_blank" rel="noopener">Raft Exteded</a> 里 Figure 8 所提到的。如果这个 log entry 的 term 比 currentTerm 小，存在被覆盖的可能。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line">(1)</span><br><span class="line">index 1 2</span><br><span class="line">server log</span><br><span class="line">------------------------</span><br><span class="line">s1(l) 1 2</span><br><span class="line">s2 1 2</span><br><span class="line">s3 1</span><br><span class="line">s4 1</span><br><span class="line">s5 1</span><br><span class="line"></span><br><span class="line">(2)</span><br><span class="line">index 1 2</span><br><span class="line">server log</span><br><span class="line">------------------------</span><br><span class="line">s1 1 2</span><br><span class="line">s2 1 2</span><br><span class="line">s3 1</span><br><span class="line">s4 1</span><br><span class="line">s5(l) 1 3</span><br><span class="line"></span><br><span class="line">(3)</span><br><span class="line">index 1 2 3</span><br><span class="line">server log</span><br><span class="line">------------------------</span><br><span class="line">s1(l) 1 2 4</span><br><span class="line">s2 1 2</span><br><span class="line">s3 1 2</span><br><span class="line">s4 1</span><br><span class="line">s5 1 3</span><br><span class="line"></span><br><span class="line">(4)</span><br><span class="line">index 1 2</span><br><span class="line">server log</span><br><span class="line">------------------------</span><br><span class="line">s1 1 3</span><br><span class="line">s2 1 3</span><br><span class="line">s3 1 3</span><br><span class="line">s4 1 3</span><br><span class="line">s5(l) 1 3</span><br><span class="line"></span><br><span class="line">(3)</span><br><span class="line">index 1 2 3</span><br><span class="line">server log</span><br><span class="line">------------------------</span><br><span class="line">s1(l) 1 2 4</span><br><span class="line">s2 1 2 4</span><br><span class="line">s3 1 2 4</span><br><span class="line">s4 1</span><br><span class="line">s5 1 3</span><br></pre></td></tr></table></figure><p>(1) s1 成为 term 2 的 leader，添加了 <code>Entry(index 2, term 2)</code> 到 s1，s2。<br>(2) s1 挂了，s5 成为 term 3 的 leader，添加了 <code>Entry(index 2, term 3)</code> 到 s5。<br>(3) s5 挂了，s1 成为 term 4 的 leader，添加了 <code>Entry(index 3, term 4)</code> 到 s1。<br>下面有两种可能<br>(4) s1 挂了，s5 成为 term 5 的 leader，将 <code>Entry(index 2, term 2)</code> 复制到所有 server。<br>(5) s1 没挂，将 <code>Entry(index 2, term 2)</code>、<code>Entry(index 3, term 4)</code> 复制到所有 server。</p><p>我们分析下，如果 s1 在 (3) 的情况下把 <code>Entry(index 2, term 2)</code> commit 并且 apply 了，之后又出现 (4) 的情况，就会导致 server 之间状态的不一致，s1 执行了 <code>Entry(index 2, term2)</code>，而其他 server 都没有。<br>因此 server 不能依赖于过去 term 中的 log entry 进行 commit。</p><p>做 Lab 2 时一定要多看看论文的 Figure 2，等我踩了无数的坑之后，才发现这上面讲的真是字字珠玑，漏一个条件多 Debug 半天。<br>Debug 时一定要打详细的日志，不然排查问题只能靠猜想了。下面是我日志的示例。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">252403ms 0 read persist[term 0, votedFor -1, log length 0]</span><br><span class="line">252406ms 1 read persist[term 0, votedFor -1, log length 0]</span><br><span class="line">252408ms 2 read persist[term 0, votedFor -1, log length 0]</span><br><span class="line">253015ms 0 begin term 1 election, before state follower</span><br><span class="line">253018ms 0->1 Vote[id 1, term 1, lastLogIndex 0, lastLogTerm 0]</span><br><span class="line">253022ms 0->2 Vote[id 2, term 1, lastLogIndex 0, lastLogTerm 0]</span><br><span class="line">253027ms 0->2 Vote[id 2, term 1, lastLogIndex 0, lastLogTerm 0] Reply[true, term 1]</span><br><span class="line">253028ms 0 now I'm the leader with term 1</span><br><span class="line">253029ms 0->1 Vote[id 1, term 1, lastLogIndex 0, lastLogTerm 0] Reply[true, term 1]</span><br><span class="line">253030ms 0->1 AE[id 3, term 1, prevLogIndex 0, prevLogTerm 0, commit 0, cmd 0]</span><br><span class="line">253030ms 0->2 AE[id 4, term 1, prevLogIndex 0, prevLogTerm 0, commit 0, cmd 0]</span><br><span class="line">253036ms 0->2 AE[id 4, term 1, prevLogIndex 0, prevLogTerm 0, commit 0, cmd 0] Reply[true, term 1, XTerm 0, XIndex 0, XLen 0]</span><br><span class="line">253036ms 0->1 AE[id 3, term 1, prevLogIndex 0, prevLogTerm 0, commit 0, cmd 0] Reply[true, term 1, XTerm 0, XIndex 0, XLen 0]</span><br></pre></td></tr></table></figure></p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="http://thesecretlivesofdata.com/raft/" target="_blank" rel="noopener">Raft 动画演示</a></li><li><a href="http://nil.csail.mit.edu/6.824/2020/papers/raft-extended.pdf" target="_blank" rel="noopener">Raft Extended</a></li></ul>]]></content>
<summary type="html">
<p>先简单介绍下 Raft 是什么。Raft 是一种共识算法，解决的是分布式系统对某个提案，大部分节点达成一致意见的过程。<br>上面这样讲感觉很抽象，我们举一个具体的例子。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"> server</span><br><span class="line"> _________</span><br><span class="line">c1 -&gt; | (k, v1) |</span><br><span class="line">c2 -&gt; | (k, v2) |</span><br><span class="line"> |_________|</span><br></pre></td></tr></table></figure>
<p>假如我们有一个分布式的 KV 系统，有两个客户端 c1, c2，有两个服务端 s1, s2。客户端向服务端发请求修改 k 的值。如上所示，c1 发出将 k 修改为 v1 的请求，c2 发出将 k 修改为 v2 的请求。那么请问这个 KV 系统 k 的值最后为多少呢？</p>
<p>你肯定会说这依赖于 v1 和 v2 的顺序，v1 先 v2 后则最终值为 v2，反之则为 v1。但这要求 v1 和 v2 之间是有顺序关系的。一种做法是给 v1、v2 附上一个顺序，比如 vector clock。</p>
<p>还有一种更简单的做法，就是我们让其中一个 server 作为 leader 处理请求，而在单一 server 上，命令执行是自然有序的，之后我们再把这种次序转发给其他 server 就可以了。</p>
<p>但这又带来了其他的问题，怎么确定这个 leader 呢？简单点的可以直接固定一个，但这就会导致 leader 挂掉后群龙无首。我们也可以让服务器选一个 leader 出来，leader 挂掉后剩下的服务器再次选举就可以了。而这时，就轮到 Raft 出场了，Raft 会选举出一个 leader 出来，leader 执行的命令会以 log 的形式传给剩下的 server，使 server 的状态保持一致。</p>
</summary>
<category term="分布式系统" scheme="http://kaywu.xyz/tags/%E5%88%86%E5%B8%83%E5%BC%8F%E7%B3%BB%E7%BB%9F/"/>
<category term="Raft" scheme="http://kaywu.xyz/tags/Raft/"/>
</entry>
<entry>
<title>MIT 6.824 分布式系统笔记 — MapReduce</title>
<link href="http://kaywu.xyz/2020/03/12/distributed-system-mapreduce/"/>
<id>http://kaywu.xyz/2020/03/12/distributed-system-mapreduce/</id>
<published>2020-03-12T03:16:36.000Z</published>
<updated>2020-03-23T02:52:55.978Z</updated>
<content type="html"><![CDATA[<p>最近抽空学了下 MIT 的著名分布式系统课 <a href="[http://nil.csail.mit.edu/6.824/2020/schedule.html]">6.824</a>。这次 MIT 终于加上了课堂录像。有兴趣的同学可以去学下，熟悉的国外计算机的味道。读论文，上课，写 Lab，每个 Lab 还有详细的测试。让我不经回忆起刚开始学算法公开课的时候，在被虐中成长。</p><p>刚学到第三节课，做下笔记。我最近越发发现，写了笔记的不容易忘，正是好记性不如烂笔头。即使笔记写得有种给自己看的意思，还是尽量写写看。</p><p>前三节课主要讲了分布式的一些基本概念，比如容错性、性能和一致性的矛盾，Go 相关知识，还有两个 Case Study，MapReduce 和 GFS。MapReduce 和 GFS 都是 Google 大数据的三驾马车，这让我对这门课充满了敬意，刚开始就学这个。后来读了论文之后，发现确实没想象中的难。</p><p>MapReduce 是为了解决大规模数据的计算问题。简单来讲你要处理大量的数据，而这已经超出了单机的承载能力，所以你必须先将数据分块，然后使用大量的机器进行并行计算，最后再汇总结果。但你作为个开发者，不想每次都考虑任务的分发和调度的问题，只想处理和数据有关的逻辑，而把其他的事情都交给框架处理，而这就是 MapReduce。<br>MapReduce 将过程分成两部，第一步 Map，对数据进行处理得出中间结果。第二部 Reduce，对中间结果进行汇总。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">Abstract view of a MapReduce job</span><br><span class="line"> input is (already) split into M files</span><br><span class="line"> Input1 -> Map -> a,1 b,1</span><br><span class="line"> Input2 -> Map -> b,1</span><br><span class="line"> Input3 -> Map -> a,1 c,1</span><br><span class="line"> | | |</span><br><span class="line"> | | -> Reduce -> c,1</span><br><span class="line"> | -----> Reduce -> b,2</span><br><span class="line"> ---------> Reduce -> a,2</span><br></pre></td></tr></table></figure></p><a id="more"></a><p>其实 MapReduce 本质是分治的分布式应用，隐藏了使用者不需要关心的分发调度、容错等细节。接下来来看下它是怎么实现的。<br><img src="/img/mapreduce-execution.jpg" alt="MapReduce 流程总览"><br>该图来自于 <a href="[http://nil.csail.mit.edu/6.824/2020/papers/mapreduce.pdf]">MapReduce 论文</a>，显示了 MapReduce 是如何工作的。我们先不关注任何细节，最简单地讲下它的流程。<br>有一个 master 节点，它负责将 map 或者 reduce 的任务分发给 worker，worker 执行完任务后将结果提交给 master，之后再进行分发-执行-提交的循环，直至所有任务都完成。在这过程中，worker 不需要知道其他信息，只需要完成 master 分发的任务即可。而 master 是统筹一切的关键，它需要分配任务，保存提交结果，记录每个任务的状态，判断整个任务的完成情况。<br>接下来我们讨论下 master 的一些细节。master 将 map 或者 reduce 的任务分发给空闲的 worker 去执行，具体是 map 或者 reduce 由整个任务的完成情况来决定。若是所有 map 任务都已经完成，那么就会分发 reduce 任务。master 会通过探活来监控 worker 的运行情况，如果发现 worker 无法响应，则会把分配给该 worker 的任务重新进行分配。这增强了容错性，因为在大量机器的情况下单一机器出错几乎是个必然事件。但这同时会导致一个任务被执行多次。 MapReduce 通过提交结果的原子性来避免该影响。如果之前已经有 worker 提交了该任务，那么之后的提交将被忽略。<br>除此之外，还有很多细节的优化，比如 master 在分配任务时会考虑输入数据与 worker 之间的距离，来减少数据传输的时间以及对中心路由器的压力。当临近 map 或 reduce 阶段结束的时候，为了减少个别任务未完成的影响，master 会对未完成的任务进行再分配。更多细节还是看论文来得实在，我就不多讲了。</p><p>从上面的分析可以看出 MapReduce 其实并不复杂，Lab1 就是要求你用 go 来实现一个简易的 MapReduce。代码难度不高，需要注意的有几点，一是分发任务、提交任务时注意原子性，master 可以通过加锁来保证这点。二是分发任务时需要判断整个任务的阶段，是 map 还是 reduce。有一个 corner case，是所有 map 都已分发完成但有部分未完成的情况下，如果开始 reduce 会导致最后结果不完整，我在这边就踩了坑。三是需要定时去刷下任务的状态，如果某个任务超过一定时间没有完成的话，需要重新将该任务的状态变为可分发。master 可以通过 channel 定时刷新来解决这点。写完了还是很有成就感的。</p><p>上面讲了 MapReduce 的优点，但 MapReduce 还是有一定的局限性的。像是工作中对大数据的处理大多是多个阶段的，因此需要用户使用多次 MapReduce 自行串联。此外，MapReduce 也不支持基于实时数据流的数据处理。</p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="[http://nil.csail.mit.edu/6.824/2020/schedule.html]">6.824 分布式系统</a></li><li><a href="[http://nil.csail.mit.edu/6.824/2020/papers/mapreduce.pdf]">MapReduce 论文</a></li></ul>]]></content>
<summary type="html">
<p>最近抽空学了下 MIT 的著名分布式系统课 <a href="[http://nil.csail.mit.edu/6.824/2020/schedule.html]">6.824</a>。这次 MIT 终于加上了课堂录像。有兴趣的同学可以去学下，熟悉的国外计算机的味道。读论文，上课，写 Lab，每个 Lab 还有详细的测试。让我不经回忆起刚开始学算法公开课的时候，在被虐中成长。</p>
<p>刚学到第三节课，做下笔记。我最近越发发现，写了笔记的不容易忘，正是好记性不如烂笔头。即使笔记写得有种给自己看的意思，还是尽量写写看。</p>
<p>前三节课主要讲了分布式的一些基本概念，比如容错性、性能和一致性的矛盾，Go 相关知识，还有两个 Case Study，MapReduce 和 GFS。MapReduce 和 GFS 都是 Google 大数据的三驾马车，这让我对这门课充满了敬意，刚开始就学这个。后来读了论文之后，发现确实没想象中的难。</p>
<p>MapReduce 是为了解决大规模数据的计算问题。简单来讲你要处理大量的数据，而这已经超出了单机的承载能力，所以你必须先将数据分块，然后使用大量的机器进行并行计算，最后再汇总结果。但你作为个开发者，不想每次都考虑任务的分发和调度的问题，只想处理和数据有关的逻辑，而把其他的事情都交给框架处理，而这就是 MapReduce。<br>MapReduce 将过程分成两部，第一步 Map，对数据进行处理得出中间结果。第二部 Reduce，对中间结果进行汇总。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">Abstract view of a MapReduce job</span><br><span class="line"> input is (already) split into M files</span><br><span class="line"> Input1 -&gt; Map -&gt; a,1 b,1</span><br><span class="line"> Input2 -&gt; Map -&gt; b,1</span><br><span class="line"> Input3 -&gt; Map -&gt; a,1 c,1</span><br><span class="line"> | | |</span><br><span class="line"> | | -&gt; Reduce -&gt; c,1</span><br><span class="line"> | -----&gt; Reduce -&gt; b,2</span><br><span class="line"> ---------&gt; Reduce -&gt; a,2</span><br></pre></td></tr></table></figure></p>
</summary>
<category term="分布式系统" scheme="http://kaywu.xyz/tags/%E5%88%86%E5%B8%83%E5%BC%8F%E7%B3%BB%E7%BB%9F/"/>
<category term="MapReduce" scheme="http://kaywu.xyz/tags/MapReduce/"/>
</entry>
<entry>
<title>记一次 GC 优化</title>
<link href="http://kaywu.xyz/2019/11/29/gc-optimize/"/>
<id>http://kaywu.xyz/2019/11/29/gc-optimize/</id>
<published>2019-11-29T02:30:00.000Z</published>
<updated>2020-02-20T16:24:43.000Z</updated>
<content type="html"><![CDATA[<h2 id="现象"><a href="#现象" class="headerlink" title="现象"></a>现象</h2><p>近一段时间，项目 account-service 出现高峰期经常性报警的问题，且发布越久情况愈发严重。</p><p>从 Service Metrics 图表 来看，报警的时段 account-service 确实存在大量接口超时的情况，且单一时间集中于某一个 Pod 上。<br><img src="/img/response-time.png" alt=""></p><p>观察 Grafana 上的图标，发现在该时间段 Pod 出现 Major GC 时间增加的情况，高达几秒，猜测是由于 GC 所引发的接口超时现象。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">// prometheus 查询</span><br><span class="line">increase(jvm_gc_pause_seconds_sum{kubernetes_pod_name=~"$app_name-$env-.+", action="end of major GC"}[2m])</span><br></pre></td></tr></table></figure><p>那么接下来我们要解决两个问题，一是 GC 为什么会花费这么长的时间，二是情况为什么会变得严重。<br><a id="more"></a></p><h3 id="GC"><a href="#GC" class="headerlink" title="GC"></a>GC</h3><p>GC，也就是垃圾收集器，是有很多选择的，比较常见的有 Serial GC、CMS、Parallel GC、G1 等。Java 8 中 server 模式下默认的 GC 为 Parallel GC，由于 account-service 未对 GC 做任何特殊的配置，所以这也是 account-service 所使用的 GC。</p><p>Parallel GC 又被称作吞吐量优先的 GC，其特点是新生代和老年代 GC 都是并行进行的。而 CMS、G1 是以停顿时间优先考虑的，会为了减少停顿时间而牺牲一定的性能。</p><p>这话怎么理解呢？打个简单的比方，使用 Parallel GC 的话，一分钟可以处理请求 100 个，但最长时间的 GC 可能高达 1s。而使用 CMS、G1 的话处理的请求只有 90 个，但最长时间的 GC 控制在 200ms 以内。PS：以上数据仅做说明，不具实际意义。</p><p>而 account-service 作为 API server，响应时间的优先更高，为了避免 GC 带来的数秒停顿的问题，应该选择 CMS 或 G1。其实 Java 8 API server 都需要更换默认的 GC，因为高达秒级别的停顿基本不可承受。而 Java 9 之后就没这种坑了，默认的 GC 变成了 G1。</p><p>那么 CMS 和 G1 我们应该选哪个呢？我做了本地测试，发现在 account-service 上 CMS、G1 性能上没有大的差异，由于 CMS 在 Java 9 之后被标记成 deprecated，所以选择了 G1。</p><p>通过如下的启动命令切换 GC 到 G1。PS：不要显式设置新生代的大小，这会导致 G1 目标时间这个参数失效。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">java -XX:+UseG1GC -jar $APP_JAR</span><br></pre></td></tr></table></figure><p>上线后观察了一段时间，目前 GC 均在 200ms 以内，高峰期因为 GC 而报警的情况没有再出现了。</p><h3 id="深挖"><a href="#深挖" class="headerlink" title="深挖"></a>深挖</h3><p>目前为止我们解决了第一个问题，接下来深挖第二个问题。</p><p>如果说报警是由于 GC 带来的长时间停顿影响的，那么报警变得频繁很有可能是因为 GC 变得更加频繁了。这时在 Grafana 观察到，随着时间， Live Data 越来越高。Live Data 表示老年代在 full GC 后所占用的空间，一直上涨表明 Server 可能存在内存泄漏问题，有一部内存无法释放且随着时间占用的空间越来越大。</p><p><img src="/img/live-data.png" alt=""></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">// prometheus 查询</span><br><span class="line">jvm_gc_live_data_size_bytes{kubernetes_pod_name=~"$app_name-$env-.+"}</span><br></pre></td></tr></table></figure><p>从线上 dump heap 并通过 MAT 分析。PS：线上 dump heap 会引发线上服务一段时间内停止工作，可以通过灰度发布进行流量分配的方式降低对线上的影响。</p><p>发现主要有两个问题，一是 org.hibernate.internal.SessionFactoryImp 占用了 120 MB 的内存。二是其中一个 ThreadPool 中的 Thread 的 ThreadLocalMap 在不断增长。</p><p>查询完相关资料后，发现第一个问题是因为 Hibernate 的 QueryPlanCache。Hibernate 内部维护了 HQL 到 Query Plan 的缓存，默认大小 2048 个。但是 in 查询参数数量的不同会导致多个不同的 Query Plan。比如 select <em> from User where id in (:id0) 和 select </em> from User where id in (:id0, id1) 是两个 Query Plan。而 account-service 本身 in 查询的数量不少，使得这块占用了不小的堆内存。这问题可以通过配置 spring.jpa.properties.hibernate.query.in_clause_parameter_padding=true(hibernate 5.2.18 及以上生效)，使 Hibernate 的 in 查询根据 2 的倍数，也就是 1-2-4-8-16 生成 Query Plan，减少相关内存占用。关于该问题的详细分析，可参考 <a href="https://stackoverflow.com/questions/31557076/spring-hibernate-query-plan-cache-memory-usage" target="_blank" rel="noopener">Query Plan Cache Memory usage</a> 和 <a href="https://vladmihalcea.com/improve-statement-caching-efficiency-in-clause-parameter-padding" target="_blank" rel="noopener">How to improve statement caching efficiency with IN clause parameter padding</a>。</p><p>第二个通过 MAT 查到 ThreadLocalMap 中的 io.opentracing.util.ThreadLocalScope 没有释放，每一次 AysncTask 的执行都会使得 ThreadLocalScope 增加一个 span。由于 account-service 没有使用到 jaeger 的相应功能，是老版本基础库的依赖引入了相关的包。新版本的基础库将 opentracing 相关的功能独立成另一个包，通过升级解决了这个内存泄漏问题。</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><ol><li>Java Server 可以通过切换 GC 为 G1 或 CMS 来降低停顿时间。</li><li>使用 Hibernate 并且 in 查询很多的 Server 可以通过配置 spring.jpa.properties.hibernate.query.in_clause_parameter_padding 来降低 QueryPlanCache 所占用的内存。</li><li>通过 Live Data Size 观察是否有内存泄漏情况，通过 MAT 分析内存泄漏的原因。</li></ol>]]></content>
<summary type="html">
<h2 id="现象"><a href="#现象" class="headerlink" title="现象"></a>现象</h2><p>近一段时间，项目 account-service 出现高峰期经常性报警的问题，且发布越久情况愈发严重。</p>
<p>从 Service Metrics 图表 来看，报警的时段 account-service 确实存在大量接口超时的情况，且单一时间集中于某一个 Pod 上。<br><img src="/img/response-time.png" alt=""></p>
<p>观察 Grafana 上的图标，发现在该时间段 Pod 出现 Major GC 时间增加的情况，高达几秒，猜测是由于 GC 所引发的接口超时现象。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">// prometheus 查询</span><br><span class="line">increase(jvm_gc_pause_seconds_sum&#123;kubernetes_pod_name=~&quot;$app_name-$env-.+&quot;, action=&quot;end of major GC&quot;&#125;[2m])</span><br></pre></td></tr></table></figure>
<p>那么接下来我们要解决两个问题，一是 GC 为什么会花费这么长的时间，二是情况为什么会变得严重。<br>
</summary>
<category term="Java" scheme="http://kaywu.xyz/tags/Java/"/>
<category term="GC" scheme="http://kaywu.xyz/tags/GC/"/>
</entry>
<entry>
<title>Kubernetes 通过 Ingress 实现灰度发布以及 CI 流程</title>
<link href="http://kaywu.xyz/2019/07/25/kubernetes-ingress/"/>
<id>http://kaywu.xyz/2019/07/25/kubernetes-ingress/</id>
<published>2019-07-25T07:05:11.000Z</published>
<updated>2020-02-20T16:24:43.000Z</updated>
<content type="html"><![CDATA[<p>ingress-nginx 在 0.22 添加了灰度发布的功能，可以通过简单的配置实现。这篇文章主要讲解如何配置以及如何和 CI 流程结合。<br>PS：简单说明下，我司的发布流程是通过 Gitlab 和 Kubernetes 实现的。在 Gitlab - Operations - Kubernetes 添加 Kubernetes 相关配置，在 .gitlab-ci.yml 配置 CI 流程，在 Gitlab CI Variables 里配置敏感信息。</p><h2 id="Ingress-配置"><a href="#Ingress-配置" class="headerlink" title="Ingress 配置"></a>Ingress 配置</h2><p>Ingress 需要增加的配置比较简单，只需要添加几个 annotation 就可以。<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># ingress annotation</span></span><br><span class="line"><span class="string">nginx.ingress.kubernetes.io/canary:</span> <span class="string">"true"</span></span><br><span class="line"><span class="string">nginx.ingress.kubernetes.io/canary-weight:</span> <span class="string">"20"</span></span><br></pre></td></tr></table></figure></p><p>上面的配置表示开启 ingress canary 功能，设置的流量为 20%。除了基本的配置之外，还可以根据 Header、Cookie 进行流量配置，可以参考<a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary" target="_blank" rel="noopener">官方文档</a>。</p><p>接下来，只要发布的服务和原来的服务 Ingress host 保持一致就可以。现在发往 <a href="http://example.beta.com" target="_blank" rel="noopener">http://example.beta.com</a> 的请求，有 80% 的流量发往原服务，有 20% 的流量发往新的服务。</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># .gitlab-ci.yml</span></span><br><span class="line"><span class="attr">deploy_beta:</span></span><br><span class="line"> <span class="string"><<:</span> <span class="meta">*DEPLOY</span></span><br><span class="line"><span class="attr"> environment:</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">beta</span></span><br><span class="line"><span class="attr"> url:</span> <span class="attr">http://example.beta.com</span></span><br><span class="line"> </span><br><span class="line"><span class="attr">deploy_beta_canary:</span></span><br><span class="line"> <span class="string"><<:</span> <span class="meta">*DEPLOY</span></span><br><span class="line"><span class="attr"> environment:</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">beta-canary</span></span><br><span class="line"><span class="attr"> url:</span> <span class="attr">http://example.beta.com</span></span><br></pre></td></tr></table></figure><a id="more"></a><h2 id="CI-流程的改造"><a href="#CI-流程的改造" class="headerlink" title="CI 流程的改造"></a>CI 流程的改造</h2><p>上面只是说明了如何进行配置，但如何与原来的 CI 流程结合是一个需要解决的问题，比如环境变量的配置、灰度发布流程。接下来一一讨论。</p><h3 id="环境变量的配置"><a href="#环境变量的配置" class="headerlink" title="环境变量的配置"></a>环境变量的配置</h3><p>由于我司 deployment 的 release 是由项目名以及 Gitlab environment 组成的，我这边通过区分灰度发布的 environment 使得 deployment 的 release 不同。但这会导致原先在 Gitlab CI 上配置的环境变量失效，需要对原来的配置进行修改。在原来的环境后添加通配符 *，可以使得灰度发布的服务也能使用之前的环境变量配置。<br><img src="/img/gitlab-variables.png" alt=""><br>当然也可以通过更改 deployment 的 release 规则达到同样的效果，不需要修改 Gitlab 环境变量的配置。</p><h3 id="发布流程"><a href="#发布流程" class="headerlink" title="发布流程"></a>发布流程</h3><p>考虑到灰度发布后，需要进行正常的发布，我个人倾向于使用同一分支进行灰度发布和正常发布的操作，而不是使用一个例外的分支进行灰度发布，这样可以避免重复构建，也能减少对原来 CI 流程的影响。</p><p>这是 master 分支在添加了灰度发布功能后的 pipeline，添加了 deploy_production_canary，stop_production_canary 等 manual action，通过点击就可执行灰度发布、撤销灰度发布的功能。<br><img src="/img/gitlab-ci-deploy.png" alt=""></p><p>如果都在同一分支上进行灰度发布和正常发布，那么是如何区分这两个操作的？</p><p>可以通过在 Gitlab CI 的灰度发布上添加 CANARY_RELEASE 表明是灰度发布，然后在 CI 发布脚本中根据该变量添加相关的 ingress 配置。以下是 .gitlab-ci.yml 代码示例。</p><p>因为灰度发布的流量是由 CANARY_WEIGHT 控制的，当要修改灰度流量比例时，我们只需要在 CI Variables 更改 CANARY_WEIGHT 的值，然后再次点击 deploy。</p><p>可以通过执行 kubectl describe ingress ${ingress name} 来确认 CANARY_WEIGHT 是否生效。</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># .gitlab-ci.yml</span></span><br><span class="line"> </span><br><span class="line"><span class="string">.deploy:</span> <span class="meta">&DEPLOY</span></span><br><span class="line"> <span class="string"><<:</span> <span class="meta">*DEPLOY_BASE</span></span><br><span class="line"><span class="attr"> script:</span></span><br><span class="line"> <span class="comment"># 在 CANARY_RELEASE 为 true 时，向 ingress 添加 nginx.ingress.kubernetes.io/canary=true 以及 nginx.ingress.kubernetes.io/canary-weight=$CANARY_WEIGHT 的配置。</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">if</span> <span class="string">[</span> <span class="string">$CANARY_RELEASE</span> <span class="string">==</span> <span class="string">'true'</span> <span class="string">];</span> <span class="string">then</span> <span class="string">export</span> <span class="string">HELM_ARGS="$HELM_ARGS,ingress.annotations.nginx\.ingress\.kubernetes\.io/canary=true,ingress.annotations.nginx\.ingress\.kubernetes\.io/canary-weight=$CANARY_WEIGHT";</span> <span class="string">fi</span></span><br><span class="line"> </span><br><span class="line"><span class="string">.deploy_canary:</span> <span class="meta">&DEPLOY_CANARY</span></span><br><span class="line"> <span class="string"><<:</span> <span class="meta">*DEPLOY</span></span><br><span class="line"><span class="attr"> variables:</span></span><br><span class="line"> <span class="string"><<:</span> <span class="meta">*DEPLOY_BASE_VARIABLES</span></span><br><span class="line"> <span class="comment"># CANARY_RELEASE 表示为灰度发布。CANARY_WEIGHT 为配置灰度发布的流量比例。</span></span><br><span class="line"><span class="attr"> CANARY_RELEASE:</span> <span class="string">'true'</span></span><br><span class="line"><span class="attr"> CANARY_WEIGHT:</span> <span class="string">'10'</span></span><br><span class="line"><span class="attr"> when:</span> <span class="string">manual</span></span><br><span class="line"> </span><br><span class="line"><span class="attr">deploy_production:</span></span><br><span class="line"> <span class="string"><<:</span> <span class="meta">*DEPLOY</span></span><br><span class="line"><span class="attr"> environment:</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">production</span></span><br><span class="line"><span class="attr"> url:</span> <span class="attr">http://example.internal-k8s.com</span></span><br><span class="line"><span class="attr"> when:</span> <span class="string">manual</span></span><br><span class="line"><span class="attr"> only:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">master</span></span><br><span class="line"><span class="attr"> tags:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">production</span></span><br><span class="line"> </span><br><span class="line"><span class="attr">deploy_production_canary:</span></span><br><span class="line"> <span class="string"><<:</span> <span class="meta">*DEPLOY_CANARY</span></span><br><span class="line"><span class="attr"> environment:</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">production-canary</span></span><br><span class="line"><span class="attr"> url:</span> <span class="attr">http://example.internal-k8s.com</span></span><br><span class="line"><span class="attr"> only:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">master</span></span><br><span class="line"><span class="attr"> tags:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">production</span></span><br><span class="line"> </span><br><span class="line"><span class="attr">stop_production_canary:</span></span><br><span class="line"> <span class="string"><<:</span> <span class="meta">*STOP_DEPLOY</span></span><br><span class="line"><span class="attr"> environment:</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">production-canary</span></span><br><span class="line"><span class="attr"> action:</span> <span class="string">stop</span></span><br><span class="line"><span class="attr"> only:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">master</span></span><br><span class="line"><span class="attr"> tags:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">production</span></span><br></pre></td></tr></table></figure><p>在灰度发布一段时间后，需要全量发布时，步骤如下：</p><ol><li>执行 deploy_production</li><li>待 production 的 pod 发布完毕后，执行 stop_production_canary，将灰度发布的 pod 删除。此时线上的情况就跟直接执行 deploy_production 的情况相同。</li></ol><p>当你第一次 deploy_production_canary 时，可能会发现没有部署到生产环境，而是部署到开发环境的 Kubernetes。那你需要更改 Gitlab - Operations - Kubernetes 的配置，将生产环境 Kubernetes cluster 的 Environment scope 从 production 改为 production*。</p><h3 id="监控"><a href="#监控" class="headerlink" title="监控"></a>监控</h3><p>当线上有两个版本的服务存在时，我们需要能区分这个请求到底是哪个版本的服务。可以通过在灰度发布的版本上添加特殊的 Header 来区分的。Spring 的示例代码如下。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// values.yaml</span></span><br><span class="line">CANARY_RELEASE: <span class="keyword">false</span></span><br><span class="line"> </span><br><span class="line"><span class="comment">// application.properties</span></span><br><span class="line">spring.account-service.canary-release=${CANARY_RELEASE:<span class="keyword">false</span>}</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line"><span class="meta">@Configuration</span></span><br><span class="line"><span class="meta">@ConditionalOnProperty</span>(name = <span class="string">"spring.account-service.canary-release"</span>)</span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">CanaryConfig</span> </span>{</span><br><span class="line"> </span><br><span class="line"> <span class="meta">@Bean</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> CanaryFilter <span class="title">canaryFilter</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> CanaryFilter();</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">CanaryFilter</span> <span class="keyword">extends</span> <span class="title">OncePerRequestFilter</span> </span>{</span><br><span class="line"> </span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">doFilterInternal</span><span class="params">(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)</span></span></span><br><span class="line"><span class="function"> <span class="keyword">throws</span> IOException, ServletException </span>{</span><br><span class="line"> response.addHeader(<span class="string">"From-Canary"</span>, <span class="string">"true"</span>);</span><br><span class="line"> filterChain.doFilter(request, response);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>可以在 Grafana 添加图表使各个 Pod 接收的请求量可视化，Query 如下。</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">sum(increase(http_server_requests_seconds_count{kubernetes_pod_name=~"$app_name-$env.+"}[2m]))</span> <span class="string">by</span> <span class="string">(kubernetes_pod_name)</span></span><br></pre></td></tr></table></figure><h3 id="分支策略"><a href="#分支策略" class="headerlink" title="分支策略"></a>分支策略</h3><p>灰度发布不走单独的分支，按原来的分支流程走。<br><img src="/img/canary-branch.png" alt=""></p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary" target="_blank" rel="noopener">https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary</a></li><li><a href="https://www.elvinefendi.com/2018/11/25/canary-deployment-with-ingress-nginx.html" target="_blank" rel="noopener">https://www.elvinefendi.com/2018/11/25/canary-deployment-with-ingress-nginx.html</a></li></ul>]]></content>
<summary type="html">
<p>ingress-nginx 在 0.22 添加了灰度发布的功能，可以通过简单的配置实现。这篇文章主要讲解如何配置以及如何和 CI 流程结合。<br>PS：简单说明下，我司的发布流程是通过 Gitlab 和 Kubernetes 实现的。在 Gitlab - Operations - Kubernetes 添加 Kubernetes 相关配置，在 .gitlab-ci.yml 配置 CI 流程，在 Gitlab CI Variables 里配置敏感信息。</p>
<h2 id="Ingress-配置"><a href="#Ingress-配置" class="headerlink" title="Ingress 配置"></a>Ingress 配置</h2><p>Ingress 需要增加的配置比较简单，只需要添加几个 annotation 就可以。<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># ingress annotation</span></span><br><span class="line"><span class="string">nginx.ingress.kubernetes.io/canary:</span> <span class="string">"true"</span></span><br><span class="line"><span class="string">nginx.ingress.kubernetes.io/canary-weight:</span> <span class="string">"20"</span></span><br></pre></td></tr></table></figure></p>
<p>上面的配置表示开启 ingress canary 功能，设置的流量为 20%。除了基本的配置之外，还可以根据 Header、Cookie 进行流量配置，可以参考<a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary" target="_blank" rel="noopener">官方文档</a>。</p>
<p>接下来，只要发布的服务和原来的服务 Ingress host 保持一致就可以。现在发往 <a href="http://example.beta.com" target="_blank" rel="noopener">http://example.beta.com</a> 的请求，有 80% 的流量发往原服务，有 20% 的流量发往新的服务。</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># .gitlab-ci.yml</span></span><br><span class="line"><span class="attr">deploy_beta:</span></span><br><span class="line"> <span class="string">&lt;&lt;:</span> <span class="meta">*DEPLOY</span></span><br><span class="line"><span class="attr"> environment:</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">beta</span></span><br><span class="line"><span class="attr"> url:</span> <span class="attr">http://example.beta.com</span></span><br><span class="line"> </span><br><span class="line"><span class="attr">deploy_beta_canary:</span></span><br><span class="line"> <span class="string">&lt;&lt;:</span> <span class="meta">*DEPLOY</span></span><br><span class="line"><span class="attr"> environment:</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">beta-canary</span></span><br><span class="line"><span class="attr"> url:</span> <span class="attr">http://example.beta.com</span></span><br></pre></td></tr></table></figure>
</summary>
<category term="Kubernetes" scheme="http://kaywu.xyz/tags/Kubernetes/"/>
</entry>
<entry>
<title>Kubernetes 使用 Zuul 网关时上游 Tomcat 400 报错问题排查</title>
<link href="http://kaywu.xyz/2019/05/05/k8s-zuul-400/"/>
<id>http://kaywu.xyz/2019/05/05/k8s-zuul-400/</id>
<published>2019-05-05T05:44:18.000Z</published>
<updated>2020-02-20T16:24:43.000Z</updated>
<content type="html"><![CDATA[<p>在 Kubernetes 里，使用 Zuul 转发请求到上游的 Tomcat 服务时，Tomcat 报了一个很诡异的错误：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">The host [zuul.host,zuul.host] is not valid</span><br></pre></td></tr></table></figure></p><p>其中 zuul.host 是 Zuul 服务的域名。</p><p>看到这个报错时，有两个点很奇怪。一是 <code>[zuul.host, zuul.host]</code> 是怎么来的，为什么它重复了一遍。二是为什么 host 的值会是这个，应该是 Tomcat 服务的域名。</p><a id="more"></a><p>网上搜了一圈，没找到相关资料，只能一步步来挖了。</p><p>由于是 Tomcat 直接报错，未进入 Spring Boot 内部，配置的 Sentry 也没捕捉到该错误，只能通过添加 Tomcat 日志配置打印相应的 Header。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"># application.properties</span><br><span class="line">server.tomcat.basedir=logs/tomcat-logs</span><br><span class="line">server.tomcat.accesslog.enabled=true</span><br><span class="line">server.tomcat.accesslog.pattern=%t "%r" %s (%D ms) -Host (%{Host}i) -X-Forwarded-host (%{X-Forwarded-Host}i)</span><br></pre></td></tr></table></figure><p>日志打印出了两个很有意思的值，Host 和 X-Forwarded-Host 的值是同样的，都为 <code>[zuul.host, zuul.host]</code>。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Host: [zuul.host, zuul.host]</span><br><span class="line">X-Forwarded-Host: [zuul.host, zuul.host]</span><br></pre></td></tr></table></figure></p><p>这时可以确定，Tomcat 确实收到了一个诡异的请求，由于 Host Header 值无效导致 Tomcat 400 报错。但问题是，这请求是怎么来的呢？只能一步步往前追查，在 Zuul 服务中添加相应的日志配置。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// ZuulFilter.java</span></span><br><span class="line">log.info(<span class="string">"LOG_HEADER: {} Host-({}) X-Forwarded-Host-({})"</span>, request.getRequestURI(), request.getHeader(<span class="string">"Host"</span>), request.getHeader(<span class="string">"X-Forwarded-Host"</span>));</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">LOG_HEADER: /api/v1/login/app Host-(zuul.host) X-Forwarded-Host-(zuul.host)</span><br></pre></td></tr></table></figure><p>从日志中可以看出，Zuul 服务接收到的请求 Host 和 X-Forwarded-Host 都为 <code>zuul.host</code>，再加上 Zuul 默认情况下也会为转发的请求添加 X-Forwarded-Host，所以 Tomcat 收到的时候 X-Forwarded-Host 为两次的 <code>zuul.host</code>。现在已经知道 X-Forwarded-Host 的值是如何来的，但 Host 值的原因还没找到。</p><p>没其他线索的情况下，我本地往 Tomcat 服务发送了几次请求，X-Forwarded-Host 的值从零个到多个，发现了些许端倪。在 Tomcat 打印的日志中，Host 和 X-Forwarded-Host 的值都是一样的。而当请求的 X-Forwarded-Host 值零个或一个时，请求返回 200。X-Forwarded-Host 值大于一个时，请求返回 400。这时我有了一个猜想，难道 X-Forwarded-Host 的值会被复制到 Host 吗？</p><p>按照这个想法，找到了相关的 <a href="https://github.com/kubernetes/ingress-nginx/issues/2463" target="_blank" rel="noopener">ISSUE</a>，原来 Ingress 在开启 <code>use-forwarded-headers</code> 后，会将 X-Forwarded-Host 的值复制到 Host，但由于没有考虑 X-Forwarded-Host 为多个值的情况，导致 Host 不合法。这个问题已经在 <code>ingress-nginx</code> 0.24 版本被修复。</p><p>请求的整个流程，如下图所示。<br><img src="/img/ingress.png" alt=""></p><p>找到了问题根源，那解决起来就简单了。有以下三种方法：</p><ol><li>升级 <code>ingress-nginx</code> 至 0.24 及以上版本，从根本上解决这个问题。</li><li>配置 <code>zuul.addProxyHeaders = false</code>，使得 Zuul 不添加额外的 <code>X-Forwarded-Host</code>。</li><li>Zuul 直接使用 Tomcat 服务的 Service Name 进行访问，不经过 Ingress。</li></ol><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="https://github.com/kubernetes/ingress-nginx/issues/2463" target="_blank" rel="noopener">X-Forwarded-Host header is copied into the Host header</a></li><li><a href="https://github.com/kubernetes/ingress-nginx/issues/3790" target="_blank" rel="noopener">Host header copied from X-Forwarded-Host (when forwarded headers are enabled)</a></li><li><a href="https://github.com/kubernetes/ingress-nginx/pull/3950" target="_blank" rel="noopener">Fix forwarded host parsing</a></li></ul>]]></content>
<summary type="html">
<p>在 Kubernetes 里，使用 Zuul 转发请求到上游的 Tomcat 服务时，Tomcat 报了一个很诡异的错误：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">The host [zuul.host,zuul.host] is not valid</span><br></pre></td></tr></table></figure></p>
<p>其中 zuul.host 是 Zuul 服务的域名。</p>
<p>看到这个报错时，有两个点很奇怪。一是 <code>[zuul.host, zuul.host]</code> 是怎么来的，为什么它重复了一遍。二是为什么 host 的值会是这个，应该是 Tomcat 服务的域名。</p>
</summary>
<category term="Kubernetes" scheme="http://kaywu.xyz/tags/Kubernetes/"/>
<category term="Zuul" scheme="http://kaywu.xyz/tags/Zuul/"/>
<category term="Tomcat" scheme="http://kaywu.xyz/tags/Tomcat/"/>
</entry>
<entry>
<title>图解密码技术</title>
<link href="http://kaywu.xyz/2018/04/30/cryptography/"/>
<id>http://kaywu.xyz/2018/04/30/cryptography/</id>
<published>2018-04-30T05:48:55.000Z</published>
<updated>2018-04-30T11:28:57.000Z</updated>
<content type="html"><![CDATA[<h2 id="对称密码"><a href="#对称密码" class="headerlink" title="对称密码"></a>对称密码</h2><p>对称密码是一种用相同的密钥进行加密和解密的技术，用于确保消息的机密性。尽管对称密码能够确保消息的机密性，但需要解决将解密密钥配送给接受者的密钥配送问题。<br>一次性密码本的原理是将明文与一串随机的比特序列进行 XOR 运算。即使用暴力破解法遍历整个密钥空间，也绝对无法被破译。<br>随着计算机的进步，现在 DES 已经能够被暴力破解，强度大不如前了。<br>AES 是取代其 DES 而成为新标准的一种对称密码算法。在 2000 年从候选算法中选出了一种名为 Rijndael 的对称密码算法，并将其确定为了 AES。</p><a id="more"></a><h2 id="分组密码的模式"><a href="#分组密码的模式" class="headerlink" title="分组密码的模式"></a>分组密码的模式</h2><p>密码算法可以分为分组密码和流密码两种。<br>分组密码是每次只能处理特定长度的一块数据的一类密码算法，这里的“一块”就称为分组。此外，一个分组的比特数就称为分组长度。例如，DES 和三重 DES 的分组长度都是 64 比特。AES 的分组长度可以从 128 比特、192 比特、256 比特中进行选择。<br>流密码是对数据流进行连续处理的一类密码算法。一般以 1 比特、8 比特或 32 比特等为单位进行加密和解密。<br>分组密码算法只能加密固定长度的分组，当明文长度超过分组密码的分组长度时，就需要对分组密码算法进行迭代。而迭代的方式就成为分组密码的模式。</p><h3 id="ECB-模式"><a href="#ECB-模式" class="headerlink" title="ECB 模式"></a>ECB 模式</h3><p>Electronic CodeBook，电子密码本模式，不应使用。<br>将明文分组加密之后的结果将直接成为密文分组。</p><h3 id="CBC-模式"><a href="#CBC-模式" class="headerlink" title="CBC 模式"></a>CBC 模式</h3><p>Cipher Block Chaining，密文分组链接模式，推荐使用。<br>将明文分组与前一个密文分组进行 XOR 运算，然后再进行加密。</p><h3 id="CFB-模式"><a href="#CFB-模式" class="headerlink" title="CFB 模式"></a>CFB 模式</h3><p>Ciper-Feedback，密文反馈模式，现在已不使用，推荐用 CTR 模式代替。<br>前一个密文分组被送回到密码算法的输入端。</p><h3 id="OFB-模式"><a href="#OFB-模式" class="headerlink" title="OFB 模式"></a>OFB 模式</h3><p>Output-Feedback 输出反馈模式，推荐用 CTR 模式代替。<br>密码算法的输出会反馈到密码算法的输入中，属于流密码。</p><h3 id="CTR-模式"><a href="#CTR-模式" class="headerlink" title="CTR 模式"></a>CTR 模式</h3><p>CounteT，计数器模式，推荐使用。<br>通过逐次累加的计数器进行加密来生成密钥流的流密码。</p><h2 id="公钥密码"><a href="#公钥密码" class="headerlink" title="公钥密码"></a>公钥密码</h2><p>密钥配送问题解决方法</p><ul><li>事先共享密钥</li><li>密钥分配中心</li><li>Diffie-Hellman 密钥交换</li><li>公钥密码</li></ul><p>公钥密码是一种用不同的密钥进行加密和解密的技术。公钥密码解决了密钥配送问题，但没有解决公钥认证问题，且它的处理速度只有对称密钥的几百分之一。<br>使用最广泛的一种公钥密码算法是 RSA。</p><h3 id="RSA"><a href="#RSA" class="headerlink" title="RSA"></a>RSA</h3><p>$密文 = 明文^E mod N$<br>$明文 = 密文^D mod N$</p><p>私钥：(D, N)<br>公钥：(E, N)</p><h3 id="中间人攻击"><a href="#中间人攻击" class="headerlink" title="中间人攻击"></a>中间人攻击</h3><p>中间人攻击就是主动攻击者混入发送者和接收者的中间，对发送者伪装成接收者，对接收者伪装成发送者的攻击方式。</p><h2 id="混合密码系统"><a href="#混合密码系统" class="headerlink" title="混合密码系统"></a>混合密码系统</h2><ul><li>用公钥密码加密对称密码中所使用的密钥</li><li>用对称密码来加密明文</li></ul><h2 id="单向散列函数"><a href="#单向散列函数" class="headerlink" title="单向散列函数"></a>单向散列函数</h2><p>单向散列函数有一个输入和一个输出，其中输入称为消息，输出称为散列值。单向散列函数可以根据信息的内容计算出散列值，而散列值就可以被用来检查消息的完整性。<br>难以发现碰撞的性质称为抗碰撞性。密码技术中所使用的单向散列函数，都需要具备抗碰撞性。<br>当给定某条消息的散列值时，单向散列函数必须确保要找到和该条消息具有相同散列值的另外一条消息是非常困难的，这一性质称为弱抗碰撞性。所谓强抗碰撞性，是指要找到散列值相同的两条不同的消息是非常困难的。<br>单向散列函数必须具备单向性。单向性指的是无法通过散列值反算出消息的性质。<br>使用单向散列函数可以辨别出篡改，但无法辨别出伪装。</p><h2 id="消息认证码"><a href="#消息认证码" class="headerlink" title="消息认证码"></a>消息认证码</h2><p>消息认证码（message athentiation code）是一种确认完整性并进行认证的技术，取三个单词的首字母，简称为 MAC。<br>消息认证码的输入包括任意长度的消息和一个发送者与接受者之间的共享的密钥，它可以输出固定长度的数据，这个数据称为 MAC 值。<br>由于发送者和接收者共享相同的密钥，因为会产生无法对第三方证明以及无法防止否认等问题。</p><h2 id="数字签名"><a href="#数字签名" class="headerlink" title="数字签名"></a>数字签名</h2><p>数字签名使用公钥和私钥组成的密钥对，用私钥加密相当于生成签名，用公钥解密则相当于验证签名。<br>通过数字签名我们可以识别篡改和伪装，还可以防止否认。</p><h2 id="证书"><a href="#证书" class="headerlink" title="证书"></a>证书</h2><p>公钥证书包括姓名、组织、邮箱地址等个人信息以及属于此人的公钥，并由认证机构施加数字签名。</p><h2 id="随机数"><a href="#随机数" class="headerlink" title="随机数"></a>随机数</h2><p>将随机数的性质分为以下三类：</p><ul><li>随机性：不存在统计学偏差，是完全杂乱的数列。弱伪随机数</li><li>不可预测性：不能从过去的数列推测出下一个出现的数。强伪随机数</li><li>不可重现性：除非将数列本身保存下来，否则不能重现相同的数列。真随机数<br>在密码技术中使用的伪随机数生成器，是以具备不可重现性的真随机数作为伪随机数的种子，来生成具备不可预测性的强伪随机数。</li></ul><h2 id="SSL-TLS"><a href="#SSL-TLS" class="headerlink" title="SSL/TLS"></a>SSL/TLS</h2><h3 id="握手阶段"><a href="#握手阶段" class="headerlink" title="握手阶段"></a>握手阶段</h3><p><img src="/img/tls_handshake.png" alt=""><br><a href="https://blog.cloudflare.com/announcing-keyless-ssl-all-the-benefits-of-cloudflare-without-having-to-turn-over-your-private-ssl-keys/" target="_blank" rel="noopener">来源</a></p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li>图解密码技术</li><li><a href="https://blog.cloudflare.com/announcing-keyless-ssl-all-the-benefits-of-cloudflare-without-having-to-turn-over-your-private-ssl-keys/" target="_blank" rel="noopener">Announcing Keyless SSL™</a></li></ul>]]></content>
<summary type="html">
<h2 id="对称密码"><a href="#对称密码" class="headerlink" title="对称密码"></a>对称密码</h2><p>对称密码是一种用相同的密钥进行加密和解密的技术，用于确保消息的机密性。尽管对称密码能够确保消息的机密性，但需要解决将解密密钥配送给接受者的密钥配送问题。<br>一次性密码本的原理是将明文与一串随机的比特序列进行 XOR 运算。即使用暴力破解法遍历整个密钥空间，也绝对无法被破译。<br>随着计算机的进步，现在 DES 已经能够被暴力破解，强度大不如前了。<br>AES 是取代其 DES 而成为新标准的一种对称密码算法。在 2000 年从候选算法中选出了一种名为 Rijndael 的对称密码算法，并将其确定为了 AES。</p>
</summary>
<category term="读书笔记" scheme="http://kaywu.xyz/tags/%E8%AF%BB%E4%B9%A6%E7%AC%94%E8%AE%B0/"/>
</entry>
<entry>
<title>转码微信 speex</title>
<link href="http://kaywu.xyz/2018/04/22/wechat-speex/"/>
<id>http://kaywu.xyz/2018/04/22/wechat-speex/</id>
<published>2018-04-22T06:29:35.000Z</published>
<updated>2018-04-22T08:23:12.000Z</updated>
<content type="html"><![CDATA[<p>H5 可以使用微信 jssdk 提供的录音接口，将录音上传到微信的服务器。而后端可以通过<a href="https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1444738727" target="_blank" rel="noopener">获取微信录音</a>的接口下载录音。<br>录音有两种格式，一种是 8K 采样率的 amr 格式。还有一种是 16K 采样率的 speex 格式。speex 格式的录音更清晰，当然文件也更大。同样的录音，speex 格式大约是 amr 格式的 4 倍。但同时，amr 格式的录音失真十分严重。如果有播放、语音识别的需求，建议还是采用 speex 格式的录音。<br>不知出于什么考虑，微信对 speex 格式的录音做了加工，得使用 speex 官方解码库结合微信的解码库才能进行转码。<br>下文介绍如何使用 docker 编译可转码微信 speex 的程序。</p><a id="more"></a><h2 id="编译"><a href="#编译" class="headerlink" title="编译"></a>编译</h2><p>首先 <code>git clone https://github.com/ppninja/wechat-speex-declib</code>。该仓库在<a href="http://wximg.gtimg.com/shake_tv/mpwiki/declib.zip" target="_blank" rel="noopener">微信解码库示例</a> 的基础上添加了 Makefile。<br>由于编译需要使用 linux，下文使用 ubuntu 16.04 作为示例。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"># 启动 ubuntu 16.04 的 docker 并运行 bash，绑定本机 wechat-speex-declib 的地址到 docker 的 /opt/speex 上</span><br><span class="line">docker pull ubuntu:16.04</span><br><span class="line">docker run -i -t --mount type=bind,source=(wechat-speex-declib 的绝对地址替换),target=/opt/speex ubuntu:16.04 /bin/bash</span><br><span class="line"></span><br><span class="line"># 修改为阿里云的镜像源，提升 apt-get 的速度</span><br><span class="line">cat > /etc/apt/sources.list << END</span><br><span class="line">deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted universe multiverse</span><br><span class="line">deb http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted universe multiverse</span><br><span class="line">deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted universe multiverse</span><br><span class="line">deb http://mirrors.aliyun.com/ubuntu/ xenial-proposed main restricted universe multiverse</span><br><span class="line">deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse</span><br><span class="line">deb-src http://mirrors.aliyun.com/ubuntu/ xenial main restricted universe multiverse</span><br><span class="line">deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted universe multiverse</span><br><span class="line">deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted universe multiverse</span><br><span class="line">deb-src http://mirrors.aliyun.com/ubuntu/ xenial-proposed main restricted universe multiverse</span><br><span class="line">deb-src http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse</span><br><span class="line">END</span><br><span class="line"></span><br><span class="line"># 安装 gcc 等编译工具</span><br><span class="line">apt-get build-essential</span><br><span class="line"># 安装 speex 开发库</span><br><span class="line">apt-get libspeex-dev</span><br><span class="line"></span><br><span class="line">cd /opt/speex</span><br><span class="line">make</span><br></pre></td></tr></table></figure><p>注意这里安装的 speex 开发库是 libspeex-dev，而不是 speex。speex 库提供了 speex 编码、解码的命令行，而 libspeex-dev 库提供了开发所需的文件。</p><p>执行完后在 bin 目录下会新增 speex_decode，也就是编译出的可执行文件。<br>通过执行 <code>./speex_decode test.speex test.wav</code> 就可以将微信定制的 speex 转码成标准的 wav 格式了。<br>提醒下，转码产生的 wav 格式未压缩，体积很大。可以通过 ffmpeg 等工具进行处理。</p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1444738727" target="_blank" rel="noopener">获取微信录音</a></li><li><a href="http://wximg.gtimg.com/shake_tv/mpwiki/declib.zip" target="_blank" rel="noopener">微信解码库示例</a></li><li><a href="https://github.com/ppninja/wechat-speex-declib" target="_blank" rel="noopener">wechat speex declib</a></li><li><a href="https://packages.qa.debian.org/s/speex.html" target="_blank" rel="noopener">speex binaries</a></li></ul><!-- 获取临时素材 -->]]></content>
<summary type="html">
<p>H5 可以使用微信 jssdk 提供的录音接口，将录音上传到微信的服务器。而后端可以通过<a href="https://mp.weixin.qq.com/wiki?t=resource/res_main&amp;id=mp1444738727" target="_blank" rel="noopener">获取微信录音</a>的接口下载录音。<br>录音有两种格式，一种是 8K 采样率的 amr 格式。还有一种是 16K 采样率的 speex 格式。speex 格式的录音更清晰，当然文件也更大。同样的录音，speex 格式大约是 amr 格式的 4 倍。但同时，amr 格式的录音失真十分严重。如果有播放、语音识别的需求，建议还是采用 speex 格式的录音。<br>不知出于什么考虑，微信对 speex 格式的录音做了加工，得使用 speex 官方解码库结合微信的解码库才能进行转码。<br>下文介绍如何使用 docker 编译可转码微信 speex 的程序。</p>
</summary>
<category term="微信" scheme="http://kaywu.xyz/tags/%E5%BE%AE%E4%BF%A1/"/>
</entry>
<entry>
<title>如何用 Markdown 写 PPT</title>
<link href="http://kaywu.xyz/2017/10/19/reveal-md/"/>
<id>http://kaywu.xyz/2017/10/19/reveal-md/</id>
<published>2017-10-19T06:21:21.000Z</published>
<updated>2018-03-12T15:38:10.000Z</updated>
<content type="html"><![CDATA[<p><a href="https://github.com/hakimel/reveal.js/" target="_blank" rel="noopener">reveal-js</a> 是通过网页来制作 PPT 的 JavaScript 框架，通过它可以轻松制作精致的网页 PPT。它自身也支持 Markdown。<br>但使用过几次之后，感觉对于日常使用还是稍重了些。每次新建 PPT 都需要复制粘贴 boilerplate code。</p><p><a href="https://github.com/webpro/reveal-md" target="_blank" rel="noopener">reveal-md</a> 让你从这些繁琐的步骤中解放出来。它使用 reveal-js 自带的 Markdown 功能，让你只需写 Markdown，其他的事它帮你搞定。</p><h3 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h3><p>简单地说下如何使用。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">mkdir slides && cd slides</span><br><span class="line">yarn init -y</span><br><span class="line">yarn add reveal-md</span><br><span class="line">yarn exec reveal-md demo</span><br></pre></td></tr></table></figure></p><p>此时炫酷的 PPT 网页就在你眼前呈现了。</p><h3 id="语法"><a href="#语法" class="headerlink" title="语法"></a>语法</h3><p>标准的 Markdown 语法，默认 <code>---</code> 作为 PPT 页面的分割线。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"># Title</span><br><span class="line"></span><br><span class="line">* Point 1</span><br><span class="line">* Point 2</span><br><span class="line"></span><br><span class="line">---</span><br><span class="line"></span><br><span class="line">## Second slide</span><br><span class="line"></span><br><span class="line">> Best quote ever.</span><br><span class="line"></span><br><span class="line">Note: speaker notes FTW!</span><br></pre></td></tr></table></figure></p><h3 id="演示命令"><a href="#演示命令" class="headerlink" title="演示命令"></a>演示命令</h3><p>演示命令 <code>reveal-md</code>，启动本地 server 并使用 reveal.js 渲染 Markdown 文件。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yarn exec reveal-md demo.md</span><br></pre></td></tr></table></figure></p><p>更详细的配置请查看 <a href="https://github.com/webpro/reveal-md" target="_blank" rel="noopener">reveal-md 文档</a> 和 <a href="https://github.com/hakimel/reveal.js/" target="_blank" rel="noopener">reveal-js 文档</a>。</p>]]></content>
<summary type="html">
<p><a href="https://github.com/hakimel/reveal.js/" target="_blank" rel="noopener">reveal-js</a> 是通过网页来制作 PPT 的 JavaScript 框架，通过它可以轻松制作精致的网页
</summary>
<category term="Tools" scheme="http://kaywu.xyz/tags/Tools/"/>
</entry>
<entry>
<title>Rails 通过 path 实现 subdomain —— default_url_options 的妙用</title>
<link href="http://kaywu.xyz/2017/10/17/default_url_options/"/>
<id>http://kaywu.xyz/2017/10/17/default_url_options/</id>
<published>2017-10-17T14:18:46.000Z</published>
<updated>2018-03-12T15:38:10.000Z</updated>
<content type="html"><![CDATA[<p>最近在开发微信开放平台，需要通过 url 来区别不同的第三方。最简单直接的方法，就是通过 subdomain 来实现。<br>比如有两个第三方 A、B，那么 A 的域名为 A.example.com，B 的域名为 B.example.com。但这次由于 https 证书的问题只能通过 path 来实现，也就是 A 的域名为 <a href="http://www.example.com/A，B" target="_blank" rel="noopener">www.example.com/A，B</a> 的域名为 <a href="http://www.example.com/B。" target="_blank" rel="noopener">www.example.com/B。</a></p><p>但通过 path 实现会有一个严重的问题，就是如何保证生成的 url 带有第三方的信息。</p><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">scope <span class="symbol">path:</span> <span class="string">'/:third_party'</span> <span class="keyword">do</span></span><br><span class="line"> resources <span class="symbol">:orders</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>上面的配置使用 orders_path 时会报错 <code>missing required keys: [:third_path]</code>。<br>最笨的办法莫过于给所有的 url 手动加上，但这方法随着项目增大会变得不可行。</p><p>我们希望 url 的生成可以更加智能，当访问 url 为 <a href="http://www.example.com/A" target="_blank" rel="noopener">www.example.com/A</a> 时，使用 orders_path 可以自动生成 /A/orders。从逻辑上，这是行得通的。</p><p>首先想到 route 可以添加 default 配置，于是做了以下尝试，可惜报错了。在 route 时只能配置静态的默认值。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># undefined local variable or method `params'</span></span><br><span class="line">scope <span class="symbol">path:</span> <span class="string">'/:third_party'</span>, <span class="symbol">default:</span> {<span class="symbol">third_party:</span> params[<span class="symbol">:third_party</span>]} <span class="keyword">do</span></span><br><span class="line"> resources <span class="symbol">:orders</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>那么，有没有类似的方法，提供 url default 配置的功能呢？这时就轮到 default_url_options 出场了。<br>default_url_options 的用处不仅仅是在 config 时设置默认的 host，每个 Controller 都有该方法，调用 url_helper 时会结合该方法生成最终的 url。</p><p>在对应的 Controller 里添加以下代码，这时调用 orders_path 就会根据域名中的 third_parth 生成对应的域名了。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">default_url_options</span></span></span><br><span class="line"> {<span class="symbol">third_party:</span> params[<span class="symbol">:third_party</span>]}</span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>解决了生成 url 的问题，我们还需要解决测试时 url 的问题。这里我对 ActionController::TestCase 进行了 monkey patch，供大家参考。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ActionController::TestCase</span></span></span><br><span class="line"></span><br><span class="line"> alias_method <span class="symbol">:orig_process</span>, <span class="symbol">:process</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">process</span><span class="params">(action, <span class="symbol">method:</span> <span class="string">"GET"</span>, <span class="symbol">params:</span> {}, <span class="symbol">session:</span> <span class="literal">nil</span>, <span class="symbol">body:</span> <span class="literal">nil</span>, <span class="symbol">flash:</span> {}, <span class="symbol">format:</span> <span class="literal">nil</span>, <span class="symbol">xhr:</span> <span class="literal">false</span>, <span class="symbol">as:</span> <span class="literal">nil</span>)</span></span></span><br><span class="line"> <span class="keyword">if</span> @controller.is_a? WechatBaseController</span><br><span class="line"> params.merge!({<span class="symbol">app_alias:</span> <span class="string">'test'</span>})</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> orig_process(action, <span class="symbol">method:</span> method, <span class="symbol">params:</span> params, <span class="symbol">session:</span> session, <span class="symbol">body:</span> body, <span class="symbol">flash:</span> flash, <span class="symbol">format:</span> format, <span class="symbol">xhr:</span> xhr, <span class="symbol">as:</span> as)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<p>最近在开发微信开放平台，需要通过 url 来区别不同的第三方。最简单直接的方法，就是通过 subdomain 来实现。<br>比如有两个第三方 A、B，那么 A 的域名为 A.example.com，B 的域名为 B.example.com。但这次由于 https 证书的问
</summary>
<category term="Rails" scheme="http://kaywu.xyz/tags/Rails/"/>
</entry>
<entry>
<title>微信开发总结</title>
<link href="http://kaywu.xyz/2017/09/10/wechat-development/"/>
<id>http://kaywu.xyz/2017/09/10/wechat-development/</id>
<published>2017-09-10T07:47:00.000Z</published>
<updated>2017-09-17T15:40:42.000Z</updated>
<content type="html"><![CDATA[<p>最近开发了基于微信的相关项目，主要是微信的网页开发、微信支付以及发送消息这块的内容。项目本身没什么难度，但由于微信封闭的体系、混乱的配置以及分散的文档，爬了不少的坑，这里总结下经验。</p><h2 id="文档"><a href="#文档" class="headerlink" title="文档"></a>文档</h2><p>先说说微信的文档。微信的文档不是统一的，比如公众平台技术文档在<a href="https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1445241432" target="_blank" rel="noopener">一个网站</a>，而支付文档在<a href="https://pay.weixin.qq.com/wiki/doc/api/jsapi.php?chapter=7_1" target="_blank" rel="noopener">另一个网站</a>，但支付文档所使用的 js 的相关文档又在前一个网站。除此之外，还有<a href="https://mp.weixin.qq.com/debug/wxadoc/dev/index.html" target="_blank" rel="noopener">小程序文档</a>、<a href="https://open.weixin.qq.com/cgi-bin/showdocument?action=dir_list&t=resource/res_list&verify=1&id=open1419318292&token=&lang=zh_CN" target="_blank" rel="noopener">开放平台文档</a>。从这混乱的文档相信你也能或多或少体会到开发的难处。</p><h2 id="配置"><a href="#配置" class="headerlink" title="配置"></a>配置</h2><p>出于安全性（更多是封闭性）的考虑，微信的配置繁琐且复杂，这里我把整个流程都记录下来以供参考。<br>首先登录公众平台，在开发-基本配置里获得 <code>AppSecret</code> 并保存下来，之后网站就不会再显示开发者密码，只能重置。<br>同样在该页面，配置 IP 白名单。如果你需要接收用户公众号的消息以及事件推送，同一页面添加服务器配置。<br>然后，在设置-公众号设置-功能设置，按照说明对业务域名、JS 接口安全域名、网页授权域名进行设置。<br>如果有微信支付的需求，去<a href="https://pay.weixin.qq.com" target="_blank" rel="noopener">微信商户平台</a>，商户平台-产品中心-开发配置中设置公众号支付的授权目录。注意，授权目录必须是发起支付网址的上一级目录。举例来说，<br>发起支付的网址为 <code>www.xxx.com/orders/22/pay</code>，那么支付目录就必须为 <code>www.xxx.com/orders/22/</code>，填 <code>www.xxx.com/</code> 或者 <code>www.xxx.com/orders/</code> 都是不行的。再加上支付的授权目录只能填 5 个，发起支付的域名得注意设计。</p><p>PS: 没有公众号的朋友可以通过微信的<a href="https://mp.weixin.qq.com/debug/cgi-bin/sandbox?t=sandbox/login" target="_blank" rel="noopener">测试号</a>来试验除微信支付以外大部分的功能。</p><a id="more"></a><h2 id="开发"><a href="#开发" class="headerlink" title="开发"></a>开发</h2><p>先简单地介绍下常见的开发概念。</p><ul><li>access_token：通过 AppID 和 AppSecret 获得，是公众号的全局唯一接口调用凭据，调用各接口时都需使用。由于是全局唯一，所以建议正式环境、测试环境使用两个公众号，不然会各自冲突。</li><li>jsapi_ticket：通过 access_token 获取，是公众号用于调用微信 JS 接口的临时票据。网页在使用 JS-SDK 的时需要先调用 config 接口，其中的参数 signature 需要用到 jsapi_ticket 生成。</li></ul><p>注意下 access_token、jsapi_ticket 有效期都为 2 小时，且每天的调用次数有限，需要做全局缓存以及过期自动刷新的处理。</p><p>我开发的项目使用了 Rails + <a href="https://github.com/Eric-Guo/wechat" target="_blank" rel="noopener">wechat gem</a> + <a href="https://github.com/jasl/wx_pay" target="_blank" rel="noopener">wx_pay gem</a>。wechat gem 自动管理 access_token、jsapi_ticket，封装了公众号的接口并提供了授权地址、调用 JS-SDK 的便捷方法，wx_pay gem 封装了支付相关的接口。强烈推荐使用，能够省去不少功夫。</p><h3 id="网页授权"><a href="#网页授权" class="headerlink" title="网页授权"></a>网页授权</h3><p>微信的网页授权也是常见的 OAuth 2.0，与其他网页不同的是有两个不同的 scope。</p><ul><li>以 snsapi_base 为 scope 发起的网页授权，可以获取进入页面的用户的openid，静默授权并自动跳转到回调页的。用户感知的就是直接进入了回调页。</li><li>以 snsapi_userinfo 为 scope 发起的网页授权，可以获取用户的基本信息。但这种授权需要用户手动同意，并且由于用户同意过，所以无须关注，就可在授权后获取该用户的基本信息。不同于用户管理类接口中的“获取用户基本信息接口”，需要该用户关注了公众号后，才能调用成功。</li></ul><h3 id="微信支付"><a href="#微信支付" class="headerlink" title="微信支付"></a>微信支付</h3><p>在调用微信支付前，需要从后端发起 unifiedorder 统一下单请求获取到 prepay_id，通过 prepay_id、AppID、商品平台的 key 等参数生成签名，最后调用 wx.chooseWXPay 发起微信支付。<br>注意，生成支付签名这边有很多的坑，比如 timestamp 在 wx.chooseWxPay 里是小写，而在生成签名时使用的是 timeStamp。尽量使用成熟的库来避免这种不知所谓的坑。<br>统一下单请求时会要求传入参数 notify_url，这个是异步接受微信支付结果通知的回调地址。注意该地址不能携带参数。举例来说 <code>https://www.xxx.com?a=1</code>，回调时 <code>a=1</code>会被省略。</p><h3 id="发送消息"><a href="#发送消息" class="headerlink" title="发送消息"></a>发送消息</h3><p>发送消息主要有两大部分，其一是模板消息。<br>微信对模板消息有着严格的管理，所在行业有相应的模板库，你只能从模板库里选择添加到我的模板，向模板库里新增模板需要经过审核。<br>新增模板时最好记录下模板编号，因为添加到我的模板后，只能看见模板 ID，看不到原始的模板编号。而每一个公众号添加相同的模板时对应的模板 ID 是不一样的。</p><p>除模板消息之外，还有客服消息。客服消息又分文本消息、图片消息、语音消息等。文档里虽然没提，文本消息支持 <code><br></code> 换行、<code><a href="#"></code> 超链接这些功能。</p><h2 id="调试"><a href="#调试" class="headerlink" title="调试"></a>调试</h2><p>刚开始开发微信时有一个很头疼的事，怎么才能做到本机调试呢。这里说下我个人的方法。<br>假设开发网站的域名是 <code>http://www.example.com</code>，通过修改 hosts、nginx 转发等方法，将 <code>http://www.example.com</code> 映射到本地端口如 <code>http://localhost:3000</code> 上。之后通过 <a href="https://mp.weixin.qq.com/debug/wxadoc/dev/devtools/download.html" target="_blank" rel="noopener">微信开发者工具</a> 就可以本机调试了。<br>但是微信开发者工具也有其限制，比如不能模拟微信支付，这时候就需要用到手机了。手机通过 charles 代理，由于本机 <code>http://www.example.com</code> 的请求已经转发到本地端口，也就实现了本机调试。<br>同时还有 TBS Studio，可以通过 USB 连接手机实现真机调试。实际使用感觉效率一般，建议补充使用。</p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1445241432" target="_blank" rel="noopener">微信公众平台技术文档</a></li><li><a href="https://pay.weixin.qq.com/wiki/doc/api/jsapi.php?chapter=7_1" target="_blank" rel="noopener">微信支付开发文档</a></li></ul><!-- 公众平台文档 --><!-- 微信支付文档 --><!-- 小程序文档 --><!-- 开放平台文档 --><!-- 微信测试号 --><!-- 微信商户平台 --><!-- wechat gem --><!-- wx_pay gem --><!-- 支付签名算法 --><!-- 微信开发工具 -->]]></content>
<summary type="html">
<p>最近开发了基于微信的相关项目，主要是微信的网页开发、微信支付以及发送消息这块的内容。项目本身没什么难度，但由于微信封闭的体系、混乱的配置以及分散的文档，爬了不少的坑，这里总结下经验。</p>
<h2 id="文档"><a href="#文档" class="headerlink" title="文档"></a>文档</h2><p>先说说微信的文档。微信的文档不是统一的，比如公众平台技术文档在<a href="https://mp.weixin.qq.com/wiki?t=resource/res_main&amp;id=mp1445241432" target="_blank" rel="noopener">一个网站</a>，而支付文档在<a href="https://pay.weixin.qq.com/wiki/doc/api/jsapi.php?chapter=7_1" target="_blank" rel="noopener">另一个网站</a>，但支付文档所使用的 js 的相关文档又在前一个网站。除此之外，还有<a href="https://mp.weixin.qq.com/debug/wxadoc/dev/index.html" target="_blank" rel="noopener">小程序文档</a>、<a href="https://open.weixin.qq.com/cgi-bin/showdocument?action=dir_list&amp;t=resource/res_list&amp;verify=1&amp;id=open1419318292&amp;token=&amp;lang=zh_CN" target="_blank" rel="noopener">开放平台文档</a>。从这混乱的文档相信你也能或多或少体会到开发的难处。</p>
<h2 id="配置"><a href="#配置" class="headerlink" title="配置"></a>配置</h2><p>出于安全性（更多是封闭性）的考虑，微信的配置繁琐且复杂，这里我把整个流程都记录下来以供参考。<br>首先登录公众平台，在开发-基本配置里获得 <code>AppSecret</code> 并保存下来，之后网站就不会再显示开发者密码，只能重置。<br>同样在该页面，配置 IP 白名单。如果你需要接收用户公众号的消息以及事件推送，同一页面添加服务器配置。<br>然后，在设置-公众号设置-功能设置，按照说明对业务域名、JS 接口安全域名、网页授权域名进行设置。<br>如果有微信支付的需求，去<a href="https://pay.weixin.qq.com" target="_blank" rel="noopener">微信商户平台</a>，商户平台-产品中心-开发配置中设置公众号支付的授权目录。注意，授权目录必须是发起支付网址的上一级目录。举例来说，<br>发起支付的网址为 <code>www.xxx.com/orders/22/pay</code>，那么支付目录就必须为 <code>www.xxx.com/orders/22/</code>，填 <code>www.xxx.com/</code> 或者 <code>www.xxx.com/orders/</code> 都是不行的。再加上支付的授权目录只能填 5 个，发起支付的域名得注意设计。</p>
<p>PS: 没有公众号的朋友可以通过微信的<a href="https://mp.weixin.qq.com/debug/cgi-bin/sandbox?t=sandbox/login" target="_blank" rel="noopener">测试号</a>来试验除微信支付以外大部分的功能。</p>
</summary>
<category term="Rails" scheme="http://kaywu.xyz/tags/Rails/"/>
<category term="微信" scheme="http://kaywu.xyz/tags/%E5%BE%AE%E4%BF%A1/"/>
</entry>
<entry>
<title>互联网创业核心技术：构建可伸缩的 Web 应用</title>
<link href="http://kaywu.xyz/2017/07/13/core-tech/"/>
<id>http://kaywu.xyz/2017/07/13/core-tech/</id>
<published>2017-07-13T09:23:30.000Z</published>
<updated>2017-09-10T08:13:51.000Z</updated>
<content type="html"><![CDATA[<p>《互联网创业核心技术：构建可伸缩的 Web 应用》整本书围绕“可伸缩”三个字，对 Web 应用的每一层展开了全面细致的解说。<br>如果你和我一样，对负载均衡、水平伸缩之类的概念不是很了解，或者不清楚它在整个架构中的使用，那么你应该来读下这本书。<br>读完之后你会发现架构这东西是有迹可循的，是围绕一系列基本的原则建立起来的。</p><p>一图胜千言，这本书不仅有着易懂的语言，而且有大量简洁的示意图。<br><a id="more"></a><br><img src="/img/core_tech.png" alt=""><br>上图大致说明了本书涉及的内容，从前后端设计、存储设计，再到消息队列和缓存等的使用场景。</p><p>我读到这本书时颇有意外之喜，毕竟这是凑单的书，可见这书挺冷门的。在这里推荐一下，希望能有更多的人读到。<br>可以搭配 <a href="https://www.youtube.com/watch?v=-W9F__D3oY4" target="_blank" rel="noopener">Scalability Harvard Web Development (需翻墙)</a> 一起使用。</p>]]></content>
<summary type="html">
<p>《互联网创业核心技术：构建可伸缩的 Web 应用》整本书围绕“可伸缩”三个字，对 Web 应用的每一层展开了全面细致的解说。<br>如果你和我一样，对负载均衡、水平伸缩之类的概念不是很了解，或者不清楚它在整个架构中的使用，那么你应该来读下这本书。<br>读完之后你会发现架构这东西是有迹可循的，是围绕一系列基本的原则建立起来的。</p>
<p>一图胜千言，这本书不仅有着易懂的语言，而且有大量简洁的示意图。<br>
</summary>
<category term="读书笔记" scheme="http://kaywu.xyz/categories/%E8%AF%BB%E4%B9%A6%E7%AC%94%E8%AE%B0/"/>
<category term="读书笔记" scheme="http://kaywu.xyz/tags/%E8%AF%BB%E4%B9%A6%E7%AC%94%E8%AE%B0/"/>
</entry>
<entry>
<title>FactoryGirl 源码浅析</title>
<link href="http://kaywu.xyz/2017/06/28/factory-girl-analysis/"/>
<id>http://kaywu.xyz/2017/06/28/factory-girl-analysis/</id>
<published>2017-06-28T09:21:20.000Z</published>
<updated>2017-09-10T08:13:51.000Z</updated>
<content type="html"><![CDATA[<p>FactoryGirl 是我个人十分喜欢的 Gem，它能很方便地模拟测试数据。使用技巧可见 <a href="http://www.jianshu.com/p/cca80f341d77" target="_blank" rel="noopener">FactoryGirl 技巧</a>。出于兴趣我研究了下它的源代码，说实话比想象得要复杂。由于 FactoryGirl 配置的灵活性以及 Ruby 本身的语言特点，使得它代码整体上比较飘逸。<br>这里我会对它最基础的方法进行分析，版本为 4.8.0。</p><h3 id="FactoryGirl-define"><a href="#FactoryGirl-define" class="headerlink" title="FactoryGirl.define"></a>FactoryGirl.define</h3><p>先从定义 Factory 开始。下面的代码定义了名为 user 的 Factory，它有一个名为 name 的属性，值为 Kay。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">FactoryGirl.define <span class="keyword">do</span></span><br><span class="line"> factory <span class="symbol">:user</span> <span class="keyword">do</span></span><br><span class="line"> name <span class="string">'Kay'</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><a id="more"></a><p>我们先找到 FactoryGirl.define 的入口，在 syntax/default.rb 里。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># syntax/default.rb</span></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">FactoryGirl</span></span></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">Syntax</span></span></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">Default</span></span></span><br><span class="line"> <span class="keyword">include</span> Methods</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">define</span><span class="params">(&block)</span></span></span><br><span class="line"> DSL.run(block)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="class"><span class="keyword">class</span> <span class="title">DSL</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">factory</span><span class="params">(name, options = {}, &block)</span></span></span><br><span class="line"> factory = Factory.new(name, options)</span><br><span class="line"> proxy = FactoryGirl::DefinitionProxy.new(factory.definition)</span><br><span class="line"> proxy.instance_eval(&block) <span class="keyword">if</span> block_given?</span><br><span class="line"></span><br><span class="line"> FactoryGirl.register_factory(factory)</span><br><span class="line"></span><br><span class="line"> proxy.child_factories.each <span class="keyword">do</span> <span class="params">|(child_name, child_options, child_block)|</span></span><br><span class="line"> parent_factory = child_options.delete(<span class="symbol">:parent</span>) <span class="params">||</span> name</span><br><span class="line"> factory(child_name, child_options.merge(<span class="symbol">parent:</span> parent_factory), &child_block)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">run</span><span class="params">(block)</span></span></span><br><span class="line"> new.instance_eval(&block)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> extend Syntax::Default</span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>从上面的代码分析可得，define 会调用 DSL.run，之后调用了 DSL#factory 方法，传入的参数 name 为 <code>:user</code>，block 为 <code>{ name 'Kay' }</code>。我们重点看 factory 方法其中 4 行：<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># DSL#factory</span></span><br><span class="line">factory = Factory.new(name, options)</span><br><span class="line">proxy = FactoryGirl::DefinitionProxy.new(factory.definition)</span><br><span class="line">proxy.instance_eval(&block) <span class="keyword">if</span> block_given?</span><br><span class="line"></span><br><span class="line">FactoryGirl.register_factory(factory)</span><br></pre></td></tr></table></figure></p><p>首先，会创建一个新的 Factory 对象，然后通过代理类 FactoryGirl::DefinitionProxy 对其进行封装，并执行其中的 block，最后注册这个 factory。其中 <code>proxy.instance_eval(&block) if block_given?</code> 是属性赋值的关键。</p><p><code>{ name 'Kay' }</code> 会调用 FactoryGirl::DefinitionProxy 的 name 方法，但由于代理类没有该方法，最终会执行 method_missing，而该方法实现了赋值的逻辑。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">method_missing</span><span class="params">(name, *args, &block)</span></span></span><br><span class="line"> <span class="keyword">if</span> args.empty? && block.<span class="literal">nil</span>?</span><br><span class="line"> @definition.declare_attribute(Declaration::Implicit.new(name, @definition, @ignore))</span><br><span class="line"> <span class="keyword">elsif</span> args.first.respond_to?(<span class="symbol">:has_key?</span>) && args.first.has_key?(<span class="symbol">:factory</span>)</span><br><span class="line"> association(name, *args)</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> add_attribute(name, *args, &block)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>由于赋值除了固定值，还可能是 block 或者 association，这里对几种情况分别进行了处理。</p><p>FactoryGirl.define 的分析差不多结束了，method_missing 这一元编程的魔法在这里又发挥了巨大的作用。</p><h3 id="FactoryGirl-create"><a href="#FactoryGirl-create" class="headerlink" title="FactoryGirl.create"></a>FactoryGirl.create</h3><p>讲完了 Factory 是如何定义的，我们来研究下如何创建一个 Factory。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">user = FactoryGirl.create(<span class="symbol">:user</span>, <span class="symbol">name:</span> <span class="string">'Link'</span>)</span><br></pre></td></tr></table></figure></p><p>我们首先得找到 create 的入口。当搜寻了一遍之后会发现，并没有显式定义 create 的地方。看来该方法是动态定义的了。从官方文档上来看，create 定义在 FactoryGirl::Syntax::Methods。搜索之后发现 StrategySyntaxMethodRegistrar 里出现了给它动态添加方法的代码。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># strategy_syntax_method_registrar.rb</span></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">FactoryGirl</span></span></span><br><span class="line"> private</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">define_singular_strategy_method</span></span></span><br><span class="line"> strategy_name = @strategy_name</span><br><span class="line"></span><br><span class="line"> define_syntax_method(strategy_name) <span class="keyword">do</span> <span class="params">|name, *traits_and_overrides, &block|</span></span><br><span class="line"> FactoryRunner.new(name, strategy_name, traits_and_overrides).run(&block)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">define_syntax_method</span><span class="params">(name, &block)</span></span></span><br><span class="line"> FactoryGirl::Syntax::Methods.module_exec <span class="keyword">do</span></span><br><span class="line"> <span class="keyword">if</span> method_defined?(name) <span class="params">||</span> private_method_defined?(name)</span><br><span class="line"> undef_method(name)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> define_method(name, &block)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>用 create 替换 strategy_name，FactoryGirl.create 实质调用了 <code>FactoryRunner.new(name, :create, : traits_and_overrides).run(&block)</code>。而 FactoryRunner#run 在进行了一些准备后，最终调用了 <code>factory.run(runner_strategy, @overrides, &block)</code> 来创建对象。</p><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Factory#run</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(build_strategy, overrides, &block)</span></span></span><br><span class="line"> block <span class="params">||</span>= ->(result) { result }</span><br><span class="line"> compile</span><br><span class="line"> strategy = StrategyCalculator.new(build_strategy).strategy.new</span><br><span class="line"> evaluator = evaluator_class.new(strategy, overrides.symbolize_keys)</span><br><span class="line"> attribute_assigner = AttributeAssigner.new(evaluator, build_class, &compiled_constructor)</span><br><span class="line"> evaluation = Evaluation.new(attribute_assigner, compiled_to_create)</span><br><span class="line"> evaluation.add_observer(CallbacksObserver.new(callbacks, evaluator))</span><br><span class="line"> strategy.result(evaluation).tap(&block)</span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>最后一行 <code>strategy.result(evaluation).tap(&block)</code> 最终返回的是 evaluation.object，而 evaluation 把 object 委托给 attribute_assigner。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># AttributeAssigner#object</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">object</span></span></span><br><span class="line"> @evaluator.instance = build_class_instance</span><br><span class="line"> build_class_instance.tap <span class="keyword">do</span> <span class="params">|instance|</span></span><br><span class="line"> attributes_to_set_on_instance.each <span class="keyword">do</span> <span class="params">|attribute|</span></span><br><span class="line"> instance.public_send(<span class="string">"<span class="subst">#{attribute}</span>="</span>, get(attribute))</span><br><span class="line"> @attribute_names_assigned << attribute</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>build_class_instance 实质上调用了 User.new 方法，创建了 User 对象。<code>instance.public_send</code> 对该对象的属性进行赋值。</p><p>至此最关键的创建步骤说完了，我们简单地说下其他几行的作用。<br><code>compile</code> 主要处理 Factory 之间的继承关系。<br><code>StrategyCalculator.new(build_strategy).strategy.new</code> 为简单工厂，可以根据 Strategy 的名字找到对应的类。<br>evaluator 保存了属性的赋值。为什么不直接进行赋值，而要使用一个中间类？我觉得原因是，FactoryGirl 创建 user 时不仅可以给 user 的属性赋值，还可以给 evaluator 赋值。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">factory <span class="symbol">:user</span> <span class="keyword">do</span></span><br><span class="line"> name <span class="string">"John Doe"</span></span><br><span class="line"> factory <span class="symbol">:user_with_posts</span> <span class="keyword">do</span></span><br><span class="line"> transient <span class="keyword">do</span></span><br><span class="line"> posts_count <span class="number">5</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> after(<span class="symbol">:create</span>) <span class="keyword">do</span> <span class="params">|user, evaluator|</span></span><br><span class="line"> create_list(<span class="symbol">:post</span>, evaluator.posts_count, <span class="symbol">user:</span> user)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 调用</span></span><br><span class="line">create(<span class="symbol">:user_with_posts</span>, <span class="symbol">posts_count:</span> <span class="number">15</span>)</span><br></pre></td></tr></table></figure></p><p>这里的 posts_count 不是 user 本身的属性，而属于 evaluator。<br>AttributeAssigner 创建了 user 对象，并给它赋值。<br>Evaluation 是对 Strategy 回调方法的一个封装，比如 Strategy::Create，都是直接调用的 evaluation 的方法。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Strategy::Create#result</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">result</span><span class="params">(evaluation)</span></span></span><br><span class="line"> evaluation.object.tap <span class="keyword">do</span> <span class="params">|instance|</span></span><br><span class="line"> evaluation.notify(<span class="symbol">:after_build</span>, instance)</span><br><span class="line"> evaluation.notify(<span class="symbol">:before_create</span>, instance)</span><br><span class="line"> evaluation.create(instance)</span><br><span class="line"> evaluation.notify(<span class="symbol">:after_create</span>, instance)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>顺带说下，compiled_constructor、compiled_to_create 在经过千辛万苦的查找后，是两个非常简单的 block。compiled_constructor 为 <code>{ new }</code>，而 compiled_to_create 为 <code>{ |instance| instance.save! }</code>，来自 configuration.rb 19、20 行。</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>由于水平精力有限，只简单地分析了最基础的 define 和 create 方法。<br>define 使用了 method_missing 来实现属性的赋值。由于 FactoryGirl create 对象时能通过很灵活的方式，比如 trait，使得其代码在创建对象时要考虑各方面的配置。这里只抽出了最主要的流程进行说明。</p>]]></content>
<summary type="html">
<p>FactoryGirl 是我个人十分喜欢的 Gem，它能很方便地模拟测试数据。使用技巧可见 <a href="http://www.jianshu.com/p/cca80f341d77" target="_blank" rel="noopener">FactoryGirl 技巧</a>。出于兴趣我研究了下它的源代码，说实话比想象得要复杂。由于 FactoryGirl 配置的灵活性以及 Ruby 本身的语言特点，使得它代码整体上比较飘逸。<br>这里我会对它最基础的方法进行分析，版本为 4.8.0。</p>
<h3 id="FactoryGirl-define"><a href="#FactoryGirl-define" class="headerlink" title="FactoryGirl.define"></a>FactoryGirl.define</h3><p>先从定义 Factory 开始。下面的代码定义了名为 user 的 Factory，它有一个名为 name 的属性，值为 Kay。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">FactoryGirl.define <span class="keyword">do</span></span><br><span class="line"> factory <span class="symbol">:user</span> <span class="keyword">do</span></span><br><span class="line"> name <span class="string">'Kay'</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p>
</summary>
<category term="Ruby" scheme="http://kaywu.xyz/categories/Ruby/"/>
<category term="Ruby" scheme="http://kaywu.xyz/tags/Ruby/"/>
</entry>
<entry>
<title>正则表达式必知必会</title>
<link href="http://kaywu.xyz/2017/05/30/regex/"/>
<id>http://kaywu.xyz/2017/05/30/regex/</id>
<published>2017-05-30T09:37:21.000Z</published>
<updated>2018-05-26T02:57:46.000Z</updated>
<content type="html"><![CDATA[<h2 id="匹配单个字符"><a href="#匹配单个字符" class="headerlink" title="匹配单个字符"></a>匹配单个字符</h2><h3 id="匹配纯文本"><a href="#匹配纯文本" class="headerlink" title="匹配纯文本"></a>匹配纯文本</h3><p>正则：<code>Ben</code></p><p>Hello, my name is <code>Ben</code>. Please visit my website at <a href="http://www.forta.com/" target="_blank" rel="noopener">http://www.forta.com/</a>.</p><h3 id="匹配任意字符"><a href="#匹配任意字符" class="headerlink" title="匹配任意字符"></a>匹配任意字符</h3><p>. 匹配任意单个字符。在绝大多数的正则表达式实现里，. 只能匹配除换行符以外的任何单个字符。<br>正则：<code>sales.</code></p><p><code>sales1</code>.xls<br>orders3.xls<br><code>sales2</code>.xls<br><code>sales3</code>.xls<br>apac1.xls<br>europe2.xls<br>na1.xls<br>na2.xls<br>sa1.xls</p><a id="more"></a><h3 id="匹配特殊字符"><a href="#匹配特殊字符" class="headerlink" title="匹配特殊字符"></a>匹配特殊字符</h3><p>元字符是一些在正则表达式里有着特殊含义的字符。如英语句号（.）是一个元字符，它可以用来匹配任何一个单个字符。因为元字符在正则表达式里有着特殊的含义，所以这些字符就无法用来代表它们本身。比如你不能使用 . 类匹配 . 本身。<br>在元字符的前面加上一个反斜杠就可以对它进行转移，转义序列 . 将匹配 . 本身。<br>正则：<code>.a.\.xls</code></p><p>sales1.xls<br>orders3.xls<br>sales2.xls<br>sales3.xls<br>apac1.xls<br>europe2.xls<br><code>na1.xls</code><br><code>na2.xls</code><br><code>sa1.xls</code></p><h2 id="匹配一组字符"><a href="#匹配一组字符" class="headerlink" title="匹配一组字符"></a>匹配一组字符</h2><h3 id="匹配多个字符中的某一个"><a href="#匹配多个字符中的某一个" class="headerlink" title="匹配多个字符中的某一个"></a>匹配多个字符中的某一个</h3><p>可以使用元字符 [ 和 ] 来定义一个字符集合，在使用 [ 和 ] 定义的字符集合里，这两个元字符之间的所有字符都是该集合的组成部分，字符集合的匹配结果是能够与该集合里的任意一个成员想匹配的文本。<br>正则：<code>[ns]a.\.xls</code></p><p>sales1.xls<br>orders3.xls<br>sales2.xls<br>sales3.xls<br>apac1.xls<br>europe2.xls<br><code>na1.xls</code><br><code>na2.xls</code><br><code>sa1.xls</code><br>ca1.xls</p><h3 id="利用字符集合区间"><a href="#利用字符集合区间" class="headerlink" title="利用字符集合区间"></a>利用字符集合区间</h3><p>模式 [0-9] 的功能与 [0123456789] 完全等价。</p><p>正则：<code>[ns]a[0-9]\.xls</code></p><p>sales1.xls<br>orders3.xls<br>sales2.xls<br>sales3.xls<br>apac1.xls<br>europe2.xls<br><code>na1.xls</code><br><code>na2.xls</code><br><code>sa1.xls</code><br>ca1.xls</p><p>其他常用的字符区间：<br>A-Z：匹配从 A 到 Z 的所有大写字母。<br>a-z：匹配从 a 到 z 的所有小写字母。</p><h3 id="取非匹配"><a href="#取非匹配" class="headerlink" title="取非匹配"></a>取非匹配</h3><p>用元字符 ^ 来表明你相对一个字符集合进行取非匹配，也就是除了那个字符集里的字符，其他字符都可以匹配。<br>正则：<code>[ns]a[^0-9]\.xls</code></p><p>sales1.xls<br>orders3.xls<br>sales2.xls<br>sales3.xls<br>apac1.xls<br>europe2.xls<br><code>sam.xls</code><br>na1.xls<br>na2.xls<br>sa1.xls<br>ca1.xls</p><h3 id="匹配特定的字符类型"><a href="#匹配特定的字符类型" class="headerlink" title="匹配特定的字符类型"></a>匹配特定的字符类型</h3><table><thead><tr><th>元字符</th><th>说明</th></tr></thead><tbody><tr><td>\d</td><td>任何一个数字字符（等价于[0-9]）</td></tr><tr><td>\D</td><td>任何一个非数字字符（等价于[^0-9]）</td></tr><tr><td>\w</td><td>任何一个字母数字字符（大小写均可）或下划线字符（等价于[a-zA-Z0-9_]）</td></tr><tr><td>\W</td><td>任何一个非字母数字或非下划线字符（等价于[^a-zA-Z0-9_]）</td></tr></tbody></table><p>正则：<code>myArray\[\d\]</code></p><p>var myArray = new Array();<br>…<br>if (<code>myArray[0]</code> == 0) {<br>…<br>}</p><p>正则：<code>\w\d\w\d\w\d</code></p><p>11213<br><code>A1C2E3</code><br>48075<br>48237<br><code>M1B4F2</code><br>90046<br><code>H1H2H2</code></p><h3 id="匹配空白字符"><a href="#匹配空白字符" class="headerlink" title="匹配空白字符"></a>匹配空白字符</h3><table><thead><tr><th>元字符</th><th>说明</th></tr></thead><tbody><tr><td>[\b]</td><td>回退（并删除）一个字符（Backspace键）</td></tr><tr><td>\f</td><td>换页符</td></tr><tr><td>\n</td><td>换行符</td></tr><tr><td>\r</td><td>回车符</td></tr><tr><td>\t</td><td>制表符</td></tr><tr><td>\v</td><td>垂直制表符</td></tr><tr><td>\s</td><td>任何一个空白字符（等价于[\f\n\r\t\v]）</td></tr><tr><td>\S</td><td>任何一个非空白字符（等价于[^\f\n\r\t\v]）</td></tr></tbody></table><h2 id="重复匹配"><a href="#重复匹配" class="headerlink" title="重复匹配"></a>重复匹配</h2><h3 id="匹配一个或多个字符"><a href="#匹配一个或多个字符" class="headerlink" title="匹配一个或多个字符"></a>匹配一个或多个字符</h3><ul><li>匹配一个或多个字符（至少一个；不匹配零个字符的情况）。<br>正则：<code>\w+@\w+\.\w+</code></li></ul><p>Send personal email to <code>ben@forta.com</code>. For questions about a book use <code>support@forta.com</code>. Feel free to send unsolicited email to <code>spam@forta.com</code> (wouldn’t it be nice if it were that simple, huh?).</p><h3 id="匹配零个或多个字符"><a href="#匹配零个或多个字符" class="headerlink" title="匹配零个或多个字符"></a>匹配零个或多个字符</h3><ul><li>元字符匹配零个或多个字符。<br>正则：<code>\w+[\w.]*@[\w.]+\.\w+</code></li></ul><p>Hello, <code>.ben@forta.com</code> is my email address.</p><h3 id="匹配零个或一个字符"><a href="#匹配零个或一个字符" class="headerlink" title="匹配零个或一个字符"></a>匹配零个或一个字符</h3><p>? 只能匹配一个字符（或字符集合）的零次或一次出现，最多不超过一次。<br>正则：<code>https?://[\w./]+</code></p><p>The URL is <code>http://www.forta.com/</code>, to connect securely use <code>https://www.forta.com/</code> instead.</p><h3 id="匹配的重复次数"><a href="#匹配的重复次数" class="headerlink" title="匹配的重复次数"></a>匹配的重复次数</h3><p>{3}意味着模式里的前一个字符（或字符集合）必须在原始文本里连续重复出现 3 次才算是一个匹配。<br>{2, 4} 的含义是最少重复 2 次、最多重复 4 次。<br>{3，} 表示至少重复 3 次。</p><h3 id="懒惰型元字符"><a href="#懒惰型元字符" class="headerlink" title="懒惰型元字符"></a>懒惰型元字符</h3><p>正则：<code><[Bb]>.*</[Bb]></code></p><p>This offer is not available to customers living in <code><B>AK</B> and <B>HI</B></code>.</p><p>因为 * 和 + 都是所谓的“贪婪型”元字符，它们在进行匹配时的行为模式是多多益善而不是适可而止的，会尽可能地从一段文本的开头一直匹配到这段文本的末尾。<br>在不需要这种“贪婪行为”的时候该怎么办？答案是使用这些元字符的“懒惰型”版本。懒惰型元字符只要给贪婪性元字符加上一个 ? 后缀即可。</p><table><thead><tr><th>贪婪型元字符</th><th>懒惰型元字符</th></tr></thead><tbody><tr><td>*</td><td>*?</td></tr><tr><td>+</td><td>+?</td></tr><tr><td>{n, }</td><td>{n, }?</td></tr></tbody></table><p>正则：<code><[Bb]>.*?</[Bb]></code></p><p>This offer is not available to customers living in <code><B>AK</B></code> and <code><B>HI</B></code>.</p><h2 id="位置匹配"><a href="#位置匹配" class="headerlink" title="位置匹配"></a>位置匹配</h2><h3 id="单词边界"><a href="#单词边界" class="headerlink" title="单词边界"></a>单词边界</h3><p>\b 用来匹配一个单词的开始或结尾。<br>简单地说，\b 匹配的是一个这样的位置，这个位置位于一个能够用户构成单词的字符（字母、数字和下划线，也就是与 \w 相匹配的字符）和一个不能用来构成单词的字符（也就是与 \W 相匹配的字符）之间。<br>正则：<code>\bcat\b</code></p><p>The <code>cat</code> scattered his food all over the room.</p><h3 id="字符串边界"><a href="#字符串边界" class="headerlink" title="字符串边界"></a>字符串边界</h3><p>^ 用来定义字符串开头，$ 用来定义字符串结尾。<br>注意：不同语言对 ^ $ 的处理有所不同。比如 Javascript 是按照以上定义实现的，而 Ruby 中 ^ 匹配一行的开始，$ 匹配一行的结束。具体可参考 <a href="http://www.regular-expressions.info/anchors.html。" target="_blank" rel="noopener">http://www.regular-expressions.info/anchors.html。</a></p><h2 id="使用子表达式"><a href="#使用子表达式" class="headerlink" title="使用子表达式"></a>使用子表达式</h2><p>子表达式是一个更大的表达式的一部分，把一个表达式划分为一系列子表达式的目的是为了把那些子表达式当做一个独立元素来使用。子表达式必须用 ( 和 ) 括起来。<br>正则：<code>(19|20)\d{2}</code></p><p>ID: 042<br>SEX: M<br>DOB: <code>1967</code>-08-17<br>Status: Active</p><p>| 字符表示或操作符，19|20 将匹配数字序列 19 或 20。</p><h2 id="回溯引用：前后一致匹配"><a href="#回溯引用：前后一致匹配" class="headerlink" title="回溯引用：前后一致匹配"></a>回溯引用：前后一致匹配</h2><p>正则：<code>[ ]+(\w+)[ ]+\1</code></p><p>This is a block <code>of of</code> text, several words here <code>are are</code> repeated, <code>and and</code> they should not be.</p><p>\1 是一个回溯引用，而它引用的正是前面划分出来的那个子表达式。当 (\w+) 匹配到单词 of 的时候，\1 也匹配单词 of；当 (\w+) 匹配到单词 and 的时候，\1 也匹配单词 and。<br>为了方便理解，可以把回溯引用想象成变量。</p><h2 id="前后查找"><a href="#前后查找" class="headerlink" title="前后查找"></a>前后查找</h2><p>前后查找包含的匹配本身并不返回，而是用于确定正确的匹配位置，它并不是匹配结果的一部分。</p><h3 id="向前查找"><a href="#向前查找" class="headerlink" title="向前查找"></a>向前查找</h3><p>从语法上看，一个向前查找模式其实就是一个以 ?= 开头的子表达式，需要匹配的文本跟在 = 后面。</p><p>正则：<code>.+(?=:)</code></p><p><code>http</code>://<a href="http://www.forta.com/" target="_blank" rel="noopener">www.forta.com/</a><br><code>https</code>://mail.forta.com/<br><code>ftp</code>://ftp.forta.com/</p><p>与子表达的对比。<br>正则：<code>.+(:)</code></p><p><code>http:</code>//<a href="http://www.forta.com/" target="_blank" rel="noopener">www.forta.com/</a><br><code>https:</code>//mail.forta.com/<br><code>ftp:</code>//ftp.forta.com/</p><h3 id="向后查找"><a href="#向后查找" class="headerlink" title="向后查找"></a>向后查找</h3><p>向后查找，以 ?<= 开头的子表达式。<br>正则： <code>(?=\$)[0-9.]+</code></p><p>ABC01: $<code>23.45</code><br>Total items found: 4</p><h3 id="对前后查找取非"><a href="#对前后查找取非" class="headerlink" title="对前后查找取非"></a>对前后查找取非</h3><p>负向前查找讲向前查找不与给定模式相匹配的文本，负向后查找将向后查找不与给定模式相匹配的文本。</p><table><thead><tr><th>操作符</th><th>说明</th><th>示例</th></tr></thead><tbody><tr><td>(?=)</td><td>正向前查找</td><td>\d+(?= dollars) matches 100 in “100 dollars”</td></tr><tr><td>(?!)</td><td>负向前查找</td><td>d+(?! dollars) matches 100 if it is NOT followed by the word “dollars”</td></tr><tr><td>(?<=)</td><td>正向后查找</td><td>(?<=lucky )\d matches 7 in “lucky 7”</td></tr><tr><td>(?<!)</td><td>负向后查找</td><td>(?<!furious )\d matches 7 in “lucky 7”</td></tr></tbody></table><h2 id="嵌入条件"><a href="#嵌入条件" class="headerlink" title="嵌入条件"></a>嵌入条件</h2><p>(?(backreference)true-regex)：? 表示这是一个条件，括号里的 backreference 是一个回溯引用， backreference 存在时会执行true-regex。<br>(?(backreference)true-regex|false-regex)： backreference 不存在时会执行 false-regex。</p><h3 id="回溯引用条件"><a href="#回溯引用条件" class="headerlink" title="回溯引用条件"></a>回溯引用条件</h3><p>正则：<code>(\()?\d{3}(?(1)\)|-)\d{3}-\d{4}</code></p><p><code>123-456-7890</code><br><code>(123)456-7890</code><br>(123)-456-7890<br>(123-456-7890<br>1234567890<br>123 456 7890</p><p><code>(\()?</code> 匹配一个可选的左括号,<code>(?(1)\)|-)</code> 将根据条件是否满足而去匹配 ) 或者 -。如果 (1) 存在，<code>\)</code> 必须被匹配，否则 - 必须被匹配。</p><h3 id="前后查找条件"><a href="#前后查找条件" class="headerlink" title="前后查找条件"></a>前后查找条件</h3><p>前后查找条件的语法与回溯引用条件的语法大同小异，只需把回溯引用替换为一个完整的前后查找表达式就行了。<br>正则：<code>\d{5}(?(?=-)-\d{4})</code></p><p>11111<br>22222<br>33333-<br>44444-4444</p><p>(?(?=-)-\d{4}) 使用了 ?=- 来匹配（但不消费）一个连字符，如果条件得到满足（那个连字符存在），-\d{4} 将匹配那个连字符和随后的 4 位数字。</p>]]></content>
<summary type="html">
<h2 id="匹配单个字符"><a href="#匹配单个字符" class="headerlink" title="匹配单个字符"></a>匹配单个字符</h2><h3 id="匹配纯文本"><a href="#匹配纯文本" class="headerlink" title="匹配纯文本"></a>匹配纯文本</h3><p>正则：<code>Ben</code></p>
<p>Hello, my name is <code>Ben</code>. Please visit my website at <a href="http://www.forta.com/" target="_blank" rel="noopener">http://www.forta.com/</a>.</p>
<h3 id="匹配任意字符"><a href="#匹配任意字符" class="headerlink" title="匹配任意字符"></a>匹配任意字符</h3><p>. 匹配任意单个字符。在绝大多数的正则表达式实现里，. 只能匹配除换行符以外的任何单个字符。<br>正则：<code>sales.</code></p>
<p><code>sales1</code>.xls<br>orders3.xls<br><code>sales2</code>.xls<br><code>sales3</code>.xls<br>apac1.xls<br>europe2.xls<br>na1.xls<br>na2.xls<br>sa1.xls</p>
</summary>
<category term="读书笔记" scheme="http://kaywu.xyz/categories/%E8%AF%BB%E4%B9%A6%E7%AC%94%E8%AE%B0/"/>
<category term="读书笔记" scheme="http://kaywu.xyz/tags/%E8%AF%BB%E4%B9%A6%E7%AC%94%E8%AE%B0/"/>
</entry>
<entry>
<title>Rails 多台服务器上 js 不一致的问题</title>
<link href="http://kaywu.xyz/2017/05/14/rails-js-compress/"/>
<id>http://kaywu.xyz/2017/05/14/rails-js-compress/</id>
<published>2017-05-14T10:00:27.000Z</published>
<updated>2017-09-10T08:13:51.000Z</updated>
<content type="html"><![CDATA[<h3 id="发现问题"><a href="#发现问题" class="headerlink" title="发现问题"></a>发现问题</h3><p>周二发布正式的时候，发生了匪夷所思的事情。<br>打开发布的网页，有一定的几率操作会没有反应。但在测试的时候却从没有这个问题发生。通过调试工具查看网页，发现一半的网页会加载 application-b1a 开头的文件，而另一半会加载 application-745 开头的文件。加载 applicaton-745 时会报 404，使得操作没有反应。</p><p>这里简单地说下发布的情况。该服务会发布到 A、B 两台服务器上，nginx 接收请求并转发到 A、B 上的实例。<br>查看正式服务器上的文件发现，A 上存在 application-b1a 文件，B 上存在 application-745 文件。看来 nginx 会查找 服务器 A 上的文件，由于找不到 application-745 而返回 404。先把 B 上的 applicaiton-b1a 复制到 A，临时修复这个问题。</p><h3 id="排查原因"><a href="#排查原因" class="headerlink" title="排查原因"></a>排查原因</h3><p>为什么两个服务器上生成的 js 文件名会不一致？我们先简单地回顾下 Asset Pipeline 生成 js 的过程。正式环境上， Asset Pipeline 会预编译文件，生成类似于 <code>application-908e25f4bf641868d8683022a5b62f54.js</code> 的文件。其中 908e 这串表示摘要，是根据文件的内容生成的 MD5，当 js 文件发生改变时生成的摘要也会发生变化。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">/<span class="regexp">/ 引用 js</span></span><br><span class="line"><span class="regexp"><%= javascript_include_tag "application" %></span></span><br><span class="line"><span class="regexp"></span></span><br><span class="line"><span class="regexp">/</span><span class="regexp">/ 生成的 html</span></span><br><span class="line"><span class="regexp"><script src="/assets</span><span class="regexp">/application-908e25f4bf641868d8683022a5b62f54.js"></script</span>></span><br></pre></td></tr></table></figure></p><a id="more"></a><p>换句话说，若 js 内容一样，生成的摘要也应该是一样的。不应该啊！两个服务器部署的代码应该是一致的。为了保险起见，我还特定查看了相关文件的 MD5 值，完全相同。</p><p>输入是相同的，而产生的结果却不一样，问题应该出在处理步骤上。于是我查看了正式环境的配置，发现了以下这条。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">config.assets.js_compressor = <span class="symbol">:uglifier</span></span><br></pre></td></tr></table></figure></p><p>uglifier 是用 ruby 封装 UglifyJS 的 gem，而 UglifyJS 是依赖 node.js 对 js 进行压缩的。查看了下两台服务器的 node 版本，A 是 0.10，B 是 6.10。问题的原因终于找到了。node 的版本不一致，导致其压缩的结果不一样，使得生成的最终文件也不相同。</p><h3 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h3><p>先在 A 上升级了 node 版本，使得 A、B 两台服务器的 node 版本一致。然后删除 <code>tmp/cache/assets/sprockets</code> 下的缓存，再执行 <code>RAILS_ENV=production bin/rake assets:precompile</code>，最后重新发布，使最新生成的结果能被实例加载。<br>注意这里必须先删除缓存，不然执行 precompile 时不会生成最新的结果。</p><h3 id="反思"><a href="#反思" class="headerlink" title="反思"></a>反思</h3><p>为什么这个问题会突然出现呢？原来是 B 上的 node 版本由于其他应用的需求进行了升级，使得 A、B 两台版本不一致了。<br>这种一个馒头引发的血案防不胜防，可见应用之间依赖的隔离是多么重要。联想到近几年类似 docker 的解决方案大受欢迎也就不足为怪了。</p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="http://guides.rubyonrails.org/asset_pipeline.html" target="_blank" rel="noopener">Asset Pipeline</a></li></ul>]]></content>
<summary type="html">
<h3 id="发现问题"><a href="#发现问题" class="headerlink" title="发现问题"></a>发现问题</h3><p>周二发布正式的时候，发生了匪夷所思的事情。<br>打开发布的网页，有一定的几率操作会没有反应。但在测试的时候却从没有这个问题发生。通过调试工具查看网页，发现一半的网页会加载 application-b1a 开头的文件，而另一半会加载 application-745 开头的文件。加载 applicaton-745 时会报 404，使得操作没有反应。</p>
<p>这里简单地说下发布的情况。该服务会发布到 A、B 两台服务器上，nginx 接收请求并转发到 A、B 上的实例。<br>查看正式服务器上的文件发现，A 上存在 application-b1a 文件，B 上存在 application-745 文件。看来 nginx 会查找 服务器 A 上的文件，由于找不到 application-745 而返回 404。先把 B 上的 applicaiton-b1a 复制到 A，临时修复这个问题。</p>
<h3 id="排查原因"><a href="#排查原因" class="headerlink" title="排查原因"></a>排查原因</h3><p>为什么两个服务器上生成的 js 文件名会不一致？我们先简单地回顾下 Asset Pipeline 生成 js 的过程。正式环境上， Asset Pipeline 会预编译文件，生成类似于 <code>application-908e25f4bf641868d8683022a5b62f54.js</code> 的文件。其中 908e 这串表示摘要，是根据文件的内容生成的 MD5，当 js 文件发生改变时生成的摘要也会发生变化。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">/<span class="regexp">/ 引用 js</span></span><br><span class="line"><span class="regexp">&lt;%= javascript_include_tag "application" %&gt;</span></span><br><span class="line"><span class="regexp"></span></span><br><span class="line"><span class="regexp">/</span><span class="regexp">/ 生成的 html</span></span><br><span class="line"><span class="regexp">&lt;script src="/assets</span><span class="regexp">/application-908e25f4bf641868d8683022a5b62f54.js"&gt;&lt;/script</span>&gt;</span><br></pre></td></tr></table></figure></p>
</summary>
<category term="Rails" scheme="http://kaywu.xyz/categories/Rails/"/>
<category term="Rails" scheme="http://kaywu.xyz/tags/Rails/"/>
</entry>
<entry>
<title>Rails 如何在开发模式下重新加载源代码</title>
<link href="http://kaywu.xyz/2017/05/01/rails-dev-reload/"/>
<id>http://kaywu.xyz/2017/05/01/rails-dev-reload/</id>
<published>2017-05-01T10:04:50.000Z</published>
<updated>2017-09-10T08:13:51.000Z</updated>
<content type="html"><![CDATA[<p>在 Rails development 环境下，若更改了部分代码，只需要重新发起请求，就能看到最新代码的结果，不需要重启服务器。<br>下文简述 Rails 是如何做到这点的。</p><p>在 <code>config/development.rb</code>，也就是 development 的环境配置文件，我们可以看到不同于其他环境的一行：<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># In the development environment your application's code is reloaded on</span></span><br><span class="line"><span class="comment"># every request. This slows down response time but is perfect for development</span></span><br><span class="line"><span class="comment"># since you don't have to restart the webserver when you make code changes.</span></span><br><span class="line">config.cache_classes = <span class="literal">false</span></span><br></pre></td></tr></table></figure></p><p>正如注释所说的，它会使得 Rails 在每次接收请求时都重新加载源代码。</p><h3 id="初始化"><a href="#初始化" class="headerlink" title="初始化"></a>初始化</h3><p>我们先从 Rails 的初始化说起，以 Rails 4.2.6 为例。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># rails/application/default_middleware_stack.rb</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">build_stack</span></span></span><br><span class="line"> ...</span><br><span class="line"> <span class="keyword">unless</span> config.cache_classes</span><br><span class="line"> middleware.use <span class="symbol">:</span><span class="symbol">:ActionDispatch</span><span class="symbol">:</span><span class="symbol">:Reloader</span>, lambda { reload_dependencies? }</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> ...</span><br><span class="line"> private</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">reload_dependencies?</span></span></span><br><span class="line"> config.reload_classes_only_on_change != <span class="literal">true</span> <span class="params">||</span> app.reloaders.map(&<span class="symbol">:updated?</span>).any?</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>当 config.cache_classses 为 true 时，middleware 会增加 <code>ActionDispatch::Reloader</code>，而这正是重新加载源代码的关键。<br>ps： middleware 可以通过 <code>rake middleware</code> 来查看。</p><h3 id="处理请求"><a href="#处理请求" class="headerlink" title="处理请求"></a>处理请求</h3><p>当服务器接收到请求时，中间件 <code>ActionDispatch::Reloader</code> 使用回调 prepare、cleanup 来实现重载源代码。其中 prepare 在处理请求前被调用，cleanup 在处理请求后被调用。</p><a id="more"></a><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ActionDispatch::Reloader</span></span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">initialize</span><span class="params">(app, condition=<span class="literal">nil</span>)</span></span></span><br><span class="line"> @app = app</span><br><span class="line"> @condition = condition <span class="params">||</span> lambda { <span class="literal">true</span> }</span><br><span class="line"> @validated = <span class="literal">true</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">call</span><span class="params">(env)</span></span></span><br><span class="line"> @validated = @condition.call</span><br><span class="line"> prepare!</span><br><span class="line"></span><br><span class="line"> response = @app.call(env)</span><br><span class="line"> response[<span class="number">2</span>] = <span class="symbol">:</span><span class="symbol">:Rack</span><span class="symbol">:</span><span class="symbol">:BodyProxy</span>.new(response[<span class="number">2</span>]) { cleanup! }</span><br><span class="line"></span><br><span class="line"> response</span><br><span class="line"> <span class="keyword">rescue</span> Exception</span><br><span class="line"> cleanup!</span><br><span class="line"> raise</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">prepare!</span> <span class="comment">#:nodoc:</span></span></span><br><span class="line"> run_callbacks <span class="symbol">:prepare</span> <span class="keyword">if</span> validated?</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">cleanup!</span> <span class="comment">#:nodoc:</span></span></span><br><span class="line"> run_callbacks <span class="symbol">:cleanup</span> <span class="keyword">if</span> validated?</span><br><span class="line"> <span class="keyword">ensure</span></span><br><span class="line"> @validated = <span class="literal">true</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> private</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">validated?</span></span></span><br><span class="line"> @validated</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>当 @validated 为 true 时会执行 prepare、cleanup 等回调。那么 @validated 的值是怎么得到的？<br>结合初始化的代码我们发现，@condition 其实就是创建 <code>ActionDispatch::Reloader</code> 时的参数 <code>lambda { reload_dependencies? }</code>。而 reload_dependencies? 具体代码如下。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">reload_dependencies?</span></span></span><br><span class="line"> config.reload_classes_only_on_change != <span class="literal">true</span> <span class="params">||</span> app.reloaders.map(&<span class="symbol">:updated?</span>).any?</span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>可以通过以下几种情况来分析<br>若 config.reload_clases_only_on_change 为 false，会执行回调。<br>若 config.reload_clases_only_on_change 为 true 且代码发生了变动（任一 reloader 调用 updated? 返回 true），会执行回调。<br>若 config.reload_clases_only_on_change 为 true 且代码未发生变动，不会执行回调。</p><h3 id="回调"><a href="#回调" class="headerlink" title="回调"></a>回调</h3><p>弄清楚了回调调用的时机，我们来继续研究回调的内容是什么。</p><p><code>Rails::Application::Finisher</code> 负责结束 Rails 的初始化，它会给 <code>ActionDispatch::Reloader</code> 增加回调。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># rails/application/finisher.rb</span></span><br><span class="line">initializer <span class="symbol">:set_clear_dependencies_hook</span>, <span class="symbol">group:</span> <span class="symbol">:all</span> <span class="keyword">do</span></span><br><span class="line"> callback = lambda <span class="keyword">do</span></span><br><span class="line"> ActiveSupport::DescendantsTracker.clear</span><br><span class="line"> ActiveSupport::Dependencies.clear</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">if</span> config.reload_classes_only_on_change</span><br><span class="line"> reloader = config.file_watcher.new(*watchable_args, &callback)</span><br><span class="line"> <span class="keyword">self</span>.reloaders << reloader</span><br><span class="line"> <span class="comment"># Prepend this callback to have autoloaded constants cleared before</span></span><br><span class="line"> <span class="comment"># any other possible reloading, in case they need to autoload fresh</span></span><br><span class="line"> <span class="comment"># constants.</span></span><br><span class="line"> ActionDispatch::Reloader.to_prepare(<span class="symbol">prepend:</span> <span class="literal">true</span>) <span class="keyword">do</span></span><br><span class="line"> <span class="comment"># In addition to changes detected by the file watcher, if routes</span></span><br><span class="line"> <span class="comment"># or i18n have been updated we also need to clear constants,</span></span><br><span class="line"> <span class="comment"># that's why we run #execute rather than #execute_if_updated, this</span></span><br><span class="line"> <span class="comment"># callback has to clear autoloaded constants after any update.</span></span><br><span class="line"> reloader.execute</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> ActionDispatch::Reloader.to_cleanup(&callback)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>当 config.reload_clases_only_on_change 为 true，会向 <code>ActionDispatch::Reloader</code> 的 prepare 添加回调，而 false 时会向 cleanup 添加回调。<br>该回调会清理所有的依赖，更确切地说，使用内置的 remove_const 清除所有加载的常量。<br>由于所有常量都被移除，<code>ActiveSupport::Dependencies</code> 使用 const_missing 并再次加载相关类，从而使得修改后的代码被加载。</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>当 config.cache_class 为 false 时 middleware 会增加中间件 <code>ActionDispatch::Reloader</code>。</p><p>当 config.reload_clases_only_on_change 为 true 时，<code>Rails::Application::Finisher</code> 会在 <code>ActionDispatch::Reloader</code> 增加 <code>prepare</code> 的回调。<br>请求处理前 Reloader 会根据代码是否更新来执行 <code>prepare</code> 回调，执行回调后会清理所有的依赖，<code>ActiveSupport::Dependencies</code> 使用 const_missing 并再次加载相关类，从而使得修改后的代码被加载。</p><p>当 config.reload_clases_only_on_change 为 false 时， <code>Rails::Application::Finisher</code> 会在 <code>ActionDispatch::Reloader</code> 增加 <code>cleanup</code> 的回调。Reloader 会在每次请求处理后执行回调清理所有的依赖。其他步骤类似。</p><p>关于更详细的步骤说明可以参考 <a href="http://crypt.codemancers.com/posts/2013-10-03-rails-reloading-in-dev-mode/" target="_blank" rel="noopener">How rails reloads your source code in development mode?</a>，关于 Rails 的 autoload 机制可以参考 <a href="http://guides.rubyonrails.org/autoloading_and_reloading_constants.html" target="_blank" rel="noopener">Autoloading and Reloading Constants</a>。</p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="http://crypt.codemancers.com/posts/2013-10-03-rails-reloading-in-dev-mode/" target="_blank" rel="noopener">How rails reloads your source code in development mode?</a></li><li><a href="http://guides.rubyonrails.org/autoloading_and_reloading_constants.html" target="_blank" rel="noopener">Autoloading and Reloading Constants</a></li></ul>]]></content>
<summary type="html">
<p>在 Rails development 环境下，若更改了部分代码，只需要重新发起请求，就能看到最新代码的结果，不需要重启服务器。<br>下文简述 Rails 是如何做到这点的。</p>
<p>在 <code>config/development.rb</code>，也就是 development 的环境配置文件，我们可以看到不同于其他环境的一行：<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># In the development environment your application's code is reloaded on</span></span><br><span class="line"><span class="comment"># every request. This slows down response time but is perfect for development</span></span><br><span class="line"><span class="comment"># since you don't have to restart the webserver when you make code changes.</span></span><br><span class="line">config.cache_classes = <span class="literal">false</span></span><br></pre></td></tr></table></figure></p>
<p>正如注释所说的，它会使得 Rails 在每次接收请求时都重新加载源代码。</p>
<h3 id="初始化"><a href="#初始化" class="headerlink" title="初始化"></a>初始化</h3><p>我们先从 Rails 的初始化说起，以 Rails 4.2.6 为例。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># rails/application/default_middleware_stack.rb</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">build_stack</span></span></span><br><span class="line"> ...</span><br><span class="line"> <span class="keyword">unless</span> config.cache_classes</span><br><span class="line"> middleware.use <span class="symbol">:</span><span class="symbol">:ActionDispatch</span><span class="symbol">:</span><span class="symbol">:Reloader</span>, lambda &#123; reload_dependencies? &#125;</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> ...</span><br><span class="line"> private</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">reload_dependencies?</span></span></span><br><span class="line"> config.reload_classes_only_on_change != <span class="literal">true</span> <span class="params">||</span> app.reloaders.map(&amp;<span class="symbol">:updated?</span>).any?</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p>
<p>当 config.cache_classses 为 true 时，middleware 会增加 <code>ActionDispatch::Reloader</code>，而这正是重新加载源代码的关键。<br>ps： middleware 可以通过 <code>rake middleware</code> 来查看。</p>
<h3 id="处理请求"><a href="#处理请求" class="headerlink" title="处理请求"></a>处理请求</h3><p>当服务器接收到请求时，中间件 <code>ActionDispatch::Reloader</code> 使用回调 prepare、cleanup 来实现重载源代码。其中 prepare 在处理请求前被调用，cleanup 在处理请求后被调用。</p>
</summary>
<category term="Rails" scheme="http://kaywu.xyz/categories/Rails/"/>
<category term="Rails" scheme="http://kaywu.xyz/tags/Rails/"/>
</entry>
<entry>
<title>MySQL EXPLAIN 解读</title>
<link href="http://kaywu.xyz/2017/03/26/mysql-explain/"/>
<id>http://kaywu.xyz/2017/03/26/mysql-explain/</id>
<published>2017-03-26T09:55:35.000Z</published>
<updated>2017-09-10T08:13:51.000Z</updated>
<content type="html"><![CDATA[<p>EXPLAIN 解释了 MySQL 是如何执行 SQL 语句的。使用的方法很简单，在 SQL 语句前加上 <code>EXPLAIN</code> 关键字就可以。<br>下面是一个简单的例子，测试数据在文章末尾。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"># 例 1</span><br><span class="line">mysql> EXPLAIN SELECT name FROM users WHERE id = 1\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: users</span><br><span class="line"> type: const</span><br><span class="line">possible_keys: PRIMARY</span><br><span class="line"> key: PRIMARY</span><br><span class="line"> key_len: 4</span><br><span class="line"> ref: const</span><br><span class="line"> rows: 1</span><br><span class="line"> Extra: NULL</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure></p><p>EXPLAIN 列的解释：</p><ul><li>id：SELECT 标识符，下面具体分析</li><li>select_type: SELECT 类型，下面会具体分析</li><li>table: 查询所使用的表</li><li>type: JOIN 的类型，下面会具体分析</li><li>possible_keys: 可能使用的索引，但不一定会真正使用</li><li>key: 真正使用的索引</li><li>key_len: 所使用的索引长度</li><li>ref: 与索引比较的列</li><li>rows: 预估需要扫描的行数</li><li>Extra: 额外信息</li></ul><a id="more"></a><h3 id="id"><a href="#id" class="headerlink" title="id"></a>id</h3><p>个人理解表示了 SELECT 的执行顺序。id 大的优先执行，id 相同的从上往下执行。</p><h3 id="select-type"><a href="#select-type" class="headerlink" title="select_type"></a>select_type</h3><p>select_type 表示查询的类型，具体种类见官方图表。<br><img src="/img/select_type.png" alt="select_type.png-来自官方文档"><br>SIMPLE 是最常见的种类，表示它未使用 UNION 及子查询。例 1 的查询就属于 SIMPLE。</p><p>当使用了关键字 UNION，查询的类型就会发生变化。<br><img src="/img/select_union.png" alt="select_union.png-41.3kB"></p><p>在这个查询中，我们可以看到 3 种类型的查询。 PRIMARY 表示最外层的查询，也就是 UNION 之前的 <code>SELECT name FROM users WHERE id = 1</code>。UNION 之后的 <code>SELECT name FROM users WHERE id = 2</code> 归为 UNION 类型。最后 UNION RESULT 将两次查询的结果归总。</p><p>下面是其他查询类型的例子。</p><h4 id="PRIMARY-amp-SUBQUERY"><a href="#PRIMARY-amp-SUBQUERY" class="headerlink" title="PRIMARY & SUBQUERY"></a>PRIMARY & SUBQUERY</h4><p>PRIMARY 为最外层的查询，而 SUBQUERY 则指子查询。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">EXPLAIN SELECT * FROM users WHERE id = (SELECT user_id FROM orders WHERE id = 3);</span><br></pre></td></tr></table></figure></p><h4 id="PRIMARY-amp-DEPENDENT-SUBQUERY"><a href="#PRIMARY-amp-DEPENDENT-SUBQUERY" class="headerlink" title="PRIMARY & DEPENDENT SUBQUERY"></a>PRIMARY & DEPENDENT SUBQUERY</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">EXPLAIN SELECT * FROM users WHERE EXISTS (SELECT user_id FROM orders WHERE orders.id = 3 and orders.user_id = users.id);</span><br></pre></td></tr></table></figure><h4 id="MATERIALIZED"><a href="#MATERIALIZED" class="headerlink" title="MATERIALIZED"></a>MATERIALIZED</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">EXPLAIN SELECT DISTINCT user_id FROM orders WHERE id IN (SELECT DISTINCT order_id FROM order_items WHERE product_name = 'p1');</span><br></pre></td></tr></table></figure><h4 id="DERIVED"><a href="#DERIVED" class="headerlink" title="DERIVED"></a>DERIVED</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">EXPLAIN SELECT * FROM (SELECT * FROM orders WHERE id = 3) o;</span><br></pre></td></tr></table></figure><h3 id="type"><a href="#type" class="headerlink" title="type"></a>type</h3><p>type 表示 JOIN 的类型，是查询是否高效的重要依据。<br>效率从高到低排列为 system > const > eq_ref > ref > range > index > all。</p><ul><li>system: 表中只有一条数据，const 连接的特殊类型。</li><li><p>const: 主键或唯一索引的等值比较，由于表中至多有一条符合的数据，所以速度很快。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">mysql> EXPLAIN SELECT * FROM users WHERE id = 2\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: users</span><br><span class="line"> type: const</span><br><span class="line">possible_keys: PRIMARY</span><br><span class="line"> key: PRIMARY</span><br><span class="line"> key_len: 4</span><br><span class="line"> ref: const</span><br><span class="line"> rows: 1</span><br><span class="line"> Extra: NULL</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure></li><li><p>eq_ref: 上表的每一个行至多会匹配到该表的一行，是除 system 和 const 之外最高效的 join type。当索引为 PRIMARY KEY 或 UNIQUE NOT NULL 且被全部使用时会用到。常见于索引列的等值比较。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line">mysql> EXPLAIN SELECT * FROM users, orders WHERE orders.user_id = users.id\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: orders</span><br><span class="line"> type: index</span><br><span class="line">possible_keys: index_orders_on_user_id_and_price</span><br><span class="line"> key: index_orders_on_user_id_and_price</span><br><span class="line"> key_len: 8</span><br><span class="line"> ref: NULL</span><br><span class="line"> rows: 4</span><br><span class="line"> Extra: Using index</span><br><span class="line">*************************** 2. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: users</span><br><span class="line"> type: eq_ref</span><br><span class="line">possible_keys: PRIMARY</span><br><span class="line"> key: PRIMARY</span><br><span class="line"> key_len: 4</span><br><span class="line"> ref: explain_test.orders.user_id</span><br><span class="line"> rows: 1</span><br><span class="line"> Extra: NULL</span><br><span class="line">2 rows in set (0.00 sec)</span><br></pre></td></tr></table></figure></li><li><p>ref: 如果 join 不能根据键值只匹配一行时则会使用该 join type。常见于不是 UNIQUE 或 PRIMARY KEY的索引等值比较，或者是最左前缀规则的索引查询。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line">mysql> EXPLAIN SELECT orders.id FROM orders JOIN users ON orders.user_id = users.id WHERE users.name = 'Amy'\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: users</span><br><span class="line"> type: ref</span><br><span class="line">possible_keys: PRIMARY,index_users_on_names</span><br><span class="line"> key: index_users_on_names</span><br><span class="line"> key_len: 152</span><br><span class="line"> ref: const</span><br><span class="line"> rows: 1</span><br><span class="line"> Extra: Using where; Using index</span><br><span class="line">*************************** 2. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: orders</span><br><span class="line"> type: ref</span><br><span class="line">possible_keys: index_orders_on_user_id_and_price</span><br><span class="line"> key: index_orders_on_user_id_and_price</span><br><span class="line"> key_len: 4</span><br><span class="line"> ref: explain_test.users.id</span><br><span class="line"> rows: 1</span><br><span class="line"> Extra: Using index</span><br><span class="line">2 rows in set (0.00 sec)</span><br></pre></td></tr></table></figure></li><li><p>range: 使用索引进行范围查询，输出的 key 字段表示使用哪个索引，key_len 表示所使用索引中最长的索引长度。注意此类型下 ref 字段为 NULL。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">mysql> EXPLAIN SELECT * FROM users WHERE users.id IN (2, 3)\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: users</span><br><span class="line"> type: range</span><br><span class="line">possible_keys: PRIMARY</span><br><span class="line"> key: PRIMARY</span><br><span class="line"> key_len: 4</span><br><span class="line"> ref: NULL</span><br><span class="line"> rows: 2</span><br><span class="line"> Extra: Using where</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure></li><li><p>index: 全索引扫描, index 类型仅仅扫描所有的索引, 而不扫描数据。一般两种情况会出现。<br>一种是出现在所要查询的数据直接在索引树中就可以获取, 此时 Extra 字段会显示 Using index。另一种是全表扫描时按索引的顺序查找数据，此时 Extra 字段不会显示 Using index。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">mysql> EXPLAIN SELECT users.name FROM users\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: users</span><br><span class="line"> type: index</span><br><span class="line">possible_keys: NULL</span><br><span class="line"> key: index_users_on_names</span><br><span class="line"> key_len: 152</span><br><span class="line"> ref: NULL</span><br><span class="line"> rows: 10</span><br><span class="line"> Extra: Using index</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure></li><li><p>ALL: 全表扫描。</p></li></ul><h3 id="Extra"><a href="#Extra" class="headerlink" title="Extra"></a>Extra</h3><p>Extra 字段提供了关于查询的额外信息，种类很多，具体可以看<a href="https://dev.mysql.com/doc/refman/5.7/en/explain-output.html#explain-extra-information" target="_blank" rel="noopener">官方文档</a>。<br>除了上文提到的 Using index，这里再额外说两种。<br>Using filesort 表示 MySQL 需要遍历所有符合条件的行然后按照排序的 key 来使得最终的查询结果是有序的。<br>Using temporary 表示 MySQL 需要创建一个临时表来存储结果，通常发生在查询包含不同列的 GROUP BY 和 ORDER BY 子句。<br>看到这两者时，可以考虑对查询进行优化。</p><h3 id="使用的测试数据"><a href="#使用的测试数据" class="headerlink" title="使用的测试数据"></a>使用的测试数据</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line">CREATE TABLE `users` (</span><br><span class="line"> `id` int(11) NOT NULL AUTO_INCREMENT,</span><br><span class="line"> `name` varchar(50) NOT NULL,</span><br><span class="line"> PRIMARY KEY (`id`),</span><br><span class="line"> KEY `index_users_on_names` (`name`)</span><br><span class="line">) ENGINE=InnoDB DEFAULT CHARSET=utf8;</span><br><span class="line"></span><br><span class="line">INSERT INTO users(name) VALUES ('Amy');</span><br><span class="line">INSERT INTO users(name) VALUES ('Bob');</span><br><span class="line">INSERT INTO users(name) VALUES ('Cindy');</span><br><span class="line">INSERT INTO users(name) VALUES ('Duke');</span><br><span class="line">INSERT INTO users(name) VALUES ('Kay');</span><br><span class="line">INSERT INTO users(name) VALUES ('Lucy');</span><br><span class="line">INSERT INTO users(name) VALUES ('Mike');</span><br><span class="line">INSERT INTO users(name) VALUES ('Nancy');</span><br><span class="line">INSERT INTO users(name) VALUES ('Ted');</span><br><span class="line">INSERT INTO users(name) VALUES ('Van');</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">CREATE TABLE `orders` (</span><br><span class="line"> `id` int(11) NOT NULL AUTO_INCREMENT,</span><br><span class="line"> `user_id` int(11) NOT NULL,</span><br><span class="line"> `price` decimal(8,2) NOT NULL,</span><br><span class="line"> PRIMARY KEY (`id`),</span><br><span class="line"> KEY `index_orders_on_user_id_and_price` (`user_id`,`price`)</span><br><span class="line">) ENGINE=InnoDB DEFAULT CHARSET=utf8;</span><br><span class="line">INSERT INTO orders(user_id, price) VALUES (1, 80);</span><br><span class="line">INSERT INTO orders(user_id, price) VALUES (1, 100);</span><br><span class="line">INSERT INTO orders(user_id, price) VALUES (2, 90);</span><br><span class="line">INSERT INTO orders(user_id, price) VALUES (2, 120);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">CREATE TABLE `order_items` (</span><br><span class="line"> `id` int(11) NOT NULL AUTO_INCREMENT,</span><br><span class="line"> `order_id` int(11) NOT NULL,</span><br><span class="line"> `product_name` varchar(50) NOT NULL,</span><br><span class="line"> `quantity` int(11) NOT NULL,</span><br><span class="line"> PRIMARY KEY (`id`),</span><br><span class="line"> KEY `index_order_items_on_order_id` (`order_id`),</span><br><span class="line"> KEY `index_order_items_on_product_name` (`product_name`)</span><br><span class="line">) ENGINE=InnoDB DEFAULT CHARSET=utf8;</span><br><span class="line">INSERT INTO order_items(order_id, product_name, quantity) VALUES (1, 'p1', 1);</span><br><span class="line">INSERT INTO order_items(order_id, product_name, quantity) VALUES (1, 'p2', 2);</span><br><span class="line">INSERT INTO order_items(order_id, product_name, quantity) VALUES (2, 'p3', 1);</span><br><span class="line">INSERT INTO order_items(order_id, product_name, quantity) VALUES (2, 'p4', 1);</span><br><span class="line">INSERT INTO order_items(order_id, product_name, quantity) VALUES (3, 'p5', 3);</span><br><span class="line">INSERT INTO order_items(order_id, product_name, quantity) VALUES (4, 'p6', 2);</span><br></pre></td></tr></table></figure><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="https://dev.mysql.com/doc/refman/5.7/en/explain-output.html" target="_blank" rel="noopener">EXPLAIN Output Format</a></li><li><a href="https://segmentfault.com/a/1190000008131735" target="_blank" rel="noopener">MySQL 性能优化神器 Explain 使用分析</a></li><li><a href="http://www.cnitblog.com/aliyiyi08/archive/2008/09/09/48878.html" target="_blank" rel="noopener">Mysql Explain 详解</a></li></ul>]]></content>
<summary type="html">
<p>EXPLAIN 解释了 MySQL 是如何执行 SQL 语句的。使用的方法很简单，在 SQL 语句前加上 <code>EXPLAIN</code> 关键字就可以。<br>下面是一个简单的例子，测试数据在文章末尾。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"># 例 1</span><br><span class="line">mysql&gt; EXPLAIN SELECT name FROM users WHERE id = 1\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> id: 1</span><br><span class="line"> select_type: SIMPLE</span><br><span class="line"> table: users</span><br><span class="line"> type: const</span><br><span class="line">possible_keys: PRIMARY</span><br><span class="line"> key: PRIMARY</span><br><span class="line"> key_len: 4</span><br><span class="line"> ref: const</span><br><span class="line"> rows: 1</span><br><span class="line"> Extra: NULL</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure></p>
<p>EXPLAIN 列的解释：</p>
<ul>
<li>id：SELECT 标识符，下面具体分析</li>
<li>select_type: SELECT 类型，下面会具体分析</li>
<li>table: 查询所使用的表</li>
<li>type: JOIN 的类型，下面会具体分析</li>
<li>possible_keys: 可能使用的索引，但不一定会真正使用</li>
<li>key: 真正使用的索引</li>
<li>key_len: 所使用的索引长度</li>
<li>ref: 与索引比较的列</li>
<li>rows: 预估需要扫描的行数</li>
<li>Extra: 额外信息</li>
</ul>
</summary>
<category term="MySQL" scheme="http://kaywu.xyz/categories/MySQL/"/>
<category term="MySQL" scheme="http://kaywu.xyz/tags/MySQL/"/>
</entry>
<entry>
<title>Rails Concern 源码研究</title>
<link href="http://kaywu.xyz/2017/03/19/rails-concern/"/>
<id>http://kaywu.xyz/2017/03/19/rails-concern/</id>
<published>2017-03-19T10:07:19.000Z</published>
<updated>2018-06-11T16:43:56.000Z</updated>
<content type="html"><![CDATA[<p>ActiveSupport::Concern 是为了更方便地 include 模块而推出的工具类。</p><h3 id="使用方法"><a href="#使用方法" class="headerlink" title="使用方法"></a>使用方法</h3><p>首先来看下它的使用方法。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 传统的 Module 引入</span></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">M</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">included</span><span class="params">(base)</span></span></span><br><span class="line"> base.extend ClassMethods</span><br><span class="line"> base.class_eval <span class="keyword">do</span></span><br><span class="line"> scope <span class="symbol">:disabled</span>, -> { where(<span class="symbol">disabled:</span> <span class="literal">true</span>) }</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">ClassMethods</span></span></span><br><span class="line"> ...</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 使用 Concern</span></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">M</span></span></span><br><span class="line"> extend ActiveSupport::Concern</span><br><span class="line"></span><br><span class="line"> included <span class="keyword">do</span></span><br><span class="line"> scope <span class="symbol">:disabled</span>, -> { where(<span class="symbol">disabled:</span> <span class="literal">true</span>) }</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> class_methods <span class="keyword">do</span></span><br><span class="line"> ...</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>可见，通过 included 和 class_method 两个类方法使得 Module 的写法更加清晰。<br>你可能有些疑问，对比传统的引入，使用 Concern 虽然更加清晰了，但没什么巨大的优点。而传统的引入也可以通过将方法分成两个 module，如 InstanceMethods、ClassMethods，来达到同样的效果。<br>在这个简单的例子上，确实如此。但在一些嵌套的 include 上 Concern 的优势就体现出来了。</p><a id="more"></a><h3 id="嵌套的-include"><a href="#嵌套的-include" class="headerlink" title="嵌套的 include"></a>嵌套的 include</h3><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Foo</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">included</span><span class="params">(base)</span></span></span><br><span class="line"> base.class_eval <span class="keyword">do</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">method_injected_by_foo</span></span></span><br><span class="line"> ...</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Bar</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">included</span><span class="params">(base)</span></span></span><br><span class="line"> base.method_injected_by_foo</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Host</span></span></span><br><span class="line"> <span class="keyword">include</span> Foo <span class="comment"># We need to include this dependency for Bar</span></span><br><span class="line"> <span class="keyword">include</span> Bar <span class="comment"># Bar is the module that Host really needs</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>我们需要在 Host 中引入 Bar，但由于 Bar 又需要 Foo，导致必须在 Host 中引入 Foo。也就是说，当我们 include module 时也必须把它的依赖同时 include 进来。这将随着依赖关系的复杂而变得艰难。<br>为什么不让 Bar 来负责自己的依赖呢？如以下的代码。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Bar</span></span></span><br><span class="line"> <span class="keyword">include</span> Foo</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">included</span><span class="params">(base)</span></span></span><br><span class="line"> base.method_injected_by_foo</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Host</span></span></span><br><span class="line"> <span class="keyword">include</span> Bar</span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>然而愿望很美好，实际运行时会报错，让我们来分析下流程。<br>当 Bar include Foo 时，Foo.included 方法被回调，而此时的 base 为 Bar。也就是说，method_injected_by_foo 会被添加到 Bar 上而不是 Host。当 Host include Bar 时，Bar.included 会调用 Host.method_injected_by_foo，而 Host 上没有相关方法，导致报错。</p><p>但是使用 Concern 就可以完美地解决这个问题。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">'active_support/concern'</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Foo</span></span></span><br><span class="line"> extend ActiveSupport::Concern</span><br><span class="line"> included <span class="keyword">do</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">method_injected_by_foo</span></span></span><br><span class="line"> ...</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Bar</span></span></span><br><span class="line"> extend ActiveSupport::Concern</span><br><span class="line"> <span class="keyword">include</span> Foo</span><br><span class="line"></span><br><span class="line"> included <span class="keyword">do</span></span><br><span class="line"> <span class="keyword">self</span>.method_injected_by_foo</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Host</span></span></span><br><span class="line"> <span class="keyword">include</span> Bar <span class="comment"># It works, now Bar takes care of its dependencies</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><h3 id="源码分析"><a href="#源码分析" class="headerlink" title="源码分析"></a>源码分析</h3><p>Concern 的源码很短，不过 40 多行，但涉及到不少元编程的知识，要看明白得花点功夫。<br>复习下基本的知识，A extend B，B.extended(base) 会被回调，参数 base 为 A。A include B，B#included(base)、B#append_features(base) 都会被回调，参数 base 为 A。</p><p>接下来，以上文 Concern 代码为例子来说明下实现原理。<br>我们先从 <code>module Foo extend ActiveSupport::Concern</code> 开始，此时 Concern.extended 会被回调，初始化 Foo 类实例变量 @_dependencies。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">extended</span><span class="params">(base)</span></span> <span class="comment">#:nodoc:</span></span><br><span class="line"> base.instance_variable_set(<span class="symbol">:</span>@_dependencies, [])</span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>之后第 5 行调用 included 时，会将 @_included_block 设置为传入的 block。<br>注意这里的 included 是显式调用的，而不是被回调的，参数 base 为 nil。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">included</span><span class="params">(base = <span class="literal">nil</span>, &block)</span></span></span><br><span class="line"> <span class="keyword">if</span> base.<span class="literal">nil</span>?</span><br><span class="line"> raise MultipleIncludedBlocks <span class="keyword">if</span> instance_variable_defined?(<span class="symbol">:</span>@_included_block)</span><br><span class="line"></span><br><span class="line"> @_included_block = block</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="keyword">super</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p>第 13 行 <code>module Bar extend ActiveSupport::Concern</code> 与 Foo extend ActiveSupport::Concern 同理。<br>第 14 行 <code>include Foo</code>，使得 Foo 的 append_features 和 included 被调用。included 由于 base 不为空只是简单地调用 super。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">append_features</span><span class="params">(base)</span></span></span><br><span class="line"> <span class="keyword">if</span> base.instance_variable_defined?(<span class="symbol">:</span>@_dependencies)</span><br><span class="line"> base.instance_variable_get(<span class="symbol">:</span>@_dependencies) << <span class="keyword">self</span></span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span> <span class="keyword">if</span> base < <span class="keyword">self</span></span><br><span class="line"> @_dependencies.each { <span class="params">|dep|</span> base.send(<span class="symbol">:include</span>, dep) }</span><br><span class="line"> <span class="keyword">super</span></span><br><span class="line"> base.extend const_get(<span class="symbol">:ClassMethods</span>) <span class="keyword">if</span> const_defined?(<span class="symbol">:ClassMethods</span>)</span><br><span class="line"> base.class_eval(&@_included_block) <span class="keyword">if</span> instance_variable_defined?(<span class="symbol">:</span>@_included_block)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p><p><code>append_features</code> 是 Concern 实现其 magic 最重要的部分。<br>由于 Bar 初始化了 @_dependencies，<code>base.instance_variable_get(:@_dependencies) << self</code> 会被运行，所以 Bar @_dependencies 就变为了 [Foo]。<br>Bar 调用 included 和 Foo 同理。</p><p>第 22 行，<code>Host include Bar</code> 使得 Bar 的 append_features 被回调。注意，重头戏来了。<br><code>@_dependencies.each { |dep| base.send(:include, dep) }</code> 会被执行，通过之前的分析 Bar.@_dependencies 为 [Foo]，所以也就是 <code>base.send(:include, Foo)</code>，这里的 base 为 Host。Host include Foo 会回调 Foo#append_features，此时 Host 会 extend Foo::ClassMethods 和 class_eval(&@_included_block)，从而实现了 Host 在 include Bar 时自动 include Foo。</p><p>简单来说，Bar include Foo 时并没有产生 include 原本的效果，而是把 Foo 添加到 Bar 的 @_dependencies 中。当 Host include Bar 时， Bar 中 @_dependencies 的依赖才一一生效。</p><p>最后附上 Concern 的源码。</p><h4 id="Concern-源码"><a href="#Concern-源码" class="headerlink" title="Concern 源码"></a>Concern 源码</h4><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">ActiveSupport</span></span></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">Concern</span></span></span><br><span class="line"> <span class="class"><span class="keyword">class</span> <span class="title">MultipleIncludedBlocks</span> < StandardError <span class="comment">#:nodoc:</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">initialize</span></span></span><br><span class="line"> <span class="keyword">super</span> <span class="string">"Cannot define multiple 'included' blocks for a Concern"</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">extended</span><span class="params">(base)</span></span> <span class="comment">#:nodoc:</span></span><br><span class="line"> base.instance_variable_set(<span class="symbol">:</span>@_dependencies, [])</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">append_features</span><span class="params">(base)</span></span></span><br><span class="line"> <span class="keyword">if</span> base.instance_variable_defined?(<span class="symbol">:</span>@_dependencies)</span><br><span class="line"> base.instance_variable_get(<span class="symbol">:</span>@_dependencies) << <span class="keyword">self</span></span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span> <span class="keyword">if</span> base < <span class="keyword">self</span></span><br><span class="line"> @_dependencies.each { <span class="params">|dep|</span> base.send(<span class="symbol">:include</span>, dep) }</span><br><span class="line"> <span class="keyword">super</span></span><br><span class="line"> base.extend const_get(<span class="symbol">:ClassMethods</span>) <span class="keyword">if</span> const_defined?(<span class="symbol">:ClassMethods</span>)</span><br><span class="line"> base.class_eval(&@_included_block) <span class="keyword">if</span> instance_variable_defined?(<span class="symbol">:</span>@_included_block)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">included</span><span class="params">(base = <span class="literal">nil</span>, &block)</span></span></span><br><span class="line"> <span class="keyword">if</span> base.<span class="literal">nil</span>?</span><br><span class="line"> raise MultipleIncludedBlocks <span class="keyword">if</span> instance_variable_defined?(<span class="symbol">:</span>@_included_block)</span><br><span class="line"></span><br><span class="line"> @_included_block = block</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="keyword">super</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">class_methods</span><span class="params">(&class_methods_module_definition)</span></span></span><br><span class="line"> mod = const_defined?(<span class="symbol">:ClassMethods</span>, <span class="literal">false</span>) ?</span><br><span class="line"> const_get(<span class="symbol">:ClassMethods</span>) <span class="symbol">:</span></span><br><span class="line"> const_set(<span class="symbol">:ClassMethods</span>, Module.new)</span><br><span class="line"></span><br><span class="line"> mod.module_eval(&class_methods_module_definition)</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="http://api.rubyonrails.org/classes/ActiveSupport/Concern.html" target="_blank" rel="noopener">ActiveSupport::Concern 源码</a></li><li><a href="http://elfxp.com/intro-of-concerns-in-rails/" target="_blank" rel="noopener">Rails 源码赏析之 Concern</a></li></ul>]]></content>
<summary type="html">
<p>ActiveSupport::Concern 是为了更方便地 include 模块而推出的工具类。</p>
<h3 id="使用方法"><a href="#使用方法" class="headerlink" title="使用方法"></a>使用方法</h3><p>首先来看下它的使用方法。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 传统的 Module 引入</span></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">M</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">included</span><span class="params">(base)</span></span></span><br><span class="line"> base.extend ClassMethods</span><br><span class="line"> base.class_eval <span class="keyword">do</span></span><br><span class="line"> scope <span class="symbol">:disabled</span>, -&gt; &#123; where(<span class="symbol">disabled:</span> <span class="literal">true</span>) &#125;</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">ClassMethods</span></span></span><br><span class="line"> ...</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 使用 Concern</span></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">M</span></span></span><br><span class="line"> extend ActiveSupport::Concern</span><br><span class="line"></span><br><span class="line"> included <span class="keyword">do</span></span><br><span class="line"> scope <span class="symbol">:disabled</span>, -&gt; &#123; where(<span class="symbol">disabled:</span> <span class="literal">true</span>) &#125;</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> class_methods <span class="keyword">do</span></span><br><span class="line"> ...</span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure></p>
<p>可见，通过 included 和 class_method 两个类方法使得 Module 的写法更加清晰。<br>你可能有些疑问，对比传统的引入，使用 Concern 虽然更加清晰了，但没什么巨大的优点。而传统的引入也可以通过将方法分成两个 module，如 InstanceMethods、ClassMethods，来达到同样的效果。<br>在这个简单的例子上，确实如此。但在一些嵌套的 include 上 Concern 的优势就体现出来了。</p>
</summary>
<category term="Rails" scheme="http://kaywu.xyz/categories/Rails/"/>
<category term="Rails" scheme="http://kaywu.xyz/tags/Rails/"/>
</entry>
<entry>
<title>Ruby 常量查找</title>
<link href="http://kaywu.xyz/2017/02/19/ruby-constant/"/>
<id>http://kaywu.xyz/2017/02/19/ruby-constant/</id>
<published>2017-02-19T09:08:10.000Z</published>
<updated>2017-10-10T15:25:40.000Z</updated>
<content type="html"><![CDATA[<p>对 Ruby 中常量查找只有基础的认识，使用上还是有不少疑问，比如 “::A” 的含义、为什么有时使用 “A::B” 而有时直接用 “B”。<br>花时间查了资料来加深对此的理解。</p><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Record</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">method</span></span></span><br><span class="line"> puts <span class="string">'outer'</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">Music</span></span></span><br><span class="line"></span><br><span class="line"> Record.method <span class="comment"># outer</span></span><br><span class="line"></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">Record</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">method</span></span></span><br><span class="line"> puts <span class="string">'inner'</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> Record.method <span class="comment"># inner</span></span><br><span class="line"> <span class="symbol">:</span><span class="symbol">:Record</span>.method <span class="comment"># outer</span></span><br><span class="line"> Music::Record.method <span class="comment"># inner</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>我们从简单的例子出发。在上文的代码中，共存在着两个名为 Record 的 module。一个是在最外层，一个是在 Music 内部。<br>当第 9 行调用 Record.method 时，由于 Music 内的 Record 还未被定义，因此调用的只能是最外层的 Record module。<br>而在第 17 行，当我们重新调用时，却发现调用的已经是内部的 Record 了。<br>可见常量的调用时是有一个先后顺序的，简单的说就是从近及远，先使用近的。虽然我们从直观上很容易理解近的含义，但严格上的顺序是通过 <code>Module.nesting</code> 得到的。在 Music 内部调用得到 [Record::Music, Record]，可见 Record::Music 确实比 Record 有更高的优先级。</p><a id="more"></a><p>我们再来看看 <code>::Record.method</code>，在使用 <code>::</code> 之后，调用的就变成了外层的 Record 。<code>::</code>的作用是从 top level 来查找相关常量，也就是调用最外层的 Record。</p><p><code>Record.method</code> 和 <code>Music::Record.method</code> 有什么区别吗？<br>若都能找到相同的对象，这两者是没有区别的，如 16、18 行。<br>但在 top level 中调用 <code>Music::Record.method</code> 时，需使用完整的路径 <code>Record::Music::Record.method</code>，不然会报 <code>uninitialized constant</code>。</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>Module.nesting 只是查找常量的一部分，更完整的常量查找步骤如下：</p><ol><li>向外找，从 Module.nesting 依次查找</li><li>向上找，从打开类 的 ancestors 依次查找（打开类：Module.nesting.first，若为空则为 Object）<br><a href="https://cirw.in/blog/constant-lookup" target="_blank" rel="noopener">Everything you ever wanted to know about constant lookup in Ruby</a> 讲得非常详情，就不多述了。</li></ol><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="https://cirw.in/blog/constant-lookup" target="_blank" rel="noopener">Everything you ever wanted to know about constant lookup in Ruby</a></li><li><a href="http://stackoverflow.com/questions/5032844/ruby-what-does-prefix-do" target="_blank" rel="noopener">Ruby: what does :: prefix do?</a></li></ul>]]></content>
<summary type="html">
<p>对 Ruby 中常量查找只有基础的认识，使用上还是有不少疑问，比如 “::A” 的含义、为什么有时使用 “A::B” 而有时直接用 “B”。<br>花时间查了资料来加深对此的理解。</p>
<figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Record</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">method</span></span></span><br><span class="line"> puts <span class="string">'outer'</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">Music</span></span></span><br><span class="line"></span><br><span class="line"> Record.method <span class="comment"># outer</span></span><br><span class="line"></span><br><span class="line"> <span class="class"><span class="keyword">module</span> <span class="title">Record</span></span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">method</span></span></span><br><span class="line"> puts <span class="string">'inner'</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"> Record.method <span class="comment"># inner</span></span><br><span class="line"> <span class="symbol">:</span><span class="symbol">:Record</span>.method <span class="comment"># outer</span></span><br><span class="line"> Music::Record.method <span class="comment"># inner</span></span><br><span class="line"> <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure>
<p>我们从简单的例子出发。在上文的代码中，共存在着两个名为 Record 的 module。一个是在最外层，一个是在 Music 内部。<br>当第 9 行调用 Record.method 时，由于 Music 内的 Record 还未被定义，因此调用的只能是最外层的 Record module。<br>而在第 17 行，当我们重新调用时，却发现调用的已经是内部的 Record 了。<br>可见常量的调用时是有一个先后顺序的，简单的说就是从近及远，先使用近的。虽然我们从直观上很容易理解近的含义，但严格上的顺序是通过 <code>Module.nesting</code> 得到的。在 Music 内部调用得到 [Record::Music, Record]，可见 Record::Music 确实比 Record 有更高的优先级。</p>
</summary>
<category term="Ruby" scheme="http://kaywu.xyz/categories/Ruby/"/>
<category term="Ruby" scheme="http://kaywu.xyz/tags/Ruby/"/>
</entry>
<entry>
<title>mac 上 MySQL system error 32 解决方案</title>
<link href="http://kaywu.xyz/2017/01/21/mysql-system-error-32/"/>
<id>http://kaywu.xyz/2017/01/21/mysql-system-error-32/</id>
<published>2017-01-21T09:53:22.000Z</published>
<updated>2017-09-10T08:13:51.000Z</updated>
<content type="html"><![CDATA[<p>不知为何，在升级了 macOS 系统之后，Rails 连接 MySQL 经常性会报 <code>Mysql2::Error: Lost connection to MySQL server at 'sending authentication information', system error: 32</code> 的错误。重启 MySQL 就好了，过段时间又会报错，烦不胜烦。下定决心解决下。</p><p>参考了<a href="http://bugs.mysql.com/bug.php?id=71960" target="_blank" rel="noopener">这个帖子</a>后，大致看到有两种解决方法。一种是调大系统 <code>ulimit -n</code> 的值，另一种是调小 <code>table_open_cache</code> 的值。<br>从帖子的回复以及 <code>ulimit -n</code> 表示系统的文件句柄限制来看，个人认为调节 <code>ulimit -n</code> 的值是一个治标不治本的方法。本机的值已经 4k+，应该足够了。</p><p>正巧有次出现问题时，MySQL 还连接着。查了下当时的 status，发现 <code>open_tables</code> 不过几十，而 <code>open_files</code> 却几千。<br>执行 <code>flush tables</code> 清除缓存后就可以重新连接了。确实和 <code>table_open_cache</code> 有一定的关系。</p><p>查看了下 <code>table_open_cache</code> 的值，默认值是 2000。通过在 <code>my.cnf</code> 中添加 <code>table_open_cache = 500</code>，MySQL 这几天就没有闹别扭，希望能一直安稳下去。</p><h4 id="MySQL-相关命令"><a href="#MySQL-相关命令" class="headerlink" title="MySQL 相关命令"></a>MySQL 相关命令</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">show status like '%open_files%'; # 查看 open_files 的值，open_tables 同理</span><br><span class="line">show variables like '%table_open_cache%'; # 查看 table_open_cache 的值</span><br></pre></td></tr></table></figure><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="http://bugs.mysql.com/bug.php?id=71960" target="_blank" rel="noopener">http://bugs.mysql.com/bug.php?id=71960</a></li></ul>]]></content>
<summary type="html">
<p>不知为何，在升级了 macOS 系统之后，Rails 连接 MySQL 经常性会报 <code>Mysql2::Error: Lost connection to MySQL server at &#39;sending authentication information
</summary>
<category term="MySQL" scheme="http://kaywu.xyz/categories/MySQL/"/>
<category term="MySQL" scheme="http://kaywu.xyz/tags/MySQL/"/>
</entry>
<entry>
<title>ActiveRecord select vs pluck</title>
<link href="http://kaywu.xyz/2017/01/15/rails-select-vs-pluck/"/>
<id>http://kaywu.xyz/2017/01/15/rails-select-vs-pluck/</id>
<published>2017-01-15T10:09:31.000Z</published>
<updated>2017-09-10T08:13:51.000Z</updated>
<content type="html"><![CDATA[<p><code>select</code>、<code>pluck</code> 都可以从数据库读取指定的字段，但两者存在不小的差别。</p><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">Product.select(<span class="symbol">:id</span>).to_a</span><br><span class="line"><span class="comment"># Product Load (0.5ms) SELECT `products`.`id` FROM `products`</span></span><br><span class="line"><span class="comment"># [#<Product id: 2>, #<Product id: 1>]</span></span><br><span class="line"></span><br><span class="line">Product.pluck(<span class="symbol">:id</span>)</span><br><span class="line"><span class="comment"># Product Load (0.4ms) SELECT `products`.`id` FROM `products`</span></span><br><span class="line"><span class="comment"># [2, 1]</span></span><br></pre></td></tr></table></figure><p><code>select</code> 返回的是仅含有 id 的 Product Model 数组，而 <code>pluck</code> 返回的是 id 的数组。<br>两者相比较，<code>pluck</code> 省却了构造 ActiveRecord 的过程，效率更优。我们可以通过 Benchmark.measure 来验证下。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">puts Benchmark.measure {Product.select(<span class="symbol">:id</span>).to_a}</span><br><span class="line">puts Benchmark.measure {Product.pluck(<span class="symbol">:id</span>)}</span><br></pre></td></tr></table></figure></p><table><thead><tr><th>-</th><th>user CPU time</th><th>system CPU time</th><th>sum CPU time</th><th>elapsed real time</th></tr></thead><tbody><tr><td>select</td><td>0.050000</td><td>0.020000</td><td>0.070000</td><td>0.095440</td></tr><tr><td>pluck</td><td>0.000000</td><td>0.000000</td><td>0.000000</td><td>0.001845</td></tr></tbody></table><p>除此之外，两者还有一个区别，即查询时机的不同。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">ProductOrder.where.<span class="keyword">not</span>(<span class="symbol">id:</span> SubOrder.where(<span class="symbol">sub_order_no:</span> <span class="string">'001'</span>).pluck(<span class="symbol">:order_id</span>))</span><br><span class="line"> <span class="comment"># SELECT `orders`.* FROM `orders` WHERE `orders`.`type` IN ('ProductOrder') AND (`orders`.`id` NOT IN (SELECT `sub_orders`.`order_id` FROM `sub_orders` WHERE `sub_orders`.`sub_order_no` = '001'))</span></span><br><span class="line"></span><br><span class="line">ProductOrder.where.<span class="keyword">not</span>(<span class="symbol">id:</span> SubOrder.where(<span class="symbol">sub_order_no:</span> <span class="string">'001'</span>).pluck(<span class="symbol">:order_id</span>))</span><br><span class="line"><span class="comment"># SELECT `sub_orders`.`order_id` FROM `sub_orders` WHERE `sub_orders`.`sub_order_no` = '001'</span></span><br><span class="line"><span class="comment"># SELECT `orders`.* FROM `orders` WHERE `orders`.`type` IN ('ProductOrder') AND (`orders`.`id` != 3)</span></span><br></pre></td></tr></table></figure></p><p>在上面这个例子中，通过 <code>pluck</code> 的调用进行了两次查询，而 <code>select</code> 只进行了一次查询。可见调用 <code>pluck</code> 会立即进行数据库查询。</p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><a href="http://gavinmiller.io/2013/getting-to-know-pluck-and-select/" target="_blank" rel="noopener">Getting to Know Pluck and Select</a></li><li><a href="http://findnerd.com/list/view/Select-Vs-Pluck-in-Rails/19258/" target="_blank" rel="noopener">Select Vs Pluck in Rails</a></li></ul>]]></content>
<summary type="html">
<p><code>select</code>、<code>pluck</code> 都可以从数据库读取指定的字段，但两者存在不小的差别。</p>
<figure class="highlight ruby"><table><tr><td class="gutter"><pre><
</summary>
<category term="Rails" scheme="http://kaywu.xyz/categories/Rails/"/>
<category term="Rails" scheme="http://kaywu.xyz/tags/Rails/"/>
</entry>
</feed>