-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
532 lines (341 loc) · 446 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>淦</title>
<subtitle>n*m*lg(b)</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="https://tankeryang.github.io/"/>
<updated>2019-10-09T17:06:09.611Z</updated>
<id>https://tankeryang.github.io/</id>
<author>
<name>tankeryang</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>python-eve 最佳实践</title>
<link href="https://tankeryang.github.io/posts/python-eve-%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5/"/>
<id>https://tankeryang.github.io/posts/python-eve-最佳实践/</id>
<published>2019-08-22T23:06:57.000Z</published>
<updated>2019-10-09T17:06:09.611Z</updated>
<content type="html"><![CDATA[<img src="/posts/python-eve-最佳实践/pic0.png"><p>实在找不到图了……</p><a id="more"></a><p>最近因为工作的关系，被要求用到 <a href="https://docs.python-eve.org/en/stable/" target="_blank" rel="noopener">Eve</a> 这个框架，需求简要概括就是对一个 mongo 表的 crud，本来用 flask 直接一把梭完就一个下午的事，但因为 “工作” 的关系，不得不盘一下 <strong>Eve</strong></p><h1 id="Eve-简介"><a href="#Eve-简介" class="headerlink" title="Eve 简介"></a>Eve 简介</h1><p>详细看了 Eve 的源码，他是一个对 <strong>Flask</strong> 高度定制化的，以 <strong>配置</strong> 为驱动的 restful 框架，根据配置文件生成 <strong>endpoint url</strong> 和 <strong>endpoint method</strong>，以 <strong>http method</strong> 对应 <strong>crud</strong> 操作。<strong>Eve</strong> 的数据持久化原生支持 <strong>mongodb</strong>，配置了 <strong>mongo</strong> 相关的参数后，对 <strong>endpoint url</strong> 的请求操作相当于直接对对应的 mongo 表进行 crud</p><p>下面上官方文档的例子进行说明</p><h2 id="快速开始"><a href="#快速开始" class="headerlink" title="快速开始"></a>快速开始</h2><ul> <li> <p>先启动一个 mongodb，这里用 docker 启一个 mongodb 容器 (这里选 3.4 版本的和公司一致，你可以自行选择其他版本) </p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ docker run --name mongodb -p 27017:27017 \</span><br><span class="line"> -v /srv/mongodb/etc:/etc/mongo \</span><br><span class="line"> -v /srv/mongo/data/db:/data/db \</span><br><span class="line"> -e MONGO_INITDB_ROOT_USERNAME=admin \</span><br><span class="line"> -e MONGO_INITDB_ROOT_PASSWORD=admin123 \</span><br><span class="line"> -d mongo:3.4-xenial</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>进入容器，创建 <code>test</code> 数据库，并设置用户 <code>test</code>，密码 <code>test123</code></p> <ul> <li> <p>进入容器的 mongodb 交互终端</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ docker <span class="built_in">exec</span> -it mongodb mongo -u admin -p admin123 --authenticationDatabase admin</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>创建 <code>test</code> 数据库，并设置用户 <code>test</code>，密码 <code>test123</code></p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line">> use <span class="built_in">test</span></span><br><span class="line">> db.createUser({user: <span class="string">'test'</span>, <span class="built_in">pwd</span>: <span class="string">'test123'</span>, roles:[{role: <span class="string">'readWrite'</span>, db: <span class="string">'test'</span>}]})</span><br></pre> </td> </tr> </table> </figure> </li> </ul> </li> <li> <p>假设你当前目录为 <code>workspace</code>，在这个目录下新建 <code>settings.py</code> 文件，顾名思义，这就是 <strong>配置文件</strong></p> <blockquote> <p>文件名可以自定义，在初始化 Eve 对象的时候传进去就好了，默认是 <code>settings.py</code></p> </blockquote> <figure class="highlight python"> <figcaption><span>settings.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># 下面为 mongo 相关配置</span></span><br><span class="line">MONGO_HOST = <span class="string">'localhost'</span></span><br><span class="line">MONGO_PORT = <span class="number">27017</span></span><br><span class="line">MONGO_USERNAME = <span class="string">'test'</span></span><br><span class="line">MONGO_PASSWORD = <span class="string">'test123'</span></span><br><span class="line">MONGO_AUTH_SOURCE = <span class="string">'test'</span></span><br><span class="line">MONGO_DBNAME = <span class="string">'test'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 定义 resource 和 item 能够接受的 http method</span></span><br><span class="line"><span class="comment"># 这里统一规定，resource 相当于 mongo collection，item 相当于 mongo document</span></span><br><span class="line">RESOURCE_METHODS = [<span class="string">'GET'</span>, <span class="string">'POST'</span>, <span class="string">'DELETE'</span>]</span><br><span class="line">ITEM_METHODS = [<span class="string">'GET'</span>, <span class="string">'PATCH'</span>, <span class="string">'PUT'</span>, <span class="string">'DELETE'</span>]</span><br><span class="line"></span><br><span class="line"><span class="comment"># 定义 item schema，schema 相当于 mongo document 的字段定义</span></span><br><span class="line">schema = {</span><br><span class="line"> <span class="string">'firstname'</span>: {</span><br><span class="line"> <span class="string">'type'</span>: <span class="string">'string'</span>,</span><br><span class="line"> <span class="string">'minlength'</span>: <span class="number">1</span>,</span><br><span class="line"> <span class="string">'maxlength'</span>: <span class="number">10</span>,</span><br><span class="line"> },</span><br><span class="line"> <span class="string">'lastname'</span>: {</span><br><span class="line"> <span class="string">'type'</span>: <span class="string">'string'</span>,</span><br><span class="line"> <span class="string">'minlength'</span>: <span class="number">1</span>,</span><br><span class="line"> <span class="string">'maxlength'</span>: <span class="number">15</span>,</span><br><span class="line"> <span class="string">'required'</span>: <span class="literal">True</span>,</span><br><span class="line"> <span class="string">'unique'</span>: <span class="literal">True</span>,</span><br><span class="line"> },</span><br><span class="line"> <span class="string">'role'</span>: {</span><br><span class="line"> <span class="string">'type'</span>: <span class="string">'list'</span>,</span><br><span class="line"> <span class="string">'allowed'</span>: [<span class="string">"author"</span>, <span class="string">"contributor"</span>, <span class="string">"copy"</span>],</span><br><span class="line"> },</span><br><span class="line"> <span class="string">'location'</span>: {</span><br><span class="line"> <span class="string">'type'</span>: <span class="string">'dict'</span>,</span><br><span class="line"> <span class="string">'schema'</span>: {</span><br><span class="line"> <span class="string">'address'</span>: {<span class="string">'type'</span>: <span class="string">'string'</span>},</span><br><span class="line"> <span class="string">'city'</span>: {<span class="string">'type'</span>: <span class="string">'string'</span>}</span><br><span class="line"> },</span><br><span class="line"> },</span><br><span class="line"> <span class="string">'born'</span>: {</span><br><span class="line"> <span class="string">'type'</span>: <span class="string">'datetime'</span>,</span><br><span class="line"> },</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># 定义 resource，名为 people</span></span><br><span class="line">people = {</span><br><span class="line"> <span class="string">'item_title'</span>: <span class="string">'person'</span>,</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 默认的请求 item 的 endpoint path 为 '/people/<mongo ObjectId>'，这样不太方便，</span></span><br><span class="line"> <span class="comment"># 我们可以自定义 item 的 endpoint path，这里用 lastname，因为是 unique 字段</span></span><br><span class="line"> <span class="string">'additional_lookup'</span>: {</span><br><span class="line"> <span class="string">'url'</span>: <span class="string">'regex("[\w]+")'</span>,</span><br><span class="line"> <span class="string">'field'</span>: <span class="string">'lastname'</span></span><br><span class="line"> },</span><br><span class="line"></span><br><span class="line"> <span class="comment"># cache 相关</span></span><br><span class="line"> <span class="string">'cache_control'</span>: <span class="string">'max-age=10,must-revalidate'</span>,</span><br><span class="line"> <span class="string">'cache_expires'</span>: <span class="number">10</span>,</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 这里会覆盖上面定义的全局 RESOURCE_METHOD</span></span><br><span class="line"> <span class="string">'resource_methods'</span>: [<span class="string">'GET'</span>, <span class="string">'POST'</span>],</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 指定 schema</span></span><br><span class="line"> <span class="string">'schema'</span>: schema</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># 定义 domain，相当于将 people resource 绑定到 ‘people’ 这个 resource endpoint</span></span><br><span class="line">DOMAIN = {<span class="string">'people'</span>: people}</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>接着新建 <code>main.py</code> 文件</p> <blockquote> <p>同样文件名可自定义，这个文件主要实例化 Eve 对象，相当于入口文件</p> </blockquote> <figure class="highlight python"> <figcaption><span>main.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> eve <span class="keyword">import</span> Eve</span><br><span class="line">app = Eve()</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> app.run()</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>执行 <code>main.py</code> 这个脚本</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ python main.py</span><br><span class="line"></span><br><span class="line">* Serving Flask app <span class="string">"eve"</span> (lazy loading)</span><br><span class="line">* Environment: production</span><br><span class="line">WARNING: This is a development server. Do not use it <span class="keyword">in</span> a production deployment.</span><br><span class="line">Use a production WSGI server instead.</span><br><span class="line">* Debug mode: off</span><br><span class="line">* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p><code>curl</code> 测试一下 api</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ curl -i http://127.0.0.1:5000/</span><br></pre> </td> </tr> </table> </figure> <p> 如果没有意外的话，你会得到下面的返回</p> <figure class="highlight json"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line">HTTP/1.0 200 OK</span><br><span class="line">Content-Type: application/json</span><br><span class="line">Content-Length: 62</span><br><span class="line">Server: Eve/0.9.2 Werkzeug/0.15.4 Python/3.6.8</span><br><span class="line">Date: Fri, 23 Aug 2019 09:32:08 GMT</span><br><span class="line"></span><br><span class="line">{<span class="attr">"_links"</span>: {<span class="attr">"child"</span>: [{<span class="attr">"href"</span>: <span class="string">"people"</span>, <span class="attr">"title"</span>: <span class="string">"people"</span>}]}}</span><br></pre> </td> </tr> </table> </figure> <p> 请求一下 <code>people</code> 资源</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ curl http://127.0.0.1:5000/people</span><br></pre> </td> </tr> </table> </figure> <p> 如果没有意外的话，你会得到下面的返回</p> <figure class="highlight json"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre> </td> <td class="code"> <pre><span class="line">{</span><br><span class="line"> <span class="attr">"_items"</span>: [],</span><br><span class="line"> <span class="attr">"_links"</span>: {</span><br><span class="line"> <span class="attr">"self"</span>: {</span><br><span class="line"> <span class="attr">"href"</span>: <span class="string">"people"</span>,</span><br><span class="line"> <span class="attr">"title"</span>: <span class="string">"people"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"parent"</span>: {</span><br><span class="line"> <span class="attr">"href"</span>: <span class="string">"/"</span>,</span><br><span class="line"> <span class="attr">"title"</span>: <span class="string">"home"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre> </td> </tr> </table> </figure> <p> 试一下 <code>DELETE</code> http method</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ curl -X DELETE http://127.0.0.1:5000/people</span><br></pre> </td> </tr> </table> </figure> <p> 你应该会得到</p> <figure class="highlight json"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line">{</span><br><span class="line"> <span class="attr">"_status"</span>: <span class="string">"ERR"</span>,</span><br><span class="line"> <span class="attr">"_error"</span>: {</span><br><span class="line"> <span class="attr">"code"</span>: <span class="number">405</span>,</span><br><span class="line"> <span class="attr">"message"</span>: <span class="string">"The method is not allowed for the requested URL."</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre> </td> </tr> </table> </figure> <p> 因为我们在 <code>people</code> 里配置了 <code>'resource_methods': ['GET', 'POST']</code>，除了 <code>GET</code>、<code>POST</code> 外，其他请求都是不成功的</p> <p> 我们 <code>POST</code> 一些数据过去</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># POST 多条数据传 list，单条直接 json</span></span><br><span class="line">> $ curl -X POST \</span><br><span class="line"> -H <span class="string">'Content-Type: application/json'</span> \</span><br><span class="line"> -d <span class="string">'[{"firstname": "barack", "lastname": "obama"}, {"firstname": "mitt", "lastname": "romney"}]'</span> \</span><br><span class="line"> http://127.0.0.1:5000/people</span><br></pre> </td> </tr> </table> </figure> <p> 返回结果为</p> <figure class="highlight json"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre> </td> <td class="code"> <pre><span class="line">{</span><br><span class="line"> <span class="attr">"_status"</span>: <span class="string">"OK"</span>,</span><br><span class="line"> <span class="attr">"_items"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"_updated"</span>: <span class="string">"Fri, 23 Aug 2019 10:45:30 GMT"</span>,</span><br><span class="line"> <span class="attr">"_created"</span>: <span class="string">"Fri, 23 Aug 2019 10:45:30 GMT"</span>,</span><br><span class="line"> <span class="attr">"_etag"</span>: <span class="string">"fc6d5b3a95813465269c57d180ef1716f700cc6a"</span>,</span><br><span class="line"> <span class="attr">"_id"</span>: <span class="string">"5d5fc3cae85c86602304ef3b"</span>,</span><br><span class="line"> <span class="attr">"_links"</span>: {</span><br><span class="line"> <span class="attr">"self"</span>: {</span><br><span class="line"> <span class="attr">"title"</span>: <span class="string">"person"</span>,</span><br><span class="line"> <span class="attr">"href"</span>: <span class="string">"people/5d5fc3cae85c86602304ef3b"</span></span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"_status"</span>: <span class="string">"OK"</span></span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"_updated"</span>: <span class="string">"Fri, 23 Aug 2019 10:45:30 GMT"</span>,</span><br><span class="line"> <span class="attr">"_created"</span>: <span class="string">"Fri, 23 Aug 2019 10:45:30 GMT"</span>,</span><br><span class="line"> <span class="attr">"_etag"</span>: <span class="string">"30761f80c263beb3b96fef0e07f5ec60ca4086e2"</span>,</span><br><span class="line"> <span class="attr">"_id"</span>: <span class="string">"5d5fc3cae85c86602304ef3c"</span>,</span><br><span class="line"> <span class="attr">"_links"</span>: {</span><br><span class="line"> <span class="attr">"self"</span>: {</span><br><span class="line"> <span class="attr">"title"</span>: <span class="string">"person"</span>,</span><br><span class="line"> <span class="attr">"href"</span>: <span class="string">"people/5d5fc3cae85c86602304ef3c"</span></span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"_status"</span>: <span class="string">"OK"</span>}</span><br><span class="line"> ]</span><br><span class="line">}</span><br></pre> </td> </tr> </table> </figure> <p> 查看一下 mongo</p> <img src="/posts/python-eve-最佳实践/pic1.png"> <p> 已经自动创建了 <code>people</code> 表，并插入了刚刚传的两条数据</p> <p> 我们请求一下 <code>item</code>，也就是 <code>people</code> 里的 <code>document</code></p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ curl -i http://127.0.0.1:5000/people/obama</span><br></pre> </td> </tr> </table> </figure> <p> 可以看到，我们直接用 <code>people/obama</code> 来请求资源，因为我们之前在 <code>people</code> 里定义了 <code>additional_lookup</code>，因此我们直接用 <code>last_name</code> 作为 <code>endpoint</code> 来访问，没有意外应该能得到下面的返回</p> <figure class="highlight json"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre> </td> <td class="code"> <pre><span class="line">{</span><br><span class="line"> <span class="attr">"_id"</span>: <span class="string">"5d5fc3cae85c86602304ef3b"</span>,</span><br><span class="line"> <span class="attr">"firstname"</span>: <span class="string">"barack"</span>,</span><br><span class="line"> <span class="attr">"lastname"</span>: <span class="string">"obama"</span>,</span><br><span class="line"> <span class="attr">"_updated"</span>: <span class="string">"Fri, 23 Aug 2019 10:45:30 GMT"</span>,</span><br><span class="line"> <span class="attr">"_created"</span>: <span class="string">"Fri, 23 Aug 2019 10:45:30 GMT"</span>,</span><br><span class="line"> <span class="attr">"_etag"</span>: <span class="string">"fc6d5b3a95813465269c57d180ef1716f700cc6a"</span>,</span><br><span class="line"> <span class="attr">"_links"</span>: {</span><br><span class="line"> <span class="attr">"self"</span>: {</span><br><span class="line"> <span class="attr">"title"</span>: <span class="string">"person"</span>,</span><br><span class="line"> <span class="attr">"href"</span>: <span class="string">"people/5d5fc3cae85c86602304ef3b"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"parent"</span>: {</span><br><span class="line"> <span class="attr">"title"</span>: <span class="string">"home"</span>,</span><br><span class="line"> <span class="attr">"href"</span>: <span class="string">"/"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"collection"</span>: {</span><br><span class="line"> <span class="attr">"title"</span>: <span class="string">"people"</span>,</span><br><span class="line"> <span class="attr">"href"</span>: <span class="string">"people"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre> </td> </tr> </table> </figure> </li></ul><h2 id="源码分析"><a href="#源码分析" class="headerlink" title="源码分析"></a>源码分析</h2><p>Eve 的目录结构如下</p><figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre> </td> <td class="code"> <pre><span class="line">eve</span><br><span class="line">├── io/</span><br><span class="line">├── methods/</span><br><span class="line">├── tests/</span><br><span class="line">├── __init__.py</span><br><span class="line">├── auth.py</span><br><span class="line">├── default_settings.py</span><br><span class="line">├── defaults.py</span><br><span class="line">├── endpoints.py</span><br><span class="line">├── exceptions.py</span><br><span class="line">├── flaskapp.py</span><br><span class="line">├── logging.py</span><br><span class="line">├── render.py</span><br><span class="line">├── utils.py</span><br><span class="line">├── validation.py</span><br><span class="line">└── versioning.py</span><br></pre> </td> </tr> </table></figure><p>这里不会一一详解，只会分析和配置比较相关的，影响开发的部分，部分分析会在注释里呈现</p><h3 id="endpoints"><a href="#endpoints" class="headerlink" title="endpoints"></a>endpoints</h3><p>Eve 初始化的时候 <code>__init__()</code> 方法会调用 <code>register_resource()</code> 方法来注册 <code>resource</code> 信息</p><figure class="highlight python"> <figcaption><span>flaskapp.py __init__()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># Use a snapshot of the DOMAIN setup for iteration so</span></span><br><span class="line"><span class="comment"># further insertion of versioned resources do not</span></span><br><span class="line"><span class="comment"># cause a RuntimeError due to the change of size of</span></span><br><span class="line"><span class="comment"># the dict</span></span><br><span class="line">domain_copy = copy.deepcopy(self.config[<span class="string">"DOMAIN"</span>])</span><br><span class="line"><span class="keyword">for</span> resource, settings <span class="keyword">in</span> domain_copy.items():</span><br><span class="line"> self.register_resource(resource, settings)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 这里 self.config 就是 settings.py 里的配置，self.config['DOMAIN'] 就是 DOMAIN = {'people': people}</span></span><br></pre> </td> </tr> </table></figure><p><code>self.register_resource()</code> 方法会调用 <code>self._add_resource_url_rules(resource, settings)</code>，绑定 <code>endpoint url</code> 和 <code>endpoint method</code></p><figure class="highlight python"> <figcaption><span>flaskapp.py register_resource()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># set up resource</span></span><br><span class="line">self._set_resource_defaults(resource, settings)</span><br><span class="line">self._validate_resource_settings(resource, settings)</span><br><span class="line">self._add_resource_url_rules(resource, settings) <span class="comment"># 就是它</span></span><br></pre> </td> </tr> </table></figure><p><code>_add_resource_url_rules()</code> 实现如下</p><figure class="highlight python"> <figcaption><span>_add_resource_url_rules()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">_add_resource_url_rules</span><span class="params">(self, resource, settings)</span>:</span></span><br><span class="line"> <span class="string">""" Builds the API url map for one resource. Methods are enabled for</span></span><br><span class="line"><span class="string"> each mapped endpoint, as configured in the settings.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.5</span></span><br><span class="line"><span class="string"> Don't add resource to url rules if it's flagged as internal.</span></span><br><span class="line"><span class="string"> Strip regexes out of config.URLS helper. Closes #466.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionadded:: 0.2</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> self.config[<span class="string">"SOURCES"</span>][resource] = settings[<span class="string">"datasource"</span>]</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> settings[<span class="string">"internal_resource"</span>]:</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"></span><br><span class="line"> url = <span class="string">"%s/%s"</span> % (self.api_prefix, settings[<span class="string">"url"</span>])</span><br><span class="line"></span><br><span class="line"> pretty_url = settings[<span class="string">"url"</span>]</span><br><span class="line"> <span class="keyword">if</span> <span class="string">"<"</span> <span class="keyword">in</span> pretty_url:</span><br><span class="line"> pretty_url = (</span><br><span class="line"> pretty_url[: pretty_url.index(<span class="string">"<"</span>) + <span class="number">1</span>]</span><br><span class="line"> + pretty_url[pretty_url.rindex(<span class="string">":"</span>) + <span class="number">1</span> :]</span><br><span class="line"> )</span><br><span class="line"> self.config[<span class="string">"URLS"</span>][resource] = pretty_url</span><br><span class="line"></span><br><span class="line"> <span class="comment"># resource endpoint</span></span><br><span class="line"> <span class="comment"># 这里调用 flask 的 add_url_rule() 方法绑定 resource url 和 method</span></span><br><span class="line"> <span class="comment"># collection_endpoint 函数是对 http method 的一些封装，下面会讲到</span></span><br><span class="line"> endpoint = resource + <span class="string">"|resource"</span></span><br><span class="line"> self.add_url_rule(</span><br><span class="line"> url,</span><br><span class="line"> endpoint,</span><br><span class="line"> view_func=collections_endpoint,</span><br><span class="line"> methods=settings[<span class="string">"resource_methods"</span>] + [<span class="string">"OPTIONS"</span>],</span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> <span class="comment"># item endpoint</span></span><br><span class="line"> <span class="comment"># 这里调用 flask 的 add_url_rule() 方法绑 item url 和 method</span></span><br><span class="line"> <span class="keyword">if</span> settings[<span class="string">"item_lookup"</span>]:</span><br><span class="line"> item_url = <span class="string">"%s/<%s:%s>"</span> % (</span><br><span class="line"> url,</span><br><span class="line"> settings[<span class="string">"item_url"</span>],</span><br><span class="line"> settings[<span class="string">"item_lookup_field"</span>],</span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> <span class="comment"># item_endpoint 函数是对 http method 的一些封装，下面会讲到</span></span><br><span class="line"> endpoint = resource + <span class="string">"|item_lookup"</span></span><br><span class="line"> self.add_url_rule(</span><br><span class="line"> item_url,</span><br><span class="line"> endpoint,</span><br><span class="line"> view_func=item_endpoint,</span><br><span class="line"> methods=settings[<span class="string">"item_methods"</span>] + [<span class="string">"OPTIONS"</span>],</span><br><span class="line"> )</span><br><span class="line"> <span class="keyword">if</span> <span class="string">"PATCH"</span> <span class="keyword">in</span> settings[<span class="string">"item_methods"</span>]:</span><br><span class="line"> <span class="comment"># support for POST with X-HTTP-Method-Override header for</span></span><br><span class="line"> <span class="comment"># clients not supporting PATCH. Also see item_endpoint() in</span></span><br><span class="line"> <span class="comment"># endpoints.py</span></span><br><span class="line"> endpoint = resource + <span class="string">"|item_post_override"</span></span><br><span class="line"> self.add_url_rule(</span><br><span class="line"> item_url, endpoint, view_func=item_endpoint, methods=[<span class="string">"POST"</span>]</span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> <span class="comment"># also enable an alternative lookup/endpoint if allowed</span></span><br><span class="line"> lookup = settings.get(<span class="string">"additional_lookup"</span>)</span><br><span class="line"> <span class="keyword">if</span> lookup:</span><br><span class="line"> l_type = settings[<span class="string">"schema"</span>][lookup[<span class="string">"field"</span>]][<span class="string">"type"</span>]</span><br><span class="line"> <span class="keyword">if</span> l_type == <span class="string">"integer"</span>:</span><br><span class="line"> item_url = <span class="string">"%s/<int:%s>"</span> % (url, lookup[<span class="string">"field"</span>])</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> item_url = <span class="string">"%s/<%s:%s>"</span> % (url, lookup[<span class="string">"url"</span>], lookup[<span class="string">"field"</span>])</span><br><span class="line"> endpoint = resource + <span class="string">"|item_additional_lookup"</span></span><br><span class="line"> self.add_url_rule(</span><br><span class="line"> item_url,</span><br><span class="line"> endpoint,</span><br><span class="line"> view_func=item_endpoint,</span><br><span class="line"> methods=[<span class="string">"GET"</span>, <span class="string">"OPTIONS"</span>],</span><br><span class="line"> )</span><br></pre> </td> </tr> </table></figure><p><code>collections_endpoint</code> 和 <code>item_endpoint</code> 封装了 <strong>http method</strong> 对应的 <strong>mongo io 操作</strong></p><figure class="highlight python"> <figcaption><span>endpoints.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">collections_endpoint</span><span class="params">(**lookup)</span>:</span></span><br><span class="line"> <span class="string">""" Resource endpoint handler</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> :param url: the url that led here</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.3</span></span><br><span class="line"><span class="string"> Pass lookup query down to delete_resource, so it can properly process</span></span><br><span class="line"><span class="string"> sub-resources.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.2</span></span><br><span class="line"><span class="string"> Relying on request.endpoint to retrieve the resource being consumed.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.1.1</span></span><br><span class="line"><span class="string"> Relying on request.path for determining the current endpoint url.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.0.7</span></span><br><span class="line"><span class="string"> Using 'utils.request_method' helper function now.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.0.6</span></span><br><span class="line"><span class="string"> Support for HEAD requests</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.0.2</span></span><br><span class="line"><span class="string"> Support for DELETE resource method.</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"> resource = _resource()</span><br><span class="line"> response = <span class="literal">None</span></span><br><span class="line"> method = request.method</span><br><span class="line"> <span class="keyword">if</span> method <span class="keyword">in</span> (<span class="string">"GET"</span>, <span class="string">"HEAD"</span>):</span><br><span class="line"> response = get(resource, lookup)</span><br><span class="line"> <span class="keyword">elif</span> method == <span class="string">"POST"</span>:</span><br><span class="line"> response = post(resource)</span><br><span class="line"> <span class="keyword">elif</span> method == <span class="string">"DELETE"</span>:</span><br><span class="line"> response = delete(resource, lookup)</span><br><span class="line"> <span class="keyword">elif</span> method == <span class="string">"OPTIONS"</span>:</span><br><span class="line"> send_response(resource, response)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> abort(<span class="number">405</span>)</span><br><span class="line"> <span class="keyword">return</span> send_response(resource, response)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">item_endpoint</span><span class="params">(**lookup)</span>:</span></span><br><span class="line"> <span class="string">""" Item endpoint handler</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> :param url: the url that led here</span></span><br><span class="line"><span class="string"> :param lookup: sub resource query</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.2</span></span><br><span class="line"><span class="string"> Support for sub-resources.</span></span><br><span class="line"><span class="string"> Relying on request.endpoint to retrieve the resource being consumed.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.1.1</span></span><br><span class="line"><span class="string"> Relying on request.path for determining the current endpoint url.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.1.0</span></span><br><span class="line"><span class="string"> Support for PUT method.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.0.7</span></span><br><span class="line"><span class="string"> Using 'utils.request_method' helper function now.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.0.6</span></span><br><span class="line"><span class="string"> Support for HEAD requests</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> resource = _resource()</span><br><span class="line"> response = <span class="literal">None</span></span><br><span class="line"> method = request.method</span><br><span class="line"> <span class="keyword">if</span> method <span class="keyword">in</span> (<span class="string">"GET"</span>, <span class="string">"HEAD"</span>):</span><br><span class="line"> response = getitem(resource, **lookup)</span><br><span class="line"> <span class="keyword">elif</span> method == <span class="string">"PATCH"</span>:</span><br><span class="line"> response = patch(resource, **lookup)</span><br><span class="line"> <span class="keyword">elif</span> method == <span class="string">"PUT"</span>:</span><br><span class="line"> response = put(resource, **lookup)</span><br><span class="line"> <span class="keyword">elif</span> method == <span class="string">"DELETE"</span>:</span><br><span class="line"> response = deleteitem(resource, **lookup)</span><br><span class="line"> <span class="keyword">elif</span> method == <span class="string">"OPTIONS"</span>:</span><br><span class="line"> send_response(resource, response)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> abort(<span class="number">405</span>)</span><br><span class="line"> <span class="keyword">return</span> send_response(resource, response)</span><br></pre> </td> </tr> </table></figure><p>对应 <code>get</code>、<code>put</code> 等具体实现在 <code>method</code> 文件夹下</p><h2 id="使用建议"><a href="#使用建议" class="headerlink" title="使用建议"></a>使用建议</h2><p>在我的需求场景中，mongo 表有一个唯一标识字段 <code>urs</code>，<strong>查询</strong> 操作直接通过这个字段查询，<strong>插入</strong> 数据时，已存在则覆盖(upsert)，不存在则直接插入(insert)，<strong>删除</strong> 时根据这个字段删除</p><p>但是因为 <code>POST</code> 只有 <code>collections_endpoint</code> 实现，而且还不能够 <code>upsert</code> 只能 <code>insert</code>，所以插入需要用 <code>PUT</code> 来做，因此 <code>settings.py</code> 里只设置 <code>item_methods</code></p><p>settings 大致如下:</p><figure class="highlight python"> <figcaption><span>settings.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">import</span> os</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> schemas <span class="keyword">import</span> urs_schema</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># config =========================================================</span></span><br><span class="line"><span class="comment"># MONGO_HOST = os.environ.get('MONGO_HOST')</span></span><br><span class="line"><span class="comment"># MONGO_PORT = int(os.environ.get('MONGO_PORT'))</span></span><br><span class="line">MONGO_URI = os.environ.get(<span class="string">'MONGO_URI'</span>)</span><br><span class="line">MONGO_USERNAME = os.environ.get(<span class="string">'MONGO_USERNAME'</span>)</span><br><span class="line">MONGO_PASSWORD = os.environ.get(<span class="string">'MONGO_PASSWORD'</span>)</span><br><span class="line">MONGO_AUTH_SOURCE = os.environ.get(<span class="string">'MONGO_AUTH_SOURCE'</span>)</span><br><span class="line">MONGO_DBNAME = os.environ.get(<span class="string">'MONGO_DBNAME'</span>)</span><br><span class="line"></span><br><span class="line">IF_MATCH = <span class="literal">False</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># domain =========================================================</span></span><br><span class="line">offline_trading_account = {</span><br><span class="line"> <span class="string">'item_title'</span>: <span class="string">'urs'</span>,</span><br><span class="line"> <span class="string">'id_field'</span>: <span class="string">'urs'</span>, <span class="comment"># 比较关键的设置，将 id_field 设为你的唯一字段</span></span><br><span class="line"> <span class="string">'item_url'</span>: <span class="string">'regex("[\s\S]+")'</span>,</span><br><span class="line"> <span class="string">'item_lookup_field'</span>: <span class="string">'urs'</span>,</span><br><span class="line"> <span class="string">'additional_lookup'</span>: {</span><br><span class="line"> <span class="string">'url'</span>: <span class="string">'regex("[\s\S]+")'</span>,</span><br><span class="line"> <span class="string">'field'</span>: <span class="string">'urs'</span></span><br><span class="line"> },</span><br><span class="line"> <span class="string">'resource_methods'</span>: [], <span class="comment"># resource_methods 为空</span></span><br><span class="line"> <span class="string">'item_methods'</span>: [<span class="string">'GET'</span>, <span class="string">'PUT'</span>, <span class="string">'DELETE'</span>],</span><br><span class="line"> <span class="string">'schema'</span>: urs_schema</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">DOMAIN = {os.environ.get(<span class="string">'MONGO_COLLECTION'</span>): offline_trading_account}</span><br></pre> </td> </tr> </table></figure><p>在上面的快速开始中，你可能注意到，插入数据时，eve 会自动给你加上 <code>_updated</code>、<code>_created</code> 的 date 格式字段，你想动它又不知如何下手；还有它返回的 json 格式，可能并不符合需求。这时你就要利用到 <strong>hook</strong> 的特性了。Eve 提供了一系列 <strong>hook 函数</strong> 让你进行操作，这里贴上我定义的 hook 函数供参考，具体用法跟如何定义参考 Eve 文档</p><figure class="highlight python"> <figcaption><span>hooks.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">import</span> json</span><br><span class="line"><span class="keyword">from</span> datetime <span class="keyword">import</span> datetime</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> request, abort</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">before_insert_urs</span><span class="params">(items)</span>:</span></span><br><span class="line"> <span class="keyword">for</span> item <span class="keyword">in</span> items:</span><br><span class="line"> item[<span class="string">'updateTime'</span>] = datetime.now().strftime(<span class="string">'%Y-%m-%d %H:%M:%S'</span>)</span><br><span class="line"> item.pop(<span class="string">'_created'</span>)</span><br><span class="line"> item.pop(<span class="string">'_updated'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">before_replace_urs</span><span class="params">(item, original)</span>:</span></span><br><span class="line"> item[<span class="string">'updateTime'</span>] = datetime.now().strftime(<span class="string">'%Y-%m-%d %H:%M:%S'</span>)</span><br><span class="line"> item.pop(<span class="string">'_created'</span>)</span><br><span class="line"> item.pop(<span class="string">'_updated'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">after_fetched_urs</span><span class="params">(response)</span>:</span></span><br><span class="line"> response.pop(<span class="string">'_id'</span>)</span><br><span class="line"> response[<span class="string">'_status'</span>] = <span class="string">'OK'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">before_get_put_delete</span><span class="params">(resource, _request, lookup)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> _request.headers.get(<span class="string">'X-NEProxy-User'</span>) <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> abort(<span class="number">403</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">after_get_put_delete</span><span class="params">(resoure, _request, payload)</span>:</span></span><br><span class="line"> code = payload.status_code</span><br><span class="line"> data = payload.json.copy()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> len(data) == <span class="number">0</span>:</span><br><span class="line"> data[<span class="string">'_status'</span>] = <span class="string">'OK'</span></span><br><span class="line"> <span class="keyword">if</span> data.get(<span class="string">'_created'</span>):</span><br><span class="line"> data.pop(<span class="string">'_created'</span>)</span><br><span class="line"> <span class="keyword">if</span> data.get(<span class="string">'_updated'</span>):</span><br><span class="line"> data.pop(<span class="string">'_updated'</span>)</span><br><span class="line"> <span class="keyword">if</span> data.get(<span class="string">'_links'</span>):</span><br><span class="line"> data.pop(<span class="string">'_links'</span>)</span><br><span class="line"></span><br><span class="line"> payload.json.clear()</span><br><span class="line"> payload.json[<span class="string">'code'</span>] = code</span><br><span class="line"> payload.json[<span class="string">'data'</span>] = data</span><br><span class="line"></span><br><span class="line"> payload.status_code = <span class="number">200</span></span><br><span class="line"> payload.data = json.dumps(payload.json).encode()</span><br><span class="line"> payload.headers[<span class="string">'Access-Control-Allow-Origin'</span>] = <span class="string">'*'</span></span><br></pre> </td> </tr> </table></figure><p>在 app 对象里注册 hook 函数</p><figure class="highlight python"> <figcaption><span>init.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> eve <span class="keyword">import</span> Eve</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> jsonify</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> hooks <span class="keyword">import</span> (</span><br><span class="line"> before_get_put_delete,</span><br><span class="line"> before_insert_urs,</span><br><span class="line"> before_replace_urs,</span><br><span class="line"> after_fetched_urs,</span><br><span class="line"> after_get_put_delete</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">register_event_hook</span><span class="params">(app)</span>:</span></span><br><span class="line"> <span class="comment"># 这里 req131313_offline_trading_account 要和你定义的 resource 名一致，也就是 DOMAIN 的 key</span></span><br><span class="line"> app.on_insert_req131313_offline_trading_account += before_insert_urs</span><br><span class="line"> app.on_replace_req131313_offline_trading_account += before_replace_urs</span><br><span class="line"> app.on_fetched_item_req131313_offline_trading_account += after_fetched_urs</span><br><span class="line"> app.on_pre_GET += before_get_put_delete</span><br><span class="line"> app.on_pre_PUT += before_get_put_delete</span><br><span class="line"> app.on_pre_DELETE += before_get_put_delete</span><br><span class="line"> app.on_post_GET += after_get_put_delete</span><br><span class="line"> app.on_post_PUT += after_get_put_delete</span><br><span class="line"> app.on_post_DELETE += after_get_put_delete</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">create_app</span><span class="params">()</span>:</span></span><br><span class="line"> app = Eve()</span><br><span class="line"> register_event_hook(app)</span><br><span class="line"> <span class="keyword">return</span> app</span><br></pre> </td> </tr> </table></figure><p>hook 的原理其实就是在执行 <code>method</code> 里的函数时(如 <code>get</code>、<code>put</code>)，通过 <code>getattr</code> 获取到 <code>app</code> 对象注册的 hook 函数，然后调用执行，下面是代码片段</p><figure class="highlight python"> <figcaption><span>methods/common.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">pre_event</span><span class="params">(f)</span>:</span></span><br><span class="line"> <span class="string">""" Enable a Hook pre http request.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.6</span></span><br><span class="line"><span class="string"> Enable callback hooks for HEAD requests.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionchanged:: 0.4</span></span><br><span class="line"><span class="string"> Merge 'sub_resource_lookup' (args[1]) with kwargs, so http methods can</span></span><br><span class="line"><span class="string"> all enjoy the same signature, and data layer find methods can seemingly</span></span><br><span class="line"><span class="string"> process both kind of queries.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> .. versionadded:: 0.2</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"><span class="meta"> @wraps(f)</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">decorated</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line"> method = request.method</span><br><span class="line"> <span class="keyword">if</span> method == <span class="string">"HEAD"</span>:</span><br><span class="line"> method = <span class="string">"GET"</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 这里拿到的 event_name 就是我们 app 注册的 hook 函数名</span></span><br><span class="line"> event_name = <span class="string">"on_pre_"</span> + method</span><br><span class="line"> resource = args[<span class="number">0</span>] <span class="keyword">if</span> args <span class="keyword">else</span> <span class="literal">None</span></span><br><span class="line"> gh_params = ()</span><br><span class="line"> rh_params = ()</span><br><span class="line"> combined_args = kwargs</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> len(args) > <span class="number">1</span>:</span><br><span class="line"> combined_args.update(args[<span class="number">1</span>].items())</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> method <span class="keyword">in</span> (<span class="string">"GET"</span>, <span class="string">"PATCH"</span>, <span class="string">"DELETE"</span>, <span class="string">"PUT"</span>):</span><br><span class="line"> gh_params = (resource, request, combined_args)</span><br><span class="line"> rh_params = (request, combined_args)</span><br><span class="line"> <span class="keyword">elif</span> method <span class="keyword">in</span> (<span class="string">"POST"</span>,):</span><br><span class="line"> <span class="comment"># POST hook does not support the kwargs argument</span></span><br><span class="line"> gh_params = (resource, request)</span><br><span class="line"> rh_params = (request,)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># general hook</span></span><br><span class="line"> <span class="comment"># 通过 getattr 调用</span></span><br><span class="line"> getattr(app, event_name)(*gh_params)</span><br><span class="line"> <span class="keyword">if</span> resource:</span><br><span class="line"> <span class="comment"># resource hook</span></span><br><span class="line"> getattr(app, event_name + <span class="string">"_"</span> + resource)(*rh_params)</span><br><span class="line"></span><br><span class="line"> r = f(resource, **combined_args)</span><br><span class="line"> <span class="keyword">return</span> r</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> decorated</span><br></pre> </td> </tr> </table></figure><h1 id="结语"><a href="#结语" class="headerlink" title="结语"></a>结语</h1><p>在看了大部分的源码和文档之后，总结一下，Eve 这个框架实现的并不复杂，思想上非常符合 restful 风格，可以说是一个通用的，以配置为驱动的 restful 框架，但是也仅仅是通用而已了，因为它的 endpoint method 已经封装好了，你想要做一些改动，比如入参或返回，要通过 hook 来做，这样就很不友好而且很不规范，还仅支持 mongo，那就只能写些小接口玩玩了。或许这就是它原本的初衷呢？</p>]]></content>
<summary type="html">
<img src="/posts/python-eve-最佳实践/pic0.png">
<p>实在找不到图了……</p>
</summary>
<category term="python" scheme="https://tankeryang.github.io/tags/python/"/>
<category term="python-eve" scheme="https://tankeryang.github.io/tags/python-eve/"/>
<category term="web" scheme="https://tankeryang.github.io/tags/web/"/>
</entry>
<entry>
<title>docker配置TLS认证开启远程访问</title>
<link href="https://tankeryang.github.io/posts/docker%E9%85%8D%E7%BD%AETLS%E8%AE%A4%E8%AF%81%E5%BC%80%E5%90%AF%E8%BF%9C%E7%A8%8B%E8%AE%BF%E9%97%AE/"/>
<id>https://tankeryang.github.io/posts/docker配置TLS认证开启远程访问/</id>
<published>2018-10-24T15:36:27.000Z</published>
<updated>2019-10-09T17:06:09.603Z</updated>
<content type="html"><![CDATA[<img src="/posts/docker配置TLS认证开启远程访问/pic0.png"><a id="more"></a><h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p> docker在今天对应用开发，运维管理来说已经是必备组件了，web应用的部署，微服务搭建，CI/CD，集群管理哪都有它。关于docker使用这里就不细说了，参考<a href="https://docs.docker.com/" target="_blank" rel="noopener">官方文档</a>。</p><p>当我们有多台服务器，每台服务器有多个docker容器，统一管理与监控docker集群就变得非常重要。轻量级的开源docker管理工具有<a href="https://www.portainer.io/" target="_blank" rel="noopener">portainer</a>，重量级的有<a href="https://www.cnrancher.com" target="_blank" rel="noopener">rancher</a>。如果服务不多，集群节点不多的话一般 <strong>portainer</strong> 足以胜任。如果是大型集群的话可以考虑 <strong>rancher</strong>。</p><p>想要中心化的管理docker容器，那么就需要docker容器开启远程访问。在讲怎么开启远程访问之前，要先稍微讲一下 <strong>docker容器进程</strong> 和 <strong>docker守护进程</strong>：</p><ul> <li> <p><strong>docker容器进程</strong></p> <p> 顾名思义，就是运行在docker容器中的应用进程，比如你用docker启动的web应用，数据库等</p> </li> <li> <p><strong>docker守护进程</strong></p> <p> docker的设计是<code>C/S</code>模式，docker守护进程（server/服务端）运行在宿主机上，就是你运行docker容器的服务器。当你装完docker并敲下<code>service docker start</code>时，docker守护进程就启动了，它默认监听<code>/var/run.docker.sock</code>（unix域套接字）文件, 你可以通过<code>docker</code>命令（client/客户端）来对容器进行一系列操作，因为 <strong>本机</strong> 的 <strong>docker客户端</strong> 是默认通过<code>/var/run.docker.sock</code>来与 <strong>docker守护进程</strong> 进行通信的</p> </li></ul><p>稍微明白了 <strong>docker容器进程</strong> 与 <strong>docker守护进程</strong> 后，你会发现，只有通过docker守护进程，才能与docker容器进行交互。而docker守护进程默认只监听本地的<code>/va/run/docker.sock</code>，因此只能在本地通过<code>docker</code>命令操作容器。</p><p>能让远程docker客户端访问容器，我们就要让docker守护进程能监听远程访问的端口。docker是采用 <strong>socket</strong> 进行server和client的连接的，它提供了<code>tcp</code>和<code>fd</code>两种方式，一般采用<code>tcp</code>方式。</p><h1 id="配置远程访问"><a href="#配置远程访问" class="headerlink" title="配置远程访问"></a>配置远程访问</h1><h2 id="不安全的配置"><a href="#不安全的配置" class="headerlink" title="不安全的配置"></a>不安全的配置</h2><p>直接通过<code>dockerd</code>命令配置守护进程：</p><figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># 默认监听 6379 端口</span></span><br><span class="line">> $ dockerd -H 0.0.0.0</span><br></pre> </td> </tr> </table></figure><p>这样所有ip都能通过<code>docker -H <remote-dcoker-server_ip>:6379 [OPTION]</code>命令与远程的docker守护进程通信，操作docker容器，生产上不提倡这种做法。</p><h2 id="安全的配置"><a href="#安全的配置" class="headerlink" title="安全的配置"></a>安全的配置</h2><p>通过 <strong>TLS（Transport Layer Security - 安全传输层协议）</strong> 来进行远程访问（<a href="https://baike.baidu.com/item/TLS" target="_blank" rel="noopener">百度百科 - TLS</a>）。我们需要在远程docker服务器（运行docker守护进程的服务器）生成 <strong>CA证书，服务器证书，服务器密钥</strong>，然后自签名，再颁发给需要连接远程docker容器的服务器。下面是具体操作：</p><div class="note info"> <p>以下操作在<code>ubuntu 16.04 server</code>上测试成功，其他系统自行参考</p></div><h3 id="修改openssl配置的CA部分"><a href="#修改openssl配置的CA部分" class="headerlink" title="修改openssl配置的CA部分"></a>修改<code>openssl</code>配置的CA部分</h3><blockquote> <p><strong>ubuntu 16.04</strong> 下 <strong>openssl</strong> 配置文件位置：<code>/usr/lib/ssl/openssl.cnf</code>，其他系统可参考</p></blockquote><ul> <li> <p>修改如下部分：</p> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre> </td> <td class="code"> <pre><span class="line">[ CA_default ]</span><br><span class="line"></span><br><span class="line">dir = <填你的路径> # 下面的操作以 /etc/openssl 为例</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>创建相应文件和文件夹：</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ <span class="built_in">cd</span> /etc/openssl <span class="comment"># 没有就新建</span></span><br><span class="line">> $ mkdir -p {certs,private,tls,crl,newcerts}</span><br><span class="line">> $ <span class="built_in">echo</span> 00 > serial <span class="comment"># 注意serial要有数字，不然会报错</span></span><br><span class="line">> $ touch index.txt</span><br></pre> </td> </tr> </table> </figure> </li></ul><h3 id="生成私钥并自签证书"><a href="#生成私钥并自签证书" class="headerlink" title="生成私钥并自签证书"></a>生成私钥并自签证书</h3><figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ openssl genrsa -out private/cakey.pem -des 1024 <span class="comment"># 生成私钥</span></span><br><span class="line">> $ openssl req -new -x509 -key private/cakey.pem -days 3650 -out cacert.pem <span class="comment"># 自签证书: 执行后会让你填写一些配置</span></span><br></pre> </td> </tr> </table></figure><h3 id="颁发证书"><a href="#颁发证书" class="headerlink" title="颁发证书"></a>颁发证书</h3><ul> <li> <p>生成要颁发证书的密钥文件</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ openssl genrsa -out private/test.key 1024</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>生成证书请求</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ openssl req -new -key private/test.key -days 3650 -out test.csr</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>颁发证书</p> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ openssl ca -<span class="keyword">in</span> test.csr -out certs/test.crt -days 3650 <span class="comment"># 这里也需要填写一些配置，最好和前面自签证书的配置保持一致</span></span><br></pre> </td> </tr> </table> </figure> </li></ul><h3 id="配置docker守护进程使用TLS认证并监听tcp端口"><a href="#配置docker守护进程使用TLS认证并监听tcp端口" class="headerlink" title="配置docker守护进程使用TLS认证并监听tcp端口"></a>配置docker守护进程使用TLS认证并监听tcp端口</h3><figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ service docker stop <span class="comment"># 需先停止 docker 服务</span></span><br><span class="line">> $ dockerd --tls \</span><br><span class="line">> --tlscacert /etc/openssl/cacert.pem \</span><br><span class="line">> --tlscert /etc/openssl/certs/test.crt \</span><br><span class="line">> --tlskey /etc/openssl/private/test.key \</span><br><span class="line">> -H 0.0.0.0:2375 <span class="comment"># 默认 6379 端口，可自定义</span></span><br><span class="line">> $ service docker start <span class="comment"># 重启 docker 服务</span></span><br></pre> </td> </tr> </table></figure><h3 id="保存CA证书，服务器证书，服务器密钥"><a href="#保存CA证书，服务器证书，服务器密钥" class="headerlink" title="保存CA证书，服务器证书，服务器密钥"></a>保存CA证书，服务器证书，服务器密钥</h3><p> 重启docker后，你会发现在远程docker服务器上使用<code>docker</code>命令不起作用了，这是因为docker守护进程不再监听<code>var/run/docker.sock</code>文件，而是监听tcp端口了。所以如果要在远程docker服务器使用<code>docker</code>命令就要加上上面的参数<code>--tls</code>，<code>--tlscacert</code>，<code>--tlscert</code>，<code>--tlskey</code>，例如：</p><figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ docker --tls \</span><br><span class="line">> --tlscacert /etc/openssl/cacert.pem \</span><br><span class="line">> --tlscert /etc/openssl/certs/test.crt \</span><br><span class="line">> --tlskey /etc/openssl/private/test.key \</span><br><span class="line">> -H <span class="variable">$HOSTNAME</span>:2375 ps</span><br></pre> </td> </tr> </table></figure><p>这样就太麻烦了，所以我们用下面的做法，将证书，密钥保存到<code>~/.docker</code>文件夹，再配置环境变量：</p><figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line">> $ <span class="built_in">cd</span> ~/.docker <span class="comment"># 没有就新建</span></span><br><span class="line">> $ cp /etc/openssl/cacert.pem ./ca.pem</span><br><span class="line">> $ cp /etc/openssl/newcerts/00.pem ./cert.pem</span><br><span class="line">> $ cp /etc/openssl/private/test.key ./key.pem</span><br><span class="line">> $ <span class="built_in">echo</span> <span class="string">"export DOCKER_HOST=tcp://<span class="variable">$HOSTNAME</span>:2375"</span> >> /etc/profile</span><br><span class="line">> $ <span class="built_in">echo</span> <span class="string">"export DOCKER_TLS_VERIFY=1"</span> >> /etc/profile</span><br><span class="line">> $ <span class="built_in">source</span> /etc/profile</span><br></pre> </td> </tr> </table></figure><p>这时你再使用<code>docker</code>命令就没问题了。</p><h1 id="配置portainer添加远程docker集群"><a href="#配置portainer添加远程docker集群" class="headerlink" title="配置portainer添加远程docker集群"></a>配置portainer添加远程docker集群</h1><ul> <li> <p>添加节点</p> <img src="/posts/docker配置TLS认证开启远程访问/pic1.png"> </li> <li> <p>配置节点属性</p> <img src="/posts/docker配置TLS认证开启远程访问/pic2.png"> <blockquote> <p>需要上传的文件对应为（以下目录在远程docker服务器上）</p> <ul> <li>TLS CA certificate: <code>~/.docker/ca.pem</code></li> <li>TLS certificate: <code>~/.docker/cert.pem</code></li> <li>TLS key: <code>~/.docker/key.pem</code></li> </ul> </blockquote> </li></ul><p>点击<code>Add endpoint</code>，完成添加。</p><p>关于怎么安装 <strong>portainer</strong>，这里就不介绍了，详细内容参考<a href="https://www.portainer.io/install.html" target="_blank" rel="noopener">官方文档</a></p>]]></content>
<summary type="html">
<img src="/posts/docker配置TLS认证开启远程访问/pic0.png">
</summary>
<category term="docker" scheme="https://tankeryang.github.io/categories/docker/"/>
<category term="docker" scheme="https://tankeryang.github.io/tags/docker/"/>
<category term="portainer" scheme="https://tankeryang.github.io/tags/portainer/"/>
</entry>
<entry>
<title>安装Ubuntu 18.04 LTS之后的N件事</title>
<link href="https://tankeryang.github.io/posts/%E5%AE%89%E8%A3%85Ubuntu-18-04-LTS%E4%B9%8B%E5%90%8E%E7%9A%84N%E4%BB%B6%E4%BA%8B/"/>
<id>https://tankeryang.github.io/posts/安装Ubuntu-18-04-LTS之后的N件事/</id>
<published>2018-06-24T23:51:06.000Z</published>
<updated>2019-10-09T17:06:09.623Z</updated>
<content type="html"><![CDATA[<img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic0.jpeg"><p>这两天装机，记录下装完系统后的一些后续事项<br><a id="more"></a></p><h1 id="软件和更新"><a href="#软件和更新" class="headerlink" title="软件和更新"></a>软件和更新</h1><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic1.png"><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic2.png"><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic3.png"><p>显卡驱动换成官方的之后可以重启一下，执行下面的命令看看是否安装成功:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">nvidia-smi</span><br></pre> </td> </tr> </table></figure><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic4.png"><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">ubuntu-drivers devices</span><br></pre> </td> </tr> </table></figure><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic5.png"><h1 id="设置sudo无密码"><a href="#设置sudo无密码" class="headerlink" title="设置sudo无密码"></a>设置sudo无密码</h1><p>执行:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo visudo</span><br></pre> </td> </tr> </table></figure><p>找到</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">%</span><span class="bash">sudo ALL=(ALL:ALL) ALL</span></span><br></pre> </td> </tr> </table></figure><p>改成</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">%</span><span class="bash">sudo ALL=(ALL:ALL) NOPASSWD:ALL</span></span><br></pre> </td> </tr> </table></figure><h1 id="安装neofetch"><a href="#安装neofetch" class="headerlink" title="安装neofetch"></a>安装neofetch</h1><p>添加<code>PPA</code>:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo add-apt-repository ppa:dawidd0811/neofetch</span><br></pre> </td> </tr> </table></figure><p>更新<code>apt</code>软件列表索引，并下载<code>neofetch</code>:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo apt update && sudo apt install neofetch</span><br></pre> </td> </tr> </table></figure><p>执行:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">neofetch</span><br></pre> </td> </tr> </table></figure><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic6.png"><h1 id="安装gnome-tweak"><a href="#安装gnome-tweak" class="headerlink" title="安装gnome-tweak"></a>安装gnome-tweak</h1><p>执行:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo apt install gnome-tweak-tool</span><br></pre> </td> </tr> </table></figure><p>装完后 <strong>程序中文名</strong> 叫<code>优化</code></p><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic7.png"><h1 id="安装Opera浏览器（针对博主本人，本人常用浏览器）"><a href="#安装Opera浏览器（针对博主本人，本人常用浏览器）" class="headerlink" title="安装Opera浏览器（针对博主本人，本人常用浏览器）"></a>安装Opera浏览器（针对博主本人，本人常用浏览器）</h1><p>去官网下<code>deb</code>包安装</p><h1 id="安装主题和图标"><a href="#安装主题和图标" class="headerlink" title="安装主题和图标"></a>安装主题和图标</h1><p>ubuntu 18.04 将桌面环境换成了 <strong>Gnome</strong>，所以我们的主题是基于 <strong>Gnome/Gtk</strong> 的</p><p>上<a href="https://www.gnome-look.org/browse/cat/135/ord/latest/" target="_blank" rel="noopener">这里</a>挑一款自己喜欢的主题:</p><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic8.png"><p>下面介绍3种安装主题和图标的方法</p><h2 id="通过apt安装"><a href="#通过apt安装" class="headerlink" title="通过apt安装"></a>通过apt安装</h2><p>这种方法需要所下载的主题或图标 <strong>有加入官方PPA源</strong> 或者 <strong>有提供第三方PPA源</strong><br>下面以<a href="https://www.gnome-look.org/p/1013714/" target="_blank" rel="noopener">macOS High Sierra</a>这款主题为例说明:</p><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic9.png"><p>看到它标题下方的 <strong>github地址</strong>: <a href="https://github.com/vinceliuice/Sierra-gtk-theme" target="_blank" rel="noopener">https://github.com/vinceliuice/Sierra-gtk-theme</a>:</p><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic10.png"><p>根据<code>README</code>进行安装:</p><ul> <li>Ubuntu PPA (maintained by @igor-dyatlov): <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo add-apt-repository ppa:dyatlov-igor/sierra-theme</span><br><span class="line">sudo apt update</span><br><span class="line">sudo apt install sierra-gtk-theme # point releases</span><br><span class="line">sudo apt install sierra-gtk-theme-git # git master branch</span><br></pre> </td> </tr> </table> </figure> </li></ul><h2 id="通过ocs-url-tools安装"><a href="#通过ocs-url-tools安装" class="headerlink" title="通过ocs-url-tools安装"></a>通过ocs-url-tools安装</h2><p> 我们可以看到，每个主题页面右边都有提供<code>Download</code>和<code>Install</code>，但是点完<code>Install</code>却发现浏览器会弹出询问 <strong>如何处理ocs: 链接</strong>，但是你点了执行却什么都没发生。这是因为你还需要安装 <strong>能处理ocs链接的工具</strong></p><p>首先，访问<a href="https://www.linux-apps.com/p/1136805/" target="_blank" rel="noopener">这里</a>，下载<a href="https://www.linux-apps.com/p/1136805/startdownload?file_id=1517920714&file_name=ocs-url_3.0.3-0ubuntu1_amd64.deb&file_type=application/x-debian-package&file_size=54198&url=https%3A%2F%2Fdl.opendesktop.org%2Fapi%2Ffiles%2Fdownloadfile%2Fid%2F1517920714%2Fs%2F976aa85dcaba33fafa7ee0c3074f03ad%2Ft%2F1529946099%2Fu%2F%2Focs-url_3.0.3-0ubuntu1_amd64.deb" target="_blank" rel="noopener">ocs-url_3.0.3-0ubuntu1_amd64.deb</a>（你可以直接点这个链接下载），双击运行安装</p><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic11.png"><p>回到下载主题的地方，以<a href="https://www.gnome-look.org/p/1215199/" target="_blank" rel="noopener">Arrongin themes</a>这款主题为例:</p><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic12.png"><p>直接点<code>Install</code>安装，选择某个系列（如: <code>Telinkrin-Buttons-Right.tar.xz</code>），浏览器会弹出提示，选择<code>启动应用程序</code>，弹出如下提示框，点<code>ok</code>安装:</p><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic13.png"><p> 装完之后点击<code>open</code>就是打开安装位置，一般在<code>~/.theme</code>下；点击<code>close</code>就是关闭窗口</p><h2 id="通过下载压缩包解压安装"><a href="#通过下载压缩包解压安装" class="headerlink" title="通过下载压缩包解压安装"></a>通过下载压缩包解压安装</h2><p>安装主题或图标实质上只是把 <strong>主题文件</strong> 或 <strong>图标文件</strong> 解压到制定文件夹下，因此我们只要下载某款主题或图标的压缩文件，然后分别解压到下面的位置:</p><ul> <li>主题文件夹: <code>~/.theme/</code></li> <li>图标文件夹: <code>~/.local/share/icons/</code></li></ul><h1 id="安装gnome-extension"><a href="#安装gnome-extension" class="headerlink" title="安装gnome-extension"></a>安装gnome-extension</h1><p>ubuntu 18.04 将gnome-shell-theme变成了一个gnome-extension插件，导致gnome-tweak不能直接换shell的主题，因此我们要先安装gnome-extension，下面说明详细步骤</p><ul> <li> <p>先本地安装支持下载gnome-extension的模块:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo apt install chrome-gnome-shell</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>访问<a href="https://extensions.gnome.org/" target="_blank" rel="noopener">gnome插件商店</a>，点开<a href="https://extensions.gnome.org/extension/19/user-themes/" target="_blank" rel="noopener">User theme</a>插件页面，点击<code>Click here to install browser extension</code>安装浏览器端的插件下载模块</p> </li> <li> <p>刷新一下页面，你就能看到右方有个开关按钮，点开就能自动安装插件了。关闭就是禁用插件</p> </li></ul><p>现在你不仅可以下载gnome-shell-theme插件，还能下载安装商店里的很多插件了</p><h1 id="安装输入法"><a href="#安装输入法" class="headerlink" title="安装输入法"></a>安装输入法</h1><p>执行如下命令安装fcitx和googlepinyin:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo apt install fcitx fcitx-googlepinyin im-config</span><br></pre> </td> </tr> </table></figure><p>执行:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">im-config</span><br></pre> </td> </tr> </table></figure><p>指定fcitx的配置即可</p><h1 id="安装git-zsh-oh-my-zsh"><a href="#安装git-zsh-oh-my-zsh" class="headerlink" title="安装git/zsh/oh-my-zsh"></a>安装git/zsh/oh-my-zsh</h1><ul> <li> <p>安装<code>git</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo apt install git</span><br></pre> </td> </tr> </table> </figure> <p> 顺便配置ssh:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line">cd ~/.ssh</span><br><span class="line">ssh-keygen -C "${你的git仓库(如github)的email}"</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>安装<code>zsh</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo apt install zsh</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>安装<code>oh-my-zsh</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> wget</span></span><br><span class="line">sh -c "$(wget https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh -O -)"</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> curl</span></span><br><span class="line">sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"</span><br></pre> </td> </tr> </table> </figure> </li></ul><h1 id="安装pip"><a href="#安装pip" class="headerlink" title="安装pip"></a>安装pip</h1><ul> <li> <p><code>pip2</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> 安装pip2</span></span><br><span class="line">sudo apt update</span><br><span class="line">sudo apt install python-pip</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看版本</span></span><br><span class="line">pip --verion</span><br><span class="line"><span class="meta">#</span><span class="bash"> 输出: pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)</span></span><br></pre> </td> </tr> </table> </figure> </li> <li> <p><code>pip3</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> 安装pip3</span></span><br><span class="line">sudo apt update</span><br><span class="line">sudo apt install python3-pip</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看版本</span></span><br><span class="line">pip3 --verion</span><br><span class="line"><span class="meta">#</span><span class="bash"> 输出: pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)</span></span><br></pre> </td> </tr> </table> </figure> </li></ul><h1 id="安装vim-tmux"><a href="#安装vim-tmux" class="headerlink" title="安装vim/tmux"></a>安装vim/tmux</h1><ul> <li> <p><code>vim</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo apt install vim</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p><code>tmux</code></p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo apt install tmux</span><br></pre> </td> </tr> </table> </figure> </li></ul><h1 id="安装powerline终端美化和字体库"><a href="#安装powerline终端美化和字体库" class="headerlink" title="安装powerline终端美化和字体库"></a>安装powerline终端美化和字体库</h1><h2 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h2><ul> <li> <p>先安装<code>powerline</code>插件:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">pip install --user powerline-status</span><br></pre> </td> </tr> </table> </figure> <p> 这是一个Python的模块，安装位置一般在<code>~/.local/lib/python2.7/site-packages</code> </p> <p> 装完之后，将<code>~/.local/bin</code>下的<code>powerline</code>相关的可执行脚本复制到<code>~/.local/lib/python2.7/site-packages/scripts</code>下: </p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> 先新建scripts文件夹</span></span><br><span class="line">mkdir ~/.local/lib/python2.7/site-packages/scripts</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 拷贝脚本到scripts下</span></span><br><span class="line">cp ~/.local/bin/powerline* ~/.local/lib/python2.7/site-packages/scripts</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>再安装<code>powerline</code>字体</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> <span class="built_in">clone</span></span></span><br><span class="line">git clone https://github.com/powerline/fonts.git --depth=1</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> install</span></span><br><span class="line">cd fonts</span><br><span class="line">./install.sh</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> clean-up a bit</span></span><br><span class="line">cd ..</span><br><span class="line">rm -rf fonts</span><br></pre> </td> </tr> </table> </figure> </li></ul><h2 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h2><p>再下面的文件添加对应配置</p><ul> <li> <p><code>~/.zshrc</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> 开头添加</span></span><br><span class="line">. /home/${填你的用户名}/.local/lib/python2.7/site-packages/powerline/bindings/zsh/powerline.zsh</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p><code>~/.vimrc</code>:</p> <figure class="highlight vim"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">set</span> <span class="keyword">shell</span>=/bin/zsh</span><br><span class="line"><span class="keyword">set</span> rtp+=/home/${填你的用户名}/.local/lib/python2.<span class="number">7</span>/site-packages/powerline/bindings/<span class="keyword">vim</span></span><br><span class="line"><span class="keyword">set</span> guifont=Source\ Code\ Pro\ <span class="keyword">for</span>\ Powerline:h14</span><br><span class="line"><span class="keyword">set</span> t_Co=<span class="number">256</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:Powerline_symbols</span> = <span class="string">'fancy'</span></span><br><span class="line"><span class="keyword">set</span> fillchars+=<span class="keyword">st</span><span class="variable">l:</span>\ ,stlnc:\</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p><code>~/.tmux.conf</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">source "/home/${填你的用户名}/.local/lib/python2.7/site-packages/powerline/bindings/tmux/powerline.conf"</span><br></pre> </td> </tr> </table> </figure> </li></ul><h1 id="安装dotfile（本人的配置文件，针对博主本人，有兴趣可以参考）"><a href="#安装dotfile（本人的配置文件，针对博主本人，有兴趣可以参考）" class="headerlink" title="安装dotfile（本人的配置文件，针对博主本人，有兴趣可以参考）"></a>安装dotfile（本人的配置文件，针对博主本人，有兴趣可以参考）</h1><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> <span class="built_in">clone</span></span></span><br><span class="line">git clone git@github.com:tankeryang/dotfile.git</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 切换分支</span></span><br><span class="line">git checkout -b ubuntu-18.04 origin/ubuntu-18.04</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 复制配置文件</span></span><br><span class="line">cd dotfile</span><br><span class="line">cp .zshrc .vimrc .tmux.conf ~/</span><br><span class="line">cp .ssh/config ~/.ssh/</span><br></pre> </td> </tr> </table></figure><h1 id="安装ShadowSocks-Qt5-SwitchyOmega"><a href="#安装ShadowSocks-Qt5-SwitchyOmega" class="headerlink" title="安装ShadowSocks-Qt5/SwitchyOmega"></a>安装ShadowSocks-Qt5/SwitchyOmega</h1><p>访问<a href="https://github.com/shadowsocks/shadowsocks-qt5/releases" target="_blank" rel="noopener">这里</a>下载 <strong>release</strong> 版本的<code>ShadowSocks-Qt5</code>可执行程序<a href="https://github.com/shadowsocks/shadowsocks-qt5/releases/download/v3.0.1/Shadowsocks-Qt5-3.0.1-x86_64.AppImage" target="_blank" rel="noopener">Shadowsocks-Qt5-3.0.1-x86_64.AppImage</a>，或者直接<code>wget</code>下载:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">wget https://github.com/shadowsocks/shadowsocks-qt5/releases/download/v3.0.1/Shadowsocks-Qt5-3.0.1-x86_64.AppImage</span><br></pre> </td> </tr> </table></figure><p>顺便搞一张<code>icon</code>图标:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">wget https://avatars1.githubusercontent.com/u/3006190?s=200&v=4 shadowsocks.png</span><br></pre> </td> </tr> </table></figure><p>将下载的可执行程序 <strong>提权</strong>，与图标一起拷贝到<code>/opt/ShadowSocks-Qt5</code>:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> 提权</span></span><br><span class="line">chmod a+x ${你的下载路径}/Shadowsocks-Qt5-3.0.1-x86_64.AppImage</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 将可执行程序放到/opt/ShadowSocks-Qt5</span></span><br><span class="line">sudo mkdir /opt/ShadowSocks-Qt5</span><br><span class="line">mv ${你的下载路径}/Shadowsocks-Qt5-3.0.1-x86_64.AppImage /opt/ShadowSocks-Qt5</span><br><span class="line">mv ${你的下载路径}/shaodwsocks.png /opt/ShadowSocks-Qt5</span><br></pre> </td> </tr> </table></figure><p>建立启动器</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> 进入启动器文件夹</span></span><br><span class="line">cd /usr/share/applications</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 新建desktop文件</span></span><br><span class="line">sudo vim ShadowSocks.desktop</span><br></pre> </td> </tr> </table></figure><p>写入如下内容</p><figure class="highlight ini"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="section">[Desktop Entry]</span></span><br><span class="line"><span class="attr">Name</span>=ShadowSocks</span><br><span class="line"><span class="attr">Comment</span>=ShadowSocks</span><br><span class="line"><span class="attr">Type</span>=Application</span><br><span class="line"><span class="attr">Exec</span>=/opt/ShadowSocks-Qt5/Shadowsocks-Qt5-<span class="number">3.0</span>.<span class="number">1</span>-x<span class="number">86_64</span>.AppImage</span><br><span class="line"><span class="attr">Icon</span>=/opt/ShadowSocks-Qt5/shadowsocks.png</span><br><span class="line"><span class="attr">Terminal</span>=<span class="literal">false</span></span><br><span class="line"><span class="attr">StartupNotify</span>=<span class="literal">true</span></span><br><span class="line"><span class="attr">Categories</span>=Application<span class="comment">;</span></span><br></pre> </td> </tr> </table></figure><p>保存退出，你应该在应用程序里能见到Shadowsocks的启动器图标了</p><p>然后安装浏览器代理插件<a href="https://github.com/FelisCatus/SwitchyOmega/releases" target="_blank" rel="noopener">SwitchyOmega</a>:</p><ul> <li> <p>点击下载<code>crx</code>文件:<a href="https://github.com/FelisCatus/SwitchyOmega/releases/download/v2.5.15/SwitchyOmega.crx" target="_blank" rel="noopener">SwitchyOmega.crx</a>，然后将其拖到浏览器中安装 </p> </li> <li> <p>用<code>wget</code>下载:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">wget https://github.com/FelisCatus/SwitchyOmega/releases/download/v2.5.15/SwitchyOmega.crx</span><br></pre> </td> </tr> </table> </figure> <p> 然后拖到浏览器安装</p> </li></ul><p>接下来配置<code>SwitchyOmega</code>，参考<a href="https://www.sundabao.com/ubuntu使用shadowsocks/" target="_blank" rel="noopener">这篇博客</a></p><h1 id="安装Shutter截图软件"><a href="#安装Shutter截图软件" class="headerlink" title="安装Shutter截图软件"></a>安装Shutter截图软件</h1><ul> <li> <p>添加源，并安装 <strong>shutter</strong>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo add-apt-repository ppa:shutter/ppa</span><br><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install shutter</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>设置快捷键</p> </li></ul><img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic14.png"><h1 id="安装微信"><a href="#安装微信" class="headerlink" title="安装微信"></a>安装微信</h1><p>访问<a href="https://github.com/geeeeeeeeek/electronic-wechat/releases" target="_blank" rel="noopener">这里</a>下载 <strong>release</strong> 64位版本的<a href="https://github.com/geeeeeeeeek/electronic-wechat/releases/download/V2.0/linux-x64.tar.gz" target="_blank" rel="noopener">linux-x64.tar.gz</a>，或者用<code>wget</code>:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">wget https://github.com/geeeeeeeeek/electronic-wechat/releases/download/V2.0/linux-x64.tar.gz</span><br></pre> </td> </tr> </table></figure><p>下载完后解压至<code>/opt</code>文件夹下:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo tar -xzvf ${你的下载路径}/linux-x64.tar.gz -C /opt</span><br></pre> </td> </tr> </table></figure><p>顺便搞一张<a href="https://images2018.cnblogs.com/blog/1127869/201806/1127869-20180602105354254-1327395543.png" target="_blank" rel="noopener">图标</a>:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">wget -O wechat.png https://images2018.cnblogs.com/blog/1127869/201806/1127869-20180602105354254-1327395543.png</span><br></pre> </td> </tr> </table></figure><p>将图标文件放到刚刚解压的路径:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo mv ${你的下载路径}/wechat.png /opt/electronic-wechat-linux-x64/resources</span><br></pre> </td> </tr> </table></figure><p>建立启动器:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> 进入启动器文件夹</span></span><br><span class="line">cd /usr/share/applications</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 新建desktop文件</span></span><br><span class="line">sudo vim wechat.desktop</span><br></pre> </td> </tr> </table></figure><p>填入以下内容:</p><figure class="highlight ini"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="section">[Desktop Entry]</span></span><br><span class="line"><span class="attr">Name</span>=wechat</span><br><span class="line"><span class="attr">Comment</span>=wechat</span><br><span class="line"><span class="attr">Type</span>=Application</span><br><span class="line"><span class="attr">Exec</span>=/opt/electronic-wechat-linux-x64/electronic-wechat</span><br><span class="line"><span class="attr">Icon</span>=/opt/electronic-wechat-linux-x64/resources/wechat.png</span><br><span class="line"><span class="attr">Terminal</span>=<span class="literal">false</span></span><br><span class="line"><span class="attr">StartupNotify</span>=<span class="literal">true</span></span><br><span class="line"><span class="attr">Categories</span>=Application<span class="comment">;</span></span><br></pre> </td> </tr> </table></figure><p>赶快去试试吧～</p><h1 id="安装VSCode"><a href="#安装VSCode" class="headerlink" title="安装VSCode"></a>安装VSCode</h1><p>官网下载<code>deb</code>包安装</p><p>安装后将<code>dotfile</code>里的<code>vscode</code>配置copy到相应位置（针对博主本人，有兴趣可以参考）:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">cp ~/dotfile/.vscode/* ~/.config/Code/User/</span><br></pre> </td> </tr> </table></figure><h1 id="安装java"><a href="#安装java" class="headerlink" title="安装java"></a>安装java</h1><ul> <li> <p>添加源，更新:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo add-apt-repository ppa:webupd8team/java</span><br><span class="line">sudo apt update</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>安装:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> java8</span></span><br><span class="line">sudo apt install oracle-java8-set-default</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash">java9</span></span><br><span class="line">sudo apt install oracle-java9-set-default</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>管理java/javac:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> java</span></span><br><span class="line">sudo update-alternatives --config java</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> javac</span></span><br><span class="line">sudo update-alternatives --config javac</span><br></pre> </td> </tr> </table> </figure> <p> 输出，自行选择版本:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre> </td> <td class="code"> <pre><span class="line">有 1 个候选项可用于替换 java (提供 /usr/bin/java)。</span><br><span class="line"></span><br><span class="line">选择 路径 优先级 状态</span><br><span class="line">------------------------------------------------------------</span><br><span class="line"> 0 /usr/lib/jvm/java-8-oracle/jre/bin/java 1081 自动模式</span><br><span class="line">* 1 /usr/lib/jvm/java-8-oracle/jre/bin/java 1081 手动模式</span><br><span class="line"></span><br><span class="line">要维持当前值[*]请按<回车键>，或者键入选择的编号：</span><br></pre> </td> </tr> </table> </figure> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre> </td> <td class="code"> <pre><span class="line">有 1 个候选项可用于替换 javac (提供 /usr/bin/javac)。</span><br><span class="line"></span><br><span class="line"> 选择 路径 优先级 状态</span><br><span class="line">------------------------------------------------------------</span><br><span class="line"> 0 /usr/lib/jvm/java-8-oracle/bin/javac 1081 自动模式</span><br><span class="line">* 1 /usr/lib/jvm/java-8-oracle/bin/javac 1081 手动模式</span><br><span class="line"></span><br><span class="line">要维持当前值[*]请按<回车键>，或者键入选择的编号：</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>配置<code>JAVA_HOME</code>:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">sudo vim /etc/environment</span><br></pre> </td> </tr> </table> </figure> <p> 填入:</p> <figure class="highlight ini"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="attr">JAVA_HOME</span>=<span class="string">"/usr/lib/jvm/java-8-oracle/jre/bin/java"</span></span><br></pre> </td> </tr> </table> </figure> <p> 使其生效:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">source /etc/environment</span><br></pre> </td> </tr> </table> </figure> <p> 检查是否生效:</p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">echo $JAVA_HOME</span><br></pre> </td> </tr> </table> </figure> </li></ul><h1 id="安装Anaconda"><a href="#安装Anaconda" class="headerlink" title="安装Anaconda"></a>安装Anaconda</h1><p>访问<a href="https://mirrors4.tuna.tsinghua.edu.cn/anaconda/archive/" target="_blank" rel="noopener">这里</a>，下载你想要的anaconda版本，或者直接<code>wget</code>:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> Anaconda3-5.2.0-Linux-x86_64.sh</span></span><br><span class="line">wget https://mirrors4.tuna.tsinghua.edu.cn/anaconda/archive/Anaconda3-5.2.0-Linux-x86_64.sh</span><br></pre> </td> </tr> </table></figure><p> 然后执行脚本安装，安装位置推荐放在<code>/opt</code>或<code>/usr/local</code>下，注意不要添加环境变量，以免与系统python冲突<br>安装完后将<code>anaconda/bin</code>下的可执行程序软连接到<code>/usr/local/bin</code>下，比如:</p><figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> conda</span></span><br><span class="line">sudo ln -s /opt/anaconda3/bin/conda /usr/local/bin/conda</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> python</span></span><br><span class="line">sudo ln -s /opt/anaconda3/bin/python /usr/local/bin/python36 # 命名python36是为了不与系统python冲突</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> jupyter</span></span><br><span class="line">sudo ln -s /opt/anaconda3/bin/jupyter /usr/local/bin/jupyter</span><br></pre> </td> </tr> </table></figure><p>终端执行一下<code>conda</code>，<code>python36</code>，<code>jupyter</code>，看有没有问题</p><h1 id="安装JetBrain系列"><a href="#安装JetBrain系列" class="headerlink" title="安装JetBrain系列"></a>安装JetBrain系列</h1><p>官网下载<code>deb</code>包安装</p>]]></content>
<summary type="html">
<img src="/posts/安装Ubuntu-18-04-LTS之后的N件事/pic0.jpeg">
<p>这两天装机，记录下装完系统后的一些后续事项<br>
</summary>
<category term="Ubuntu" scheme="https://tankeryang.github.io/categories/Ubuntu/"/>
<category term="linux" scheme="https://tankeryang.github.io/tags/linux/"/>
<category term="ubuntu" scheme="https://tankeryang.github.io/tags/ubuntu/"/>
</entry>
<entry>
<title>半次元爬虫 banciyuan-downloader v2.0 更新</title>
<link href="https://tankeryang.github.io/posts/%E5%8D%8A%E6%AC%A1%E5%85%83%E7%88%AC%E8%99%AB-banciyuan-downloader-v2-0-%E6%9B%B4%E6%96%B0/"/>
<id>https://tankeryang.github.io/posts/半次元爬虫-banciyuan-downloader-v2-0-更新/</id>
<published>2018-06-20T11:01:33.000Z</published>
<updated>2019-10-09T17:06:09.623Z</updated>
<content type="html"><![CDATA[<img src="/posts/半次元爬虫-banciyuan-downloader-v2-0-更新/pic0.png"><a id="more"></a><hr><p>这段时间<a href="https://bcy.net" target="_blank" rel="noopener">半次元</a>前端做了很大的改动，先后更新了两版，之前的小版本<code>v1.1</code>也不能用了，索性更新了<a href="https://github.com/tankeryang/banciyuan-downloader" target="_blank" rel="noopener">banciyuan-downloader v2.0</a>版本，做了很大的改变，代码全部封装到<code>Downloader</code>类里，下面是更新内容</p><h1 id="banciyuan-downloader-v2-0"><a href="#banciyuan-downloader-v2-0" class="headerlink" title="banciyuan-downloader v2.0"></a>banciyuan-downloader v2.0</h1><ul> <li>更新时间: <code>2018-06-18</code></li></ul><h2 id="更新原因"><a href="#更新原因" class="headerlink" title="更新原因"></a>更新原因</h2><ul> <li>半次元前端改版，需要重写脚本以适应HTML解析</li> <li>半次元登录认证改为通过cookie表单传递认证，需重写登录方法</li> <li>脚本太乱，需进行封装</li></ul><h2 id="更新内容"><a href="#更新内容" class="headerlink" title="更新内容"></a>更新内容</h2><h3 id="模块化"><a href="#模块化" class="headerlink" title="模块化"></a>模块化</h3><p>将方法封装到类<code>Downloader</code>里，优化接口方法，可提供外部调用</p><h3 id="实现功能"><a href="#实现功能" class="headerlink" title="实现功能"></a>实现功能</h3><ul> <li style="list-style: none"><input type="checkbox" checked> 根据<code>coser_id</code>批量下载某个coser发布的作品的所有图片</li> <li style="list-style: none"><input type="checkbox" checked> 图片保存在以 <strong>coser名</strong> 命名的文件夹内</li> <li style="list-style: none"><input type="checkbox" checked> 图片按 <strong>coser发布的作品</strong> 分文件夹保存，文件夹命名以 <strong>作品标签</strong> 拼接而成</li> <li style="list-style: none"><input type="checkbox" checked> 若有相同标题的作品，则命名文件夹时会加上 <strong>顺序编号后缀</strong> 防止文件名冲突</li> <li style="list-style: none"><input type="checkbox" checked> 图片命名格式为<code>%num%.jpg</code>/<code>%num%.png</code>，其中<code>%num%</code>为从<code>1</code>开始的 <strong>编号</strong></li> <li style="list-style: none"><input type="checkbox" checked> 支持智能下载 <strong>本地没有的作品</strong>，本地已有的作品 <strong>不会</strong> 重复下载</li> <li style="list-style: none"><input type="checkbox" checked> 支持 <strong>断点续传</strong> （从断连作品的下一个作品开始下载，若断连作品没下载完，则会丢失一部分断连作品的图片，其余均不影响）</li> <li style="list-style: none"><input type="checkbox" checked> 支持超时自动重试 </li> <li style="list-style: none"><input type="checkbox" checked> 支持下载指定作品 </li> <li style="list-style: none"><input type="checkbox" checked> 根据频繁I/O进行 <strong>多线程优化</strong></li> <li style="list-style: none"><input type="checkbox"> 不支持无半次元账号的下载，因为有 <strong>只有粉丝可见</strong> 的限制，最好注册一个半次元账号</li> <li style="list-style: none"><input type="checkbox"> 只有粉丝可见的作品需关注该coser后方可下载</li> <li style="list-style: none"><input type="checkbox"> 暂时不支持只下载<code>COS</code>类的作品，因为半次元改版后没有对<code>COS</code>和<code>绘画</code>之类的做分类，都在同一<code>url</code>下 </li></ul><h3 id="依赖的库"><a href="#依赖的库" class="headerlink" title="依赖的库"></a>依赖的库</h3><p>我的测试环境:</p><ul> <li>python 3.6.4</li> <li>beautifulsoup 4.5.3</li> <li>requests 2.13.0</li> <li>lxml 3.7.2</li></ul><h3 id="如何使用"><a href="#如何使用" class="headerlink" title="如何使用"></a>如何使用</h3><ul> <li> <p>直接运行<code>run.py</code></p> <p> 先执行:</p> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">python run.py</span><br></pre> </td> </tr> </table> </figure> <p> 后续步骤参考<a href="https://github.com/tankeryang/banciyuan-downloader#usage" target="_blank" rel="noopener">v1.0的Usage</a> </p> </li> <li> <p>自定义实现</p> <p> 因为进行了 <strong>模块化</strong>，所以可以自行实例化一个<code>Downloader</code>对象来调用方法实现功能，下面给出简单例子: </p> <p> <strong>分部执行</strong></p> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> bcy_downloader <span class="keyword">import</span> Downloader</span><br><span class="line"></span><br><span class="line"><span class="comment"># 实例化Downloader对象</span></span><br><span class="line"><span class="comment"># 用户名: test</span></span><br><span class="line"><span class="comment"># 密码: 123</span></span><br><span class="line"><span class="comment"># coser_id: 770554</span></span><br><span class="line"><span class="comment"># 下载目录: E:/banciyuan</span></span><br><span class="line">dl = Downloader(account=<span class="string">'test'</span>, password=<span class="string">'123'</span>, coser_id=<span class="string">'770554'</span>, bcy_home_dir=<span class="string">'E:/banciyuan'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 获取作品url列表</span></span><br><span class="line">dl.get_post_url_list()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 或者自定义下载作品列表</span></span><br><span class="line">dl.post_url_list = [<span class="string">"https://bcy.net/item/detail/6558754255610577155"</span>, <span class="string">"https://bcy.net/item/detail/6554677621064466692"</span>]</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看作品url列表</span></span><br><span class="line">print(dl.post_url_list)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看本地已下载作品url列表</span></span><br><span class="line">print(dl.local_post_url_list)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 获取每个作品下所有图片url，得到download_data</span></span><br><span class="line">dl.get_pics_url_list()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看download_data</span></span><br><span class="line"><span class="comment"># 格式如下</span></span><br><span class="line"><span class="comment"># {</span></span><br><span class="line"><span class="comment"># '$(post_url)':</span></span><br><span class="line"><span class="comment"># {</span></span><br><span class="line"><span class="comment"># 'post_name': $(post_name),</span></span><br><span class="line"><span class="comment"># 'pics_url_list': $(pics_url_list)</span></span><br><span class="line"><span class="comment"># }</span></span><br><span class="line"><span class="comment"># }</span></span><br><span class="line"><span class="comment"># 例子:</span></span><br><span class="line"><span class="comment"># {</span></span><br><span class="line"><span class="comment"># "https://bcy.net/item/detail/6558754255610577155":</span></span><br><span class="line"><span class="comment"># {</span></span><br><span class="line"><span class="comment"># 'post_name': "碧蓝航线-COS-舰娘-场照-返图-cp22-三三笠",</span></span><br><span class="line"><span class="comment"># 'pics_url_list': [</span></span><br><span class="line"><span class="comment"># 'https://img5.bcyimg.com/user/770554/item/c0je3/63y6vuq8hhgfmge7nrqcaqkpspyfszj5.jpg?1',</span></span><br><span class="line"><span class="comment"># 'https://img9.bcyimg.com/user/770554/item/c0je3/esdrtvchzzkfzm74ezd8idx04ennjjfr.jpg?2',</span></span><br><span class="line"><span class="comment"># ......</span></span><br><span class="line"><span class="comment"># 'https://img9.bcyimg.com/user/770554/item/c0je3/zfiyptpdcpasyz0itzh7ndtmpiysw8uy.jpg?9'</span></span><br><span class="line"><span class="comment"># ]</span></span><br><span class="line"><span class="comment"># }</span></span><br><span class="line"><span class="comment"># }</span></span><br><span class="line">print(dl.download_data)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 根据download_data获取图片</span></span><br><span class="line">dl.get_pics()</span><br></pre> </td> </tr> </table> </figure> <p> <strong>一键执行</strong></p> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> bcy_downloader <span class="keyword">import</span> Downloader</span><br><span class="line"></span><br><span class="line"><span class="comment"># 实例化Downloader对象</span></span><br><span class="line"><span class="comment"># 用户名: test</span></span><br><span class="line"><span class="comment"># 密码: 123</span></span><br><span class="line"><span class="comment"># coser_id: 770554</span></span><br><span class="line"><span class="comment"># 下载目录: E:/banciyuan</span></span><br><span class="line">dl = Downloader(account=<span class="string">'test'</span>, password=<span class="string">'123'</span>, coser_id=<span class="string">'770554'</span>, bcy_home_dir=<span class="string">'E:/banciyuan'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 自动下载</span></span><br><span class="line">dl.run()</span><br></pre> </td> </tr> </table> </figure> </li></ul><h2 id="注意"><a href="#注意" class="headerlink" title="注意"></a>注意</h2><p><code>v2.0</code>版本下载时会在 <strong>每个作品对应的文件夹</strong> 里建一个<code>url.local</code>文件，里面写入的是该作品对应的<code>url</code>。该文件主要用于 <strong>判断本地是否已经下载过该<code>url</code>对应的作品，以实现智能无重复下载，请不要随意删除</strong></p><h1 id="FAQ"><a href="#FAQ" class="headerlink" title="FAQ"></a>FAQ</h1><p>有何疑问可在<a href="https://github.com/tankeryang/banciyuan-downloader" target="_blank" rel="noopener">github</a>发布<a href="https://github.com/tankeryang/banciyuan-downloader/issues" target="_blank" rel="noopener">issue</a> ，本人会尽量及时查看</p>]]></content>
<summary type="html">
<img src="/posts/半次元爬虫-banciyuan-downloader-v2-0-更新/pic0.png">
</summary>
<category term="python" scheme="https://tankeryang.github.io/categories/python/"/>
<category term="python" scheme="https://tankeryang.github.io/tags/python/"/>
<category term="爬虫" scheme="https://tankeryang.github.io/tags/%E7%88%AC%E8%99%AB/"/>
<category term="二次元" scheme="https://tankeryang.github.io/tags/%E4%BA%8C%E6%AC%A1%E5%85%83/"/>
</entry>
<entry>
<title>python模块fabric踩坑记录</title>
<link href="https://tankeryang.github.io/posts/python%E6%A8%A1%E5%9D%97fabric%E8%B8%A9%E5%9D%91%E8%AE%B0%E5%BD%95/"/>
<id>https://tankeryang.github.io/posts/python模块fabric踩坑记录/</id>
<published>2018-06-11T16:50:41.000Z</published>
<updated>2019-10-09T17:06:09.615Z</updated>
<content type="html"><![CDATA[<img src="/posts/python模块fabric踩坑记录/pic0.jpeg"><p>最近都在狂写脚本，好像变成 <strong>半个运维</strong> 一样…<br>刚好有需求写些部署工具，于是了解到了<a href="http://www.fabfile.org/" target="_blank" rel="noopener">fabric</a>这个模块。下面简单带过一下</p><a id="more"></a><h1 id="什么是Fabric"><a href="#什么是Fabric" class="headerlink" title="什么是Fabric"></a>什么是Fabric</h1><p><strong>引用fabric主页的介绍</strong></p><blockquote> <p>Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return</p></blockquote><p>意思就是Fabric是基于SSH的远程执行命令，并返回可调用的python对象的框架<br> <div class="note info"> <p> Fabric<code>2.x</code>的版本与<code>1.x</code>相比，除了支持<code>python3</code>之外，还做了很多改动。网上很多博客都写的是<code>1.x</code>的版本，参考时要注意。<br><strong>这里主要分析<code>2.x</code>的版本</strong> </p> </div></p><p>安装什么的就不废话了，下面来用一下<br>参照他的示例:<br> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> fabric <span class="keyword">import</span> Connection</span><br><span class="line"><span class="meta">>>> </span>result = Connection(<span class="string">'web1.example.com'</span>).run(<span class="string">'uname -s'</span>)</span><br><span class="line"><span class="meta">>>> </span>msg = <span class="string">"Ran {.command!r} on {.host}, got this stdout:\n{.stdout}"</span></span><br><span class="line"><span class="meta">>>> </span>print(msg.format(result))</span><br><span class="line"></span><br><span class="line">Ran <span class="string">"uname -s"</span> on web1.example.com, got this stdout:</span><br><span class="line">Linux</span><br></pre> </td> </tr> </table> </figure></p><p>一目了然</p><p>下面在公司服务器上测试一下<br> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># 服务器ip: 10.10.22.13</span></span><br><span class="line"><span class="comment"># 用户: root</span></span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> fabric <span class="keyword">import</span> Connection</span><br><span class="line"><span class="meta">>>> </span>result = Connection(<span class="string">'10.10.22.13'</span>, user=<span class="string">'root'</span>).run(<span class="string">'uname -s'</span>)</span><br><span class="line"><span class="meta">>>> </span>msg = <span class="string">"Ran {.command!r} on {.host}, got this stdout:\n{.stdout}"</span></span><br><span class="line"><span class="meta">>>> </span>print(msg.format(result))</span><br><span class="line"></span><br><span class="line">Traceback (most recent call last):</span><br><span class="line"> File <span class="string">"/Users/yang/workspace/PycharmProjects/FP-project/inventory_allocate/test/fabric_test.py"</span>, line <span class="number">10</span>, <span class="keyword">in</span> <module></span><br><span class="line"> result = Connection(<span class="string">"10.10.22.13"</span>, user=<span class="string">'root'</span>).run(<span class="string">"uname -s"</span>)</span><br><span class="line"> File <span class="string">"<decorator-gen-3>"</span>, line <span class="number">2</span>, <span class="keyword">in</span> run</span><br><span class="line"> File <span class="string">"/Users/yang/anaconda3/lib/python3.6/site-packages/fabric/connection.py"</span>, line <span class="number">29</span>, <span class="keyword">in</span> opens</span><br><span class="line"> self.open()</span><br><span class="line"> File <span class="string">"/Users/yang/anaconda3/lib/python3.6/site-packages/fabric/connection.py"</span>, line <span class="number">501</span>, <span class="keyword">in</span> open</span><br><span class="line"> self.client.connect(**kwargs)</span><br><span class="line"> File <span class="string">"/Users/yang/anaconda3/lib/python3.6/site-packages/paramiko/client.py"</span>, line <span class="number">424</span>, <span class="keyword">in</span> connect</span><br><span class="line"> passphrase,</span><br><span class="line"> File <span class="string">"/Users/yang/anaconda3/lib/python3.6/site-packages/paramiko/client.py"</span>, line <span class="number">715</span>, <span class="keyword">in</span> _auth</span><br><span class="line"> <span class="keyword">raise</span> SSHException(<span class="string">'No authentication methods available'</span>)</span><br><span class="line">paramiko.ssh_exception.SSHException: No authentication methods available</span><br></pre> </td> </tr> </table> </figure></p><p>呵呵，我他妈就知道，代码里一行ssh的参数毛都没见到，这么容易连上就有鬼了（微笑<br>没的说，填坑要紧</p><hr><h1 id="源码分析"><a href="#源码分析" class="headerlink" title="源码分析"></a>源码分析</h1><p>分析报错<br> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre> </td> <td class="code"> <pre><span class="line">Traceback (most recent call last):</span><br><span class="line"> File <span class="string">"/Users/yang/workspace/PycharmProjects/FP-project/inventory_allocate/test/fabric_test.py"</span>, line <span class="number">10</span>, <span class="keyword">in</span> <module></span><br><span class="line"> result = Connection(<span class="string">"10.10.22.13"</span>, user=<span class="string">'root'</span>).run(<span class="string">"uname -s"</span>)</span><br><span class="line"> File <span class="string">"<decorator-gen-3>"</span>, line <span class="number">2</span>, <span class="keyword">in</span> run</span><br><span class="line"> File <span class="string">"/Users/yang/anaconda3/lib/python3.6/site-packages/fabric/connection.py"</span>, line <span class="number">29</span>, <span class="keyword">in</span> opens</span><br><span class="line"> self.open()</span><br><span class="line"> File <span class="string">"/Users/yang/anaconda3/lib/python3.6/site-packages/fabric/connection.py"</span>, line <span class="number">501</span>, <span class="keyword">in</span> open</span><br><span class="line"> self.client.connect(**kwargs)</span><br><span class="line"> File <span class="string">"/Users/yang/anaconda3/lib/python3.6/site-packages/paramiko/client.py"</span>, line <span class="number">424</span>, <span class="keyword">in</span> connect</span><br><span class="line"> passphrase,</span><br><span class="line"> File <span class="string">"/Users/yang/anaconda3/lib/python3.6/site-packages/paramiko/client.py"</span>, line <span class="number">715</span>, <span class="keyword">in</span> _auth</span><br><span class="line"> <span class="keyword">raise</span> SSHException(<span class="string">'No authentication methods available'</span>)</span><br><span class="line">paramiko.ssh_exception.SSHException: No authentication methods available</span><br></pre> </td> </tr> </table> </figure></p><p>我们可以看到调用堆栈上的错误回溯，定位到<code>line 501</code>，在实例化<code>Connection</code>对象后调用<code>client.connect(**kwargs)</code>时抽了…</p><p> 直接摸过去，从<code>Connection</code>类出发往他祖宗上刨，下面先给出继承关系和<code>Connection</code>的关键部分，然后逐块拆分说明:</p><img src="http://www.plantuml.com/plantuml/svg/oq_AIaqkKN19B4aCACglgEJAXmWuv-ULf5QK877JCjEHU7b0EZC_ZuiBhbF1faPN5wuFKnYa7oeB5uEQ0W00"> <figure class="highlight python"><figcaption><span>connection.py</span></figcaption><table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Connection</span><span class="params">(Context)</span>:</span></span><br><span class="line"> host = <span class="literal">None</span></span><br><span class="line"> original_host = <span class="literal">None</span></span><br><span class="line"> user = <span class="literal">None</span></span><br><span class="line"> port = <span class="literal">None</span></span><br><span class="line"> ssh_config = <span class="literal">None</span></span><br><span class="line"> gateway = <span class="literal">None</span></span><br><span class="line"> forward_agent = <span class="literal">None</span></span><br><span class="line"> connect_timeout = <span class="literal">None</span></span><br><span class="line"> connect_kwargs = <span class="literal">None</span></span><br><span class="line"> client = <span class="literal">None</span></span><br><span class="line"> transport = <span class="literal">None</span></span><br><span class="line"> _sftp = <span class="literal">None</span></span><br><span class="line"> _agent_handler = <span class="literal">None</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params"> self,</span></span></span><br><span class="line"><span class="function"><span class="params"> host,</span></span></span><br><span class="line"><span class="function"><span class="params"> user=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> port=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> config=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> gateway=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> forward_agent=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> connect_timeout=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> connect_kwargs=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> )</span>:</span></span><br><span class="line"> <span class="comment"># config</span></span><br><span class="line"> super(Connection, self).__init__(config=config)</span><br><span class="line"> <span class="keyword">if</span> config <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> config = Config()</span><br><span class="line"> <span class="keyword">elif</span> <span class="keyword">not</span> isinstance(config, Config):</span><br><span class="line"> config = config.clone(into=Config)</span><br><span class="line"> self._set(_config=config)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># host相关</span></span><br><span class="line"> shorthand = self.derive_shorthand(host)</span><br><span class="line"> host = shorthand[<span class="string">"host"</span>]</span><br><span class="line"> err = (</span><br><span class="line"> <span class="string">"You supplied the {} via both shorthand and kwarg! Please pick one."</span> <span class="comment"># noqa</span></span><br><span class="line"> )</span><br><span class="line"> <span class="keyword">if</span> shorthand[<span class="string">"user"</span>] <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">if</span> user <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">raise</span> ValueError(err.format(<span class="string">"user"</span>))</span><br><span class="line"> user = shorthand[<span class="string">"user"</span>]</span><br><span class="line"> <span class="keyword">if</span> shorthand[<span class="string">"port"</span>] <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">if</span> port <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">raise</span> ValueError(err.format(<span class="string">"port"</span>))</span><br><span class="line"> port = shorthand[<span class="string">"port"</span>]</span><br><span class="line"></span><br><span class="line"> <span class="comment"># ssh_config</span></span><br><span class="line"> self.ssh_config = self.config.base_ssh_config.lookup(host)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># original_host</span></span><br><span class="line"> self.original_host = host</span><br><span class="line"></span><br><span class="line"> <span class="comment"># host</span></span><br><span class="line"> self.host = host</span><br><span class="line"> <span class="keyword">if</span> <span class="string">"hostname"</span> <span class="keyword">in</span> self.ssh_config:</span><br><span class="line"> self.host = self.ssh_config[<span class="string">"hostname"</span>]</span><br><span class="line"></span><br><span class="line"> <span class="comment"># user</span></span><br><span class="line"> self.user = user <span class="keyword">or</span> self.ssh_config.get(<span class="string">"user"</span>, self.config.user)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># port</span></span><br><span class="line"> self.port = port <span class="keyword">or</span> int(self.ssh_config.get(<span class="string">"port"</span>, self.config.port))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> gateway <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">if</span> <span class="string">"proxyjump"</span> <span class="keyword">in</span> self.ssh_config:</span><br><span class="line"> hops = reversed(self.ssh_config[<span class="string">"proxyjump"</span>].split(<span class="string">","</span>))</span><br><span class="line"> prev_gw = <span class="literal">None</span></span><br><span class="line"> <span class="keyword">for</span> hop <span class="keyword">in</span> hops:</span><br><span class="line"> <span class="keyword">if</span> prev_gw <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> cxn = Connection(hop)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> cxn = Connection(hop, gateway=prev_gw)</span><br><span class="line"> prev_gw = cxn</span><br><span class="line"> gateway = prev_gw</span><br><span class="line"> <span class="keyword">elif</span> <span class="string">"proxycommand"</span> <span class="keyword">in</span> self.ssh_config:</span><br><span class="line"> gateway = self.ssh_config[<span class="string">"proxycommand"</span>]</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> gateway = self.config.gateway</span><br><span class="line"> self.gateway = gateway</span><br><span class="line"></span><br><span class="line"> <span class="comment"># forward_agent</span></span><br><span class="line"> <span class="keyword">if</span> forward_agent <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> forward_agent = self.config.forward_agent</span><br><span class="line"> <span class="keyword">if</span> <span class="string">"forwardagent"</span> <span class="keyword">in</span> self.ssh_config:</span><br><span class="line"> map_ = {<span class="string">"yes"</span>: <span class="literal">True</span>, <span class="string">"no"</span>: <span class="literal">False</span>}</span><br><span class="line"> forward_agent = map_[self.ssh_config[<span class="string">"forwardagent"</span>]]</span><br><span class="line"> self.forward_agent = forward_agent</span><br><span class="line"></span><br><span class="line"> <span class="comment"># connect_timeout</span></span><br><span class="line"> <span class="keyword">if</span> connect_timeout <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> connect_timeout = self.ssh_config.get(</span><br><span class="line"> <span class="string">"connecttimeout"</span>, self.config.timeouts.connect</span><br><span class="line"> )</span><br><span class="line"> <span class="keyword">if</span> connect_timeout <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> connect_timeout = int(connect_timeout)</span><br><span class="line"> self.connect_timeout = connect_timeout</span><br><span class="line"></span><br><span class="line"> <span class="comment"># connect_kwargs</span></span><br><span class="line"> self.connect_kwargs = self.resolve_connect_kwargs(connect_kwargs)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># client</span></span><br><span class="line"> client = SSHClient()</span><br><span class="line"> client.set_missing_host_key_policy(AutoAddPolicy())</span><br><span class="line"> self.client = client</span><br><span class="line"></span><br><span class="line"> <span class="comment"># transport</span></span><br><span class="line"> self.transport = <span class="literal">None</span></span><br></pre> </td> </tr></table></figure><h2 id="重要的成员变量"><a href="#重要的成员变量" class="headerlink" title="重要的成员变量"></a>重要的成员变量</h2><figure class="highlight python"> <figcaption><span>class Connection</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre> </td> <td class="code"> <pre><span class="line">host = <span class="literal">None</span> <span class="comment"># 主机名或IP地址: www.host.com, 66.66.66.66</span></span><br><span class="line">original_host = <span class="literal">None</span> <span class="comment"># 同host</span></span><br><span class="line">user = <span class="literal">None</span> <span class="comment"># 系统用户名: root, someone</span></span><br><span class="line">port = <span class="literal">None</span> <span class="comment"># 端口号（远程执行某些应用需提供）</span></span><br><span class="line">gateway = <span class="literal">None</span> <span class="comment"># 网关</span></span><br><span class="line">forward_agent = <span class="literal">None</span> <span class="comment"># 代理</span></span><br><span class="line">connect_timeout = <span class="literal">None</span> <span class="comment"># 超时时间</span></span><br><span class="line">connect_kwargs = <span class="literal">None</span> <span class="comment"># 连接参数（记住这个，非常重要）</span></span><br><span class="line">client = <span class="literal">None</span> <span class="comment"># 客户端</span></span><br></pre> </td> </tr> </table></figure><h2 id="构造函数参数"><a href="#构造函数参数" class="headerlink" title="构造函数参数"></a>构造函数参数</h2><figure class="highlight python"> <figcaption><span>Connection.__init__()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre> </td> <td class="code"> <pre><span class="line">host</span><br><span class="line">user=<span class="literal">None</span></span><br><span class="line">port=<span class="literal">None</span></span><br><span class="line">config=<span class="literal">None</span></span><br><span class="line">gateway=<span class="literal">None</span></span><br><span class="line">forward_agent=<span class="literal">None</span></span><br><span class="line">connect_timeout=<span class="literal">None</span></span><br><span class="line">connect_kwargs=<span class="literal">None</span></span><br></pre> </td> </tr> </table></figure><p> 这些就是我们在实例化<code>Connection</code>对象时可以控制的一些部分，比较重要的有<code>config</code>和<code>connection_kwargs</code></p><h2 id="构造函数主体"><a href="#构造函数主体" class="headerlink" title="构造函数主体"></a>构造函数主体</h2><h3 id="config"><a href="#config" class="headerlink" title="config"></a>config</h3><figure class="highlight python"> <figcaption><span>Connection.__init__()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre> </td> <td class="code"> <pre><span class="line">super(Connection, self).__init__(config=config)</span><br><span class="line"><span class="keyword">if</span> config <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> config = Config()</span><br><span class="line"><span class="keyword">elif</span> <span class="keyword">not</span> isinstance(config, Config):</span><br><span class="line"> config = config.clone(into=Config)</span><br><span class="line">self._set(_config=config)</span><br></pre> </td> </tr> </table></figure><p> <code>config</code>成员变量是一个<code>Config</code>对象，它是调用父类<code>Context.__init__()</code>方法来初始化的。<code>Context.__init__()</code>定义如下:<br> <figure class="highlight python"> <figcaption><span>class Context</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Context</span><span class="params">(DataProxy)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, config=None)</span>:</span></span><br><span class="line"> config = config <span class="keyword">if</span> config <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span> <span class="keyword">else</span> Config()</span><br><span class="line"> self._set(_config=config)</span><br><span class="line"></span><br><span class="line"> command_prefixes = list()</span><br><span class="line"> self._set(command_prefixes=command_prefixes)</span><br><span class="line"></span><br><span class="line"> command_cwds = list()</span><br><span class="line"> self._set(command_cwds=command_cwds)</span><br></pre> </td> </tr> </table> </figure></p><p> 具体过程是<code>Context.__init__()</code>初始化时调用<code>_set()</code>绑定了<code>Config</code>成员对象<code>_config</code>:<br> <figure class="highlight python"> <figcaption><span>class DataProxy</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">_set</span><span class="params">(self, *args, **kwargs)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> args:</span><br><span class="line"> object.__setattr__(self, *args)</span><br><span class="line"> <span class="keyword">for</span> key, value <span class="keyword">in</span> six.iteritems(kwargs):</span><br><span class="line"> object.__setattr__(self, key, value)</span><br></pre> </td> </tr> </table> </figure></p><p> 再通过加了<code>@property</code>的<code>config()</code>函数，使得<code>connection</code>对象能直接用<code>self.config</code>来引用<code>_config</code>:<br> <figure class="highlight python"> <figcaption><span>class DataProxy</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">@property</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">config</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> self._config</span><br><span class="line"></span><br><span class="line"><span class="meta">@config.setter</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">config</span><span class="params">(self, value)</span>:</span></span><br><span class="line"> self._set(_config=value)</span><br></pre> </td> </tr> </table> </figure></p><h3 id="host-user-port"><a href="#host-user-port" class="headerlink" title="host, user, port"></a>host, user, port</h3><figure class="highlight python"> <figcaption><span>Connection.__init__()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre> </td> <td class="code"> <pre><span class="line">shorthand = self.derive_shorthand(host)</span><br><span class="line">host = shorthand[<span class="string">"host"</span>]</span><br><span class="line">err = (</span><br><span class="line"> <span class="string">"You supplied the {} via both shorthand and kwarg! Please pick one."</span> <span class="comment"># noqa</span></span><br><span class="line">)</span><br><span class="line"><span class="keyword">if</span> shorthand[<span class="string">"user"</span>] <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">if</span> user <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">raise</span> ValueError(err.format(<span class="string">"user"</span>))</span><br><span class="line"> user = shorthand[<span class="string">"user"</span>]</span><br><span class="line"><span class="keyword">if</span> shorthand[<span class="string">"port"</span>] <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">if</span> port <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">raise</span> ValueError(err.format(<span class="string">"port"</span>))</span><br><span class="line"> port = shorthand[<span class="string">"port"</span>]</span><br></pre> </td> </tr> </table></figure><p>这段是处理<code>host</code>参数的。<code>host</code>可以有下面集中传入形式:<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre> </td> <td class="code"> <pre><span class="line">user@host:port # 例如: root@10.10.10.10:6666</span><br><span class="line">user@host # 例如: root@10.10.10.10</span><br><span class="line">host:port # 例如: 10.10.10.10:6666</span><br><span class="line">host # 例如: 10.10.10.10</span><br></pre> </td> </tr> </table> </figure></p><p> 前三种会调用<code>self.derive_shorthand(host)</code>分别解析出<code>self.host</code>，<code>self.user</code>和<code>self.port</code>，最后一种需单独传入<code>user</code>，<code>port</code>。<br>如果用前三种传入方式的话，记得不要再重复传入<code>user</code>或<code>port</code>了，会抛出异常</p><p>源码是真他妈长啊，注释就占了一两百行 (微笑</p><p> ok，初始化的参数我们先到这里，因为它给的示例也就只有个<code>host</code>而已，一会再讲<code>config_kwags</code>和深入下<code>config</code>。我们现在先去报错的地方:<br> <figure class="highlight python"> <figcaption><span>class Connection</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre> </td> <td class="code"> <pre><span class="line">kwargs = dict(</span><br><span class="line"> self.connect_kwargs,</span><br><span class="line"> username=self.user,</span><br><span class="line"> hostname=self.host,</span><br><span class="line"> port=self.port,</span><br><span class="line">)</span><br><span class="line"><span class="keyword">if</span> self.gateway:</span><br><span class="line"> kwargs[<span class="string">"sock"</span>] = self.open_gateway()</span><br><span class="line"><span class="keyword">if</span> self.connect_timeout:</span><br><span class="line"> kwargs[<span class="string">"timeout"</span>] = self.connect_timeout</span><br><span class="line"><span class="comment"># Strip out empty defaults for less noisy debugging</span></span><br><span class="line"><span class="keyword">if</span> <span class="string">"key_filename"</span> <span class="keyword">in</span> kwargs <span class="keyword">and</span> <span class="keyword">not</span> kwargs[<span class="string">"key_filename"</span>]:</span><br><span class="line"> <span class="keyword">del</span> kwargs[<span class="string">"key_filename"</span>]</span><br><span class="line"><span class="comment"># Actually connect!</span></span><br><span class="line">self.client.connect(**kwargs) <span class="comment"># 就是你了</span></span><br></pre> </td> </tr> </table> </figure></p><p> 看到最后一行，传参时将字典<code>kwargs</code>传了过去，<code>kwargs</code>里除了<code>usr</code>，<code>host</code>，<code>port</code>之外，还有一个<code>connect_kwargs</code>。我们看看<code>client.connect()</code>的定义:<br> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">connect</span><span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params"> self,</span></span></span><br><span class="line"><span class="function"><span class="params"> hostname,</span></span></span><br><span class="line"><span class="function"><span class="params"> port=SSH_PORT,</span></span></span><br><span class="line"><span class="function"><span class="params"> username=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> password=None, # 你</span></span></span><br><span class="line"><span class="function"><span class="params"> pkey=None, # 你</span></span></span><br><span class="line"><span class="function"><span class="params"> key_filename=None, # 还有你</span></span></span><br><span class="line"><span class="function"><span class="params"> timeout=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> allow_agent=True,</span></span></span><br><span class="line"><span class="function"><span class="params"> look_for_keys=True,</span></span></span><br><span class="line"><span class="function"><span class="params"> compress=False,</span></span></span><br><span class="line"><span class="function"><span class="params"> sock=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> gss_auth=False,</span></span></span><br><span class="line"><span class="function"><span class="params"> gss_kex=False,</span></span></span><br><span class="line"><span class="function"><span class="params"> gss_deleg_creds=True,</span></span></span><br><span class="line"><span class="function"><span class="params"> gss_host=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> banner_timeout=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> auth_timeout=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> gss_trust_dns=True,</span></span></span><br><span class="line"><span class="function"><span class="params"> passphrase=None,</span></span></span><br><span class="line"><span class="function"><span class="params"> )</span></span></span><br></pre> </td> </tr> </table> </figure></p><p>看到<code>6, 7, 8</code>行没？that’s it.</p><p>接下来我们改写一下一开始的示例:</p><ul> <li> <p>使用<code>password</code>:</p> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> fabric <span class="keyword">import</span> Connection</span><br><span class="line"><span class="meta">>>> </span><span class="comment"># my_password为10.10.22.13的root用户密码</span></span><br><span class="line"><span class="meta">>>> </span>conn = Connection(<span class="string">'10.10.22.13'</span>, user=<span class="string">'root'</span>, connect_kwargs={<span class="string">'password'</span>: <span class="string">'${my_password}'</span>})</span><br><span class="line"><span class="meta">>>> </span>conn.run(<span class="string">"uname -s"</span>)</span><br><span class="line"></span><br><span class="line">Linux</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>使用<code>key_filename</code>:</p> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> fabric <span class="keyword">import</span> Connection</span><br><span class="line"><span class="meta">>>> </span><span class="comment"># 使用key_filename参数前提需将你的ssh公钥（.pub后缀）添加到远程服务器的.ssh/authorized_keys file里</span></span><br><span class="line"><span class="meta">>>> </span><span class="comment"># id_rsa为私钥</span></span><br><span class="line"><span class="meta">>>> </span>conn = Connection(<span class="string">'10.10.22.13'</span>, user=<span class="string">'root'</span>, connect_kwargs={<span class="string">'key_filename'</span>: <span class="string">'${path to local .ssh dir}/${your id_rsa file}'</span>})</span><br><span class="line"><span class="meta">>>> </span>conn.run(<span class="string">"uname -s"</span>)</span><br><span class="line"></span><br><span class="line">Linux</span><br></pre> </td> </tr> </table> </figure> </li></ul><p>朋友们，让我们举杯庆祝一下吧</p><h3 id="connect-kwargs"><a href="#connect-kwargs" class="headerlink" title="connect_kwargs"></a>connect_kwargs</h3><p>我们趁此机会窥视一下<code>connect_keargs</code>的相关部分<br> <figure class="highlight python"> <figcaption><span>Connection.__init__()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">self.connect_kwargs = self.resolve_connect_kwargs(connect_kwargs)</span><br></pre> </td> </tr> </table> </figure></p><p> 我们看到，当<code>connect_kwargs</code>为<code>None</code>时，会通过<code>config</code>成员变量动态增加属性<code>connect_kwargs</code>属性:<br> <figure class="highlight python"> <figcaption><span>Connection.resolve_connect_kwargs()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">resolve_connect_kwargs</span><span class="params">(self, connect_kwargs)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> connect_kwargs <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> connect_kwargs = self.config.connect_kwargs <span class="comment"># 调用__getattr__()动态增加属性</span></span><br><span class="line"> <span class="keyword">elif</span> <span class="string">"key_filename"</span> <span class="keyword">in</span> self.config.connect_kwargs:</span><br><span class="line"> kwarg_val = connect_kwargs.get(<span class="string">"key_filename"</span>, [])</span><br><span class="line"> conf_val = self.config.connect_kwargs[<span class="string">"key_filename"</span>]</span><br><span class="line"> connect_kwargs[<span class="string">"key_filename"</span>] = conf_val + kwarg_val</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> <span class="string">"identityfile"</span> <span class="keyword">in</span> self.ssh_config:</span><br><span class="line"> connect_kwargs.setdefault(<span class="string">"key_filename"</span>, [])</span><br><span class="line"> connect_kwargs[<span class="string">"key_filename"</span>].extend(</span><br><span class="line"> self.ssh_config[<span class="string">"identityfile"</span>]</span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> connect_kwargs</span><br></pre> </td> </tr> </table> </figure></p><p> 在<code>__getattr__</code>方法里又调用了类方法<code>_get()</code>将<code>connect_kwargs</code>传到<code>key</code>:<br> <figure class="highlight python"> <figcaption><span>DataProxy.__getattr__()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">__getattr__</span><span class="params">(self, key)</span>:</span></span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> <span class="keyword">return</span> self._get(key) <span class="comment"># 调用_get()</span></span><br><span class="line"> <span class="keyword">except</span> KeyError:</span><br><span class="line"> <span class="keyword">if</span> key <span class="keyword">in</span> self._proxies:</span><br><span class="line"> <span class="keyword">return</span> getattr(self._config, key)</span><br><span class="line"> err = <span class="string">"No attribute or config key found for {!r}"</span>.format(key)</span><br><span class="line"> attrs = [x <span class="keyword">for</span> x <span class="keyword">in</span> dir(self.__class__) <span class="keyword">if</span> <span class="keyword">not</span> x.startswith(<span class="string">'_'</span>)]</span><br><span class="line"> err += <span class="string">"\n\nValid keys: {!r}"</span>.format(</span><br><span class="line"> sorted(list(self._config.keys()))</span><br><span class="line"> )</span><br><span class="line"> err += <span class="string">"\n\nValid real attributes: {!r}"</span>.format(attrs)</span><br><span class="line"> <span class="keyword">raise</span> AttributeError(err)</span><br></pre> </td> </tr> </table> </figure></p><p>调用<code>_get()</code>后返回一个<code>DataProxy</code>对象<code>value</code>:<br> <figure class="highlight python"> <figcaption><span>DataProxy._get()</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">_get</span><span class="params">(self, key)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> key <span class="keyword">in</span> (</span><br><span class="line"> <span class="string">'__setstate__'</span>,</span><br><span class="line"> ):</span><br><span class="line"> <span class="keyword">raise</span> AttributeError(key)</span><br><span class="line"> value = self._config[key]</span><br><span class="line"> <span class="keyword">if</span> isinstance(value, dict):</span><br><span class="line"> keypath = (key,)</span><br><span class="line"> <span class="keyword">if</span> hasattr(self, <span class="string">'_keypath'</span>):</span><br><span class="line"> keypath = self._keypath + keypath</span><br><span class="line"> root = getattr(self, <span class="string">'_root'</span>, self)</span><br><span class="line"> value = DataProxy.from_data(</span><br><span class="line"> data=value,</span><br><span class="line"> root=root,</span><br><span class="line"> keypath=keypath,</span><br><span class="line"> )</span><br><span class="line"> <span class="keyword">return</span> value</span><br></pre> </td> </tr> </table> </figure></p><p>是不是晕了？呵呵没关系，我他妈也是。就让他随风而去吧</p><h2 id="fab命令"><a href="#fab命令" class="headerlink" title="fab命令"></a>fab命令</h2><p> 安装完<code>fabric</code>后会连同<code>fab</code>工具一起装到<code>python/bin</code>下，在终端输入<code>fab -h</code>查看命令参数<br>简单来说，<code>fab</code>干这么件事，<strong>直接执行当前目录下的<code>fabfile.py</code>脚本里的函数</strong>，下面介绍具体如何写<code>fabfile.py</code></p><p>首先按照国际惯例，导入包:<br> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> fabric <span class="keyword">import</span> Connection</span><br><span class="line"><span class="keyword">from</span> invoke <span class="keyword">import</span> task</span><br></pre> </td> </tr> </table> </figure></p><p>实例化<code>COnnection</code>对象:<br> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># ${}里的内容自行填充</span></span><br><span class="line">conn = Connection(<span class="string">"${remote host}"</span>, user=<span class="string">'${remote user}'</span>, connect_kwargs={<span class="string">'password'</span>: <span class="string">"${remote user's password}"</span>})</span><br></pre> </td> </tr> </table> </figure></p><p>定义一个功能函数:<br> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">@task</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">execute</span><span class="params">(c)</span>:</span></span><br><span class="line"> conn.run(<span class="string">"uname -s"</span>)</span><br></pre> </td> </tr> </table> </figure></p><p> 这个<code>@task</code>装饰器是必须加的，保证<code>fab</code>能直接执行，参数<code>c</code>不用管它，但不能定义成与你实例化的<code>conn</code>同名，原因后面会说</p><p>保存为<code>fabfile.py</code>:<br> <figure class="highlight python"> <figcaption><span>fabfile.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> fabric <span class="keyword">import</span> Connection</span><br><span class="line"><span class="keyword">from</span> invoke <span class="keyword">import</span> task</span><br><span class="line"></span><br><span class="line">conn = Connection(<span class="string">"${remote host}"</span>, user=<span class="string">'${remote user}'</span>, connect_kwargs={<span class="string">'password'</span>: <span class="string">"${remote user's password}"</span>})</span><br><span class="line"></span><br><span class="line"><span class="meta">@task</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">execute</span><span class="params">(c)</span>:</span></span><br><span class="line"> conn.run(<span class="string">"uname -s"</span>)</span><br></pre> </td> </tr> </table> </figure></p><p>在终端执行（服务器系统为Linux）:<br> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">$</span><span class="bash"> fab execute</span></span><br><span class="line">Linux # Linux下的输出</span><br></pre> </td> </tr> </table> </figure></p><p> 说回参数<code>c</code>，<code>c</code>在这里实际上是本地连接，是<code>localhost</code>的一个<code>Connection</code>对象。我们可以试一下:<br> <figure class="highlight python"> <figcaption><span>fabfile.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> fabric <span class="keyword">import</span> Connection</span><br><span class="line"><span class="keyword">from</span> invoke <span class="keyword">import</span> task</span><br><span class="line"></span><br><span class="line"><span class="meta">@task</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">execute</span><span class="params">(c)</span>:</span></span><br><span class="line"> c.run(<span class="string">"uname -s"</span>)</span><br></pre> </td> </tr> </table> </figure></p><p>在终端执行（我的系统为Mac）:<br> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">$</span><span class="bash"> fab execute</span></span><br><span class="line">Darwin # Mac下的输出</span><br></pre> </td> </tr> </table> </figure></p><p>一目了然</p><p>到这里大家应该大致明白<code>fab</code>是干啥的了。这样的玩法就多了，下面举几个例子:<br> <figure class="highlight python"> <figcaption><span>fabfile.py</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> fabric <span class="keyword">import</span> Connection</span><br><span class="line"><span class="keyword">from</span> invoke <span class="keyword">import</span> task</span><br><span class="line"></span><br><span class="line">conn = Connection(<span class="string">"${remote host}"</span>, user=<span class="string">'${remote user}'</span>, connect_kwargs={<span class="string">'password'</span>: <span class="string">"${remote user's password}"</span>})</span><br><span class="line"></span><br><span class="line"><span class="meta">@task</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">uname_local</span><span class="params">(c)</span>:</span></span><br><span class="line"> c.run(<span class="string">"uname -s"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 列出某路径下的文件</span></span><br><span class="line"><span class="meta">@task</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">ls_remote</span><span class="params">(c, dir_path)</span>:</span></span><br><span class="line"> <span class="keyword">with</span> conn.cd(dir_path):</span><br><span class="line"> conn.run(<span class="string">"ls -la"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 还能在函数里实例化Connection对象</span></span><br><span class="line"><span class="meta">@task</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">uname_rmt</span><span class="params">(c, host, user, password)</span>:</span></span><br><span class="line"> con = Connection(host, user=user, connect_kwargs={<span class="string">'password'</span>: password})</span><br><span class="line"></span><br><span class="line"> con.run(<span class="string">"uname -s"</span>)</span><br></pre> </td> </tr> </table> </figure></p><p>执行:<br> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">$</span><span class="bash"> fab uname_local</span></span><br><span class="line">Darwin</span><br><span class="line"></span><br><span class="line"><span class="meta">$</span><span class="bash"> fab ls_remote /home</span></span><br><span class="line">trendy</span><br><span class="line">td_root</span><br><span class="line"></span><br><span class="line"><span class="meta">$</span><span class="bash"> fab uname_rmt 10.10.22.13 root ******* <span class="comment"># 这个密码就不放出来了</span></span></span><br><span class="line">Linux</span><br></pre> </td> </tr> </table> </figure></p><p>具体的<code>Connection</code>还封装了哪些命令，就需要大家在使用中探索了</p><h1 id="fab-1-x-与-fab-2-x"><a href="#fab-1-x-与-fab-2-x" class="headerlink" title="fab 1.x 与 fab 2.x"></a>fab 1.x 与 fab 2.x</h1><p><code>fab 1.x</code>与<code>fab 2.x</code>最大的不同就是<code>fab 2.x</code>没有了<code>env</code>模块，所有的操作都基于<code>Connection</code>对象完成，对于<code>fab</code>命令的调用，将暴露方法封装到<code>invoke</code>模块，使其独立出来。这是我觉得最大的不同。只是现在还没有很多的博客写到<code>fab 2.x</code>的一些特性，官方文档也很简单，还是需要大家花时间阅读下源码才能清楚里面的逻辑</p><p>有时间的话会再对<code>fabric</code>做个详细剖析</p>]]></content>
<summary type="html">
<img src="/posts/python模块fabric踩坑记录/pic0.jpeg">
<p>最近都在狂写脚本，好像变成 <strong>半个运维</strong> 一样…<br>刚好有需求写些部署工具，于是了解到了<a href="http://www.fabfile.org/" target="_blank" rel="noopener">fabric</a>这个模块。下面简单带过一下
</p>
</summary>
<category term="python" scheme="https://tankeryang.github.io/categories/python/"/>
<category term="python" scheme="https://tankeryang.github.io/tags/python/"/>
<category term="fabric" scheme="https://tankeryang.github.io/tags/fabric/"/>
<category term="远程部署" scheme="https://tankeryang.github.io/tags/%E8%BF%9C%E7%A8%8B%E9%83%A8%E7%BD%B2/"/>
</entry>
<entry>
<title>服务器安装anaconda与配置jupyter反向代理</title>
<link href="https://tankeryang.github.io/posts/%E6%9C%8D%E5%8A%A1%E5%99%A8%E5%AE%89%E8%A3%85anaconda%E4%B8%8E%E9%85%8D%E7%BD%AEjupyter%E5%8F%8D%E5%90%91%E4%BB%A3%E7%90%86/"/>
<id>https://tankeryang.github.io/posts/服务器安装anaconda与配置jupyter反向代理/</id>
<published>2018-05-23T23:54:26.000Z</published>
<updated>2019-10-09T17:06:09.647Z</updated>
<content type="html"><![CDATA[<img src="/posts/服务器安装anaconda与配置jupyter反向代理/pic0.png"><p>最近在公司的服务器上搭环境和工具，折腾了一小会，记录一下</p><a id="more"></a><hr><h1 id="安装Anaconda"><a href="#安装Anaconda" class="headerlink" title="安装Anaconda"></a>安装Anaconda</h1><p><strong>公司服务器系统信息:</strong></p><blockquote> <ul> <li>Linux version: <code>4.4.0-116-generic (buildd@lgw01-amd64-021)</code> </li> <li>Distribution: <code>Ubuntu 16.04.4 LTS</code></li> <li>gcc version: <code>5.4.0 20160609</code></li> <li>CPU info: <code>32 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz</code> (双物理cpu，单个cpu 8核16线程…)</li> <li>内网ip: <code>10.10.22.13</code></li> </ul></blockquote><ul> <li> <p> 首先<code>ssh</code>到服务器，<code>cd</code>到一个非系统级的目录，下载anaconda安装脚本:<br><code>wget https://mirrors4.tuna.tsinghua.edu.cn/anaconda/archive/Anaconda3-5.1.0-Linux-x86_64.sh</code> </p> </li> <li> <p>执行安装脚本:<br><code>sh Anaconda3-5.1.0-Linux-x86_64.sh</code></p> <div class="note warning"> <ul> <li>若中途问你是否添加环境变量，请选择否，不然会覆盖系统的python环境变量</li> <li>安装位置自选</li> </ul> </div> </li> <li> <p> 设置软链接至<code>usr/local/bin</code>:<br>因为不设置环境变量，所以这一步是方便我们可以在命令行直接输入<code>可执行文件名</code> 执行<code>anaconda3/bin</code> 下的可执行文件，如<code>jupyter</code></p> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">#</span><span class="bash"> 将anaconda3/bin/python解释器运行脚本软链接至/usr/<span class="built_in">local</span>/bin/python36，python36可换成其他你喜欢的名字</span></span><br><span class="line">ln -s ${path to anaconda3}/bin/python /usr/local/bin/python36</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 将anaconda3/bin/jupyter运行脚本软链接至/usr/<span class="built_in">local</span>/bin/jupyter</span></span><br><span class="line">ln -s ${path to anaconda3}/bin/jupyter /usr/local/bin/jupyter</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 还有什么要软链过去的请参照上面自便</span></span><br></pre> </td> </tr> </table> </figure> </li></ul><p>在终端输入<code>python36</code>，<code>jupyter</code>查看是否生效</p><hr><h1 id="服务器配置jupyter"><a href="#服务器配置jupyter" class="headerlink" title="服务器配置jupyter"></a>服务器配置jupyter</h1><p>到重头戏了</p><p>这里有两个需求:</p><ul> <li>在公司内网能访问到juoyter服务的端口</li> <li>在家时连上公司VPN能访问到jupyter服务的端口</li></ul><h2 id="首先生成jupyter配置文件"><a href="#首先生成jupyter配置文件" class="headerlink" title="首先生成jupyter配置文件"></a>首先生成jupyter配置文件</h2><p>执行下面的命令:<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">jupyter notebook --generate-config</span><br></pre> </td> </tr> </table> </figure></p><p>配置文件会生成在以下路径:</p><ul> <li>Windows: <code>C:\Users\USERNAME\.jupyter\jupyter_notebook_config.py</code></li> <li>OS X: <code>/Users/USERNAME/.jupyter/jupyter_notebook_config.py</code> </li> <li>Linux: <code>/home/USERNAME/.jupyter/jupyter_notebook_config.py</code> </li></ul><h2 id="生成密码与加密后的hash串"><a href="#生成密码与加密后的hash串" class="headerlink" title="生成密码与加密后的hash串"></a>生成密码与加密后的hash串</h2><p>执行下面的命令，输入设定的密码<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">jupyter notebook password</span><br></pre> </td> </tr> </table> </figure></p><p>hash文件生成在以下路径:</p><ul> <li>Windows: <code>C:\Users\USERNAME\.jupyter\jupyter_notebook_config.json</code></li> <li>OS X: <code>/Users/USERNAME/.jupyter/jupyter_notebook_config.json</code> </li> <li>Linux: <code>/home/USERNAME/.jupyter/jupyter_notebook_config.json</code> </li></ul><h2 id="配置jupyter-notebook-config-py"><a href="#配置jupyter-notebook-config-py" class="headerlink" title="配置jupyter_notebook_config.py"></a>配置<code>jupyter_notebook_config.py</code></h2><p>先<code>cd</code>进<code>.jupyter</code>目录，然后<code>vim jupyter_notebook_config.py</code>进行修改</p><p>在顶部添加下面的配置:<br> <figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre> </td> <td class="code"> <pre><span class="line">c.NotebookApp.allow_origin = <span class="string">'*'</span></span><br><span class="line">c.NotebookApp.ip = <span class="string">'*'</span></span><br><span class="line">c.NotebookApp.open_browser = <span class="literal">False</span></span><br><span class="line">c.NotebookApp.password = <span class="string">u"sha1:bb7b06..."</span> <span class="comment"># 这里将jupyter_notebook_config.json里的sha1串复制过来</span></span><br><span class="line">c.NotebookApp.port = <span class="number">9999</span> <span class="comment"># 端口可设为其他，注意不要冲突</span></span><br><span class="line">c.NotebookApp.trust_xheaders = <span class="literal">True</span></span><br></pre> </td> </tr> </table> </figure></p><h2 id="配置hosts"><a href="#配置hosts" class="headerlink" title="配置hosts"></a>配置hosts</h2><p>以我的为例<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">10.10.22.13 fp-bd13</span><br></pre> </td> </tr> </table> </figure></p><h2 id="启动jupyter"><a href="#启动jupyter" class="headerlink" title="启动jupyter"></a>启动jupyter</h2><p>用nohup在后台启动:<br> <figure class="highlight shell"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">nohup jupyter notebook --allow-root &</span><br></pre> </td> </tr> </table> </figure></p><p>浏览器访问<code>http://fp-bd13:9999</code>，看看效果。<br> <div class="note info"> <p>Tips: jupyter新的jupyter lab比notebook做了一些优化，界面也高端很多。只要在url后面加<code>/lab</code>就能立即享用<br><code>http://fp-bd13:9999/lab</code> </p> </div></p><h2 id="配置反向代理"><a href="#配置反向代理" class="headerlink" title="配置反向代理"></a>配置反向代理</h2><p>这一步是最坑爹的</p><p> 我们公司内部的vpn统一登上这台服务器<code>172.17.22.229</code>，然后通过它再连<code>10.10.22.*</code>的机器。所以需要在<code>172.17.22.229</code>的nginx上配置反向代理，这样我们就能在外面访问到jupyter了</p><p>折腾了N久，最终配置如下:<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre> </td> <td class="code"> <pre><span class="line">map $http_upgrade $connection_upgrade {</span><br><span class="line"> default upgrade;</span><br><span class="line"> '' close;</span><br><span class="line">}</span><br><span class="line">server{</span><br><span class="line"> server_name fp.bd13.dev.jupyter;</span><br><span class="line"></span><br><span class="line"> location / {</span><br><span class="line"> proxy_pass http://10.10.22.13:9999;</span><br><span class="line"> proxy_redirect off;</span><br><span class="line"> proxy_set_header HOST $host;</span><br><span class="line"> proxy_set_header X-Real-IP $remote_addr;</span><br><span class="line"> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</span><br><span class="line"> proxy_http_version 1.1;</span><br><span class="line"> proxy_set_header Upgrade $http_upgrade;</span><br><span class="line"> proxy_set_header Connection "upgrade";</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre> </td> </tr> </table> </figure></p><p>重新加载下nginx的配置，在浏览器访问<code>fp.bd13.dev.jupyter</code>就能愉快地玩耍啦</p>]]></content>
<summary type="html">
<img src="/posts/服务器安装anaconda与配置jupyter反向代理/pic0.png">
<p>最近在公司的服务器上搭环境和工具，折腾了一小会，记录一下</p>
</summary>
<category term="技术杂项" scheme="https://tankeryang.github.io/categories/%E6%8A%80%E6%9C%AF%E6%9D%82%E9%A1%B9/"/>
<category term="anaconda" scheme="https://tankeryang.github.io/tags/anaconda/"/>
<category term="jupyter" scheme="https://tankeryang.github.io/tags/jupyter/"/>
<category term="nginx" scheme="https://tankeryang.github.io/tags/nginx/"/>
<category term="反向代理" scheme="https://tankeryang.github.io/tags/%E5%8F%8D%E5%90%91%E4%BB%A3%E7%90%86/"/>
</entry>
<entry>
<title>Machine Learning (Week4)</title>
<link href="https://tankeryang.github.io/posts/Machine-Learning%20(Week4)/"/>
<id>https://tankeryang.github.io/posts/Machine-Learning (Week4)/</id>
<published>2018-01-26T12:29:41.000Z</published>
<updated>2019-10-09T17:06:09.579Z</updated>
<content type="html"><![CDATA[<p>第四周的课程开始介绍 <strong>Neural Networks - 神经网络</strong> 的概念，包括它的 <strong>结构</strong> 和 <strong>表示方法</strong>。</p><img src="/posts/Machine-Learning%20(Week4)/pic0.jpg"><a id="more"></a><h1 id="Neural-Networks-Representation-神经网络：表述"><a href="#Neural-Networks-Representation-神经网络：表述" class="headerlink" title="Neural Networks: Representation - 神经网络：表述"></a>Neural Networks: Representation - 神经网络：表述</h1><h2 id="Non-linear-hypotheses-非线性假设"><a href="#Non-linear-hypotheses-非线性假设" class="headerlink" title="Non-linear hypotheses - 非线性假设"></a>Non-linear hypotheses - 非线性假设</h2><p>我们之前所讲到的，线性回归和逻辑回归，它们在进行 <strong>非线性预测和分类</strong> 时，在 <strong>特征比较少</strong> 的情况下，用 <strong>特征多项式</strong> 来训练会有不错的表现（如下式，两个特征）。</p><p> <center><br><span>$\theta_{0} + \theta_{1}x_{1}^{2} + \theta_{2}x_{1}x_{2} + \theta_{3}x_{2}^{2}$</span><!-- Has MathJax --><br></center><br><br></p><p> 但是一旦特征变多，比如几十甚至上百个时，利用特征多项式训练，所组合出来的新特征将会是一个非常庞大的数字（如下式，100个特征，仅两两组合）。这无疑是行不通的。</p><p> <center><br><span>$\theta_{0} + \theta_{1}x_{1}^{2} + \theta_{2}x_{2}^{2} + \theta_{3}x_{3}^{2} + \cdots + \theta_{100}x_{100}^{2} + \theta_{101}x_{1}x_{2} + \theta_{102}x_{1}x_{3} + \cdots + \theta_{10000}x_{99}x_{100}$</span><!-- Has MathJax --><br></center> <br><br></p><p>所以，我们需要一种新的模型，就是 <strong>神经网络</strong>。</p><h2 id="Neurons-and-the-brain-神经元和大脑"><a href="#Neurons-and-the-brain-神经元和大脑" class="headerlink" title="Neurons and the brain - 神经元和大脑"></a>Neurons and the brain - 神经元和大脑</h2><blockquote> <p> <font color="#aeaeae">这部分是介绍神经网络的一些背景，若不感兴趣可</font><a href="#Model-representation-I-模型表示-1">点此跳至下一节</a> </p></blockquote><p> 首先神经网络是模仿大脑结构来建立的。我们大脑可以学习很多事情，小到一个行为，一个动作，大到一门学科，一门语言，如果我们想让计算机来处理不同的学习任务，似乎我们需要针对性的编写不同的程序来实现。可是人脑在学习的时候真的用了这么多不同的“算法”么？我们能不能假设其实大脑只有一种学习算法，但却可以处理很多的事情？下面我们来看一下关于这个假设的一些证据。</p><ul> <li> <p>科学家将小白鼠的 <strong>耳朵</strong> 到 <strong>听觉皮层</strong> 的 <strong>听觉神经</strong> 剪断，然后将 <strong>视觉神经</strong> 接到 <strong>听觉皮层</strong> 上，结果，<strong>听觉皮层</strong> 学会了 <strong>看</strong>。 </p> <img src="/posts/Machine-Learning%20(Week4)/pic1.png"> </li> <li> <p>美国食品和药物管理局在临床实验一个名为brainport的系统，能帮助失明人士 <strong>看见</strong> 东西。它是这样工作的：在失明人士头上带一个灰度摄像头，获取面前场景的低分辨率灰度图像，然后将信号连接到 <strong>舌头</strong> 上的一个很薄的 <strong>电极阵列</strong> 上，这样 <strong>每个像素点都能映射到舌头的某个位置</strong>，这种系统能让人在几十分钟内用舌头学会 <strong>看东西</strong>。</p> <img src="/posts/Machine-Learning%20(Week4)/pic2.png"> </li></ul><p>这些例子说明，我们大脑的每一块区域，都能处理不同的信息，图像，声音，触觉，嗅觉等等，就好比一个机器可以接受任何传感器输入的信号。如果我们找出 <strong>大脑的学习方法</strong>，然后在计算机上实现，这也许是真正的迈向人工智能。而 <strong>神经网络</strong> 则是第一步。</p><p>下面我们将深入到神经网络的细节。</p><h2 id="Model-representation-I-模型表示-1"><a href="#Model-representation-I-模型表示-1" class="headerlink" title="Model representation I - 模型表示 1"></a>Model representation I - 模型表示 1</h2><p>每一个神经元，也可以叫一个 <strong>计算节点</strong>，它接受来自 <strong>前一个神经元的输出</strong> 作为 <strong>它的输入</strong>，然后经过 <strong>计算</strong>，将 <strong>输出</strong> 送给 <strong>下一个神经元</strong> 作为其 <strong>输入</strong>。</p><div class="mermaid"> graph LR; a1((a1)) -- input --> a2((a2)); a2((a2)) -- output --> a3((a3));</div><p>比如一个以 <strong>逻辑回归</strong> 为模型的网络可以表示成这样：<br><img src="/posts/Machine-Learning%20(Week4)/pic3.png"></p><p>这是一个只有 <strong>两层</strong> 的神经网络，只包含 <strong>输入层</strong> 和 <strong>输出层</strong>。</p><p>下面我们看一个3层的神经网络：<br><img src="/posts/Machine-Learning%20(Week4)/pic4.png"></p><p>其中layer 2是 <strong>隐藏层</strong>，是中间的计算单元，将结果反馈到下一层。这里我们定义一下符号表示：</p><ul> <li><span>$a_{i}^{(j)}$</span><!-- Has MathJax -->为第j层第i个节点</li> <li><span>$h_{\Theta}(x)$</span><!-- Has MathJax -->为输出结果</li></ul><div class="note danger"> <p><i class="fa fa-spinner fa-pulse fa-lg margin-bottom" aria-hidden="true"></i> 未完待续…有空继续… </p></div>]]></content>
<summary type="html">
<p>第四周的课程开始介绍 <strong>Neural Networks - 神经网络</strong> 的概念，包括它的
<strong>结构</strong> 和 <strong>表示方法</strong>。</p>
<img src="/posts/Machine-Learning%20(Week4)/pic0.jpg">
</summary>
<category term="机器学习笔记" scheme="https://tankeryang.github.io/categories/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/"/>
<category term="机器学习" scheme="https://tankeryang.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="ML" scheme="https://tankeryang.github.io/tags/ML/"/>
<category term="数学" scheme="https://tankeryang.github.io/tags/%E6%95%B0%E5%AD%A6/"/>
</entry>
<entry>
<title>Hexo+NexT+GithubPages+CodingPages+GiteePages+Travis全攻略</title>
<link href="https://tankeryang.github.io/posts/Hexo%20+%20NexT%20+%20Github%20Pages%20+%20Coding%20Pages%20+%20Gitee%20Pages%20+%20Travis%20%E5%85%A8%E6%94%BB%E7%95%A5/"/>
<id>https://tankeryang.github.io/posts/Hexo + NexT + Github Pages + Coding Pages + Gitee Pages + Travis 全攻略/</id>
<published>2018-01-22T15:34:54.000Z</published>
<updated>2019-10-09T17:06:09.547Z</updated>
<content type="html"><</code>的方法访问它们。</p><p> 如果你想要更有规律地提供图片和其他资源以及想要将他们的资源分布在各个文章上的人来说，Hexo也提供了更组织化的方式来管理资源。这个稍微有些复杂但是管理资源非常方便的功能可以通过将<code>config.yml</code>文件中的<code>post_asset_folder</code>选项设为<code>true</code>来打开：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="attr">post_asset_folder:</span> <span class="literal">true</span> <span class="comment"># 设置为true</span></span><br></pre> </td> </tr> </table> </figure></p><p>设置为<code>true</code>后，当你新建一篇文章时，hexo同时会新建一个 <strong>和文章标题一样名字</strong> 的文件夹，你的文章所引用的图片等资源就可以放在这里面了。</p><p>将所有与你的文章有关的资源放在这个关联文件夹中之后，你可以通过<a href="#标签插件"><strong>标签插件</strong></a>来引用它们，这样你就得到了一个更简单而且方便得多的工作流。关于什么是标签插件，接下来的内容会说明。请耐心阅读。（你也可以点击上面的链接浏览一下标签插件的内容）</p><hr><blockquote> <p><i class="fa fa-hand-o-down" aria-hidden="true"></i> <font color="#aeaeae"> 以下内容更新于2018/1/25，承接上文（不好意思，本人很懒…）</font> </p></blockquote><h1 id="写作"><a href="#写作" class="headerlink" title="写作"></a>写作</h1><h2 id="新建文章"><a href="#新建文章" class="headerlink" title="新建文章"></a>新建文章</h2><p>执行如下命令新建文章：<br> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">hexo new [layout] <title></span><br></pre> </td> </tr> </table> </figure></p><p>其中<code>[layout]</code>字段是文章的 <strong>布局</strong>，默认为<code>post</code>，可以通过修改<code>_config.yml</code>中的<code>default_layout</code>参数来指定默认布局。<code><title></code>则是文章标题。</p><h3 id="布局"><a href="#布局" class="headerlink" title="布局"></a>布局</h3><p>Hexo 有三种默认布局：<code>post</code>、<code>page</code>和<code>draft</code>，它们分别对应不同的路径，而您 <strong>自定义的其他布局</strong> 和<code>post</code>相同，都将储存到<code>source/_posts</code>文件夹。</p><table> <thead> <tr> <th style="text-align:center">布局</th> <th style="text-align:center">路径</th> </tr> </thead> <tbody> <tr> <td style="text-align:center"><code>post</code></td> <td style="text-align:center"><code>source/_posts</code></td> </tr> <tr> <td style="text-align:center"><code>page</code></td> <td style="text-align:center"><code>source</code></td> </tr> <tr> <td style="text-align:center"><code>draft</code></td> <td style="text-align:center"><code>source/_draft</code></td> </tr> </tbody></table><p>其实布局我到现在也不是很清楚是什么，<strong>我是这样认为的：</strong></p><div class="note primary"> <p>如果你执行了这条命令<code>hexo new post "new article"</code>，hexo会新建一个<code>new article.md</code>文件在<code>source/_post</code>文件夹下。同理，其他两个布局参照上面的路径新建文章。 </p></div><h3 id="文件名称"><a href="#文件名称" class="headerlink" title="文件名称"></a>文件名称</h3><p>Hexo默认以 <strong>标题</strong> 作为文件名称，你也可编辑<code>myblog/_config.yml</code>中的<code>new_post_name</code>参数来改变默认的文件名称，比如设置为：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="attr">new_post_name:</span> <span class="string">:year-:month-:day-:title.md</span> <span class="comment"># File name of new posts</span></span><br></pre> </td> </tr> </table> </figure></p><p>这样在 <strong>文章名</strong> 前面就会加上日期和时间共同组成 <strong>文件名</strong>。</p><p>下面是一些可用来配置文件名的变量：</p><table> <thead> <tr> <th style="text-align:center">变量</th> <th style="text-align:center">描述</th> </tr> </thead> <tbody> <tr> <td style="text-align:center"><code>:title</code></td> <td style="text-align:center">标题（小写，空格将会被替换为短杠）</td> </tr> <tr> <td style="text-align:center"><code>:year</code></td> <td style="text-align:center">建立的年份,比如，2015</td> </tr> <tr> <td style="text-align:center"><code>:month</code></td> <td style="text-align:center">建立的月份（有前导零），比如，<code>04</code></td> </tr> <tr> <td style="text-align:center"><code>i_month</code></td> <td style="text-align:center">建立的月份（无前导零），比如，<code>4</code></td> </tr> <tr> <td style="text-align:center"><code>:day</code></td> <td style="text-align:center">建立的日期（有前导零），比如，<code>07</code></td> </tr> <tr> <td style="text-align:center"><code>i_day</code></td> <td style="text-align:center">建立的日期（无前导零），比如，<code>7</code></td> </tr> </tbody></table><div class="note info"> <p>更多有关 <strong>基本写作设置</strong> 的内容请参考<a href="https://hexo.io/zh-cn/docs" target="_blank" rel="noopener">hexo官方文档</a>的<a href="https://hexo.io/zh-cn/docs/writing.html" target="_blank" rel="noopener">写作</a>部分。 </p></div><h3 id="下面，我们就来尝试一下吧！"><a href="#下面，我们就来尝试一下吧！" class="headerlink" title="下面，我们就来尝试一下吧！"></a>下面，我们就来尝试一下吧！</h3><p>首先，执行<br> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">hexo new post <span class="string">"caonima"</span></span><br></pre> </td> </tr> </table> </figure></p><p>你会看到输出<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">INFO Created: C:/hexo/myblog/source/_posts/caonima.md</span><br></pre> </td> </tr> </table> </figure></p><p> 同时，在<code>source/_post</code>文件夹下多了<code>caonima.md</code>文件和<code>caonima</code>文件夹。<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre> </td> <td class="code"> <pre><span class="line">.</span><br><span class="line">├── caonima.md</span><br><span class="line">├── hello-world.md</span><br><span class="line">├── caonima</span><br></pre> </td> </tr> </table> </figure></p><p>打开<code>caonima.md</code>，你会看到这些<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="meta">---</span></span><br><span class="line"><span class="attr">title:</span> <span class="string">caonima</span></span><br><span class="line"><span class="attr">date:</span> <span class="number">2018</span><span class="bullet">-01</span><span class="bullet">-25</span> <span class="number">13</span><span class="string">:06:28</span></span><br><span class="line"><span class="attr">tags:</span></span><br><span class="line"><span class="meta">---</span></span><br></pre> </td> </tr> </table> </figure></p><p>这些是 <strong>front-matter</strong>，下面我会对它进行说明，请耐心阅读（<del>这句话我到底讲了几次</del>）。<br>接下来你可以随便在里面写点内容，比如：<br> <figure class="highlight markdown"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="section"># caonima</span></span><br><span class="line"><span class="section">## caonima</span></span><br><span class="line"><span class="section">### caonima</span></span><br><span class="line"><span class="strong">__caonima__</span></span><br><span class="line"><span class="emphasis">_caonima_</span></span><br><span class="line"><span class="bullet">* </span>caonima</span><br><span class="line"></span><br><span class="line"><span class="quote">> caonima</span></span><br><span class="line"></span><br><span class="line">[<span class="string">caonima</span>](<span class="link"></span>)</span><br><span class="line">~~caonima~~</span><br><span class="line"><span class="code">`caonima`</span></span><br></pre> </td> </tr> </table> </figure></p><p>保存。执行下面语句生成博客文件并运行本地hexo服务端：<br> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">hexo g && hexo s</span><br></pre> </td> </tr> </table> </figure></p><p>你会看到如下输出：<br> <figure class="highlight bash"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre> </td> <td class="code"> <pre><span class="line">INFO Start processing</span><br><span class="line">INFO Files loaded <span class="keyword">in</span> 642 ms</span><br><span class="line">INFO Generated: 2018/01/22/hello-world/index.html</span><br><span class="line">INFO Generated: archives/index.html</span><br><span class="line">INFO Generated: index.html</span><br><span class="line">INFO Generated: archives/2018/01/index.html</span><br><span class="line">INFO Generated: archives/2018/index.html</span><br><span class="line">INFO Generated: images/avatar.gif</span><br><span class="line">INFO Generated: images/apple-touch-icon-next.png</span><br><span class="line">INFO Generated: images/cc-by-nc-nd.svg</span><br><span class="line">INFO Generated: images/algolia_logo.svg</span><br><span class="line">INFO Generated: images/cc-by-nc-sa.svg</span><br><span class="line">INFO Generated: images/cc-by-nd.svg</span><br><span class="line">INFO Generated: images/cc-by-nc.svg</span><br><span class="line">INFO Generated: images/cc-by.svg</span><br><span class="line">INFO Generated: images/favicon-16x16-next.png</span><br><span class="line">INFO Generated: images/cc-zero.svg</span><br><span class="line">INFO Generated: images/favicon-32x32-next.png</span><br><span class="line">INFO Generated: images/cc-by-sa.svg</span><br><span class="line">INFO Generated: images/logo.svg</span><br><span class="line">INFO Generated: images/placeholder.gif</span><br><span class="line">INFO Generated: images/quote-l.svg</span><br><span class="line">INFO Generated: images/loading.gif</span><br><span class="line">INFO Generated: images/quote-r.svg</span><br><span class="line">INFO Generated: lib/font-awesome/HELP-US-OUT.txt</span><br><span class="line">INFO Generated: lib/font-awesome/css/font-awesome.css.map</span><br><span class="line">INFO Generated: images/searchicon.png</span><br><span class="line">INFO Generated: lib/font-awesome/fonts/fontawesome-webfont.woff2</span><br><span class="line">INFO Generated: lib/font-awesome/fonts/fontawesome-webfont.woff</span><br><span class="line">INFO Generated: js/src/algolia-search.js</span><br><span class="line">INFO Generated: js/src/exturl.js</span><br><span class="line">INFO Generated: js/src/affix.js</span><br><span class="line">INFO Generated: js/src/post-details.js</span><br><span class="line">INFO Generated: js/src/motion.js</span><br><span class="line">INFO Generated: js/src/scroll-cookie.js</span><br><span class="line">INFO Generated: js/src/utils.js</span><br><span class="line">INFO Generated: js/src/scrollspy.js</span><br><span class="line">INFO Generated: lib/font-awesome/bower.json</span><br><span class="line">INFO Generated: js/src/js.cookie.js</span><br><span class="line">INFO Generated: js/src/bootstrap.js</span><br><span class="line">INFO Generated: lib/velocity/velocity.ui.min.js</span><br><span class="line">INFO Generated: lib/ua-parser-js/dist/ua-parser.pack.js</span><br><span class="line">INFO Generated: js/src/schemes/pisces.js</span><br><span class="line">INFO Generated: lib/ua-parser-js/dist/ua-parser.min.js</span><br><span class="line">INFO Generated: css/main.css</span><br><span class="line">INFO Generated: lib/jquery/index.js</span><br><span class="line">INFO Generated: lib/velocity/velocity.ui.js</span><br><span class="line">INFO Generated: lib/font-awesome/css/font-awesome.css</span><br><span class="line">INFO Generated: lib/velocity/velocity.min.js</span><br><span class="line">INFO Generated: lib/font-awesome/css/font-awesome.min.css</span><br><span class="line">INFO Generated: lib/velocity/velocity.js</span><br><span class="line">INFO Generated: lib/font-awesome/fonts/fontawesome-webfont.eot</span><br><span class="line">INFO Generated: 2018/01/25/caonima/index.html</span><br><span class="line">INFO 51 files generated <span class="keyword">in</span> 1 s</span><br><span class="line">INFO Start processing</span><br><span class="line">INFO Hexo is running at http://localhost:4000/. Press Ctrl+C to stop.</span><br></pre> </td> </tr> </table> </figure></p><p>在浏览器输入<code>http://localhost:4000/</code>并访问，你会看到这样的页面：<br><img src="/posts/Hexo%20+%20NexT%20+%20Github%20Pages%20+%20Coding%20Pages%20+%20Gitee%20Pages%20+%20Travis%20全攻略/pic4.png"></p><div class="note success"> <p><i class="fa fa-thumbs-o-up" aria-hidden="true"></i> <strong>恭喜你！你已经可以进行简单的文字创作了！下面的任务就是让你的写作流程规范化的细节。请耐心阅读。</strong> </p></div><h2 id="标签插件"><a href="#标签插件" class="headerlink" title="标签插件"></a>标签插件</h2><blockquote> <p> <font color="#aeaeae">从上方跳转过来的朋友，</font><a href="#配置资源文件夹"><i class="fa fa-hand-o-up" aria-hidden="true"></i> 点此返回<strong>资源配置文件夹处</strong></a> </p></blockquote><p>标签插件是用于在文章中快速插入特定内容的插件。<br>它的在文章中用法一般是这样：<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre> </td> <td class="code"> <pre><span class="line">{% [某种标签] %}</span><br><span class="line"><你想插入的内容></span><br><span class="line">{% end[某种标签] %}</span><br></pre> </td> </tr> </table> </figure></p><p>或者这样：<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">{% [某种标签] <你想插入的内容> %}</span><br></pre> </td> </tr> </table> </figure></p><p>用例子说明最快：<br>比如像前面提到的，<strong>用标签插件在文章引用图片</strong>，你只需这样写<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">{% asset_img <图片文件名> %}</span><br></pre> </td> </tr> </table> </figure></p><p>示例<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">{% asset_img caonima.jpg %}</span><br></pre> </td> </tr> </table> </figure></p><p>这是我最常用的标签了，<code>asset_img</code>，顾名思义，就是图片资源。一般我都用它来插入图片。因为我们在前面配置了 <strong>资源文件夹</strong>，所以<code><图片文件名></code>这里我们不用输入绝对路径，<strong>只需输入图片文件名</strong> 就ok了，<strong>hexo会自动在资源文件夹里寻找你的图片</strong>。</p><p> 你可以在<code>caonima</code>文件夹里放一张图片，然后在<code>caonima.md</code>里用上面的<code>asset_img</code>标签插件来引用它，看看效果。</p><div class="note info"> <p>更多有关标签插件的内容，请参考<a href="https://hexo.io/zh-cn/docs" target="_blank" rel="noopener">hexo官方文档</a>中的<a href="https://hexo.io/zh-cn/docs/tag-plugins.html" target="_blank" rel="noopener">标签插件</a>部分。 </p></div><hr><h1 id="配置主题配置文件"><a href="#配置主题配置文件" class="headerlink" title="配置主题配置文件"></a>配置主题配置文件</h1><p>NexT主题作为hexo众多主题里最火的一款，除了简约美观的设计之外，最重要的一点就是 <strong>可定制化的程度高</strong>。你可以很轻松的 <strong>开启或关闭某些功能</strong>，甚至 <strong>自己尝试添加一些功能</strong>也比其他主题简单，因为它的 <strong>源文件组织得很清晰</strong>，主题的 <strong>布局</strong>，<strong>js</strong>，<strong>css</strong>，<strong>字体</strong>，<strong>语言</strong>，等文件都独立区分。</p><p>下面我会参照我的配置来详细介绍如何配置NexT主题。</p><h2 id="重要更新"><a href="#重要更新" class="headerlink" title="重要更新"></a>重要更新</h2><p>在<span class="label info">v6.0.x</span>的版本里，NexT新增了 <strong>缓存</strong> 这样一个特性：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># Allow to cache content generation. Introduced in NexT v6.0.0.</span></span><br><span class="line"><span class="attr">cache:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br></pre> </td> </tr> </table> </figure></p><p>这是一个非常强大的改进！也就是说，当我们执行了<code>hexo s</code>预览博客内容时，同时对 <strong>文章内容</strong> 或 <strong>主题配置</strong> 做了一些修改，我们只需 <strong>刷新一下</strong> 页面就能实时看到更改效果，而不用重新执行<code>hexo clean && hexo g</code>来重新生成页面。对此我只能说<span class="label default">6</span><span class="label primary">6</span><span class="label success">6</span><span class="label info">6</span><span class="label warning">6</span><span class="label danger">6</span></p><h2 id="网站图标"><a href="#网站图标" class="headerlink" title="网站图标"></a>网站图标</h2><p>下面就是网站图标的配置项：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># For example, you put your favicons into `hexo-site/source/images` directory.</span></span><br><span class="line"><span class="comment"># Then need to rename & redefine they on any other names, otherwise icons from Next will rewrite your custom icons in Hexo.</span></span><br><span class="line"><span class="attr">favicon:</span></span><br><span class="line"><span class="attr"> small:</span> <span class="string">/images/favicon-16x16-next.png</span></span><br><span class="line"><span class="attr"> medium:</span> <span class="string">/images/favicon-32x32-next.png</span></span><br><span class="line"><span class="attr"> apple_touch_icon:</span> <span class="string">/images/apple-touch-icon-next.png</span></span><br><span class="line"><span class="attr"> safari_pinned_tab:</span> <span class="string">/images/logo.svg</span></span><br><span class="line"> <span class="comment">#android_manifest: /images/manifest.json</span></span><br><span class="line"> <span class="comment">#ms_browserconfig: /images/browserconfig.xml</span></span><br></pre> </td> </tr> </table> </figure></p><p> 参照注释，先在<code>myblog/source/</code>路径下新建<code>images</code>文件夹，找一张<code>16x16</code>的<code>ico</code>或者<code>png</code>图标，放进<code>images</code>文件夹（在哪里找图标请自行百度），比如<code>caonima.ico</code>。<br>然后将<code>small</code>选项设置为<code>/images/caonima.ico</code>：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="attr">favicon:</span></span><br><span class="line"><span class="attr"> small:</span> <span class="string">/images/caonima.ico</span></span><br></pre> </td> </tr> </table> </figure></p><p>再将其他的选项注释掉（因为基本用不到）：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="attr">favicon:</span></span><br><span class="line"><span class="attr"> small:</span> <span class="string">/images/caonima.ico</span></span><br><span class="line"> <span class="comment">#medium: /images/favicon-32x32-next.png</span></span><br><span class="line"> <span class="comment">#apple_touch_icon: /images/apple-touch-icon-next.png</span></span><br><span class="line"> <span class="comment">#safari_pinned_tab: /images/logo.svg</span></span><br><span class="line"> <span class="comment">#android_manifest: /images/manifest.json</span></span><br><span class="line"> <span class="comment">#ms_browserconfig: /images/browserconfig.xml</span></span><br></pre> </td> </tr> </table> </figure></p><p>网站图标配置就完成了。关于其他选项你可以有空自己放些图标文件来玩玩看什么效果。</p><h2 id="网站底部内容"><a href="#网站底部内容" class="headerlink" title="网站底部内容"></a>网站底部内容</h2><p>这些在<code>footer</code>选项的配置里：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="attr">footer:</span></span><br><span class="line"> <span class="comment"># Specify the date when the site was setup.</span></span><br><span class="line"> <span class="comment"># If not defined, current year will be used.</span></span><br><span class="line"> <span class="comment">#since: 2015</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Icon between year and copyright info.</span></span><br><span class="line"><span class="attr"> icon:</span> <span class="string">user</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># If not defined, will be used `author` from Hexo main config.</span></span><br><span class="line"><span class="attr"> copyright:</span></span><br><span class="line"> <span class="comment"># -------------------------------------------------------------</span></span><br><span class="line"> <span class="comment"># Hexo link (Powered by Hexo).</span></span><br><span class="line"><span class="attr"> powered:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="attr"> theme:</span></span><br><span class="line"> <span class="comment"># Theme & scheme info link (Theme - NexT.scheme).</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"> <span class="comment"># Version info of NexT after scheme info (vX.X.X).</span></span><br><span class="line"><span class="attr"> version:</span> <span class="literal">true</span></span><br><span class="line"> <span class="comment"># -------------------------------------------------------------</span></span><br><span class="line"> <span class="comment"># Any custom text can be defined here.</span></span><br><span class="line"> <span class="comment">#custom_text: Hosted by <a target="_blank" rel="external nofollow" href="https://pages.coding.me"><b>Coding Pages</b></a></span></span><br></pre> </td> </tr> </table> </figure></p><p>默认图标是<a href="http://www.fontawesome.com.cn/icons/user/" target="_blank" rel="noopener"><code>user</code></a>。假如你想个性化的话可以参照<a href="http://www.fontawesome.com.cn/" target="_blank" rel="noopener">fontawsome</a>提供的图标来进行选择。下面是我的配置：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="attr">footer:</span></span><br><span class="line"> <span class="comment"># Specify the date when the site was setup.</span></span><br><span class="line"> <span class="comment"># If not defined, current year will be used.</span></span><br><span class="line"><span class="attr"> since:</span> <span class="number">2017</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Icon between year and copyright info.</span></span><br><span class="line"><span class="attr"> icon:</span> <span class="string">cog</span></span><br><span class="line"> <span class="comment"># spiner icon</span></span><br><span class="line"><span class="attr"> rotate:</span> <span class="string">fa-spin</span> <span class="string">fa-lg</span> <span class="string">margin-bottom</span></span><br><span class="line"> <span class="comment">#rotate_plus : fa-plus fa-lg margin-bottom</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># If not defined, will be used `author` from Hexo main config.</span></span><br><span class="line"><span class="attr"> copyright:</span></span><br><span class="line"> <span class="comment"># -------------------------------------------------------------</span></span><br><span class="line"> <span class="comment"># Hexo link (Powered by Hexo).</span></span><br><span class="line"><span class="attr"> powered:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="attr"> theme:</span></span><br><span class="line"> <span class="comment"># Theme & scheme info link (Theme - NexT.scheme).</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"> <span class="comment"># Version info of NexT after scheme info (vX.X.X).</span></span><br><span class="line"><span class="attr"> version:</span> <span class="literal">true</span></span><br><span class="line"> <span class="comment"># -------------------------------------------------------------</span></span><br><span class="line"> <span class="comment"># Any custom text can be defined here.</span></span><br><span class="line"> <span class="comment">#custom_text: Hosted by <a target="_blank" href="https://pages.github.com">GitHub Pages</a></span></span><br></pre> </td> </tr> </table> </figure></p><h2 id="菜单栏设置"><a href="#菜单栏设置" class="headerlink" title="菜单栏设置"></a>菜单栏设置</h2><p>这些在<code>menu</code>选项里：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># ---------------------------------------------------------------</span></span><br><span class="line"><span class="comment"># Menu Settings</span></span><br><span class="line"><span class="comment"># ---------------------------------------------------------------</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># When running the site in a subdirectory (e.g. domain.tld/blog), remove the leading slash from link value (/archives -> archives).</span></span><br><span class="line"><span class="comment"># Usage: `Key: /link/ || icon`</span></span><br><span class="line"><span class="comment"># Key is the name of menu item. If translate for this menu will find in languages - this translate will be loaded; if not - Key name will be used. Key is case-senstive.</span></span><br><span class="line"><span class="comment"># Value before `||` delimeter is the target link.</span></span><br><span class="line"><span class="comment"># Value after `||` delimeter is the name of FontAwesome icon. If icon (with or without delimeter) is not specified, question icon will be loaded.</span></span><br><span class="line"><span class="attr">menu:</span></span><br><span class="line"><span class="attr"> home:</span> <span class="string">/</span> <span class="string">||</span> <span class="string">home</span></span><br><span class="line"> <span class="comment">#about: /about/ || user</span></span><br><span class="line"> <span class="comment">#tags: /tags/ || tags</span></span><br><span class="line"> <span class="comment">#categories: /categories/ || th</span></span><br><span class="line"><span class="attr"> archives:</span> <span class="string">/archives/</span> <span class="string">||</span> <span class="string">archive</span></span><br><span class="line"> <span class="comment">#schedule: /schedule/ || calendar</span></span><br><span class="line"> <span class="comment">#sitemap: /sitemap.xml || sitemap</span></span><br><span class="line"> <span class="comment">#commonweal: /404/ || heartbeat</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Enable/Disable menu icons / item badges.</span></span><br><span class="line"><span class="attr">menu_settings:</span></span><br><span class="line"><span class="attr"> icons:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> badges:</span> <span class="literal">false</span></span><br></pre> </td> </tr> </table> </figure></p><p><code>||</code>后面的<code>icon</code>对应的是<a href="http://www.fontawesome.com.cn/" target="_blank" rel="noopener">fontawsome</a>相应的图标名。如果你想添加<code>about</code>菜单的话，按照以下步骤：</p><ul> <li>在<code>myblog</code>下新建<code>about</code>文件夹</li> <li>在<code>about</code>文件夹下新增<code>index.md</code>文件</li> <li>在<code>index.md</code>里添加内容</li> <li>在<code>menu</code>选项里添加<code>about: /about/ || user</code></li></ul><p>刷新一下看看效果。你会发现多了一个<code>about</code>菜单。用这样的办法可以自定义很多菜单目录。</p><h2 id="主题布局"><a href="#主题布局" class="headerlink" title="主题布局"></a>主题布局</h2><p>NexT主题提供了4种不同风格的主题布局，按需设置：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># ---------------------------------------------------------------</span></span><br><span class="line"><span class="comment"># Scheme Settings</span></span><br><span class="line"><span class="comment"># ---------------------------------------------------------------</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Schemes</span></span><br><span class="line"><span class="attr">scheme:</span> <span class="string">Muse</span></span><br><span class="line"><span class="comment">#scheme: Mist</span></span><br><span class="line"><span class="comment">#scheme: Pisces</span></span><br><span class="line"><span class="comment">#scheme: Gemini</span></span><br></pre> </td> </tr> </table> </figure></p><h2 id="侧边栏设置"><a href="#侧边栏设置" class="headerlink" title="侧边栏设置"></a>侧边栏设置</h2><p>这一项的内容有点多，先放配置文件，我们一个个看：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># ---------------------------------------------------------------</span></span><br><span class="line"><span class="comment"># Sidebar Settings</span></span><br><span class="line"><span class="comment"># ---------------------------------------------------------------</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Posts / Categories / Tags in sidebar.</span></span><br><span class="line"><span class="attr">site_state:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Social Links.</span></span><br><span class="line"><span class="comment"># Usage: `Key: permalink || icon`</span></span><br><span class="line"><span class="comment"># Key is the link label showing to end users.</span></span><br><span class="line"><span class="comment"># Value before `||` delimeter is the target permalink.</span></span><br><span class="line"><span class="comment"># Value after `||` delimeter is the name of FontAwesome icon. If icon (with or without delimeter) is not specified, globe icon will be loaded.</span></span><br><span class="line"><span class="comment">#social:</span></span><br><span class="line"> <span class="comment">#GitHub: https://github.com/yourname || github</span></span><br><span class="line"> <span class="comment">#E-Mail: mailto:yourname@gmail.com || envelope</span></span><br><span class="line"> <span class="comment">#Google: https://plus.google.com/yourname || google</span></span><br><span class="line"> <span class="comment">#Twitter: https://twitter.com/yourname || twitter</span></span><br><span class="line"> <span class="comment">#FB Page: https://www.facebook.com/yourname || facebook</span></span><br><span class="line"> <span class="comment">#VK Group: https://vk.com/yourname || vk</span></span><br><span class="line"> <span class="comment">#StackOverflow: https://stackoverflow.com/yourname || stack-overflow</span></span><br><span class="line"> <span class="comment">#YouTube: https://youtube.com/yourname || youtube</span></span><br><span class="line"> <span class="comment">#Instagram: https://instagram.com/yourname || instagram</span></span><br><span class="line"> <span class="comment">#Skype: skype:yourname?call|chat || skype</span></span><br><span class="line"></span><br><span class="line"><span class="attr">social_icons:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> icons_only:</span> <span class="literal">false</span></span><br><span class="line"><span class="attr"> transition:</span> <span class="literal">false</span></span><br><span class="line"> <span class="comment"># Dependencies: exturl: true in Tags Settings section below.</span></span><br><span class="line"> <span class="comment"># To encrypt links above use https://www.base64encode.org</span></span><br><span class="line"> <span class="comment"># Example encoded link: `GitHub: aHR0cHM6Ly9naXRodWIuY29tL3RoZW1lLW5leHQ= || github`</span></span><br><span class="line"><span class="attr"> exturl:</span> <span class="literal">false</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Follow me on GitHub banner in right-top corner.</span></span><br><span class="line"><span class="comment"># Usage: `permalink || title`</span></span><br><span class="line"><span class="comment"># Value before `||` delimeter is the target permalink.</span></span><br><span class="line"><span class="comment"># Value after `||` delimeter is the title and aria-label name.</span></span><br><span class="line"><span class="comment">#github_banner: https://github.com/yourname || Follow me on GitHub</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Blog rolls</span></span><br><span class="line"><span class="attr">links_icon:</span> <span class="string">link</span></span><br><span class="line"><span class="attr">links_title:</span> <span class="string">Links</span></span><br><span class="line"><span class="attr">links_layout:</span> <span class="string">block</span></span><br><span class="line"><span class="comment">#links_layout: inline</span></span><br><span class="line"><span class="comment">#links:</span></span><br><span class="line"> <span class="comment">#Title: http://example.com/</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Sidebar Avatar</span></span><br><span class="line"><span class="attr">avatar:</span></span><br><span class="line"> <span class="comment"># in theme directory(source/images): /images/avatar.gif</span></span><br><span class="line"> <span class="comment"># in site directory(source/uploads): /uploads/avatar.gif</span></span><br><span class="line"> <span class="comment"># You can also use other linking images.</span></span><br><span class="line"><span class="attr"> url:</span> <span class="comment">#/images/avatar.gif</span></span><br><span class="line"> <span class="comment"># If true, the avatar would be dispalyed in circle. </span></span><br><span class="line"><span class="attr"> rounded:</span> <span class="literal">false</span></span><br><span class="line"> <span class="comment"># The value of opacity should be choose from 0 to 1 to set the opacity of the avatar.</span></span><br><span class="line"><span class="attr"> opacity:</span> <span class="number">1</span></span><br><span class="line"> <span class="comment"># If true, the avatar would be rotated with the cursor.</span></span><br><span class="line"><span class="attr"> rotated:</span> <span class="literal">false</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Table Of Contents in the Sidebar</span></span><br><span class="line"><span class="attr">toc:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Automatically add list number to toc.</span></span><br><span class="line"><span class="attr"> number:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># If true, all words will placed on next lines if header width longer then sidebar width.</span></span><br><span class="line"><span class="attr"> wrap:</span> <span class="literal">false</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Creative Commons 4.0 International License.</span></span><br><span class="line"><span class="comment"># http://creativecommons.org/</span></span><br><span class="line"><span class="comment"># Available: by | by-nc | by-nc-nd | by-nc-sa | by-nd | by-sa | zero</span></span><br><span class="line"><span class="comment">#creative_commons: by-nc-sa</span></span><br><span class="line"><span class="comment">#creative_commons:</span></span><br><span class="line"></span><br><span class="line"><span class="attr">sidebar:</span></span><br><span class="line"> <span class="comment"># Sidebar Position, available value: left | right (only for Pisces | Gemini).</span></span><br><span class="line"><span class="attr"> position:</span> <span class="string">left</span></span><br><span class="line"> <span class="comment">#position: right</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Sidebar Display, available value (only for Muse | Mist):</span></span><br><span class="line"> <span class="comment"># - post expand on posts automatically. Default.</span></span><br><span class="line"> <span class="comment"># - always expand for all pages automatically</span></span><br><span class="line"> <span class="comment"># - hide expand only when click on the sidebar toggle icon.</span></span><br><span class="line"> <span class="comment"># - remove Totally remove sidebar including sidebar toggle.</span></span><br><span class="line"><span class="attr"> display:</span> <span class="string">post</span></span><br><span class="line"> <span class="comment">#display: always</span></span><br><span class="line"> <span class="comment">#display: hide</span></span><br><span class="line"> <span class="comment">#display: remove</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Sidebar offset from top menubar in pixels (only for Pisces | Gemini).</span></span><br><span class="line"><span class="attr"> offset:</span> <span class="number">12</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Back to top in sidebar (only for Pisces | Gemini).</span></span><br><span class="line"><span class="attr"> b2t:</span> <span class="literal">false</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Scroll percent label in b2t button.</span></span><br><span class="line"><span class="attr"> scrollpercent:</span> <span class="literal">false</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Enable sidebar on narrow view (only for Muse | Mist).</span></span><br><span class="line"><span class="attr"> onmobile:</span> <span class="literal">false</span></span><br></pre> </td> </tr> </table> </figure></p><ul> <li><code>social</code>: 这一项被注释掉了，我们先取消注释。配置内容就是对应的社交账号链接。</li> <li><code>github_banner</code>: 右上角的 <strong>follow me on github</strong></li> <li><code>links</code>: 放一些友情链接或其它的你想放的链接</li> <li><code>avatar</code>: 头像的一些设置</li></ul><p>其余的设置参考他给出的注释说明配置即可。下面是我的配置：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># ---------------------------------------------------------------</span></span><br><span class="line"><span class="comment"># Sidebar Settings</span></span><br><span class="line"><span class="comment"># ---------------------------------------------------------------</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Posts / Categories / Tags in sidebar.</span></span><br><span class="line"><span class="attr">site_state:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Social Links.</span></span><br><span class="line"><span class="comment"># Usage: `Key: permalink || icon`</span></span><br><span class="line"><span class="comment"># Key is the link label showing to end users.</span></span><br><span class="line"><span class="comment"># Value before `||` delimeter is the target permalink.</span></span><br><span class="line"><span class="comment"># Value after `||` delimeter is the name of FontAwesome icon. If icon (with or without delimeter) is not specified, globe icon will be loaded.</span></span><br><span class="line"><span class="attr">social:</span></span><br><span class="line"><span class="attr"> GitHub:</span> <span class="attr">https://github.com/tankeryang</span> <span class="string">||</span> <span class="string">github</span></span><br><span class="line"><span class="attr"> E-Mail:</span> <span class="attr">mailto:youngzyang@outlook.com</span> <span class="string">||</span> <span class="string">envelope</span></span><br><span class="line"> <span class="comment">#Google: https://plus.google.com/yourname || google</span></span><br><span class="line"> <span class="comment">#Twitter: https://twitter.com/yourname || twitter</span></span><br><span class="line"> <span class="comment">#FB Page: https://www.facebook.com/yourname || facebook</span></span><br><span class="line"> <span class="comment">#VK Group: https://vk.com/yourname || vk</span></span><br><span class="line"> <span class="comment">#StackOverflow: https://stackoverflow.com/yourname || stack-overflow</span></span><br><span class="line"> <span class="comment">#YouTube: https://youtube.com/yourname || youtube</span></span><br><span class="line"> <span class="comment">#Instagram: https://instagram.com/yourname || instagram</span></span><br><span class="line"> <span class="comment">#Skype: skype:yourname?call|chat || skype</span></span><br><span class="line"></span><br><span class="line"><span class="attr">social_icons:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> icons_only:</span> <span class="literal">false</span></span><br><span class="line"><span class="attr"> transition:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Follow me on GitHub banner in right-top corner.</span></span><br><span class="line"><span class="comment"># Usage: `permalink || title`</span></span><br><span class="line"><span class="comment"># Value before `||` delimeter is the target permalink.</span></span><br><span class="line"><span class="comment"># Value after `||` delimeter is the title and aria-label name.</span></span><br><span class="line"><span class="attr">github_banner:</span> <span class="attr">https://github.com/tankeryang</span> <span class="string">||</span> <span class="string">Follow</span> <span class="string">me</span> <span class="string">on</span> <span class="string">GitHub</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Blog rolls</span></span><br><span class="line"><span class="attr">links_icon:</span> <span class="string">link</span></span><br><span class="line"><span class="attr">links_title:</span> <span class="string">博客镜像</span> <span class="string">&</span> <span class="string">友情链接</span></span><br><span class="line"><span class="attr">links_layout:</span> <span class="string">block</span></span><br><span class="line"><span class="comment">#links_layout: inline</span></span><br><span class="line"><span class="attr">links:</span></span><br><span class="line"> <span class="string">淦</span> <span class="bullet">-</span> <span class="attr">github:</span> <span class="attr">https://tankeryang.github.io</span></span><br><span class="line"> <span class="string">淦</span> <span class="bullet">-</span> <span class="attr">coding:</span> <span class="attr">https://tankeryang.coding.me</span></span><br><span class="line"> <span class="string">淦</span> <span class="bullet">-</span> <span class="attr">gitee:</span> <span class="attr">http://tankeryang.gitee.io</span></span><br><span class="line"> <span class="string">未知:</span> <span class="attr">https://deeeeeeeee.github.io</span></span><br><span class="line"> <span class="string">简单可依赖:</span> <span class="attr">https://www.tiexo.cn/</span></span><br><span class="line"> <span class="string">刘伟的博客:</span> <span class="attr">https://darrenliuwei.com</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Sidebar Avatar</span></span><br><span class="line"><span class="comment"># in theme directory(source/images): /images/avatar.gif</span></span><br><span class="line"><span class="comment"># in site directory(source/uploads): /uploads/avatar.gif</span></span><br><span class="line"><span class="attr">avatar:</span> </span><br><span class="line"><span class="attr"> url:</span> <span class="string">/uploads/my.jpg</span></span><br><span class="line"><span class="attr"> rounded:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Table Of Contents in the Sidebar</span></span><br><span class="line"><span class="attr">toc:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Automatically add list number to toc.</span></span><br><span class="line"><span class="attr"> number:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># If true, all words will placed on next lines if header width longer then sidebar width.</span></span><br><span class="line"><span class="attr"> wrap:</span> <span class="literal">false</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Creative Commons 4.0 International License.</span></span><br><span class="line"><span class="comment"># http://creativecommons.org/</span></span><br><span class="line"><span class="comment"># Available: by | by-nc | by-nc-nd | by-nc-sa | by-nd | by-sa | zero</span></span><br><span class="line"><span class="attr">creative_commons:</span> <span class="string">by-nc-sa</span></span><br><span class="line"><span class="comment">#creative_commons:</span></span><br><span class="line"></span><br><span class="line"><span class="attr">sidebar:</span></span><br><span class="line"> <span class="comment"># Sidebar Position, available value: left | right (only for Pisces | Gemini).</span></span><br><span class="line"><span class="attr"> position:</span> <span class="string">left</span></span><br><span class="line"> <span class="comment">#position: right</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Sidebar Display, available value (only for Muse | Mist):</span></span><br><span class="line"> <span class="comment"># - post expand on posts automatically. Default.</span></span><br><span class="line"> <span class="comment"># - always expand for all pages automatically</span></span><br><span class="line"> <span class="comment"># - hide expand only when click on the sidebar toggle icon.</span></span><br><span class="line"> <span class="comment"># - remove Totally remove sidebar including sidebar toggle.</span></span><br><span class="line"> <span class="comment">#display: post</span></span><br><span class="line"><span class="attr"> display:</span> <span class="string">always</span></span><br><span class="line"> <span class="comment">#display: hide</span></span><br><span class="line"> <span class="comment">#display: remove</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Sidebar offset from top menubar in pixels (only for Pisces | Gemini).</span></span><br><span class="line"><span class="attr"> offset:</span> <span class="number">12</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Back to top in sidebar (only for Pisces | Gemini).</span></span><br><span class="line"><span class="attr"> b2t:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Scroll percent label in b2t button.</span></span><br><span class="line"><span class="attr"> scrollpercent:</span> <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Enable sidebar on narrow view (only for Muse | Mist).</span></span><br><span class="line"><span class="attr"> onmobile:</span> <span class="literal">true</span></span><br></pre> </td> </tr> </table> </figure></p><h2 id="个性化设置"><a href="#个性化设置" class="headerlink" title="个性化设置"></a>个性化设置</h2><p>因为NexT可配置的选项太多，在这里我就不一一展开了，下面在介绍两个个性化的设置</p><h3 id="主题标签插件"><a href="#主题标签插件" class="headerlink" title="主题标签插件"></a>主题标签插件</h3><p>关于标签插件，大家可以回顾一下前面的内容。NexT也自带了一些标签插件供用户使用</p><h4 id="note"><a href="#note" class="headerlink" title="note"></a>note</h4><div class="note default"> <p>提示块标签，效果就像你看到的这个提示块</p></div><p>配置如下：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># Note tag (bs-callout).</span></span><br><span class="line"><span class="attr">note:</span></span><br><span class="line"> <span class="comment"># Note tag style values:</span></span><br><span class="line"> <span class="comment"># - simple bs-callout old alert style. Default.</span></span><br><span class="line"> <span class="comment"># - modern bs-callout new (v2-v3) alert style.</span></span><br><span class="line"> <span class="comment"># - flat flat callout style with background, like on Mozilla or StackOverflow.</span></span><br><span class="line"> <span class="comment"># - disabled disable all CSS styles import of note tag.</span></span><br><span class="line"><span class="attr"> style:</span> <span class="string">flat</span></span><br><span class="line"><span class="attr"> icons:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> border_radius:</span> <span class="number">2</span></span><br><span class="line"> <span class="comment"># Offset lighter of background in % for modern and flat styles (modern: -12 | 12; flat: -18 | 6).</span></span><br><span class="line"> <span class="comment"># Offset also applied to label tag variables. This option can work with disabled note tag.</span></span><br><span class="line"><span class="attr"> light_bg_offset:</span> <span class="number">3</span></span><br></pre> </td> </tr> </table> </figure></p><p>使用方法：<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre> </td> <td class="code"> <pre><span class="line">{% note class %}</span><br><span class="line">Any content (support inline tags too).</span><br><span class="line">{% endnote %}</span><br><span class="line"></span><br><span class="line"># 支持的class: class : default | primary | success | info | warning | danger.</span><br></pre> </td> </tr> </table> </figure></p><h4 id="label"><a href="#label" class="headerlink" title="label"></a>label</h4><span class="label primary">label标签，给文字加底色</span><p>配置如下：<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line"># Label tag.</span><br><span class="line">label: true</span><br></pre> </td> </tr> </table> </figure></p><p>使用方法：<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre> </td> <td class="code"> <pre><span class="line">{% label class@Text %}</span><br><span class="line"></span><br><span class="line"># 支持的class : default | primary | success | info | warning | danger.</span><br></pre> </td> </tr> </table> </figure></p><h4 id="选项卡"><a href="#选项卡" class="headerlink" title="选项卡"></a>选项卡</h4><p>效果如下：<br> <div class="tabs" id="选项卡1-选项卡2"> <ul class="nav-tabs"> <li class="tab active"><a href="#选项卡1-选项卡2-1">选项卡1</a></li> <li class="tab"><a href="#选项卡1-选项卡2-2">选项卡2</a></li> </ul> <div class="tab-content"> <div class="tab-pane active" id="选项卡1-选项卡2-1"> <p>我是选项卡1</p> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">我是代码框1</span><br></pre> </td> </tr> </table> </figure> </div> <div class="tab-pane" id="选项卡1-选项卡2-2"> <p>我是选项卡2</p> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br></pre> </td> <td class="code"> <pre><span class="line">我是代码框2</span><br></pre> </td> </tr> </table> </figure> </div> </div> </div><br>配置如下：<br> <figure class="highlight yaml"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="comment"># Tabs tag.</span></span><br><span class="line"><span class="attr">tabs:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> transition:</span></span><br><span class="line"><span class="attr"> tabs:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> labels:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> border_radius:</span> <span class="number">3</span></span><br></pre> </td> </tr> </table> </figure></p><p>使用方法：<br> <figure class="highlight plain"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre> </td> <td class="code"> <pre><span class="line">{% tabs 选项卡1 选项卡2 %}</span><br><span class="line"><!-- tab 选项卡1 --></span><br><span class="line">我是选项卡1</span><br><span class="line">{% codeblock %}</span><br><span class="line">我是代码框1</span><br><span class="line">{% endcodeblock %}</span><br><span class="line"> <!-- endtab --></span><br><span class="line"></span><br><span class="line"> <!-- tab 选项卡2 --></span><br><span class="line">我是选项卡2</span><br><span class="line">{% codeblock %}</span><br><span class="line">我是代码框2</span><br><span class="line">{% endcodeblock %}</span><br><span class="line"> <!-- endtab --></span><br><span class="line">{% endtabs %}</span><br></pre> </td> </tr> </table> </figure></p><div class="note info"> <p>更多设置请参考<span class="label danger">v5.x</span>版本的<a href="http://theme-next.iissnan.com/" target="_blank" rel="noopener">NexT文档</a>和最新的<span class="label primary">v6.x</span>版本的<a href="https://github.com/theme-next/hexo-theme-next/tree/master/docs/zh-CN" target="_blank" rel="noopener">NexT文档</a> </p></div><hr><h1 id="配置Git与Travic-CI持续集成"><a href="#配置Git与Travic-CI持续集成" class="headerlink" title="配置Git与Travic-CI持续集成"></a>配置Git与Travic-CI持续集成</h1><div class="note warning"> <p>这一步是最能体现 <strong>自动化博客写作流程</strong> 的关键。务必仔细阅读。</p></div><h2 id="github，gitee，coding新建博客项目"><a href="#github，gitee，coding新建博客项目" class="headerlink" title="github，gitee，coding新建博客项目"></a>github，gitee，coding新建博客项目</h2><p>在github上新开一个repo，<strong>repo名: <code>USERNAME.github.io</code></strong><br>同理，<strong>coding repo名: <code>USERNAME</code></strong>， <strong>gitee repo名: <code>USERNAME</code></strong></p><blockquote> <p><code>USERNAME</code>为你的github，gitee，coding用户名<br>pages服务参考：<a href="https://pages.github.com/" target="_blank" rel="noopener">github</a>，<a href="https://coding.net/help/doc/pages/creating-pages.html" target="_blank" rel="noopener">coding</a>，<a href="http://git.mydoc.io/?t=154714" target="_blank" rel="noopener">gitee</a> </p></blockquote><h2 id="添加-gitignore"><a href="#添加-gitignore" class="headerlink" title="添加.gitignore"></a>添加.gitignore</h2><p>进入<span class="label default">myblog</span>文件夹，新增<span class="label default">.gitignore</span>文件，若存在则检查是否与如下一致：<br> <figure class="highlight plain"> <figcaption><span>.gitignore</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre> </td> <td class="code"> <pre><span class="line">.DS_Store</span><br><span class="line">Thumbs.db</span><br><span class="line">db.json</span><br><span class="line">*.log</span><br><span class="line">node_modules/</span><br><span class="line">public/</span><br><span class="line">.deploy*/</span><br></pre> </td> </tr> </table> </figure></p><h2 id="设置travis关联USERNAME-github-io"><a href="#设置travis关联USERNAME-github-io" class="headerlink" title="设置travis关联USERNAME.github.io"></a>设置travis关联USERNAME.github.io</h2><ul> <li> <p>先用github账号登陆<a href="https://www.travis-ci.org/" target="_blank" rel="noopener">travis</a></p> </li> <li> <p>勾选你的博客项目，顺便点旁边的<span class="label primary">setting</span>进入设置，参照图2来勾选： </p> <img src="/posts/Hexo%20+%20NexT%20+%20Github%20Pages%20+%20Coding%20Pages%20+%20Gitee%20Pages%20+%20Travis%20全攻略/pic5.png"> <img src="/posts/Hexo%20+%20NexT%20+%20Github%20Pages%20+%20Coding%20Pages%20+%20Gitee%20Pages%20+%20Travis%20全攻略/pic6.png"> </li></ul><h2 id="为博客项目设置deploy-key"><a href="#为博客项目设置deploy-key" class="headerlink" title="为博客项目设置deploy key"></a>为博客项目设置deploy key</h2><p>这一步比较重要，这是travis能否访问你的repo的关键。按照下面的步骤配置：</p><ul> <li> <p> 在<code>myblog</code>下新建一个<code>.travis</code>文件夹，并进入<code>.travis</code>文件夹 </p> </li> <li> <p>在<code>myblog/.travis</code>下生成一个专门给travis用的<code>ssh key</code>:<br>执行<code>ssh-keygen -t rsa -f travis_rsa -C "your@email.com"</code>，遇到密码输入直接<code>enter回车</code>，email记得替换 </p> </li> <li> <p> github博客项目添加travis_rsa.pub公钥<br>将<code>.travis</code>下的<code>travis_rsa.pub</code>的内容复制，按下图进入<code>Add deploy key</code>界面，将内容粘贴，<strong>并勾选可读写权限（一定不要遗漏!!!）</strong></p> <img src="/posts/Hexo%20+%20NexT%20+%20Github%20Pages%20+%20Coding%20Pages%20+%20Gitee%20Pages%20+%20Travis%20全攻略/pic7.png"> </li> <li> <p>coding过程同上</p> </li> <li> <p>gitee因为项目的<code>deploy key</code>不能设置可写入权限，所以只能将<code>travis_rsa.pub</code>添加到个人的公钥</p> </li></ul><h2 id="加密travis-rsa"><a href="#加密travis-rsa" class="headerlink" title="加密travis_rsa"></a>加密travis_rsa</h2><ul> <li> <p>先安装<a href="https://rubygems.org/" target="_blank" rel="noopener">gem</a>（关于ruby，gem相关的安装这里就不列出来了，我相信你的动手能力:)） </p> </li> <li> <p>接着安装travis命令行工具: <code>sudo gem install travis</code></p> </li> <li> <p>装好后，进入<code>myblog/.travis</code>文件夹，在当前路径下执行如下命令：</p> <figure class="highlight sh"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br></pre> </td> <td class="code"> <pre><span class="line">travis login --auto</span><br><span class="line">travis encrypt-file travis_rsa -add</span><br></pre> </td> </tr> </table> </figure> </li></ul><div class="note warning"> <p> 需要注意的是，travis命令行工具貌似在Windows下不太好使…因此需要换另外的方法使得travis能访问到github，coding和gitee上的博客项目，主要是通过 <strong>token</strong> 来访问。相关的配置百度一下就有很多教程，比ssh来的更简单。就是在 <strong>安全</strong> 性上差了一点，因为 <strong>token</strong> 是有操作所有 <strong>repo</strong> 的权限的…</p></div><p> 加密完成后你会发现<code>.travis</code>文件夹下多了个<code>travis_rsa.enc</code>的文件，这个就是加密后的<code>travis_rsa</code>。同时<code>myblog</code>下也多了<code>.travis.yml</code>文件。下面我们就配置一下<code>.travis.yml</code>。</p><ul> <li> <p> 先将<code>.travis</code>下的<code>travis_rsa</code>，<code>travis_rsa.pub</code>删掉。<strong>请记住一定要删掉！！！只保留加密文件</strong> </p> </li> <li> <p>按照下面配置<code>.travis.yml</code>:</p> <figure class="highlight yaml"> <figcaption><span>.travis.yml</span></figcaption> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="attr">language:</span> <span class="string">node_js</span></span><br><span class="line"><span class="attr">node_js:</span> <span class="string">stable</span></span><br><span class="line"></span><br><span class="line"><span class="attr">branches:</span></span><br><span class="line"><span class="attr"> only:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">dev</span></span><br><span class="line"></span><br><span class="line"><span class="attr">before_install:</span></span><br><span class="line"><span class="comment"># 解密SSH</span></span><br><span class="line"><span class="comment"># encrypted_key和encrypted_id可以在travis里setting查看，具体位置参考下图</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">openssl</span> <span class="string">aes-256-cbc</span> <span class="bullet">-K</span> <span class="string">${你的encrypted_xxxx_key}</span> <span class="bullet">-iv</span> <span class="string">${你的encrypted_xxxx_iv}</span></span><br><span class="line"><span class="bullet"> -</span><span class="string">in</span> <span class="string">.travis/travis_rsa.enc</span> <span class="bullet">-out</span> <span class="string">~/.ssh/travis_rsa</span> <span class="bullet">-d</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">chmod</span> <span class="number">600</span> <span class="string">~/.ssh/travis_rsa</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">mv</span> <span class="bullet">-fv</span> <span class="string">.travis/config</span> <span class="string">~/.ssh/config</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 安装hexo以及一些插件，没用到的可移除相关命令</span></span><br><span class="line"><span class="attr">install:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">hexo-cli</span> <span class="bullet">-g</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">hexo-math</span> <span class="bullet">--save</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">hexo-generator-searchdb</span> <span class="bullet">--save</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">hexo-symbols-count-time</span> <span class="bullet">--save</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">hexo-generator-feed</span> <span class="bullet">--save</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">hexo-wordcount</span> <span class="bullet">--save</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">mermaid</span> <span class="bullet">--save</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">hexo-tag-mermaid</span> <span class="bullet">--save</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">npm</span> <span class="string">install</span> <span class="string">hexo-tag-plantuml</span> <span class="bullet">--save</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 安装next主题插件，没用到的可移除相关命令</span></span><br><span class="line"><span class="attr">before_script:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">cd</span> <span class="string">./themes/next</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">clone</span> <span class="attr">https://github.com/theme-next/theme-next-fancybox3</span> <span class="string">source/lib/fancybox</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">clone</span> <span class="attr">https://github.com/theme-next/theme-next-jquery-lazyload</span> <span class="string">source/lib/jquery_lazyload</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">clone</span> <span class="attr">https://github.com/theme-next/theme-next-needmoreshare2</span> <span class="string">source/lib/needsharebutton</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">clone</span> <span class="attr">https://github.com/theme-next/theme-next-pace</span> <span class="string">source/lib/pace</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">clone</span> <span class="attr">https://github.com/theme-next/theme-next-pangu.git</span> <span class="string">source/lib/pangu</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">clone</span> <span class="attr">https://github.com/theme-next/theme-next-reading-progress</span> <span class="string">source/lib/reading_progress</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">cd</span> <span class="string">..</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">cd</span> <span class="string">..</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 构建博客</span></span><br><span class="line"><span class="attr">script:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">hexo</span> <span class="bullet">-version</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">hexo</span> <span class="string">clean</span> <span class="string">&&</span> <span class="string">hexo</span> <span class="string">g</span></span><br><span class="line"></span><br><span class="line"><span class="attr">after_script:</span></span><br><span class="line"><span class="comment"># 设置环境</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">set</span> <span class="bullet">-ev</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">export</span> <span class="string">TZ='Asia/Shanghai'</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">cd</span> <span class="string">./public</span></span><br><span class="line"><span class="comment"># 提交</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">config</span> <span class="string">user.name</span> <span class="string">"USERNAME"</span> <span class="comment"># 你的github用户名</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">config</span> <span class="string">user.email</span> <span class="string">"your@email.com"</span> <span class="comment"># 你的github邮箱</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">init</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">add</span> <span class="string">.</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">commit</span> <span class="bullet">-m</span> <span class="string">"Site updated:`date +"</span><span class="string">%Y-%m-%d</span> <span class="string">%H:%M:%S"`"</span></span><br><span class="line"><span class="comment"># push</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">remote</span> <span class="string">add</span> <span class="string">coding</span> <span class="string">git@git.coding.net:USERNAME/USERNAME.git</span> <span class="comment"># USERNAME为你的coding用户名</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">remote</span> <span class="string">add</span> <span class="string">gitee</span> <span class="string">git@gitee.com:USERNAME/USERNAME.git</span> <span class="comment"># USERNAME为你的gitee用户名</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">push</span> <span class="bullet">-u</span> <span class="string">origin</span> <span class="string">master</span> <span class="bullet">-f</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">push</span> <span class="bullet">-u</span> <span class="string">coding</span> <span class="string">master</span> <span class="bullet">-f</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">git</span> <span class="string">push</span> <span class="bullet">-u</span> <span class="string">gitee</span> <span class="string">master</span> <span class="bullet">-f</span></span><br></pre> </td> </tr> </table> </figure> </li></ul><img src="/posts/Hexo%20+%20NexT%20+%20Github%20Pages%20+%20Coding%20Pages%20+%20Gitee%20Pages%20+%20Travis%20全攻略/pic8.jpeg"><h2 id="最后，将你的本地博客目录与远程github博客项目关联"><a href="#最后，将你的本地博客目录与远程github博客项目关联" class="headerlink" title="最后，将你的本地博客目录与远程github博客项目关联"></a>最后，将你的本地博客目录与远程github博客项目关联</h2><ul> <li> <p>先初始化本地博客目录用git管理</p> <figure class="highlight sh"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre> </td> <td class="code"> <pre><span class="line">git init</span><br><span class="line">git remote add origin git @github.com:USERNAME/USERNAME.github.io.git</span><br><span class="line">git checkout -b dev</span><br><span class="line">git add .</span><br><span class="line">git commit -m <span class="string">"first commit"</span></span><br><span class="line">git push -u origin dev</span><br></pre> </td> </tr> </table> </figure> </li> <li> <p>上<a href="https://www.travis-ci.org/" target="_blank" rel="noopener">travis</a>检查构建流程，看看是否有问题，有问题再根据日志进行<code>.travis.yml</code>的修改 </p> </li> <li> <p>构建完成后，分享到朋友圈:)</p> </li> <li> <p>之后写完文章只要<code>git push origin dev</code>就ok了。注意要在<code>dev</code>分支下进行。可以通过<code>git checkout dev</code>切换分支</p> </li></ul><p>从年初开始写，一直到今天（2018-05-23）才写完…足以证明我是一个多么<del>懒到没谱</del>持之以恒的人，这样的博主还不赶快献爱心一个？</p>]]></content>
<summary type="html">
<img src="/posts/Hexo%20+%20NexT%20+%20Github%20Pages%20+%20Coding%20Pages%20+%20Gitee%20Pages%20+%20Travis%20全攻略/pic1.png">
</summary>
<category term="技术杂项" scheme="https://tankeryang.github.io/categories/%E6%8A%80%E6%9C%AF%E6%9D%82%E9%A1%B9/"/>
<category term="Hexo" scheme="https://tankeryang.github.io/tags/Hexo/"/>
<category term="NexT" scheme="https://tankeryang.github.io/tags/NexT/"/>
<category term="Github Pages" scheme="https://tankeryang.github.io/tags/Github-Pages/"/>
<category term="Coding Pages" scheme="https://tankeryang.github.io/tags/Coding-Pages/"/>
<category term="Gitee Pages" scheme="https://tankeryang.github.io/tags/Gitee-Pages/"/>
<category term="Travis" scheme="https://tankeryang.github.io/tags/Travis/"/>
</entry>
<entry>
<title>半次元爬虫 banciyuan-downloader v1.0 发布</title>
<link href="https://tankeryang.github.io/posts/%E5%8D%8A%E6%AC%A1%E5%85%83%E7%88%AC%E8%99%AB-banciyuan-downloader-v1-0%20%E5%8F%91%E5%B8%83/"/>
<id>https://tankeryang.github.io/posts/半次元爬虫-banciyuan-downloader-v1-0 发布/</id>
<published>2017-10-29T17:14:13.000Z</published>
<updated>2019-10-09T17:06:09.615Z</updated>
<content type="html"><![CDATA[<p><strong> <center> <font color="#ff0000"> <h1>请勿用于商业用途 尊重coser的版权 转载图片请注明Cn及链接</h1> </font> </center> </strong></p><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/16.jpg"><a id="more"></a><hr><p>今天突然看到一组 <a href="https://bcy.net/coser/detail/99095/1511757" target="_blank" rel="noopener">《小魔女学院》的cos图</a>，小姐姐美炸了。</p><blockquote> <p><i class="fa fa-wheelchair-alt" aria-hidden="true"></i> 因为加载速度太慢，我只能调低了图片的分辨率，各位将就一下，想看原图的到上面的链接慢慢欣赏。 </p></blockquote><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/1.jpg"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/3.jpg"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/4.jpg"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/6.jpg"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/9.jpg"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/10.jpg"><p><br></p><p>于是乎搜到了这位小姐姐<a href="https://bcy.net/u/770554" target="_blank" rel="noopener">【cn:犬神洛洛子】</a><br>cos粉毛那位</p><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/5.jpg"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/6_.jpg"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/42.jpg"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/45.jpg"><p><br></p><p>这是她的<a href="https://bcy.net/u/770554" target="_blank" rel="noopener">半次元主页</a></p><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/coser_page.png"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/post_page.png"><p><br></p><p>关注一波。想着有空写个爬虫来爬她的原图吧。结果越写添加的功能越多。现在这个版本我push到了<a href="https://github.com/tankeryang/banciyuan-downloader" target="_blank" rel="noopener">github</a>，上面有详细的使用方法。</p><h1 id="目前版本支持功能情况"><a href="#目前版本支持功能情况" class="headerlink" title="目前版本支持功能情况"></a>目前版本支持功能情况</h1><ul> <li style="list-style: none"><input type="checkbox" checked> 根据<code>coser_id</code>批量下载某个coser发布的主题的所有图片</li> <li style="list-style: none"><input type="checkbox" checked> 图片保存在以 <strong>coser名</strong> 命名的文件夹内</li> <li style="list-style: none"><input type="checkbox" checked> 图片按 <strong>coser发布的主题</strong> 分文件夹保存，文件夹以主题标题命名</li> <li style="list-style: none"><input type="checkbox" checked> 若有相同标题的主题，则命名文件夹时会加上随机后缀防止文件名冲突</li> <li style="list-style: none"><input type="checkbox" checked> 图片命名格式为<code>%num%.jpg</code>/<code>%num%.png</code>，其中<code>%num%</code>为从<code>1</code>开始的 <strong>编号</strong> </li> <li style="list-style: none"><input type="checkbox" checked> 只下载 <strong>最新</strong> 发布的图片，本地已有的 <strong>不会</strong> 重复下载</li> <li style="list-style: none"><input type="checkbox" checked> 支持 <strong>断点续传</strong> （从断连主题的下一个主题开始下载，若断连主题没下载完，则会丢失一部分断连主题的图片，其余均不影响）</li> <li style="list-style: none"><input type="checkbox"> <del>无须提供半次元账号即可下载</del></li> <li style="list-style: none"><input type="checkbox"> <del>下载指定主题</del> </li> <li style="list-style: none"><input type="checkbox"> <del>智能下载 <strong>未下载过的主题</strong></del></li> <li style="list-style: none"><input type="checkbox"> <del>无须提前关注就可下载 <strong>粉丝可见的主题</strong></del></li></ul><h1 id="运行环境及python包版本（本人）"><a href="#运行环境及python包版本（本人）" class="headerlink" title="运行环境及python包版本（本人）"></a>运行环境及python包版本（本人）</h1><ul> <li><strong>windows</strong> 10 1703-15063.674</li> <li><strong>python</strong> 3.6.1</li> <li><strong>beautifulsoup</strong> 4.5.3</li> <li><strong>requests</strong> 2.13.0</li> <li><strong>lxml</strong> 3.7.2</li></ul><h1 id="运行结果"><a href="#运行结果" class="headerlink" title="运行结果"></a>运行结果</h1><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/home_folder.png"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/coser_folder.png"><img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/post_folder.png"><h1 id="FAQ"><a href="#FAQ" class="headerlink" title="FAQ"></a>FAQ</h1><p>有何疑问可在<a href="https://github.com/tankeryang/banciyuan-downloader" target="_blank" rel="noopener">github</a>发布<a href="https://github.com/tankeryang/banciyuan-downloader/issues" target="_blank" rel="noopener">issue</a> ，本人会尽量及时查看</p><h1 id="疯狂打call"><a href="#疯狂打call" class="headerlink" title="疯狂打call"></a>疯狂打call</h1><p>原图出处: <a href="https://bcy.net/coser/detail/99095/1511757" target="_blank" rel="noopener">小魔女学园-主角三人搞事组cos</a><br>苏西·曼芭芭拉(粉毛) <strong>cn</strong> :<a href="https://bcy.net/u/770554" target="_blank" rel="noopener">犬神洛洛子</a><br>亚可·卡嘉莉(黑毛) <strong>cn</strong> :real__yami<br>洛蒂·杨森(黄毛) <strong>cn</strong> :<a href="https://bcy.net/u/48018" target="_blank" rel="noopener">樱群</a></p>]]></content>
<summary type="html">
<p><strong>
<center>
<font color="#ff0000">
<h1>请勿用于商业用途 尊重coser的版权 转载图片请注明Cn及链接</h1>
</font>
</center>
</strong></p>
<img src="/posts/半次元爬虫-banciyuan-downloader-v1-0%20发布/16.jpg">
</summary>
<category term="python" scheme="https://tankeryang.github.io/categories/python/"/>
<category term="python" scheme="https://tankeryang.github.io/tags/python/"/>
<category term="爬虫" scheme="https://tankeryang.github.io/tags/%E7%88%AC%E8%99%AB/"/>
<category term="二次元" scheme="https://tankeryang.github.io/tags/%E4%BA%8C%E6%AC%A1%E5%85%83/"/>
</entry>
<entry>
<title>Machine Learning (Week3)</title>
<link href="https://tankeryang.github.io/posts/Machine%20Learning%20(Week3)/"/>
<id>https://tankeryang.github.io/posts/Machine Learning (Week3)/</id>
<published>2017-10-03T22:26:22.000Z</published>
<updated>2019-10-09T17:06:09.571Z</updated>
<content type="html"><![CDATA[<p>在第三周的课程里，介绍了<strong>Logistic Regression - 逻辑斯谛回归</strong>问题，主要应用在<strong>Classification - 分类</strong>上。还有<strong>Regularization - 正则化</strong>，如何用来解决<strong>Overfitting - 过拟合</strong>问题。</p><img src="/posts/Machine%20Learning%20(Week3)/pic0.png"><a id="more"></a><h1 id="Logistic-Regression-逻辑斯谛回归"><a href="#Logistic-Regression-逻辑斯谛回归" class="headerlink" title="Logistic Regression - 逻辑斯谛回归"></a>Logistic Regression - 逻辑斯谛回归</h1><h2 id="Classification-分类问题"><a href="#Classification-分类问题" class="headerlink" title="Classification - 分类问题"></a>Classification - 分类问题</h2><p> 分类在日常中用在很多地方，比如邮件是否垃圾邮件，肿瘤是否良性。通过给定的特征与对应的类别，我们可以训练出一个能够进行分类的算法。相应的，垃圾邮件的特征可以是某些关键词，比如推销类的；而肿瘤的特征可以是肿瘤大小或者别的什么（不懂就不胡说了）。<br>这时我们的结果$y$就是离散化的数字。</p><ul> <li>如果是<strong>Binary Classification - 二分类</strong>问题，$y$可以离散化为$y = 0 or 1$，对应<strong>是/否</strong>，<strong>大/小</strong>等抽象的结果。</li> <li>如果是<strong>Multiple Classification - 多分类</strong>问题，$y$可以离散化为<strong>元素个数为类别个数的向量</strong>。比如我们需要对某组数据进行分类，训练样本中一共有$n$类，当前训练样本的$y$是属于第$i$类的，则令<span>$y = [0_{1},0_{2},\cdots,1_{i},\cdots,0_{n}]^{T}$</span> <!-- Has MathJax -->，其中下标位置为对应类别。</li></ul><p>这里我们先讨论<strong>Binary Classification - 二分类</strong>问题。同样先来个例子。</p><p>假如我们有一组肿瘤大小与其对应性质（良性/恶性）的数据，特征只有一个，就是肿瘤大小，$y=1$和<font color="#ff0000">红色X点 </font>代表恶性。如下图：<br><img src="/posts/Machine%20Learning%20(Week3)/pic1.png"><br>我们看到，假如我们用单纯的线性方程来拟合这组数据，按照图示定义，当<span>$h_{\theta} = \theta^{T}x >0.5$</span> <!-- Has MathJax -->时预测为恶性，则会与原数据<strong>误差较大</strong>。显然单纯的线性方程很难做到精准的分类。</p><p> 我们尝试换一种思路。可以看到，两种类型的数据在某一$x$值上会有明显的区分。在$x$左边是良性的，在$x$右边是恶性的。于是我们做如下分析：<br><img src="/posts/Machine%20Learning%20(Week3)/pic2.png"><br>令分界点<span>$x = \frac{-\theta_{0}}{\theta_{1}}$</span><!-- Has MathJax -->，则当<span>$x > \frac{-\theta_{0}}{\theta_{1}}$</span> <!-- Has MathJax -->即<span>$\theta^{T}x = \theta_{0}+\theta_{1}x > 0$</span><!-- Has MathJax -->时，预测$y=1$，当<span>$x < \frac{-\theta_{0}}{\theta_{1}}$</span> <!-- Has MathJax -->即<span>$\theta^{T}x = \theta_{0}+\theta_{1}x < 0$</span><!-- Has MathJax -->时，预测$y=0$。这样就能得到很好的分类效果。</p><p>同样的，多特征时也可以如此这般。比如两个特征时的情况：<br><img src="/posts/Machine%20Learning%20(Week3)/pic3.png"></p><p>因此我们只要一个函数<span>$g(\theta^{T}x)$</span><!-- Has MathJax --></p><ul> <li>当<span>$\theta^{T}x > 0$</span> <!-- Has MathJax -->时，<span>$g(\theta^{T}x) > 0.5$</span> <!-- Has MathJax --> </li> <li>当<span>$\theta^{T}x < 0$</span> <!-- Has MathJax -->时，<span>$g(\theta^{T}x) < 0.5$</span> <!-- Has MathJax --> </li></ul><p>最后令<span>$h_{\theta}(x) = g(\theta^{T}x)$</span><!-- Has MathJax -->，就ok了。</p><h2 id="Hypothesis-Representation-假设函数的表示"><a href="#Hypothesis-Representation-假设函数的表示" class="headerlink" title="Hypothesis Representation - 假设函数的表示"></a>Hypothesis Representation - 假设函数的表示</h2><p>接着上面的问题。我们给出这样一个函数</p><center><br><span>$g(z) = \frac{1}{1+e^{-z}}$</span><!-- Has MathJax --><br></center><p>它的图像如下<br><img src="/posts/Machine%20Learning%20(Week3)/pic4.jpg"></p><p>它满足下述性质</p><ul> <li>当<span>$z > 0$</span><!-- Has MathJax -->时，<span>$g(z) > 0.5$</span> <!-- Has MathJax --> </li> <li>当<span>$z < 0$</span><!-- Has MathJax -->时，<span>$g(z) < 0.5$</span> <!-- Has MathJax --> </li></ul><p>因此，我们只需令<span>$\theta^{T}x = z$</span><!-- Has MathJax -->，即</p><center><br><span>$g(\theta^{T}x) = \frac{1}{1+e^{-\theta^{T}x}}$</span> <!-- Has MathJax --><br></center><p>则可得我们的假设函数</p><center><br><span>$h_{\theta}(x) = g(\theta^{T}x) = \frac{1}{1+e^{-\theta^{T}x}}$</span><!-- Has MathJax --><br></center><p>这里<span>$h_{\theta}(x)$</span><!-- Has MathJax -->实际上可以理解为如下</p><center><br><span>$h_{\theta}(x) = p(y = 1|x;\theta)$</span> <!-- Has MathJax --><br></center><p>即输入<span>$x$</span><!-- Has MathJax -->的情况下，预测结果为<span>$1$</span> <!-- Has MathJax -->的概率为<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->。<br>比如说这是一个良性/恶性肿瘤的分类问题，我们训练出了<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->。现在有一个肿瘤的各项特征为<span>$x$</span> <!-- Has MathJax -->，我们要判断它是良性<span>$(y=1)$</span> <!-- Has MathJax -->还是恶性<span>$(y=0)$</span> <!-- Has MathJax -->，把<span>$x$</span> <!-- Has MathJax -->丢进<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->里，结果为<span>$0.8$</span> <!-- Has MathJax -->，那么我们就可以说这个肿瘤有<span>$80\%$</span> <!-- Has MathJax -->的概率是良性的。假如我们设置了一个<strong>阈值</strong>为<span>$0.7$</span> <!-- Has MathJax -->，计算结果超过这个阈值就可以声明预测为真，那么在上面的情况中，我们就可以直接对病人说你的肿瘤是良性的，不用担心。</p><h2 id="Decision-Boundary-判定边界"><a href="#Decision-Boundary-判定边界" class="headerlink" title="Decision Boundary - 判定边界"></a>Decision Boundary - 判定边界</h2><p>在逻辑斯谛回归中，我们一般</p><ul> <li>当<span>$h_{\theta}(x) > 0.5$</span> <!-- Has MathJax -->，即当<span>$\theta^{T}x > 0$</span> <!-- Has MathJax -->时，预测<span>$y=1$</span><!-- Has MathJax --> </li> <li>当<span>$h_{\theta}(x) < 0.5$</span> <!-- Has MathJax -->，即当<span>$\theta^{T}x < 0$</span> <!-- Has MathJax -->时，预测<span>$y=0$</span><!-- Has MathJax --> </li></ul><p>如下图（见上节）<br><img src="/posts/Machine%20Learning%20(Week3)/pic2.png"><br><img src="/posts/Machine%20Learning%20(Week3)/pic3.png"></p><p>此时<strong>阈值</strong>为<span>$0.5$</span><!-- Has MathJax --></p><p>对于一般的问题，<span>$0.5$</span> <!-- Has MathJax -->的阈值足以胜任。可是假如是一些精确度要求很高的问题，就比如刚刚的判断肿瘤是否良性，或者判断是否得癌症等，那么就应该把<strong>阈值</strong>设置得高点，预测结果更准确。毕竟是人命关天的事嘛:)</p><p>假如我们的样本分布不能线性划分，如下图所示<br><img src="/posts/Machine%20Learning%20(Week3)/pic5.png"></p><p>我们可以用一个非线性的边界（高次多项式）来划分。</p><h2 id="Cost-function-代价函数"><a href="#Cost-function-代价函数" class="headerlink" title="Cost function - 代价函数"></a>Cost function - 代价函数</h2><p>对于前面的回归问题，我们的代价函数是<strong>所有误差的平方和取均值</strong>（实际上就是<strong>方差</strong>）</p><center> <br><span>$J(\theta)=\frac{1}{2m}\sum_{i=1}^{m}\left(h_{\theta}(x^{(i)})-y^{( i)}\right)^{2}$</span><!-- Has MathJax --><br></center><p><br><br>如果我们沿用这个计算方法，将逻辑斯谛回归的<span>$h_{\theta}(x) = g(\theta^{T}x) = \frac{1}{1+e^{-\theta^{T}x}}$</span> <!-- Has MathJax -->代入进上式的话，我们的函数图像会呈现出下面一种状况<br><img src="/posts/Machine%20Learning%20(Week3)/pic6.png"></p><p> 这是一个<strong>非凸函数</strong>，它有许多的局部最小值，不利于用梯度下降法寻找全局最小值。<br>所以我们要对逻辑回归重新定义一个代价函数。</p><p>根据我们<span>$h_{\theta}(x)$</span><!-- Has MathJax -->的性质</p><ul> <li>当<span>$\theta^{T}x > 0$</span> <!-- Has MathJax -->时，<span>$h_{\theta}(x) > 0.5$</span> <!-- Has MathJax --> </li> <li>当<span>$\theta^{T}x < 0$</span> <!-- Has MathJax -->时，<span>$h_{\theta}(x) < 0.5$</span> <!-- Has MathJax --> </li> <li><span>$h_{\theta}(x) \in (0, 1)$</span><!-- Has MathJax --> </li></ul><p>我们对代价函数作出如下定义</p><center><br><span>$$Cost\left( h_{\theta}\left( x \right), y \right) =\begin{cases} -log\left( h_{\theta}\left( x \right ) \right )& \text{ if } y = 1 \\ -log\left( 1 - h_{\theta}\left( x \right ) \right )& \text{ if } y = 0 \end{cases}$$</span><!-- Has MathJax --><br></center><p>它的函数图像和意义如下所示</p><ul> <li>当<span>$y = 1$</span><!-- Has MathJax -->时<img src="/posts/Machine%20Learning%20(Week3)/pic8.jpg"> </li></ul><p>它所反映的就是当样本结果<span>$y = 1$</span> <!-- Has MathJax -->时，如果我们<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->输出也<span>$\rightarrow 1$</span> <!-- Has MathJax -->的话，我们的误差就<span>$\rightarrow 0$</span> <!-- Has MathJax -->；反之，如果我们<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->输出<span>$\rightarrow 0$</span> <!-- Has MathJax -->的话，我们的误差就<span>$\rightarrow \infty$</span> <!-- Has MathJax --><br><br></p><ul> <li>当<span>$y = 0$</span><!-- Has MathJax -->时<img src="/posts/Machine%20Learning%20(Week3)/pic7.jpg"> </li></ul><p>它所反映的就是当样本结果<span>$y = 0$</span> <!-- Has MathJax -->时，如果我们<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->输出也<span>$\rightarrow 0$</span> <!-- Has MathJax -->的话，我们的误差就<span>$\rightarrow 0$</span> <!-- Has MathJax -->；反之，如果我们<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->输出<span>$\rightarrow 1$</span> <!-- Has MathJax -->的话，我们的误差就<span>$\rightarrow \infty$</span> <!-- Has MathJax --></p><p>没毛病:)</p><h2 id="Simplified-cost-function-and-gradient-descent-化简代价函数与梯度下降"><a href="#Simplified-cost-function-and-gradient-descent-化简代价函数与梯度下降" class="headerlink" title="Simplified cost function and gradient descent - 化简代价函数与梯度下降"></a>Simplified cost function and gradient descent - 化简代价函数与梯度下降</h2><p>对于<span>$Cost\left( h_{\theta}\left( x \right), y \right)$</span> <!-- Has MathJax -->，我们可以化简成</p><center><br><span>$Cost\left( h_{\theta}\left( x \right), y \right) = -ylog\left( h_{\theta}\left( x \right) \right) - \left( 1-y \right)log\left( 1-h_{\theta}\left( x \right) \right)$</span><!-- Has MathJax --><br></center><p><br><br>对于<span>$J(\theta)$</span><!-- Has MathJax -->我们作如下定义</p><center><br><span>$J(\theta) = \frac{1}{m} \sum_{i=1}^{m}Cost\left( h_{\theta}\left( x^{\left( i \right)} \right), y^{\left( i \right) }\right)$</span><!-- Has MathJax --><br></center><p><br><br>将<span>$Cost\left( h_{\theta}\left( x \right), y \right)$</span> <!-- Has MathJax -->代入上式，则可得</p><center><br><span>$J(\theta) = - \frac{1}{m} \sum_{i=1}^{m} \left[ y^{\left( i \right)}log\left( h_{\theta}\left( x^{\left( i \right)} \right) \right) + \left( 1-y^{\left( i \right)} \right)log\left( 1-h_{\theta}\left( x^{\left( i \right)} \right) \right) \right]$</span><!-- Has MathJax --><br></center><p><br><br>这时我们就可以用<strong>梯度下降</strong>来求<span>$min_{\theta}J(\theta)$</span> <!-- Has MathJax -->。</p><p>与回归一样，我们要做的就是不断更新<span>$\theta$</span><!-- Has MathJax --></p><center><br><span>$\theta := \theta - \alpha \frac{\partial J}{\partial \theta}$</span><!-- Has MathJax --><br></center><p><br><br>接下来我们来推导一下<span>$\frac{\partial J}{\partial \theta}$</span> <!-- Has MathJax -->。</p><p>首先，我们有</p><ul> <li><span>$X_{m \times (n+1)}\begin{bmatrix}x_{0}^{(1)} & x_{1}^{(1)} & \cdots & x_{n}^{(1)}\\ \vdots & \vdots & \ddots & \vdots\\ x_{0}^{(m)} & x_{1}^{(m)} & \cdots & x_{n}^{(m)}\end{bmatrix} = \begin{bmatrix}1 & x_{1}^{(1)} & \cdots & x_{n}^{(1)}\\ \vdots & \vdots & \ddots & \vdots\\ 1 & x_{1}^{(m)} & \cdots & x_{n}^{(m)}\end{bmatrix}$</span> <!-- Has MathJax --> </li> <li><span>$Y_{m\times 1} = \begin{bmatrix} y^{(1)} & \cdots & y^{(m)}\end{bmatrix}^{T}$</span><!-- Has MathJax --> </li> <li><span>$\theta = \begin{bmatrix}\theta_{0} & \theta_{1} & \cdots & \theta_{n} \end{bmatrix}^{T}$</span><!-- Has MathJax --> </li> <li><span>$\frac{\partial J}{\partial \theta} = - \frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)}\frac{\partial log(h_{\theta}(x^{(i)}))}{\partial \theta} + ( 1-y^{(i)})\frac{\partial log(1-h_{\theta}( x^{(i)} ))}{\partial \theta} \right]$</span><!-- Has MathJax --> </li> <li><span>$h_{\theta}(x^{(i)}) = g(\theta^{T}x^{(i)}) = \frac{1}{1+e^{-\theta^{T}x^{(i)}}}$</span><!-- Has MathJax --> </li></ul><p><br><br>因为</p><center><br><span>$y^{(i)} \frac{\partial log(h_{\theta}(x^{(i)}))}{\partial \theta} = \frac{y^{(i)}}{h_{\theta}(x^{(i)})} \cdot \frac{x^{(i)}e^{-\theta^{T}x^{(i)}}}{(1+e^{-\theta^{T}x^{(i)}})^{2}}$</span> <!-- Has MathJax --><br><br><br><span>$= y^{(i)}(1+e^{-\theta^{T}x^{(i)}}) \cdot \frac{x^{(i)}e^{-\theta^{T}x^{(i)}}}{(1+e^{-\theta^{T}x^{(i)}})^{2}}$</span> <!-- Has MathJax --><br><br><br><span>$= y^{(i)} \frac{x^{(i)}e^{-\theta^{T}x^{(i)}}}{1+e^{-\theta^{T}x^{(i)}}}$</span> <!-- Has MathJax --><br><br><br><span>$= y^{(i)} h_{\theta}(x^{(i)})x^{(i)}e^{-\theta^{T}x^{(i)}}$</span> <!-- Has MathJax --><br></center><p><br><br>又因为</p><center><br><span>$(1-y^{(i)}) \frac{\partial log(1-h_{\theta}( x^{(i)} ))}{\partial \theta} = - (1-y^{(i)}) \frac{1}{1-h_{\theta}(x^{(i)})} \cdot \frac{x^{(i)}e^{-\theta^{T}x^{(i)}}}{(1+e^{-\theta^{T}x^{(i)}})^{2}}$</span> <!-- Has MathJax --><br><br><br><span>$= - (1-y^{(i)}) \frac{1+e^{-\theta^{T}x^{(i)}}}{e^{-\theta^{T}x^{(i)}}} \cdot \frac{x^{(i)}e^{-\theta^{T}x^{(i)}}}{(1+e^{-\theta^{T}x^{(i)}})^{2}}$</span> <!-- Has MathJax --><br><br><br><span>$= - (1-y^{(i)}) \frac{x^{(i)}}{1+e^{-\theta^{T}x^{(i)}}}$</span> <!-- Has MathJax --><br><br><br><span>$= - (1-y^{(i)})x^{(i)}h_{\theta}(x^{(i)})$</span><!-- Has MathJax --><br></center><p><br><br>将上面两式相加，提出<span>$h_{\theta}(x^{(i)})x^{(i)}$</span> <!-- Has MathJax -->，得</p><center><br><span>$h_{\theta}(x^{(i)})x^{(i)} \left[y^{(i)}(1+e^{-\theta^{T}x^{(i)}}) - 1\right]$</span> <!-- Has MathJax --><br><br><br><span>$= x^{(i)}y^{(i)} - x^{(i)}h_{\theta}(x^{(i)})$</span><!-- Has MathJax --><br><br><br><span>$= x^{(i)} (y^{(i)} - h_{\theta}(x^{(i)}))$</span><!-- Has MathJax --><br></center><p><br><br>将上式代入<span>$\frac{\partial J}{\partial \theta}$</span> <!-- Has MathJax -->，则</p><center><br><span>$\frac{\partial J}{\partial \theta} = \frac{1}{m} \sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})x^{(i)}$</span> <!-- Has MathJax --><br></center><p><br><br>我们惊奇地发现，<span>$\frac{\partial J}{\partial \theta}$</span> <!-- Has MathJax -->的表达式居然跟回归是一样的！这就是数学的魅力！<br>因此，参考<a href="https://tankeryang.github.io/posts/Machine%20Learning%20(Week1)/#Gradient-Descent-for-Liner-Regression-线性回归中的梯度下降">week1</a>的推导，可得</p><center><br><span>$\frac{\partial J}{\partial \theta} = \frac{1}{m} X^{T}(X\theta - Y)$</span><!-- Has MathJax --><br><br><span>$$\theta := \theta - \alpha \frac{1}{m} X^{T}(X\theta - Y)$$</span> <!-- Has MathJax --><br></center><h2 id="Multi-class-classification-One-vs-all-多分类问题：一对多"><a href="#Multi-class-classification-One-vs-all-多分类问题：一对多" class="headerlink" title="Multi-class classification: One-vs-all - 多分类问题：一对多"></a>Multi-class classification: One-vs-all - 多分类问题：一对多</h2><p> 在实际分类问题中，我们遇到的大多是需要<strong>分多个类</strong>的问题，比如联系人的分类有家人，朋友，同事，同学等等。在可视化图像中，它们可能会呈现出如下的分布<br><img src="/posts/Machine%20Learning%20(Week3)/pic9.png"></p><p>一对多的做法就是我们分别对这三个类训练<span>$h_{\theta}^{(i)}(x)$</span> <!-- Has MathJax -->，其中<span>$i$</span><!-- Has MathJax -->为类别序号。如下图所示<br><img src="/posts/Machine%20Learning%20(Week3)/pic10.png"></p><p>训练完所有的<span>$h_{\theta}^{(i)}(x)$</span> <!-- Has MathJax -->后，当我们给一组输入特征<span>$x$</span> <!-- Has MathJax -->时，取<strong>最大的</strong><span>$h_{\theta}^{(i)}(x)$</span> <!-- Has MathJax -->作为我们的预测结果。</p><h1 id="Regularization-正则化"><a href="#Regularization-正则化" class="headerlink" title="Regularization - 正则化"></a>Regularization - 正则化</h1><h2 id="The-problem-of-overfitting-过拟合问题"><a href="#The-problem-of-overfitting-过拟合问题" class="headerlink" title="The problem of overfitting - 过拟合问题"></a>The problem of overfitting - 过拟合问题</h2><p>假如我们样本有非常多的特征，我们也许能训练出一个在样本集上表现得很好的假设函数<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->，但是对于新的输入，我们可能不能很好地进行拟合（预测）。这类问题，我们称之为<strong>过拟合</strong>。<br><img src="/posts/Machine%20Learning%20(Week3)/pic11.png"><br><img src="/posts/Machine%20Learning%20(Week3)/pic12.png"></p><p>对于过拟合问题我们一般有下面一些解决方法</p><ul> <li>减少特征数量</li></ul><blockquote> <p>手动剔除一些不必要的特征，或者用一些降维算法（PCA）来自动减少特征数</p></blockquote><ul> <li>正则化</li></ul><blockquote> <p>保留所有的特征，同时减小参数<span>$\theta$</span><!-- Has MathJax -->的大小</p></blockquote><h2 id="Cost-function-代价函数-1"><a href="#Cost-function-代价函数-1" class="headerlink" title="Cost function - 代价函数"></a>Cost function - 代价函数</h2><p>首先看下面这两种预测函数在样本集上的结果<br><img src="/posts/Machine%20Learning%20(Week3)/pic13.png"></p><p>我们能看到，左边是比较合适的预测函数，而右边则明显过拟合了。</p><p>这时我们用一个小小的技巧，在我们的误差函数<span>$J(\theta)$</span> <!-- Has MathJax -->后面对<span>$\theta_{3}$</span> <!-- Has MathJax -->和<span>$\theta_{4}$</span> <!-- Has MathJax -->加一个<strong>惩罚系数（或者说补偿反馈）</strong>，使之变为</p><center> <br><span>$J(\theta)=\frac{1}{2m}\sum_{i=1}^{m}\left(h_{\theta}(x^{(i)})-y^{(i)}\right)^{2} + 1000\theta_{3} + 1000\theta_{4}$</span><!-- Has MathJax --><br></center><p><br><br>由上可知，我们要求最优的<span>$\theta$</span> <!-- Has MathJax -->，使得<span>$J(\theta)$</span> <!-- Has MathJax -->取得最小值，那么我们在优化过程中（比如梯度下降）<span>$\theta_{3}$</span> <!-- Has MathJax -->和<span>$\theta_{4}$</span> <!-- Has MathJax -->一定<span>$\rightarrow 0$</span> <!-- Has MathJax -->，因为他们占比很大。这样<span>$\theta_{3}$</span> <!-- Has MathJax -->和<span>$\theta_{4}$</span> <!-- Has MathJax -->对<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->的贡献就非常小，<span>$x^{3}$</span> <!-- Has MathJax -->和<span>$x^{4}$</span> <!-- Has MathJax -->这些高次项在<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->所占的权重就小很多，有效地防止了<strong>过拟合</strong>。</p><p>假如我们有非常多的特征，不知道要对哪些对应的参数<span>$\theta$</span> <!-- Has MathJax -->作惩罚，那么最好的办法就是对所有的<span>$\theta$</span> <!-- Has MathJax -->作惩罚，然后让程序自己迭代优化。所以我们的代价函数<span>$J(\theta)$</span> <!-- Has MathJax -->就变成下面这种形式</p><center><br><span>$J(\theta) = \frac{1}{2m} \left[ \sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})^{2} + \lambda \sum_{j=1}^{n} \theta_{j}^{2} \right]$</span><!-- Has MathJax --><br></center><p><br><br>其中<span>$\lambda$</span> <!-- Has MathJax -->称为<strong>正则化参数</strong>。一般我们不对<span>$\theta_{0}$</span> <!-- Has MathJax -->进行惩罚。</p><p>正则化后的假设函数如下图所示<br><img src="/posts/Machine%20Learning%20(Week3)/pic14.png"></p><p>其中<font color="#538fca">蓝色</font>曲线是过拟合的情况，<font color="#ee34e1">紫色</font> 曲线是正则化后的假设函数曲线，而<font color="#ef9a3d">橙色</font>直线则是 <strong>正则化参数过大</strong> 导致的 <strong>欠拟合</strong>。为什么会这样呢？因为正则化参数过大，会对<span>$(\theta_{1} \cdots \theta_{n})$</span><!-- Has MathJax -->惩罚过重，以至于<span>$(\theta_{1} \cdots \theta_{n}) \rightarrow 0$</span><!-- Has MathJax -->，使得<span>$h_{\theta}(x) \approx \theta_{0}$</span><!-- Has MathJax -->。</p><p>因此对于正则化，我们要选一个合适的值，才有好的效果。</p><h2 id="Regularized-linear-regression-正则化后的线性回归"><a href="#Regularized-linear-regression-正则化后的线性回归" class="headerlink" title="Regularized linear regression - 正则化后的线性回归"></a>Regularized linear regression - 正则化后的线性回归</h2><p>正则化后我们的代价函数变成</p><center><br><span>$J(\theta) = \frac{1}{2m} \left\{ \left[\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})^{2} \right] + \lambda \sum_{j=1}^{n} \theta_{j}^{2} \right\}$</span><!-- Has MathJax --><br></center><p><br><br>如果我们用梯度下降来求最优<span>$\theta$</span> <!-- Has MathJax -->，我们更新<span>$\theta$</span> <!-- Has MathJax -->就要分别更新<span>$\theta_{0}$</span> <!-- Has MathJax -->和<span>$\theta_{1} \cdots \theta_{n}$</span> <!-- Has MathJax --></p><center><br><span>$$\begin{cases} \theta_{0} := \theta_{0} - \alpha \frac{1}{m} \sum_{i=1}^{m}( h_{\theta}(x^{(i)})-y^{(i)} )x_{0}^{(i)} \\ \theta_{j} := \theta_{j} - \alpha \left\{ \frac{1}{m} \left[\sum_{i=1}^{m}( h_{\theta}(x^{(i)})-y^{(i)} )x_{j}^{(i)} \right]+\frac{\lambda}{m}\theta_{j}\right\} & j=1,2, \cdots ,n \end{cases}$$</span><!-- Has MathJax --><br></center><p><br><br>其中<span>$\theta_{j}$</span><!-- Has MathJax -->可以化简成</p><center><br><span>$\theta_{j}(1-\alpha\frac{\lambda}{m}) - \alpha\frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})x_{j}^{(i)}$</span> <!-- Has MathJax --><br></center><p><br><br>我们看到，<span>$(1-\alpha\frac{\lambda}{m}) < 1$</span> <!-- Has MathJax -->，所以正则化后的梯度下降实际上就是让<span>$\theta_{j}$</span> <!-- Has MathJax -->减少一定的比例后再进行原来的梯度下降。</p><p>我们知道，梯度<span>$\frac{\partial J}{\partial \theta}$</span> <!-- Has MathJax -->为<span>$0$</span> <!-- Has MathJax -->时，<span>$J(\theta)$</span><!-- Has MathJax -->取得极小值，所以我们令</p><center><br><span>$$\begin{cases} \sum_{i=1}^{m}( h_{\theta}(x^{(i)})-y^{(i)} )x_{0}^{(i)} = 0\\ \sum_{i=1}^{m}( h_{\theta}(x^{(i)})-y^{(i)} )x_{j}^{(i)} + \lambda\theta_{j} = 0 & j=1,2, \cdots ,n \end{cases}$$</span><!-- Has MathJax --><br></center><p>即</p><center><br><span>$\frac{\partial J}{\partial \theta} =\begin{bmatrix}x_{0}^{(1)} & x_{0}^{(2)} &\cdots & x_{0}^{(m)}\\ x_{1}^{(1)} & x_{1}^{(2)} & \cdots & x_{1}^{(m)}\\ \vdots & \vdots & \ddots & \vdots \\ x_{n}^{(1)} & x_{n}^{(2)} & \cdots & x_{n}^{(m)}\end{bmatrix} \begin{bmatrix}h_{\theta}(x^{(1)})-y^{(1)}\\h_{\theta}(x^{(2)})-y^{(2)}\\ \vdots\\h_{\theta}(x^{(m)})-y^{(m)}\end{bmatrix} + \lambda \begin{bmatrix}0\\ \theta_{1}\\ \vdots \\ \theta_{n}\end{bmatrix} = 0$</span><!-- Has MathJax --><br></center><p><br><br>也即</p><center><br><span>$X^{T}(X\theta - Y) + \lambda \begin{bmatrix}0 & & & \\ & 1 & & \\ & & \ddots & \\ & & & 1\end{bmatrix}_{(n+1)^{2}} \theta = 0$</span><!-- Has MathJax --><br></center><p><br><br>去掉括号，并提出<span>$\theta$</span><!-- Has MathJax -->，整理等式</p><center><br><span>$(X^{T}X + \lambda\begin{bmatrix}0 & & & \\ & 1 & & \\ & & \ddots & \\ & & & 1\end{bmatrix}_{(n+1)^{2}}) \theta = X^{T}Y$</span><!-- Has MathJax --><br></center><p><br><br>最后我们可得</p><center><br><span>$\theta = (X^{T}X + \lambda\begin{bmatrix}0 & & & \\ & 1 & & \\ & & \ddots & \\ & & & 1\end{bmatrix}_{(n+1)^{2}})^{-1} X^{T}Y$</span><!-- Has MathJax --><br></center><p><br><br>上式就是正则化后的 <strong>Normal equation - 正规方程</strong>，其中<span>$(X^{T}X + \lambda\begin{bmatrix}0 & & & \\ & 1 & & \\ & & \ddots & \\ & & & 1\end{bmatrix}_{(n+1)^{2}})$</span> <!-- Has MathJax -->一定是可逆的。这个就不在此作证明了。</p><h2 id="Regularized-logistic-regression-正则化后的逻辑斯谛回归"><a href="#Regularized-logistic-regression-正则化后的逻辑斯谛回归" class="headerlink" title="Regularized logistic regression - 正则化后的逻辑斯谛回归"></a>Regularized logistic regression - 正则化后的逻辑斯谛回归</h2><p>与线性回归一样，我们在原来的代价函数<span>$J(\theta)$</span> <!-- Has MathJax -->后面加上一个惩罚项，则<span>$J(\theta)$</span><!-- Has MathJax -->变成</p><center><br><span>$J(\theta) = - \left\{ \frac{1}{m} \sum_{i=1}^{m} \left[ y^{\left( i \right)}log\left( h_{\theta}\left( x^{\left( i \right)} \right) \right) + \left( 1-y^{\left( i \right)} \right)log\left( 1-h_{\theta}\left( x^{\left( i \right)} \right) \right) \right] \right\} + \frac{\lambda}{2m} \sum_{j=1}^{2} \theta_{j}^{2}$</span><!-- Has MathJax --><br></center><p><br><br>因为其<span>$\frac{\partial J(\theta)}{\partial \theta}$</span> <!-- Has MathJax -->的形式与上面线性回归一样，所以梯度下降的过程同上。</p><p><br><br> <div class="note info"> <p> <center><strong>课程资料</strong></center> </p> <ul> <li><a href="https://github.com/tankeryang/Coursera-machine-learning-lecture-note/tree/master/week3" target="_blank" rel="noopener">week3课程讲义</a> </li> <li><a href="https://github.com/tankeryang/Coursera-machine-learning-assignment/tree/master/machine-learning-ex2" target="_blank" rel="noopener">编程作业ex2</a> </li> </ul> </div></p>]]></content>
<summary type="html">
<p>在第三周的课程里，介绍了<strong>Logistic Regression -
逻辑斯谛回归</strong>问题，主要应用在<strong>Classification -
分类</strong>上。还有<strong>Regularization -
正则化</strong>，如何用来解决<strong>Overfitting - 过拟合</strong>问题。</p>
<img src="/posts/Machine%20Learning%20(Week3)/pic0.png">
</summary>
<category term="机器学习笔记" scheme="https://tankeryang.github.io/categories/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/"/>
<category term="机器学习" scheme="https://tankeryang.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="ML" scheme="https://tankeryang.github.io/tags/ML/"/>
<category term="数学" scheme="https://tankeryang.github.io/tags/%E6%95%B0%E5%AD%A6/"/>
</entry>
<entry>
<title>Machine Learning (Week2)</title>
<link href="https://tankeryang.github.io/posts/Machine%20Learning%20(Week2)/"/>
<id>https://tankeryang.github.io/posts/Machine Learning (Week2)/</id>
<published>2017-10-03T14:43:21.000Z</published>
<updated>2019-10-09T17:06:09.571Z</updated>
<content type="html"><![CDATA[<p> 在第二周的课程里，主要讲了<strong>多变量线性回归</strong>以及相应的<strong>梯度下降实践</strong>，一些梯度下降的技巧如<strong>学习速率的选择</strong>，<strong>Feature Scaling - 特征缩放</strong>等，最后介绍了<strong>Polynomial Regression - 多项式回归</strong>和<strong>Normal Equation - 正规方程</strong>。</p><img src="/posts/Machine%20Learning%20(Week2)/pic0.png"><a id="more"></a><h1 id="Linear-Regression-with-multiple-variables-多变量线性回归"><a href="#Linear-Regression-with-multiple-variables-多变量线性回归" class="headerlink" title="Linear Regression with multiple variables - 多变量线性回归"></a>Linear Regression with multiple variables - 多变量线性回归</h1><h2 id="Multiple-Feature-多变量"><a href="#Multiple-Feature-多变量" class="headerlink" title="Multiple Feature - 多变量"></a>Multiple Feature - 多变量</h2><ul> <li>参考<a href="https://tankeryang.github.io/posts/Coursera机器学习笔记-1-Week1"><strong>week1</strong></a>第二节内容。 </li></ul><h2 id="Gradient-Descent-for-multiple-variable-多变量梯度下降"><a href="#Gradient-Descent-for-multiple-variable-多变量梯度下降" class="headerlink" title="Gradient Descent for multiple variable - 多变量梯度下降"></a>Gradient Descent for multiple variable - 多变量梯度下降</h2><ul> <li>参考<a href="https://tankeryang.github.io/posts/Coursera机器学习笔记-1-Week1"><strong>week1</strong></a>第三节内容。 </li></ul><h2 id="Gradient-Descent-in-practise-1-Feature-Scaling-梯度下降实践1：特征缩放"><a href="#Gradient-Descent-in-practise-1-Feature-Scaling-梯度下降实践1：特征缩放" class="headerlink" title="Gradient Descent in practise 1: Feature Scaling - 梯度下降实践1：特征缩放"></a>Gradient Descent in practise 1: Feature Scaling - 梯度下降实践1：特征缩放</h2><p>首先还是用房价预测回归的例子来说明：<br>假设我们有两个特征：</p><ul> <li><span>$x_{1} = size(feet_{2}) \in (0,2000)$</span><!-- Has MathJax --> </li> <li><span>$x_{2} = number of bedrooms \in (1,5)$</span><!-- Has MathJax --> </li></ul><p>可以看到<span>$x_{1}$</span><!-- Has MathJax -->比<span>$x_{2}$</span> <!-- Has MathJax -->的取值范围要大了几个数量级。这么做的直接后果就是，只要<span>$x_{1}$</span> <!-- Has MathJax -->的参数<span>$\theta_{1}$</span> <!-- Has MathJax -->稍微变化一下，预测值<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->与实际价格之间的误差就会偏移得很厉害，也就导致<span>$J(\theta)$</span> <!-- Has MathJax -->在<span>$\theta_{1}$</span> <!-- Has MathJax -->偏移得很厉害，如下图所示：<br><img src="/posts/Machine%20Learning%20(Week2)/pic1.png"><br>我们看到，假如对此进行梯度下降，须迭代多次才能达到极值点。显然这是不ok的。</p><p>所以我们要对<span>$x_{1}$</span><!-- Has MathJax -->和<span>$x_{2}$</span> <!-- Has MathJax -->进行缩放，让他们的取值范围落在同一个区间，或者相近区间。</p><p>一般我们会用下面的方法进行缩放：</p><ul> <li><span>$x_{1} = \frac{size(feet_{2})}{2000}$</span><!-- Has MathJax --> </li> <li><span>$x_{2} = \frac{number of bedrooms}{5}$</span><!-- Has MathJax --> </li></ul><p>这样我们就能令<span>$x_{1},x_{2} \in [0,1]$</span> <!-- Has MathJax -->，这种方法也叫<strong>归一化</strong>。<br>我们在做梯度下降时，速度就会快很多：<br><img src="/posts/Machine%20Learning%20(Week2)/pic2.png"></p><p>更普遍的，我们采用如下方法：</p><center><br><span>$x := \frac{x-\mu}{s}$</span><!-- Has MathJax --><br></center><p>其中$\mu$为$x$的均值，$s$为$x$的方差。这样就能令$x\in (-1,1)$。</p><h2 id="Gradient-Descent-in-practise-2-Learning-rate-梯度下降实践2：学习速率"><a href="#Gradient-Descent-in-practise-2-Learning-rate-梯度下降实践2：学习速率" class="headerlink" title="Gradient Descent in practise 2: Learning rate - 梯度下降实践2：学习速率"></a>Gradient Descent in practise 2: Learning rate - 梯度下降实践2：学习速率</h2><p>如何选择学习速率是做梯度下降的很关键的一步。在<a href="https://tankeryang.github.io/posts/Coursera机器学习笔记-1-Week1"><strong>week1</strong></a>第三节的内容里，我们了解到学习速率$\alpha$设置不当会有怎样的结果。所以当我们做梯度下降时一定要确保每一次迭代时$J(\theta)$都在<strong>减小</strong>，最后收敛于某个值。一般我们可以作<strong>迭代次数 - $J(\theta)$图</strong>来观察$J(\theta)$是否收敛：<br> <div class="note info"> <ul> <li> <p>收敛<br><img src="/posts/Machine%20Learning%20(Week2)/pic3.png"></p> </li> <li> <p>发散<br><img src="/posts/Machine%20Learning%20(Week2)/pic4.png"></p> </li> </ul> </div> <br>或者设置某个阈值（比如$0.001$）来检测，<strong>当</strong>$J(\theta)$<strong>减小的差值小于阈值时</strong>，可以认为$J(\theta)$已收敛到极值。</p><h2 id="Features-and-Polynomial-Regression-特征与多项式回归"><a href="#Features-and-Polynomial-Regression-特征与多项式回归" class="headerlink" title="Features and Polynomial Regression - 特征与多项式回归"></a>Features and Polynomial Regression - 特征与多项式回归</h2><p> 线性回归顾名思义，用于特征与结果有明显的线性关系时的情况。假如我们的某些特征与结果是非线性关系的，比如下图，我们只要观察散点的分布趋向哪种多项式函数然后做拟合就好了：<br><img src="/posts/Machine%20Learning%20(Week2)/pic5.png"></p><h2 id="Normal-Equation-正规方程"><a href="#Normal-Equation-正规方程" class="headerlink" title="Normal Equation - 正规方程"></a>Normal Equation - 正规方程</h2><p>根据<a href="https://tankeryang.github.io/posts/Coursera机器学习笔记-1-Week1"><strong>week1</strong></a>第三节最后的内容，我们得到：</p><center><br><span>$$\frac{\partial J}{\partial \theta} = grad_{(n+1)\times 1} = \frac{1}{m} X^{T}(X\theta - Y)$$</span><!-- Has MathJax --><br></center><br>令其为$0$，可得：<br><center><br><span>$$X^{T}X\theta - X^{T}Y = 0$$</span> <!-- Has MathJax --><br><span>$X^{T}X\theta = X^{T}Y$</span> <!-- Has MathJax --><br></center><br>则得：<br><center><br><span>$\theta = (X^{T}X)^{-1}X^{T}Y$</span><!-- Has MathJax --><br></center><p> <br><br>其中要注意的是，$X^{T}X$不一定是可逆的。比如<strong>特征数过多</strong>$(m<<n)$，此时$r(X)\leq m$，$r(X^{T}X)\leq min(r(X),r(X^{T}))\leq m < n$。而$X^{T}X\in n\times n$，因此$X^{T}X$是不可逆的。<br>因此在<strong>样本数量远大于特征数量</strong>时才能用正规方程。</p><h1 id="小结"><a href="#小结" class="headerlink" title="小结"></a>小结</h1><p>具体问题具体分析，梯度下降虽然基础，但在许多问题上仍然是很有效的。正规方程用在合适的问题上则能非常快的得出结果。各司其职。</p><p><br><br> <div class="note info"> <p> <center><strong>课程资料</strong></center> </p> <ul> <li><a href="https://github.com/tankeryang/Coursera-machine-learning-lecture-note/tree/master/week2" target="_blank" rel="noopener">week2课程讲义</a> </li> <li><a href="https://github.com/tankeryang/Coursera-machine-learning-assignment/tree/master/machine-learning-ex1" target="_blank" rel="noopener">编程作业ex1</a> </li> </ul> </div></p>]]></content>
<summary type="html">
<p>
在第二周的课程里，主要讲了<strong>多变量线性回归</strong>以及相应的<strong>梯度下降实践</strong>，一些梯度下降的技巧如<strong>学习速率的选择</strong>，<strong>Feature
Scaling - 特征缩放</strong>等，最后介绍了<strong>Polynomial Regression -
多项式回归</strong>和<strong>Normal Equation - 正规方程</strong>。</p>
<img src="/posts/Machine%20Learning%20(Week2)/pic0.png">
</summary>
<category term="机器学习笔记" scheme="https://tankeryang.github.io/categories/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/"/>
<category term="机器学习" scheme="https://tankeryang.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="ML" scheme="https://tankeryang.github.io/tags/ML/"/>
<category term="数学" scheme="https://tankeryang.github.io/tags/%E6%95%B0%E5%AD%A6/"/>
</entry>
<entry>
<title>deeplearning.ai 1 - Neural Networks and Deep Learning (week1)</title>
<link href="https://tankeryang.github.io/posts/deeplearning.ai%201%20-%20Neural%20Networks%20and%20Deep%20Learning%20(week1)/"/>
<id>https://tankeryang.github.io/posts/deeplearning.ai 1 - Neural Networks and Deep Learning (week1)/</id>
<published>2017-09-30T19:03:30.000Z</published>
<updated>2019-10-09T17:06:09.591Z</updated>
<content type="html"><![CDATA[<p>第一周的课程里主要是简单介绍<strong>deeplearning - 深度学习</strong>，<strong>Neural NetWork - 神经网络</strong>的概念。</p><a id="more"></a><h1 id="Introduction-to-Deep-Learning-深度学习简介"><a href="#Introduction-to-Deep-Learning-深度学习简介" class="headerlink" title="Introduction to Deep Learning - 深度学习简介"></a>Introduction to Deep Learning - 深度学习简介</h1><h2 id="What-is-Neural-Network-什么是神经网络？"><a href="#What-is-Neural-Network-什么是神经网络？" class="headerlink" title="What is Neural Network? - 什么是神经网络？"></a>What is Neural Network? - 什么是神经网络？</h2><p>其实<strong>神经网络</strong>并没有想象中那么复杂，它由三部分组成：</p><ul> <li>输入层</li> <li>隐藏层</li> <li>输出层</li></ul><p>其中每一层都包含数个<strong>neuron - 神经元</strong>，我称之为<strong>计算节点</strong>。每一层的每一个节点都完成一种计算任务，上一层的节点计算的结果会作为当前层的节点的输入，当前层节点的输出则作为下一层节点的输入。</p><p>我们用一个简单的例子来解释：<br>假如现在有一组数据，是关于房子面积和价格的表格，如下：</p><table> <thead> <tr> <th style="text-align:center">$Size in feet^{2}$ (x)</th> <th style="text-align:center">$prize$ (y)</th> </tr> </thead> <tbody> <tr> <td style="text-align:center">2104</td> <td style="text-align:center">460</td> </tr> <tr> <td style="text-align:center">1416</td> <td style="text-align:center">232</td> </tr> <tr> <td style="text-align:center">1534</td> <td style="text-align:center">315</td> </tr> <tr> <td style="text-align:center">852</td> <td style="text-align:center">178</td> </tr> </tbody></table><p>我们可以作线性回归，可能画出来的图是这样的：</p><img src="/posts/deeplearning.ai%201%20-%20Neural%20Networks%20and%20Deep%20Learning%20(week1)/pic0.png"><p>那么我们可以作如下定义：</p><ul> <li><strong>输入层节点</strong>为房子面积$(x)$</li> <li><strong>隐藏层节点</strong>为<font color="#50ccf1">蓝色直线</font>所代表的 <strong>拟合函数</strong> $(f)$</li> <li><strong>输出层节点</strong>为 <strong>拟合函数</strong>在对应输入上的输出$(\hat{y})$</li></ul><p>可作下图：</p><div class="mermaid"> graph LR; x((x)) --> f((f)); f((f)) --> y((yhat));</div><p>这就是最简单的神经网络，它只有$3$层，每层只有$1$个节点。只要我们将随便一个房屋子的面积作为输入丢进去，它就会帮我们预测输出价格。</p><p> 可是只靠面积来预测价格肯定是不严谨的，可能会有多个因素，比如地段，交通便利情况等。这时我们就会有多个输入，也就意味着输入层节点有多个，与输入的特征一一对应。<br><img src="/posts/deeplearning.ai%201%20-%20Neural%20Networks%20and%20Deep%20Learning%20(week1)/pic01.png"></p><p>我的总结，<strong>神经网络就是一个多层的嵌套函数</strong>。这句话会在后面的笔记得以充分体现。</p><hr><h2 id="Supervised-Learning-with-Neural-Networks-有监督学习神经网络"><a href="#Supervised-Learning-with-Neural-Networks-有监督学习神经网络" class="headerlink" title="Supervised Learning with Neural Networks - 有监督学习神经网络"></a>Supervised Learning with Neural Networks - 有监督学习神经网络</h2><p>有监督学习，简要的说，就是我们有一个 <strong>数据集</strong>，并且我们知道这对数据的 <strong>输出</strong>是张什么样的，而且可以肯定作为变量的输入数据与作为结果的输出数据之间有着必然的联系。</p><p>有监督学习通常分为<strong>regression - 回归</strong>与<strong>classification - 分类</strong>两类问题。</p><ul> <li> <p>对于回归问题，一般我们的数据都是 <strong>Structured - 结构化</strong> 的，有着明显的特征：</p> <div class="note info"> <p><img src="/posts/deeplearning.ai%201%20-%20Neural%20Networks%20and%20Deep%20Learning%20(week1)/pic02.png"> </p> </div> </li> <li> <p>对于分类问题，数据一般是 <strong>Unstructured - 无结构化</strong>的，特征不明显：</p> <div class="note info"> <p><img src="/posts/deeplearning.ai%201%20-%20Neural%20Networks%20and%20Deep%20Learning%20(week1)/pic03.png"> </p> </div> </li></ul><hr><p>关于神经网络的发展请阅读课程资料，这里不在赘述。<br> <div class="note info"> <p> <center><strong>课程资料</strong></center><br><a href="https://github.com/tankeryang/Coursera-deeplearning.ai-lecture-note/tree/master/week1/PDF" target="_blank" rel="noopener">week1课程讲义</a> </p> </div></p>]]></content>
<summary type="html">
<p>第一周的课程里主要是简单介绍<strong>deeplearning - 深度学习</strong>，<strong>Neural NetWork -
神经网络</strong>的概念。</p>
</summary>
<category term="deeplearning.ai笔记" scheme="https://tankeryang.github.io/categories/deeplearning-ai%E7%AC%94%E8%AE%B0/"/>
<category term="机器学习" scheme="https://tankeryang.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="ML" scheme="https://tankeryang.github.io/tags/ML/"/>
<category term="数学" scheme="https://tankeryang.github.io/tags/%E6%95%B0%E5%AD%A6/"/>
<category term="深度学习" scheme="https://tankeryang.github.io/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/"/>
<category term="DL" scheme="https://tankeryang.github.io/tags/DL/"/>
</entry>
<entry>
<title>deeplearning.ai 0 (openning)</title>
<link href="https://tankeryang.github.io/posts/deeplearning.ai%200%20(openning)/"/>
<id>https://tankeryang.github.io/posts/deeplearning.ai 0 (openning)/</id>
<published>2017-09-30T18:06:39.000Z</published>
<updated>2019-10-09T17:06:09.591Z</updated>
<content type="html"><![CDATA[<p> <center> <h1>开篇的话</h1> </center><br>大约上个月，在<strong>Coursera</strong>上申请<strong>Andrew Ng</strong>的<strong>deeplearning.ai</strong>系列课程助学金通过了。想着偏偏在我复习考研的时间段里开课，又不想转换班次，于是“狠下心来”开始上课，这下负担又重了，不仅要复习，还要兼顾Coursera上两门重量级课程，虽然有做笔记和推导，但是实在没有时间再去做整理发布了。<br>那么为什么会有这个开篇呢？因为我实在忍不住。今天刚上完<strong>系列课程1–Neural Networks and Deep Learning</strong>的<strong>week3</strong>课程，而<strong>Machine Learning</strong>那边也进行到<strong>week6</strong>。因为在<strong>Machine Learning</strong>里也教到<strong>Back Propagation - 反向传播算法</strong>后面一点的内容，同样的<strong>deeplearning</strong>那边也讲到，感觉对<strong>BP</strong>有了非常清晰的思路。迫不及待想整理上来。趁着国庆做一次详细整理吧。</p><p>以下是一些关于此课程笔记的事项：</p><ul> <li>这里记录了我在Coursera上学习deeplearning.ai专项课程的笔记</li> <li>同时会把课程相关资料整理一并发布，包括视频，lecturenote，homework，和我的答案解释。</li> <li><a href="https://www.coursera.org/specializations/deep-learning" target="_blank" rel="noopener">课程链接</a>:可以申请助学金以免费听课，需15个工作日处理。 </li></ul>]]></content>
<summary type="html">
<p>
<center>
<h1>开篇的话</h1>
</center><br>大约上个月，在<strong>Coursera</strong>上申请<strong>Andrew
Ng</strong>的<strong>deeplearning.ai</s
</summary>
<category term="deeplearning.ai笔记" scheme="https://tankeryang.github.io/categories/deeplearning-ai%E7%AC%94%E8%AE%B0/"/>
<category term="机器学习" scheme="https://tankeryang.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="ML" scheme="https://tankeryang.github.io/tags/ML/"/>
<category term="数学" scheme="https://tankeryang.github.io/tags/%E6%95%B0%E5%AD%A6/"/>
<category term="深度学习" scheme="https://tankeryang.github.io/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/"/>
<category term="DL" scheme="https://tankeryang.github.io/tags/DL/"/>
</entry>
<entry>
<title>Machine Learning (Week1)</title>
<link href="https://tankeryang.github.io/posts/Machine%20Learning%20(Week1)/"/>
<id>https://tankeryang.github.io/posts/Machine Learning (Week1)/</id>
<published>2017-09-06T15:38:01.000Z</published>
<updated>2019-10-09T17:06:09.555Z</updated>
<content type="html"><![CDATA[<p>在第一周的课程里简要介绍了<strong>什么是机器学习</strong>，<strong>Model - 模型</strong>和<strong>Cost Function - 代价函数</strong>的概念，以及一些必要的<strong>线性代数</strong>的知识。</p><img src="/posts/Machine%20Learning%20(Week1)/pic0.png"><div class="note warning"> <p><strong>特别声明</strong>：这里不会对线性代数基础进行记录，有需要了解的请自行学习。</p> <center><strong>敬请留意</strong></center></div><a id="more"></a><h1 id="Introduction-简介"><a href="#Introduction-简介" class="headerlink" title="Introduction - 简介"></a>Introduction - 简介</h1><h2 id="What-is-Machine-Learning-何为机器学习？"><a href="#What-is-Machine-Learning-何为机器学习？" class="headerlink" title="What is Machine Learning? - 何为机器学习？"></a>What is Machine Learning? - 何为机器学习？</h2><p>引用Tom Mitchell的经典解释（有点像绕口令）</p><blockquote> <p>Two definitions of Machine Learning are offered. Arthur Samuel described it as: “the field of study that gives computers the ability to learn without being explicitly programmed.” This is an older, informal definition.</p></blockquote><p>Tom Mitchell provides a more modern definition: “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.”</p><blockquote></blockquote><p>Example: playing checkers.</p><blockquote></blockquote><p>E = the experience of playing many games of checkers</p><blockquote></blockquote><p>T = the task of playing checkers.</p><blockquote></blockquote><p>P = the probability that the program will win the next game.</p><blockquote></blockquote><p>In general, any machine learning problem can be assigned to one of two broad classifications:</p><blockquote></blockquote><p>Supervised learning and Unsupervised learning.</p><h2 id="Supervised-Learning-有监督学习"><a href="#Supervised-Learning-有监督学习" class="headerlink" title="Supervised Learning - 有监督学习"></a>Supervised Learning - 有监督学习</h2><p> 有监督学习，简要的说，就是我们有一个<strong>数据集</strong>，并且我们知道这对数据的<strong>输出</strong>是张什么样的，而且可以肯定作为变量的输入数据与作为结果的输出数据之间有着必然的联系。</p><p>有监督学习通常分为<strong>regression - 回归</strong>与<strong>classification - 分类</strong>两类问题。</p><ul> <li> 在回归问题中，我们要做的就是得到一个连续的<strong>预测函数</strong>去拟合离散的数据，也就是要把<strong>输入变量映射到连续函数上</strong>，从而实现未知数据的预测。 </li></ul><p><strong>例子：</strong><br>Given data about the size of houses on the real estate market, try to predict their price. Price as a function of size is a continuous output, so this is a regression problem.<br>We could turn this example into a classification problem by instead making our output about whether the house “sells for more or less than the asking price.” Here we are classifying the houses based on price into two discrete categories.<br><img src="/posts/Machine%20Learning%20(Week1)/pic1.jpg"></p><ul> <li> 在分类问题中，输出结果是离散的，比如二分类问题，每一类别分别对应0,1两个离散数值，我们要做的就是得到一个<strong>预测函数</strong>，能够根据输入变量得到离散的输出结果，也就是要把<strong>输入变量映射到离散的类别中</strong> </li></ul><p><strong>例子：</strong><br>(a) Classification - Given a patient with a tumor, we have to predict whether the tumor is malignant or benign.<br><img src="/posts/Machine%20Learning%20(Week1)/pic2.jpg"></p><p>(b) Regression - Given a picture of a person, we have to predict their age on the basis of the given picture<br><img src="/posts/Machine%20Learning%20(Week1)/pic3.jpg"></p><h2 id="Unsupervised-Learning-无监督学习"><a href="#Unsupervised-Learning-无监督学习" class="headerlink" title="Unsupervised Learning - 无监督学习"></a>Unsupervised Learning - 无监督学习</h2><p> 无监督学习就是有监督学习的反面情况，即我们有一组数据集，但我们并不知道它的输出结果是什么，或者它根本就没有输出，甚至它本身代表什么我们都不知道。我们要做的就是从这堆数据中<strong>找出它的规律或者结构</strong>，来确定这些输入变量产生的影响，比如将这堆数据分组。</p><p>特别的是，无监督学习并不像有监督学习那样有基于预测结果的反馈。</p><p>现实生活中有很多这样的例子，比如你有一堆新闻内容的数据，你要把有关联的分成一组。像这样的算法叫做<strong>聚类</strong>，就如字面意思一样。</p><p><strong>例子：</strong><br><img src="/posts/Machine%20Learning%20(Week1)/pic4.jpg"><br><img src="/posts/Machine%20Learning%20(Week1)/pic5.jpg"><br><img src="/posts/Machine%20Learning%20(Week1)/pic6.jpg"><br><img src="/posts/Machine%20Learning%20(Week1)/pic7.jpg"></p><hr><h1 id="Model-and-Cost-Function-模型与代价函数"><a href="#Model-and-Cost-Function-模型与代价函数" class="headerlink" title="Model and Cost Function - 模型与代价函数"></a>Model and Cost Function - 模型与代价函数</h1><h2 id="Model-Representation-模型的表示方法"><a href="#Model-Representation-模型的表示方法" class="headerlink" title="Model Representation - 模型的表示方法"></a>Model Representation - 模型的表示方法</h2><p>对于<strong>有监督学习</strong>，在这门课程里有一套专门的符号，参数，公式的表示方法：</p><ul> <li>$vector$: 向量（都指列向量）</li> <li>$m$: 训练样本组数</li> <li>$x^{\left(i \right)}$ : 第$i$组输入变量（一般为向量）</li> <li>$y^{\left(i \right)}$: 第$i$组输出变量（一般为向量）</li> <li>$\left(x^{\left(i \right)}, y^{\left(i \right)} \right)$: 第$i$组训练样本</li> <li>$\left(x, y\right)$: 全体训练样本数据</li> <li>$X$: 输入变量空间（一般为矩阵）</li> <li>$Y$: 输出变量空间（一般为矩阵）</li> <li>$h_{\theta} \left(x \right)$: 预测函数</li> <li>$\theta_{j}$: 第$j$组学习参数</li></ul><p><strong>例子：</strong><br>假设我们有一组（房子面积, 价格）数据集，对应下表:</p><table> <thead> <tr> <th style="text-align:center">$Size in feet^{2}$ (x)</th> <th style="text-align:center">$prize$ (y)</th> </tr> </thead> <tbody> <tr> <td style="text-align:center">2104</td> <td style="text-align:center">460</td> </tr> <tr> <td style="text-align:center">1416</td> <td style="text-align:center">232</td> </tr> <tr> <td style="text-align:center">1534</td> <td style="text-align:center">315</td> </tr> <tr> <td style="text-align:center">852</td> <td style="text-align:center">178</td> </tr> </tbody></table><p>其中</p><ul> <li>$m = 4$</li> <li>$x^{\left(1 \right)} = 2104$</li> <li>$y^{\left(1 \right)} = 460$</li> <li>$\left(x^{\left(1 \right)}, y^{\left(1 \right)} \right) = (2104, 460)$ </li> <li>$X$ = $\begin{bmatrix}2104 & 1416 & 1534 & 852\end{bmatrix}^{T}$</li> <li>$Y$ = $\begin{bmatrix}460 & 232 & 315 & 178\end{bmatrix}^{T}$ </li> <li><span>$h_{\theta}\left(x\right)=\theta_{0}+\theta_{1}x$</span> <!-- Has MathJax --> </li> <li>$\theta_{1}$: 第$1$组学习参数</li></ul><p>对于多变量（或者叫<strong>feature - 特征</strong>）的表示方法，如下</p><p><strong>例子：</strong><br>假设我们有一组多个特征的数据，每组特征对应一个确定的输出：</p><table> <thead> <tr> <th style="text-align:center">$X_{0}$</th> <th style="text-align:center">$X_{1}$</th> <th style="text-align:center">$X_{2}$</th> <th style="text-align:center">$X_{3}$</th> <th style="text-align:center">…</th> <th style="text-align:center">$X_{n}$</th> <th style="text-align:center">$Y$</th> </tr> </thead> <tbody> <tr> <td style="text-align:center">$1$</td> <td style="text-align:center">$x_{1}^{(1)}$</td> <td style="text-align:center">$x_{2}^{(1)}$</td> <td style="text-align:center">$x_{3}^{(1)}$</td> <td style="text-align:center">…</td> <td style="text-align:center">$x_{n}^{(1)}$</td> <td style="text-align:center">$y^{(1)}$</td> </tr> <tr> <td style="text-align:center">$1$</td> <td style="text-align:center">$x_{1}^{(2)}$</td> <td style="text-align:center">$x_{2}^{(2)}$</td> <td style="text-align:center">$x_{3}^{(2)}$</td> <td style="text-align:center">…</td> <td style="text-align:center">$x_{n}^{(2)}$</td> <td style="text-align:center">$y^{(2)}$</td> </tr> <tr> <td style="text-align:center">$1$</td> <td style="text-align:center">$x_{1}^{(3)}$</td> <td style="text-align:center">$x_{2}^{(3)}$</td> <td style="text-align:center">$x_{3}^{(3)}$</td> <td style="text-align:center">…</td> <td style="text-align:center">$x_{n}^{(3)}$</td> <td style="text-align:center">$y^{(3)}$</td> </tr> <tr> <td style="text-align:center">…</td> <td style="text-align:center">…</td> <td style="text-align:center">…</td> <td style="text-align:center">…</td> <td style="text-align:center">…</td> <td style="text-align:center">…</td> <td style="text-align:center">…</td> </tr> <tr> <td style="text-align:center">$1$</td> <td style="text-align:center">$x_{1}^{(m)}$</td> <td style="text-align:center">$x_{2}^{(m)}$</td> <td style="text-align:center">$x_{3}^{(m)}$</td> <td style="text-align:center">…</td> <td style="text-align:center">$x_{n}^{(m)}$</td> <td style="text-align:center">$y^{(m)}$</td> </tr> </tbody></table><p>其中</p><ul> <li><span>$X_{m\times (n+1)} = \begin{bmatrix}1 & x_{1}^{(1)} & \cdots &x_{n}^{(1)} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{1}^{(m)} & \cdots & x_{n}^{(m)} \\ \end{bmatrix}$</span> <!-- Has MathJax --> </li> <li> <p>$Y_{m\times 1} = \begin{bmatrix} y^{(1)} & \cdots & y^{(m)}\end{bmatrix}^{T}$</p> </li> <li><span>$\theta = \begin{bmatrix} \theta_{0} & \theta_{1} & \cdots & \theta_{n} \end{bmatrix}^{T}$</span><!-- Has MathJax --> </li> <li> <span>$h_{\theta}\left(x\right)=\begin{bmatrix}h_{\theta}\left(x^{(1)}\right)&h_{\theta}\left(x^{(2)}\right)&\cdots&h_{\theta}\left(x^{(m)}\right)\end{bmatrix}^{T}=\begin{bmatrix}\theta^{T}x^{(1)}&\theta^{T}x^{(2)}&\cdots&\theta^{T}x^{(m)}\end{bmatrix}^{T}$</span> <!-- Has MathJax --> </li></ul><p>整个监督学习的过程，就是找到最优的$\theta$，从而得到最优的<span>$h_{\theta}\left(x\right)$</span> <!-- Has MathJax -->。对于<strong>回归问题</strong>，预测就是我们把一组$x$丢进 <span>$h_{\theta}\left(x\right)$</span><!-- Has MathJax --> 中，得到的结果就是预测值。对于<strong>分类问题</strong>，$h_{\theta}\left(x\right)$得到的结果是一个概率，即输入$x$属于某一类的概率值是多少。或许你觉得我解释得很抽象，因为这里的内容只是让你的大脑对机器学习有一个大致的轮廓。详细的过程将记录在后面的笔记中，请读者放心。</p><h2 id="Cost-Function-代价函数"><a href="#Cost-Function-代价函数" class="headerlink" title="Cost Function - 代价函数"></a>Cost Function - 代价函数</h2><p>回到我们上面的那个（房子面积, 价格）数据集的例子中。这是一个<strong>单变量</strong>的回归问题。如下<br><img src="/posts/Machine%20Learning%20(Week1)/pic8.jpg"></p><p>在这里，我们的预测函数为<span>$h_{\theta}(x)=\theta_{0}+\theta_{1}x$</span> <!-- Has MathJax -->。当$\theta$取不同值时，对应如下图：</p><img src="/posts/Machine%20Learning%20(Week1)/pic9.jpg"><p> 我们要找到最优的$\theta$去拟合$(x,y)$，首先就要定义一个能判断当前$\theta$是否最优的函数，这个函数就是<strong>代价函数</strong>。在这里我们将它定义为$J(\theta)$。</p><p>那么它等于什么呢？下面给个直观的图例辅助解释：</p><img src="/posts/Machine%20Learning%20(Week1)/pic10.jpg"><p>在这幅图里，有3组样本数据为</p><table> <thead> <tr> <th style="text-align:center">$X$</th> <th style="text-align:center">$Y$</th> </tr> </thead> <tbody> <tr> <td style="text-align:center">1</td> <td style="text-align:center">1</td> </tr> <tr> <td style="text-align:center">2</td> <td style="text-align:center">2</td> </tr> <tr> <td style="text-align:center">3</td> <td style="text-align:center">3</td> </tr> </tbody></table><p>分别对应上图三个<font color="#fc0c0c">红色×点</font>。<font color="#000000">黑色斜线</font>为</p><center><br><span>$h_{\theta}(x)=0+0.5x$</span><!-- Has MathJax --><br></center><br>过这3点分别作垂直于$x$轴的<strong>垂线段</strong>交于<span>$h_{\theta}(x)$</span><!-- Has MathJax -->。则第$i$个样本点的误差（即代价）就是该样本点对应<strong>垂线段的长度</strong>，为<br><center><br><span>$\left|h_{\theta}(x^{(i)})-y^{(i)}\right|$</span> <!-- Has MathJax --><br></center><br>为方便处理，我们将绝对值去掉，重新定义误差为<br><center><br><span>$\left(h_{\theta}(x^{(i)})-y^{(i)}\right)^{2}$</span> <!-- Has MathJax --><br></center><br>则总误差为<br><center> <br><span>$\sum_{i=1}^{3}\left(h_{\theta}(x^{(i)})-y^{(i)}\right)^{2}$</span> <!-- Has MathJax --><br></center><br>平均误差为<br><center> <br><span>$\frac{1}{3}\sum_{i=1}^{3}\left(h_{\theta}(x^{(i)})-y^{(i)}\right)^{2}$</span> <!-- Has MathJax --><br></center><br>为了方便后面处理，这里我们一般将平均误差乘一个$\frac{1}{2}$，即<br><center><br><span>$\frac{1}{2 \times 3}\sum_{i=1}^{3}\left(h_{\theta}(x^{(i)})-y^{(i)}\right)^{2}$</span> <!-- Has MathJax --><br></center><br>上式就是我们的代价函数，即<br><center> <br><span>$J(\theta)=\frac{1}{6}\sum_{i=1}^{3}\left(h_{\theta}(x^{(i)})-y^{(i)}\right)^{2}$</span> <!-- Has MathJax --><br></center><br>将上式扩展到$m$个样本点的一般情况，即<br><center> <br><span>$J(\theta)=\frac{1}{2m}\sum_{i=1}^{m}\left(h_{\theta}(x^{(i)})-y^{(i)}\right)^{2}$</span> <!-- Has MathJax --><br></center><p> 当我们的$\theta$能令$J(\theta)$取到最小值时，我们就认为这是最优的$\theta$。<br>是不是很直观？<br>有了代价函数之后，我们要做的就是找出令它取得最小值的$\theta$，下图就是我们的任务：<br><img src="/posts/Machine%20Learning%20(Week1)/pic11.jpg"></p><hr><h1 id="Parameter-Learning-参数学习"><a href="#Parameter-Learning-参数学习" class="headerlink" title="Parameter Learning - 参数学习"></a>Parameter Learning - 参数学习</h1><h2 id="Gradient-Descent-梯度下降"><a href="#Gradient-Descent-梯度下降" class="headerlink" title="Gradient Descent - 梯度下降"></a>Gradient Descent - 梯度下降</h2><p>这是典型的极值问题，在数学方法中，我们可以求导解决，可是在计算机程序中，我们要用一种<strong>通用的数值方法</strong>，去逼近。</p><p>我们看只有一个学习参数的情况，假设<span>$h_{\theta}(x)=\theta x$</span><!-- Has MathJax --></p><p>当<span>$\theta$</span><!-- Has MathJax -->比较小时：<br><img src="/posts/Machine%20Learning%20(Week1)/pic12.png"><br>我们看到<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->没有很好地拟合数据，<span>$J(\theta)$</span><!-- Has MathJax -->比较大。</p><p>当<span>$\theta$</span><!-- Has MathJax -->比较大时：<br><img src="/posts/Machine%20Learning%20(Week1)/pic13.png"><br>同样的，<span>$h_{\theta}(x)$</span> <!-- Has MathJax -->没有很好地拟合数据，<span>$J(\theta)$</span><!-- Has MathJax -->比较大。</p><p>当<span>$\theta$</span><!-- Has MathJax -->比取到能使<span>$h_{\theta}(x)=\theta x$</span><!-- Has MathJax -->很好地拟合数据时：<br><img src="/posts/Machine%20Learning%20(Week1)/pic14.png"><br>这时的<span>$\theta$</span> <!-- Has MathJax -->就是<span>$J(\theta)$</span> <!-- Has MathJax -->的极小值点。也就是最优的<span>$\theta$</span> <!-- Has MathJax -->。<br>接下来我们就来讲，如何让计算机自动训练出最优的$\theta$</p><p>我们继续用上面的例子，<br>当<span>$\theta$</span><!-- Has MathJax -->比较小时：<br><img src="/posts/Machine%20Learning%20(Week1)/pic15.png"><br>我们求得<span>$J(\theta)$</span> <!-- Has MathJax -->在当前<span>$\theta$</span> <!-- Has MathJax -->的导数，<strong>小于</strong><span>$0$</span> <!-- Has MathJax -->。此时我们把<span>$\theta$</span> <!-- Has MathJax -->更新为<span>$\theta - \alpha\frac{dJ(\theta)}{d\theta}$</span> <!-- Has MathJax -->，<span>$\theta$</span> <!-- Has MathJax -->就会<strong>变大</strong>，往极值点靠近。其中<span>$\alpha$</span> <!-- Has MathJax -->为<strong>学习速率</strong>。</p><p>当<span>$\theta$</span><!-- Has MathJax -->比较大时：<br><img src="/posts/Machine%20Learning%20(Week1)/pic16.png"><br>我们求得<span>$J(\theta)$</span> <!-- Has MathJax -->在当前<span>$\theta$</span> <!-- Has MathJax -->的导数，<strong>大于</strong><span>$0$</span> <!-- Has MathJax -->。此时我们把<span>$\theta$</span> <!-- Has MathJax -->更新为<span>$\theta - \alpha\frac{dJ(\theta)}{d\theta}$</span> <!-- Has MathJax -->，<span>$\theta$</span><!-- Has MathJax -->就会 <strong>减小</strong>，往极值点靠近。其中<span>$\alpha$</span> <!-- Has MathJax -->为<strong>学习速率</strong>。</p><p>这就是<strong>梯度下降</strong>算法。通过多次的迭代，更新<span>$\theta$</span> <!-- Has MathJax -->，我们就能无限逼近最优值。</p><p>将<span>$\theta$</span> <!-- Has MathJax -->拓展到<strong>二维向量</strong>（即有两个参数）的情形，我们可能会得到如下的<span>$J(\theta)$</span> <!-- Has MathJax -->：<br><img src="/posts/Machine%20Learning%20(Week1)/pic17.jpg"><br>这是一个二维曲面，这种情况我们就要分别对<span>$\theta_{0},\theta_{1}$</span> <!-- Has MathJax -->求偏导来进行梯度下降。</p><p>对于梯度下降，还有一些要注意的地方：</p><ul> <li>关于<strong>学习速率</strong><span>$\alpha$</span> <!-- Has MathJax -->，怎样设置学习速率也是很关键的问题，如果<span>$\alpha$</span> <!-- Has MathJax -->设置的<strong>过小</strong>，则梯度下降就会收敛得很慢，训练时间会过长。如果<span>$\alpha$</span> <!-- Has MathJax -->设置的过大，则梯度下降有可能会发散，就是越过了极值点：<img src="/posts/Machine%20Learning%20(Week1)/pic18.png"> </li></ul><p>所以我们在做迭代时一定要关注着$J(\theta)$，确保它是在下降的。</p><ul> <li>在实际问题中，我们的<span>$J(\theta)$</span> <!-- Has MathJax -->一般不会是<strong>凸函数</strong>，也就是说我们做梯度下降得到的只是<strong>局部最优值</strong>，而不是<strong>全局最优值</strong>：<img src="/posts/Machine%20Learning%20(Week1)/pic19.png"> <img src="/posts/Machine%20Learning%20(Week1)/pic20.png"> </li></ul><h2 id="Gradient-Descent-for-Liner-Regression-线性回归中的梯度下降"><a href="#Gradient-Descent-for-Liner-Regression-线性回归中的梯度下降" class="headerlink" title="Gradient Descent for Liner Regression - 线性回归中的梯度下降"></a>Gradient Descent for Liner Regression - 线性回归中的梯度下降</h2><p>对于线性回归，我们有如下定义：</p><ul> <li><span>$X_{m\times (n+1)} = \begin{bmatrix}1 & x_{1}^{(1)} & \cdots &x_{n}^{(1)} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{1}^{(m)} & \cdots & x_{n}^{(m)} \\ \end{bmatrix}$</span> <!-- Has MathJax --> </li> <li><span>$Y_{m\times 1} = \begin{bmatrix} y^{(1)} & \cdots & y^{(m)}\end{bmatrix}^{T}$</span><!-- Has MathJax --> </li> <li><span>$\theta = \begin{bmatrix} \theta_{0} & \theta_{1} & \cdots & \theta_{n} \end{bmatrix}^{T}$</span><!-- Has MathJax --> </li> <li><span>$h_{\theta}\left(x\right)=\begin{bmatrix}h_{\theta}\left(x^{(1)}\right)&h_{\theta}\left(x^{(2)}\right)&\cdots&h_{\theta}\left(x^{(m)}\right)\end{bmatrix}^{T}=\begin{bmatrix}\theta^{T}x^{(1)}&\theta^{T}x^{(2)}&\cdots&\theta^{T}x^{(m)}\end{bmatrix}^{T} = X \theta$</span><!-- Has MathJax --> </li> <li><span>$h_{\theta}\left(x^{(i)}\right) = \theta_{0} + \theta_{1}x_{1}^{(i)} + \cdots + \theta_{n}x_{n}^{(i)}$</span><!-- Has MathJax --> </li> <li> <span>$J(\theta)=\frac{1}{2m}\sum_{i=1}^{m}\left(h_{\theta}(x^{(i)})-y^{(i)}\right)^{2}$</span> <!-- Has MathJax --> </li></ul><p>我们对<span>$J(\theta)$</span><!-- Has MathJax -->求所有<span>$\theta$</span> <!-- Has MathJax -->的偏导：</p><center><br><span>$\frac{\partial J}{\partial \theta_{j}} = \frac{1}{m} \sum_{i=1}^{m} \left[\left( h_{\theta}(x^{(i)})-y^{i} \right) \frac{\partial h_{\theta}(x^{(i)})}{\partial \theta_{j}} \right]$</span> <!-- Has MathJax --><br></center><br>当<span>$j=0$</span><!-- Has MathJax -->时：<br><center><br><span>$\frac{\partial h_{\theta}(x^{(i)})}{\partial \theta_{0}} = 1$</span><!-- Has MathJax --><br></center><br>当<span>$j=1 \cdots n$</span><!-- Has MathJax -->时：<br><center><br><span>$\frac{\partial h_{\theta}(x^{(i)})}{\partial \theta_{j}} = x_{j}^{(i)}$</span><!-- Has MathJax --><br></center><br>综上：<br><center><br><span>$\frac{\partial J}{\partial \theta_{0}} = \frac{1}{m} \sum_{i=1}^{m} \left[\left( h_{\theta}(x^{(i)})-y^{(i)} \right)\right]$</span><!-- Has MathJax --><br><br><span>$\frac{\partial J}{\partial \theta_{1}} = \frac{1}{m} \sum_{i=1}^{m} \left[\left( h_{\theta}(x^{(i)})-y^{(i)} \right) x_{1}^{(i)} \right]$</span> <!-- Has MathJax --><br><br><span>$\vdots$</span> <!-- Has MathJax --><br><br><span>$\frac{\partial J}{\partial \theta_{n}} = \frac{1}{m} \sum_{i=1}^{m} \left[\left( h_{\theta}(x^{(i)})-y^{(i)} \right) x_{n}^{(i)} \right]$</span><!-- Has MathJax --><br></center><br>更新<span>$\theta$</span><!-- Has MathJax -->：<br><center><br><span>$\theta_{0}:=\theta_{0} - \alpha \frac{\partial J}{\partial \theta_{0}} = \theta_{0} - \alpha \frac{1}{m} \sum_{i=1}^{m} \left[\left( h_{\theta}(x^{(i)})-y^{(i)} \right)\right]$</span> <!-- Has MathJax --><br><br><span>$\theta_{1}:=\theta_{1} - \alpha \frac{\partial J}{\partial \theta_{1}} = \theta_{1} - \alpha \frac{1}{m} \sum_{i=1}^{m} \left[\left( h_{\theta}(x^{(i)})-y^{(i)} \right) x_{1}^{(i)} \right]$</span><!-- Has MathJax --><br><br><span>$\vdots$</span> <!-- Has MathJax --><br><br><span>$\theta_{n}:=\theta_{n} - \alpha \frac{\partial J}{\partial \theta_{n}} = \theta_{n} - \alpha \frac{1}{m} \sum_{i=1}^{m} \left[\left( h_{\theta}(x^{(i)})-y^{(i)} \right) x_{n}^{(i)} \right]$</span><!-- Has MathJax --><br></center><p>我们将上述过程向量化：</p><ul> <li> <p>首先将偏导数向量化：<br> <center> </center></p> <span>$$\frac{\partial J}{\partial \theta_{0}} = \frac{1}{m} \begin{bmatrix}1&1&\cdots&1\end{bmatrix} \begin{bmatrix}h_{\theta}(x^{(1)})-y^{(1)}\\h_{\theta}(x^{(2)})-y^{(2)}\\ \vdots\\h_{\theta}(x^{(m)})-y^{(m)}\end{bmatrix}$$</span> <!-- Has MathJax --> <span>$$\frac{\partial J}{\partial \theta_{1}} = \frac{1}{m} \begin{bmatrix}x_{1}^{(1)}&x_{1}^{(2)}&\cdots&x_{1}^{(m)}\end{bmatrix} \begin{bmatrix}h_{\theta}(x^{(1)})-y^{(1)}\\h_{\theta}(x^{(2)})-y^{(2)}\\ \vdots\\h_{\theta}(x^{(m)})-y^{(m)}\end{bmatrix}$$</span> <!-- Has MathJax --> <span>$$\vdots$$</span><!-- Has MathJax --> <span>$$\frac{\partial J}{\partial \theta_{n}} = \frac{1}{m} \begin{bmatrix}x_{n}^{(1)}&x_{n}^{(2)}&\cdots&x_{n}^{(m)}\end{bmatrix} \begin{bmatrix}h_{\theta}(x^{(1)})-y^{(1)}\\h_{\theta}(x^{(2)})-y^{(2)}\\ \vdots\\h_{\theta}(x^{(m)})-y^{(m)}\end{bmatrix}$$</span> <!-- Has MathJax --> <p>可得：</p> <span>$$\frac{\partial J}{\partial \theta} = grad_{(n+1)\times 1} = \begin{bmatrix}\frac{\partial J}{\partial \theta_{0}}\\\frac{\partial J}{\partial \theta_{1}}\\ \vdots\\\frac{\partial J}{\partial \theta_{n}}\end{bmatrix} = \frac{1}{m} X^{T}(X\theta - Y)$$</span> <!-- Has MathJax --> </li> <li> <p>接着将梯度下降的过程向量化：</p> <span>$$\theta := \theta - \alpha \frac{1}{m} X^{T}(X\theta - Y)$$</span> <!-- Has MathJax --> <p> </p> </li></ul><hr><p>PS:其实上面的<strong>多变量线性回归梯度下降</strong> 是 <strong>week2</strong>的内容，因为不算太复杂我就搬到这里讲了，那么<strong>week2</strong>的笔记里就会跳过这部分内容，请大家注意。</p><p><br><br> <div class="note info"> <p> <center><strong>课程资料</strong></center> </p> <ul> <li><a href="https://github.com/tankeryang/Coursera-machine-learning-lecture-note/tree/master/week1" target="_blank" rel="noopener">week1课程讲义</a> </li> </ul> </div></p>]]></content>
<summary type="html">
<p>在第一周的课程里简要介绍了<strong>什么是机器学习</strong>，<strong>Model -
模型</strong>和<strong>Cost Function -
代价函数</strong>的概念，以及一些必要的<strong>线性代数</strong>的知识。</p>
<img src="/posts/Machine%20Learning%20(Week1)/pic0.png">
<div class="note warning">
<p><strong>特别声明</strong>：这里不会对线性代数基础进行记录，有需要了解的请自行学习。</p>
<center><strong>敬请留意</strong></center>
</div>
</summary>
<category term="机器学习笔记" scheme="https://tankeryang.github.io/categories/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/"/>
<category term="机器学习" scheme="https://tankeryang.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="ML" scheme="https://tankeryang.github.io/tags/ML/"/>
<category term="数学" scheme="https://tankeryang.github.io/tags/%E6%95%B0%E5%AD%A6/"/>
</entry>
<entry>
<title>Machine Learning (Week0) Openning</title>
<link href="https://tankeryang.github.io/posts/Machine%20Learning%20(Week0)%20Openning/"/>
<id>https://tankeryang.github.io/posts/Machine Learning (Week0) Openning/</id>
<published>2017-09-06T14:07:42.000Z</published>
<updated>2019-10-09T17:06:09.551Z</updated>
<content type="html"><![CDATA[<p> <center> <h1>开篇的话</h1> </center></p><ul> <li>这里记录了我在Coursera上听吴恩达机器学习课程的笔记，以表示我是有学习的:)</li> <li>同时会把课程相关资料整理一并发布，包括lecturenote，homework，和我的答案解释。</li> <li><a href="https://www.coursera.org/learn/machine-learning/home/welcome" target="_blank" rel="noopener">课程链接</a>:可以直接注册听课，但是不会有证书。 </li></ul><div class="note warning"> <p> <strong>特别声明</strong>：笔记在一些对类似什么是机器学习，机器学习的应用这些字面意义上的概念只会一笔带过，需要了解的自行谷歌或百度，笔记之着重于机器学习本身的内容，算法，推导，在一些理论细节上可能会做详细深入。 </p> <center><strong>敬请留意</strong></center></div>]]></content>
<summary type="html">
<p>
<center>
<h1>开篇的话</h1>
</center>
</p>
<ul>
<li>这里记录了我在Coursera上听吴恩达机器学习课程的笔记，以表示我是有学习的:)</li>
<li>同时会把课程相关资料整理一并发布，包括lecture
</summary>
<category term="机器学习笔记" scheme="https://tankeryang.github.io/categories/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/"/>
<category term="机器学习" scheme="https://tankeryang.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="ML" scheme="https://tankeryang.github.io/tags/ML/"/>
<category term="数学" scheme="https://tankeryang.github.io/tags/%E6%95%B0%E5%AD%A6/"/>
</entry>
<entry>
<title>发布测试</title>
<link href="https://tankeryang.github.io/posts/%E5%8F%91%E5%B8%83%E6%B5%8B%E8%AF%95/"/>
<id>https://tankeryang.github.io/posts/发布测试/</id>
<published>2017-09-03T00:00:00.000Z</published>
<updated>2019-10-09T17:06:09.623Z</updated>
<content type="html"><![CDATA[<h1 id="博客发布测试"><a href="#博客发布测试" class="headerlink" title="博客发布测试"></a>博客发布测试</h1><h2 id="2级标题"><a href="#2级标题" class="headerlink" title="2级标题"></a>2级标题</h2><figure class="highlight python"> <table> <tr> <td class="gutter"> <pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre> </td> <td class="code"> <pre><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">func</span><span class="params">(args)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> args</span><br></pre> </td> </tr> </table></figure><div class="note "> <p>Content (md partial supported) </p></div><div class="note default"> <p>Content (md partial supported) </p></div><div class="note primary"> <p>Content (md partial supported) </p></div><div class="note success"> <p>Content (md partial supported) </p></div><div class="note info"> <p>Content (md partial supported) </p></div><div class="note warning"> <p>Content (md partial supported) </p></div><div class="note danger"> <p>Content (md partial supported) </p></div><div class="note danger"> <p><i class="fa fa-spinner fa-pulse fa-lg margin-bottom" aria-hidden="true"></i> 未完待续…有空继续… </p></div><hr><blockquote class="blockquote-center"> <p>$$n\cdot m\cdot \lg b$$ </p></blockquote><span class="exturl" data-url="aHR0cHM6Ly9oZXhvLmlvL3poLWNuL2RvY3Mv" title="hexo doc"><i class="fa fa-external-link"></i></span><a id="more"></a><p>Simple inline $a = b + c$.</p><p>$$\frac{\partial u}{\partial t}<br>= h^2 \left( \frac{\partial^2 u}{\partial x^2} +<br>\frac{\partial^2 u}{\partial y^2} +<br>\frac{\partial^2 u}{\partial z^2}\right)$$</p><span>$$J\left(\theta\right) = -\left[\frac{1}{m}\sum_{i=1}^{m}y^\left(i\right)log\left(h_{\theta}\left(x^{\left(i\right)}\right)\right)+ \left(1-y^{\left(i\right)}\right)log\left(1-h_{\theta}\left(x^{\left(i\right)}\right)\right)\right]+ \frac{\lambda}{2m}\sum_{j=1}^{n}\theta_{j}^{2}$$</span><!-- Has MathJax --><table> <thead> <tr> <th style="text-align:center">$Size in feet^{2}$ (x)</th> <th style="text-align:center">$prize$ (y)</th> </tr> </thead> <tbody> <tr> <td style="text-align:center">2104</td> <td style="text-align:center">460</td> </tr> <tr> <td style="text-align:center">1416</td> <td style="text-align:center">232</td> </tr> <tr> <td style="text-align:center">1534</td> <td style="text-align:center">315</td> </tr> <tr> <td style="text-align:center">852</td> <td style="text-align:center">178</td> </tr> </tbody></table><img src="http://www.plantuml.com/plantuml/svg/qqWqDDHKqBLJq4WqDDBaqa1mZL2vna0ka6-4gje8LRKHccfZLDN6gAgDaLG30000"> <p><br></p><img src="http://www.plantuml.com/plantuml/svg/POv12iCm30Jl-mfz81y8mM0xBoYKdesZWod4gQ8M-ljQJY65NdJGxixkbaHBBbjQbQux2s2N1aT-fBdUgjoHifPNKYOFU9F2IDD4MRWzB_nuygxatEaBG31V_4T4H1eEX0bpNHbBoqjzDyrYH_KN6tl7qYo_y3x2TXg_0yw97m00"> <p><br></p><div class="mermaid"> graph LR; a11(("a1[0](i)")) --> a12(("a1[1](i)")); a11(("a1[0](i)")) --> a22(("a2[1](i)")); a11(("a1[0](i)")) --> a32(("a3[1](i)")); a21(("a2[0](i)")) --> a12(("a1[1](i)")); a21(("a2[0](i)")) --> a22(("a2[1](i)")); a21(("a2[0](i)")) --> a32(("a3[1](i)")); a31(("a3[0](i)")) --> a12(("a1[1](i)")); a31(("a3[0](i)")) --> a22(("a2[1](i)")); a31(("a3[0](i)")) --> a32(("a3[1](i)")); a12(("a1[1](i)")) --> y(("a1[2](i)")); a22(("a2[1](i)")) --> y(("a1[2](i)")); a32(("a3[1](i)")) --> y(("a1[2](i)")); y(("a1[2](i)")) --> yh((yhat));</div><div class="mermaid"> graph TB; subgraph L0; a11(("a1[0](i)")) --- a21(("a2[0](i)")); a21(("a2[0](i)")) --- a31(("a3[0](i)")); end;</div>]]></content>
<summary type="html">
<h1 id="博客发布测试"><a href="#博客发布测试" class="headerlink" title="博客发布测试"></a>博客发布测试
</h1>
<h2 id="2级标题"><a href="#2级标题" class="headerlink" title="2级标题"></a>2级标题</h2>
<figure class="highlight python">
<table>
<tr>
<td class="gutter">
<pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre>
</td>
<td class="code">
<pre><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">func</span><span class="params">(args)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> args</span><br></pre>
</td>
</tr>
</table>
</figure>
<div class="note ">
<p>Content (md partial supported) </p>
</div>
<div class="note default">
<p>Content (md partial supported) </p>
</div>
<div class="note primary">
<p>Content (md partial supported) </p>
</div>
<div class="note success">
<p>Content (md partial supported) </p>
</div>
<div class="note info">
<p>Content (md partial supported) </p>
</div>
<div class="note warning">
<p>Content (md partial supported) </p>
</div>
<div class="note danger">
<p>Content (md partial supported) </p>
</div>
<div class="note danger">
<p><i class="fa fa-spinner fa-pulse fa-lg margin-bottom" aria-hidden="true"></i>&nbsp;未完待续…有空继续…
</p>
</div>
<hr>
<blockquote class="blockquote-center">
<p>$$n\cdot m\cdot \lg b$$ </p>
</blockquote>
<span class="exturl" data-url="aHR0cHM6Ly9oZXhvLmlvL3poLWNuL2RvY3Mv" title="hexo doc"><i class="fa fa-external-link"></i></span>
</summary>
<category term="回收站" scheme="https://tankeryang.github.io/categories/%E5%9B%9E%E6%94%B6%E7%AB%99/"/>
<category term="博客测试" scheme="https://tankeryang.github.io/tags/%E5%8D%9A%E5%AE%A2%E6%B5%8B%E8%AF%95/"/>
</entry>
</feed>