-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
1628 lines (1147 loc) · 331 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html>
<head><meta name="generator" content="Hexo 3.8.0">
<meta charset="utf-8">
<meta name="renderer" content="webkit">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<link rel="dns-prefetch" href="https://ansgoo.github.io/Ranger">
<title>派对实验室</title>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
<meta property="og:type" content="website">
<meta property="og:title" content="派对实验室">
<meta property="og:url" content="https://ansgoo.github.io/Ranger/index.html">
<meta property="og:site_name" content="派对实验室">
<meta property="og:locale" content="default">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="派对实验室">
<link rel="alternative" href="/atom.xml" title="派对实验室" type="application/atom+xml">
<link rel="icon" href="/favicon.png">
<link rel="stylesheet" type="text/css" href="/Ranger/./main.0cf68a.css">
<style type="text/css">
#container.show {
background: linear-gradient(200deg,#a0cfe4,#e8c37e);
}
</style>
</head>
</html>
<body>
<div id="container" q-class="show:isCtnShow">
<canvas id="anm-canvas" class="anm-canvas"></canvas>
<div class="left-col" q-class="show:isShow">
<div class="overlay" style="background: #4d4d4d"></div>
<div class="intrude-less">
<header id="header" class="inner">
<a href="/Ranger/" class="profilepic">
<img src="/Ranger/avatar/Avrt.jpg" class="js-avatar">
</a>
<hgroup>
<h1 class="header-author"><a href="/Ranger/">独行侠</a></h1>
</hgroup>
<p class="header-subtitle">Hi,外星人</p>
<nav class="header-menu">
<ul>
<li><a href="/Ranger/">主页</a></li>
</ul>
</nav>
<nav class="header-smart-menu">
<a q-on="click: openSlider(e, 'innerArchive')" href="javascript:void(0)">所有文章</a>
<a q-on="click: openSlider(e, 'aboutme')" href="javascript:void(0)">关于我</a>
</nav>
<nav class="header-nav">
<div class="social">
<a class="github" target="_blank" href="https://github.com/AnsGoo" title="github"><i class="icon-github"></i></a>
<a class="qq" target="_blank" href="/Ranger/724998120" title="qq"><i class="icon-qq"></i></a>
<a class="mail" target="_blank" href="/Ranger/haiven_123@163.com" title="mail"><i class="icon-mail"></i></a>
</div>
</nav>
</header>
</div>
</div>
<div class="mid-col" q-class="show:isShow,hide:isShow|isFalse">
<nav id="mobile-nav">
<div class="overlay js-overlay" style="background: #4d4d4d"></div>
<div class="btnctn js-mobile-btnctn">
<div class="slider-trigger list" q-on="click: openSlider(e)"><i class="icon icon-sort"></i></div>
</div>
<div class="intrude-less">
<header id="header" class="inner">
<div class="profilepic">
<img src="/Ranger/avatar/Avrt.jpg" class="js-avatar">
</div>
<hgroup>
<h1 class="header-author js-header-author">独行侠</h1>
</hgroup>
<p class="header-subtitle"><i class="icon icon-quo-left"></i>Hi,外星人<i class="icon icon-quo-right"></i></p>
<nav class="header-nav">
<div class="social">
<a class="github" target="_blank" href="https://github.com/AnsGoo" title="github"><i class="icon-github"></i></a>
<a class="qq" target="_blank" href="/Ranger/724998120" title="qq"><i class="icon-qq"></i></a>
<a class="mail" target="_blank" href="/Ranger/haiven_123@163.com" title="mail"><i class="icon-mail"></i></a>
</div>
</nav>
<nav class="header-menu js-header-menu">
<ul style="width: 50%">
<li style="width: 100%"><a href="/Ranger/">主页</a></li>
</ul>
</nav>
</header>
</div>
<div class="mobile-mask" style="display:none" q-show="isShow"></div>
</nav>
<div id="wrapper" class="body-wrap">
<div class="menu-l">
<div class="canvas-wrap">
<canvas data-colors="#eaeaea" data-sectionHeight="100" data-contentId="js-content" id="myCanvas1" class="anm-canvas"></canvas>
</div>
<div id="js-content" class="content-ll">
<article id="post-实用python工具包锦集" class="article article-type-post article-index" itemscope itemprop="blogPost">
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/Ranger/2019/10/10/实用python工具包锦集/">实用python工具包锦集</a>
</h1>
<a href="/Ranger/2019/10/10/实用python工具包锦集/" class="archive-article-date">
<time datetime="2019-10-10T14:25:02.000Z" itemprop="datePublished"><i class="icon-calendar icon"></i>2019-10-10</time>
</a>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="sh"><a href="#sh" class="headerlink" title="sh"></a>sh</h2><p>sh 是一个成熟,用于替代 subprocess,它允许你调用任何程序,就像它是一个函数,支持 Python2.6 - 3.5</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> sh <span class="keyword">import</span> ifconfig</span><br><span class="line"><span class="keyword">print</span> ifconfig(<span class="string">"eth0"</span>)</span><br></pre></td></tr></table></figure>
<h2 id="mycli"><a href="#mycli" class="headerlink" title="mycli"></a>mycli</h2><p><code>mycli</code> 是一个带语法高亮、自动补全的 MySQL 命令行客户端工具。例如,连接数据库方法:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mycli -h localhost -u 用户名 数据库</span><br></pre></td></tr></table></figure>
<h2 id="python-fire"><a href="#python-fire" class="headerlink" title="python-fire"></a>python-fire</h2><p>Fire 是 Google 开源的 Python 库,可自动将您的代码转变成 CLI,无需您做任何额外工作。您不必定义参数,设置帮助信息,或者编写定义代码运行方式的 main 函数。相反,您只需从 main 模块调用“Fire”函数,其余工作全部交由 Python Fire 来完成。示例代码如下:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> fire</span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Example</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(self, name=<span class="string">'world'</span>)</span>:</span></span><br><span class="line"> <span class="string">"""Says hello to the specified name."""</span></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'Hello {name}!'</span>.format(name=name)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"> fire.Fire(Example)</span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> main()</span><br></pre></td></tr></table></figure>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> 在终端中调用效果如下:</span><br><span class="line"><span class="meta">$</span> ./example.py hello</span><br><span class="line">Hello world!</span><br><span class="line"><span class="meta">$</span> ./example.py hello David</span><br><span class="line">Hello David!</span><br><span class="line"><span class="meta">$</span> ./example.py hello --name=Google</span><br><span class="line">Hello Google!</span><br></pre></td></tr></table></figure>
<h2 id="pdir2"><a href="#pdir2" class="headerlink" title="pdir2"></a>pdir2</h2><p>Python 程序员需要一个更好的 dir() —— 以更加友好的显示 dir() 输出的结果</p>
<h2 id="freezegun"><a href="#freezegun" class="headerlink" title="freezegun"></a>freezegun</h2><p>时间漫步模块,模拟到某一个时间,使用简单方式多样,实现了装饰器、上下文等调用方式。示例代码如下:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> freezegun <span class="keyword">import</span> freeze_time</span><br><span class="line"><span class="keyword">import</span> datetime</span><br><span class="line"><span class="keyword">import</span> unittest</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@freeze_time("2012-01-14")</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">assert</span> datetime.datetime.now() == datetime.datetime(<span class="number">2012</span>, <span class="number">1</span>, <span class="number">14</span>)</span><br></pre></td></tr></table></figure>
<h2 id="records"><a href="#records" class="headerlink" title="records"></a>records</h2><p>Kenneth Reitz 大神的for Humans™系列,Records 是一个支持大多数主流关系数据库的原生 SQL 查询第三方库。API 友好,使用简单、支持命令行模式、功能多样。与此同时该库只有 500 行代码,可以当作入门阅读源码的项目,同时学习大神的编程技巧与习惯,示例代码如下:</p>
<h2 id="pygorithm"><a href="#pygorithm" class="headerlink" title="pygorithm"></a>pygorithm</h2><p>一个帮助学习主要算法的库,可以通过理解这些算法的实现,提高自己的算法水平</p>
<h2 id="huey"><a href="#huey" class="headerlink" title="huey"></a>huey</h2><p>结合 redis 实现的轻量任务队列,但是支持功能还是很多的:</p>
<ul>
<li>多进程、多线程、协程</li>
<li>任务定时执行</li>
<li>任务执行失败重试</li>
<li>结果存储</li>
</ul>
<h2 id="joblib"><a href="#joblib" class="headerlink" title="joblib"></a>joblib</h2><p>使用 Python 方便的进行并行计算,示例代码如下:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> joblib <span class="keyword">import</span> Parallel, delayed</span><br><span class="line"><span class="keyword">from</span> math <span class="keyword">import</span> sqrt</span><br><span class="line">Parallel(n_jobs=<span class="number">1</span>)(delayed(sqrt)(i**<span class="number">2</span>) <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">10</span>))</span><br></pre></td></tr></table></figure>
<h2 id="thefuck"><a href="#thefuck" class="headerlink" title="thefuck"></a>thefuck</h2><p>在 Linux 命令行中,当你输入的命令有错误后,直接输入 fuck 就可以自动执行修复后的命令</p>
<h2 id="pook"><a href="#pook" class="headerlink" title="pook"></a>pook</h2><p>模拟 HTTP 请求结果的库,可用于单元测试等场景。采用装饰器方式调用的示例代码如下:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> pook</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line"><span class="meta">@pook.get('http://httpbin.org/status/500', reply=204)</span></span><br><span class="line"><span class="meta">@pook.get('http://httpbin.org/status/400', reply=200)</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fetch</span><span class="params">(url)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> requests.get(url)</span><br><span class="line"></span><br><span class="line">res = fetch(<span class="string">'http://httpbin.org/status/400'</span>)</span><br><span class="line">print(<span class="string">'#1 status:'</span>, res.status_code)</span><br><span class="line"></span><br><span class="line">res = fetch(<span class="string">'http://httpbin.org/status/500'</span>)</span><br><span class="line">print(<span class="string">'#2 status:'</span>, res.status_code)</span><br></pre></td></tr></table></figure>
<h2 id="tenacity"><a href="#tenacity" class="headerlink" title="tenacity"></a>tenacity</h2><p>使用该库可以优雅地实现各种需求的重试。示例代码如下:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> tenacity <span class="keyword">import</span> retry, stop_after_attempt</span><br><span class="line"></span><br><span class="line"><span class="comment"># 通过装饰器,实现遇到异常重试3次</span></span><br><span class="line"><span class="meta">@retry(stop=stop_after_attempt(3)) </span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_data</span><span class="params">(url)</span>:</span></span><br><span class="line"> response = requests.get(url)</span><br><span class="line"> response_json = response.json()</span><br></pre></td></tr></table></figure>
<h2 id="pudb"><a href="#pudb" class="headerlink" title="pudb"></a>pudb</h2><p>基于控制台的全屏 Python 可视化调试器。比 pdb 好用太多了,特性:</p>
<ul>
<li>源码语法高亮,栈、断点、变量可见并且一直动态更新。变量展示还有很多可以定制化的功能。</li>
<li>基于键盘,简单高效。支持 VI 的鼠标移动。还支持 PDB 的某些命令</li>
<li>支持查找源代码,可以使用 m 代用 module browser 查看载入的模块</li>
<li>断点设置。鼠标移到某行代码,按 b,然后可以在断点窗口编辑断点</li>
</ul>
<h2 id="loguru"><a href="#loguru" class="headerlink" title="loguru"></a>loguru</h2><p>一个让 Python 记录日志变得简单的库</p>
<h2 id="click"><a href="#click" class="headerlink" title="click"></a>click</h2><p>Python 的第三方库,用于快速创建命令行。支持装饰器方式调用、多种参数类型、自动生成帮助信息等。示例代码如下:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> click</span><br><span class="line"></span><br><span class="line"><span class="meta">@click.command()</span></span><br><span class="line"><span class="meta">@click.option("--count", default=1, help="Number of greetings.")</span></span><br><span class="line"><span class="meta">@click.option("--name", prompt="Your name",</span></span><br><span class="line"> help=<span class="string">"The person to greet."</span>)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(count, name)</span>:</span></span><br><span class="line"> <span class="string">"""Simple program that greets NAME for a total of COUNT times."""</span></span><br><span class="line"> <span class="keyword">for</span> _ <span class="keyword">in</span> range(count):</span><br><span class="line"> click.echo(<span class="string">"Hello, %s!"</span> % name)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> hello()</span><br></pre></td></tr></table></figure>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="meta">#</span> 下面为运行效果</span><br><span class="line"><span class="meta">$</span> python hello.py --count=3</span><br><span class="line">Your name: Click</span><br><span class="line">Hello, Click!</span><br><span class="line">Hello, Click!</span><br><span class="line">Hello, Click!</span><br></pre></td></tr></table></figure>
<h2 id="pyright"><a href="#pyright" class="headerlink" title="pyright"></a>pyright</h2><p>微软出品的 Python 静态类型检查工具。执行速度快,适合大型 Python 项目,引用一句话:动态语言一时爽,重构火葬场</p>
<h2 id="PySnooper"><a href="#PySnooper" class="headerlink" title="PySnooper"></a>PySnooper</h2><p>Python 的第三方调试库。让你通过装饰器方法,方便的知道每一行程序运行后的结果,而不需要再手动增加 print 展示过程数据、调试程序。示例代码:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pysnooper</span><br><span class="line"></span><br><span class="line"><span class="meta">@pysnooper.snoop()</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">number_to_bits</span><span class="params">(number)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> number:</span><br><span class="line"> bits = []</span><br><span class="line"> <span class="keyword">while</span> number:</span><br><span class="line"> number, remainder = divmod(number, <span class="number">2</span>)</span><br><span class="line"> bits.insert(<span class="number">0</span>, remainder)</span><br><span class="line"> <span class="keyword">return</span> bits</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">return</span> [<span class="number">0</span>]</span><br><span class="line"> </span><br><span class="line">number_to_bits(<span class="number">6</span>)</span><br></pre></td></tr></table></figure>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> 输出如下</span><br><span class="line">Starting var:.. number = 6</span><br><span class="line">15:29:11.327032 call 4 def number_to_bits(number):</span><br><span class="line">15:29:11.327032 line 5 if number:</span><br><span class="line">15:29:11.327032 line 6 bits = []</span><br><span class="line">New var:....... bits = []</span><br><span class="line">15:29:11.327032 line 7 while number:</span><br><span class="line">15:29:11.327032 line 8 number, remainder = divmod(number, 2)</span><br><span class="line">New var:....... remainder = 0</span><br><span class="line">Modified var:.. number = 3</span><br><span class="line">....</span><br></pre></td></tr></table></figure>
<h2 id="装饰器妙用"><a href="#装饰器妙用" class="headerlink" title="装饰器妙用"></a>装饰器妙用</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> datetime</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">count_time</span><span class="params">(func)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">int_time</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line"> start_time = datetime.datetime.now() <span class="comment"># 程序开始时间</span></span><br><span class="line"> func()</span><br><span class="line"> over_time = datetime.datetime.now() <span class="comment"># 程序结束时间</span></span><br><span class="line"> total_time = (over_time-start_time).total_seconds()</span><br><span class="line"> print(<span class="string">'程序共计%s秒'</span> % total_time)</span><br><span class="line"> <span class="keyword">return</span> int_time</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@count_time</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'>>>>开始计算函数运行时间'</span>)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">1</span>, <span class="number">1000</span>): <span class="comment"># 可以是任意函数 , 这里故意模拟函数的运行时间</span></span><br><span class="line"> <span class="keyword">for</span> j <span class="keyword">in</span> range(i):</span><br><span class="line"> print(j)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> main()</span><br></pre></td></tr></table></figure>
<h2 id="arrow"><a href="#arrow" class="headerlink" title="arrow"></a>arrow</h2><p>还在为处理时间、时区、转化、夏令时等问题而头疼吗?这个 Python 的第三方时间库。提供了更便捷的方式来创建、操作和格式化时间和日期,用更少的代码来处理时间和日期</p>
</div>
<div class="article-info article-info-index">
<div class="article-tag tagcloud">
<i class="icon-price-tags icon"></i>
<ul class="article-tag-list">
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">python</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">第三方工具包</a>
</li>
</ul>
</div>
<p class="article-more-link">
<a class="article-more-a" href="/Ranger/2019/10/10/实用python工具包锦集/">展开全文 >></a>
</p>
<div class="clearfix"></div>
</div>
</div>
</article>
<aside class="wrap-side-operation">
<div class="mod-side-operation">
<div class="jump-container" id="js-jump-container" style="display:none;">
<a href="javascript:void(0)" class="mod-side-operation__jump-to-top">
<i class="icon-font icon-back"></i>
</a>
<div id="js-jump-plan-container" class="jump-plan-container" style="top: -11px;">
<i class="icon-font icon-plane jump-plane"></i>
</div>
</div>
</div>
</aside>
<article id="post-Celery4-1-中文文档" class="article article-type-post article-index" itemscope itemprop="blogPost">
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/Ranger/2019/08/01/Celery4-1-中文文档/">Celery4.1 中文文档</a>
</h1>
<a href="/Ranger/2019/08/01/Celery4-1-中文文档/" class="archive-article-date">
<time datetime="2019-08-01T15:01:36.000Z" itemprop="datePublished"><i class="icon-calendar icon"></i>2019-08-01</time>
</a>
</header>
<div class="article-entry" itemprop="articleBody">
<h1 id="Application-应用"><a href="#Application-应用" class="headerlink" title="Application(应用)"></a>Application(应用)</h1><p>Celery 库在使用之前必须初始化,一个celery实例被称为一个应用(或者缩写 app)。</p>
<p>Celery 应用是线程安全的,所以多个不同配置、不同组件、不同任务的 应用可以在一个进程空间里共存。<br>下面创建一个 celery 应用:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> celery <span class="keyword">import</span> Celery</span><br><span class="line"><span class="meta">>>> </span>app = Celery()</span><br><span class="line"><span class="meta">>>> </span>app</span><br><span class="line"><Celery __main__:<span class="number">0x100469fd0</span>></span><br></pre></td></tr></table></figure>
<p>最后一行显示的是 celery 应用的文本表示: 包含应用类的名称(Celery),当前主模块的名称(main),以及应用对象的内存地址(0x100469fd0)。</p>
<h2 id="Main-Name"><a href="#Main-Name" class="headerlink" title="Main Name"></a>Main Name</h2><p>上述文本表示中只有一部分是重要的,那就是主模块名称。下面分析下它为何重要。</p>
<p>当你发送一个消息给 Celery,消息中不会包含任何源码,而只有你想要执行的任务的名称。这就好像因特网上的域名映射原理一般:每个执行单元维护着一个任务名称到实际任务函数的映射,这个映射被称为任务注册表。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>@app.task</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">add</span><span class="params">(x, y)</span>:</span></span><br><span class="line"><span class="meta">... </span> <span class="keyword">return</span> x + y</span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>add</span><br><span class="line"><@task: __main__.add></span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>add.name</span><br><span class="line">__main__.add</span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>app.tasks[<span class="string">'__main__.add'</span>]</span><br><span class="line"><@task: __main__.add></span><br></pre></td></tr></table></figure>
<p>现在,你又看到显示 <strong>main</strong> 模块名称;当Celery不能探查到这个任务函数属于哪个模块时,它将使用主模块名称来产生任务名称的前缀。</p>
<p>这在有些情况下会产生问题: </p>
<ol>
<li>定义任务的主模块作为一个程序运行。 </li>
<li>应用在python交互终端创建。</li>
</ol>
<p>例如下面程序,定义任务的模块还调用 <code>app.worker_main()</code> 来启动一个工作单元:<br>task.py</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> celery <span class="keyword">import</span> Celery</span><br><span class="line">app = Celery()</span><br><span class="line"></span><br><span class="line"><span class="meta">@app.task</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">add</span><span class="params">(x, y)</span>:</span> <span class="keyword">return</span> x + y</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> app.worker_main()</span><br></pre></td></tr></table></figure>
<p>当这个模块运行,任务将以前缀<code>__main__</code> 命名,但是当该模块被其他进程引入来运行一个任务,这个任务的名称将以前缀 tasks 命名(即这个模块的真实名称)。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> tasks <span class="keyword">import</span> add</span><br><span class="line"><span class="meta">>>> </span>add.name</span><br><span class="line">tasks.add</span><br></pre></td></tr></table></figure>
<p>你可以给主模块声明另外一个名称</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>app = Celery(<span class="string">'tasks'</span>)</span><br><span class="line"><span class="meta">>>> </span>app.main</span><br><span class="line"><span class="string">'tasks'</span></span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>@app.task</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">add</span><span class="params">(x, y)</span>:</span></span><br><span class="line"><span class="meta">... </span> <span class="keyword">return</span> x + y</span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>add.name</span><br><span class="line">tasks.add</span><br></pre></td></tr></table></figure>
<p>具体可以查看名称这节</p>
</div>
<div class="article-info article-info-index">
<div class="article-tag tagcloud">
<i class="icon-price-tags icon"></i>
<ul class="article-tag-list">
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">python</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">Celery</a>
</li>
</ul>
</div>
<p class="article-more-link">
<a class="article-more-a" href="/Ranger/2019/08/01/Celery4-1-中文文档/">展开全文 >></a>
</p>
<div class="clearfix"></div>
</div>
</div>
</article>
<aside class="wrap-side-operation">
<div class="mod-side-operation">
<div class="jump-container" id="js-jump-container" style="display:none;">
<a href="javascript:void(0)" class="mod-side-operation__jump-to-top">
<i class="icon-font icon-back"></i>
</a>
<div id="js-jump-plan-container" class="jump-plan-container" style="top: -11px;">
<i class="icon-font icon-plane jump-plane"></i>
</div>
</div>
</div>
</aside>
<article id="post-Gevent-简明教程" class="article article-type-post article-index" itemscope itemprop="blogPost">
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/Ranger/2019/07/28/Gevent-简明教程/">Gevent 简明教程</a>
</h1>
<a href="/Ranger/2019/07/28/Gevent-简明教程/" class="archive-article-date">
<time datetime="2019-07-28T14:05:19.000Z" itemprop="datePublished"><i class="icon-calendar icon"></i>2019-07-28</time>
</a>
</header>
<div class="article-entry" itemprop="articleBody">
<h1 id="前述"><a href="#前述" class="headerlink" title="前述"></a>前述</h1><h3 id="进程-线程-协程-异步"><a href="#进程-线程-协程-异步" class="headerlink" title="进程 线程 协程 异步"></a>进程 线程 协程 异步</h3><p>并发编程(不是并行)目前有四种方式:多进程、多线程、协程和异步。</p>
<ul>
<li>多进程编程在python中有类似C的os.fork,更高层封装的有multiprocessing标准库</li>
<li>多线程编程python中有Thread和threading</li>
<li>异步编程在linux下主+要有三种实现select,poll,epoll</li>
<li><p>协程在python中通常会说到yield,关于协程的库主要有greenlet,stackless,gevent,eventlet等实现。</p>
<h3 id="进程"><a href="#进程" class="headerlink" title="进程"></a>进程</h3></li>
<li><p>不共享任何状态</p>
</li>
<li>调度由操作系统完成</li>
<li>有独立的内存空间(上下文切换的时候需要保存栈、cpu寄存器、虚拟内存、以及打开的相关句柄等信息,开销大)</li>
<li><p>通讯主要通过信号传递的方式来实现(实现方式有多种,信号量、管道、事件等,通讯都需要过内核,效率低)</p>
<h3 id="线程"><a href="#线程" class="headerlink" title="线程"></a>线程</h3></li>
<li><p>调度完全由用户控制</p>
</li>
<li>一个线程(进程)可以有多个协程</li>
<li>每个线程(进程)循环按照指定的任务清单顺序完成不同的任务(当任务被堵塞时,执行下一个任务;当恢复时,再回来执行这个任务;任务间切换只需要保存任务的上下文,没有内核的开销,可以不加锁的访问全局变量)</li>
<li>协程需要保证是非堵塞的且没有相互依赖</li>
<li>协程基本上不能同步通讯,多采用异步的消息通讯,效率比较高</li>
</ul>
<h3 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h3><ul>
<li>进程拥有自己独立的堆和栈,既不共享堆,亦不共享栈,进程由操作系统调度</li>
<li>线程拥有自己独立的栈和共享的堆,共享堆,不共享栈,线程亦由操作系统调度(标准线程是的)</li>
<li>协程和线程一样共享堆,不共享栈,协程由程序员在协程的代码里显示调度</li>
</ul>
<h3 id="聊聊协程"><a href="#聊聊协程" class="headerlink" title="聊聊协程"></a>聊聊协程</h3><p>协程,又称微线程,纤程。</p>
<p>Python的线程并不是标准线程,是系统级进程,线程间上下文切换有开销,而且Python在执行多线程时默认加了一个全局解释器锁(GIL),因此Python的多线程其实是串行的,所以并不能利用多核的优势,也就是说一个进程内的多个线程只能使用一个CPU。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">coroutine</span><span class="params">(func)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">ret</span><span class="params">()</span>:</span></span><br><span class="line"> f = func()</span><br><span class="line"> f.next()</span><br><span class="line"> <span class="keyword">return</span> f</span><br><span class="line"> <span class="keyword">return</span> ret</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">consumer</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Wait to getting a task"</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> n = (<span class="keyword">yield</span>)</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Got %s"</span>,n</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">producer</span><span class="params">()</span>:</span></span><br><span class="line"> c = consumer()</span><br><span class="line"> task_id = <span class="number">0</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> time.sleep(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Send a task to consumer"</span> % task_id</span><br><span class="line"> c.send(<span class="string">"task %s"</span> % task_id)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</span><br><span class="line"> producer()</span><br></pre></td></tr></table></figure>
<p>结果:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">Wait to getting a task</span><br><span class="line">Send a task 0 to consumer</span><br><span class="line">Got task 0</span><br><span class="line">Send a task 1 to consumer</span><br><span class="line">Got task 1</span><br><span class="line">Send a task 2 to consumer</span><br><span class="line">Got task 2</span><br><span class="line">...</span><br></pre></td></tr></table></figure>
<p>传统的生产者-消费者模型是一个线程写消息,一个线程取消息,通过锁机制控制队列和等待,但容易死锁。<br>如果改用协程,生产者生产消息后,直接通过yield跳转到消费者开始执行,待消费者执行完毕后,切换回生产者继续生产,效率极高。</p>
<h2 id="Gevent"><a href="#Gevent" class="headerlink" title="Gevent"></a>Gevent</h2><h3 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h3><p>gevent是基于协程的Python网络库。特点:</p>
<ul>
<li>基于libev的快速事件循环(Linux上epoll,FreeBSD上kqueue)。</li>
<li>基于greenlet的轻量级执行单元。</li>
<li>API的概念和Python标准库一致(如事件,队列)。</li>
<li>可以配合socket,ssl模块使用。</li>
<li>能够使用标准库和第三方模块创建标准的阻塞套接字(gevent.monkey)。</li>
<li>默认通过线程池进行DNS查询,也可通过c-are(通过GEVENT_RESOLVER=ares环境变量开启)。</li>
<li>TCP/UDP/HTTP服务器</li>
<li>子进程支持(通过gevent.subprocess)</li>
<li>线程池</li>
</ul>
<h3 id="安装和依赖"><a href="#安装和依赖" class="headerlink" title="安装和依赖"></a>安装和依赖</h3><p>依赖于greenlet library</p>
<p>支持python 2.6+ 、3.3+</p>
<h3 id="核心部分"><a href="#核心部分" class="headerlink" title="核心部分"></a>核心部分</h3><ul>
<li>Greenlets</li>
<li>同步和异步执行</li>
<li>确定性</li>
<li>创建Greenlets</li>
<li>Greenlet状态</li>
<li>程序停止</li>
<li>超时</li>
<li>猴子补丁</li>
</ul>
<h4 id="Greenlets"><a href="#Greenlets" class="headerlink" title="Greenlets"></a>Greenlets</h4><p>gevent中的主要模式, 它是以C扩展模块形式接入Python的轻量级协程。 全部运行在主程序操作系统进程的内部,但它们被程序员协作式地调度。</p>
<p>在任何时刻,只有一个协程在运行。</p>
<p>区别于multiprocessing、threading等提供真正并行构造的库, 这些库轮转使用操作系统调度的进程和线程,是真正的并行</p>
<h4 id="同步和异步执行"><a href="#同步和异步执行" class="headerlink" title="同步和异步执行"></a>同步和异步执行</h4><p>并发的核心思想在于,大的任务可以分解成一系列的子任务,后者可以被调度成 同时执行或异步执行,而不是一次一个地或者同步地执行。两个子任务之间的 切换也就是上下文切换。</p>
<p>在gevent里面,上下文切换是通过yielding来完成的.</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">foo</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'Running in foo'</span>)</span><br><span class="line"> gevent.sleep(<span class="number">0</span>)</span><br><span class="line"> print(<span class="string">'Explicit context switch to foo again'</span>)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">bar</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'Explicit context to bar'</span>)</span><br><span class="line"> gevent.sleep(<span class="number">0</span>)</span><br><span class="line"> print(<span class="string">'Implicit context switch back to bar'</span>)</span><br><span class="line">gevent.joinall([</span><br><span class="line"> gevent.spawn(foo),</span><br><span class="line"> gevent.spawn(bar),</span><br><span class="line">])</span><br></pre></td></tr></table></figure>
<p>执行结果:<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Running <span class="keyword">in</span> foo</span><br><span class="line">Explicit context to bar</span><br><span class="line">Explicit context switch to foo again</span><br><span class="line">Implicit context switch back to bar</span><br></pre></td></tr></table></figure></p>
<p>网络延迟或IO阻塞隐式交出greenlet上下文的执行权。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> select</span><br><span class="line">start = time.time()</span><br><span class="line">tic = <span class="keyword">lambda</span>: <span class="string">'at %1.1f seconds'</span> % (time.time() - start)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">gr1</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'Started Polling: %s'</span> % tic())</span><br><span class="line"> select.select([], [], [], <span class="number">1</span>)</span><br><span class="line"> print(<span class="string">'Ended Polling: %s'</span> % tic())</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">gr2</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'Started Polling: %s'</span> % tic())</span><br><span class="line"> select.select([], [], [], <span class="number">2</span>)</span><br><span class="line"> print(<span class="string">'Ended Polling: %s'</span> % tic())</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">gr3</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">"Hey lets do some stuff while the greenlets poll, %s"</span> % tic())</span><br><span class="line"> gevent.sleep(<span class="number">1</span>)</span><br><span class="line">gevent.joinall([</span><br><span class="line"> gevent.spawn(gr1),</span><br><span class="line"> gevent.spawn(gr2),</span><br><span class="line"> gevent.spawn(gr3),</span><br><span class="line">])</span><br></pre></td></tr></table></figure>
<p>执行结果</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Started Polling: at 0.0 seconds</span><br><span class="line">Started Polling: at 0.0 seconds</span><br><span class="line">Hey lets do some stuff while the greenlets poll, at 0.0 seconds</span><br><span class="line">Ended Polling: at 1.0 seconds</span><br><span class="line">Ended Polling: at 2.0 seconds</span><br></pre></td></tr></table></figure>
<p>同步VS异步</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">task</span><span class="params">(pid)</span>:</span></span><br><span class="line"> gevent.sleep(random.randint(<span class="number">0</span>,<span class="number">2</span>)*<span class="number">0.001</span>)</span><br><span class="line"> print(<span class="string">'Task %s done'</span> % pid)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">synchronous</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> xrange(<span class="number">5</span>):</span><br><span class="line"> task(i)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">asynchronous</span><span class="params">()</span>:</span></span><br><span class="line"> threads = [gevent.spawn(task, i) <span class="keyword">for</span> i <span class="keyword">in</span> xrange(<span class="number">5</span>)]</span><br><span class="line"> gevent.joinall(threads)</span><br><span class="line"> print(<span class="string">'Synchronous:'</span>)</span><br><span class="line"> synchronous()</span><br><span class="line"> print(<span class="string">'Asynchronous:'</span>)</span><br><span class="line"> asynchronous()</span><br></pre></td></tr></table></figure>
<p>执行结果<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">Synchronous:</span><br><span class="line">Task 0 done</span><br><span class="line">Task 1 done</span><br><span class="line">Task 2 done</span><br><span class="line">Task 3 done</span><br><span class="line">Task 4 done</span><br><span class="line">Asynchronous:</span><br><span class="line">Task 2 done</span><br><span class="line">Task 0 done</span><br><span class="line">Task 1 done</span><br><span class="line">Task 3 done</span><br><span class="line">Task 4 done</span><br></pre></td></tr></table></figure></p>
<h4 id="确定性"><a href="#确定性" class="headerlink" title="确定性"></a>确定性</h4><p>greenlet具有确定性。在相同配置相同输入的情况下,它们总是会产生相同的输出。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">echo</span><span class="params">(i)</span>:</span></span><br><span class="line"> time.sleep(<span class="number">0.001</span>)</span><br><span class="line"> <span class="keyword">return</span> i</span><br><span class="line"><span class="comment"># Non Deterministic Process Pool</span></span><br><span class="line"><span class="keyword">from</span> multiprocessing.pool <span class="keyword">import</span> Pool</span><br><span class="line">p = Pool(<span class="number">10</span>)</span><br><span class="line">run1 = [a <span class="keyword">for</span> a <span class="keyword">in</span> p.imap_unordered(echo, xrange(<span class="number">10</span>))]</span><br><span class="line">run2 = [a <span class="keyword">for</span> a <span class="keyword">in</span> p.imap_unordered(echo, xrange(<span class="number">10</span>))]</span><br><span class="line">run3 = [a <span class="keyword">for</span> a <span class="keyword">in</span> p.imap_unordered(echo, xrange(<span class="number">10</span>))]</span><br><span class="line">run4 = [a <span class="keyword">for</span> a <span class="keyword">in</span> p.imap_unordered(echo, xrange(<span class="number">10</span>))]</span><br><span class="line">print(run1 == run2 == run3 == run4)</span><br><span class="line"><span class="comment"># Deterministic Gevent Pool</span></span><br><span class="line"><span class="keyword">from</span> gevent.pool <span class="keyword">import</span> Pool</span><br><span class="line">p = Pool(<span class="number">10</span>)</span><br><span class="line">run1 = [a <span class="keyword">for</span> a <span class="keyword">in</span> p.imap_unordered(echo, xrange(<span class="number">10</span>))]</span><br><span class="line">run2 = [a <span class="keyword">for</span> a <span class="keyword">in</span> p.imap_unordered(echo, xrange(<span class="number">10</span>))]</span><br><span class="line">run3 = [a <span class="keyword">for</span> a <span class="keyword">in</span> p.imap_unordered(echo, xrange(<span class="number">10</span>))]</span><br><span class="line">run4 = [a <span class="keyword">for</span> a <span class="keyword">in</span> p.imap_unordered(echo, xrange(<span class="number">10</span>))]</span><br><span class="line">print(run1 == run2 == run3 == run4)</span><br></pre></td></tr></table></figure>
<p>执行结果:<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">False</span><br><span class="line">True</span><br></pre></td></tr></table></figure></p>
<p>即使gevent通常带有确定性,当开始与如socket或文件等外部服务交互时, 不确定性也可能溜进你的程序中。因此尽管gevent线程是一种“确定的并发”形式, 使用它仍然可能会遇到像使用POSIX线程或进程时遇到的那些问题。</p>
<p>涉及并发长期存在的问题就是竞争条件(race condition)(当两个并发线程/进程都依赖于某个共享资源同时都尝试去修改它的时候, 就会出现竞争条件),这会导致资源修改的结果状态依赖于时间和执行顺序。 这个问题,会导致整个程序行为变得不确定。</p>
<p>解决办法: 始终避免所有全局的状态.</p>
<h4 id="创建Greenlets"><a href="#创建Greenlets" class="headerlink" title="创建Greenlets"></a>创建Greenlets</h4><p>gevent对Greenlet初始化提供了一些封装.</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> Greenlet</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">foo</span><span class="params">(message, n)</span>:</span></span><br><span class="line"> gevent.sleep(n)</span><br><span class="line"> print(message)</span><br><span class="line"> thread1 = Greenlet.spawn(foo, <span class="string">"Hello"</span>, <span class="number">1</span>)</span><br><span class="line"> thread2 = gevent.spawn(foo, <span class="string">"I live!"</span>, <span class="number">2</span>)</span><br><span class="line"> thread3 = gevent.spawn(<span class="keyword">lambda</span> x: (x+<span class="number">1</span>), <span class="number">2</span>)</span><br><span class="line"> threads = [thread1, thread2, thread3]</span><br><span class="line"> gevent.joinall(threads)</span><br></pre></td></tr></table></figure>
<p>执行结果:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Hello</span><br><span class="line">I live!</span><br></pre></td></tr></table></figure>
<p>除使用基本的Greenlet类之外,你也可以子类化Greenlet类,重载它的_run方法。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> Greenlet</span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">MyGreenlet</span><span class="params">(Greenlet)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, message, n)</span>:</span></span><br><span class="line"> Greenlet.__init__(self)</span><br><span class="line"> self.message = message</span><br><span class="line"> self.n = n</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">_run</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(self.message)</span><br><span class="line"> gevent.sleep(self.n)</span><br><span class="line">g = MyGreenlet(<span class="string">"Hi there!"</span>, <span class="number">3</span>)</span><br><span class="line">g.start()</span><br><span class="line">g.join()</span><br></pre></td></tr></table></figure>
<p>执行结果:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Hi there!</span><br></pre></td></tr></table></figure>
<h4 id="Greenlet状态"><a href="#Greenlet状态" class="headerlink" title="Greenlet状态"></a>Greenlet状态</h4><p>greenlet的状态通常是一个依赖于时间的参数:</p>
<ul>
<li>started – Boolean, 指示此Greenlet是否已经启动</li>
<li>ready() – Boolean, 指示此Greenlet是否已经停止</li>
<li>successful() – Boolean, 指示此Greenlet是否已经停止而且没抛异常</li>
<li>value – 任意值, 此Greenlet代码返回的值</li>
<li>exception – 异常, 此Greenlet内抛出的未捕获异常</li>
</ul>
<h4 id="程序停止"><a href="#程序停止" class="headerlink" title="程序停止"></a>程序停止</h4><p>程序<br>当主程序(main program)收到一个SIGQUIT信号时,不能成功做yield操作的 Greenlet可能会令意外地挂起程序的执行。这导致了所谓的僵尸进程, 它需要在Python解释器之外被kill掉。</p>
<p>通用的处理模式就是在主程序中监听SIGQUIT信号,调用gevent.shutdown退出程序。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">import</span> signal</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run_forever</span><span class="params">()</span>:</span></span><br><span class="line"> gevent.sleep(<span class="number">1000</span>)</span><br><span class="line"> <span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> gevent.signal(signal.SIGQUIT, gevent.shutdown)</span><br><span class="line"> thread = gevent.spawn(run_forever)</span><br><span class="line"> thread.join()</span><br></pre></td></tr></table></figure>
<h4 id="超时"><a href="#超时" class="headerlink" title="超时"></a>超时</h4><p>通过超时可以对代码块儿或一个Greenlet的运行时间进行约束。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> Timeout</span><br><span class="line">seconds = <span class="number">10</span></span><br><span class="line">timeout = Timeout(seconds)</span><br><span class="line">timeout.start()</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">wait</span><span class="params">()</span>:</span></span><br><span class="line"> gevent.sleep(<span class="number">10</span>)</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> gevent.spawn(wait).join()</span><br><span class="line"> <span class="keyword">except</span> Timeout:</span><br><span class="line"> print(<span class="string">'Could not complete'</span>)</span><br></pre></td></tr></table></figure>
<p>超时类</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> Timeout</span><br><span class="line">time_to_wait = <span class="number">5</span> <span class="comment"># seconds</span></span><br><span class="line"> <span class="class"><span class="keyword">class</span> <span class="title">TooLong</span><span class="params">(Exception)</span>:</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">with</span> Timeout(time_to_wait, TooLong):</span><br><span class="line"> gevent.sleep(<span class="number">10</span>)</span><br></pre></td></tr></table></figure>
<p>另外,对各种Greenlet和数据结构相关的调用,gevent也提供了超时参数。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> Timeout</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">wait</span><span class="params">()</span>:</span></span><br><span class="line"> gevent.sleep(<span class="number">2</span>)</span><br><span class="line">timer = Timeout(<span class="number">1</span>).start()</span><br><span class="line">thread1 = gevent.spawn(wait)</span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> thread1.join(timeout=timer)</span><br><span class="line"><span class="keyword">except</span> Timeout:</span><br><span class="line"> print(<span class="string">'Thread 1 timed out'</span>)</span><br><span class="line"><span class="comment"># --</span></span><br><span class="line">timer = Timeout.start_new(<span class="number">1</span>)</span><br><span class="line">thread2 = gevent.spawn(wait)</span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> thread2.get(timeout=timer)</span><br><span class="line"><span class="keyword">except</span> Timeout:</span><br><span class="line"> print(<span class="string">'Thread 2 timed out'</span>)</span><br><span class="line"><span class="comment"># --</span></span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> gevent.with_timeout(<span class="number">1</span>, wait)</span><br><span class="line"><span class="keyword">except</span> Timeout:</span><br><span class="line"> print(<span class="string">'Thread 3 timed out'</span>)</span><br></pre></td></tr></table></figure>
<p>执行结果:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Thread 1 timed out</span><br><span class="line">Thread 2 timed out</span><br><span class="line">Thread 3 timed out</span><br></pre></td></tr></table></figure>
<h4 id="猴子补丁-Monkey-patching"><a href="#猴子补丁-Monkey-patching" class="headerlink" title="猴子补丁(Monkey patching)"></a>猴子补丁(Monkey patching)</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> socket</span><br><span class="line">print(socket.socket)</span><br><span class="line">print(<span class="string">"After monkey patch"</span>)</span><br><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> monkey</span><br><span class="line">monkey.patch_socket()</span><br><span class="line">print(socket.socket)</span><br><span class="line"><span class="keyword">import</span> select</span><br><span class="line">print(select.select)</span><br><span class="line">monkey.patch_select()</span><br><span class="line">print(<span class="string">"After monkey patch"</span>)</span><br><span class="line">print(select.select)</span><br></pre></td></tr></table></figure>
<p>执行结果:<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">class 'socket.socket'</span><br><span class="line">After monkey patch</span><br><span class="line">class 'gevent.socket.socket'</span><br><span class="line">built-in function select</span><br><span class="line">After monkey patch</span><br><span class="line">function select at 0x1924de8</span><br></pre></td></tr></table></figure></p>
<p>Python的运行环境允许我们在运行时修改大部分的对象,包括模块,类甚至函数。 这是个一般说来令人惊奇的坏主意,因为它创造了“隐式的副作用”,如果出现问题 它很多时候是极难调试的。虽然如此,在极端情况下当一个库需要修改Python本身 的基础行为的时候,猴子补丁就派上用场了。在这种情况下,gevent能够修改标准库里面大部分的阻塞式系统调用,包括socket、ssl、threading和 select等模块,而变为协作式运行。</p>
<p>例如,Redis的python绑定一般使用常规的tcp socket来与redis-server实例通信。 通过简单地调用gevent.monkey.patch_all(),可以使得redis的绑定协作式的调度 请求,与gevent栈的其它部分一起工作。</p>
<p>这让我们可以将一般不能与gevent共同工作的库结合起来,而不用写哪怕一行代码。 虽然猴子补丁仍然是邪恶的(evil),但在这种情况下它是“有用的邪恶(useful evil)”。</p>
<h3 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h3><ul>
<li>事件</li>
<li>队列</li>
<li>组和池</li>
<li>锁和信号量</li>
<li>线程局部变量</li>
<li>子进程</li>
<li>Actors<h4 id="事件"><a href="#事件" class="headerlink" title="事件"></a>事件</h4></li>
</ul>
<p>事件(event)是一个在Greenlet之间异步通信的形式。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.event <span class="keyword">import</span> Event</span><br><span class="line"> </span><br><span class="line">evt = Event()</span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">setter</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'A: Hey wait for me, I have to do something'</span>)</span><br><span class="line"> gevent.sleep(<span class="number">3</span>)</span><br><span class="line"> print(<span class="string">"Ok, I'm done"</span>)</span><br><span class="line"> evt.set()</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">waiter</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">"I'll wait for you"</span>)</span><br><span class="line"> evt.wait() <span class="comment"># blocking</span></span><br><span class="line"> print(<span class="string">"It's about time"</span>)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"> gevent.joinall([</span><br><span class="line"> gevent.spawn(setter),</span><br><span class="line"> gevent.spawn(waiter),</span><br><span class="line"> gevent.spawn(waiter),</span><br><span class="line"> gevent.spawn(waiter)</span><br><span class="line"> ])</span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>: </span><br><span class="line"> main()</span><br></pre></td></tr></table></figure>
<p>执行结果:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">A: Hey wait for me, I have to do something</span><br><span class="line">I'll wait for you</span><br><span class="line">I'll wait for you</span><br><span class="line">I'll wait for you</span><br><span class="line">Ok, I'm done</span><br><span class="line">It's about time</span><br><span class="line">It's about time</span><br><span class="line">It's about time</span><br></pre></td></tr></table></figure>
<p>事件对象的一个扩展是AsyncResult,它允许你在唤醒调用上附加一个值。 它有时也被称作是future或defered,因为它持有一个指向将来任意时间可设置为任何值的引用。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.event <span class="keyword">import</span> AsyncResult</span><br><span class="line">a = AsyncResult()</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">setter</span><span class="params">()</span>:</span></span><br><span class="line"> gevent.sleep(<span class="number">3</span>)</span><br><span class="line"> a.set(<span class="string">'Hello!'</span>)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">waiter</span><span class="params">()</span>:</span></span><br><span class="line"> print(a.get())</span><br><span class="line">gevent.joinall([</span><br><span class="line"> gevent.spawn(setter),</span><br><span class="line"> gevent.spawn(waiter),</span><br><span class="line">])</span><br></pre></td></tr></table></figure>
<h4 id="队列"><a href="#队列" class="headerlink" title="队列"></a>队列</h4><p>队列是一个排序的数据集合,它有常见的put / get操作, 但是它是以在Greenlet之间可以安全操作的方式来实现的。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.queue <span class="keyword">import</span> Queue</span><br><span class="line">tasks = Queue()</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">worker</span><span class="params">(n)</span>:</span></span><br><span class="line"> <span class="keyword">while</span> <span class="keyword">not</span> tasks.empty():</span><br><span class="line"> task = tasks.get()</span><br><span class="line"> print(<span class="string">'Worker %s got task %s'</span> % (n, task))</span><br><span class="line"> gevent.sleep(<span class="number">0</span>)</span><br><span class="line"> print(<span class="string">'Quitting time!'</span>)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">boss</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> xrange(<span class="number">1</span>,<span class="number">10</span>):</span><br><span class="line"> tasks.put_nowait(i)</span><br><span class="line">gevent.spawn(boss).join()</span><br><span class="line">gevent.joinall([</span><br><span class="line"> gevent.spawn(worker, <span class="string">'steve'</span>),</span><br><span class="line"> gevent.spawn(worker, <span class="string">'john'</span>),</span><br><span class="line"> gevent.spawn(worker, <span class="string">'nancy'</span>),</span><br><span class="line">])</span><br></pre></td></tr></table></figure>
<p>执行结果:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">Worker steve got task 1</span><br><span class="line">Worker john got task 2</span><br><span class="line">Worker nancy got task 3</span><br><span class="line">Worker steve got task 4</span><br><span class="line">Worker john got task 5</span><br><span class="line">Worker nancy got task 6</span><br><span class="line">Worker steve got task 7</span><br><span class="line">Worker john got task 8</span><br><span class="line">Worker nancy got task 9</span><br><span class="line">Quitting time!</span><br><span class="line">Quitting time!</span><br><span class="line">Quitting time!</span><br></pre></td></tr></table></figure>
<p>put和get操作都是阻塞的,put_nowait和get_nowait不会阻塞, 然而在操作不能完成时抛出gevent.queue.Empty或gevent.queue.Full异常。</p>
<h4 id="组和池"><a href="#组和池" class="headerlink" title="组和池"></a>组和池</h4><p>组(group)是一个运行中greenlet集合,集合中的greenlet像一个组一样会被共同管理和调度。 它也兼饰了像Python的multiprocessing库那样的平行调度器的角色,主要用在在管理异步任务的时候进行分组。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.pool <span class="keyword">import</span> Group</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">talk</span><span class="params">(msg)</span>:</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> xrange(<span class="number">2</span>):</span><br><span class="line"> print(msg)</span><br><span class="line">g1 = gevent.spawn(talk, <span class="string">'bar'</span>)</span><br><span class="line">g2 = gevent.spawn(talk, <span class="string">'foo'</span>)</span><br><span class="line">g3 = gevent.spawn(talk, <span class="string">'fizz'</span>)</span><br><span class="line">group = Group()</span><br><span class="line">group.add(g1)</span><br><span class="line">group.add(g2)</span><br><span class="line">group.join()</span><br><span class="line">group.add(g3)</span><br><span class="line">group.join()</span><br></pre></td></tr></table></figure>
<p>执行结果:<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">bar</span><br><span class="line">bar</span><br><span class="line">foo</span><br><span class="line">foo</span><br><span class="line">fizz</span><br><span class="line">fizz</span><br></pre></td></tr></table></figure></p>
<p>池(pool)是一个为处理数量变化并且需要限制并发的greenlet而设计的结构。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.pool <span class="keyword">import</span> Pool</span><br><span class="line">pool = Pool(<span class="number">2</span>)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello_from</span><span class="params">(n)</span>:</span></span><br><span class="line"> print(<span class="string">'Size of pool %s'</span> % len(pool))</span><br><span class="line"> pool.map(hello_from, range(<span class="number">3</span>))</span><br></pre></td></tr></table></figure>
<p>执行结果</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Size of pool 2</span><br><span class="line">Size of pool 2</span><br><span class="line">Size of pool 1</span><br></pre></td></tr></table></figure>
<p>构造一个socket池的类,在各个socket上轮询。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> gevent.pool <span class="keyword">import</span> Pool</span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">SocketPool</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line"> self.pool = Pool(<span class="number">10</span>)</span><br><span class="line"> self.pool.start()</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">listen</span><span class="params">(self, socket)</span>:</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> socket.recv()</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">add_handler</span><span class="params">(self, socket)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> self.pool.full():</span><br><span class="line"> <span class="keyword">raise</span> Exception(<span class="string">"At maximum pool size"</span>)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> self.pool.spawn(self.listen, socket)</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">shutdown</span><span class="params">(self)</span>:</span></span><br><span class="line"> self.pool.kill()</span><br></pre></td></tr></table></figure>
<h4 id="锁和信号量"><a href="#锁和信号量" class="headerlink" title="锁和信号量"></a>锁和信号量</h4><p>信号量是一个允许greenlet相互合作,限制并发访问或运行的低层次的同步原语。 信号量有两个方法,acquire和release。在信号量是否已经被 acquire或release,和拥有资源的数量之间不同,被称为此信号量的范围 (the bound of the semaphore)。如果一个信号量的范围已经降低到0,它会 阻塞acquire操作直到另一个已经获得信号量的greenlet作出释放。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> sleep</span><br><span class="line"><span class="keyword">from</span> gevent.pool <span class="keyword">import</span> Pool</span><br><span class="line"><span class="keyword">from</span> gevent.coros <span class="keyword">import</span> BoundedSemaphore</span><br><span class="line">sem = BoundedSemaphore(<span class="number">2</span>)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">worker1</span><span class="params">(n)</span>:</span></span><br><span class="line"> sem.acquire()</span><br><span class="line"> print(<span class="string">'Worker %i acquired semaphore'</span> % n)</span><br><span class="line"> sleep(<span class="number">0</span>)</span><br><span class="line"> sem.release()</span><br><span class="line"> print(<span class="string">'Worker %i released semaphore'</span> % n)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">worker2</span><span class="params">(n)</span>:</span></span><br><span class="line"> <span class="keyword">with</span> sem:</span><br><span class="line"> print(<span class="string">'Worker %i acquired semaphore'</span> % n)</span><br><span class="line"> sleep(<span class="number">0</span>)</span><br><span class="line"> print(<span class="string">'Worker %i released semaphore'</span> % n)</span><br><span class="line">pool = Pool()</span><br><span class="line">pool.map(worker1, range(<span class="number">0</span>,<span class="number">2</span>))</span><br><span class="line">`</span><br></pre></td></tr></table></figure>
<p>执行结果:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Worker 0 acquired semaphore</span><br><span class="line">Worker 1 acquired semaphore</span><br><span class="line">Worker 0 released semaphore</span><br><span class="line">Worker 1 released semaphore</span><br></pre></td></tr></table></figure>
<p>锁(lock)是范围为1的信号量。它向单个greenlet提供了互斥访问。 信号量和锁常被用来保证资源只在程序上下文被单次使用。</p>
<h4 id="线程局部变量"><a href="#线程局部变量" class="headerlink" title="线程局部变量"></a>线程局部变量</h4><p>Gevent允许程序员指定局部于greenlet上下文的数据。 在内部,它被实现为以greenlet的getcurrent()为键, 在一个私有命名空间寻址的全局查找。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.local <span class="keyword">import</span> local</span><br><span class="line">stash = local()</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">f1</span><span class="params">()</span>:</span></span><br><span class="line"> stash.x = <span class="number">1</span></span><br><span class="line"> print(stash.x)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">f2</span><span class="params">()</span>:</span></span><br><span class="line"> stash.y = <span class="number">2</span></span><br><span class="line"> print(stash.y)</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> stash.x</span><br><span class="line"> <span class="keyword">except</span> AttributeError:</span><br><span class="line"> print(<span class="string">"x is not local to f2"</span>)</span><br><span class="line">g1 = gevent.spawn(f1)</span><br><span class="line">g2 = gevent.spawn(f2)</span><br><span class="line">gevent.joinall([g1, g2])</span><br></pre></td></tr></table></figure>
<p>执行结果:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">x is not local to f2</span><br></pre></td></tr></table></figure>
<p>很多集成了gevent的web框架将HTTP会话对象以线程局部变量的方式存储在gevent内。 例如使用Werkzeug实用库和它的proxy对象,我们可以创建Flask风格的请求对象。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">from</span> gevent.local <span class="keyword">import</span> local</span><br><span class="line"><span class="keyword">from</span> werkzeug.local <span class="keyword">import</span> LocalProxy</span><br><span class="line"><span class="keyword">from</span> werkzeug.wrappers <span class="keyword">import</span> Request</span><br><span class="line"><span class="keyword">from</span> contextlib <span class="keyword">import</span> contextmanager</span><br><span class="line"><span class="keyword">from</span> gevent.wsgi <span class="keyword">import</span> WSGIServer</span><br><span class="line">_requests = local()</span><br><span class="line">request = LocalProxy(<span class="keyword">lambda</span>: _requests.request)</span><br><span class="line"><span class="meta">@contextmanager</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">sessionmanager</span><span class="params">(environ)</span>:</span></span><br><span class="line"> _requests.request = Request(environ)</span><br><span class="line"> <span class="keyword">yield</span></span><br><span class="line"> _requests.request = <span class="literal">None</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">logic</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> <span class="string">"Hello "</span> + request.remote_addr</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">application</span><span class="params">(environ, start_response)</span>:</span></span><br><span class="line"> status = <span class="string">'200 OK'</span></span><br><span class="line"> <span class="keyword">with</span> sessionmanager(environ):</span><br><span class="line"> body = logic()</span><br><span class="line"> headers = [</span><br><span class="line"> (<span class="string">'Content-Type'</span>, <span class="string">'text/html'</span>)</span><br><span class="line"> ]</span><br><span class="line"> start_response(status, headers)</span><br><span class="line"> <span class="keyword">return</span> [body]</span><br><span class="line"> WSGIServer((<span class="string">''</span>, <span class="number">8000</span>), application).serve_forever()</span><br></pre></td></tr></table></figure>
<h4 id="子进程"><a href="#子进程" class="headerlink" title="子进程"></a>子进程</h4><p>从gevent 1.0起,支持gevent.subprocess,支持协作式的等待子进程。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.subprocess <span class="keyword">import</span> Popen, PIPE</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">cron</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> print(<span class="string">"cron"</span>)</span><br><span class="line"> gevent.sleep(<span class="number">0.2</span>)</span><br><span class="line">g = gevent.spawn(cron)</span><br><span class="line">sub = Popen([<span class="string">'sleep 1; uname'</span>], stdout=PIPE, shell=<span class="literal">True</span>)</span><br><span class="line">out, err = sub.communicate()</span><br><span class="line">g.kill()</span><br><span class="line">print(out.rstrip())</span><br><span class="line"></span><br><span class="line">``` </span><br><span class="line"></span><br><span class="line"> 执行结果:</span><br></pre></td></tr></table></figure>
<p>cron<br>cron<br>cron<br>cron<br>cron<br>Linux<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">很多人也想将gevent和multiprocessing一起使用。最明显的挑战之一 就是multiprocessing提供的进程间通信默认不是协作式的。由于基于 multiprocessing.Connection的对象(例如Pipe)暴露了它们下面的 文件描述符(file descriptor),gevent.socket.wait_read和wait_write 可以用来在直接读写之前协作式的等待ready-to-read/ready-to-write事件。</span><br><span class="line"></span><br><span class="line">```python</span><br><span class="line">import gevent</span><br><span class="line">from multiprocessing import Process, Pipe</span><br><span class="line">from gevent.socket import wait_read, wait_write</span><br><span class="line"># To Process</span><br><span class="line">a, b = Pipe()</span><br><span class="line"># From Process</span><br><span class="line">c, d = Pipe()</span><br><span class="line">def relay():</span><br><span class="line"> for i in xrange(5):</span><br><span class="line"> msg = b.recv()</span><br><span class="line"> c.send(msg + " in " + str(i))</span><br><span class="line">def put_msg():</span><br><span class="line"> for i in xrange(5):</span><br><span class="line"> wait_write(a.fileno())</span><br><span class="line"> a.send('hi')</span><br><span class="line">def get_msg():</span><br><span class="line"> for i in xrange(5):</span><br><span class="line"> wait_read(d.fileno())</span><br><span class="line"> print(d.recv())</span><br><span class="line">if __name__ == '__main__':</span><br><span class="line"> proc = Process(target=relay)</span><br><span class="line"> proc.start()</span><br><span class="line"> g1 = gevent.spawn(get_msg)</span><br><span class="line"> g2 = gevent.spawn(put_msg)</span><br><span class="line"> gevent.joinall([g1, g2], timeout=1)</span><br><span class="line">``` </span><br><span class="line"></span><br><span class="line"></span><br><span class="line">执行结果:</span><br><span class="line"></span><br><span class="line">```shell </span><br><span class="line">hi in 0</span><br><span class="line">hi in 1</span><br><span class="line">hi in 2</span><br><span class="line">hi in 3</span><br><span class="line">hi in 4</span><br></pre></td></tr></table></figure></p>
<p>然而要注意,组合multiprocessing和gevent必定带来 依赖于操作系统(os-dependent)的缺陷,其中有:</p>
<p>在兼容POSIX的系统创建子进程(forking)之后, 在子进程的gevent的状态是不适定的(ill-posed)。一个副作用就是, multiprocessing.Process创建之前的greenlet创建动作,会在父进程和子进程两方都运行。</p>
<p>上例的put_msg()中的a.send()可能依然非协作式地阻塞调用的线程:一个 ready-to-write事件只保证写了一个byte。在尝试写完成之前底下的buffer可能是满的。</p>
<p>上面表示的基于wait_write()/wait_read()的方法在Windows上不工作 (IOError: 3 is not a socket (files are not supported)),因为Windows不能监视 pipe事件。</p>
<p>Python包gipc以大体上透明的方式在 兼容POSIX系统和Windows上克服了这些挑战。它提供了gevent感知的基于 multiprocessing.Process的子进程和gevent基于pipe的协作式进程间通信。</p>
<h4 id="Actors"><a href="#Actors" class="headerlink" title="Actors"></a>Actors</h4><p>actor模型是一个由于Erlang变得普及的更高层的并发模型。 简单的说它的主要思想就是许多个独立的Actor,每个Actor有一个可以从 其它Actor接收消息的收件箱。Actor内部的主循环遍历它收到的消息,并根据它期望的行为来采取行动。</p>
<p>Gevent没有原生的Actor类型,但在一个子类化的Greenlet内使用队列, 我们可以定义一个非常简单的。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.queue <span class="keyword">import</span> Queue</span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Actor</span><span class="params">(gevent.Greenlet)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line"> self.inbox = Queue()</span><br><span class="line"> Greenlet.__init__(self)</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">receive</span><span class="params">(self, message)</span>:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> Define in your subclass.</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="keyword">raise</span> <span class="built_in">NotImplemented</span>()</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">_run</span><span class="params">(self)</span>:</span></span><br><span class="line"> self.running = <span class="literal">True</span></span><br><span class="line"> <span class="keyword">while</span> self.running:</span><br><span class="line"> message = self.inbox.get()</span><br><span class="line"> self.receive(message)</span><br></pre></td></tr></table></figure>
<p>下面是一个使用的例子:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> gevent</span><br><span class="line"><span class="keyword">from</span> gevent.queue <span class="keyword">import</span> Queue</span><br><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> Greenlet</span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Pinger</span><span class="params">(Actor)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">receive</span><span class="params">(self, message)</span>:</span></span><br><span class="line"> print(message)</span><br><span class="line"> pong.inbox.put(<span class="string">'ping'</span>)</span><br><span class="line"> gevent.sleep(<span class="number">0</span>)</span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Ponger</span><span class="params">(Actor)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">receive</span><span class="params">(self, message)</span>:</span></span><br><span class="line"> print(message)</span><br><span class="line"> ping.inbox.put(<span class="string">'pong'</span>)</span><br><span class="line"> gevent.sleep(<span class="number">0</span>)</span><br><span class="line">ping = Pinger()</span><br><span class="line">pong = Ponger()</span><br><span class="line">ping.start()</span><br><span class="line">pong.start()</span><br><span class="line">ping.inbox.put(<span class="string">'start'</span>)</span><br><span class="line">gevent.joinall([ping, pong])</span><br></pre></td></tr></table></figure>
<h3 id="实际应用"><a href="#实际应用" class="headerlink" title="实际应用"></a>实际应用</h3><ul>
<li>Gevent ZeroMQ</li>
<li>简单server</li>
<li>WSGI Servers</li>
<li>流式server</li>
<li>Long Polling</li>
<li>Websockets</li>
</ul>
<h4 id="简单server"><a href="#简单server" class="headerlink" title="简单server"></a>简单server</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># On Unix: Access with ``$ nc 127.0.0.1 5000``</span></span><br><span class="line"><span class="comment"># On Window: Access with ``$ telnet 127.0.0.1 5000``</span></span><br><span class="line"><span class="keyword">from</span> gevent.server <span class="keyword">import</span> StreamServer</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">handle</span><span class="params">(socket, address)</span>:</span></span><br><span class="line"> socket.send(<span class="string">"Hello from a telnet!\n"</span>)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>):</span><br><span class="line"> socket.send(str(i) + <span class="string">'\n'</span>)</span><br><span class="line"> socket.close()</span><br><span class="line">server = StreamServer((<span class="string">'127.0.0.1'</span>, <span class="number">5000</span>), handle)</span><br><span class="line">server.serve_forever()</span><br></pre></td></tr></table></figure>
<h4 id="WSGI-Servers-And-Websockets"><a href="#WSGI-Servers-And-Websockets" class="headerlink" title="WSGI Servers And Websockets"></a>WSGI Servers And Websockets</h4><p>Gevent为HTTP内容服务提供了两种WSGI server。从今以后就称为 wsgi和pywsgi:</p>
<ul>
<li>gevent.wsgi.WSGIServer</li>
<li>gevent.pywsgi.WSGIServer<br>glb中使用</li>
</ul>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> click</span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</span><br><span class="line"><span class="keyword">from</span> gevent.pywsgi <span class="keyword">import</span> WSGIServer</span><br><span class="line"><span class="keyword">from</span> geventwebsocket.handler <span class="keyword">import</span> WebSocketHandler</span><br><span class="line"><span class="keyword">import</span> v1</span><br><span class="line"><span class="keyword">from</span> .settings <span class="keyword">import</span> Config</span><br><span class="line"><span class="keyword">from</span> .sockethandler <span class="keyword">import</span> handle_websocket</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">create_app</span><span class="params">(config=None)</span>:</span></span><br><span class="line"> app = Flask(__name__, static_folder=<span class="string">'static'</span>)</span><br><span class="line"> <span class="keyword">if</span> config:</span><br><span class="line"> app.config.update(config)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> app.config.from_object(Config)</span><br><span class="line"> app.register_blueprint(</span><br><span class="line"> v1.bp,</span><br><span class="line"> url_prefix=<span class="string">'/v1'</span>)</span><br><span class="line"> <span class="keyword">return</span> app</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">wsgi_app</span><span class="params">(environ, start_response)</span>:</span></span><br><span class="line"> path = environ[<span class="string">'PATH_INFO'</span>]</span><br><span class="line"> <span class="keyword">if</span> path == <span class="string">'/websocket'</span>:</span><br><span class="line"> handle_websocket(environ[<span class="string">'wsgi.websocket'</span>])</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">return</span> create_app()(environ, start_response)</span><br><span class="line"><span class="meta">@click.command()</span></span><br><span class="line"><span class="meta">@click.option('-h', '--host_port', type=(unicode, int),</span></span><br><span class="line"> default=(<span class="string">'0.0.0.0'</span>, <span class="number">5000</span>), help=<span class="string">'Host and port of server.'</span>)</span><br><span class="line"><span class="meta">@click.option('-r', '--redis', type=(unicode, int, int),</span></span><br><span class="line"> default=(<span class="string">'127.0.0.1'</span>, <span class="number">6379</span>, <span class="number">0</span>),</span><br><span class="line"> help=<span class="string">'Redis url of server.'</span>)</span><br><span class="line"><span class="meta">@click.option('-p', '--port_range', type=(int, int),</span></span><br><span class="line"> default=(<span class="number">50000</span>, <span class="number">61000</span>),</span><br><span class="line"> help=<span class="string">'Port range to be assigned.'</span>)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">manage</span><span class="params">(host_port, redis=None, port_range=None)</span>:</span></span><br><span class="line"> Config.REDIS_URL = <span class="string">'redis://%s:%s/%s'</span> % redis</span><br><span class="line"> Config.PORT_RANGE = port_range</span><br><span class="line"> http_server = WSGIServer(host_port,</span><br><span class="line"> wsgi_app, handler_class=WebSocketHandler)</span><br><span class="line"> print(<span class="string">'----GLB Server run at %s:%s-----'</span> % host_port)</span><br><span class="line"> print(<span class="string">'----Redis Server run at %s:%s:%s-----'</span> % redis)</span><br><span class="line"> http_server.serve_forever()</span><br></pre></td></tr></table></figure>
<h3 id="缺陷"><a href="#缺陷" class="headerlink" title="缺陷"></a>缺陷</h3><p>和其他异步I/O框架一样,gevent也有一些缺陷:</p>
<p>阻塞(真正的阻塞,在内核级别)在程序中的某个地方停止了所有的东西.这很像C代码中monkey patch没有生效<br>保持CPU处于繁忙状态.greenlet不是抢占式的,这可能导致其他greenlet不会被调度.<br>在greenlet之间存在死锁的可能.<br>一个gevent回避的缺陷是,你几乎不会碰到一个和异步无关的Python库–它将阻塞你的应用程序,因为纯Python库使用的是monkey patch的stdlib.</p>
</div>
<div class="article-info article-info-index">
<div class="article-tag tagcloud">
<i class="icon-price-tags icon"></i>
<ul class="article-tag-list">
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">python</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">Gevent</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color5">异步编程</a>
</li>
</ul>
</div>
<p class="article-more-link">
<a class="article-more-a" href="/Ranger/2019/07/28/Gevent-简明教程/">展开全文 >></a>
</p>
<div class="clearfix"></div>
</div>
</div>
</article>
<aside class="wrap-side-operation">
<div class="mod-side-operation">
<div class="jump-container" id="js-jump-container" style="display:none;">
<a href="javascript:void(0)" class="mod-side-operation__jump-to-top">
<i class="icon-font icon-back"></i>
</a>
<div id="js-jump-plan-container" class="jump-plan-container" style="top: -11px;">
<i class="icon-font icon-plane jump-plane"></i>
</div>
</div>
</div>
</aside>
<article id="post-PyZMQ请求应答模式-观察者模式" class="article article-type-post article-index" itemscope itemprop="blogPost">
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/Ranger/2019/06/22/PyZMQ请求应答模式-观察者模式/">PyZMQ请求应答模式-观察者模式</a>
</h1>
<a href="/Ranger/2019/06/22/PyZMQ请求应答模式-观察者模式/" class="archive-article-date">
<time datetime="2019-06-22T14:21:41.000Z" itemprop="datePublished"><i class="icon-calendar icon"></i>2019-06-22</time>
</a>
</header>
<div class="article-entry" itemprop="articleBody">
<p>前面讲过zmq的device,用来充当客户端与服务端的中间件,以增加灵活性,让服务端也变成可插拔。然而device是zmq封装好的,怎样才能一窥内部的数据流呢?看下图<br><img src="/Ranger/2019/06/22/PyZMQ请求应答模式-观察者模式/monitored_queue.png" alt="请求应答模式示意图"></p>
<p>一看这图就明白了,MonitoredQueue在创建Queue同时,还提供第3个PUB socket来发布途经这个Queue的进出信息。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> zmq</span><br><span class="line"><span class="keyword">from</span> zmq.devices.basedevice <span class="keyword">import</span> ProcessDevice</span><br><span class="line"><span class="keyword">from</span> zmq.devices.monitoredqueuedevice <span class="keyword">import</span> MonitoredQueue</span><br><span class="line"><span class="keyword">from</span> zmq.utils.strtypes <span class="keyword">import</span> asbytes</span><br><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"> </span><br><span class="line">frontend_port = <span class="number">5559</span> </span><br><span class="line">backend_port = <span class="number">5560</span> </span><br><span class="line">monitor_port = <span class="number">5562</span> </span><br><span class="line">number_of_workers = <span class="number">2</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">monitordevice</span><span class="params">()</span>:</span> </span><br><span class="line"> in_prefix=asbytes(<span class="string">'in'</span>) </span><br><span class="line"> out_prefix=asbytes(<span class="string">'out'</span>) </span><br><span class="line"> monitoringdevice = MonitoredQueue(zmq.XREP, zmq.XREQ, zmq.PUB, in_prefix, out_prefix) </span><br><span class="line"> </span><br><span class="line"> monitoringdevice.bind_in(<span class="string">"tcp://127.0.0.1:%d"</span> % frontend_port) </span><br><span class="line"> monitoringdevice.bind_out(<span class="string">"tcp://127.0.0.1:%d"</span> % backend_port) </span><br><span class="line"> monitoringdevice.bind_mon(<span class="string">"tcp://127.0.0.1:%d"</span> % monitor_port) </span><br><span class="line"> </span><br><span class="line"> monitoringdevice.setsockopt_in(zmq.RCVHWM, <span class="number">1</span>) <span class="comment">#对进入socket的消息设置高水位</span></span><br><span class="line"> monitoringdevice.setsockopt_out(zmq.SNDHWM, <span class="number">1</span>) <span class="comment">#对向外发送的消息设置高水位(最大缓存量)</span></span><br><span class="line"> monitoringdevice.start() </span><br><span class="line"> print(<span class="string">"Program: Monitoring device has started"</span>)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">server</span><span class="params">(backend_port)</span>:</span> </span><br><span class="line"> print(<span class="string">"Program: Server connecting to device"</span>) </span><br><span class="line"> context = zmq.Context() </span><br><span class="line"> socket = context.socket(zmq.REP) </span><br><span class="line"> socket.connect(<span class="string">"tcp://127.0.0.1:%s"</span> % backend_port) </span><br><span class="line"> server_id = random.randrange(<span class="number">1</span>,<span class="number">10005</span>) </span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>: </span><br><span class="line"> message = socket.recv() </span><br><span class="line"> print(<span class="string">"Server: Received - %s"</span> % message) </span><br><span class="line"> socket.send_string(<span class="string">"Response from server #%s"</span> % server_id)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">client</span><span class="params">(frontend_port, client_id)</span>:</span> </span><br><span class="line"> print(<span class="string">"Program: Worker #%s connecting to device"</span> % client_id)</span><br><span class="line"> context = zmq.Context() </span><br><span class="line"> socket = context.socket(zmq.REQ) </span><br><span class="line"> socket.connect(<span class="string">"tcp://127.0.0.1:%s"</span> % frontend_port) </span><br><span class="line"> request_num = <span class="number">1</span> </span><br><span class="line"> socket.send_string(<span class="string">"Request #%s from client#%s"</span> % (request_num, client_id)) </span><br><span class="line"> <span class="comment"># Get the reply. </span></span><br><span class="line"> message = socket.recv_multipart() </span><br><span class="line"> print(<span class="string">"Client: Received - %s"</span> % message)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">monitor</span><span class="params">()</span>:</span> </span><br><span class="line"> print(<span class="string">"Starting monitoring process"</span>) </span><br><span class="line"> context = zmq.Context() </span><br><span class="line"> socket = context.socket(zmq.SUB) </span><br><span class="line"> print( <span class="string">"Collecting updates from server..."</span>) </span><br><span class="line"> socket.connect(<span class="string">"tcp://127.0.0.1:%s"</span> % monitor_port) </span><br><span class="line"> socket.setsockopt(zmq.SUBSCRIBE, <span class="string">b""</span>) </span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>: </span><br><span class="line"> string = socket.recv_multipart() </span><br><span class="line"> print(<span class="string">"Monitoring Client: %s"</span> % string)</span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> monitoring_p = Process(target=monitordevice) </span><br><span class="line"> monitoring_p.start() </span><br><span class="line"> server_p = Process(target=server, args=(backend_port,)) </span><br><span class="line"> server_p.start() </span><br><span class="line"> monitorclient_p = Process(target=monitor) </span><br><span class="line"> monitorclient_p.start() </span><br><span class="line"> time.sleep(<span class="number">2</span>) </span><br><span class="line"> </span><br><span class="line"> <span class="keyword">for</span> client_id <span class="keyword">in</span> range(number_of_workers): </span><br><span class="line"> Process(target=client, args=(frontend_port, client_id,)).start()</span><br><span class="line"></span><br><span class="line"> time.sleep(<span class="number">10</span>)</span><br><span class="line"> server_p.terminate()</span><br><span class="line"> monitorclient_p.terminate()</span><br><span class="line"> monitoring_p.terminate()</span><br></pre></td></tr></table></figure>
</div>
<div class="article-info article-info-index">
<div class="article-tag tagcloud">
<i class="icon-price-tags icon"></i>
<ul class="article-tag-list">
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">python</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color4">ZMQ</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color1">PyZMQ</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">请求应答模式</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color1">观察者模式</a>
</li>
</ul>
</div>
<p class="article-more-link">
<a class="article-more-a" href="/Ranger/2019/06/22/PyZMQ请求应答模式-观察者模式/">展开全文 >></a>
</p>
<div class="clearfix"></div>
</div>
</div>
</article>
<aside class="wrap-side-operation">
<div class="mod-side-operation">
<div class="jump-container" id="js-jump-container" style="display:none;">
<a href="javascript:void(0)" class="mod-side-operation__jump-to-top">
<i class="icon-font icon-back"></i>
</a>
<div id="js-jump-plan-container" class="jump-plan-container" style="top: -11px;">
<i class="icon-font icon-plane jump-plane"></i>
</div>
</div>
</div>
</aside>
<article id="post-PyZMQ一种可靠的请求应答模式-超时重试模式" class="article article-type-post article-index" itemscope itemprop="blogPost">
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/Ranger/2019/06/22/PyZMQ一种可靠的请求应答模式-超时重试模式/">PyZMQ一种可靠的请求应答模式--超时重试模式</a>
</h1>
<a href="/Ranger/2019/06/22/PyZMQ一种可靠的请求应答模式-超时重试模式/" class="archive-article-date">
<time datetime="2019-06-22T13:35:03.000Z" itemprop="datePublished"><i class="icon-calendar icon"></i>2019-06-22</time>
</a>
</header>
<div class="article-entry" itemprop="articleBody">
<p> 提升REQ/REP模式的客户端侧可靠性,由于在此模式下,客户端与服务端严格遵循你一下我一下的乒乓规则,当然现实中不可能这么配合,比如服务器突然挂了,客户端就会阻塞在socket.recv(),然后过段时间,服务器又ok了,客户端也自动重连。但是有的场景客户端需要及时知道服务端状况而不是在那一直等,需要返回个错误之类。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">### 请求端</span></span><br><span class="line"><span class="keyword">import</span> sys </span><br><span class="line"><span class="keyword">import</span> zmq </span><br><span class="line"> </span><br><span class="line">REQUEST_TIMEOUT = <span class="number">2500</span> </span><br><span class="line">REQUEST_RETRIES = <span class="number">3</span> </span><br><span class="line">SERVER_ENDPOINT = <span class="string">"tcp://localhost:5555"</span> </span><br><span class="line"> </span><br><span class="line">context = zmq.Context(<span class="number">1</span>) </span><br><span class="line"> </span><br><span class="line">prin(<span class="string">"I: Connecting to server…"</span>) </span><br><span class="line">client = context.socket(zmq.REQ) </span><br><span class="line">client.connect(SERVER_ENDPOINT) </span><br><span class="line"> </span><br><span class="line">poll = zmq.Poller() </span><br><span class="line">poll.register(client, zmq.POLLIN) </span><br><span class="line"> </span><br><span class="line">sequence = <span class="number">0</span> </span><br><span class="line">retries_left = REQUEST_RETRIES </span><br><span class="line"><span class="keyword">while</span> retries_left: </span><br><span class="line"> sequence += <span class="number">1</span> </span><br><span class="line"> request = str(sequence) </span><br><span class="line"> print(<span class="string">"I: Sending (%s)"</span> % request)</span><br><span class="line"> client.send(request) </span><br><span class="line"> </span><br><span class="line"> expect_reply = <span class="literal">True</span> </span><br><span class="line"> <span class="keyword">while</span> expect_reply: </span><br><span class="line"> socks = dict(poll.poll(REQUEST_TIMEOUT)) </span><br><span class="line"> <span class="keyword">if</span> socks.get(client) == zmq.POLLIN: </span><br><span class="line"> reply = client.recv() </span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> reply: </span><br><span class="line"> <span class="keyword">break</span> </span><br><span class="line"> <span class="keyword">if</span> int(reply) == sequence: </span><br><span class="line"> prin(<span class="string">"I: Server replied OK (%s)"</span> % reply) </span><br><span class="line"> retries_left = REQUEST_RETRIES </span><br><span class="line"> expect_reply = <span class="literal">False</span> </span><br><span class="line"> <span class="keyword">else</span>: </span><br><span class="line"> prin(<span class="string">"E: Malformed reply from server: %s"</span> % reply)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">else</span>: </span><br><span class="line"> print(<span class="string">"W: No response from server, retrying…"</span>) </span><br><span class="line"> <span class="comment"># Socket is confused. Close and remove it. </span></span><br><span class="line"> client.setsockopt(zmq.LINGER, <span class="number">0</span>) </span><br><span class="line"> client.close() </span><br><span class="line"> poll.unregister(client) </span><br><span class="line"> retries_left -= <span class="number">1</span> </span><br><span class="line"> <span class="keyword">if</span> retries_left == <span class="number">0</span>: </span><br><span class="line"> print(<span class="string">"E: Server seems to be offline, abandoning"</span>) </span><br><span class="line"> <span class="keyword">break</span> </span><br><span class="line"> print(<span class="string">"I: Reconnecting and resending (%s)"</span> % request) </span><br><span class="line"> <span class="comment"># Create new connection </span></span><br><span class="line"> client = context.socket(zmq.REQ) </span><br><span class="line"> client.connect(SERVER_ENDPOINT) </span><br><span class="line"> poll.register(client, zmq.POLLIN) </span><br><span class="line"> client.send(request) </span><br><span class="line"> </span><br><span class="line">context.term()</span><br></pre></td></tr></table></figure>
<p>重点是:使用poll.poll(REQUEST_TIMEOUT),来执行一个限时读操作,如果时间段内该REQ socket没收到数据,视作服务端故障,这时就尝试先关闭当前socket,然后重连服务端,重发request。</p>
<p>另外,注意client.setsockopt(zmq.LINGER, 0)设置linger为0,即马上丢掉socket中未处理的消息,否则context,term()啥时能执行完就不知道了。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 服务于端</span></span><br><span class="line"><span class="comment">#coding=utf-8 </span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">from</span> random <span class="keyword">import</span> randint </span><br><span class="line"><span class="keyword">import</span> time </span><br><span class="line"><span class="keyword">import</span> zmq </span><br><span class="line"> </span><br><span class="line">context = zmq.Context(<span class="number">1</span>) </span><br><span class="line">server = context.socket(zmq.REP) </span><br><span class="line">server.bind(<span class="string">"tcp://*:5555"</span>) </span><br><span class="line"> </span><br><span class="line">cycles = <span class="number">0</span> </span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>: </span><br><span class="line"> request = server.recv() </span><br><span class="line"> cycles += <span class="number">1</span> </span><br><span class="line"> <span class="comment"># Simulate various problems, after a few cycles </span></span><br><span class="line"> <span class="keyword">if</span> cycles > <span class="number">3</span> <span class="keyword">and</span> randint(<span class="number">0</span>, <span class="number">3</span>) == <span class="number">0</span>: </span><br><span class="line"> print(<span class="string">"I: Simulating a crash"</span>) </span><br><span class="line"> <span class="keyword">break</span> </span><br><span class="line"> <span class="keyword">elif</span> cycles > <span class="number">3</span> <span class="keyword">and</span> randint(<span class="number">0</span>, <span class="number">3</span>) == <span class="number">0</span>: </span><br><span class="line"> print(<span class="string">"I: Simulating CPU overload"</span>)</span><br><span class="line"> time.sleep(<span class="number">2</span>) </span><br><span class="line"> <span class="keyword">break</span> </span><br><span class="line"> print(<span class="string">"I: Normal request (%s)"</span> % request) </span><br><span class="line"> time.sleep(<span class="number">1</span>) <span class="comment"># Do some heavy work </span></span><br><span class="line"> server.send(request) </span><br><span class="line"> </span><br><span class="line">server.close() </span><br><span class="line">context.term()</span><br></pre></td></tr></table></figure>
</div>
<div class="article-info article-info-index">
<div class="article-tag tagcloud">
<i class="icon-price-tags icon"></i>
<ul class="article-tag-list">
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">python</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color4">ZMQ</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color1">PyZMQ</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color4">可靠性</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">请求应答模式</a>
</li>
</ul>
</div>
<p class="article-more-link">
<a class="article-more-a" href="/Ranger/2019/06/22/PyZMQ一种可靠的请求应答模式-超时重试模式/">展开全文 >></a>
</p>
<div class="clearfix"></div>
</div>
</div>
</article>
<aside class="wrap-side-operation">
<div class="mod-side-operation">
<div class="jump-container" id="js-jump-container" style="display:none;">
<a href="javascript:void(0)" class="mod-side-operation__jump-to-top">
<i class="icon-font icon-back"></i>
</a>
<div id="js-jump-plan-container" class="jump-plan-container" style="top: -11px;">
<i class="icon-font icon-plane jump-plane"></i>
</div>
</div>
</div>
</aside>
<article id="post-ZeroMQ简介" class="article article-type-post article-index" itemscope itemprop="blogPost">
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/Ranger/2019/06/22/ZeroMQ简介/">ZeroMQ简介</a>
</h1>
<a href="/Ranger/2019/06/22/ZeroMQ简介/" class="archive-article-date">
<time datetime="2019-06-22T11:14:18.000Z" itemprop="datePublished"><i class="icon-calendar icon"></i>2019-06-22</time>
</a>
</header>
<div class="article-entry" itemprop="articleBody">
<h3 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h3><p>引用官方的说法: “ZMQ (以下 ZeroMQ 简称 ZMQ)是一个简单好用的传输层,像框架一样的一个 socket library,他使得 Socket 编程更加简单、简洁和性能更高。是一个消息处理队列库,可在多个线程、内核和主机盒之间弹性伸缩。ZMQ 的明确目标是“成为标准网络协议栈的一部分,之后进入 Linux 内核”。现在还未看到它们的成功。但是,它无疑是极具前景的、并且是人们更加需要的“传统”BSD 套接字之上的一层封装。ZMQ 让编写高性能网络应用程序极为简单和有趣。”</p>
<p>近几年有关”Message Queue”的项目层出不穷,知名的就有十几种,这主要是因为后摩尔定律时代,分布式处理逐渐成为主流,业界需要一套标准来解决分布式计算环境中节点之间的消息通信。几年的竞争下来,Apache 基金会旗下的符合 AMQP/1.0标准的 RabbitMQ 已经得到了广泛的认可,成为领先的 MQ 项目。</p>
<p>与 RabbitMQ 相比,ZMQ 并不像是一个传统意义上的消息队列服务器,事实上,它也根本不是一个服务器,它更像是一个底层的网络通讯库,在 Socket API 之上做了一层封装,将网络通讯、进程通讯和线程通讯抽象为统一的 API 接口。</p>
<h3 id="消息模型"><a href="#消息模型" class="headerlink" title="消息模型"></a>消息模型</h3><p>ZMQ 提供了三个基本的通信模型,分别是<code>Request-Reply</code>,<code>Publisher-Subscriber</code>,<code>Parallel Pipeline</code>,我们从这三种模式一窥 ZMQ 的究竟</p>
<h4 id="请求应答模式"><a href="#请求应答模式" class="headerlink" title="请求应答模式"></a>请求应答模式</h4><p>由 Client 发起请求,并等待 Server 回应请求。请求端发送一个简单的 hello,服务端则回应一个 world。请求端和服务端都可以是 1:N 的模型。通常把 1 认为是 Server ,N 认为是 Client 。ZMQ 可以很好的支持路由功能(实现路由功能的组件叫作 Device),把 1:N 扩展为N:M (只需要加入若干路由节点。如下图</p>
<p><img src="/Ranger/2019/06/22/ZeroMQ简介/rep-req.jpeg" alt="请求应答模式示意图"></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#响应端代码</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"><span class="keyword">import</span> zmq</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line">context = zmq.Context()</span><br><span class="line">socket = context.socket(zmq.REP)</span><br><span class="line">socket.bind(<span class="string">"tcp://*:5555"</span>)</span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> message = socket.recv()</span><br><span class="line"> print(<span class="string">"Received request:%s"</span>, %(message))</span><br><span class="line"> time.sleep (<span class="number">1</span>) </span><br><span class="line"> socket.send(time_utils.get_format_timestamp()+<span class="string">" World"</span>)</span><br></pre></td></tr></table></figure>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#请求端代码</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"><span class="keyword">import</span> zmq</span><br><span class="line">context = zmq.Context()</span><br><span class="line">socket = context.socket(zmq.REQ)</span><br><span class="line">socket.connect (<span class="string">"tcp://localhost:5555"</span>)</span><br><span class="line"><span class="keyword">for</span> request <span class="keyword">in</span> range (<span class="number">1</span>,<span class="number">10</span>):</span><br><span class="line"> socket.send (<span class="string">"Hello"</span>)</span><br><span class="line"> message = socket.recv()</span><br><span class="line"> print(<span class="string">"Received reply :%s"</span>%(message))</span><br></pre></td></tr></table></figure>
<blockquote>
<ol>
<li>服务端和客户端无论谁先启动,效果是相同的,这点不同于 Socket。</li>
<li>在服务端收到信息以前,程序是阻塞的,会一直等待客户端连接上来。</li>
<li>服务端收到信息以后,会 send 一个“World”给客户端。值得注意的是一定是 client 连接上来以后,send 消息给 Server,然后 Server 再 rev 然后响应 client,这种一问一答式的。如果 Server 先 send,client 先 rev 是会报错的。</li>
<li>ZMQ 通信通信单元是消息,他除了知道 Bytes 的大小,他并不关心的消息格式。因此,你可以使用任何你觉得好用的数据格式。Xml、Protocol Buffers、Thrift、json 等等。<br>虽然可以使用 ZMQ 实现 HTTP 协议,但是,这绝不是他所擅长的。</li>
<li>通常bind为绑定本地的端口,而connect为连接远程端口,不一定是服务端绑定,客户端连接,至于谁绑定谁连接要根据实际情况来决定。</li>
<li>请求发出必须有应答不然会报错,相应端尚未完成一个请求响应不能接受第二请求</li>
</ol>
</blockquote>
<h3 id="发布订阅模型"><a href="#发布订阅模型" class="headerlink" title="发布订阅模型"></a>发布订阅模型</h3><p>我们可以想象一下天气预报的订阅模式,由一个节点提供信息源,由其他的节点,接受信息源的信息</p>
<p><img src="/Ranger/2019/06/22/ZeroMQ简介/pub-sub.jpeg" alt="请求应答模式示意图"></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#发布端代码</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"><span class="keyword">import</span> zmq</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">from</span> common <span class="keyword">import</span> time_utils</span><br><span class="line">context = zmq.Context()</span><br><span class="line">socket = context.socket(zmq.PUB)</span><br><span class="line">socket.bind(<span class="string">"tcp://*:5555"</span>)</span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> message = time_utils.get_format_timestamp()+<span class="string">":Publish message"</span></span><br><span class="line"> print(<span class="string">"Publish Message:%s "</span>, %(message))</span><br><span class="line"> time.sleep (<span class="number">1</span>)</span><br><span class="line"> socket.send(message)</span><br></pre></td></tr></table></figure>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> zmq</span><br><span class="line">context = zmq.Context()</span><br><span class="line">socket = context.socket(zmq.SUB)</span><br><span class="line">socket.connect (<span class="string">"tcp://localhost:5555"</span>)</span><br><span class="line">socket.setsockopt(zmq.SUBSCRIBE, <span class="string">''</span>)</span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> message = socket.recv()</span><br><span class="line"> print(<span class="string">"Receive Message:%s"</span>+%(message))</span><br></pre></td></tr></table></figure>
<blockquote>
<ol>
<li>与请求应答模式不同的是Socket 的类型变成 SOCKET_PUB 和 SOCKET_SUB 类型。客户端需要$subscriber->setSockOpt (ZMQ::SOCKOPT_SUBSCRIBE, $filter);设置一个过滤值,相当于设定一个订阅频道,否则什么信息也收不到。</li>
<li>服务器端一直不断的广播中,如果中途有 Subscriber 端退出,并不影响他继续的广播,当 Subscriber 再连接上来的时候,收到的就是后来发送的新的信息了。这对比较晚加入的,或者是中途离开的订阅者,必然会丢失掉一部分信息。<br>但是,如果 Publisher 中途离开,所有的 Subscriber 会阻塞住,等待 Publisher 再上线的时候,会继续接受信息。</li>
</ol>
</blockquote>
<h3 id="管道模型"><a href="#管道模型" class="headerlink" title="管道模型"></a>管道模型</h3><p>这个是 PUSH/PULL 模式,又叫做pipeline管道模式,取其一去不回头之意,一推一拉,数据滚滚向前。<br>这种socket封装的原本用意呢,是把数据交给一组worker端干活,PUSH会把任务均匀的(这个好像是zmq的招牌)的分配给下游的worker们,保证大家都有活干,如下图:<br><img src="/Ranger/2019/06/22/ZeroMQ简介/push-pull.jpeg" alt="请求应答模式示意图"></p>
<blockquote>
<p>Producer:产生代处理数据,并将数据push出去</p>
</blockquote>
<blockquote>
<p>consumer(worker):pull到来自producer的数据,处理后,push出去</p>
</blockquote>
<blockquote>
<p>resultcolltor:pull到consumer的数据处理结果</p>
</blockquote>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 生产者</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> time </span><br><span class="line"><span class="keyword">import</span> zmq </span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">producer</span><span class="params">()</span>:</span> </span><br><span class="line"> context = zmq.Context() </span><br><span class="line"> zmq_socket = context.socket(zmq.PUSH) </span><br><span class="line"> zmq_socket.bind(<span class="string">"tcp://127.0.0.1:5557"</span>) </span><br><span class="line"> <span class="keyword">for</span> num <span class="keyword">in</span> xrange(<span class="number">2000</span>): </span><br><span class="line"> work_message = { <span class="string">'num'</span> : num } </span><br><span class="line"> zmq_socket.send_json(work_message) </span><br><span class="line">producer()</span><br></pre></td></tr></table></figure>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 消费者</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> time </span><br><span class="line"><span class="keyword">import</span> zmq </span><br><span class="line"><span class="keyword">import</span> random </span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">consumer</span><span class="params">()</span>:</span> </span><br><span class="line"> consumer_id = random.randrange(<span class="number">1</span>,<span class="number">10005</span>) </span><br><span class="line"> <span class="keyword">print</span> <span class="string">"I am consumer #%s"</span> % (consumer_id) </span><br><span class="line"> context = zmq.Context() </span><br><span class="line"> <span class="comment"># recieve work </span></span><br><span class="line"> consumer_receiver = context.socket(zmq.PULL) </span><br><span class="line"> consumer_receiver.connect(<span class="string">"tcp://127.0.0.1:5557"</span>) </span><br><span class="line"> <span class="comment"># send work </span></span><br><span class="line"> consumer_sender = context.socket(zmq.PUSH) </span><br><span class="line"> consumer_sender.connect(<span class="string">"tcp://127.0.0.1:5558"</span>) </span><br><span class="line"> </span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>: </span><br><span class="line"> work = consumer_receiver.recv_json() </span><br><span class="line"> data = work[<span class="string">'num'</span>] </span><br><span class="line"> result = { <span class="string">'consumer'</span> : consumer_id, <span class="string">'num'</span> : data} </span><br><span class="line"> <span class="keyword">if</span> data%<span class="number">2</span> == <span class="number">0</span>: </span><br><span class="line"> consumer_sender.send_json(result) </span><br><span class="line">consumer()</span><br></pre></td></tr></table></figure>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#汇总收集者</span></span><br><span class="line"><span class="keyword">import</span> time </span><br><span class="line"><span class="keyword">import</span> zmq </span><br><span class="line"><span class="keyword">import</span> pprint </span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">result_collector</span><span class="params">()</span>:</span> </span><br><span class="line"> context = zmq.Context() </span><br><span class="line"> results_receiver = context.socket(zmq.PULL) </span><br><span class="line"> results_receiver.bind(<span class="string">"tcp://127.0.0.1:5558"</span>) </span><br><span class="line"> collecter_data = {} </span><br><span class="line"> <span class="keyword">for</span> x <span class="keyword">in</span> xrange(<span class="number">1000</span>): </span><br><span class="line"> result = results_receiver.recv_json() </span><br><span class="line"> <span class="keyword">if</span> collecter_data.has_key(result[<span class="string">'consumer'</span>]): </span><br><span class="line"> collecter_data[result[<span class="string">'consumer'</span>]] = collecter_data[result[<span class="string">'consumer'</span>]] + <span class="number">1</span> </span><br><span class="line"> <span class="keyword">else</span>: </span><br><span class="line"> collecter_data[result[<span class="string">'consumer'</span>]] = <span class="number">1</span> </span><br><span class="line"> <span class="keyword">if</span> x == <span class="number">999</span>: </span><br><span class="line"> pprint.pprint(collecter_data) </span><br><span class="line"> </span><br><span class="line">result_collector()</span><br></pre></td></tr></table></figure>
<blockquote>
<p>生产者 使用的是 SOCKET_PUSH,将任务分给消费者节点上。而消费者使用 SOCKET_PULL 从上游接受任务,并使用 SOCKET_PUSH 将结果汇集到 收集者。值得注意的是,任务的分发的时候也同样有一个负载均衡的路由功能,消费者可以随时自由加入,生产者可以均衡将任务分发出去。</p>
</blockquote>
<blockquote>
<p>与发布订阅不同的是,生产者发布出去消息如果没有被即使消费,那么会直接保存在缓存中,直到消费者上线将消息消费了。但是如果长期没有消费者消费消息,当达到一定的标准是,生产者会将最开始的数据丢弃,整个消息的消费遵循先进先出原则</p>
</blockquote>
</div>
<div class="article-info article-info-index">
<div class="article-tag tagcloud">
<i class="icon-price-tags icon"></i>
<ul class="article-tag-list">
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color4">ZMQ</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">ZeroMQ</a>
</li>
</ul>
</div>
<p class="article-more-link">
<a class="article-more-a" href="/Ranger/2019/06/22/ZeroMQ简介/">展开全文 >></a>
</p>
<div class="clearfix"></div>
</div>
</div>
</article>
<aside class="wrap-side-operation">
<div class="mod-side-operation">
<div class="jump-container" id="js-jump-container" style="display:none;">
<a href="javascript:void(0)" class="mod-side-operation__jump-to-top">
<i class="icon-font icon-back"></i>
</a>
<div id="js-jump-plan-container" class="jump-plan-container" style="top: -11px;">
<i class="icon-font icon-plane jump-plane"></i>
</div>
</div>
</div>
</aside>
<article id="post-一种基于django-restFramework通用型增删改查方案" class="article article-type-post article-index" itemscope itemprop="blogPost">
<div class="article-inner">
<header class="article-header">
<h1 itemprop="name">
<a class="article-title" href="/Ranger/2019/06/21/一种基于django-restFramework通用型增删改查方案/">一种基于django-restFramework通用型增删改查方案</a>
</h1>
<a href="/Ranger/2019/06/21/一种基于django-restFramework通用型增删改查方案/" class="archive-article-date">
<time datetime="2019-06-21T14:23:57.000Z" itemprop="datePublished"><i class="icon-calendar icon"></i>2019-06-21</time>
</a>
</header>
<div class="article-entry" itemprop="articleBody">
<h2 id="方案理论基础"><a href="#方案理论基础" class="headerlink" title="方案理论基础"></a>方案理论基础</h2><p>在Web开发中对于数据模型的操作不可避免,常见的数据资源的对象一般有两种:</p>
<ol>
<li>对某一数据资源集合的操作</li>
<li>对某一数据资源对象的的操作</li>
</ol>
<p>对于数据资源操作及其前端HTTP请求方法通常有:</p>
<ol>
<li><code>增</code>—-> <code>POST</code> or <code>PUT</code></li>
<li><code>删</code>—-> <code>DELETE</code> </li>
<li><code>改</code>—-> <code>POST</code> or <code>PUT</code></li>
<li><code>查</code>—-> <code>POST</code> or <code>GET</code></li>
</ol>
<p>而业界<code>RESTful API</code>设计风格的对资源的操作类型和HTTP动词关系设计通常采用如下方案:</p>
<ol>
<li><code>增</code>—-> <code>POST</code> (在服务器新建资源)</li>
<li><code>删</code>—-> <code>DELETE</code> (从服务器删除资源) </li>
<li><code>改</code>—-> <code>PUT</code> (在服务器更新资源,客户端提供资源改变属性)</li>
<li><code>查</code>—-> <code>GET</code> (从服务器获去资源)</li>
<li><code>改</code>—-> <code>PATCH</code> (在服务器更新资源,服务器提供资源改变属性)</li>
</ol>
<p>利用上述原理和<code>django-restFramework</code>的序列化器<code>serialiaer</code>我们设计一种适用于<code>django</code>框架项目的通用型增删改查方案。</p>
<h2 id="项目架构"><a href="#项目架构" class="headerlink" title="项目架构"></a>项目架构</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">project---</span><br><span class="line"> |--project--|</span><br><span class="line"> |__init__.py</span><br><span class="line"> |urls.py</span><br><span class="line"> |settings.py</span><br><span class="line"> |wsgi.py</span><br><span class="line"></span><br><span class="line"> |---app-----|__init__.py</span><br><span class="line"> |app.py</span><br><span class="line"> |urls.py</span><br><span class="line"> |views.py</span><br><span class="line"> |serializers.py </span><br><span class="line"> |models.py</span><br><span class="line"> |admin.py</span><br><span class="line"> ....</span><br><span class="line"></span><br><span class="line"> |--manage.py</span><br></pre></td></tr></table></figure>
<h2 id="实现方案"><a href="#实现方案" class="headerlink" title="实现方案"></a>实现方案</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#views.py</span></span><br><span class="line"><span class="keyword">from</span> django.shortcuts <span class="keyword">import</span> render</span><br><span class="line"><span class="keyword">from</span> django.http <span class="keyword">import</span> HttpResponse</span><br><span class="line"><span class="keyword">from</span> rest_framework <span class="keyword">import</span> status</span><br><span class="line"><span class="keyword">from</span> django.db.models <span class="keyword">import</span> Q</span><br><span class="line"><span class="keyword">from</span> functools <span class="keyword">import</span> reduce</span><br><span class="line"><span class="keyword">import</span> operator</span><br><span class="line"><span class="keyword">from</span> app <span class="keyword">import</span> serializers</span><br><span class="line"><span class="keyword">from</span> app <span class="keyword">import</span> model</span><br><span class="line"><span class="keyword">from</span> app.serializers <span class="keyword">import</span> JSONResponse</span><br><span class="line"><span class="keyword">from</span> rest_framework.parsers <span class="keyword">import</span> JSONParserl</span><br><span class="line"><span class="keyword">import</span> importlib</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">model_list</span><span class="params">(request)</span>:</span></span><br><span class="line"> </span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 列出所有的code snippet,或创建一个新的snippet。</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> model_str=request.META.get(<span class="string">'HTTP_MODEL'</span>)</span><br><span class="line"> <span class="keyword">global</span> model</span><br><span class="line"> <span class="keyword">global</span> serializers</span><br><span class="line"> <span class="keyword">if</span> model_str:</span><br><span class="line"> Model=getattr(model,model_str)</span><br><span class="line"> ModelSerializer=getattr(serializers,model_str+<span class="string">'Serializer'</span>)</span><br><span class="line"> <span class="keyword">if</span> request.method == <span class="string">'GET'</span>:</span><br><span class="line"> <span class="comment"># 获取资源列表</span></span><br><span class="line"> models = Model.objects.ALL()</span><br><span class="line"> serializer = ModelSerializer(models, many=<span class="literal">True</span>)</span><br><span class="line"> <span class="keyword">return</span> JSONResponse(serializer.data)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">elif</span> request.method == <span class="string">'POST'</span>:</span><br><span class="line"> <span class="comment"># 根据条件获取资源列表</span></span><br><span class="line"> data = JSONParser().parse(request))</span><br><span class="line"> serializer = ModelSerializer(data=data)</span><br><span class="line"> fields=Model.get_fields()</span><br><span class="line"> q_list=[]</span><br><span class="line"> <span class="keyword">for</span> field <span class="keyword">in</span> fields:</span><br><span class="line"> value=data.get(field)</span><br><span class="line"> <span class="keyword">if</span> value:</span><br><span class="line"> q_list.append(Q(**{field+<span class="string">'__icontains'</span>:value}))</span><br><span class="line"> id_list=Model.get_indexs()</span><br><span class="line"> <span class="keyword">for</span> field <span class="keyword">in</span> id_list:</span><br><span class="line"> value=data.get(field)</span><br><span class="line"> <span class="keyword">if</span> isinstance(value,list):</span><br><span class="line"> <span class="keyword">if</span> data.get(field):</span><br><span class="line"> q_list.append(Q(**{field+<span class="string">'__in'</span>:value}))</span><br><span class="line"> <span class="keyword">if</span> isinstance(value,int):</span><br><span class="line"> <span class="keyword">if</span> data.get(field):</span><br><span class="line"> q_list.append(Q(**{field+<span class="string">''</span>:value}))</span><br><span class="line"> <span class="keyword">if</span> isinstance(value,str):</span><br><span class="line"> <span class="keyword">if</span> data.get(field):</span><br><span class="line"> q_list.append(Q(**{field+<span class="string">''</span>:int(value)}))</span><br><span class="line"> models=Model.objects.filter((reduce(operator.and_, q_list)))</span><br><span class="line"> serializer = ModelSerializer(models, many=<span class="literal">True</span>)</span><br><span class="line"> <span class="keyword">return</span> JSONResponse(serializer.data)</span><br><span class="line"> <span class="keyword">elif</span> request.method == <span class="string">'PUT'</span>:</span><br><span class="line"> <span class="comment"># 新建资源</span></span><br><span class="line"> data = JSONParser().parse(request)</span><br><span class="line"> serializer = ModelSerializer(data=data)</span><br><span class="line"> <span class="keyword">if</span> serializer.is_valid():</span><br><span class="line"> serializer.save()</span><br><span class="line"> <span class="keyword">return</span> JSONResponse(serializer.data, status=status.HTTP_201_CREATED)</span><br><span class="line"> <span class="keyword">return</span> JSONResponse(serializer.errors, status=status.HTTP_400_BAD_REQUEST)</span><br><span class="line"> <span class="keyword">return</span> JSONResponse(<span class="string">'not definde request headers model'</span>, status=status.HTTP_400_BAD_REQUEST)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">model_detail</span><span class="params">(request,pk)</span>:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 获取,更新或删除一个 code snippet。</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> model_str=request.META.get(<span class="string">'HTTP_MODEL'</span>)</span><br><span class="line"> <span class="keyword">global</span> model</span><br><span class="line"> <span class="keyword">global</span> serializers</span><br><span class="line"> <span class="keyword">if</span> model_str:</span><br><span class="line"> Model=getattr(model,model_str)</span><br><span class="line"> ModelSerializer=getattr(serializers,model_str+<span class="string">'Serializer'</span>)</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> model = Model.objects.get(pk=pk)</span><br><span class="line"> <span class="keyword">except</span> Model.DoesNotExist:</span><br><span class="line"> <span class="keyword">return</span> HttpResponse(status=status.HTTP_404_NOT_FOUND)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> request.method == <span class="string">'GET'</span>:</span><br><span class="line"> <span class="comment">#获取一个资源对象</span></span><br><span class="line"> serializer = ModelSerializer(model)</span><br><span class="line"> <span class="keyword">return</span> JSONResponse(serializer.data)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">elif</span> request.method == <span class="string">'PUT'</span>:</span><br><span class="line"> <span class="comment"># 更新一个资源对象</span></span><br><span class="line"> data = JSONParser().parse(request)</span><br><span class="line"> serializer = ModelSerializer(model, data=data)</span><br><span class="line"> <span class="keyword">if</span> serializer.is_valid():</span><br><span class="line"> serializer.save()</span><br><span class="line"> <span class="keyword">return</span> Response(serializer.data)</span><br><span class="line"> <span class="keyword">return</span> JSONResponse(serializer.errors, status=status.HTTP_400_BAD_REQUEST)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">elif</span> request.method == <span class="string">'DELETE'</span>:</span><br><span class="line"> <span class="comment">#删除一个资源对象</span></span><br><span class="line"> model.delete()</span><br><span class="line"> <span class="keyword">return</span> HttpResponse(status=status.HTTP_204_NO_CONTENT)</span><br><span class="line"> <span class="keyword">return</span> JSONResponse(<span class="string">'not definde request headers model'</span>, status=status.HTTP_400_BAD_REQUEST)</span><br></pre></td></tr></table></figure>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># serializers.py</span></span><br><span class="line"><span class="keyword">from</span> rest_framework <span class="keyword">import</span> serializers</span><br><span class="line"><span class="keyword">from</span> rest_framework.renderers <span class="keyword">import</span> JSONRenderer</span><br><span class="line"><span class="keyword">from</span> app.model <span class="keyword">import</span> Modal</span><br><span class="line"><span class="keyword">from</span> django.http <span class="keyword">import</span> HttpResponse</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">JSONResponse</span><span class="params">(HttpResponse)</span>:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> An HttpResponse that renders its content into JSON.</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, data, **kwargs)</span>:</span></span><br><span class="line"> content = JSONRenderer().render(data)</span><br><span class="line"> kwargs[<span class="string">'content_type'</span>] = <span class="string">'application/json'</span></span><br><span class="line"> super(JSONResponse, self).__init__(content, **kwargs)</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ModalSerializer</span><span class="params">(serializers.ModelSerializer)</span>:</span></span><br><span class="line"> <span class="class"><span class="keyword">class</span> <span class="title">Meta</span>:</span></span><br><span class="line"> model = Modal</span><br><span class="line"> fields = <span class="string">'__all__'</span></span><br></pre></td></tr></table></figure>
<h3 id="请求封装"><a href="#请求封装" class="headerlink" title="请求封装"></a>请求封装</h3><p>首先在前端请求的headers中添加我们需要获取的资源类名<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line">headers={<span class="string">'model'</span>:<span class="string">'Model'</span>,<span class="string">'Content-Type'</span>:<span class="string">'application/json'</span>}</span><br><span class="line">url=<span class="string">'http://127.0.0.1:8000/app/model/model_list/'</span></span><br><span class="line">response=requests.get(url, headers=headers)</span><br><span class="line">print(response.text)</span><br></pre></td></tr></table></figure></p>
<p>为了防止<code>django</code>拦截headers中的参数,我们需要在<code>settings</code>文件中配置<code>request</code>跨域的允许访问的<code>headers</code>字段<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">CORS_ALLOW_HEADERS = (</span><br><span class="line"> <span class="string">'accept'</span>,</span><br><span class="line"> ...,</span><br><span class="line"> <span class="string">'model'</span>,</span><br><span class="line">)</span><br></pre></td></tr></table></figure></p>
<h3 id="解析并获取资源类"><a href="#解析并获取资源类" class="headerlink" title="解析并获取资源类"></a>解析并获取资源类</h3><p>前台请求制定资源对象类名和请求HTTP动词即可通过反射定位到对应方法,完成对资源的相关操作<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># app 下的model文件</span></span><br><span class="line"><span class="keyword">from</span> app <span class="keyword">import</span> model</span><br><span class="line"><span class="keyword">from</span> app <span class="keyword">import</span> serializers</span><br><span class="line"><span class="comment"># 反射获取model 类</span></span><br><span class="line">Model=getattr(model,model_str)</span><br><span class="line"><span class="comment"># 反射获取 model类对象的序列化对象</span></span><br><span class="line">ModelSerializer=getattr(serializers,model_str+<span class="string">'Serializer'</span>)</span><br></pre></td></tr></table></figure></p>
<h3 id="动态构建资源对象的查询条件"><a href="#动态构建资源对象的查询条件" class="headerlink" title="动态构建资源对象的查询条件"></a>动态构建资源对象的查询条件</h3><p>通常我们对资源查询对于像<code>id</code>、<code>pk</code>、<code>fk</code>等索引性条件采用<code>field__in=list_obj</code>或者<code>field=int_obj</code>,对于其他字段通常采用<code>field__icontains=obj</code>的模糊匹配的方法,进行条件查询,利用这些特性,我们使用<code>Q</code>查询动态构建查询条件。</p>
<p>为了让改方法适用于所有的模型,我们讲查询条件定义到 `model.py`中的资源对象类中静态方法<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">UserDetail</span><span class="params">(models.Model)</span>:</span></span><br><span class="line"> name=models.CharField(max_length=<span class="number">50</span>, null=<span class="literal">True</span>)</span><br><span class="line"> sex=models.CharField(max_length=<span class="number">2</span>, null=<span class="literal">True</span>)</span><br><span class="line"> age=models.IntegerField()</span><br><span class="line"> address=models.CharField(max_length=<span class="number">100</span>, null=<span class="literal">True</span>)</span><br><span class="line"> user=models.ForeignKey(User,null=<span class="literal">True</span>, blank=<span class="literal">True</span>, on_delete=models.SET_NULL))</span><br><span class="line"></span><br><span class="line"><span class="meta"> @staticmethod</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">get_indexs</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> [<span class="string">'id'</span><span class="string">'age'</span>,<span class="string">'pk'</span>]</span><br><span class="line"></span><br><span class="line"><span class="meta"> @staticmethod</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">get_fields</span><span class="params">(Self)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> [<span class="string">'name'</span>,<span class="string">'sex'</span>,<span class="string">'address'</span>]</span><br></pre></td></tr></table></figure></p>
<p>至此我们基本完成了jango项目通用型增删改查的方案,所有跟资源模型相关在<code>model</code>文件中定义,如果要使用<code>django-restframwork</code>序列化方法的相关则在<code>serializers.py</code>中定义,对资源模型的操作在<code>views</code>文件中定义,当然你完全可以重新定义当多个APP使用同一个<code>model_list()</code>和<code>model_detail()</code>方法,也可以不采用<code>django-restframwork</code></p>
</div>
<div class="article-info article-info-index">
<div class="article-tag tagcloud">
<i class="icon-price-tags icon"></i>
<ul class="article-tag-list">
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color2">django</a>
</li>
<li class="article-tag-list-item">
<a href="javascript:void(0)" class="js-tag article-tag-list-link color1">django-restFramework</a>
</li>
</ul>
</div>