/
tutorial_original.html
2753 lines (2739 loc) · 121 KB
/
tutorial_original.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
</head>
<body>
<h1>Weave Documentation</h1>
<p>
By Eric Jones eric@enthought.com
</p>
<p></p>
<h2>Outline</h2>
<dl>
<dd> <a href="#Introduction">Introduction</a> </dd>
<dd> <a href="#Requirements">Requirements</a> </dd>
<dd> <a href="#Installation">Installation</a> </dd>
<dd> <a href="#Testing">Testing</a> </dd>
<dd> <a href="#Benchmarks">Benchmarks</a> </dd>
<dd> <a href="#Inline">Inline</a>
<dl>
<dd><a href="#More%20with%20printf">More with printf</a> </dd>
<dd> <a href="#More%20examples">More examples</a>
<dl>
<dd><a href="#Binary%20search">Binary search</a> </dd>
<dd><a href="#Dictionary%20sort">Dictionary sort</a> </dd>
<dd><a href="#Numeric%20--%20cast/copy/transpose">NumPy --
cast/copy/transpose</a> </dd>
<dd><a href="#wxPython">wxPython</a></dd>
</dl>
</dd>
<dd><a href="#Keyword%20options">Keyword options</a> </dd>
<dd><a href="#Returning%20values">Returning values</a>
<dl>
<dd><a href="#The%20issue%20with%20locals%28%29"> The issue
with <code>locals()</code></a></dd>
</dl>
</dd>
<dd><a href="#inline_quick_look_at_code">A quick look at the code</a>
</dd>
<dd> <a href="#inline_technical_details">Technical Details</a>
<dl>
<dd><a href="#Converting%20Types">Converting Types</a>
<dl>
<dd><a href="#inline_numeric_argument_conversion"> NumPy
Argument Conversion</a> </dd>
<dd><a href="#inline_python_argument_conversion"> String,
List, Tuple, and Dictionary Conversion</a> </dd>
<dd><a href="#inline_callable_argument_conversion">File
Conversion</a> </dd>
<dd><a href="#inline_callable_argument_conversion">Callable,
Instance, and Module Conversion</a> </dd>
<dd><a href="#Customizing%20Conversions">Customizing
Conversions</a> </dd>
</dl>
</dd>
<dd><a href="#Compiling%20Code">Compiling Code</a> </dd>
<dd><a href="#The%20Catalog">"Cataloging" functions</a>
<dl>
<dd><a href="#function%20storage">Function Storage</a> </dd>
<dd><a href="#PYTHONCOMPILED">The PYTHONCOMPILED
evnironment variable</a></dd>
</dl>
</dd>
</dl>
</dd>
</dl>
</dd>
<dd><a href="#Blitz">Blitz</a>
<dl>
<dd><a href="#blitz_requirements">Requirements</a> </dd>
<dd><a href="#blitz_limitations">Limitations</a> </dd>
<dd><a href="#Numeric%20Efficiency">NumPy Efficiency Issues</a> </dd>
<dd><a href="#blitz_tools">The Tools</a>
<dl>
<dd><a href="#blitz_parser">Parser</a> </dd>
<dd><a href="#blitz_blitz">Blitz and NumPy</a> </dd>
</dl>
</dd>
<dd><a href="#blitz_type_conversions">Type defintions and coersion</a>
</dd>
<dd><a href="#blitz_catalog">Cataloging Compiled Functions</a> </dd>
<dd><a href="#blitz_array_sizes">Checking Array Sizes</a> </dd>
<dd><a href="#blitz_extension_module">Creating the Extension
Module</a> </dd>
</dl>
</dd>
<dd> <a href="#Extension%20Modules"> Extension Modules</a>
<dl>
<dd><a href="#A%20Simple%20Example">A Simple Example</a> </dd>
<dd><a href="#Fibonacci%20Example">Fibonacci Example</a> </dd>
</dl>
</dd>
<dd> <a href="#Type%20Factories"> Customizing Type Conversions --
Type Factories (not written)</a>
<dl>
<dd>Type Specifications </dd>
<dd>Type Information </dd>
<dd>The Conversion Process </dd>
</dl>
</dd>
</dl>
<a name="Introduction"></a>
<h1>Introduction</h1>
<p>
The <code>weave</code> package provides tools for including C/C++ code
within
in Python code. This offers both another level of optimization to those
who need it, and an easy way to modify and extend any supported
extension libraries such as wxPython and hopefully VTK soon. Inlining
C/C++ code within Python generally
results in speed ups of 1.5x to 30x speed-up over algorithms written in
pure
Python (However, it is also possible to slow things down...). Generally
algorithms that require a large number of calls to the Python API don't
benefit
as much from the conversion to C/C++ as algorithms that have inner
loops completely convertable to C.
</p>
<p> There are three basic ways to use <code>weave</code>. The <code>weave.inline()</code>
function executes C code directly within Python, and <code>weave.blitz()</code>
translates Python NumPy expressions to C++ for fast execution. <code>blitz()</code>
was the original reason <code>weave</code> was built. For those
interested in building extension
libraries, the <code>ext_tools</code> module provides classes for
building extension modules within Python. </p>
<p>Most of <code>weave's</code> functionality should work on Windows
and Unix, although some of its functionality requires <code>gcc</code>
or a similarly modern C++ compiler that handles templates well. Up to
now, most testing has been done on Windows 2000 with Microsoft's C++
compiler (MSVC) and with gcc (mingw32 2.95.2 and 2.95.3-6). All tests
also pass on Linux (RH 7.1 with gcc 2.96), and I've had reports that it
works on Debian also (thanks Pearu).
</p>
<p>The <code>inline</code> and <code>blitz</code> provide new
functionality to Python (although I've recently learned about the <a
href="http://pyinline.sourceforge.net/">PyInline</a> project which may
offer similar functionality to <code>inline</code>). On the other
hand, tools for building Python extension modules already exists (SWIG,
SIP, pycpp, CXX, and others). As of yet, I'm not sure where <code>weave</code>
fits in this spectrum. It is closest in flavor to CXX in that it makes
creating new C/C++ extension modules pretty easy. However, if you're
wrapping a gaggle of legacy functions or classes, SWIG and friends are
definitely the better choice. <code>weave</code> is set up so that you
can customize how Python types are converted to C types in <code>weave</code>.
This is great for <code>inline()</code>, but, for wrapping legacy
code, it is more flexible to specify things the other way around --
that is how C types map to Python types. This <code>weave</code> does
not do. I guess it would be possible to build such a tool on top of <code>weave</code>,
but with good tools like SWIG around, I'm not sure the effort produces
any new capabilities. Things like function overloading are probably
easily implemented in <code>weave</code> and it might be easier to mix
Python/C code in function calls, but nothing beyond this comes to mind.
So, if you're developing new extension modules or optimizing Python
functions in C, <code>weave.ext_tools()</code> might be the tool for
you. If you're wrapping legacy code, stick with SWIG.
</p>
<p>The next several sections give the basics of how to use <code>weave</code>.
We'll discuss what's happening under the covers in more detail later
on. Serious users will need to at least look at the type conversion
section to understand how Python variables map to C/C++ types and how
to customize this behavior. One other note. If you don't know C or C++
then these docs are probably of very little help to you. Further, it'd
be helpful if you know something about writing Python extensions. <code>weave</code>
does quite a bit for you, but for anything complex, you'll need to do
some conversions, reference counting, etc.
</p>
<p><em>
Note: </em><code>weave</code><em> is actually part of the <a
href="http://www.scipy.org">SciPy</a> package. However, it also works
fine as a standalone package (you can check out the sources using svn
co http://svn.scipy.org/svn/scipy/trunk/Lib/weave weave and install as
python setup.py install). The examples here are given as if it is used
as a stand alone package. If you are using from within scipy, you can
use <code> from scipy import weave</code> and the examples will work
identically.</em>
<a name="Requirements"></a></p>
<h1>Requirements</h1>
<ul>
<li> Python
<p> I use 2.1.1. Probably 2.0 or higher should work. </p>
<p> </p>
</li>
<li> C++ compiler
<p> <code>weave</code> uses <code>distutils</code> to actually
build extension modules, so it uses whatever compiler was originally
used to build Python. <code>weave</code> itself requires a C++
compiler. If you used a C++ compiler to build Python, your probably
fine. </p>
<p> On Unix gcc is the preferred choice because I've done a little
testing with it. All testing has been done with gcc, but I expect the
majority of compilers should work for <code>inline</code> and <code>ext_tools</code>.
The one issue I'm not sure about is that I've hard coded things so that
compilations are linked with the <code>stdc++</code> library. <em>Is
this standard across Unix compilers, or is this a gcc-ism?</em> </p>
<p> For <code>blitz()</code>, you'll need a reasonably recent
version of gcc. 2.95.2 works on windows and 2.96 looks fine on Linux.
Other versions are likely to work. Its likely that KAI's C++ compiler
and maybe some others will work, but I haven't tried. My advice is to
use gcc for now unless your willing to tinker with the code some. </p>
<p> On Windows, either MSVC or gcc (<a
href="http://www.mingw.org%3Ewww.mingw.org"> mingw32</a>) should work.
Again, you'll need gcc for <code>blitz()</code> as the MSVC compiler
doesn't handle templates well. </p>
<p> I have not tried Cygwin, so please report success if it works
for you. </p>
<p> </p>
</li>
<li> NumPy
<p>The python NumPy module from <a href="http://numeric.scipy.org/">here</a>.
is required for <code>blitz()</code> to work and for numpy.distutils
which is used by weave. </p>
<p> </p>
</li>
</ul>
<p>
<a name="Installation"></a></p>
<h1>Installation</h1>
<p>
There are currently two ways to get <code>weave</code>. Fist, <code>weave</code>
is part of SciPy and installed automatically (as a sub-
package) whenever SciPy is installed. Second, since <code>weave</code>
is useful outside of the scientific community, it has been setup so
that it can be
used as a stand-alone module. </p>
<p>The stand-alone version can be downloaded from <a
href="http://www.scipy.org/Weave">here</a>. Instructions for
installing should be found there as well. setup.py file to
simplify
installation. </p>
<p><a name="Testing"></a></p>
<h1>Testing</h1>
Once <code>weave</code> is installed, fire up python and run its unit
tests.
<blockquote>
<pre><code>
>>> import weave
>>> weave.test()
runs long time... spews tons of output and a few warnings
.
.
.
..............................................................
................................................................
..................................................
----------------------------------------------------------------------
Ran 184 tests in 158.418s
OK
<unittest.texttestrunner
instance="" at="" 01562934="">
>>>
</unittest.texttestrunner></code></pre>
</blockquote>
This takes a while, usually several minutes.
On Unix with remote file systems, I've had it take 15 or so minutes. In
the end, it should run about 180 tests and spew some speed results
along the way. If you get errors, they'll be reported at the end of the
output. Please report erros that you find. Some tests are
known to fail at this point. <br>
<p>If you only want to test a single module of the package, you can do
this by
running test() for that specific module. </p>
<blockquote>
<pre><code>
>>> import weave.scalar_spec
>>> weave.scalar_spec.test()
.......
----------------------------------------------------------------------
Ran 7 tests in 23.284s
</code></pre>
</blockquote>
<em>Testing Notes:
</em>
<ul>
<em> </em>
<li><em> Windows 1
<p> I've had some test fail on windows machines where I have msvc,
gcc-2.95.2 (in c:\gcc-2.95.2), and gcc-2.95.3-6 (in c:\gcc) all
installed. My environment has c:\gcc in the path and does not have
c:\gcc-2.95.2 in the path. The test process runs very smoothly until
the end where several test using gcc fail with cpp0 not found by g++.
If I check os.system('gcc -v') before running tests, I get
gcc-2.95.3-6. If I check after running tests (and after failure), I get
gcc-2.95.2. ??huh??. The os.environ['PATH'] still has c:\gcc first in
it and is not corrupted (msvc/distutils messes with the environment
variables, so we have to undo its work in some places). If anyone else
sees this, let me know - - it may just be an quirk on my machine
(unlikely). Testing with the gcc- 2.95.2 installation always works. </p>
<p> </p>
</em></li>
<em> </em>
<li><em> Windows 2
<p> If you run the tests from PythonWin or some other GUI tool,
you'll get a ton of DOS windows popping up periodically as <code>weave</code>
spawns the compiler multiple times. Very annoying. Anyone know how to
fix this? </p>
<p> </p>
</em></li>
<em> </em>
<li><em> wxPython
<p> wxPython tests are not enabled by default because importing
wxPython on a Unix machine without access to a X-term will cause the
program to exit. Anyone know of a safe way to detect whether wxPython
can be imported and whether a display exists on a machine? </p>
<p> </p>
<p></p>
</em></li>
<em> </em>
</ul>
<a name="Benchmarks"></a>
<h1>Benchmarks</h1>
This section has not been updated from old scipy weave and Numeric....<br>
<br>
This section has a few benchmarks -- thats all people want to see
anyway right? These are mostly taken from running files in the <code>weave/example</code>
directory and also from the test scripts. Without more information
about what the test actually do, their value is limited. Still, their
here for the curious. Look at the example scripts for more specifics
about what problem was actually solved by each run. These examples are
run under windows 2000 using Microsoft Visual C++ and python2.1 on a
850 MHz PIII laptop with 320 MB of RAM.
Speed up is the improvement (degredation) factor of <code>weave</code>
compared to conventional Python functions. <code>The blitz()</code>
comparisons are shown
compared to NumPy.
<p></p>
<center>
<table border="1" width="100%">
<tbody>
<tr>
<td colspan="2" width="100%">
<p align="center">inline and ext_tools</p>
</td>
</tr>
<tr>
<td>
<p align="center">Algorithm</p>
</td>
<td>
<p align="center">Speed up </p>
</td>
</tr>
<tr>
<td>binary search</td>
<td> 1.50 </td>
</tr>
<tr>
<td>fibonacci (recursive)</td>
<td> 82.10 </td>
</tr>
<tr>
<td>fibonacci (loop)</td>
<td> 9.17 </td>
</tr>
<tr>
<td>return None</td>
<td> 0.14 </td>
</tr>
<tr>
<td>map</td>
<td> 1.20 </td>
</tr>
<tr>
<td>dictionary sort</td>
<td> 2.54 </td>
</tr>
<tr>
<td>vector quantization</td>
<td> 37.40 </td>
</tr>
<tr>
<td colspan="2" width="100%">
<p align="center">blitz -- double precision</p>
</td>
</tr>
<tr>
<td>
<p align="center">Algorithm</p>
</td>
<td>
<p align="center">Speed up </p>
</td>
</tr>
<tr>
<td>a = b + c 512x512</td>
<td> 3.05 </td>
</tr>
<tr>
<td>a = b + c + d 512x512</td>
<td> 4.59 </td>
</tr>
<tr>
<td>5 pt avg. filter, 2D Image 512x512</td>
<td> 9.01 </td>
</tr>
<tr>
<td>Electromagnetics (FDTD) 100x100x100</td>
<td> 8.61 </td>
</tr>
</tbody>
</table>
</center>
<p>
The benchmarks shown <code>blitz</code> in the best possible light.
NumPy (at least on my machine) is significantly worse for double
precision than it is for single precision calculations. If your
interested in single precision results, you can pretty much divide the
double precision speed up by 3 and you'll
be close.
<a name="Inline"></a></p>
<h1>Inline</h1>
<p>
<code>inline()</code> compiles and executes C/C++ code on the fly.
Variables in the local and global Python scope are also available in
the C/C++ code. Values are passed to the C/C++ code by assignment much
like variables are passed into a standard Python function. Values are
returned from the C/C++ code through a special argument called
return_val. Also, the contents of mutable objects can be changed within
the C/C++ code and the changes remain after the C code exits and
returns to Python. (more on this later)
</p>
<p> Here's a trivial <code>printf</code> example using <code>inline()</code>:
</p>
<blockquote>
<pre><code>
>>> import weave
>>> a = 1
>>> weave.inline('printf("%d\\n",a);',['a'])
1
</code></pre>
</blockquote>
<p>In this, its most basic form, <code>inline(c_code, var_list)</code>
requires two arguments. <code>c_code</code> is a string of valid C/C++
code. <code>var_list</code> is a list of variable names that are
passed from Python into C/C++. Here we have a simple <code>printf</code>
statement that writes the Python variable <code>a</code> to the
screen. The first time you run this, there will be a pause while the
code is written to a .cpp file, compiled into an extension module,
loaded into Python, cataloged for future use, and executed. On windows
(850 MHz PIII), this takes about 1.5 seconds when using Microsoft's C++
compiler (MSVC) and 6-12 seconds using gcc (mingw32 2.95.2). All
subsequent executions of the code will happen very quickly because the
code only needs to be compiled once. If you kill and restart the
interpreter and then execute the same code fragment again, there will
be a much shorter delay in the fractions of seconds range. This is
because <code>weave</code> stores a catalog of all previously compiled
functions in an on disk cache. When it sees a string that has been
compiled, it loads the already compiled module and executes the
appropriate function. </p>
<p><em>
Note: If you try the <code>printf</code> example in a GUI shell such
as IDLE, PythonWin, PyShell, etc., you're unlikely to see the output.
This is because the C code is writing to stdout, instead of to the GUI
window. This doesn't mean that inline doesn't work in these
environments -- it only means that standard out in C is not the same as
the standard out for Python in these cases. Non input/output functions
will work as expected.
</em></p>
<p>Although effort has been made to reduce the overhead associated with
calling inline, it is still less efficient for simple code snippets
than using equivalent Python code. The simple <code>printf</code>
example is actually slower by 30% or so than using Python <code>print</code>
statement. And, it is not difficult to create code fragments that are
8-10 times slower using inline than equivalent Python. However, for
more complicated algorithms, the speed up can be worth while --
anywhwere from 1.5- 30 times faster. Algorithms that have to manipulate
Python objects (sorting a list) usually only see a factor of 2 or so
improvement. Algorithms that are highly computational or manipulate
NumPy arrays can see much larger improvements. The examples/vq.py file
shows a factor of 30 or more improvement on the vector quantization
algorithm that is used heavily in information theory and classification
problems.
</p>
<p><a name="More with printf"></a>
</p>
<h2>More with printf</h2>
<p>
MSVC users will actually see a bit of compiler output that distutils
does not
supress the first time the code executes: </p>
<blockquote>
<pre><code> <br> >>> weave.inline(r'printf("%d\n",a);',['a'])<br> sc_e013937dbc8c647ac62438874e5795131.cpp<br> Creating library C:\DOCUME~1\eric\LOCALS~1\Temp\python21_compiled\temp<br> \Release\sc_e013937dbc8c647ac62438874e5795131.lib and object C:\DOCUME<br> ~1\eric\LOCALS~1\Temp\python21_compiled\temp\Release\sc_e013937dbc8c64<br> 7ac62438874e5795131.exp<br> 1<br> </code></pre>
</blockquote>
<p>
Nothing bad is happening, its just a bit annoying. <em> Anyone know
how to turn this off?</em> </p>
<p>This example also demonstrates using 'raw strings'. The <code>r</code>
preceeding the code string in the last example denotes that this is a
'raw string'. In raw strings, the backslash character is not
interpreted as an escape character, and so it isn't necessary to use a
double backslash to indicate that the '\n' is meant to be interpreted
in the C <code>printf</code> statement instead of by Python. If your C
code contains a lot
of strings and control characters, raw strings might make things
easier.
Most of the time, however, standard strings work just as well.
</p>
<p>The <code>printf</code> statement in these examples is formatted to
print out integers. What happens if <code>a</code> is a string? <code>inline</code>
will happily, compile a new version of the code to accept strings as
input,
and execute the code. The result? </p>
<blockquote>
<pre><code> <br> >>> a = 'string'<br> >>> weave.inline(r'printf("%d\n",a);',['a'])<br> 32956972<br> </code></pre>
</blockquote>
<p>In this case, the result is non-sensical, but also non-fatal. In
other situations, it might produce a compile time error because <code>a</code>
is required to be an integer at some point in the code, or it could
produce a segmentation fault. Its possible to protect against passing <code>inline</code>
arguments of the wrong data type by using asserts in Python. </p>
<blockquote>
<pre><code> <br> >>> a = 'string'<br> >>> def protected_printf(a): <br> ... assert(type(a) == type(1))<br> ... weave.inline(r'printf("%d\n",a);',['a'])<br> >>> protected_printf(1)<br> 1<br> >>> protected_printf('string')<br> AssertError...<br> </code></pre>
</blockquote>
<p>For printing strings, the format statement needs to be changed.
Also, weave
doesn't convert strings to char*. Instead it uses CXX Py::String type,
so you have to do a little more work. Here we convert it to a C++
std::string
and then ask cor the char* version. </p>
<blockquote>
<pre><code> <br> >>> a = 'string' <br> >>> weave.inline(r'printf("%s\n",std::string(a).c_str());',['a'])<br> string<br> </code></pre>
</blockquote>
<p><em> This is a little convoluted. Perhaps strings should convert to
std::string
objects instead of CXX objects. Or maybe to char*.
</em></p>
<p>As in this case, C/C++ code fragments often have to change to accept
different types. For the given printing task, however, C++ streams
provide a way of a single statement that works for integers and
strings. By default, the stream objects live in the std (standard)
namespace and thus require the use of <code>std::</code>. </p>
<blockquote>
<pre><code> <br> >>> weave.inline('std::cout << a << std::endl;',['a'])<br> 1 <br> >>> a = 'string'<br> >>> weave.inline('std::cout << a << std::endl;',['a'])<br> string<br> </code></pre>
</blockquote>
<p>Examples using <code>printf</code> and <code>cout</code> are
included in examples/print_example.py.
<a name="More examples"></a></p>
<h2> More examples </h2>
This section shows several more advanced uses of <code>inline</code>.
It includes a few algorithms from the <a
href="http://aspn.activestate.com/ASPN/Cookbook/Python">Python Cookbook</a>
that have been re-written in inline C to improve speed as well as a
couple examples using NumPy and wxPython.
<a name="Binary search"></a>
<h3> Binary search</h3>
Lets look at the example of searching a sorted list of integers for a
value. For inspiration, we'll use Kalle Svensson's <a
href="http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81188">
binary_search()</a> algorithm from the Python Cookbook. His recipe
follows:
<blockquote>
<pre><code>
def binary_search(seq, t):
min = 0; max = len(seq) - 1
while 1:
if max < min:
return -1
m = (min + max) / 2
if seq[m] < t:
min = m + 1
elif seq[m] > t:
max = m - 1
else:
return m
</code></pre>
</blockquote>
This Python version works for arbitrary Python data types. The C
version below is specialized to handle integer values. There is a
little type checking done in Python to assure that we're working with
the correct data types before heading into C. The variables <code>seq</code>
and <code>t</code> don't need to be declared beacuse <code>weave</code>
handles converting and declaring them in the C code. All other
temporary variables such as <code>min, max</code>, etc. must be
declared -- it is C after all. Here's the new mixed Python/C function:
<blockquote>
<pre><code> <br> def c_int_binary_search(seq,t):<br> # do a little type checking in Python<br> assert(type(t) == type(1))<br> assert(type(seq) == type([]))<br> <br> # now the C code<br> code = """<br> #line 29 "binary_search.py"<br> int val, m, min = 0; <br> int max = seq.length() - 1;<br> PyObject *py_val; <br> for(;;)<br> {<br> if (max < min ) <br> { <br> return_val = Py::new_reference_to(Py::Int(-1)); <br> break;<br> } <br> m = (min + max) /2;<br> val = py_to_int(PyList_GetItem(seq.ptr(),m),"val"); <br> if (val < t) <br> min = m + 1;<br> else if (val > t)<br> max = m - 1;<br> else<br> {<br> return_val = Py::new_reference_to(Py::Int(m));<br> break;<br> }<br> }<br> """<br> return inline(code,['seq','t'])<br> </code></pre>
</blockquote>
<p>We have two variables <code>seq</code> and <code>t</code> passed
in. <code>t</code> is guaranteed (by the <code>assert</code>) to be
an integer. Python integers are converted to C int types in the
transition from Python to C. <code>seq</code> is a Python list. By
default, it is translated to a CXX list object. Full documentation for
the CXX library can be found at its <a
href="http://cxx.sourceforge.net/">website</a>. The basics are that
the CXX provides C++ class equivalents for Python objects that
simplify, or at least object orientify, working with Python objects in
C/C++. For example, <code>seq.length()</code> returns the length of
the list. A little more about
CXX and its class methods, etc. is in the ** type conversions **
section.
</p>
<p><em>
Note: CXX uses templates and therefore may be a little less portable
than another alternative by Gordan McMillan called SCXX which was
inspired by
CXX. It doesn't use templates so it should compile faster and be more
portable.
SCXX has a few less features, but it appears to me that it would mesh
with
the needs of weave quite well. Hopefully xxx_spec files will be written
for SCXX in the future, and we'll be able to compare on a more
empirical
basis. Both sets of spec files will probably stick around, it just a
question
of which becomes the default.
</em></p>
<p>Most of the algorithm above looks similar in C to the original
Python code. There are two main differences. The first is the setting
of <code>return_val</code> instead of directly returning from the C
code with a <code>return</code> statement. <code>return_val</code> is
an automatically defined variable of type <code>PyObject*</code> that
is returned from the C code back to Python. You'll have to handle
reference counting issues when setting this variable. In this example,
CXX classes and functions handle the dirty work. All CXX functions and
classes live in the namespace <code>Py::</code>. The following code
converts the integer <code>m</code> to a CXX <code>Int()</code>
object and then to a <code>PyObject*</code> with an incremented
reference count using <code>Py::new_reference_to()</code>. </p>
<blockquote>
<pre><code> <br> return_val = Py::new_reference_to(Py::Int(m));<br> </code></pre>
</blockquote>
<p>The second big differences shows up in the retrieval of integer
values from the Python list. The simple Python <code>seq[i]</code>
call balloons into a C Python API call to grab the value out of the
list and then a separate call to <code>py_to_int()</code> that
converts the PyObject* to an integer. <code>py_to_int()</code>
includes both a NULL cheack and a <code>PyInt_Check()</code> call as
well as the conversion call. If either of the checks fail, an exception
is raised. The entire C++ code block is executed with in a <code>try/catch</code>
block that handles exceptions much like Python does. This removes the
need for most error checking code.
</p>
<p>It is worth note that CXX lists do have indexing operators that
result in code that looks much like Python. However, the overhead in
using them appears to be relatively high, so the standard Python API
was used on the <code>seq.ptr()</code> which is the underlying <code>PyObject*</code>
of the List object.
</p>
<p>The <code>#line</code> directive that is the first line of the C
code block isn't necessary, but it's nice for debugging. If the
compilation fails because of the syntax error in the code, the error
will be reported as an error in the Python file "binary_search.py" with
an offset from the given line number (29 here).
</p>
<p>So what was all our effort worth in terms of efficiency? Well not a
lot in this case. The examples/binary_search.py file runs both Python
and C versions of the functions As well as using the standard <code>bisect</code>
module. If we run it on a 1 million element list and run the search
3000 times (for 0-
2999), here are the results we get: </p>
<blockquote>
<pre><code> <br> C:\home\ej\wrk\scipy\weave\examples> python binary_search.py<br> Binary search for 3000 items in 1000000 length list of integers:<br> speed in python: 0.159999966621<br> speed of bisect: 0.121000051498<br> speed up: 1.32<br> speed in c: 0.110000014305<br> speed up: 1.45<br> speed in c(no asserts): 0.0900000333786<br> speed up: 1.78<br> </code></pre>
</blockquote>
<p>So, we get roughly a 50-75% improvement depending on whether we use
the Python asserts in our C version. If we move down to searching a
10000 element list, the advantage evaporates. Even smaller lists might
result in the Python version being faster. I'd like to say that moving
to NumPy lists (and getting rid of the GetItem() call) offers a
substantial speed up, but my preliminary efforts didn't produce one. I
think the log(N) algorithm is to blame. Because the algorithm is nice,
there just isn't much time spent computing things, so moving to C isn't
that big of a win. If there are ways to reduce conversion overhead of
values, this may improve the C/Python speed up. Anyone have other
explanations or faster code, please let me know.
<a name="#Dictionary sort"></a></p>
<h3> Dictionary Sort</h3>
<p>
The demo in examples/dict_sort.py is another example from the Python
CookBook. <a
href="http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52306">This
submission</a>, by Alex Martelli, demonstrates how to return the values
from a dictionary sorted by their keys: </p>
<blockquote>
<pre><code> <br> def sortedDictValues3(adict):<br> keys = adict.keys()<br> keys.sort()<br> return map(adict.get, keys)<br> </code></pre>
</blockquote>
<p>Alex provides 3 algorithms and this is the 3rd and fastest of the
set. The C version of this same algorithm follows: </p>
<blockquote>
<pre><code> <br> def c_sort(adict):<br> assert(type(adict) == type({}))<br> code = """ <br> #line 21 "dict_sort.py" <br> Py::List keys = adict.keys();<br> Py::List items(keys.length()); keys.sort(); <br> PyObject* item = NULL; <br> for(int i = 0; i < keys.length();i++)<br> {<br> item = PyList_GET_ITEM(keys.ptr(),i);<br> item = PyDict_GetItem(adict.ptr(),item);<br> Py_XINCREF(item);<br> PyList_SetItem(items.ptr(),i,item); <br> } <br> return_val = Py::new_reference_to(items);<br> """ <br> return inline_tools.inline(code,['adict'],verbose=1)<br> </code></pre>
</blockquote>
<p>Like the original Python function, the C++ version can handle any
Python dictionary regardless of the key/value pair types. It uses CXX
objects for the most part to declare python types in C++, but uses
Python API calls to manipulate their contents. Again, this choice is
made for speed. The C++ version, while
more complicated, is about a factor of 2 faster than Python. </p>
<blockquote>
<pre><code> <br> C:\home\ej\wrk\scipy\weave\examples> python dict_sort.py<br> Dict sort of 1000 items for 300 iterations:<br> speed in python: 0.319999933243<br> [0, 1, 2, 3, 4]<br> speed in c: 0.151000022888<br> speed up: 2.12<br> [0, 1, 2, 3, 4]<br> </code></pre>
</blockquote>
<p>
<a name="#Numeric -- cast/copy/transpose"></a></p>
<h3>NumPy -- cast/copy/transpose</h3>
CastCopyTranspose is a function called quite heavily by Linear Algebra
routines
in the NumPy library. Its needed in part because of the row-major
memory layout
of multi-demensional Python (and C) arrays vs. the col-major order of
the underlying
Fortran algorithms. For small matrices (say 100x100 or less), a
significant
portion of the common routines such as LU decompisition or singular
value decompostion
are spent in this setup routine. This shouldn't happen. Here is the
Python
version of the function using standard NumPy operations.
<blockquote>
<pre><code> <br> def _castCopyAndTranspose(type, array):<br> if a.typecode() == type:<br> cast_array = copy.copy(NumPy.transpose(a))<br> else:<br> cast_array = copy.copy(NumPy.transpose(a).astype(type))<br> return cast_array<br> </code></pre>
</blockquote>
And the following is a inline C version of the same function:
<blockquote>
<pre><code>
from weave.blitz_tools import blitz_type_factories
from weave import scalar_spec
from weave import inline
def _cast_copy_transpose(type,a_2d):
assert(len(shape(a_2d)) == 2)
new_array = zeros(shape(a_2d),type)
NumPy_type = scalar_spec.NumPy_to_blitz_type_mapping[type]
code = \
"""
for(int i = 0;i < _Na_2d[0]; i++)
for(int j = 0; j < _Na_2d[1]; j++)
new_array(i,j) = (%s) a_2d(j,i);
""" % NumPy_type
inline(code,['new_array','a_2d'],
type_factories = blitz_type_factories,compiler='gcc')
return new_array
</code></pre>
</blockquote>
This example uses blitz++ arrays instead of the standard representation
of NumPy arrays so that indexing is simplier to write. This is
accomplished by passing in the blitz++ "type factories" to override the
standard Python to C++ type conversions. Blitz++ arrays allow you to
write clean, fast code, but they also are sloooow to compile (20
seconds or more for this snippet). This is why they aren't the default
type used for Numeric arrays (and also because most compilers can't
compile blitz arrays...). <code>inline()</code> is also forced to use
'gcc' as the compiler because the default compiler on Windows (MSVC)
will not compile blitz code. <em> 'gcc' I think will use the standard
compiler on Unix machine instead of explicitly forcing gcc (check this)
</em>Comparisons of the Python vs inline C++ code show a factor of 3
speed
up. Also shown are the results of an "inplace" transpose routine that
can be used if the output of the linear algebra routine can overwrite
the original matrix (this is often appropriate). This provides another
factor of 2 improvement.
<blockquote>
<pre><code>
#C:\home\ej\wrk\scipy\weave\examples> python cast_copy_transpose.py
# Cast/Copy/Transposing (150,150)array 1 times
# speed in python: 0.870999932289
# speed in c: 0.25
# speed up: 3.48
# inplace transpose c: 0.129999995232
# speed up: 6.70
</code></pre>
</blockquote>
<a name="#wxPython" a=""><>
</a>
<h3><a name="#wxPython" a="">wxPython</a></h3>
<code><a name="#wxPython" a="">inline</a></code><a name="#wxPython" a="">
knows how to handle wxPython objects. Thats nice in and of
itself, but it also demonstrates that the type conversion mechanism is
reasonably flexible. Chances are, it won't take a ton of effort to
support special types
you might have. The examples/wx_example.py borrows the scrolled window
example from the wxPython demo, accept that it mixes inline C code in
the middle
of the drawing function. </a>
<blockquote>
<pre><code>
<a name="#wxPython" a=""> def DoDrawing(self, dc):<br> <br> red = wxNamedColour("RED");<br> blue = wxNamedColour("BLUE");<br> grey_brush = wxLIGHT_GREY_BRUSH;<br> code = \<br> """<br> #line 108 "wx_example.py" <br> dc->BeginDrawing();<br> dc->SetPen(wxPen(*red,4,wxSOLID));<br> dc->DrawRectangle(5,5,50,50);<br> dc->SetBrush(*grey_brush);<br> dc->SetPen(wxPen(*blue,4,wxSOLID));<br> dc->DrawRectangle(15, 15, 50, 50);<br> """<br> inline(code,['dc','red','blue','grey_brush'])<br> <br> dc.SetFont(wxFont(14, wxSWISS, wxNORMAL, wxNORMAL))<br> dc.SetTextForeground(wxColour(0xFF, 0x20, 0xFF))<br> te = dc.GetTextExtent("Hello World")<br> dc.DrawText("Hello World", 60, 65)<br><br> dc.SetPen(wxPen(wxNamedColour('VIOLET'), 4))<br> dc.DrawLine(5, 65+te[1], 60+te[0], 65+te[1])<br> ...<br> </a></code></pre>
</blockquote>
<a name="#wxPython" a="">Here, some of the Python calls to wx objects
were just converted to C++ calls. There
isn't any benefit, it just demonstrates the capabilities. You might
want to use this
if you have a computationally intensive loop in your drawing code that
you want to speed up.
On windows, you'll have to use the MSVC compiler if you use the
standard wxPython
DLLs distributed by Robin Dunn. Thats because MSVC and gcc, while
binary
compatible in C, are not binary compatible for C++. In fact, its
probably best, no matter what platform you're on, to specify that <code>inline</code>
use the same
compiler that was used to build wxPython to be on the safe side. There
isn't currently
a way to learn this info from the library -- you just have to know.
Also, at least
on the windows platform, you'll need to install the wxWindows libraries
and link to them. I think there is a way around this, but I haven't
found it yet -- I get some
linking errors dealing with wxString. One final note. You'll probably
have to
tweak weave/wx_spec.py or weave/wx_info.py for your machine's
configuration to
point at the correct directories etc. There. That should sufficiently
scare people
into not even looking at this... :)
</a><a name="Keyword Options"></a>
<h2> Keyword Options </h2>
<p>
The basic definition of the <code>inline()</code> function has a slew
of optional variables. It also takes keyword arguments that are passed
to <code>distutils</code> as compiler options. The following is a
formatted cut/paste of the argument section of <code>inline's</code>
doc-string. It explains all of the variables. Some examples using
various options will follow. </p>
<blockquote>
<pre><code> <br> def inline(code,arg_names,local_dict = None, global_dict = None, <br> force = 0, <br> compiler='',<br> verbose = 0, <br> support_code = None,<br> customize=None, <br> type_factories = None, <br> auto_downcast=1,<br> **kw):<br> </code></pre>
</blockquote>
<code>inline</code> has quite a few options as listed below. Also, the
keyword arguments for distutils extension modules are accepted to
specify extra information needed for compiling.
<blockquote></blockquote>
<h4>inline Arguments:</h4>
<blockquote>
<dl>
<dt>code </dt>
<dd>string. A string of valid C++ code. It should not specify a
return statement. Instead it should assign results that need to be
returned to Python in the return_val. </dd>
<dt>arg_names </dt>
<dd>list of strings. A list of Python variable names that should be
transferred from Python into the C/C++ code. </dd>
<dt>local_dict </dt>
<dd>optional. dictionary. If specified, it is a dictionary of
values that should be used as the local scope for the C/C++ code. If
local_dict is not specified the local dictionary of the calling
function is used. </dd>
<dt>global_dict </dt>
<dd>optional. dictionary. If specified, it is a dictionary of
values that should be used as the global scope for the C/C++ code. If
global_dict is not specified the global dictionary of the calling
function is used. </dd>
<dt>force </dt>
<dd>optional. 0 or 1. default 0. If 1, the C++ code is compiled
every time inline is called. This is really only useful for debugging,
and probably only useful if you're editing support_code a lot. </dd>
<dt>compiler </dt>
<dd>optional. string. The name of compiler to use when compiling.
On windows, it understands 'msvc' and 'gcc' as well as all the compiler
names understood by distutils. On Unix, it'll only understand the
values understoof by distutils. (I should add 'gcc' though to this).
<p>On windows, the compiler defaults to the Microsoft C++
compiler. If this isn't available, it looks for mingw32 (the gcc
compiler). </p>
<p>On Unix, it'll probably use the same compiler that was used
when compiling Python. Cygwin's behavior should be similar.</p>
</dd>
<dt>verbose </dt>
<dd>optional. 0,1, or 2. defualt 0. Speficies how much much
information is printed during the compile phase of inlining code. 0 is
silent (except on windows with msvc where it still prints some
garbage). 1 informs you when compiling starts, finishes, and how long
it took. 2 prints out the command lines for the compilation process and
can be useful if you're having problems getting code to work. Its handy
for finding the name of the .cpp file if you need to examine it.
verbose has no affect if the compilation isn't necessary. </dd>
<dt>support_code </dt>
<dd>optional. string. A string of valid C++ code declaring extra
code that might be needed by your compiled function. This could be
declarations of functions, classes, or structures. </dd>
<dt>customize </dt>
<dd>optional. base_info.custom_info object. An alternative way to
specifiy support_code, headers, etc. needed by the function see the
weave.base_info module for more details. (not sure this'll be used
much). </dd>
<dt>type_factories </dt>
<dd>optional. list of type specification factories. These guys are
what convert Python data types to C/C++ data types. If you'd like to
use a different set of type conversions than the default, specify them
here. Look in the type conversions section of the main documentation
for examples. </dd>
<dt>auto_downcast </dt>
<dd>optional. 0 or 1. default 1. This only affects functions that
have Numeric arrays as input variables. Setting this to 1 will cause
all floating point values to be cast as float instead of double if all
the NumPy arrays are of type float. If even one of the arrays has type
double or double complex, all variables maintain there standard types. </dd>
</dl>
</blockquote>
<h4> Distutils keywords:</h4>
<blockquote> <code>inline()</code> also accepts a number of <code>distutils</code>
keywords for controlling how the code is compiled. The following
descriptions have been copied from Greg Ward's <code>distutils.extension.Extension</code>
class doc-
strings for convenience:
<dl>
<dt>sources </dt>
<dd>[string] list of source filenames, relative to the distribution
root (where the setup script lives), in Unix form (slash-separated) for
portability. Source files may be C, C++, SWIG (.i), platform-specific
resource files, or whatever else is recognized by the "build_ext"
command as source for a Python extension. Note: The module_path file is
always appended to the front of this list </dd>
<dt>include_dirs </dt>
<dd>[string] list of directories to search for C/C++ header files
(in Unix form for portability) </dd>
<dt>define_macros </dt>
<dd>[(name : string, value : string|None)] list of macros to
define; each macro is defined using a 2-tuple, where 'value' is either
the string to define it to or None to define it without a particular
value (equivalent of "#define FOO" in source or -DFOO on Unix C
compiler command line) </dd>
<dt>undef_macros </dt>
<dd>[string] list of macros to undefine explicitly </dd>
<dt>library_dirs </dt>
<dd> [string] list of directories to search for C/C++ libraries at
link time </dd>
<dt>libraries </dt>
<dd> [string] list of library names (not filenames or paths) to
link against </dd>
<dt>runtime_library_dirs </dt>
<dd>[string] list of directories to search for C/C++ libraries at
run time
(for shared extensions, this is when the extension is loaded) </dd>
<dt>extra_objects </dt>
<dd>[string] list of extra files to link with (eg. object files not
implied by 'sources', static library that must be explicitly specified,
binary resource files, etc.) </dd>
<dt>extra_compile_args </dt>
<dd>[string] any extra platform- and compiler-specific information
to use when compiling the source files in 'sources'. For platforms and
compilers where "command line" makes sense, this is typically a list of
command-line arguments, but for other platforms it could be anything. </dd>
<dt>extra_link_args </dt>
<dd>[string] any extra platform- and compiler-specific information
to use when linking object files together to create the extension (or
to create a new static Python interpreter). Similar interpretation as
for 'extra_compile_args'. </dd>
<dt>export_symbols </dt>
<dd>[string] list of symbols to be exported from a shared
extension. Not used on all platforms, and not generally necessary for
Python extensions, which typically export exactly one symbol: "init" +
extension_name. </dd>
</dl>
</blockquote>
<a name="Keyword Option Examples"></a>
<h3> Keyword Option Examples</h3>
We'll walk through several examples here to demonstrate the behavior of
<code>inline</code> and also how the various arguments are used.
In the simplest (most) cases, <code>code</code> and <code>arg_names</code>
are the only arguments that need to be specified. Here's a simple
example
run on Windows machine that has Microsoft VC++ installed.
<blockquote>
<pre><code>
>>> from weave import inline
>>> a = 'string'
>>> code = """
... int l = a.length();
... return_val = Py::new_reference_to(Py::Int(l));
... """
>>> inline(code,['a'])
sc_86e98826b65b047ffd2cd5f479c627f12.cpp
Creating
library C:\DOCUME~1\eric\LOCALS~1\Temp\python21_compiled\temp\Release\sc_86e98826b65b047ffd2cd5f479c627f12.lib
and object C:\DOCUME~ 1\eric\LOCALS~1\Temp\python21_compiled\temp\Release\sc_86e98826b65b047ff
d2cd5f479c627f12.exp
6
>>> inline(code,['a'])
6
</code></pre>
</blockquote>
When <code>inline</code> is first run, you'll notice that pause and
some trash printed to the screen. The "trash" is acutually part of the
compilers
output that distutils does not supress. The name of the extension file,
<code>sc_bighonkingnumber.cpp</code>, is generated from the md5 check
sum
of the C/C++ code fragment. On Unix or windows machines with only
gcc installed, the trash will not appear. On the second call, the code
fragment is not compiled since it already exists, and only the answer
is returned. Now kill the interpreter and restart, and run the same
code with
a different string.
<blockquote>
<pre><code>
>>> from weave import inline
>>> a = 'a longer string'
>>> code = """
... int l = a.length();
... return_val = Py::new_reference_to(Py::Int(l));
... """
>>> inline(code,['a'])
15
</code></pre>
</blockquote>
<p>
Notice this time, <code>inline()</code> did not recompile the code
because it
found the compiled function in the persistent catalog of functions.
There is
a short pause as it looks up and loads the function, but it is much
shorter than compiling would require.
</p>
<p>You can specify the local and global dictionaries if you'd like
(much like <code>exec</code> or <code>eval()</code> in Python), but
if they aren't specified, the "expected" ones are used -- i.e. the ones