Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

array expressions #144

Open
wants to merge 56 commits into from

2 participants

Mark Florisson Robert Bradshaw
Mark Florisson
Collaborator

Element-wise array expressions for Cython

added some commits May 21, 2012
Mark Florisson Split buffer and memoryview operations from IndexNode fbae57b
Mark Florisson Split analyse_types of IndexNode some more 26d7c28
Mark Florisson Fix copy/copy_fortran return type & tests 1df5077
Mark Florisson Refactor memoryview dtype validation 98d0d60
Mark Florisson Fix transpose memoryview type 36f037b
Mark Florisson Refactor slice assignment and copying 922af21
Mark Florisson Fix python 3 import 9536b62
Mark Florisson Start on elementwise expressions 6244394
Mark Florisson Allow strided memoryviews to be copied to contiguous memoryviews 148c290
Mark Florisson Specialize for contiguous operands 9b0fc9e
Mark Florisson Add minivect as submodule 1217a97
Mark Florisson Choose typedefed type in widest_numeric_type & support tyepdef Cython…
… types in vector expressions
3fdcc52
Mark Florisson Allow arbitrary Cython types compatible with C expressions as vector …
…expression dtypes
31460e8
Mark Florisson Support complex numbers in vector expressions 1e53ae2
Mark Florisson Support objects in vector expressions 451c802
Mark Florisson Use array funcarg method f4496bc
Mark Florisson Select specialization at runtime & support broadcasting b8ab0f3
Mark Florisson Support overlapping memory and broadcasting assignment
    (Avoid recomputation along broadcasting axes)
    (Need some tests)
c7db788
Mark Florisson DECREF lhs in slice assignment w/ dtype object c4e72cf
Mark Florisson Don't choose contig specialization for broadcasting operations known …
…at compile time
e86e5aa
Mark Florisson Remove broadcasting leading dimensions from RHS 48672b0
Mark Florisson Split vector expression tests & add runtime broadcasting test bc22f9f
Mark Florisson Select specialization at runtime 32e6671
Mark Florisson Support tiling specializations aae2d35
Mark Florisson Let the LHS participate in determining array layout order b13c9e2
Mark Florisson Implement a code cache 46fe298
Mark Florisson Support scalar arguments a02e733
Mark Florisson Support unary operations d640297
Mark Florisson Support non-slice array expression assignment 62f8783
Mark Florisson Support non-assignment array expressions ce10532
Mark Florisson Fix broadcasting & add omitted tests 0653f04
Mark Florisson Support restrict and const qualifiers for array expression arguments 6a6e9a7
Mark Florisson Support inner-contiguous specialization selection 94ddebd
Mark Florisson Map more types to minivect, change qualify to use immutable types e19b8a7
Mark Florisson Support OpenMP in array expressions d22b3c4
Mark Florisson Filter duplicate expression arguments ef46dfd
Mark Florisson Add auto-tuner for square tiling blocksize (+caching) 2182b34
Mark Florisson Name mangle all variable and function names in the tiling utility code 842952f
Mark Florisson Make tiling blocksize type signed (avoid compiler warnings about sign…
…ed/unsigned comparison)
cbe47f8
Mark Florisson Divide tiling blocksize by itemsize 5c8409c
Mark Florisson Auto-tune OpenMP size e413b45
Mark Florisson Test various binary operators dd455de
Mark Florisson Support (partial) elementwise function calls b42b157
Mark Florisson Avoid tiled specializations if all operands are C or F contig d41d090
Mark Florisson Support vectorized specializations ebc8850
Mark Florisson Add support for graphviz AST visualisation c596986
Mark Florisson Use more sensible output filenames for graphviz .dot files 7c5f2f2
Mark Florisson Only print debug code for expression 1ef2ebd
Mark Florisson Add variable resolving mixin class adbb911
Mark Florisson Fix code caching for array expressions 53b77f7
Mark Florisson Omit type information for graphviz 1719bdb
Mark Florisson Resolve non-external typedefs for array expressions d146ac7
Mark Florisson Add some more documentation 4ae867f
Mark Florisson Create minivect function type for function calls bdaeea5
scoder scoder referenced this pull request October 12, 2012
Closed

refactor indexnode2 #137

Robert Bradshaw robertwb commented on the diff October 15, 2012
Cython/Compiler/Vector.py
((24 lines not shown))
  24
+
  25
+# Print generated minivect code
  26
+_debug = False
  27
+# Generate debug calls from specialized functions
  28
+_context_debug = False
  29
+
  30
+#
  31
+### Graphviz related things. .dot files are only written when write_graphviz
  32
+### is true.
  33
+#
  34
+graphviz_out_filename_unspecialized = os.path.expanduser("~/ast.dot")
  35
+graphviz_out_filename = os.path.expanduser("~/ast%s.dot")
  36
+write_graphviz = False
  37
+
  38
+# Macro that should be defined to enable explicit vectorization
  39
+cython_vector_size = "CYTHON_VECTOR_SIZE"
2
Robert Bradshaw Owner
robertwb added a note October 15, 2012

Meaning of this value?

Mark Florisson Collaborator

Compiling with -DCYTHON_VECTOR_SIZE=X enables vectorization, with 4 meaning SSE2 (4 floats), and 8 meaning AVX (8 floats).

The new branch is called _array_expressions_rebased, but the py3k build segfaults with a refcount error. I still need to investigate that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Robert Bradshaw robertwb commented on the diff October 15, 2012
Cython/Compiler/Vector.py
((43 lines not shown))
  43
+    Map Cython types to minitypes.
  44
+    """
  45
+
  46
+    def map_type(self, type, wrap=False):
  47
+        if type.is_typedef:
  48
+            if type.typedef_is_external:
  49
+                return minitypes.TypeWrapper(type, self.context)
  50
+            else:
  51
+                type = type.resolve()
  52
+
  53
+        if type.is_memoryviewslice:
  54
+            dtype = self.map_type(type.dtype, wrap=wrap)
  55
+            return minitypes.ArrayType(dtype, len(type.axes),
  56
+                                       is_c_contig=type.is_c_contig,
  57
+                                       is_f_contig=type.is_f_contig)
  58
+        elif type.is_float:
5
Robert Bradshaw Owner
robertwb added a note October 15, 2012

Would these be better expressed as a dictionary?

Mark Florisson Collaborator

Yes I think that would be nicer.

Mark Florisson Collaborator

Oh I remember why I didn't do that now. It's because you can create new types, and for instance c_sint_type and c_int_type are the same, but don't compare equal (I'm not sure why?).

Robert Bradshaw Owner
robertwb added a note October 29, 2012

Like typedefs I guess. Does this then assume you know the exact size at compile time? Hmm...

Mark Florisson Collaborator

No, it needs that only for the LLVM backend (which isn't used by Cython). It uses the external typedef name if the typedef is external, otherwise it uses the actual types to allow code reuse of expressions of equivalent types. I got everything working, except for one pyximport test, but I think that might be my test setup. I'll try a new PR next weekend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Robert Bradshaw robertwb commented on the diff October 15, 2012
Cython/Compiler/Vector.py
((959 lines not shown))
  959
+            code.putln("else {")
  960
+
  961
+        code.putln("/* Strided specializations */")
  962
+        self.put_ordered_specializations(code, specializers.StridedSpecializer,
  963
+                                         specializers.StridedFortranSpecializer)
  964
+
  965
+        if if_clause != "if":
  966
+            code.putln("}")
  967
+
  968
+    def generate_result_code(self, code):
  969
+        "Generate a branch and call to each specialization"
  970
+        contig, mixed_contig, c_contig, f_contig = all_c_or_f_contig(self.operands)
  971
+
  972
+        self.context.original_cython_code = code
  973
+
  974
+        if_clause = "if"
1
Robert Bradshaw Owner
robertwb added a note October 15, 2012

unused code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Robert Bradshaw
Owner

In my (albeit somewhat cursory) perusal of the code, I think it looks good. Is there any way to test that your code is being used (i.e. unnecessary intermediates are not being created) as well as getting the correct output? (Perhaps an object type that prints out arithmetic operations could verify the expected sequence of operations, though this might be too much subject to change). Also, it'd be good to have a link somewhere to your thesis or other high-level overview.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 56 unique commits by 1 author.

Jul 23, 2012
Mark Florisson Split buffer and memoryview operations from IndexNode fbae57b
Mark Florisson Split analyse_types of IndexNode some more 26d7c28
Mark Florisson Fix copy/copy_fortran return type & tests 1df5077
Mark Florisson Refactor memoryview dtype validation 98d0d60
Mark Florisson Fix transpose memoryview type 36f037b
Mark Florisson Refactor slice assignment and copying 922af21
Mark Florisson Fix python 3 import 9536b62
Mark Florisson Start on elementwise expressions 6244394
Mark Florisson Allow strided memoryviews to be copied to contiguous memoryviews 148c290
Mark Florisson Specialize for contiguous operands 9b0fc9e
Mark Florisson Add minivect as submodule 1217a97
Mark Florisson Choose typedefed type in widest_numeric_type & support tyepdef Cython…
… types in vector expressions
3fdcc52
Mark Florisson Allow arbitrary Cython types compatible with C expressions as vector …
…expression dtypes
31460e8
Mark Florisson Support complex numbers in vector expressions 1e53ae2
Mark Florisson Support objects in vector expressions 451c802
Mark Florisson Use array funcarg method f4496bc
Mark Florisson Select specialization at runtime & support broadcasting b8ab0f3
Mark Florisson Support overlapping memory and broadcasting assignment
    (Avoid recomputation along broadcasting axes)
    (Need some tests)
c7db788
Mark Florisson DECREF lhs in slice assignment w/ dtype object c4e72cf
Mark Florisson Don't choose contig specialization for broadcasting operations known …
…at compile time
e86e5aa
Mark Florisson Remove broadcasting leading dimensions from RHS 48672b0
Mark Florisson Split vector expression tests & add runtime broadcasting test bc22f9f
Mark Florisson Select specialization at runtime 32e6671
Mark Florisson Support tiling specializations aae2d35
Mark Florisson Let the LHS participate in determining array layout order b13c9e2
Mark Florisson Implement a code cache 46fe298
Mark Florisson Support scalar arguments a02e733
Mark Florisson Support unary operations d640297
Mark Florisson Support non-slice array expression assignment 62f8783
Mark Florisson Support non-assignment array expressions ce10532
Mark Florisson Fix broadcasting & add omitted tests 0653f04
Mark Florisson Support restrict and const qualifiers for array expression arguments 6a6e9a7
Mark Florisson Support inner-contiguous specialization selection 94ddebd
Mark Florisson Map more types to minivect, change qualify to use immutable types e19b8a7
Mark Florisson Support OpenMP in array expressions d22b3c4
Mark Florisson Filter duplicate expression arguments ef46dfd
Mark Florisson Add auto-tuner for square tiling blocksize (+caching) 2182b34
Mark Florisson Name mangle all variable and function names in the tiling utility code 842952f
Mark Florisson Make tiling blocksize type signed (avoid compiler warnings about sign…
…ed/unsigned comparison)
cbe47f8
Mark Florisson Divide tiling blocksize by itemsize 5c8409c
Mark Florisson Auto-tune OpenMP size e413b45
Mark Florisson Test various binary operators dd455de
Mark Florisson Support (partial) elementwise function calls b42b157
Mark Florisson Avoid tiled specializations if all operands are C or F contig d41d090
Mark Florisson Support vectorized specializations ebc8850
Mark Florisson Add support for graphviz AST visualisation c596986
Mark Florisson Use more sensible output filenames for graphviz .dot files 7c5f2f2
Aug 01, 2012
Mark Florisson Only print debug code for expression 1ef2ebd
Aug 05, 2012
Mark Florisson Add variable resolving mixin class adbb911
Aug 11, 2012
Mark Florisson Fix code caching for array expressions 53b77f7
Aug 13, 2012
Mark Florisson Omit type information for graphviz 1719bdb
Aug 21, 2012
Mark Florisson Resolve non-external typedefs for array expressions d146ac7
Aug 24, 2012
Mark Florisson Add some more documentation 4ae867f
Sep 16, 2012
Mark Florisson Create minivect function type for function calls bdaeea5
Oct 14, 2012
Mark Florisson Remove minivect submodule, fix C++ type cast 6640e99
Mark Florisson Support Cython under pdb for python 2.4 f9b33c4
Something went wrong with that request. Please try again.