proposal: spec: multidimensional slices

[dadkins]

As per Rob and Andrew's request:
https://groups.google.com/forum/#!topic/golang-nuts/Q7lwBDPmQh4

[kisielk]

Comment 1:

Some more discussion on gonum-dev:
https://groups.google.com/d/topic/gonum-dev/WnptzWjqhmk/discussion

[gopherbot]

Comment 2 by nagle@animats.com:

There's been some discussion about this.  Main points so far:
1. The general goal is to make it easy to do Matlab/Octave like work in Go.  Most
   engineering math today is prototyped in one of those languages, then translated
   to something faster for production use.   
2. It should be possible to easily convert standard libraries from "Numerical Methods" 
   (the book series, "Numerical Methods in Fortran, ... in C, in C++ etc.") to Go.
   That would get Go a needed set of standard numerical libraries.  And it should
   be reasonably easy to translate Matlab code to Go code. 
3. It's helpful to have only one standard way to represent matrices.  Otherwise,
   you get math libraries where a matrix coming out of one library won't go into
   another.  (This is one of Matlab's selling points.)
4. Performance is a major issue.  These constructs should go fast, approaching
   or exceeding C/FORTRAN performance.  
Language issues:
1. In general, multidimensional array/slice access syntax would look like "arr[i,j]".
2. Multidimensional slices would be supported, derived from multidimensional arrays.
3. While there is interest in generics and overloading, that's probably too radical.
   However, extending "+", "-", and "*" to vectors and matrices might be reasonable.
   Many machines have hardware that can speed up such operations (MMX, etc.) so
   that's something a compiler can do that a library alone cannot.
4. There's interest in "reshaping", where a subarray or sub-slice is derived from
   an array.  This can easily get overcomplicated (it did in FORTRAN 77) and needs
   to be carefully designed.  Further discussion and use cases are needed for this.
The machine learning community within Google might be consulted on what they want in
this area.  The machine learning theorists write in Matlab, and then someone has to
make that work in production code. 
It would be helpful to get agreement on needed functionality before putting too much
effort into syntax, so there are few specific syntax suggestions here yet.

[adg]

Comment 3:

Thanks for filing the issue.

Labels changed: added feature, languagechange, removed priority-triage.

Status changed to Thinking.

[dadkins]

Comment 5:

My desires are modest. I'd like to be able to write something like
// LUPDecompose performs an in-place LU factorization of a square
// matrix A. It returns a permutation vector P, such that PA = LU.
// On completion, A[i][j] = L[i][j] if i > j, U[i][j] if i <= j.
func LUPDecompose(A [][]float64) (P []float64, err error)
When writing this kind of code in C, I used to define a macro
#define A_(i,j) (A[(i)*rowsep + (j)])
That is precisely the kind of thing I wish the compiler would do for me.

[gopherbot]

Comment 6 by gexarcha1:

This document contains a collection of different proposals described in different levels
of detail:
https://docs.google.com/document/d/1gejfouITT25k29eHYTgdvi4cCLVt5dhYiJVvF46gp78/edit
It also contains links to any proposal or information relevant to the topic that we
found interesting. Feel free to edit the Document to clarify any questions relevant to
implementation.
I think most of the people in need of multi dimensional arrays in Go are not used to
writing proposals for programming language changes. If you have any input that would
make the process less awkward it would be highly appreciated.

[gopherbot]

Comment 7 by nagle@animats.com:

There's been considerable discussion of this subject on "comp.lang.go.general".
Areas of agreement:
   There seems to be a consensus that multidimensional arrays and slices in Go would be useful for numerical work.
   The array element access syntax generally proposed is 
       arr[n,m]     // 2D
       arr[n,m,o]   // 3D
etc.  Existing arrays of arrays would be retained in Go. Arrays of arrays can be
"ragged",
(rows of different length) but multidimensional arrays are always of uniform length in
each dimension, allowing FORTRAN-type optimizations. 
   Both fixed-size multidimensional array types defined at compile time and arrays sized
at run time have use cases.  The former are widely used for graphics, and the latter are
widely used for general numerical work.  It should be possible to pass both as parameters
to functions, although not necessarily the same functions.
Areas of disagreement:
   There's much disagreement over what facilities for reslicing and reshaping should
be provided. 
   The minimum capability discussed is the ability to pass an entire array or 
multidimensional slice object to a function.  Creating a slice which refers
to a portion of an array, though, is controversial. Syntactically, this refers
to expressions of the form
         arr[a0:a1, b0:b1]
The problem is that values of b0 and b1 which are smaller than the array bounds
result in the need to represent a kind of sparse array, where the memory
distance between adjacent elements is not the same as the size of an element.
This complicates the internal representation of slices; even the lowest dimension
must have a "stride" value.  There's a performance penalty for this feature when it
not being used.  But it is convenient to have.  
   The other big issue is the ability to "grow" multidimensional slices. 
Should "append" be allowed for multidimensional arrays, and if so, how
general should it be?  Is growth allowed in all dimensions?  It's a useful
capability but complicates the implementation considerably. 
   Those are the two biggest issues being discussed.
   Lesser issues include whether operators "+", "-", and "*" should be provided
as built-in operations for multidimensional arrays and slices.  Providing them
permits easy transliteration for math libraries in other languages (C++, Matlab
especially).  But they're not essential; NumPy uses "A.dot(B)" for matrix multiply.
There are a lot of cases to handle for "*" - vector*scalar, vector*matrix, 
matrix*matrix...  "+" and "-" are somewhat simpler.
   That's roughly where the discussions are.  Once the slice and growth issues are
decided, the rest should fall into place.

[adg]

Comment 8:

The discussion referred to by nagle@ is on golang-nuts ("comp.lang.go.general" is just a
name made up by gmane.org):
https://groups.google.com/forum/#!topic/golang-nuts/Q7lwBDPmQh4

[gopherbot]

Comment 9 by norman.yarvin:

My proposal, at
https://docs.google.com/document/d/1xHyOK3hSxwMtyMH-pbXSXxo7Td9A_Teoj6cbr9ECeLM/edit?pli=1#
has now reached a reasonably mature state.  I'd appreciate if people had a look at it. 
It's a rather long proposal, but that's because this is not a small addition to the
language and I've tried to be thorough.  The proposal is purely about the basics of
multidimensional slices; math operations on whole slices are not part of it.  (It does
not preclude later adding them, but does not set up a situation where they'd be
necessary, either.)
As regards comments that people have made so far on the document, please keep in mind
that most of them were made against earlier versions, in which some things were not
explained as well.  Thus if any appear to be foolish that should not be held against the
commenter, or at least not too much.  (Though some comments have been marked as
resolved, I haven't marked them that way just because I disagree and have improved my
explanation, as that seems too dictatorial; I figure people can retire their own
comments if their objections have truly been addressed.)
As a personal comment, it is rather rare to have an opportunity to add a major feature
to a language so cleanly.  Usually one either has to break compatibility or introduce
some serious ugliness or severe compromise.  But here it's almost like Go was designed
for this feature to just be plugged in and almost seamlessly fill a gap in its
capabilities.  Or so I believe; go ahead, hammer on the proposal and prove me wrong.

[gopherbot]

Comment 10 by A.Vansteenkiste:

In my opinion, instead of changing the language, it might be useful if the standard
library simply provided a standard matrix format. That should guide the creation of
idiomatic, compatible matrix libraries. Cfr. database/sql, which guided the creation of
excellent database drivers.
The image package already provides nice guidance on how to handle 2D matrices. It uses
contiguous underlying storage which is needed when passing to external libraries. I
modelled a matrix (storage) package after image, this is how it looks like:
http://godoc.org/github.com/barnex/mat. However, many people wrote such kind of packages
with slightly different, incompatible, conventions. Hence it might be nice to have on a
standard convention in the standard library, once and for all.

[rsc]

Comment 11:

Labels changed: added go1.3maybe.

[rsc]

Comment 12:

Labels changed: removed feature.

[rsc]

Comment 13:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 14:

Labels changed: added repo-main.

[btracey]

Comment 15:

About a month ago, I put together a proposal for this. Putting this comment as a
reference.
Here is a reduced proposal for multi-dimensional arrays
https://docs.google.com/document/d/1eHm7KqfKP9_s4vR1zToxq-FBazdUQ9ZYi-YhcEtdfR0/edit
Golang-nuts discussion thread
https://groups.google.com/forum/#!searchin/golang-nuts/tables$20in$20go/golang-nuts/osTLUEmB5Gk/3-f9_UKfE9MJ
In the thread, Dan Kortschak proposed a nice syntax for range which could be an addition
to the proposal (https://groups.google.com/d/msg/golang-nuts/osTLUEmB5Gk)/-A15bJpuTzsJ

[MichaelTJones]

Comment 16:

Related discussion in my go-nuts post:
https://groups.google.com/forum/#!topic/golang-nuts/ScFRRxqHTkY