std.experimental.ndslice [ready to merge] #3397
Conversation
|
ping @John-Colvin @kyllingstad |
|
put Example 0 in the unittests the is also some spacing and brace placement issues |
|
I'd love to get stuck in to this immediately, but I'm completely swamped with other work. I don't think we should be targeting phobos so quickly. Whatever the design, it needs to get some proper use before it gets set in stone. I would love to have it as the heart of https://github.com/DlangScience (see my DConf talk) for a start and then see how well it works out. I imagine a Then a matrix type can be trivially built on top of this. |
|
I completely agree. |
9il
changed the title
|
First release http://code.dlang.org/packages/dip80-ndslice. |
|
There is problem to build |
|
Why can’t you get away with N-1 strides? In terms of the underlying range, one of the N strides is always 1, no?
|
|
E.g. NxN has 2 strides, (1,N), but you only need store one of them, (N). The transpose of NxN has two strides, (N,1), but again you only need one plus a flag to say "this is transposed". |
|
In unittest Properties and methods, subsection stride property: auto tensor = 100.iota.array.sliced(3, 4, 5);
assert(tensor.stride == 20);
assert(tensor.stride!0 == 20);
assert(tensor.stride!1 == 5);
assert(tensor.stride!2 == 1);
//`matrix` can be casted to BLASMatrix
// strides: (1, 5)
auto matrix = tensor.back;
assert(matrix.stride == 5);
assert(matrix.stride!1 == 1);
//Runtime error if casting `matrix` to BLASMatrix:
// strides: (20, 5)
matrix = tensor.back!2;
assert(matrix.stride == 20);
assert(matrix.stride!1 == 5); |
|
Ah, yes, I see now, my mistake. |
9il
changed the title
|
This is awesome. |
I can add
The order of dimensions can be changed with generalized auto t0 = 1000.iota.sliced(3, 4, 5);
auto t1 = t0.transposed!(2, 0, 1); //CTFE - recommended
auto t2 = t0.transposed (2, 0, 1); //Runtime
assert(t0[1, 2, 3] == t1[3, 1, 2]);
assert(t0[1, 2, 3] == t2[3, 1, 2]);
static assert(is(typeof(t0) == typeof(t1)));
static assert(is(typeof(t0) == typeof(t2))); |
|
@9il I think what @TurkeyMan wants is a byElement method. Essentially it's just a byRow.joiner (generalised for n-dims obviously), but joiner doesn't know about rectangular-ness so you sadly only get a forward range. |
|
Instead of offering All the pointer specialisation is just premature optimisation that obscures the code. Lazy matrix operations could really take this work to the next level. The bare-bones of the design I had come up with is as follows: a = b; //shallow copy
a[] = b; //deep copy
auto c = <op>a;
//auto c = Slice!(N, typeof(a._range.map!"<op>a"))(a.lengths, a.strides, a._range.map!"<op>a");
auto d = c <op> b;
//auto d = Slice!(N, typeof(zip(c, b).map!"a[0]<op>a[1]"))(a.lengths, a.strides, zip(c, b).map!"a[0]<op>a[1]");
a <op>= b; //error
a[] <op>= b;
//(<op>b).byElement.copy(a.byElement);The comments are just for showing the semantics, obviously they could be implemented in a more specialised way. Also, any operands could be of ElementType!Range as well, e.g. adding a number to every element. The reason why this is so great, is that it saves hugely on temporaries. Temporary allocations and copies are the bane of my life when working with traditional array-based scientific tools like numpy and matlab. This is very in keeping with Walter's "don't allocate" message. |
OK, we can add both:
foreach(elem; tensor.byElement) {...}
foreach(i, j, k, elem; tensor3D) {...} |
Why not implement an n-dimensional version of |
foreach(elem; tensor.byElement) {...}
foreach(ref elem; tensor.byElement) {...}_2. _3. or this... |
|
I'll just state my use cases, and the most appropriate solution may follow. I think 1, 2, and 3 are all useful. You say opApply is slow above? Why is that? |
TypeTuple!(size_t[N], "lengths", size_t[N], "strides") shape() @property @safe pure nothrow @nogc
Already works.
Already works.
It is very ambiguous syntax. auto c = a.range.map!"fun>".sliced(a.shape.lengths);What about
ditto
Already works (error).
Already works. (Implemented in more specialised way)
I don't understand this bit.
Agreed. However your syntax looks like "I allocate" instead of "I don't allocate". |
|
EDIT: auto c = a.range.map!"<fun>".sliced(a.shape); |
Because it calls delegate. For example: if you want to set up some integer slice to simple funciton from indexes it would be approximately two times slower then naive chain of What do you think about foreach(i, j, k, pointer; tensor3D.byIndexedPointer)
{
pointer* = i*j+k;
} |
|
R.e. slicedMap, slicedZip, ambiguous syntax: It isn't super-explicit, but it is very usable. Consider this sort of function: auto imageAdjust(S0, S1)(S s, float gamma, float preOffset, float postOffset, S1 mask)
{
return (postOffset + (s + preOffset)^^gamma) & mask;
}The maths is directly visible but no allocations needed (compared to the same in numpy, where you'd be allocating like crazy). Compare to the same without the syntax sugar: auto imageAdjust(S0, S1)(S s, float gamma, float preOffset, float postOffset, S1 mask)
{
return slicedZip(s, mask).map!(x => (postOffset + (x[0] + preOffset)^^gamma) & x[1]);
}And then consider how it scales are more Slices are involved. Also, note that the lambda in the non-sugar version would have to be allocated as a closure. I'm not sure how I would sell D to a normal data scientist if I told them that they can't write simple mathematical expressions on multidimensional arrays without using template lambda functions, declaring all the arrays involved up-front and then referring to them in the expression by indexing a Tuple. The other big win of supporting these basic algebraic operations is that it allows some code to be written once and support everything from scalars through to 1000-dim arrays. The |
|
What's the advantage of the pointer api vs ref? I don't think foreach syntax and pointers should mingle. |
|
Another thought; Walter dropped the ideas of some lowering for array syntax to range functions by the compiler in his talk... consider how those ideas interact with multi-dimensional arrays? |
Nothing in terms of syntax. Only speed (Tuple vs delegates). OK, lets droop |
@TurkeyMan Please describe what do you mean. |
|
Just a heads-up that I've edited my previous post to clarify some things, I'll delete this message shortly. |
9il
changed the title
@JackStouffer Thank you for review management! |
|
I have removed compiler version check. It can be merged IMO. |
|
|
||
| ///ditto | ||
| Slice!(N, Range) transposed(size_t N, Range)(auto ref Slice!(N, Range) slice, size_t dimension) | ||
| in { |
|
I've been largely absent from Phobos development lately because of other commitments. The current PR contains at least one object file, though, which should be removed completely from history. |
Thanks! Fixed and rebased |
|
Merge failed for DAutoTest. In the same time |
|
DMD 2.070 is going to be branched 1st Jan. @andralex @MartinNowak @klickverbot @kyllingstad @burner please make a decision about Notes:
Thanks! |
Fixed now (and one step closer to fixing this permanently), thanks for letting me know. Happy new year. |
|
@CyberShadow Thanks! Happy new year) |
|
I still think that the docs need improvements |
OK, if this is hard constraint, then a decision maker from Core team can close this PR. |
|
@CyberShadow DAutoTest failed again . |
I am not much involved in this project but I think docs can be imploved post release. As long as the design is sound and backwards-incompatible changes are unlikely, it could be merged.
Fixed |
fix Slice change concept implement `sliced` change structure remove comment slice constructors add 1D case fix opSlice 1D implement ND, part1 update to modern syntax generic opIndex fix protection cleanup fix cleanup style fix rename update update implement transpose unittest unittest for properties and methods more examples & fixes move code minor fix add header update make fils update make files dmd bug workaround fix module name update imports style fix fix asserts remove createRefCountedSlice add shape property rework `shape`, add `structrure` ndarray swapped transposed opCast remove save calls everted make Slice(size_t N0, Slice(size_t N1, ...)) virtual huge rework remove space move code fix style add packEverted relax constraints add black comments remove spaces fix macros fix doc fix docs update docs reduce template bloat optimize and fix for arrays. update docs remove space update docs update docs update link update doc fix constructor add toString fix `save` add `reversed` method fix constraints for `reversed` optimisation in unittests add `strided` implement `reversed` for ranges (no arrays) reduce constraints cleanup add Iota for static foreach remove string representation DMD 2.067 support fix style fix opIndexUnary constraints move byElement add shift property add CTFE-able strided huge style fix add macros update `sliced` move private code update docs update allIndexesReversed, renamed to allReversed update docs update docs fix macros fix posix.mak update posix.mak update docs and strided move code update sliced constraints update `sliced` docs move code remove whitespaces add static assert to ndslice add one more opIndexUnary update createSlice docs update ndarray update docs for Transpose operators update docs and fix bugs add pragma inline fix docs update docs update docs add inline pragma ditto replace errors with asserts update docs update doc style update docs remove comes in docs update docs remove whitespaces update docs update docs update comment update test update docs fix typo fix docs review fix transposed description change doc tables remove unused RepeatTypeTuple remove function attributes for templates make constructor private make N and Range private make elementsCount public fix createSlice params naming add assert description to sliced [big] fix range primitives wrong asserts and update documentation regroup primitives minor docs fix minor docs fix fix typo fix Slice constraints add indexing using static arrays make byElement a random access range fix by random access primitives for ByElement update unittest fix random access primitives for ByElement remove trailing space implement slicing for ByElement make ByElement more readable update docs for subspace operators remove one See_also revert last commit update docs add descriptions to asserts add more examples minor doc update add example with allocators add makeSlice for allocators update docs table add range checks add more constructors Add description to asserts. add checks for const/immutable ditto update to DMD 2.069 minor fixes add elements property make makeSlice an unittest remove space update docs remove space update docs update doc fix makeSlice example fix strided make strided template variadic add Guide for Slice/Matrix/BLAS contributers remove unused import add better error messages update guide update docs remove space [minor] fix docs minor error messages update minor doc fix rename package split package update posix.mak update win*.mak ditto fix posix mak update *mak. update docs update man files minor doc update update module headers fix opCast rename pop*N to pop*Exactly remove primitives add popFrontN popBackN to Slice [minor] update docs add package primitives to Slice update operators prototypes add drop* primitives [minor] update docs update docs remove spaces remove allocators minor doc fix [minor] update docs add dropToNCube add diagonal add return type for template for better documentation move pack* to iterators rm allocators [minor] doc update add support of packed slice for diagonal update example for diagonal add blocks rename packed and unpacked to pack and unpack update docs renaming [minor] docs update ditto minor style update rm old files [minor] update docs update docs ditto [minor] update docs add rotated [minor] update docs add byElementInStandardSimplex add windows remove space remove structure update docs add spaces add reshape rename packEverted -> evertPack remove spaces fix ReshapeException minor doc fix update windows/blocks template params fix pack windows/blocks add index @Property remove spaces minor doc fix add Slice tuples remove spaces update docs update docs and rename dropToHypercube update docs minor doc update fix links remove version OS constraints for allocators assumeSameStructure fix minor doc fix minor doc update after review minor style fix fix Elaborate Assign add index slice fix NSeq fix bug with diagonal fix sliced slice add main example update docs translation fix comment fix style remove spaces update style fix link Vandermonde matrix remove space move example remove `opCast` add opEquals for arrays update opIndex(Op)Assign update docs update docs fix style update docs (russian will be translated) update tests fix doc style ditto ditto update docs update docs update docs update docs update unittests update docs ditto ditto ditto [major] doc update update docs update docs fix voting conditions (docs, style) minor doc update fix style add unittest for `map` ditto fix string mixins add Params and Returns ditto add headers add descriptions Fix m32 mode in example Minor description fix fix spaces ditto Add Internal Binary Representation ditto ditto ditto ditto ditto ditto ditto add description for binary representation ditto minor style fix ditto ditto ditto ditto ditto ditto inlining remove compiler version check fix braces fix docs add Quick_Start Fix English Add two examples fix style remove object file minor doc update ditto remove spaces fix indexing & add unittests ditto
I think @9il will have a hard time to improve the docs on it's own, and it's something that's better done collectively while working the module from std.experimental to std. |
|
That said the documentation doesn't need a few improvements but a complete restructure. There is no clear introduction/overview, just the table for the submodules. When I try to understand the example (which seems a bit too hard for an intro) I'm already stuck trying to understand what pack does. |
|
Auto-merge toggled on |
|
Let's move on with this, there has hardly been a module w/ more support by the community, and as there are many users (and already articles in preparation), I'm pretty confident, we can get the documentations in shape soon. |
std.experimental.ndslice [ready to merge]
|
That's a really crappy commit message (9il@54a6d72) 😞. |
|
Whoohoooo! |
|
According to http://digger.k3.1azy.net/trend/, this caused a 85KiB increase in the hello world binary!? Is DMD really including symbols from this when it's never imported? |
@JackStouffer wasn't it dlang/druntime#1453 ? edit: I had to zoom in many times and then found: dlang/dmd#5324 |
DUB package is available at Mir library: link
Documentation: link
Voting thread: http://forum.dlang.org/thread/nexiojzouxtawdwnlfvt@forum.dlang.org
Review thread: http://forum.dlang.org/thread/uesnmkgniumswfclwbgt@forum.dlang.org
Old announce at forum.
Optimisation check
Source: https://gist.github.com/9il/bc7966823d96557c566c
LDC disassembled: https://gist.github.com/9il/47aea1621a9fba609869 (all functions are inlined!)
DMD disassembled: https://gist.github.com/9il/a5c0ee9bdb4ddd25c4d6
TODO list
std.experimental.ndsliceand split.byElementa random access rangebyElementa range with slicingstridedreshapediagonalSliceblocksSlicerotatedSliceindexproperty to allby*ElementIteratorsbyElementInStandardSimplexwindows- moving window slicepop*multidimensional range primitivesdrop*multidimensional range primitivesdropToNCubeassumeSameStructureabstractionindexSliceopIndexAssignfor D arraysopIndexOpAssignfor D arraysvander