Issue 6798 - Integrate overloadings for multidimensional indexing and slicing #443

Merged
merged 10 commits into from Mar 16, 2014

Conversation

Projects
None yet
@9rnsr
Member

9rnsr commented Oct 10, 2011

http://d.puremagic.com/issues/show_bug.cgi?id=6798

This patch is additional enhancement of opDollar (issue 3474 and #442).
Enable the mixing operator overloadings of indexing and slicing

a[$-1, 2..$] is translated to a.opIndex(a.opDollar!0 - 1, a.opSlice!1(2, a.opDollar!1))

The slicing lwr..upr inside bracket is converted to a.opSlice!(dimension)(lwr, upr).
This enhancement never break existing codes.

expressionnewly added overloadingexists overloading
a[i0, ...]a.opIndex(i0, ...)
a[]a.opIndex()a.opSlice()
a[l..u]a.opIndex(a.opSlice!0(l, u))a.opSlice(l, u)
a[l..u, ...]a.opIndex(a.opSlice!0(l, u), ...)
op a[i0, ...]a.opIndexUnary!op(i0, ...)
op a[]a.opIndexUnary!op()a.opSliceUnary!op()
op a[l..u]a.opIndexUnary!op(a.opSlice!0(l, u))a.opSliceUnary!op(l, u)
op a[l..u, ...]a.opIndexUnary!op(a.opSlice!0(l, u), ...)
a[i0, ...] = va.opIndexAssign(v, i0, ...)
a[] = va.opIndexAssign(v)a.opSliceAssign(v)
a[l..u] = va.opIndexAssign(v, a.opSlice!0(l, u))a.opSliceAssign(v, l, u)
a[l..u, ...] = va.opIndexAssign(v, a.opSlice!0(l, u), ...)
a[i0, ...] op= va.opIndexOpAssign!op(v, i0, ...)
a[] op= va.opIndexOpAssign!op(v)a.opSliceOpAssign!op(v)
a[l..u] op= va.opIndexOpAssign!op(v, a.opSlice!0(l, u))a.opSliceOpAssign!op(v, l, u)
a[l..u, ...] op= va.opIndexOpAssign!op(v, a.opSlice!0(l, u), ...)
@braddr

This comment has been minimized.

Show comment
Hide comment
@braddr

braddr Dec 29, 2011

Member

This patch is failing on all 64 bit platforms:

http://d.puremagic.com/test-results/pull.ghtml?runid=12403
../druntime/import/core/stdc/stdarg.di(250): Error: cannot implicitly convert expression (parmn[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(250): Error: cannot implicitly convert expression (p[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(250): Error: expression p[(__error)] is void and has no value
../druntime/import/core/stdc/stdarg.di(283): Error: cannot implicitly convert expression (parmn[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(283): Error: cannot implicitly convert expression (p[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(283): Error: expression p[(__error)] is void and has no value
../druntime/import/core/stdc/stdarg.di(292): Error: cannot implicitly convert expression (parmn[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(292): Error: cannot implicitly convert expression (p[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(292): Error: expression p[(__error)] is void and has no value
std/format.d(3973): Error: template instance core.stdc.stdarg.va_arg!() error instantiating

Member

braddr commented Dec 29, 2011

This patch is failing on all 64 bit platforms:

http://d.puremagic.com/test-results/pull.ghtml?runid=12403
../druntime/import/core/stdc/stdarg.di(250): Error: cannot implicitly convert expression (parmn[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(250): Error: cannot implicitly convert expression (p[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(250): Error: expression p[(__error)] is void and has no value
../druntime/import/core/stdc/stdarg.di(283): Error: cannot implicitly convert expression (parmn[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(283): Error: cannot implicitly convert expression (p[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(283): Error: expression p[(__error)] is void and has no value
../druntime/import/core/stdc/stdarg.di(292): Error: cannot implicitly convert expression (parmn[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(292): Error: cannot implicitly convert expression (p[0LU..tsize]) of type void[] to ulong
../druntime/import/core/stdc/stdarg.di(292): Error: expression p[(__error)] is void and has no value
std/format.d(3973): Error: template instance core.stdc.stdarg.va_arg!() error instantiating

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr May 31, 2012

Member

Revamped changes.

Member

9rnsr commented May 31, 2012

Revamped changes.

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Sep 24, 2012

Member

(background: I'm doing a review of pull requests < 1000 as they tend to be controversial)

This is nice, self-consistent, and tastefully designed.

The question is scope and intended usage. This is a relatively hefty language addition, with a very narrow clientele. I'm not sure whether those clients actually need this stuff, and if they do, whether the current form is satisfactory to them. (For example one issue is that the slice limits are computed outside the index expression, so presumably they'd make certain usages difficult.) One other issue discussed in the enhancement request is that there might be a need for strides etc.

So I'm torn about this. One one hand it's a work of beauty. On the other it seems of low impact and unclear user base. Thoughts?

Member

andralex commented Sep 24, 2012

(background: I'm doing a review of pull requests < 1000 as they tend to be controversial)

This is nice, self-consistent, and tastefully designed.

The question is scope and intended usage. This is a relatively hefty language addition, with a very narrow clientele. I'm not sure whether those clients actually need this stuff, and if they do, whether the current form is satisfactory to them. (For example one issue is that the slice limits are computed outside the index expression, so presumably they'd make certain usages difficult.) One other issue discussed in the enhancement request is that there might be a need for strides etc.

So I'm torn about this. One one hand it's a work of beauty. On the other it seems of low impact and unclear user base. Thoughts?

@quickfur

This comment has been minimized.

Show comment
Hide comment
@quickfur

quickfur Feb 16, 2013

Member

I vote for merging this.

Or at the very least, some subset of this that supports multidimensional opDollar with slicing. Some background: I'm working on a multidimensional array implementation that uses the variadic form of opIndex to dereference array elements, and it works wonderfully (opDollar works as well). However, slicing syntax currently is not supported, so I've implemented a workaround using opSlice with two array arguments, so one could write A[[0,0,0]..[2,2,2]] for example, but the problem is that opDollar doesn't work correctly ($ always gets mapped to opDollar!0, so A[[0,0,0]..[$,$,$]] does not do the right thing when the array has different lengths along each dimension. I've tried various alternative ways to provide convenient slicing syntax, but none of them can support opDollar correctly.

It would be very nice if slicing can also support variadic arguments, since currently it's a hole in the language -- opIndex supports multidimensional arrays but opSlice doesn't.

Member

quickfur commented Feb 16, 2013

I vote for merging this.

Or at the very least, some subset of this that supports multidimensional opDollar with slicing. Some background: I'm working on a multidimensional array implementation that uses the variadic form of opIndex to dereference array elements, and it works wonderfully (opDollar works as well). However, slicing syntax currently is not supported, so I've implemented a workaround using opSlice with two array arguments, so one could write A[[0,0,0]..[2,2,2]] for example, but the problem is that opDollar doesn't work correctly ($ always gets mapped to opDollar!0, so A[[0,0,0]..[$,$,$]] does not do the right thing when the array has different lengths along each dimension. I've tried various alternative ways to provide convenient slicing syntax, but none of them can support opDollar correctly.

It would be very nice if slicing can also support variadic arguments, since currently it's a hole in the language -- opIndex supports multidimensional arrays but opSlice doesn't.

@WalterBright

This comment has been minimized.

Show comment
Hide comment
@WalterBright

WalterBright Mar 7, 2013

Member

I'd like to include with this some code that is commented out for the moment that issues a warning and corrective action when opSlice, opSliceUnary, opSliceAssign, and opSliceOpAssign as those will now be obsolete.

Member

WalterBright commented Mar 7, 2013

I'd like to include with this some code that is commented out for the moment that issues a warning and corrective action when opSlice, opSliceUnary, opSliceAssign, and opSliceOpAssign as those will now be obsolete.

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex May 12, 2013

Member

Ping on this. Now that time has passed we should be a tad wiser so maybe we have new insights in the matter. Thoughts?

Member

andralex commented May 12, 2013

Ping on this. Now that time has passed we should be a tad wiser so maybe we have new insights in the matter. Thoughts?

@MaksimZh

This comment has been minimized.

Show comment
Hide comment
@MaksimZh

MaksimZh May 14, 2013

Slices and strides make code more expressive and can save hours of debugging especially if one deals with matrices.

The design suggested is really good. It doesn't break existing code. It reduces the number of methods one has to implement (e.g. opSliceAssign is not needed anymore). Strides can be added later and will also not break anything.

I just can not understand why it is still not merged.

Slices and strides make code more expressive and can save hours of debugging especially if one deals with matrices.

The design suggested is really good. It doesn't break existing code. It reduces the number of methods one has to implement (e.g. opSliceAssign is not needed anymore). Strides can be added later and will also not break anything.

I just can not understand why it is still not merged.

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr May 14, 2013

Member

Sorry I'm failing to rebase this change to git master. Currently this change will break test/runnable/aliasthis.d...

Member

9rnsr commented May 14, 2013

Sorry I'm failing to rebase this change to git master. Currently this change will break test/runnable/aliasthis.d...

@don-clugston-sociomantic

This comment has been minimized.

Show comment
Hide comment
@don-clugston-sociomantic

don-clugston-sociomantic May 14, 2013

Contributor

This needs a rebase (perhaps a rebase -i). There are dozens of unrelated commits in here.

Otherwise, I think this is awesome. This finishes the job started with opDollar, and it's something we've needed to do eventually.

What happens if you do:
void foo( int {] x) { x[2..$, 4] = 7; }
after these changes to the parser? Do you still get a nice error message?

Contributor

don-clugston-sociomantic commented May 14, 2013

This needs a rebase (perhaps a rebase -i). There are dozens of unrelated commits in here.

Otherwise, I think this is awesome. This finishes the job started with opDollar, and it's something we've needed to do eventually.

What happens if you do:
void foo( int {] x) { x[2..$, 4] = 7; }
after these changes to the parser? Do you still get a nice error message?

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr May 14, 2013

Member

And, now I have a small performance worry. To represent slice, it would need to pack the start and end of the slice.

struct S {
    auto opSlice(int x, int y) { return tuple(x, y); }
    void opIndex(int, Tuple!(int, int) slice) {}
}
S s;
s[1, 2..3];   // 2..3 will always need creating tuple object

Is this acceptable?

Member

9rnsr commented May 14, 2013

And, now I have a small performance worry. To represent slice, it would need to pack the start and end of the slice.

struct S {
    auto opSlice(int x, int y) { return tuple(x, y); }
    void opIndex(int, Tuple!(int, int) slice) {}
}
S s;
s[1, 2..3];   // 2..3 will always need creating tuple object

Is this acceptable?

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr May 19, 2013

Member

OK. I finished rebasing.

Member

9rnsr commented May 19, 2013

OK. I finished rebasing.

@aitzkora

This comment has been minimized.

Show comment
Hide comment
@aitzkora

aitzkora Oct 1, 2013

I vote also for merging this! multidimensional slicing is very interesting for people doing numerical computations!

aitzkora commented Oct 1, 2013

I vote also for merging this! multidimensional slicing is very interesting for people doing numerical computations!

@John-Colvin

This comment has been minimized.

Show comment
Hide comment
@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Mar 12, 2014

Member

Finished rebasing.

Member

9rnsr commented Mar 12, 2014

Finished rebasing.

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Mar 16, 2014

Member

Is my understanding that this is a large breakage of existing uses of opSlice?

Member

andralex commented Mar 16, 2014

Is my understanding that this is a large breakage of existing uses of opSlice?

@andralex

View changes

src/expression.c
+ }
+ else
+ {
+ ae->error("multi-dimentional slicing requires template opSlice");

This comment has been minimized.

@andralex

andralex Mar 16, 2014

Member

s/dimentional/dimensional/

@andralex

andralex Mar 16, 2014

Member

s/dimentional/dimensional/

This comment has been minimized.

@9rnsr

9rnsr Mar 16, 2014

Member

Thanks, fixed.

@9rnsr

9rnsr Mar 16, 2014

Member

Thanks, fixed.

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Mar 16, 2014

Member

For the record Walter and I are on board with this language addition as long as there's no breakage of existing code. Thanks!

Member

andralex commented Mar 16, 2014

For the record Walter and I are on board with this language addition as long as there's no breakage of existing code. Thanks!

@CyberShadow

This comment has been minimized.

Show comment
Hide comment
@CyberShadow

CyberShadow Mar 16, 2014

Member

The pull description states:

This enhancement never break existing codes.

Looking at the tests, it does seem to break alias-this'd tuples. Kenji confirmed it in a comment:

Currently this change will break test/runnable/aliasthis.d...

Member

CyberShadow commented Mar 16, 2014

The pull description states:

This enhancement never break existing codes.

Looking at the tests, it does seem to break alias-this'd tuples. Kenji confirmed it in a comment:

Currently this change will break test/runnable/aliasthis.d...

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Mar 16, 2014

Member

@9rnsr can you work some magic to keep alias this working?

Member

andralex commented Mar 16, 2014

@9rnsr can you work some magic to keep alias this working?

9rnsr added some commits Mar 12, 2014

[Refactoring] Operator overload handling in AssignExp
Change to recursive call for later fallback mechanism.
Add IntervalExp and fix parser, and disabled test for issue 6789
All of N-dimensional array operations are now translated to ArrayExp.
But currently they are immediately translated to SliceExp, so have no effect.
Support multi-dimensional opIndex
If it fails, fall back to opSlice for backward compatibility.
Support multi-dimensional opIndexAssign
If it fails, fall back to opSliceAssign for backward compatibility.
Support multi-dimensional opIndexUnary
If it fails, fall back to opSliceUnary for backward compatibility.
Support multi-dimensional opIndexOpAssign
If it fails, fall back to opSliceOpAssign for backward compatibility.
@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Mar 16, 2014

Member

Is my understanding that this is a large breakage of existing uses of opSlice?

No. New opSlice!dim(lwe, upr) will be preferred, but if it does not exist, old signature opSlice(lwr, upr) is used.

Currently this change will break test/runnable/aliasthis.d

It's an exception. Issue 8735 & 9709 was fixed in 2.065, but in the test case, A[0] should not invoke alias this, because A is the type Tuple9709!(1,int,"foo"), so A[0] should be analyzed as the zero-length static array type. I think that is a bug, so the fix is legitimate.

Member

9rnsr commented Mar 16, 2014

Is my understanding that this is a large breakage of existing uses of opSlice?

No. New opSlice!dim(lwe, upr) will be preferred, but if it does not exist, old signature opSlice(lwr, upr) is used.

Currently this change will break test/runnable/aliasthis.d

It's an exception. Issue 8735 & 9709 was fixed in 2.065, but in the test case, A[0] should not invoke alias this, because A is the type Tuple9709!(1,int,"foo"), so A[0] should be analyzed as the zero-length static array type. I think that is a bug, so the fix is legitimate.

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Mar 16, 2014

Member

OK, great to know! What's with the red though?

Member

andralex commented Mar 16, 2014

OK, great to know! What's with the red though?

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Mar 16, 2014

Member

Auto-merge toggled on

Member

andralex commented Mar 16, 2014

Auto-merge toggled on

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Mar 16, 2014

Member

What's with the red though?

It seems to be caused by the merge of #2256. This and that are both big, so something would be changed. I'll fix it.

Member

9rnsr commented Mar 16, 2014

What's with the red though?

It seems to be caused by the merge of #2256. This and that are both big, so something would be changed. I'll fix it.

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Mar 16, 2014

Member

thanks!

Member

andralex commented Mar 16, 2014

thanks!

@braddr

This comment has been minimized.

Show comment
Hide comment
@braddr

braddr Mar 16, 2014

Member

Pull updated, auto_merge toggled off

Member

braddr commented Mar 16, 2014

Pull updated, auto_merge toggled off

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Mar 16, 2014

Member

Hmm, the red seems to have been fake. Could you re-toggle auto merge?

Member

9rnsr commented Mar 16, 2014

Hmm, the red seems to have been fake. Could you re-toggle auto merge?

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Mar 16, 2014

Member

Auto-merge toggled on

Member

andralex commented Mar 16, 2014

Auto-merge toggled on

andralex added a commit that referenced this pull request Mar 16, 2014

Merge pull request #443 from 9rnsr/fix6798
Issue 6798 - Integrate overloadings for multidimensional indexing and slicing

@andralex andralex merged commit 7148ab2 into dlang:master Mar 16, 2014

1 check passed

default Pass: 10
Details
@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Mar 16, 2014

Member

Thanks!

Member

9rnsr commented Mar 16, 2014

Thanks!

@@ -1069,6 +1069,18 @@ class ArrayLengthExp : public UnaExp
void accept(Visitor *v) { v->visit(this); }
};
+struct IntervalExp : Expression

This comment has been minimized.

@yebblies

yebblies Mar 17, 2014

Member

This should have been a class.

@yebblies

yebblies Mar 17, 2014

Member

This should have been a class.

@Orvid

This comment has been minimized.

Show comment
Hide comment
@Orvid

Orvid Mar 17, 2014

Contributor

Akk...... I see a serious design flaw with the multi-dimensional slicing, namely that it's impossible to determine whether the slice is the first or second argument from within the opSlice call, because it would change what the slice actually represents in things such as multi-dimensional arrays. I'm going to put this here for now just because there is an active topic on the mailing list over this, I'll update this momentarily to illustrate the issue better with examples.

So, it appears I have mis-understood what this PR actually does, and instead have the simple question of why? Why is this useful? Why would you want a slice operation to potentially call an opIndex, which is intended for selecting a single element? All I see in this PR is the actual addition of the ability to do multi-dimensional slicing, but even that is using an opIndex, when it should be introducing it as an opSlice. This also will cause issues due to the fact an operator can't currently be overloaded based on return type. There are situations, such as with matrices, that would mean that this syntax would require a dynamic array allocation for each index in order to both allow a user to both do that single dimensional slice and use it, as well as to still allow a multi-dimensional slice to be done.

This isn't the first addition to the D language that's been merged that really shouldn't have been, not without a more public review process. Perhaps requiring a formal review, such as what is required for the addition of Phobos modules, when adding or removing features or behaviors from the D language itself?

Contributor

Orvid commented Mar 17, 2014

Akk...... I see a serious design flaw with the multi-dimensional slicing, namely that it's impossible to determine whether the slice is the first or second argument from within the opSlice call, because it would change what the slice actually represents in things such as multi-dimensional arrays. I'm going to put this here for now just because there is an active topic on the mailing list over this, I'll update this momentarily to illustrate the issue better with examples.

So, it appears I have mis-understood what this PR actually does, and instead have the simple question of why? Why is this useful? Why would you want a slice operation to potentially call an opIndex, which is intended for selecting a single element? All I see in this PR is the actual addition of the ability to do multi-dimensional slicing, but even that is using an opIndex, when it should be introducing it as an opSlice. This also will cause issues due to the fact an operator can't currently be overloaded based on return type. There are situations, such as with matrices, that would mean that this syntax would require a dynamic array allocation for each index in order to both allow a user to both do that single dimensional slice and use it, as well as to still allow a multi-dimensional slice to be done.

This isn't the first addition to the D language that's been merged that really shouldn't have been, not without a more public review process. Perhaps requiring a formal review, such as what is required for the addition of Phobos modules, when adding or removing features or behaviors from the D language itself?

@quickfur

This comment has been minimized.

Show comment
Hide comment
@quickfur

quickfur Mar 17, 2014

Member

Huh? This pull has been referenced multiple times over the course of (at least) the past year in the forum. Is that not public enough? Or are we supposed to copy-n-paste the comments / code from the pull to the forum in order to make it more public? I'm confused.

Member

quickfur commented Mar 17, 2014

Huh? This pull has been referenced multiple times over the course of (at least) the past year in the forum. Is that not public enough? Or are we supposed to copy-n-paste the comments / code from the pull to the forum in order to make it more public? I'm confused.

@quickfur

This comment has been minimized.

Show comment
Hide comment
@quickfur

quickfur Mar 17, 2014

Member

Regarding your question about opIndex, the idea here is that opSlice!n(a,b) will return some kind of type that represents a slice from a to b in the n'th position, and opIndex will be overloaded to take arguments of this type and implement the actual slicing this way. The idea is that you should be able to call opSlice to get a type representing an index range, and then you can use that range object in multiple subsequent calls to opIndex. This is more flexible than limiting opSlice to do the actual slicing, then you have to keep track of the endpoints of the range manually each time.

Member

quickfur commented Mar 17, 2014

Regarding your question about opIndex, the idea here is that opSlice!n(a,b) will return some kind of type that represents a slice from a to b in the n'th position, and opIndex will be overloaded to take arguments of this type and implement the actual slicing this way. The idea is that you should be able to call opSlice to get a type representing an index range, and then you can use that range object in multiple subsequent calls to opIndex. This is more flexible than limiting opSlice to do the actual slicing, then you have to keep track of the endpoints of the range manually each time.

@Orvid

This comment has been minimized.

Show comment
Hide comment
@Orvid

Orvid Mar 17, 2014

Contributor

The thing is that the way it's been being done is subjective to the actual reader of the forum topics, by using the same process that phobos modules go through, it produces a non-subjective result, in the form of votes.

Contributor

Orvid commented Mar 17, 2014

The thing is that the way it's been being done is subjective to the actual reader of the forum topics, by using the same process that phobos modules go through, it produces a non-subjective result, in the form of votes.

@Orvid

This comment has been minimized.

Show comment
Hide comment
@Orvid

Orvid Mar 17, 2014

Contributor

But that still doesn't address the fact that your using opIndex, which is intended to retrieve a single element, to retrieve a slice instead. The same overload that is being called for opIndex could be done as an overload for opSlice instead, and would make much more sense to the reader of the code.

Contributor

Orvid commented Mar 17, 2014

But that still doesn't address the fact that your using opIndex, which is intended to retrieve a single element, to retrieve a slice instead. The same overload that is being called for opIndex could be done as an overload for opSlice instead, and would make much more sense to the reader of the code.

@quickfur

This comment has been minimized.

Show comment
Hide comment
@quickfur

quickfur Mar 17, 2014

Member

I don't know why you think that opIndex is only intended to retrieve a single element. In my own multidimensional array implementation, I've done opIndex(Range a, Range b, ...) because it's easier to consolidate mixed slices that way, e.g., opIndex(int a, Range b, int c, ...). It doesn't make sense to me to differentiate between indexing and slicing, because then how would you implement things like arr[1, 0..$, 2, 3, 2..5]? Rather, think of opIndex as implementing a[...], which can either be a single element (all arguments are indices), a full-dimensional slice (i.e., all arguments are ranges), or a sub-dimensional slice (some arguments are indices, some are ranges).

Member

quickfur commented Mar 17, 2014

I don't know why you think that opIndex is only intended to retrieve a single element. In my own multidimensional array implementation, I've done opIndex(Range a, Range b, ...) because it's easier to consolidate mixed slices that way, e.g., opIndex(int a, Range b, int c, ...). It doesn't make sense to me to differentiate between indexing and slicing, because then how would you implement things like arr[1, 0..$, 2, 3, 2..5]? Rather, think of opIndex as implementing a[...], which can either be a single element (all arguments are indices), a full-dimensional slice (i.e., all arguments are ranges), or a sub-dimensional slice (some arguments are indices, some are ranges).

@Orvid

This comment has been minimized.

Show comment
Hide comment
@Orvid

Orvid Mar 17, 2014

Contributor

Your use of the ranges there is born out of the lack of a proper way to do it, opIndex should only ever be used to refer to a single element, a sub-dimensional slice is still a slice, and could even be treated as a full-dimensional slice. Whatever way you look at it, they are still slices and thus should be using opSlice.

Contributor

Orvid commented Mar 17, 2014

Your use of the ranges there is born out of the lack of a proper way to do it, opIndex should only ever be used to refer to a single element, a sub-dimensional slice is still a slice, and could even be treated as a full-dimensional slice. Whatever way you look at it, they are still slices and thus should be using opSlice.

@quickfur

This comment has been minimized.

Show comment
Hide comment
@quickfur

quickfur Mar 17, 2014

Member

I find the use of opIndex for both purposes more uniform, because it allows generic implementations that handle all cases in a single interface. Subdimensional slicing, in my implementation, actually returns a different type depending on how many arguments are slices and how many are indices (the types are segmented by dimension; the 0-dimensional case aliases to the element type); the case where all are indices is only one of the many possible combinations. Arbitrarily segmenting opIndex to two names introduces a lot of inelegance in the code, because now everything has to specially treat the case where all arguments happen to be indices, whereas now, it simply handles all cases together.

In any case, you're arguing with the wrong person; you should be asking Kenji the rationale for this design, since he's the one who implemented it! :)

Member

quickfur commented Mar 17, 2014

I find the use of opIndex for both purposes more uniform, because it allows generic implementations that handle all cases in a single interface. Subdimensional slicing, in my implementation, actually returns a different type depending on how many arguments are slices and how many are indices (the types are segmented by dimension; the 0-dimensional case aliases to the element type); the case where all are indices is only one of the many possible combinations. Arbitrarily segmenting opIndex to two names introduces a lot of inelegance in the code, because now everything has to specially treat the case where all arguments happen to be indices, whereas now, it simply handles all cases together.

In any case, you're arguing with the wrong person; you should be asking Kenji the rationale for this design, since he's the one who implemented it! :)

@MasonMcGill

This comment has been minimized.

Show comment
Hide comment
@MasonMcGill

MasonMcGill Mar 17, 2014

Great work folks! I do have a concern, though: it looks like you're bringing Python 2.0-style slice handling to D, even though it turned out there were limitations to that model that could have been avoided with a different design. It worries me that Julia, a language specifically designed ~15 years later to make up for the shortcomings in existing scientific platforms, chose an alternate approach. More discussion of that here:
http://forum.dlang.org/thread/upzdamhmxrrlsexgcdva@forum.dlang.org

It looks like I timed this pretty poorly, but I actually just wrote up a DIP that addresses the problem of "advanced" indexing in a more general way, and enables more MATLAB/Julia-style numerical features, while reducing language complexity and maintaining backwards compatibility:
http://wiki.dlang.org/DIP58

Again, I know the timing's horrible, after all the work that's gone into this, but I think it's important to get numerical syntax right, and I think a bit of discussion about the pros and cons of each approach would do the language some good.

Discussion receptacle:
http://forum.dlang.org/thread/clswrooryjcudncqbkje@forum.dlang.org

Great work folks! I do have a concern, though: it looks like you're bringing Python 2.0-style slice handling to D, even though it turned out there were limitations to that model that could have been avoided with a different design. It worries me that Julia, a language specifically designed ~15 years later to make up for the shortcomings in existing scientific platforms, chose an alternate approach. More discussion of that here:
http://forum.dlang.org/thread/upzdamhmxrrlsexgcdva@forum.dlang.org

It looks like I timed this pretty poorly, but I actually just wrote up a DIP that addresses the problem of "advanced" indexing in a more general way, and enables more MATLAB/Julia-style numerical features, while reducing language complexity and maintaining backwards compatibility:
http://wiki.dlang.org/DIP58

Again, I know the timing's horrible, after all the work that's gone into this, but I think it's important to get numerical syntax right, and I think a bit of discussion about the pros and cons of each approach would do the language some good.

Discussion receptacle:
http://forum.dlang.org/thread/clswrooryjcudncqbkje@forum.dlang.org

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Mar 17, 2014

Member

@quickfur In the proposal, the member function opIndex will be used for every bracket access syntax (a[], a[i], a[i..j], a[$, i..j], etc). It's just syntactic rewriting rule, and it's not directly related to the code meaning of indexing, as the same as that opDollar is not directly related to the length or size of container.

Member

9rnsr commented Mar 17, 2014

@quickfur In the proposal, the member function opIndex will be used for every bracket access syntax (a[], a[i], a[i..j], a[$, i..j], etc). It's just syntactic rewriting rule, and it's not directly related to the code meaning of indexing, as the same as that opDollar is not directly related to the length or size of container.

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Mar 17, 2014

Member

@MasonMcGill Sometimes .. operator had been proposed in the newsgroup, but it is not yet exist. So this proposal did not consider it.

Member

9rnsr commented Mar 17, 2014

@MasonMcGill Sometimes .. operator had been proposed in the newsgroup, but it is not yet exist. So this proposal did not consider it.

@MasonMcGill

This comment has been minimized.

Show comment
Hide comment
@MasonMcGill

MasonMcGill Mar 18, 2014

@9rnsr Understood. My request was for a discussion about why this approach is better than something like http://wiki.dlang.org/DIP58 , and whether this is meant to be the "last stop" for the evolution of numerical syntax in D, or part of a larger transition.

@9rnsr Understood. My request was for a discussion about why this approach is better than something like http://wiki.dlang.org/DIP58 , and whether this is meant to be the "last stop" for the evolution of numerical syntax in D, or part of a larger transition.

@Orvid

This comment has been minimized.

Show comment
Hide comment
@Orvid

Orvid Mar 18, 2014

Contributor

I believe that it is related to the code meaning of indexing, by introducing this under the name of opIndex, it's using the wrong term for the slicing operation that is being performed.

Contributor

Orvid commented Mar 18, 2014

I believe that it is related to the code meaning of indexing, by introducing this under the name of opIndex, it's using the wrong term for the slicing operation that is being performed.

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Mar 18, 2014

Member

@Orvid I agree it was related. But, by supporting multi-dimensional slicing, a[i..j] could also be considered as one dimension indexing with an interval.
So uniformly using the name opIndex for both a[i..j] and a[i..j, k] would be practically convenient. I don't think it's so big issue.

Member

9rnsr commented Mar 18, 2014

@Orvid I agree it was related. But, by supporting multi-dimensional slicing, a[i..j] could also be considered as one dimension indexing with an interval.
So uniformly using the name opIndex for both a[i..j] and a[i..j, k] would be practically convenient. I don't think it's so big issue.

@don-clugston-sociomantic

This comment has been minimized.

Show comment
Hide comment
@don-clugston-sociomantic

don-clugston-sociomantic Mar 18, 2014

Contributor

@Orvid

I believe that it is related to the code meaning of indexing, by introducing this under the name of opIndex, it's using the wrong term for the slicing operation that is being performed.

IMHO opSlice would not be the correct term. The only sense in which it is a "slice" is that with an n-dimensional array, it doesn't reduce the dimensionality, whereas an "index" normally does. But we don't even enforce those semantics. For example, you could create something with an infinite number of dimensions, and you could make slicing reduce the dimensionality as well.

Really it is a strided slice, which is a totally different beast, neither an index nor a slice, it doesn't have much in common with normal slices. Pragmatically, the opIndex syntax generalizes, whereas the opSlice syntax does not.

Contributor

don-clugston-sociomantic commented Mar 18, 2014

@Orvid

I believe that it is related to the code meaning of indexing, by introducing this under the name of opIndex, it's using the wrong term for the slicing operation that is being performed.

IMHO opSlice would not be the correct term. The only sense in which it is a "slice" is that with an n-dimensional array, it doesn't reduce the dimensionality, whereas an "index" normally does. But we don't even enforce those semantics. For example, you could create something with an infinite number of dimensions, and you could make slicing reduce the dimensionality as well.

Really it is a strided slice, which is a totally different beast, neither an index nor a slice, it doesn't have much in common with normal slices. Pragmatically, the opIndex syntax generalizes, whereas the opSlice syntax does not.

@Orvid

This comment has been minimized.

Show comment
Hide comment
@Orvid

Orvid Mar 21, 2014

Contributor

While I do see your point that opSlice is not correct in all cases, there are cases where it would be the correct term. This PR adds the ability to use opIndex in places where opSlice was previously the preferred operator. It also adds an overload to opIndex that opSlice should have as well, for use in the cases where opSlice would be the correct term, unless part of the intention of this PR is to start a deprecation path for opSlice in favor of opIndex?

Contributor

Orvid commented Mar 21, 2014

While I do see your point that opSlice is not correct in all cases, there are cases where it would be the correct term. This PR adds the ability to use opIndex in places where opSlice was previously the preferred operator. It also adds an overload to opIndex that opSlice should have as well, for use in the cases where opSlice would be the correct term, unless part of the intention of this PR is to start a deprecation path for opSlice in favor of opIndex?

@JakobOvrum

This comment has been minimized.

Show comment
Hide comment
@JakobOvrum

JakobOvrum Mar 21, 2014

Member

Is there a corresponding documentation PR for this?

Member

JakobOvrum commented Mar 21, 2014

Is there a corresponding documentation PR for this?

@quickfur

This comment has been minimized.

Show comment
Hide comment
@quickfur

quickfur Mar 21, 2014

Member

@Orvid No, in this PR opSlice performs the duty of translating an x..y range into an aggregate index object represent that range. This aggregate index is then translated into an actual slicing operation by opIndex. The idea is to unify slicing and indexing, not to deprecate anything over the other.

Member

quickfur commented Mar 21, 2014

@Orvid No, in this PR opSlice performs the duty of translating an x..y range into an aggregate index object represent that range. This aggregate index is then translated into an actual slicing operation by opIndex. The idea is to unify slicing and indexing, not to deprecate anything over the other.

@9rnsr 9rnsr deleted the 9rnsr:fix6798 branch Apr 15, 2014

@John-Colvin

This comment has been minimized.

Show comment
Hide comment
@John-Colvin

John-Colvin Aug 6, 2014

Contributor

documentation?

Contributor

John-Colvin commented Aug 6, 2014

documentation?

@quickfur

This comment has been minimized.

Show comment
Hide comment
@quickfur

quickfur Aug 6, 2014

Member

I'll see if I can work up a PR for the docs today.

Member

quickfur commented Aug 6, 2014

I'll see if I can work up a PR for the docs today.

@quickfur quickfur referenced this pull request in dlang/dlang.org Aug 7, 2014

Merged

Document multidimensional array op overloading #625

@quickfur

This comment has been minimized.

Show comment
Hide comment
@quickfur

quickfur Aug 7, 2014

Member
static assert(a[0] == 1);
//static assert(is(A[1] == int));
//static assert(is(a[1] == int));
- static assert(A[2] == "foo");
+ //static assert(A[2] == "foo");

This comment has been minimized.

@Orvid

Orvid Jan 29, 2015

Contributor

What was the reasoning behind this? (only noticed it as I went to see how this handled both opSlice and opIndex being assigned)

@Orvid

Orvid Jan 29, 2015

Contributor

What was the reasoning behind this? (only noticed it as I went to see how this handled both opSlice and opIndex being assigned)

This comment has been minimized.

@9rnsr

9rnsr Jan 29, 2015

Member

I don't remember the precise reason, but it had been interfere with the new operator overloading rule. And the test case was added without deep thinking when issue 8735 was fixed (actually it was added by me).

So I was disable it again because it would not introduce serious use code breaking.

@9rnsr

9rnsr Jan 29, 2015

Member

I don't remember the precise reason, but it had been interfere with the new operator overloading rule. And the test case was added without deep thinking when issue 8735 was fixed (actually it was added by me).

So I was disable it again because it would not introduce serious use code breaking.

@CyberShadow

This comment has been minimized.

Show comment
Hide comment
@CyberShadow

CyberShadow May 26, 2015

Member

This pull request introduced a regression:
https://issues.dlang.org/show_bug.cgi?id=14621

Member

CyberShadow commented May 26, 2015

This pull request introduced a regression:
https://issues.dlang.org/show_bug.cgi?id=14621

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment