Make unvetted `size` throw an error for arrays with non-1 indexing #17228

timholy · 2016-07-01T15:21:46Z

For arrays that violate the assumption that indexing starts at 1, there's a risk that formerly-safe code could exit with nasty errors, see here. Discussion in #16260 led to a good suggestion for a migration strategy: during julia-0.5, size should throw an error by default for such "unconventional" arrays. This PR introduces the necessary infrastructure:

an IndicesSafety trait that can be passed as an argument through size and related functions
a macro @safeindices that you can use to wrap functions that you've certified are safe for arrays with unconventional indices.

Together with #17137, this closes #16973 and should be the end of array changes that I envision for 0.5. (It is expected that more functions will become @safeindices over 0.5.x releases as people increasingly use such unconventional arrays.)

timholy · 2016-07-01T15:25:08Z

base/abstractarray.jl

+# SafeIndices().
+size( ::IndicesSafety, A::AbstractArray)    = size(A)
+size( ::IndicesSafety, A::AbstractArray, d) = size(A,d)  # fixme
+# size(s::IndicesSafety, A::AbstractArray, d) = d <= ndims(A) ? size(s, A)[d] : 1


For some reason, uncommenting this line causes inference to bail during bootstrap. (Avoiding ndims and using the parameter variant, A::AbstractArray{T,N}, does not fix the problem either.)

As a consequence, people who write packages for unconventionally-indexed arrays have to specialize both size(s, A) and size(s, A, d), which is a little unfortunate but not too terrible.

tkelman · 2016-07-03T13:50:12Z

base/abstractarray.jl

+size( ::IndicesSafety, A)                   = size(A)
+size( ::IndicesSafety, A::AbstractArray, d) = size(A,d)  # fixme
+# size(s::IndicesSafety, A::AbstractArray, d) = d <= ndims(A) ? size(s, A)[d] : 1
+size{N}(s::IndicesSafety, A::AbstractArray, d1::Integer, d2::Integer, dx::Vararg{Integer, N}) = (size(s, A, d1), size(s, A, d2), ntuple(k->size(s, A, dx[k]), Val{N})...)


tkelman · 2016-07-03T14:04:53Z

I'm not following when SafeIndices() should be used vs UnsafeIndices() - could an explanation be written up in docs or devdocs?

tkelman · 2016-07-03T14:12:07Z

base/abstractarray.jl

@@ -1417,7 +1447,7 @@ function mapslices(f, A::AbstractArray, dims::AbstractVector)
        idx[d] = Colon()
    end

-    r1 = f(view(A, idx...))
+    r1 = f(A[idx...])


not mentioned in commit message, maybe should have a specific test

I'll split this out into a separate PR.

timholy · 2016-07-03T14:13:23Z

I'll modify the unmerged JuliaLang/www_old.julialang.org#386 when things settle out here. Bottom line is that neither is expected to be used in user-code; the idea is:

audit/fix your code for more general indexing
mark the function with @safeindices

Internally within Base, @safeindices doesn't work early in bootstrap.

tkelman · 2016-07-03T14:13:29Z

base/exports.jl

@@ -577,6 +577,7 @@ export
    rot180,
    rotl90,
    rotr90,
+    @safeindices,


needs docstring inline and in rst, and probably a dedicated test of the macro?

Will add the docstring. The test happens as a consequence of the new @test_throws in offsetarray.jl and the fact that the remaining tests work.

timholy · 2016-07-03T14:18:09Z

Test failure seems unrelated.

I'm tempted to merge. I hope people realize this PR makes the unconventionally-indexed arrays brittle-by-design: in practice, this forces you to look at basically every "unvetted" function you pass them to. I've modified the big stuff (indexing, broadcasting, show) and a number of smaller routines, but Base and definitely packages will, in the early days, require changes. I think this is the best approach, however, since uncontrolled failures are much worse.

tkelman · 2016-07-03T14:25:44Z

base/reduce.jl

-            s = op(s, f(Ai))
+    inds = linearindices(A)
+    n = length(inds)
+    @inbounds begin


does this make an assumption that f(a) is bounds-safe?

@inbounds doesn't propagate, even with inlining (unless so marked). We had inbounds previously for the n<16 case, but it missed several A[i] accesses.

I clearly need to watch https://www.youtube.com/watch?v=l0icRnmqeYs a few more times

tkelman · 2016-07-03T14:35:58Z

Bottom line is that neither is expected to be used in user-code

Okay, then I guess devdocs is better, but in the current state it'll be easy to accidentally use the wrong one in base.

timholy · 2016-07-03T14:40:13Z

Are you suggesting I move the blog post to devdocs? I can do that. I'll also clarify the meaning of the two alternatives.

tkelman · 2016-07-03T14:48:17Z

I suppose that works. Would like to get out of the habit of merging things without accompanying documentation.

timholy · 2016-07-03T22:12:13Z

OK, other than moving the blog post I think I've addressed all the review comments---and frankly I think it's best to do that once #17137 merges.

I'm having trouble building the docs locally:

tim@diva:~/src/julia-0.5$ make -C doc html
make: Entering directory '/home/tim/src/julia-0.5/doc'
. ./../deps/build/julia-env/bin/activate && pip install sphinx==1.3.1 \
              && pip install -r /home/tim/src/julia-0.5/doc/requirements.txt
Traceback (most recent call last):
  File "/home/tim/src/julia-0.5/deps/build/julia-env/bin/pip", line 7, in <module>
    from pip import main
  File "/home/tim/src/julia-0.5/deps/build/julia-env/local/lib/python2.7/site-packages/pip/__init__.py", line 14, in <module>
    from pip.utils import get_installed_distributions, get_prog
  File "/home/tim/src/julia-0.5/deps/build/julia-env/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 27, in <module>
    from pip._vendor import pkg_resources
  File "/home/tim/src/julia-0.5/deps/build/julia-env/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 36, in <module>
    import plistlib
  File "/usr/lib/python2.7/plistlib.py", line 62, in <module>
    import datetime
ImportError: No module named datetime
Makefile:33: recipe for target '../deps/build/julia-env/bin/sphinx-build' failed
make: *** [../deps/build/julia-env/bin/sphinx-build] Error 1
make: Leaving directory '/home/tim/src/julia-0.5/doc'
tim@diva:~/src/julia-0.5$ locate datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.2-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.2-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.4-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.4-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.4-py27_1/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.4-py27_1/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.11.0-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.11.0-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.11.0-py27_1/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.11.0-py27_1/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/usr/lib/python2.7/dist-packages/numpy/core/tests/test_datetime.py
/usr/lib/python2.7/dist-packages/numpy/core/tests/test_datetime.pyc
/usr/lib/python2.7/dist-packages/zope/interface/common/idatetime.py
/usr/lib/python2.7/dist-packages/zope/interface/common/idatetime.pyc
/usr/lib/python2.7/dist-packages/zope/interface/common/tests/test_idatetime.py
/usr/lib/python2.7/dist-packages/zope/interface/common/tests/test_idatetime.pyc
/usr/lib/python3.5/datetime.py

If the solution is not obvious, perhaps someone else can build the docs for me and push it to this branch?

tkelman · 2016-07-03T22:47:05Z

make -C deps distclean-virtualenv maybe? ~~I was only able to get half of them to populate...~~ the signature matching is picky, fix incoming

timholy · 2016-07-04T10:37:24Z

Thanks, @tkelman! I made a few more doc tweaks, and decided to make @safeindices extensible to user-functions.

Before merging, we should check performance: @nanosoldier runbenchmarks(ALL, vs=":master").

tkelman · 2016-07-04T16:53:40Z

base/abstractarray.jl


    trailingsize(A, n) = trailingsize(UnsafeIndices(), A, n)
+


are you able to build the docs? this may make these 2 separate code blocks?

It doesn't (but thanks for noticing that I forgot make julia-genstdlib).

timholy · 2016-07-04T20:22:42Z

@nanosoldier runbenchmarks(ALL, vs = ":master")

timholy · 2016-07-04T20:28:02Z

Nanosoldier seems to be taking a well-deserved holiday.

jrevels · 2016-07-05T13:08:38Z

Looks like there was a network issue while I was away, since the error in the server logs is getaddrinfo callback: system error (EAI_SYSTEM). I'm about to hop on a plane, but I'll investigate further when I land. In the meantime, I checked the relevant connections and things seem to be fine now, so I've restarted the server. Perhaps it'll work now:

@nanosoldier runbenchmarks(ALL, vs = ":master")

jrevels · 2016-07-06T12:01:16Z

Ugh, master broke Nanosoldier's v0.5 version of JLD, which is why the build is failing. I'll update it and see if that helps. ref JuliaCI/BenchmarkTools.jl#15

tkelman · 2016-07-07T16:11:39Z

@nanosoldier runbenchmarks(ALL, vs = ":master")

nanosoldier · 2016-07-07T18:43:24Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

tkelman · 2016-07-16T09:31:48Z

base/broadcast.jl

+
+# newindexer(shape, A) generates `keep` (for use by `newindex` above)
+# for a particular array `A`, given the broadcast_shape `shape`
+# Equivalent to map(==, indices(A), shape) (but see #17126)


also equivalent to indices(A) .== shape or?

we don't support that syntax for tuples (but conceptually, yes)

timholy · 2016-07-18T11:14:36Z

All review comments addressed.

tkelman · 2016-07-18T11:19:19Z

base/multidimensional.jl

+_maybe_reshape(::LinearFast, A::AbstractArray, i) = A
+_maybe_reshape(::LinearSlow, A::AbstractVector, i) = A
+_maybe_reshape(::LinearSlow, A::AbstractArray, i) = _maybe_reshape(LinearSlow(), index_ndims(i), A)
+_maybe_reshape{N}(::LinearIndexing, ::NTuple{N}, A) = _= reshape(A, Val{N})


= _= typo?

…ndex

…dices required)

timholy · 2016-07-18T18:08:01Z

Set again (I hope!)

tkelman · 2016-07-19T06:11:52Z

I don't entirely get what these various maybe_* functions are supposed to be doing or why they're needed.

timholy · 2016-07-19T09:36:07Z

_maybe_reshape

This is extended from an older function from 51b4bc1, _maybe_linearize. The idea is that you might need/want to reshape the parent array before extracting values. Why and when? This is in support of partial linear indexing (which is still waiting for a volunteer to deprecate---you have other things to do 😉, it should be someone else). If you try to index a LinearSlow array with fewer indices than it has dimensions, it's going to have to call div which is always a performance nightmare. If you are just asking for a single value from the array, there's no better alternative to going ahead and doing it---and that's intercepted by this method, so this path doesn't get called in such cases. However, this path is used if you're asking for multiple values from the array (at least one of the indices is a vector or Colon). Because of #15357 and #15449 it will usually be worth it to first reshape the parent array to match the dimensionality of indices, so that you get the benefit of the fast multiplicative inverses when calculating div.

Since types are involved, it's important to do all this in a type-stable manner. So basically this implements the following logic in a way the compiler can follow: IF this is a LinearSlow array, AND the dimensionality of the indices doesn't match the dimensionality of the array, reshape the array to match the dimensionality of the indices. The computation of the dimensionality of the indices is actually a little tricky because of CartesianIndex, hence index_ndims.

maybe_oneto

This is to handle the opposite case from partial linear indexing, when the user supplies an extra index (e.g., indexing a vector as v[i,1]), and helps do the right thing when you've "run out of indices" in inds. I could have dispatched on Tuple{}, but since we're also specializing on the 2nd argument of decolon and index_shape_dim that gets to be annoying pretty quickly. Since IteratorsMD.split already handles that case well, I decided to use that instead, and repackage the empty tuple as an explicit index. ~~But hmm, explaining this makes me realize this should just return 1 and not OneTo(1). Will fix.~~ EDIT: nvm, I was right the first time, because the index is Colon() which needs to be converted to a Range.

timholy · 2016-07-19T14:45:42Z

Just an FYI that from my perspective this is merge-worthy and hence closes a 0.5.0 milestone. Alternatively, I can await more reviews and perhaps pile on my promise in https://github.com/JuliaLang/julia/pull/17283/files/c68a2cd9919cdf4a23243b313670ad8dc41a9c27#r71298118.

tkelman · 2016-07-19T19:07:25Z

let's do @nanosoldier runbenchmarks(ALL, vs = ":master") again just in case

StefanKarpinski · 2016-07-19T21:32:49Z

Assuming the nanosoldier run looks good, I think you should go ahead and merge, @timholy.

nanosoldier · 2016-07-19T21:44:25Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

timholy · 2016-07-19T21:47:32Z

More than good.

JeffBezanson · 2016-07-20T17:11:14Z

Thanks @timholy, this approach looks good. Kudos especially for the excellent documentation.
(Just got back from vacation, which is why I haven't been following along for the last couple days.)

timholy · 2016-07-20T18:07:00Z

Hope you had a great vacation!

And thanks for the guidance---as usual, your taste is exquisite, and I just wasn't brave enough to throw a basic function like size under a bus. Thanks for helping me get to the right answer.

cf. JuliaLang/julia#17228 (comment)

timholy reviewed Jul 1, 2016
View reviewed changes

timholy force-pushed the teh/safe_size_length branch 2 times, most recently from 26d84cd to 9faec4e Compare July 3, 2016 11:17

tkelman reviewed Jul 3, 2016
View reviewed changes

timholy force-pushed the teh/safe_size_length branch from 9faec4e to 67a2c0d Compare July 3, 2016 22:07

tkelman force-pushed the teh/safe_size_length branch from a4e50e5 to a9be8ce Compare July 3, 2016 22:53

tkelman reviewed Jul 4, 2016
View reviewed changes

timholy force-pushed the teh/safe_size_length branch from 3008bd7 to 542b088 Compare July 4, 2016 20:20

jrevels mentioned this pull request Jul 6, 2016

Error loading immutables when using Julia master JuliaIO/JLD.jl#77

Closed

tkelman reviewed Jul 16, 2016
View reviewed changes

tkelman reviewed Jul 18, 2016
View reviewed changes

timholy added 5 commits July 18, 2016 11:37

Make size throw an error for arrays with non-1 indexing

7711aef

unsafe_length(A) -> _length(A)

88014f3

Fixes and tests to index_shape, decolon, and indexing with CartesianI…

effdc40

…ndex

Better error for array operations with axis < 1

15914b5

reductions & broadcast: allow any dimension of size 1 (no specific in…

f85f858

…dices required)

timholy force-pushed the teh/safe_size_length branch from 82b79ff to f85f858 Compare July 18, 2016 18:01

timholy merged commit c2e3494 into master Jul 19, 2016

timholy deleted the teh/safe_size_length branch July 19, 2016 21:47

mbauman added a commit to mbauman/DistributedArrays.jl that referenced this pull request Aug 3, 2016

Update for SubArray change

4fb5404

cf. JuliaLang/julia#17228 (comment)

Make unvetted size throw an error for arrays with non-1 indexing #17228

Make unvetted size throw an error for arrays with non-1 indexing #17228

Conversation

timholy commented Jul 1, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkelman commented Jul 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timholy commented Jul 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timholy commented Jul 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkelman commented Jul 3, 2016

timholy commented Jul 3, 2016

tkelman commented Jul 3, 2016

timholy commented Jul 3, 2016

tkelman commented Jul 3, 2016 • edited Loading

timholy commented Jul 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timholy commented Jul 4, 2016

timholy commented Jul 4, 2016

jrevels commented Jul 5, 2016 • edited Loading

jrevels commented Jul 6, 2016

tkelman commented Jul 7, 2016

nanosoldier commented Jul 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timholy commented Jul 18, 2016

Choose a reason for hiding this comment

timholy commented Jul 18, 2016

tkelman commented Jul 19, 2016

timholy commented Jul 19, 2016 • edited Loading

_maybe_reshape

maybe_oneto

timholy commented Jul 19, 2016

tkelman commented Jul 19, 2016

StefanKarpinski commented Jul 19, 2016

nanosoldier commented Jul 19, 2016

timholy commented Jul 19, 2016

JeffBezanson commented Jul 20, 2016

timholy commented Jul 20, 2016

Make unvetted `size` throw an error for arrays with non-1 indexing #17228

Make unvetted `size` throw an error for arrays with non-1 indexing #17228

tkelman commented Jul 3, 2016 •

edited

Loading

jrevels commented Jul 5, 2016 •

edited

Loading

timholy commented Jul 19, 2016 •

edited

Loading