require explicit predicates in `find` functions #23812

JeffBezanson · 2017-09-21T23:03:21Z

I also tried out using equalto(x) as a predicate, and I really think we should adopt it. It reads nicely (!equalto(x) also works) and will cut down the number of function types.

fix #23120, fix #19186

Part of #10593.

JeffBezanson · 2017-09-21T23:11:07Z

Also, I didn't touch findn, findnz, or find for sparse matrices. I suppose for find, we could require that predicates be false for 0 (giving an error otherwise).

ararslan · 2017-09-21T23:25:56Z

base/deprecated.jl

@@ -1855,6 +1855,12 @@ end
    nothing
 end

+@deprecate find(x::Number)            find(x->x!=0, x)


Should we encourage find(!iszero, x) for this?

Ah yes, for Number I guess we should.

ararslan · 2017-09-21T23:27:45Z

test/arrayops.jl

@@ -427,44 +427,43 @@ end
    @test X[Y[end],1] == 5
    @test X[end,Y[end]] == 11
 end
+equalto(y) = x->x==y


Would it make sense to use Base.equalto for these, or do you specifically need == rather than isequal for these tests?

ararslan · 2017-09-21T23:28:39Z

I really like equalto!

TotalVerb · 2017-09-22T00:08:43Z

While it shouldn't block anything, I think it's a little strange that equalto is partially applied == and not partially applied isequal.

JeffBezanson · 2017-09-22T00:10:29Z

That's actually only the version in the tests --- I'll replace that. The one I put in Base uses isequal.

ararslan · 2017-09-22T00:19:01Z

base/deprecated.jl

@@ -1855,6 +1855,12 @@ end
    nothing
 end

+@deprecate find(x::Number)            find(!iszero, x)
+@deprecate findnext(A, v, i::Integer) findnext(x->x==v, A, i)


Perhaps we should export equalto and use it in the deprecation targets?

JeffBezanson · 2017-09-22T02:07:37Z

Ok, I went ahead and exported equalto. It's a thing.

ararslan

Jeff does it again! 💯

ararslan · 2017-09-22T05:25:15Z

Ah, this needs a NEWS item for the addition of equalto.

KristofferC · 2017-09-22T05:27:01Z

base/array.jl

@@ -1615,8 +1615,14 @@ julia> findnext(A,3)
 function findnext(A, start::Integer)
    l = endof(A)
    i = start
+    warned = false


Won't this warn everytime the function is called?

I was counting on depwarn for that.

Yeah, this looks good to me — the warned makes sure the slower depwarn is only called once for each invocation.

Oh, sorry for the noise.

nalimilan

That's really a nice simplification of the API.

Also, I didn't touch findn, findnz, or find for sparse matrices. I suppose for find, we could require that predicates be false for 0 (giving an error otherwise).

findn and findnz are special, so it's fine to keep them (at least for now), but having find behave differently for sparse matrices doesn't sound like a good idea to me. We could at least change the signature of the current find(::SparseMatrixCSC) method to find(::Union{typeof(!iszero), typeof(!equalto(0))}, ::SparseMatrixCSC) to ensure consistency and still benefit from an optimized implementation for the common case.

That approach can be generalized this by checking whether the predicate is true or false on zero elements and assuming it's pure. I don't think throwing an error is a good idea: as discussed elsewhere, sparse matrices should behave the same as dense ones everywhere it's possible.

nalimilan · 2017-09-22T11:06:43Z

base/array.jl

@@ -1596,14 +1596,14 @@ cat(n::Integer, x::Integer...) = reshape([x...], (ntuple(x->1, n-1)..., length(x
 """
    findnext(A, i::Integer)

-Find the next linear index >= `i` of a non-zero element of `A`, or `0` if not found.
+Find the next linear index >= `i` of a true element of `A`, or `0` if not found.


Maybe add backticks around true (here and in other docstring) since "a true element" may not be completely obvious when you have no idea what the function does.

nalimilan · 2017-09-22T11:07:32Z

base/array.jl

@@ -1596,14 +1596,14 @@ cat(n::Integer, x::Integer...) = reshape([x...], (ntuple(x->1, n-1)..., length(x
 """
    findnext(A, i::Integer)

-Find the next linear index >= `i` of a non-zero element of `A`, or `0` if not found.
+Find the next linear index >= `i` of a true element of `A`, or `0` if not found.

 # Examples


It would make sense to add an example using equalto, since people will likely look at this docstring when they need this feature (same for other docstrings). Or just mention it in the description.

nalimilan · 2017-09-22T11:10:07Z

base/file.jl

@@ -271,7 +271,7 @@ function tempname(temppath::AbstractString,uunique::UInt32)
    tempp = cwstring(temppath)
    tname = Vector{UInt16}(32767)
    uunique = ccall(:GetTempFileNameW,stdcall,UInt32,(Ptr{UInt16},Ptr{UInt16},UInt32,Ptr{UInt16}), tempp,temp_prefix,uunique,tname)
-    lentname = findfirst(tname,0)-1
+    lentname = findfirst(equalto(0),tname)-1


Wouldn't iszero work here?

nalimilan · 2017-09-22T11:11:04Z

base/inference.jl

@@ -5604,7 +5604,7 @@ function _getfield_elim_pass!(e::Expr, sv::InferenceState)
            if alloc !== false
                flen, fnames = alloc
                if isa(j, QuoteNode)
-                    j = findfirst(fnames, j.value)
+                    j = findfirst(x->x == j.value, fnames)


Wouldn't equalto be enough? Same below.

nalimilan · 2017-09-22T11:11:52Z

base/operators.jl

+"""
+    equalto(x)
+
+Create a function that compares its argument to `x` using `isequal`; i.e. returns


[`isequal`](@ref)

nalimilan · 2017-09-22T11:12:24Z

base/pkg/query.jl

@@ -359,7 +359,7 @@ function prune_versions(reqs::Requires, deps::Dict{String,Dict{VersionNumber,Ava
            vmaskp[vn] = falses(luds)
        end
        for (vn,a) in fdepsp
-            vmind = findfirst(uniqdepssets, a.requires)
+            vmind = findfirst(Base.equalto(a.requires), uniqdepssets)


Why Base.?

Shouldn't be necessary since it's exported?

JeffBezanson · 2017-09-22T15:18:30Z

Thanks for the good review @nalimilan .

I don't think throwing an error is a good idea: as discussed elsewhere, sparse matrices should behave the same as dense ones everywhere it's possible.

The only reason I want to throw an error is so I don't have to implement a new algorithm for finding zeros. That case can be added later.

mbauman · 2017-09-22T15:32:03Z

base/array.jl

-        if A[i] != 0
+        a = A[i]
+        if !warned && !(a isa Bool)
+            depwarn("In the future `findnext` will only work on boolean collections. Use `findnext(x->x!=0, A)` instead.", :findnext)


When I've done things like this I've often added breadcrumbs that point to it from deprecated.jl that help make sure they get removed at the appropriate timeframe.

JeffBezanson · 2017-09-22T16:06:10Z

Ah --- previously find(f, sparse) called the generic implementation, so we can continue to do that at least for the zero case.

JeffBezanson · 2017-09-22T23:59:12Z

@nanosoldier runbenchmarks(ALL, vs=":master")

nanosoldier · 2017-09-23T04:22:24Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

garrison · 2017-09-23T13:56:18Z

In UniqueVectors.jl I override findfirst with a value so that the search can be performed in O(1) time, with a Dict lookup. There is no way to do this by passing a predicate. If this pull request lands, the usage of findfirst in that package will no longer be consistent with its use in Base, so it will probably make sense for me to change the method name to something else. It may also be worth looking through the package ecosystem to see whether other packages rely on overriding findfirst(a, elt) or similar functionality as well.

Overall I am on the fence, leaning toward being -1 about this change. findfirst,findnext,findprev with a value are more specific forms than with a predicate, and UniqueVectors is unlikely to be the only case where overriding these methods for a specific data structure (with improved efficiency) makes sense.

As for #19186, I've never found the argument order confusing, as long as I keep in mind that predicates go first so that the do-block syntax can be supported.

garrison · 2017-09-23T15:07:58Z

Or perhaps this would solve my issue:

struct EqualTo{T}
    val::T
end

(x::EqualTo)(y) = isequal(x.val,y)

equalto(x) = EqualTo(x)

Then I could override findfirst(::EqualTo, ::UniqueVector).

garrison · 2017-09-23T15:21:17Z

If this PR lands, it would also be good to see a test where !equalto(x) is used. (My EqualTo suggestion above would break this functionality, unless it comes with an additional NotEqualTo type and appropriate methods for !.)

JeffBezanson · 2017-09-23T16:06:24Z

To make ! work all we'd have to do is make EqualTo a subtype of Function, so +1 to that approach.

What I find confusing about the current methods is that in findnext(x, y), y is an index, but in findfirst(x, y), y is a value to search for. Of course this makes sense if you think about it, but I always have to do an extra mental step when I see those calls.

JeffBezanson · 2017-09-26T21:29:59Z

Done --- I just made equalto the name of the type.

ararslan · 2017-09-26T21:58:34Z

base/operators.jl

@@ -972,3 +972,22 @@ julia> filter(!isalpha, str)
 ```
 """
 !(f::Function) = (x...)->!f(x...)
+
+struct equalto{T} <: Function


This should probably use the typical upper camel case style for type names as recommended in the style guide and used in Base, especially since this is user-facing.

Disagree, since this is meant to be used like a function, equalto(x).

You can always define equalto(x) = EqualTo(x), as is done with Iterators.filter returning a Filter object.

A problem with conflating equalto with "regular" functions is that it doesn't have its own type in the same way that other functions do. In particular, to specialize on a particular function, you use ::typeof(f), but in this case typeof(equalto) will be DataType. So users will have to remember that to specialize on this function they need ::equalto instead.

I agree with @ararslan, that pattern is quite common, we also use it with zip vs. Zip1/Zip2.

True, but I'm not sure that's such a great pattern. It's confusing to have both filter and Filter. That came from a situation where filter had several methods, only one of which returned a Filter, but then the functions were separated (into Base.filter and Base.Iterators.filter). zip is also different in that it might return either a Zip1 or a Zip2, so it can't be directly identified with either type.

JeffBezanson · 2017-09-28T19:27:49Z

That's right --- @garrison 's use case would dispatch as findfirst(f::equalto, a).

ararslan · 2017-09-28T19:34:56Z

Still seems simpler and more consistent to me to do what we do in Base.Iterators: define the type using the usual conventions and make an appropriately named accessor function. I still don't see why this has to be a special case from that convention.

JeffBezanson · 2017-09-28T19:42:42Z

But then why don't we have dict(x...) = Dict(x...), rational(x) = Rational(x), etc.?

ararslan · 2017-09-28T19:53:17Z

Okay, then if this is supposed to be used as a straight type constructor rather than an accessor function, I think we should go with the stylistic conventions and call it EqualTo.

JeffBezanson · 2017-09-28T20:13:30Z

How about const EqualTo = equalto? :)

nalimilan · 2017-09-28T20:30:01Z

EqualTo sounds fine. At least it makes it clear that it's a type. The fact that it's <: Function is kind of an implementation detail in the context of find* functions.

JeffBezanson · 2017-09-28T20:32:10Z

Actually I got my last comment backwards. We could call the type EqualTo to follow the type naming convention, then have const equalto = EqualTo for nicer calling syntax for the common cases.

fix #23120, fix #19186

This fixes the deprecations due to JuliaLang/julia#23812 and makes for cleaner code anyway. (Truncation of trailing zeros is done in the `Poly` constructor so there is no need to do it here.)

GregPlowman · 2017-10-13T00:33:40Z

base/array.jl

-        if A[i] != 0
+        a = A[i]
+        if !warned && !(a isa Bool)
+            depwarn("In the future `findnext` will only work on boolean collections. Use `findnext(x->x!=0, A)` instead.", :findnext)


Should start be an argument to findnext in the depwarn message?

depwarn(" ... Use `findnext(x->x!=0, A, start)` instead.", :findnext)

Similarly for findprev?

Indeed, it should.

…e explicit Since we now need explicit predicates [1], this optimization only works if we know that the predicate is a function that is false for zero values. As suggested in that pull request, we could find out by calling `f(zero(eltype(array)))` and hoping that `f` is pure, but I like being a bit more conservative and only applying this optimization only to the case where we *know* `f` is equal to `!iszero`. For clarity, this commit also renames the helper method _sparse_findnext() to _sparse_findnextnz(), because now that the predicate-less version doesn't exist anymore, the `nz` part isn't implicit anymore either. [1]: JuliaLang#23812

timholy · 2018-01-09T11:21:47Z

Were these methods deliberately omitted, or was that an oversight? If the latter, I will likely get them as part of moving towards using nothing as a sentinel.

I'm contemplating adding a sentinel as an optional last argument, and because it could be an Integer there are all sorts of ambiguities that crop up. Do we actually need to specify start::Integer? To me it seems like it might be cleaner if the only typed argument is ::Function.

If people like that, I'd be tempted to include that change among the deprecations too, because otherwise I may have to add a bunch of specializations just to resolve ambiguities. There's some risk that this would change method sorting in a breaking way for packages

nalimilan · 2018-01-09T13:12:22Z

Good catch! These should definitely be deprecated, I must have missed them.

I'm not sure we need to specify start::Integer, and indeed as a follow-up to #24774 start should also be allowed to be of the type returned by key (which isn't known in advance).

Feel free to put everything in a PR it that makes things simpler for you.

ararslan reviewed Sep 21, 2017

View reviewed changes

ararslan added collections Data structures holding multiple items, e.g. sets deprecation This change introduces or involves a deprecation labels Sep 21, 2017

JeffBezanson force-pushed the jb/find1 branch from 060d3c4 to ebb3e05 Compare September 22, 2017 00:10

ararslan reviewed Sep 22, 2017

View reviewed changes

JeffBezanson force-pushed the jb/find1 branch 2 times, most recently from f151e89 to 6253be9 Compare September 22, 2017 02:06

ararslan approved these changes Sep 22, 2017

View reviewed changes

KristofferC reviewed Sep 22, 2017

View reviewed changes

nalimilan reviewed Sep 22, 2017

View reviewed changes

mbauman reviewed Sep 22, 2017

View reviewed changes

JeffBezanson force-pushed the jb/find1 branch from 6253be9 to 1eb0827 Compare September 22, 2017 16:45

JeffBezanson force-pushed the jb/find1 branch from d67a0b1 to 25ce69f Compare September 26, 2017 21:24

ararslan reviewed Sep 26, 2017

View reviewed changes

require explicit predicates in find functions

9bdf07f

fix #23120, fix #19186

JeffBezanson force-pushed the jb/find1 branch from 25ce69f to 9bdf07f Compare September 28, 2017 21:30

JeffBezanson merged commit 7f14cd6 into master Sep 29, 2017

JeffBezanson deleted the jb/find1 branch September 29, 2017 03:06

martinholters added a commit that referenced this pull request Sep 29, 2017

Fix deprecations of #23812 for sparse find methods

8e5c7f9

martinholters added a commit to HSU-ANT/ACME.jl that referenced this pull request Sep 29, 2017

Fix deprecations due to JuliaLang/julia#23812

c68bcbe

martinholters mentioned this pull request Sep 29, 2017

Avoid findfirst in PolynomialRatio constructor JuliaDSP/DSP.jl#181

Merged

JeffBezanson pushed a commit that referenced this pull request Sep 29, 2017

Fix deprecations of #23812 for sparse find methods (#23924)

810b0e6

Sacha0 mentioned this pull request Oct 1, 2017

Sparse vector/matrix: add fast implementation of find_next and find_prev (fixed) #23317

Merged

GregPlowman reviewed Oct 13, 2017

View reviewed changes

GregPlowman mentioned this pull request Oct 14, 2017

Fix depwarn message for findnext, findprev #24139

Merged

garrison added a commit to garrison/UniqueVectors.jl that referenced this pull request Oct 14, 2017

Migrate API to use equalto, following JuliaLang/julia#23812

a5b1238

fredrikekre mentioned this pull request Dec 6, 2017

fix deprecated array constructors JuliaCollections/DataStructures.jl#348

Merged

stevengj mentioned this pull request Dec 8, 2017

RFC: curry underscore arguments to create anonymous functions #24990

Open

5 tasks

fredrikekre mentioned this pull request Jan 3, 2018

Some corrections to findn #25365

Merged

timholy mentioned this pull request Jan 9, 2018

Deprecate a couple more find(next|prev) methods, and remove ::Integer on start #25468

Closed

nalimilan mentioned this pull request Feb 23, 2018

Performance regression from findall depwarn suggestion #26183

Closed

garrison mentioned this pull request Mar 21, 2018

Make Compat.Fix2 always exist JuliaLang/Compat.jl#521

Merged

require explicit predicates in find functions #23812

require explicit predicates in find functions #23812

Conversation

JeffBezanson commented Sep 21, 2017 • edited by nalimilan Loading

JeffBezanson commented Sep 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ararslan commented Sep 21, 2017

TotalVerb commented Sep 22, 2017

JeffBezanson commented Sep 22, 2017

Choose a reason for hiding this comment

JeffBezanson commented Sep 22, 2017

ararslan left a comment

Choose a reason for hiding this comment

ararslan commented Sep 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nalimilan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeffBezanson commented Sep 22, 2017

Choose a reason for hiding this comment

JeffBezanson commented Sep 22, 2017 • edited Loading

JeffBezanson commented Sep 22, 2017

nanosoldier commented Sep 23, 2017

garrison commented Sep 23, 2017

garrison commented Sep 23, 2017 • edited Loading

garrison commented Sep 23, 2017 • edited Loading

JeffBezanson commented Sep 23, 2017

JeffBezanson commented Sep 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeffBezanson commented Sep 28, 2017

ararslan commented Sep 28, 2017

JeffBezanson commented Sep 28, 2017

ararslan commented Sep 28, 2017

JeffBezanson commented Sep 28, 2017

nalimilan commented Sep 28, 2017

JeffBezanson commented Sep 28, 2017

GregPlowman Oct 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timholy commented Jan 9, 2018

nalimilan commented Jan 9, 2018

require explicit predicates in `find` functions #23812

require explicit predicates in `find` functions #23812

JeffBezanson commented Sep 21, 2017 •

edited by nalimilan

Loading

JeffBezanson commented Sep 22, 2017 •

edited

Loading

garrison commented Sep 23, 2017 •

edited

Loading

garrison commented Sep 23, 2017 •

edited

Loading

GregPlowman Oct 13, 2017 •

edited

Loading