[Stdlib] Implement find_first, find_last for maps and sets

[g2p]

Finds the first or last binding or element that satisfies a monotonic predicate.

See discussion in #665.

[alainfrisch]

Making the case for extending the stdlib is always difficult. Can you for instance provide pointers to publicly available code that would benefit from the new functions?

[bluddy]

I've had to implement these functions before, and because of type opaqueness, had to copy-paste the entire map implementation. I think this is a valuable addition.

[yakobowski]

@alainfrisch https://github.com/Frama-C/Frama-C-snapshot/blob/master/src/libraries/stdlib/FCSet.mli, functions nearest_elt_le and nearest_elt_ge, are very similar to what is proposed here.

[alainfrisch]

Thanks @bluddy and @yakobowski .

@yakobowski : Would the addition of the functions proposed here avoid a custom implementation of sets in Frama-C?

[yakobowski]

@alainfrisch I think so, yes. The standard library has caught up with the other changes we used to have (such as merge).

[dbuenzli]

Couldn't new functions use options rather than Not_found exceptions ?

[OvermindDL1]

As a user, please, 'something option is so much better than exceptions...

/me would give a lot to have a set of functions for the stdlib like MyMap.get myKey myMap that returned an option instead of throwing exceptions all over the place, plus an Option module that had things like map or andThen and so forth...

[bluddy]

Consistency. Unless we deprecate and rebuild the whole stdlib (not a bad idea but not the place to do it), we can't just start using options. Also, doesn't option allocate more than using exceptions? You can't always count on Flambda to eliminate the allocation.

[dbuenzli]

Consistency. Unless we deprecate and rebuild the whole stdlib (not a bad idea but not the place to do it), we can't just start using options.

Why not ? I don't think one should enshrine design errors in the name of consistency. There's absolutely no loss here at not being consistent you'll be guided by the types. It only makes the API easier and safer to use.

@OvermindDL1 you might be interested by asetmap.

[OvermindDL1]

@OvermindDL1 you might be interested by asetmap.

Lots of useful libraries out there, but I am trying to depend only on stdlib and nothing else for a library that I am building. I keep having to write wrappers around various stdlib things, it is compounding a lot. ^.^

And yes, I find 'something option significantly safer, forces you to at least think about the error cases instead of just trying to remember to Blah.mem first, or whatever lookup test there is in a module, the naming is anything but consistent; it would indeed be very nice to rewrite stdlib from scratch with all modern styles that have been learned over time for efficiency and safety both, release an OCaml 5.0.0 for that.

Why can't Flambda eliminate the allocation? It seems that both option and Result.t would be two very good cases to optimize if not generically then specifically.

EDIT0: Re-making the stdlib after Implicit Modules exist that is, and preferably concurrency and no GIL so everything can be made properly immutable or safe in some way...

EDIT1: Do feel free to ignore me, just hard to resist piping in when I saw a conversation between throwing or option, throwing is very... flow-control disrupting, makes things harder to reason...

[g2p]

I want find_first to be consistent with find and throw an exception.
Thanks to exception matching, usage is just as simple as matching on an option.
My code that depends on this isn't public yet (these functions are used to implement a B+Tree), but with this and #665 we have at least five users relying on independent reimplementations of similar functions.

[OvermindDL1]

Thanks to exception matching, usage is just as simple as matching on an option

Exception matching is nice, but it is still easily ignorable and forgettable, where an option mandates making sure to handle both cases and not worrying about something weird happening, like the callstack suddenly unwinding who knows how high just because of some quickly written code. ^.^

[alainfrisch]

About Not_found vs option: I tend to be on @dbuenzli side here. Consistency is a nice goal, but being consistent with a bad design for the sake of consistency only does not seem right. A point could be made that if we plan to add option-returning variants with a specific naming scheme (e.g. find_opt, assoc_opt, etc), the new functions should directly follow that naming scheme

[bluddy]

Wow, @alainfrisch -- that was unexpected.

If anything though, we'd want to label the exception-based functions with _exn as Jane Street does. That would hint that they're less desirable, and convey some information about their usage of exception.

[alainfrisch]

Also, doesn't option allocate more than using exceptions?

Yes, but setting up the exception handler can be costly as well, and you more or less have to do it around each call anyway.

A very quick test with:

let rec find p = function
  | [] -> raise Not_found
  | x :: l -> if p x then x else find p l

let rec find_opt p = function
  | [] -> None
  | x :: l -> if p x then Some x else find_opt p l

let f1 () =
  try find p l
  with Not_found -> 0

let f2 () =
  match find_opt p l with
  | Some x -> x
  | None -> 0

suggests that the find_opt + match..with version is about 50% faster than find + try..with with let p _ = true, and between 0% (for long lists) and 80% faster (for short lists) with let p _ = false.

[bluddy]

Good to know. Could this be FLambda at work? Because it can't ever optimize setting up exceptions, but it can optimize options being returned if the call is inlined.

[dbuenzli]

If anything though, we'd want to label the exception-based functions with _exn as Jane Street does.

This is quite ugly. Usually you can find better names conveing your intent for that. For example you can have find for the paths where you are unsure something matches (and hence returns an option) and get for the paths where you know by construction something has to match (and raises Invalid_argument if that thing was not in there).

[alainfrisch]

Could this be FLambda at work?

No, timings were without flambda. Allocating short-lived blocks (such as an immediately deconstructed option) is very cheap (at least with the official OCaml runtime system; Javascript backends are another story).

[dra27]

+1 to still having an exception version, but with a name which makes it clear why you'd use it (i.e. find+get) - assuming that it's the case that the exception version with no exception handler is faster than the 'a option version?

[alainfrisch]

(#885 proposes to add non-raising variants for existing functions.)

[OvermindDL1]

assuming that it's the case that the exception version with no exception handler is faster than the 'a option version?

Most of the time you'd still have something like a MyMap.mem check before a MyMap.get, which would likely be more costly anyway.

[dra27]

Most - but definitely not all the time. An obvious example being keyword symbols in a parser, and also any situation where you would have one mem call before more than one get call. Either Daniel's suggestion of using Invalid_argument (which is never supposed to be caught) and/or documentation should make it very clear that find_opt is probably what's wanted 95% of the time.

[alainfrisch]

So, assuming #885 is accepted, what should we do here? To have a consistent naming, the new option-returning functions should be called *_opt, but if we don't plan to include the Not-found-raising versions, the suffix would be technically useless.

[dra27]

Given your work in #885, there should obviously be no question that there must be option-returning versions. I also think there's enough here to suggest that there should also be exception-raising versions. The two big questions are:

1. Should the exception-raising versions raise Not_found or Invalid_argument _. Despite the inconsistency, I think I'd be in favour of Invalid_argument, as it moves in the direction of encouraging correct use: although that's balanced against quite a "gotcha" of catching Not_found for analogous functions elsewhere.
2. The precise names - for this, while I really prefer Daniel's suggest of intent-based naming (get/find, etc.), I think consistency of naming is probably better, so the option-returning versions should have _opt suffices, and the exception-raising versions be undecorated.

[gasche]

You could be forgiven by answering Alain's question, giving your external opinion on whether you think this is a good interface. (cc: @c-cube?)

[g2p]

The docstrings were changed per @hcarty's suggestions.

The interface was discussed in #665 as well, I picked the best option from that discussion.

[c-cube]

I don't have a strong opinion on this interface, it looks ok to me. Maybe it would be good to document more explicitely what "monotonic" is (i.e. if [f k && compare k k' <= 0] holds, then [f k'] must hold), and maybe show a possible use case?

[g2p]

I added an example; I think it's enough to illustrate what a monotonic function is, since it's not especially ambiguous.

[hcarty]

@gasche The interface looks OK to me as well. It seems reasonably clear from the type + name of each function what to expect.

[gasche]

I'll still let @alainfrisch make merge decisions. I don't want to commit to working on the standard-library-extension project, because I have too much on my plate already and it's frustrating work.

Re-reading this PR discussion, it looks like there is agreement that the interface is good, but it's not completely clear to me whether there is a consensus on the name of the exception-raising variants. I would personally be tempted to "let those who do the work decide" and go for @g2p's preference, but stdlib matters are thorny as current choices impact future choices and it's easy to have regrets in the future. Not my call!

@g2p: could you amend the Changelog entry to indicate the people whose review/discussion/proposals helped the PR move forward? You don't need to cite everyone involved in the discussion, but people whose reviewing effort you found substantial and impactful.

[dbuenzli]

but it's not completely clear to me whether there is a consensus on the name of the exception-raising variants.

Since exception-raising variants are provided I think the current PR is the best naming scheme you can find.

[g2p]

Credits added, you're right. Thanks everyone for discussing and reviewing.

[g2p]

(rebased to fix a conflict in the change log)

[alainfrisch]

Thanks to @g2p and everyone involving in the discussion for moving this forward.