RFC: add `Base.Iterators` module #18839

JeffBezanson · 2016-10-07T21:46:14Z

Wraps the definitions that were in base/iterator.jl in a module, as has been discussed.

I found that zip and enumerate are so widely used that we should probably continue to export them from Base.
Deprecating the other previously-exported iterators is tricky, since if Base exports them as part of a deprecation, then using IterTools will conflict. We should add a rule that deprecated bindings can be shadowed instead of giving the "both X and Y export Z; uses of it must be qualified" error.
I was able to remove iterator.jl from coreimg.jl entirely, which is nice.
I separated Base.filter and IterTools.filter into two functions (since the latter is lazy and the former is eager). This is up for debate.

ararslan · 2016-10-07T23:11:31Z

Perhaps the lazy filter from the submodule could be called lazyfilter? That avoids any potential name confusion.

For what it's worth (which I doubt is much), I'm personally not a huge fan of the name IterTools--sounds too "Pythonic." I think something like Iterators might be nicer. Anyway, just a thought.

iamed2 · 2016-10-08T02:19:26Z

Iterators was rejected because Iterators.jl is a package.

cstjean · 2016-10-08T15:42:39Z

I separated Base.filter and IterTools.filter into two functions (since the latter is lazy and the former is eager). This is up for debate.

Base.filter over iterators is lazy. #13712

JeffBezanson · 2016-10-09T03:04:46Z

Yes, I separated the functions so that that's no longer the case. I guess I was a bit unclear; the parenthetical applies to the new functions.

tkelman

lgtm if green

tkelman · 2016-10-12T07:41:09Z

This is probably worth mentioning in NEWS.md

kshyatt · 2016-10-12T18:24:56Z

Needs a rebase and possibly that NEWS addition?

JeffBezanson · 2016-10-12T18:30:35Z

Yep, and it occurs to me I should probably add a doc/stdlib file+section for the module.

JeffBezanson · 2016-10-13T19:17:39Z

We discussed this on the triage call. Consensus was that IterTools is an ugly name, and we don't want to get stuck with it. So now we have Base.Iterators as well as Iterators (the package). We felt that using the same name actually has better usability.

I tried this out, and the name conflict doesn't seem to be inconvenient. You can even do both using Iterators and using Base.Iterators and it works (aside from name conflicts, which in this case are easy to resolve by removing things from the Iterators package that are now in Base).

This will be more future-proof; we can potentially break up the Iterators package by (1) moving some things to Base.Iterators, (2) move more exotic iterators to their own packages. Then we'd be left with just Base.Iterators instead of just Base.IterTools.

tkelman · 2016-10-13T20:55:52Z

wouldn't it be better to move everything from Base.Iterators to Iterators.jl and make it a default package?

iamed2 · 2016-10-13T21:04:25Z

I feel like the best solution is to either move everything from Iterators.jl into Base (a la Rust) or reexport Base.Iterators from Iterators.jl so that users don't ever have to run into name conflicts.

JeffBezanson · 2016-10-13T22:07:00Z

reexport Base.Iterators from Iterators.jl

That's a good idea.

One tricky thing about Base.Iterators is that it has some functions that are very widely used in Base, plus some things that are needed for lowering generators and comprehensions, so it can't be fully separated in any meaningful way (unless we split it into ImportantIterators and OtherIterators).

tkelman · 2016-10-14T01:05:47Z

Seems that anything that isn't absolutely a dependency of all other code written in the language should eventually be a package, so important vs other is a fair dividing line between what should be here and what shouldn't.

StefanKarpinski · 2016-10-14T01:15:55Z

You can pull that thread until there's no sweater left.

tkelman · 2016-10-14T02:38:47Z

are we serious about #5155 or not?

StefanKarpinski · 2016-10-14T02:49:52Z

Yes, but my point is that there's no hard line between what's necessary for Base and what isn't – and it can change over time. We should try to make the distinction a little less transient.

JeffBezanson · 2016-10-14T03:30:26Z

Yeah I'm serious about it --- in that issue I list 26 modules to move out, without even mentioning this one.

tkelman · 2016-10-14T04:41:54Z

Yes and that's a good list, but I don't see why the parts of Iterators that aren't needed for bootstrapping or lowering shouldn't be on it too.

JeffBezanson · 2016-10-14T05:52:10Z

Maybe so; it's just a pretty small amount of code that's broadly applicable. Several of the iterators are <30 lines of code each, so moving them is a low priority. Some belong by symmetry; for example I wouldn't want take and drop in separate packages just because only one happened to be used in Base.

tkelman · 2016-10-14T05:54:25Z

base/exports.jl

@@ -23,6 +23,7 @@ export
    Docs,
    Markdown,
    Threads,
+    Iterators,


this probably shouldn't be exported, won't it conflict with the package name?

Good point. I tried both ways, and exporting it didn't change the situation much. The way using works, if you do using Iterators first it loads the package with no warnings. All the other public submodules of Base are exported, so I decided to be consistent. However I'm willing to remove all or most of these exports as part of #5155; some will become top-level modules instead of submodules, and for the remainder we can change the policy on exporting them.

if you do using Iterators without the package installed will it behave differently than with?

No. using can't see Base.Iterators unless you do e.g. import Base.Iterators.

Ah. But if exported from Base, does Iterators refer to something different before you do using Iterators vs after?

In that case yes, and you get the usual WARNING: imported binding for Iterators overwritten in module Main.

it looks like this is now confusing precompile because it finds the Main.Iterators binding (probably using the wrong logic) and ends up having that shadow the real Iterators package. This shows up as breakage of the Compose package:

julia> using Compose WARNING: The call to compilecache failed to create a usable precompiled cache file for module Compose. Got: WARNING: Module Iterators uuid did not match cache file. julia> using Iterators WARNING: imported binding for Iterators overwritten in module Main julia> using Compose julia>

GiovineItalia/Compose.jl#225

Yes, precompile should use the equivalent of eval_import_path.

lstagner · 2016-10-16T02:07:13Z

Perhaps #17935 could also be included with this

This removes a spurious warning if Base.X is deprecated to Mod.X, and one does `using Mod`. `X` is available from both `Base` and `Mod`, but we asked for the non-deprecated one so there should be no warning.

fixes deprecation warnings on v0.6. see JuliaLang/julia#18839 for more details.

cstjean · 2017-03-03T14:21:20Z

On 0.6

julia> filter(x->true, (1,2))
WARNING: filter(flt, itr) is deprecated, use Iterators.filter(flt, itr) instead.
Base.Iterators.Filter{##3#4,Tuple{Int64,Int64}}(#3, (1, 2))

Shouldn't that be handled by Base.filter instead of Base.Iterators.Filter?

yuyichao · 2017-03-03T14:25:38Z

What should be handled by the deprecated Base.filter?

cstjean · 2017-03-03T14:32:00Z

Sorry, I misunderstood the deprecation strategy. Never mind, it makes sense.

JeffBezanson force-pushed the jb/itertools branch 6 times, most recently from 510de68 to 211d14d Compare October 11, 2016 05:05

tkelman approved these changes Oct 11, 2016

View reviewed changes

JeffBezanson force-pushed the jb/itertools branch from 211d14d to b927d43 Compare October 11, 2016 15:51

JeffBezanson changed the title ~~add IterTools module~~ RFC: add IterTools module Oct 11, 2016

tkelman added the needs news A NEWS entry is required for this change label Oct 12, 2016

JeffBezanson force-pushed the jb/itertools branch 2 times, most recently from cc7bfdb to 36f1f82 Compare October 12, 2016 22:20

JeffBezanson removed the needs news A NEWS entry is required for this change label Oct 12, 2016

JeffBezanson force-pushed the jb/itertools branch from 36f1f82 to 7ade52d Compare October 13, 2016 19:10

JeffBezanson changed the title ~~RFC: add IterTools module~~ RFC: add Iterators module Oct 13, 2016

JeffBezanson changed the title ~~RFC: add Iterators module~~ RFC: add Base.Iterators module Oct 13, 2016

tkelman reviewed Oct 14, 2016

View reviewed changes

JeffBezanson mentioned this pull request Oct 14, 2016

remove iterator.jl from coreimg #18934

Merged

JeffBezanson added 3 commits October 18, 2016 10:48

return false for deprecated bindings in is_exported_from_stdlib

d149055

if a binding exists in multiple usings, prefer a non-deprecated one

fe3b081

This removes a spurious warning if Base.X is deprecated to Mod.X, and one does `using Mod`. `X` is available from both `Base` and `Mod`, but we asked for the non-deprecated one so there should be no warning.

add Iterators module around extra iterator functionality

7cc22c5

JeffBezanson force-pushed the jb/itertools branch from 7ade52d to 7cc22c5 Compare October 18, 2016 15:00

JeffBezanson merged commit 333ad95 into master Oct 18, 2016

tkelman mentioned this pull request Oct 19, 2016

remove text example from tests (2nd option for fixing #222) GiovineItalia/Compose.jl#225

Closed

This was referenced Oct 21, 2016

Reattach filter's docstring #19057

Merged

Improve(?) deprecation warnings for deprecated iterators #19058

Closed

vtjnash mentioned this pull request Oct 25, 2016

fix incremental requires #19108

Merged

vchuravy mentioned this pull request Nov 2, 2016

Import Base.Iterators on v0.6 JuliaCI/BenchmarkTools.jl#25

Merged

tlnagy added a commit to tlnagy/Lazy.jl that referenced this pull request Nov 17, 2016

switch to using new Base.Iterators module on v0.6

eba73e3

fixes deprecation warnings on v0.6. see JuliaLang/julia#18839 for more details.

This was referenced Nov 17, 2016

switch to using new Base.Iterators module on v0.6 MikeInnes/Lazy.jl#55

Merged

Preparing for v0.6 GiovineItalia/Gadfly.jl#930

Closed

tlnagy mentioned this pull request Nov 17, 2016

switch to using new Base.Iterators module on v0.6 JuliaGraphics/Showoff.jl#1

Merged

tlnagy added a commit to tlnagy/Showoff.jl that referenced this pull request Nov 22, 2016

switch to using new Base.Iterators module on v0.6

98cdf21

fixes deprecation warnings on v0.6. see JuliaLang/julia#18839 for more details.

cstjean mentioned this pull request Mar 7, 2017

Clarify iterator behaviour (filter is half-lazy) #13712

Closed

yuyichao mentioned this pull request Apr 9, 2017

filter is deprecated, but does not give a deprecation warning #21334

Closed

JeffBezanson deleted the jb/itertools branch February 1, 2018 16:50

Gyoshi mentioned this pull request Dec 29, 2020

filter doesn't work on generators #37146

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: add `Base.Iterators` module #18839

RFC: add `Base.Iterators` module #18839

JeffBezanson commented Oct 7, 2016 •

edited

ararslan commented Oct 7, 2016 •

edited

iamed2 commented Oct 8, 2016

cstjean commented Oct 8, 2016

JeffBezanson commented Oct 9, 2016

tkelman left a comment

tkelman commented Oct 12, 2016

kshyatt commented Oct 12, 2016

JeffBezanson commented Oct 12, 2016

JeffBezanson commented Oct 13, 2016

tkelman commented Oct 13, 2016

iamed2 commented Oct 13, 2016

JeffBezanson commented Oct 13, 2016

tkelman commented Oct 14, 2016

StefanKarpinski commented Oct 14, 2016

tkelman commented Oct 14, 2016

StefanKarpinski commented Oct 14, 2016

JeffBezanson commented Oct 14, 2016

tkelman commented Oct 14, 2016 •

edited

JeffBezanson commented Oct 14, 2016

tkelman Oct 14, 2016

JeffBezanson Oct 14, 2016

tkelman Oct 14, 2016

JeffBezanson Oct 14, 2016

tkelman Oct 14, 2016

JeffBezanson Oct 14, 2016

vtjnash Oct 19, 2016

tkelman Oct 19, 2016

JeffBezanson Oct 19, 2016

lstagner commented Oct 16, 2016

cstjean commented Mar 3, 2017

yuyichao commented Mar 3, 2017

cstjean commented Mar 3, 2017

RFC: add Base.Iterators module #18839

RFC: add Base.Iterators module #18839

Conversation

JeffBezanson commented Oct 7, 2016 • edited

ararslan commented Oct 7, 2016 • edited

iamed2 commented Oct 8, 2016

cstjean commented Oct 8, 2016

JeffBezanson commented Oct 9, 2016

tkelman left a comment

Choose a reason for hiding this comment

tkelman commented Oct 12, 2016

kshyatt commented Oct 12, 2016

JeffBezanson commented Oct 12, 2016

JeffBezanson commented Oct 13, 2016

tkelman commented Oct 13, 2016

iamed2 commented Oct 13, 2016

JeffBezanson commented Oct 13, 2016

tkelman commented Oct 14, 2016

StefanKarpinski commented Oct 14, 2016

tkelman commented Oct 14, 2016

StefanKarpinski commented Oct 14, 2016

JeffBezanson commented Oct 14, 2016

tkelman commented Oct 14, 2016 • edited

JeffBezanson commented Oct 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lstagner commented Oct 16, 2016

cstjean commented Mar 3, 2017

yuyichao commented Mar 3, 2017

cstjean commented Mar 3, 2017

RFC: add `Base.Iterators` module #18839

RFC: add `Base.Iterators` module #18839

JeffBezanson commented Oct 7, 2016 •

edited

ararslan commented Oct 7, 2016 •

edited

tkelman commented Oct 14, 2016 •

edited