`itertools.ilen` addition

[dg-pb]

Feature or enhancement

Proposal:

Introduction

Discussion has not suffered any terminal blows since its inception and has been silent for a while now. So I am taking an opportunity to make one more attempt to suggest addition of ilen function.
Given the current state and coverage of itertools module, I think ilen can be a useful and timely addition.

Past attmepts

#53756 (comment). There is a clear indication that it is not the first time this was proposed. Although repreated rejection does not shine a positive light on this proposal, it can also indicate a reocurring attempt to satisfy a need.

Addressing criticism

Statement: "The core issue is that it is not a very useful operation because it consumes the iterator." - #53756 (comment)
Response: "I think the utility of consuming an iterator immediately to find its length, without regard to the contents, is greater than some might expect. After all, there’s a Unix utility (wc) dedicated to exactly that." - https://discuss.python.org/t/itertools-ilen-iterable/53002/23
Support by Use Case count

56      stdlib: `ripgrep '\Wlen\(list\(' | wc -l`
305K    github: `/.*\Wlen\(list\(.*/ language:Python`
1.3K    github: `/.*\Wilen\(.*\).*/ language:Python`

Also, it is most likely that len(list(...)) is the most commonly used solution, which is the least memory efficient among them all (see below).

My personal experience of usefulness

• ilen is used 6 times in the module of iterator recipes which contains 40 functions.
• Often used as infinite consumer (instead of collections.deque(it, maxlen=0))
  • Often in iterator recipe benchmarks. Although I mostly use black_hole = collections.deque(maxlen=0).extend now.

Python packages that implement it

Any iterator library will inevitably going to re-implement it:

• more_itertools.ilen
• iteration_utilities.count_items
• iters.iter.length

Implementations in other languages

• Rust: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.count
• C++: https://cplusplus.com/reference/iterator/distance/
• JavaScript: https://github.com/stdlib-js/iter-length
• JAVA(guava): https://guava.dev/releases/snapshot/api/docs/com/google/common/collect/Iterators.html#size(java.util.Iterator)

Stack threads

• https://stackoverflow.com/questions/3345785/getting-number-of-elements-in-an-iterator-in-python
• https://stackoverflow.com/questions/390852/is-there-any-built-in-way-to-get-the-length-of-an-iterable-in-python
• https://stackoverflow.com/questions/5384570/how-can-i-count-the-number-of-items-in-an-arbitrary-iterable-such-as-a-generato

Solutions that are currently used (or have been considered)

import more_itertools
import iteration_utilities

@cython.locals(i=cython.int)
def cython_ilen(iterable):
    i = 0
    for _ in iterable:
        i += 1
    return i

a = range(100_000)
%timeit sum(1 for _ in iter(a))                     # 6.1 ms
%timeit sum(map(lambda i: 1, iter(a)))              # 5.5 ms
%timeit coll.deque(enumerate(iter(a)), maxlen=1)    # 4.4 ms
%timeit more_itertools.ilen(iter(a))                # 3.5 ms
%timeit sum(map(len, itl.batched(iter(a), 5)))      # 2.9 ms (suboptimal memory consumption)
%timeit len(list(iter(a)))                          # 2.6 ms (highest memory consumption)
%timeit sum(map(len, itl.batched(iter(a), 1000)))   # 2.2 ms (suboptimal memory consumption)
%timeit iteration_utilities.count_items(iter(a))    # 1.5 ms (more complex than it needs to be)
%timeit cython_ilen(iter(a))                        # 1.5 ms
%timeit proposed_ilen(iter(a))                      # 1.45 ms
%timeit coll.deque(iter(a), maxlen=0)               # 1.42 ms (fastest consumer in stdlib)

Couple of use cases

1. os.path.commonprefix

# Current:
def commonprefix(m):
    ...
    s1 = min(m)
    s2 = max(m)
    for i, c in enumerate(s1):
        if c != s2[i]:
            return s1[:i]
    return s1

# Alternative:
def commonprefix(m):
    ...
    s1 = min(m)
    i = ilen(takewhile(bool, map(opr.eq, s1, max(m))))
    return s1[:i]

2. takewhile_select

def takewhile_select(selectors, iterable):
    """Take while parallel elements in selectors are true
    Examples:
        >>> list(takewhile_select([1, 1, 0, 0], [0, 1, 2, 3]))
        [0, 1]
    """
    n = ilen(takewhile(bool, selectors))
    return islice(iterable, n)

Implementation considerations

Flexibility such as offered in iteration_utilities.count_items has been considered. However, I think it is best to keep it simple. Preprocessing of any kind can be done before calling ilen.

Maintenance cost

Minimal. It would be the simplest function in itertools.

Final thoughts

To me personally, this is the one missing function in itertools. Currently, I am using cython_ilen when the module is cython-compiled and a copy of more_itertools.ilen when it is not. This addition would simplify the matters.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/itertools-ilen-iterable/53002

Linked PRs

• gh-120478: itertools.ilen addition #120483

[nineteendo]

def commonprefix(m):
    ...
    s1 = min(m)
    i = ilen(takewhile(bool, map(opr.eq, s1, max(m))))
    return s1[:i]

I don't think this is a good example, it's not at all obvious what it's doing. An explicit for loop is easier to maintain.

[picnixz]

Personally, I have reasons to say yes and no (if you prefer me to go to Discourse for a follow-up discussion, tell me).

• +1 because it's useful and I'm tired of using the deque approach.
• +1 because you can avoid having a (more or less slow/fast) dependency/util function
• -1 because this only benefits code where the bottleneck is consuming the iterable (in general you usually want to work with the iterated items and ilen is mostly combined with more complex recipes that are more costly in the end, so having the slow simple util function is somewhat fine IMO).

I'd still advocate for its addition but would not recommend it for infinite consumer. Infinite consumer could be implemented separately in itertools but you do not even need to return the length (and this could consume whatever you want at C speed).

[dg-pb]

Feedback is always appreciated. Here, in discussions or PM.

To me, pre-initialized coll.deque(maxlen=0).extend is perfect and I don't think anything better can be done. It could be argued that it should be added with better alias, but personally I love that I am forced to name it. :)

I have found out why it is so fast only today. It actually has a proper exception for case when maxlen=0.

The only drawback to me was initialization cost (I couldn't trust it for very short iterables), but only because I didn't think of: more-itertools/more-itertools#848.

Having that said, ilen is only marginally slower. It would be the 2nd fastest infinite consumer. I would still use it for consumption when I am drafting, but not in any production/public code.

[picnixz]

What I meant by a fast consumer is to just have the consume_iterator call without anything else around it. A pure function that does not need to instantiate or prepare a deque object or work on one. Maybe we could open a discussion on whether a consumer in itertools could be considered or not.

(If so, I'd be interested in implementing it since I think it's good as a starting issue.)

[dg-pb]

Personally, I don't see it going through. Initialization of 1 object is not a big issue and it is as fast as it can be. Naming is a bit awkward, but anyone can rename it when pre-instantiating. To me, it is pretty perfect the way it is.

What could be a good idea is to add it to official documentation recipes. I might even dare to suggest it at the very top.

[dg-pb]

I take it back. It already exists. My bad.

[picnixz]

Yes, I know it in the recipe but I was wondering whether having a dedicated function in itertools or not. There's always the debate of whether a 3-line function should be part of the stdlib but if we gain something without the needs of a deque it might be worth considering (since you are opening the discussion on ilen). Anyway, this is probably not the place to discuss it so I'll hide this reply.

[rhettinger]

Thank you for the suggestion but I will decline this for the itertools module. I don't see it as part of a core "iterator algebra" or as a member of a toolset mostly devoted to lazy evaluation and side-effect free functional programming. Itertools was intentionally scoped at the outset to be callables that produce iterators. Tools that just consume iterators belong elsewhere.

Even if this were in the module scope, I don't think the function is advisable for general use. For me at least, it is uncommon to have only an iterator data source, to want a count, but not to be interested in the data itself. If a need did arise, the existing alternatives are good enough — not something that would warrant being reimplemented in C to solve a speed bottleneck (in the general case, data production likely dominates the counting time). Also as a code reviewer, I would worry that ilen() calls don't scale well and that the blocking behavior would be eventually reported as a bug.

The function would need a name that more clearly suggests, "run an iterator to exhaustion and return a count of the number of values". Ideally, the name should avoid the i-prefix which has historically meant "return an iterator." Also a good name would avoid false analogies with len() which is generally expected to be O(1), to never fail, to match the boolean value the input, to always terminate, and to not mutate the input.

If this is important to you, consider some alternatives. Perhaps, publish your own iterator consumer library on PyPI or ask the more-itertools project if it would expand its scope to accept C accelerators.

Apologies for taking a while to respond. It took a while to read through all of your references. Also, I wanted to see if any core developers wanted to comment.

[dg-pb]

I am still convinced ilen is currently the most needed addition.

E.g. I am writing string regular expression search and I want to add a flag to only return the count. Either I need to store all of matches in a list len(list(iterable)), use inefficient loop, depend on external library, sub-optimal performance recipe, etc...

Point is, none of those are optimal having in mind it is such a basic operation. Same as wc -l in bash.

[pochmann3]

For your regex match counting, maybe re.subn would be good.