Make itertuples really an iterator/generator in implementation, not just return type

[mitar]

itertuples is not really an iterator/generator and constructs a copy of whole DataFrame in memory. Ideally it would return just an iterator and construct row by row as it is being iterated over.

[jreback]

looks like a generator to me

In [1]: df = pd.DataFrame({'A': range(3), 'B': list('ABC')})

In [2]: df.itertuples()
Out[2]: <map at 0x10922c080>

In [3]: list(df.itertuples())
Out[3]: 
[Pandas(Index=0, A=0, B='A'),
 Pandas(Index=1, A=1, B='B'),
 Pandas(Index=2, A=2, B='C')]

In [5]: i = df.itertuples()

In [6]: next(i)
Out[6]: Pandas(Index=0, A=0, B='A')

In [7]: next(i)
Out[7]: Pandas(Index=1, A=1, B='B')

In [8]: next(i)
Out[8]: Pandas(Index=2, A=2, B='C')

In [9]: next(i)
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-9-a883b34d6d8a> in <module>()
----> 1 next(i)

StopIteration: 

[mitar]

That's just because you return a zip. But inside the function, you create whole list of values first. See here:

arrays.extend(self.iloc[:, k] for k in range(len(self.columns)))

It looks like iterator because of zip(*arrays), but arrays is not an iterator. This is a problem.

[mitar]

You can test the issue here by doing:

d = pandas.DataFrame({'a': range(100000000)})
for a in d.itertuples(index=False, name=None):
    print(a)

Do this in Python interpreter. Note the time it takes to create d. Now, when you press enter after print line, twice, note the time it takes to start producing anything. It takes this time because it is first creating all rows in memory, before iterating over them.

[jreback]

and if u want to fix it pls submit a PR
this is truly an iterator

so what u describe is an implementation detail

[mitar]

Implementation detail which blows up memory and performance?

Anything can be made look like iterator. But if does not really behave like iterator, it is not an iterator.

I think this is a bug. Please reopen this. And then me or somebody else can make a pull request.

[mitar]

This goes even deeper. Also iterating over a series constructs a list internally:

d = pandas.DataFrame({'a': range(100000000)})
for a in d['a']:
    print(a)

This will also have a large delay before starting sending results back.

[jorisvandenbossche]

I agree that ideally it would be more lazy the iteration (although not really a priority issue for me), and since we would accept a PR to fix, let's keep the issue open.

[jreback]

@mitar you are welcome to submit a PR, however, this method follows exactly the pandas paradigm. We create a new copy of things then hand it back to you, here the handing back is an iterator. If you can optimize this, great, but you are fighting standard practice. Further you may slow things down by doing this in the common case.

[TomAugspurger]

This would be a welcome fix if possible.

@mitar a couple things to watch out for, which we just hit with Categorical.__iter__:

• Scalars have to be converted from NumPy scalars to Python scalars
• We need to avoid tons of calls to Series/Frame.__getitem__, as this is relatively slow

[mitar]

I made: #20796

[mitar]

We need to avoid tons of calls to Series/Frame.__getitem__, as this is relatively slow

Calling iloc is OK? Or is this the same slow as __getitem__?

[mitar]

I think that the PR #20796 is ready to be reviewed.