Feature Request: df.apply()

[aamirkhan34]

Requesting the feature df.apply() . I did not find any issues regarding this.

Thanks.

[sethmlarson]

This is unlikely to be implemented as its no more efficient than ed_df.to_pandas().apply(). Is there a use-case in particular that's more efficient to implement and desirable?

[aamirkhan34]

Thanks @sethmlarson.
The ed_df.to_pandas() method is slow and might run out of memory for larger sample. I want to harness the power of our elasticsearch cluster to process the eland dataframe using apply method. This will be very efficient for our process.

What do you think?

Thanks.

[sethmlarson]

Unfortunately apply is very generic, you can basically pass it anything and we can't transform arbitrary Python functions into an Elasticsearch query. Is there some operation(s) in particular you're interested in?

[kxbin]

Thanks @sethmlarson.
The ed_df.to_pandas() method is slow and might run out of memory for larger sample. I want to harness the power of our elasticsearch cluster to process the eland dataframe using apply method. This will be very efficient for our process.

What do you think?

Thanks.

Yeah, I also found that ed_df.to_pandas() method is very slow for larger sample.

So, Maybe we can process the data in batches，Like this:

pd_df_iterator = ed_df.to_pandas_in_batch(batch_size=1000)
for pd_df in pd_df_iterator:
    pd_df.apply()

After testing, I found that the speed has increased a lot, because the amount of data in each batches is determined by batch_size.

Here is a pull request to handle this situation:
Add to_pandas_in_batch() DataFrame API #369