-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add iterrows()
and itertuples()
DataFrame API, its usage is similar to pandas
#369
Conversation
Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually? |
💚 CLA has been signed |
So, I am thinking with this #368 We will eliminate Since, I am thinking we can add a new parameter to Also turn existing method into iterator by default
Once this is done. I am thinking if @sethmlarson What do you think ? |
@V1NAY8 Thanks for the reply I agree with you. We can throw a warning saying 10k is maximum and fetch results with max batch size,I will modify it now. In addition, I also considered turn existing method into iterator by default. Maybe we can determine whether to return an iterator or a dataframe based on the batch_size parameters (batch_size is None or not None). |
@kxbin Don't throw a warning now in this PR. The existing code can run for both < 10k and > 10k. I will throw if needed in my other changes. |
@V1NAY8 Okay, I'll trouble you to throw a warning. |
So, we can turn internally into iterator by default. The method |
This idea is great 😃 |
Thanks @kxbin and @V1NAY8 for your interest in this feature. I discussed this with the team and I believe we're thinking it makes sense to implement The reason we'd like to do this instead of a DataFrame chunking API is it provides compatibility with pandas and only allows two ways to view your data (either "all" the data or every row) instead of any number of views based on |
@sethmlarson Thanks for the reply Yeah, If we can be compatible with We can implement the following methods: Make it similar to pandas usage: Please give me some time, I think I can finish it. 😛 |
Awesome, sounds great! 💪 Let me know if you have questions. |
Okay 😃 |
Btw you may want to wait for #370 to land and base your work off of this function that's been added. It may help you out a lot! |
Thank you for your corrections and tips. I will wait for the landing, otherwise it is possible that _es_result() will be modified at the same time, and there will be conflicts. |
I think I should have succeeded in the optimization.
|
jenkins test this please |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is really great for a first contribution, thanks for working on this! I have some comments below for you, however I also have a higher-level comment of I think our search_after_hits
implementation should be yielding "batches" of documents instead of individual documents.
This makes more sense I think because the "post-processing" phase is a lot more efficient to be called less frequently and means we'll end up creating fewer pd.DataFrame
in the end. We'll still be able to use the current setup we're doing now in itertuples
and iterrows
having yield from pd_df.itertuples()
. I can accomplish this in a separate PR.
Looks like the |
@kxbin Take a look at the change in #379 and adapt your PR to do essentially: def iterrows():
... (setup)
for hits in _search_yield_hits(...):
df = _es_results_to_pandas(hits)
df = self._post_process_...(df)
yield from df.iterrows() Perhaps you can even encapsulate the logic within |
I think this would be useful, to allow raising scroll size from eland default 1000 to elastic maximum 10K. |
Yeah, Really useful. We have now adopted a better solution, we don't use batch_size now, and change to expose Then the user can implement his own batch_size in the dataframe iteration. |
Would you mind providing an example of how this should be called? |
Such like this:
|
Related to this issues:
Can we get batch data use df.to_pandas() in the case of big data? close #345
Add to_pandas_in_batch() DataFrame API
Then, We can use below code to get batch dataframe in the case of big data
If there code is something wrong, please give some suggestions. Thank you!