Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove PandasToCSVCollector, PandasToDataFrameCollector #378

Merged
merged 1 commit into from Aug 16, 2021

Conversation

V1NAY8
Copy link
Contributor

@V1NAY8 V1NAY8 commented Aug 11, 2021

This PR is a small change that removes PandasToCSVCollector, PandasToDataFrameCollector

This has a basic functionality, But at the moment I think its not required.

If any additional functionality comes, I think we can achieve it without a class!

@sethmlarson Take a look at it :)

@elasticmachine
Copy link

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@V1NAY8 V1NAY8 changed the title Remove PandasToCSVCollector, PandasToCSVCollector Remove PandasToCSVCollector, PandasToDataFrameCollector Aug 11, 2021
@sethmlarson
Copy link
Contributor

jenkins test this please

Copy link
Contributor

@sethmlarson sethmlarson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good change, I've got a few comments for you.

Generally this seems like an in-between change towards making _es_results a true iterator but that can still be handled in a separate change too.

eland/etl.py Show resolved Hide resolved

return collector._ret
df = self._es_results(query_compiler, show_progress)
df.to_csv(**kwargs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return df.to_csv()? We should add a type and test for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, we actually won't return this, the path of the file is passed in kwargs, where the csv has to be ingested to.

We already have good tests in tests\dataframe\test_to_csv_pytest.py
What do you think ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From reading pandas.DataFrame.to_csv() it looks like if path_or_buf is None then the CSV is returned as a string. We should probably do and test that as well.

@sethmlarson
Copy link
Contributor

jenkins test this please

Copy link
Contributor

@sethmlarson sethmlarson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for this! Simplifies this logic immensely.

@sethmlarson sethmlarson merged commit 011bf29 into elastic:master Aug 16, 2021
@V1NAY8 V1NAY8 deleted the remove-collectors branch August 16, 2021 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants