-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add CSV export capability to DataFrameClient #45
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple of comments on this:
- It's returning the data as binary data. The binary can be written to a CSV file or to pandas directly, but if the user wants to get the data as a string in code, they will need to decode it themselves. We did this in anticipation of adding binary export formats in the future (TDMS).
- I didn't implement
readline
orreadlines
, since they are not required for writing to a file or to pandas. I can add them if we want this to be a fully-functional file-like object, but they were getting a bit messy with iterating the binary and normalizing line endings to what python expects.
I'm working through some type errors. |
pyproject.toml
Outdated
@@ -29,7 +29,7 @@ classifiers = [ | |||
] | |||
|
|||
[tool.poetry.dependencies] | |||
python = "^3.8" | |||
python = ">=3.8,<3.12" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was required to satisfy the mypy type checker for having pandas in the export-data example. mypy needs pandas-stubs to do type checking for pandas, and pandas-stubs requires python to be between these two versions.
If we're not happy with this, I can remove pandas-stubs and revert this line, because it just gives a warning from mypy that it's skipping type checking for pandas without it.
From https://dev.azure.com/ni/DevCentral/_workitems/edit/2347097, If the server terminates the connection for any reason, the client will get an unhelpful |
We have a couple of ideas for how to solve this, a couple of which would be changes on the DFS side. I added some details to the bug, and we can decide on the approach when we get to it. |
What does this Pull Request accomplish?
Add
export_table_data
to the DataFrameClient, which calls the export route on the DataFrame Service API.Wrap the
response.iter_content
generator in a file-like object that can either be written to a file or loaded into a pandas dataframe, without loading the data into memory all at once beforehand.Added an example of writing the export data to a file or pandas.
Why should this Pull Request be merged?
Parity of the python API with new features added to the DataFrame Service API.
What testing has been done?
Manual testing using Jupyter notebooks. Added unit tests for the generator wrapper class. Added an integration test for the export data method.