Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MATLAB] Decouple MATLAB-Arrow conversion logic from Feather file specific logic #20506

Closed
asfimport opened this issue Nov 28, 2018 · 4 comments

Comments

@asfimport
Copy link

Currently, the logic for converting between a MATLAB mxArray and various Arrow data structures (arrow::Table, arrow::Array, etc.) is tightly coupled and fairly tangled up with the logic specific to handling Feather files. It would be helpful to factor out these conversions into a more generic "mlarrow" conversion layer component so that it can be reused in the future for use cases other than Feather support. Furthermore, this would be helpful to enforce a cleaner separation of concerns.

It would be nice to start off with this refactoring work up front before adding support for more datatypes to the MATLAB featherread/featherwrite functions, so that we can start off with a clean base upon which to expand moving forward.

Reporter: Kevin Gurney / @kevingurney

Note: This issue was originally created as ARROW-3896. Please see the migration documentation for further details.

@asfimport
Copy link
Author

Wes McKinney / @wesm:
Sounds like a good idea.

@asfimport
Copy link
Author

Kevin Gurney / @kevingurney:
Assigning to myself. I'll look into factoring out the MATLAB-Arrow conversions after finishing off the Feather numeric write support changes that are currently in flight here:

mathworks@871d813

@asfimport
Copy link
Author

Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

@kevingurney kevingurney self-assigned this Aug 16, 2023
@kevingurney
Copy link
Member

I think we can mark this as resolved now that featherread and featherwrite have been re-implemented in terms of the new internal arrow.internal.io.feather.Reader (#37163) and arrow.internal.io.feather.Writer (#37047) APIs.

The logic for converting MATLAB types to Arrow types has mostly been separated out into the arrow.array.Array subclasses at this point.

Technically, we are still casting logical and integer types to double in order to substitute null values with NaN in featherread - but, this is just to preserve compatibility with the old code, and we may change this behavior in the future.

Closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants