Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tf_tensors would fail when reading a dataset with non int32 fields and ProcessPool(pyarrow_serialize=True) implementation is used #167

Open
selitvin opened this issue Sep 18, 2018 · 0 comments
Open

tf_tensors would fail when reading a dataset with non int32 fields and ProcessPool(pyarrow_serialize=True) implementation is used #167

selitvin opened this issue Sep 18, 2018 · 0 comments
Labels
bug Something isn't working

Comments

@selitvin
Copy link
Collaborator

ProcessPool uses pyarrow.serialize. However, it would map all integer fields to int32 type. Tensorflow requires exact types to be returned by a pyfunc, hence it will fail.

We did not see this issue since we are not covering the combination of ProcessPool and tf_tensors in test_tf_utils.py.

ProcessPool is going to be replaced when we fully adapt Reader v2, so probably do not have to fix the issue. Opening this issue for awareness.
@selitvin selitvin added the bug Something isn't working label Sep 18, 2018
@selitvin selitvin changed the title tf_tensors would fail when reading a dataset with non int32 fields and ProcessPool implementation is used tf_tensors would fail when reading a dataset with non int32 fields and ProcessPool(pyarrow_serialize=True) implementation is used Oct 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@selitvin