ARROW-17641: [python] Fix ParseOptions deserialization of invalid_row_handler #14061
Conversation
…_handler Signed-off-by: Kai Fricke <kai@anyscale.com>
|
|
python/pyarrow/tests/test_csv.py
Outdated
| parse_opts.invalid_row_handler = InvalidRowHandler('skip') | ||
| state = parse_opts.__getstate__() | ||
|
|
||
| parse_opts = ParseOptions() | ||
| parse_opts.__setstate__(state) |
There was a problem hiding this comment.
Instead of calling getstate/setstate explicitly, can you pickle/unpickle instead? (like new_parse_opts = pickle.loads(pickle.dumps(parse_opts)))
There was a problem hiding this comment.
Sure, updated!
I've also updated __getstate__ to return self.invalid_row_handler instead of self._invalid_row_handler, let me know if this looks good to you.
Signed-off-by: Kai Fricke <kai@anyscale.com>
|
Thanks for fixing this, this is great! @krfricke has a little more context on the bug here: ray-project/ray#28326 (comment) |
|
Benchmark runs are scheduled for baseline = 47314c3 and contender = 6ff5224. 6ff5224 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
…_handler (apache#14061) Authored-by: Kai Fricke <kai@anyscale.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
…_handler (apache#14061) Authored-by: Kai Fricke <kai@anyscale.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
No description provided.