Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-15776: [Python] Expose IpcReadOptions #12800

Closed
wants to merge 13 commits into from
Closed

ARROW-15776: [Python] Expose IpcReadOptions #12800

wants to merge 13 commits into from

Conversation

raulcd
Copy link
Member

@raulcd raulcd commented Apr 5, 2022

This PR intends to expose IpcReadOptions to pyarrow.

@github-actions
Copy link

github-actions bot commented Apr 5, 2022

https://issues.apache.org/jira/browse/ARROW-15776

@github-actions
Copy link

github-actions bot commented Apr 5, 2022

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@raulcd raulcd marked this pull request as ready for review April 6, 2022 14:30
@raulcd
Copy link
Member Author

raulcd commented Apr 6, 2022

@jorisvandenbossche As discussed I think this can start review process. I was going to assign you as reviewer but I don't have permission.

jorisvandenbossche
jorisvandenbossche reviewed Apr 7, 2022
View reviewed changes
Copy link
Member

@jorisvandenbossche jorisvandenbossche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very nice PR, thanks!

python/pyarrow/ipc.pxi Outdated Show resolved Hide resolved
python/pyarrow/ipc.pxi Outdated Show resolved Hide resolved
lidavidm
lidavidm reviewed Apr 7, 2022
View reviewed changes
Copy link
Member

@lidavidm lidavidm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. (We should file a follow-up to expose this in Flight.)

python/pyarrow/ipc.pxi Outdated Show resolved Hide resolved
python/pyarrow/ipc.pxi Outdated Show resolved Hide resolved
@raulcd
ARROW-15776: [Python] If IpcReadOptions included_fieldsis None use de…
d813029 
…fault C++ value for included_fields instead of empty list
@raulcd
ARROW-15776: [Python] Remove MemoryPool from IpcReadOptions and expos…
0193c4d 
…e as part of open_stream and open_file directly
lidavidm
lidavidm approved these changes Apr 8, 2022
View reviewed changes
python/pyarrow/ipc.pxi Outdated Show resolved Hide resolved
@lidavidm
Copy link
Member

lidavidm commented Apr 8, 2022

Actually, we should also add it to the docs here:

arrow/docs/source/python/api/ipc.rst

Line 41 in b829943

ipc.IpcWriteOptions

@raulcd
ARROW-15776: [Python] Add IpcReadOptions to ipc.rst docs and use incl…
7473cd8 
…uded_fields.clear as requested on code review
@raulcd
Copy link
Member Author

raulcd commented Apr 8, 2022

Actually, we should also add it to the docs here:

arrow/docs/source/python/api/ipc.rst

Line 41 in b829943

ipc.IpcWriteOptions

Done and validated documentation generated:
image

pitrou
pitrou reviewed Apr 12, 2022
View reviewed changes
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this @raulcd! Here are some comments.

python/pyarrow/ipc.pxi Outdated Show resolved Hide resolved
python/pyarrow/ipc.py Outdated Show resolved Hide resolved
python/pyarrow/ipc.py Show resolved Hide resolved
python/pyarrow/tests/test_ipc.py Outdated Show resolved Hide resolved
raulcd and others added 2 commits April 13, 2022 10:54
@raulcd @pitrou
Remove unnecessary empty line
c2f470f 
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
@raulcd
ARROW-15776: [Python] Review comments. Do not allow None for included…
b27269f 
…_fields and use expected values for test
@raulcd raulcd requested a review from pitrou April 19, 2022 08:12
pitrou
pitrou reviewed Apr 19, 2022
View reviewed changes
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Two nits, otherwise LGTM.

python/pyarrow/ipc.py Outdated Show resolved Hide resolved
python/pyarrow/ipc.py Outdated Show resolved Hide resolved
raulcd and others added 2 commits April 19, 2022 12:22
@raulcd @pitrou
Update python/pyarrow/ipc.py
df87898 
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
@raulcd
ARROW-15776: [Python] Update comments to be consistent and all method…
38bdde3 
…s as requested on review
pitrou
pitrou approved these changes Apr 19, 2022
View reviewed changes
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, thank you @raulcd

@pitrou pitrou closed this in 2a2c087 Apr 19, 2022
@ursabot
Copy link

ursabot commented Apr 20, 2022

Benchmark runs are scheduled for baseline = 6c10a38 and contender = 2a2c087. 2a2c087 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.29% ⬆️0.0%] test-mac-arm
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.26% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/539| 2a2c0873 ec2-t3-xlarge-us-east-2>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/526| 2a2c0873 test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/525| 2a2c0873 ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/536| 2a2c0873 ursa-thinkcentre-m75q>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/538| 6c10a389 ec2-t3-xlarge-us-east-2>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/525| 6c10a389 test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/524| 6c10a389 ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/535| 6c10a389 ursa-thinkcentre-m75q>
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@ursabot
Copy link

ursabot commented Apr 26, 2022

['Python', 'R'] benchmarks have high level of regressions.
test-mac-arm

@asfimport asfimport mentioned this pull request Apr 26, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lidavidm lidavidm lidavidm approved these changes

@jorisvandenbossche jorisvandenbossche jorisvandenbossche left review comments

@pitrou pitrou pitrou approved these changes

Assignees
No one assigned
Labels
Component: Python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@raulcd @lidavidm @ursabot @jorisvandenbossche @pitrou