Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windowing Support for the Dask Runner #32941

Merged
merged 196 commits into from
Nov 18, 2024
Merged

Windowing Support for the Dask Runner #32941

merged 196 commits into from
Nov 18, 2024

Conversation

alxmrs
Copy link
Contributor

@alxmrs alxmrs commented Oct 24, 2024

This CL adds basic Windowing support to this runner, including a few tests for side inputs – take three (#27618, #23913).

CC: @cisaacstern
Reviewers: @jrmccluskey

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

alxmrs and others added 30 commits September 19, 2022 16:34
@alxmrs
WIP: Created a skeleton dask runner implementation.
bc6b525
@alxmrs
WIP: Idea for a translation evaluator.
665ee61
@alxmrs
Added overrides and a visitor that translates operations.
d1095c7
@alxmrs
Fixed a dataclass typo.
fd13d40
@alxmrs
Expanded translations.
e3d0c8a
@alxmrs
Core idea seems to be kinda working...
69a660f
@alxmrs
First iteration on DaskRunnerResult (keep track of pipeline state).
ab58334
@alxmrs
Added minimal set of DaskRunner options.
5391cd6
@alxmrs
WIP: Alllmost got asserts to work! The current status is:
5deb598 
- CoGroupByKey is broken due to how tags are used with GroupByKey
- GroupByKey should output `[('0', None), ('1', 1)]`, however it actually outputs: [(None, ('1', 1)), (None, ('0', None))]
- Once that is fixed, we may have test pipelines work on Dask.
@alxmrs
With a great 1-liner from @pabloem, groupby is fixed! Now, all three …
1768e47 
…initial tests pass.
@alxmrs
Self-review: Cleaned up dask runner impl.
e6c7106
@alxmrs
Self-review: Remove TODOs, delete commented out code, other cleanup.
1b58d4c
@alxmrs
First pass at linting rules.
5fe9372
@alxmrs
WIP, include dask dependencies + test setup.
b98330e
@alxmrs
WIP: maybe better dask deps?
e48780a
@alxmrs
Skip dask tests depending on successful import.
0dc9e23
@alxmrs
Fixed setup.py (missing ,).
326d3a3
@alxmrs
Added an additional comma.
b4cc408
@alxmrs
Moved skipping logic to be above dask import.
40a6ebe
@alxmrs
Fix lint issues with dask runner tests.
3c4204d
@alxmrs
Adding destination for client address.
41623ec
@alxmrs
Changing to async produces a timeout error instead of stuck in infini…
676d752 
…te loop.
@alxmrs
Close client during wait_until_finish; rm async.
09365f6
@pabloem
Revert "Close client during wait_until_finish; rm async."
c6ba4ba 
This reverts commit 09365f6.
@pabloem
Revert "Changing to async produces a timeout error instead of stuck i…
a325356 
…n infinite loop."

This reverts commit 676d752.
@pabloem
Adding -dask tox targets onto the gradle build
ea13125
@alxmrs
Supporting side-inputs for ParDo.
f855ffc
@alxmrs
Merge remote-tracking branch 'origin/dask-runner-mvp' into dask-runne…
3fd966e 
…r-mvp
@alxmrs
wip - added print stmt.
173d79b
@alxmrs
wip - prove side inputs is set.
dd2d15c
@alxmrs
Copy link
Contributor Author

alxmrs commented Nov 7, 2024

@Abacn May I have some help getting the windows build to work? It looks tractable:

C:\hostedtoolcache\windows\Python\3.10.11\x64\python.exe: can't open file 
'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py310-win-dask\\Scripts\\pip': [Errno 2] No such file or directory

@alxmrs
Copy link
Contributor Author

alxmrs commented Nov 7, 2024

Ok, I think I got it working on windows, too.

@alxmrs alxmrs requested a review from Abacn November 7, 2024 22:43
@alxmrs alxmrs force-pushed the dask-runner-windowing branch from 1b41784 to 7caba84 Compare November 8, 2024 00:09
@alxmrs
Copy link
Contributor Author

alxmrs commented Nov 8, 2024

I think the failing tests are just flaky.

@shunping
Copy link
Contributor

@Abacn, could you take another look on @alxmrs 's changes?

@alxmrs
Copy link
Contributor Author

alxmrs commented Nov 14, 2024

Hey there! I'd love to get this merged. Please let me know if there is any more I can do.

@Abacn
Copy link
Contributor

Abacn commented Nov 14, 2024

Sorry for late response. My comments were most on the infrastructure / testing related components. For that part LGTM. For dask runner implementation would it be possible to find a reviewer who knows Dask also?

@alxmrs
Copy link
Contributor Author

alxmrs commented Nov 15, 2024

That sounds good. Hey, @jacobtomlinson: by any chance, do you have cycles to review this PR, focusing on the Dask aspect of the code? This one has been a long time coming! I would appreciate any and all help you can offer (and no pressure if time is sparse).

jacobtomlinson
jacobtomlinson approved these changes Nov 15, 2024
View reviewed changes
Copy link
Contributor

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally this looks good to me from the Dask side.

sdks/python/apache_beam/runners/dask/dask_runner.py Show resolved Hide resolved
@alxmrs
Copy link
Contributor Author

alxmrs commented Nov 15, 2024

Hey @Abacn! In light of Jacob's review, do you think we're good-to-go? :)

@Abacn
Copy link
Contributor

Abacn commented Nov 18, 2024

Thank you, merging for now

@Abacn Abacn merged commit e939be3 into apache:master Nov 18, 2024
106 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jacobtomlinson jacobtomlinson jacobtomlinson approved these changes

@Abacn Abacn Awaiting requested review from Abacn

Assignees
No one assigned
Labels
build Next Action: Reviewers python runners
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@alxmrs @shunping @Abacn @jacobtomlinson @pabloem @cisaacstern