Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dask plugin docs #929

Merged
merged 15 commits into from Jan 23, 2023
Merged

Dask plugin docs #929

merged 15 commits into from Jan 23, 2023

Conversation

bstadlbauer
Copy link
Member

@bstadlbauer bstadlbauer commented Dec 12, 2022 •
edited

Adds docs for the dask integration developed in flyteorg/flyte#427

Todo before merging (after flyteorg/flyteplugins#275 and flyteorg/flytekit#1366 are in):

  • Revisit all links to make sure they work
  • Add and lock cookbook/integrations/kubernetes/k8s_dask/requirements.in (on Linux)

@bstadlbauer
Add documentation for dask integration
5dfcc10 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Fix typos in pyspark_pi.py
be00af6 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Fix typo in dask_example.py
2440753 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Fix typos
9552b36 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer bstadlbauer mentioned this pull request Dec 12, 2022
Closed
6 tasks
@bstadlbauer
Update docs after flyteidl change
46b8710 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Add documentation on interruptible behavior
aa38c47 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Merge branch 'master' into dask-plugin-docs
9916652 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Address PR feedback
52a9652 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Merge remote-tracking branch 'origin/master' into dask-plugin-docs
837286d 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Small fixes
1c37e54 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Add requirement files for dask
6206a81 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Add git to Dockerfile
edb8ccd 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Copy link
Member Author

bstadlbauer commented Jan 15, 2023 •
edited

@wild-endeavor Things would be ready from my end 👍

@kumare3
Copy link
Contributor

kumare3 commented Jan 16, 2023

I think we should add a note about local execution- that you can execute the dask task locally, in a non clustered mode?

@bstadlbauer
Add note about local execution
87fbe73 
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
@bstadlbauer
Copy link
Member Author

@kumare3 Good point, I've added a section in 87fbe73

Local execution

When running the dask task locally, it will use a local distributed Client. In case you would like to connect the to a remote cluster for when developing locally, you can set the DASK_SCHEDULER_ADDRESS environment variable to the URL of the remote scheduler and the Client() will use the cluster automatically.

@cosmicBboy
minor typos
5d16eab 
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
cosmicBboy
cosmicBboy approved these changes Jan 23, 2023
View reviewed changes
Copy link
Contributor

@cosmicBboy cosmicBboy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @bstadlbauer ! 🚀

@bstadlbauer bstadlbauer merged commit 2aefe89 into flyteorg:master Jan 23, 2023
ppiegaze
ppiegaze reviewed Jan 24, 2023
View reviewed changes
cookbook/docs/integrations.rst
@@ -120,6 +120,15 @@ orchestrated by Flyte itself, within its provisioned Kubernetes clusters.

---

.. link-button:: auto/integrations/kubernetes/k8s_dask/index
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@samhita-alla I think this change addresses issue flyteorg/flyte#3259

cosmicBboy added a commit that referenced this pull request Jan 30, 2023
@cosmicBboy
Revert "Dask plugin docs (#929)"
139ddea 
This reverts commit 2aefe89.
@cosmicBboy cosmicBboy mentioned this pull request Jan 30, 2023
Closed
2 tasks
cosmicBboy added a commit that referenced this pull request Feb 8, 2023
@bstadlbauer @cosmicBboy
Dask plugin docs (#929)
aeb0b92 
* Add documentation for `dask` integration

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Fix typos in `pyspark_pi.py`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Fix typo in `dask_example.py`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Fix typos

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update docs after `flyteidl` change

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add documentation on interruptible behavior

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Address PR feedback

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Small fixes

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add requirement files for dask

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add git to Dockerfile

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add note about local execution

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* minor typos

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com>
cosmicBboy added a commit that referenced this pull request Feb 8, 2023
@pingsutw @bstadlbauer @cosmicBboy
Add databricks to index.rst (#939)
8c96962 
* Add databricks to index.rst

Signed-off-by: Kevin Su <pingsutw@gmail.com>

* Dask plugin docs (#929)

* Add documentation for `dask` integration

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Fix typos in `pyspark_pi.py`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Fix typo in `dask_example.py`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Fix typos

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update docs after `flyteidl` change

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add documentation on interruptible behavior

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Address PR feedback

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Small fixes

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add requirement files for dask

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add git to Dockerfile

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add note about local execution

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* minor typos

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com>

* fix up index

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

---------

Signed-off-by: Kevin Su <pingsutw@gmail.com>
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com>
Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ppiegaze ppiegaze ppiegaze left review comments

@cosmicBboy cosmicBboy cosmicBboy approved these changes

@kumare3 kumare3 Awaiting requested review from kumare3

@wild-endeavor wild-endeavor Awaiting requested review from wild-endeavor

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@bstadlbauer @kumare3 @ppiegaze @cosmicBboy @wild-endeavor