Add dask plugin #patch #1366

bstadlbauer · 2022-12-12T06:33:21Z

TL;DR

This adds the Python side of the dask backend plugin (flytekitplugins-dask)

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

This adds the Python side of the dask backed plugin (links to the related PRs can be found in the linked issue below).

TODO before this can be merged:
After flyteorg/flyteidl#339 is merged:

Make sure flytekit version is up to date in plugins/flytekit-dask/setup.py (needs to have buf protobufs)
Make sure flyteidl version is up to date in plugins/flytekit-dask/setup.py (needs to support dask IDL)
Add plugins/flytekit-dask/requirements.in and lock (on Linux)
Add dask[distributed] to doc-requirements.in and re-lock (on Linux)
Check to make sure documentation looks ok

Tracking Issue

flyteorg/flyte#427

Follow-up issue

NA

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

codecov · 2022-12-12T06:41:10Z

Codecov Report

Merging #1366 (8c62437) into master (581a5c6) will increase coverage by 0.01%.
The diff coverage is 81.81%.

@@            Coverage Diff             @@
##           master    #1366      +/-   ##
==========================================
+ Coverage   69.18%   69.20%   +0.01%     
==========================================
  Files         299      300       +1     
  Lines       28023    28084      +61     
  Branches     2632     2640       +8     
==========================================
+ Hits        19389    19435      +46     
- Misses       8118     8136      +18     
+ Partials      516      513       -3

Impacted Files	Coverage Δ
flytekit/core/node.py	`44.00% <37.50%> (-9.07%)`	⬇️
flytekit/core/utils.py	`50.80% <66.66%> (ø)`
flytekit/core/resources.py	`51.35% <76.00%> (+51.35%)`	⬆️
tests/flytekit/unit/core/test_resources.py	`100.00% <100.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

plugins/flytekit-dask/flytekitplugins/dask/task.py

pingsutw

btw, there is a linting error.

bstadlbauer · 2022-12-14T09:01:29Z

@pingsutw Thank you for the quick review!

We're currently working through some design changes in flyteorg/flyteidl#339 - may I ping you once that is done for a re-review?

pingsutw · 2022-12-14T09:38:41Z

Yes, of course. Thank you

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com>

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

plugins/flytekit-dask/flytekitplugins/dask/models.py

plugins/flytekit-dask/flytekitplugins/dask/resources.py

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

wild-endeavor

couple nits if you feel like changing them. if not that's okay too.

wild-endeavor · 2023-01-07T00:08:53Z

plugins/flytekit-dask/README.md

+[step 2](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/k8s_dask/index.html#step-2-environment-setup)
+
+An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/k8s_dask/index.html)
+can be found in the documentation.


Should we add one line about what happens in local execution?

flytekit/core/resources.py

plugins/flytekit-dask/flytekitplugins/dask/models.py

flytekit/core/resources.py

eapolinario

A few nits, nothing to block the release or anything.

plugins/README.md

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

bstadlbauer · 2023-01-12T06:36:45Z

@eapolinario Thanks for fixing and merging!

* Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add docs page Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask models Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Remove namespace Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update setup.py Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update docs for images Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update README.md Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>

* Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add docs page Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask models Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Remove namespace Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update setup.py Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update docs for images Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update README.md Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

This reverts commit 41a9c7a. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Force flyteidl==1.2.9 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Sanitize query template input in sqlite task (#1359) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * TypeTransformer for reading and writing from TensorFlowRecord format (#1240) * first commit Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * add tensorflow example tf record transformer Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * refactor Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * correct tfexample description Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * fix test_native.py Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * add tensorflow docs and reqs Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * add tensorflow docs and reqs1 Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * tensorflow import in init Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * fix failing tests Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * add tensorflow pinned version to reqs Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * pin grpcio-status to remove protobuf error Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * add suggested changes Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * redesign transformer Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * remove old script Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * fix type reference for TFREcordDataset Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * refactor Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * refactor Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * spacing and uppercase Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * redesign with tfdir and tfrecordfile subclass Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * fix conflicts and typos Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * address majority of comments Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * refactor Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * fix test with flytefile and metadata annotated Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * fix check for example records in directory Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * refactor and correct typing Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * lint Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * import annotated from typing_extensions Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * tweak to tests to test case when Config not passed in as type Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * add suggested changes Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * add task for tfrecord dir with no config in test Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * get filenames from local dir instead of remote Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> * update ray plugin dependency (#1361) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Kevin Su <pingsutw@apache.org> * Set default format of structured dataset to empty (#1159) * Set default format of structured dataset to empty Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * last error (#1364) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Adds CLI reference for pyflyte (#1362) * Adds pyflyte CLI reference guide Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * bump python version Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * bump python version Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * resolve docs error Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * set nested to none Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * remove flyteidl version constraint Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * update requirements Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * Signaling (#1133) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Adding created and updated at to ExecutionClosure model (#1371) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Add Databricks config to Spark Job (#1358) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add overwrite_cache option the to calls of remote and local executions (#1375) Signed-off-by: H. Furkan Vural <hfurkanvural@blackshark.ai> Implemented cache overwrite feature is added on flytekit as well for the completeness. In order to support the cache eviction RFC, an overwrite parameter was added, indicating the data store should replace an existing artifact instead of creating a new one on local calls. * Remove project/domain from being overridden with execution values in serialized context (#1378) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Use TaskSpec instead of TaskTemplate for fetch_task and avoid network when loading module (#1348) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Register Databricks config (#1379) * Register databricks plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * Update databricks plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * register databricks Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * PodSpec should not require primary_container name (#1380) For Pod tasks, if the primary_container_name is not specified, it should default. Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fix(pyflyte): change -d to -D for --destination-dir as -d is already for --domain (#1381) Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> * Handle Optional[FlyteFile] in Dataclass type transformer (#1393) * Add support for Optional to dataclass transformer Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add one more test Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add one more test Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix serialization of optional flyte types Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * add FastSerializationSettings to docs (#1386) Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Kevin Su <pingsutw@apache.org> * Added more pod tests and an example pod task (#1382) * Added more pod tests and an example pod task Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixing test and name Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Convert default dict to json string in pyflyte run (#1399) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> * docs: update register help, non-fast version is supported (#1402) Signed-off-by: Patrick Brogan <pbrogan12@gmail.com> * Update log level for structured dataset (#1394) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add Niels to code owners (#1404) Signed-off-by: Kevin Su <pingsutw@apache.org> * Signal use (#1398) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * User Documentation Proposal (#1200) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add support MLFlow plugin (#1274) * MLFlow plugin in progress Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * update test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * update readme Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * dwip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * change experiment name Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * Add mlflow to index.rst Signed-off-by: Kevin Su <pingsutw@apache.org> * use experiment name that user provided Signed-off-by: Kevin Su <pingsutw@apache.org> * update doc-requirements.txt Signed-off-by: Kevin Su <pingsutw@apache.org> * Add backend plugin deployment Signed-off-by: Kevin Su <pingsutw@apache.org> * generate doc for method Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * update docstring Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update docstring Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Update tracking.py Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> * fix remote API reference (#1405) Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Read structured dataset from a folder (#1406) * Read polars dataframe in a folder Signed-off-by: Kevin Su <pingsutw@apache.org> * Read polars dataframe in a folder Signed-off-by: Kevin Su <pingsutw@apache.org> * Load huggingface and spark plugin implicitly Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * remove _pyspark alias Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Update default config to work out-of-the-box with flytectl demo (#1384) Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Add dask plugin #patch (#1366) * Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add docs page Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask models Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Remove namespace Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update setup.py Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update docs for images Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update README.md Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add support for overriding task configurations (#1410) Signed-off-by: Kevin Su <pingsutw@apache.org> * Warning if git is not installed (#1414) * warning if git is not installed Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Kevin Su <pingsutw@apache.org> * Flip the settings for channel and logger (#1415) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Preserving Exception in the LazyEntity fetch (#1412) * Preserving Exception in the LazyEntity fetch Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated lint error Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * more tests Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * [Docs] SynchronousFlyteClient API reference #3095 (#1416) Signed-off-by: Peeter Piegaze <peeter@union.ai> Signed-off-by: Peeter Piegaze <peeter@union.ai> Co-authored-by: Peeter Piegaze <peeter@union.ai> Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com> * Return error code on fail (#1408) * AWS batch return error code once it fails Signed-off-by: Kevin Su <pingsutw@gmail.com> * AWS batch return error code once it fails Signed-off-by: Kevin Su <pingsutw@gmail.com> * update tests Signed-off-by: Kevin Su <pingsutw@gmail.com> * Update tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * wrapping flyte entity in a task node in call to flyte node constructor, not sure if integration tests are actually running (#1422) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Sqlalchemy multiline query (#1421) * SQLAlchemyTask should handle multiline strings for query template Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * sqlalchemy supports multi-line query Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update base sql task Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * remove space Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix snowflake tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix test Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Sklearn type transformer should be automatically loaded with import flytekit (#1423) * add flytekit.extras.sklearn to main __init__ import Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix docs Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add temporary docs/requirements.txt for onnx plugins Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Bump isort to 5.12.0 (#1427) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fixes guess type bug in UnionTransformer (#1426) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> * Add `pod_template` and `pod_template_name` arguments for `PythonAutoContainerTask`, its downstream tasks, and `@task`. (#1425) * Add `pod_template` and `pod_template_name` arguments for `PythonAutoContainerTask`, its downstream tasks, and `@task` Signed-off-by: byhsu <byhsu@linkedin.com> * clean Signed-off-by: byhsu <byhsu@linkedin.com> * fix test Signed-off-by: byhsu <byhsu@linkedin.com> * Fix taskmetadata Signed-off-by: byhsu <byhsu@linkedin.com> * add kubernetes in setup.py Signed-off-by: byhsu <byhsu@linkedin.com> * address comments Signed-off-by: byhsu <byhsu@linkedin.com> * Regenerate requirements using python 3.7 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: byhsu <byhsu@linkedin.com> * keep container validation Signed-off-by: byhsu <byhsu@linkedin.com> * bump idl version Signed-off-by: byhsu <byhsu@linkedin.com> * Regenerate requirements using python 3.7 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate doc-requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * fix Signed-off-by: byhsu <byhsu@linkedin.com> --------- Signed-off-by: byhsu <byhsu@linkedin.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: byhsu <byhsu@linkedin.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Auto Backfill workflow (#1420) * Fix primitive decoder when evaluating Promise (#1432) Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * set maximum python version to 3.10 (#1433) * set maximum python version to 3.10 Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * remove unneeded python version check Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Revert "Remove project/domain from being overridden with execution values in serialized context (#1378)" (#1460) * Revert "Remove project/domain from being overridden with execution values in serialized context (#1378)" This reverts commit b3bfef5. * Import os Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Support checkpointing in local mode from cached tasks (#1457) * support checkpointing in local mode from cached tasks * clear cache before tests --------- Co-authored-by: Stef Nelson-Lindall <stef@stripe.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> * Deprecate FlyteSchema (#1418) * Deprecate FlyteSchema Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove version Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> * Use scarf images (#1434) * Use scarf images Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Use scarf names in tests. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * add undocumented objects/functions to flytekit api ref (#1502) * add reference_launch_plan to flytekit api ref Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * import in init, add docstrings Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add more to references Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix up docstrings Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Samhita Alla <aallasamhita@gmail.com> * Use non-root user in default flytekit image (#1417) Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix PyTorch transformer (#1510) Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * Fix mypy errors (#1313) * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix mypy errors Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix mypy errors Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * fix test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * Update type Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * update dev-requirements.txt Signed-off-by: Kevin Su <pingsutw@apache.org> * Address comment Signed-off-by: Kevin Su <pingsutw@apache.org> * upgrade torch Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Compile the workflow only at compile time (#1311) * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * add tests Signed-off-by: Kevin Su <pingsutw@apache.org> * add tests Signed-off-by: Kevin Su <pingsutw@apache.org> * support dynamic task Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * lazy compile Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * add tests Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * update test Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Get the origin type when serializing dataclass (#1508) * Get the origin type when serializing dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * update test Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> * Fix bad merge Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Delay initialization of SynchronousFlyteClient in FlyteRemote (#1514) * Delay initialization of SynchronousFlyteClient in FlyteRemote Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix spark plugin flyteremote test. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set flytekit and flyteidl bounds in plugins tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Fix mypy errors (#1313)" This reverts commit 3798450. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix requirements in dask and ray plugins Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix papermill tests requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix doc-requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * dask plugin requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Add dask plugin #patch (#1366)" This reverts commit 41a9c7a. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Ryan Nazareth <ryankarlos@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Patrick Brogan <pbrogan12@gmail.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Peeter Piegaze <peeter@union.ai> Signed-off-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: byhsu <byhsu@linkedin.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Ryan Nazareth <ryankarlos@gmail.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Samhita Alla <aallasamhita@gmail.com> Co-authored-by: H. Furkan Vural <33652917+hfurkanvural@users.noreply.github.com> Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com> Co-authored-by: mcloney-ddm <119345186+mcloney-ddm@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: pbrogan12 <pbrogan12@gmail.com> Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com> Co-authored-by: Peeter Piegaze <peeter@piegaze.com> Co-authored-by: Peeter Piegaze <peeter@union.ai> Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com> Co-authored-by: ByronHsu <byronhsu1230@gmail.com> Co-authored-by: byhsu <byhsu@linkedin.com> Co-authored-by: Stef Lindall <bethebunny@gmail.com> Co-authored-by: Stef Nelson-Lindall <stef@stripe.com>

* Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add docs page Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask models Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Remove namespace Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update setup.py Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update docs for images Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update README.md Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* FLYTECTL_CONFIG env var higher precedence, config flag respected in pyflyte package (#1662) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Change flytekit Pytorch, TFJob and MPI plugins to use new kubeflow config (#1627) * upgrade tensorflow plugin to v1 Signed-off-by: Yubo Wang <yubwang@linkedin.com> * minor fix Signed-off-by: Yubo Wang <yubwang@linkedin.com> * fix tests and lints Signed-off-by: Yubo Wang <yubwang@linkedin.com> * move models file into task make backward compatible Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add code example in README Signed-off-by: Yubo Wang <yubwang@linkedin.com> * bump flyteidl Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add pytorch Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add mpi and fix requirements.txt Signed-off-by: Yubo Wang <yubwang@linkedin.com> * lint and fmt Signed-off-by: Yubo Wang <yubwang@linkedin.com> * Regenerate requirements files using python 3.8 Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Root cert should be byte string when loading from caCertFilePath (#1669) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Explicitly set the content type for flyte deck (#1658) * Set content type for flyte deck Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Use protos of new kubeflow.pytorch plugin instead of legacy pytorch plugin (#1678) Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> * More time info for time line deck (#1680) * more visualization Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * more visualization Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> --------- Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * Add http_proxy to client & Fix deviceflow (#1611) * Add http_proxy to client & Fix deviceflow RB=3890720 Signed-off-by: byhsu <byhsu@linkedin.com> * nit Signed-off-by: byhsu <byhsu@linkedin.com> * lint! Signed-off-by: byhsu <byhsu@linkedin.com> --------- Signed-off-by: byhsu <byhsu@linkedin.com> Co-authored-by: byhsu <byhsu@linkedin.com> * Pass verify flag to all authenticators (#1641) Signed-off-by: byhsu <byhsu@linkedin.com> * feat: Add Auth0/audience support for ClientCredentials flow (#1639) * feat: Add Auth0/audience support for ClientCredentials flow Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor: Remove unneeded variables & condense code Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor: Reduce verbosity of code Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor(chore): Remove unused commented code Signed-off-by: tnam <tommy.nam@aiq-index.com> * fix: Missing comma in input args - authenticator.py 213 Signed-off-by: tnam <tommy.nam@aiq-index.com> * style: Run pre-commit against all files Signed-off-by: tnam <tommy.nam@aiq-index.com> --------- Signed-off-by: tnam <tommy.nam@aiq-index.com> Co-authored-by: tnam <tommy.nam@aiq-index.com> * pyflyte run remote file (#1670) Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * upload deck.html only with deck enable (#1693) Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * Add dask plugin #patch (#1366) * Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add docs page Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask models Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Remove namespace Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update setup.py Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update docs for images Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update README.md Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Do not bring dask changes Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Remove readthedocs Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Linting Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Force scipy<1.11.0 is on windows Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> Signed-off-by: byhsu <byhsu@linkedin.com> Signed-off-by: tnam <tommy.nam@aiq-index.com> Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yubo Wang <yubowang2019@gmail.com> Co-authored-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Fabio M. Graetz, Ph.D <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Yicheng-Lu-llll <51814063+Yicheng-Lu-llll@users.noreply.github.com> Co-authored-by: ByronHsu <byronhsu1230@gmail.com> Co-authored-by: byhsu <byhsu@linkedin.com> Co-authored-by: TomNam <92072956+PudgyPigeon@users.noreply.github.com> Co-authored-by: tnam <tommy.nam@aiq-index.com> Co-authored-by: ChungYujoyce <46030368+ChungYujoyce@users.noreply.github.com> Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com>

bstadlbauer requested review from wild-endeavor, kumare3, eapolinario and pingsutw as code owners December 12, 2022 06:33

bstadlbauer added 9 commits December 12, 2022 07:34

Add dummy task type to test backend plugin

eddb152

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Add docs page

407d5cb

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Add dask models

992d5e7

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Add function to convert resources

d7bdf47

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Add tests to dask task

641d2ac

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Remove namespace

96162bf

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Update setup.py

6092b6f

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Add dask to plugin/README.md

1caceff

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Add README.md for dask

afa54df

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

bstadlbauer force-pushed the add-dask-plugin branch from ca596e4 to afa54df Compare December 12, 2022 06:34

bstadlbauer added 3 commits December 12, 2022 10:21

Top level export of JopPodSpec and DaskCluster

49bdab3

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Update docs for images

96270f3

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Update README.md

dfc4131

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

This was referenced Dec 12, 2022

Dask plugin docs flyteorg/flytesnacks#929

Merged

[Backend][Plugin]Support for Dask clustered tasks in Flyte flyteorg/flyte#427

Closed

pingsutw approved these changes Dec 14, 2022

View reviewed changes

plugins/flytekit-dask/flytekitplugins/dask/task.py Show resolved Hide resolved

pingsutw requested changes Dec 14, 2022

View reviewed changes

bstadlbauer added 4 commits December 17, 2022 19:02

Update models after flyteidl change

de2af6d

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Update task after flyteidl change

8633ecc

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Merge remote-tracking branch 'origin/master' into add-dask-plugin

528a085

Raise error when less than 1 worker

019529d

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

bstadlbauer changed the title ~~Add dask plugin #minor~~ Add dask plugin #patch Jan 5, 2023

Update flyteidl to >= 1.3.2

b04fb3d

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

bstadlbauer and others added 5 commits January 5, 2023 09:45

Update doc requirements

4d5e445

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Merge branch 'master' into add-dask-plugin

a5311ae

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Update doc-requirements.txt

f703c09

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Re-lock dependencies on linux

0465291

Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com>

Update dask API docs

f02d9dc

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

pingsutw reviewed Jan 7, 2023

View reviewed changes

plugins/flytekit-dask/flytekitplugins/dask/models.py Outdated Show resolved Hide resolved

plugins/flytekit-dask/flytekitplugins/dask/models.py Outdated Show resolved Hide resolved

plugins/flytekit-dask/flytekitplugins/dask/resources.py Outdated Show resolved Hide resolved

bstadlbauer added 5 commits January 9, 2023 21:24

Merge branch 'master' into add-dask-plugin

b50be3e

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Fix documentation links

167d654

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Default optional model constructor arguments to None

dab1bc6

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Refactor convert_resources_to_resource_model to core.resources

3722b0a

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Use convert_resources_to_resource_model in core.node

4f829b7

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

bstadlbauer requested a review from cosmicBboy as a code owner January 9, 2023 21:28

bstadlbauer requested a review from pingsutw January 9, 2023 21:31

bstadlbauer and others added 2 commits January 10, 2023 07:51

Merge branch 'master' into add-dask-plugin

43147cb

Merge branch 'master' into add-dask-plugin

1e6e3e1

wild-endeavor previously approved these changes Jan 11, 2023

View reviewed changes

eapolinario reviewed Jan 11, 2023

View reviewed changes

plugins/README.md Outdated Show resolved Hide resolved

Incorporate review feedback

aadf78c

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

eapolinario dismissed wild-endeavor’s stale review via aadf78c January 12, 2023 00:41

Lint

8c62437

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

wild-endeavor approved these changes Jan 12, 2023

View reviewed changes

pingsutw approved these changes Jan 12, 2023

View reviewed changes

eapolinario merged commit 531db3c into flyteorg:master Jan 12, 2023

eapolinario added a commit that referenced this pull request Feb 22, 2023

Revert "Add dask plugin #patch (#1366)"

abfcbb4

This reverts commit 41a9c7a. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dask plugin #patch #1366

Add dask plugin #patch #1366

bstadlbauer commented Dec 12, 2022 •

edited

codecov bot commented Dec 12, 2022 •

edited

pingsutw left a comment

bstadlbauer commented Dec 14, 2022

pingsutw commented Dec 14, 2022

wild-endeavor left a comment

wild-endeavor Jan 7, 2023

eapolinario left a comment

bstadlbauer commented Jan 12, 2023 •

edited

Add dask plugin #patch #1366

Add dask plugin #patch #1366

Conversation

bstadlbauer commented Dec 12, 2022 • edited

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

codecov bot commented Dec 12, 2022 • edited

Codecov Report

pingsutw left a comment

Choose a reason for hiding this comment

bstadlbauer commented Dec 14, 2022

pingsutw commented Dec 14, 2022

wild-endeavor left a comment

Choose a reason for hiding this comment

wild-endeavor Jan 7, 2023

Choose a reason for hiding this comment

eapolinario left a comment

Choose a reason for hiding this comment

bstadlbauer commented Jan 12, 2023 • edited

bstadlbauer commented Dec 12, 2022 •

edited

codecov bot commented Dec 12, 2022 •

edited

bstadlbauer commented Jan 12, 2023 •

edited