Explicitly set the content type for flyte deck #1658

pingsutw · 2023-05-25T22:19:53Z

TL;DR

Sometimes s3 or gcs will set the content type to application/octet-stream. In this pr, we explicitly set the type, so flyteconsole can render flyte deck properly.

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

from flytekit import task, workflow
from flytekit.image_spec import ImageSpec


new_flytekit = "git+https://github.com/flyteorg/flytekit@e199d01b44042a19a6f18b7942150b02dac24e01"

python_image = ImageSpec(
    registry="pingsutw",
    apt_packages=["git"],
    packages=[new_flytekit],
)


@task(container_image=python_image, disable_deck=False)
def t1():
    print("t1")


@workflow()
def wf():
    t1()


if __name__ == '__main__':
    wf()

Tracking Issue

https://unionai.slack.com/archives/C02CMUNT4PQ/p1684943237121539

Follow-up issue

NA

Signed-off-by: Kevin Su <pingsutw@apache.org>

codecov · 2023-05-25T22:35:16Z

Codecov Report

Merging #1658 (d473838) into master (eafcc82) will decrease coverage by 0.02%.
The diff coverage is 43.75%.

@@            Coverage Diff             @@
##           master    #1658      +/-   ##
==========================================
- Coverage   71.01%   71.00%   -0.02%     
==========================================
  Files         336      336              
  Lines       30756    30781      +25     
  Branches     5572     2953    -2619     
==========================================
+ Hits        21842    21855      +13     
- Misses       8368     8379      +11     
- Partials      546      547       +1

Impacted Files	Coverage Δ
flytekit/core/context_manager.py	`40.57% <25.00%> (-0.17%)`	⬇️
flytekit/core/data_persistence.py	`51.09% <50.00%> (ø)`
flytekit/deck/deck.py	`47.36% <50.00%> (-1.29%)`	⬇️

... and 5 files with indirect coverage changes

eapolinario · 2023-05-31T17:28:06Z

flytekit/bin/entrypoint.py

+    :param raw_output_data_prefix: Where to write primitive outputs
+    :param output_prefix:  Where to write offloaded data (files, directories, dataframes).


these comments are inverted, right?

updated it, thanks

flytekit/deck/deck.py

Signed-off-by: Kevin Su <pingsutw@apache.org>

pingsutw · 2023-06-01T17:24:51Z

Evan and I have tested it on GCP, it works well.

wild-endeavor · 2023-06-01T17:26:35Z

flytekit/bin/entrypoint.py

@@ -183,14 +183,16 @@ def get_one_of(*args) -> str:
 @contextlib.contextmanager
 def setup_execution(
    raw_output_data_prefix: str,
+    output_prefix: Optional[str] = None,


what's this?

Signed-off-by: Kevin Su <pingsutw@apache.org>

pingsutw · 2023-06-01T19:51:19Z

https://dogfood-gcp.cloud-staging.union.ai/console/projects/flytesnacks/domains/development/executions/faddb9b36637e423b99f

* Set content type for flyte deck Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org>

* FLYTECTL_CONFIG env var higher precedence, config flag respected in pyflyte package (#1662) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Change flytekit Pytorch, TFJob and MPI plugins to use new kubeflow config (#1627) * upgrade tensorflow plugin to v1 Signed-off-by: Yubo Wang <yubwang@linkedin.com> * minor fix Signed-off-by: Yubo Wang <yubwang@linkedin.com> * fix tests and lints Signed-off-by: Yubo Wang <yubwang@linkedin.com> * move models file into task make backward compatible Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add code example in README Signed-off-by: Yubo Wang <yubwang@linkedin.com> * bump flyteidl Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add pytorch Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add mpi and fix requirements.txt Signed-off-by: Yubo Wang <yubwang@linkedin.com> * lint and fmt Signed-off-by: Yubo Wang <yubwang@linkedin.com> * Regenerate requirements files using python 3.8 Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Root cert should be byte string when loading from caCertFilePath (#1669) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Explicitly set the content type for flyte deck (#1658) * Set content type for flyte deck Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Use protos of new kubeflow.pytorch plugin instead of legacy pytorch plugin (#1678) Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> * More time info for time line deck (#1680) * more visualization Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * more visualization Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> --------- Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * Add http_proxy to client & Fix deviceflow (#1611) * Add http_proxy to client & Fix deviceflow RB=3890720 Signed-off-by: byhsu <byhsu@linkedin.com> * nit Signed-off-by: byhsu <byhsu@linkedin.com> * lint! Signed-off-by: byhsu <byhsu@linkedin.com> --------- Signed-off-by: byhsu <byhsu@linkedin.com> Co-authored-by: byhsu <byhsu@linkedin.com> * Pass verify flag to all authenticators (#1641) Signed-off-by: byhsu <byhsu@linkedin.com> * feat: Add Auth0/audience support for ClientCredentials flow (#1639) * feat: Add Auth0/audience support for ClientCredentials flow Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor: Remove unneeded variables & condense code Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor: Reduce verbosity of code Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor(chore): Remove unused commented code Signed-off-by: tnam <tommy.nam@aiq-index.com> * fix: Missing comma in input args - authenticator.py 213 Signed-off-by: tnam <tommy.nam@aiq-index.com> * style: Run pre-commit against all files Signed-off-by: tnam <tommy.nam@aiq-index.com> --------- Signed-off-by: tnam <tommy.nam@aiq-index.com> Co-authored-by: tnam <tommy.nam@aiq-index.com> * pyflyte run remote file (#1670) Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * upload deck.html only with deck enable (#1693) Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * Add dask plugin #patch (#1366) * Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add docs page Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask models Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Remove namespace Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update setup.py Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update docs for images Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update README.md Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Do not bring dask changes Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Remove readthedocs Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Linting Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Force scipy<1.11.0 is on windows Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> Signed-off-by: byhsu <byhsu@linkedin.com> Signed-off-by: tnam <tommy.nam@aiq-index.com> Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yubo Wang <yubowang2019@gmail.com> Co-authored-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Fabio M. Graetz, Ph.D <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Yicheng-Lu-llll <51814063+Yicheng-Lu-llll@users.noreply.github.com> Co-authored-by: ByronHsu <byronhsu1230@gmail.com> Co-authored-by: byhsu <byhsu@linkedin.com> Co-authored-by: TomNam <92072956+PudgyPigeon@users.noreply.github.com> Co-authored-by: tnam <tommy.nam@aiq-index.com> Co-authored-by: ChungYujoyce <46030368+ChungYujoyce@users.noreply.github.com> Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com>

pingsutw added 12 commits May 25, 2023 14:27

Set content type for flyte deck

2742f2b

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

0725ebd

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

3ea45bd

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

898c107

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

be24a37

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

f31ccd2

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

025fd8f

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

f3ffc71

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

177d00e

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

3f97fb5

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

e5c97d9

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

e199d01

Signed-off-by: Kevin Su <pingsutw@apache.org>

pingsutw requested review from wild-endeavor, kumare3, eapolinario and cosmicBboy as code owners May 25, 2023 22:19

eapolinario reviewed May 31, 2023

View reviewed changes

pingsutw added 3 commits May 31, 2023 21:46

test

1070d30

Signed-off-by: Kevin Su <pingsutw@apache.org>

test

b9ea617

Signed-off-by: Kevin Su <pingsutw@apache.org>

nit

8a74a9a

Signed-off-by: Kevin Su <pingsutw@apache.org>

wild-endeavor reviewed Jun 1, 2023

View reviewed changes

test

3e15c0a

Signed-off-by: Kevin Su <pingsutw@apache.org>

wild-endeavor previously approved these changes Jun 1, 2023

View reviewed changes

pingsutw marked this pull request as draft June 1, 2023 18:23

nit

7c20f45

Signed-off-by: Kevin Su <pingsutw@apache.org>

pingsutw dismissed wild-endeavor’s stale review via 7c20f45 June 1, 2023 18:24

nit

d473838

Signed-off-by: Kevin Su <pingsutw@apache.org>

pingsutw marked this pull request as ready for review June 1, 2023 18:47

wild-endeavor approved these changes Jun 1, 2023

View reviewed changes

pingsutw merged commit c8433ea into master Jun 1, 2023
122 of 124 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Explicitly set the content type for flyte deck #1658

Explicitly set the content type for flyte deck #1658

pingsutw commented May 25, 2023

codecov bot commented May 25, 2023 •

edited

eapolinario May 31, 2023

pingsutw Jun 1, 2023

pingsutw commented Jun 1, 2023

wild-endeavor Jun 1, 2023

pingsutw commented Jun 1, 2023

		:param raw_output_data_prefix: Where to write primitive outputs
		:param output_prefix: Where to write offloaded data (files, directories, dataframes).

Explicitly set the content type for flyte deck #1658

Explicitly set the content type for flyte deck #1658

Conversation

pingsutw commented May 25, 2023

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

codecov bot commented May 25, 2023 • edited

Codecov Report

eapolinario May 31, 2023

Choose a reason for hiding this comment

pingsutw Jun 1, 2023

Choose a reason for hiding this comment

pingsutw commented Jun 1, 2023

wild-endeavor Jun 1, 2023

Choose a reason for hiding this comment

pingsutw commented Jun 1, 2023

codecov bot commented May 25, 2023 •

edited