Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More time info for time line deck #1680

Merged
merged 2 commits into from Jun 8, 2023

Conversation

Yicheng-Lu-llll
Copy link
Member

TL;DR

This PR:

  • allows users to view the remote URL for uploading and downloading in the time line deck
  • The process of "generating the timeline HTML file" has been moved to a later stage, allowing for the inclusion of more time-specific information. In the previous implementation, the user could not see the remote URL storing the 'output.pb' and 'deck.pb' files because the HTML file was generated too early. This adjustment ensures these important details are visible in the timeline.
    ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True)

Checks

Tested in the sandbox. We are now able to see the remote url in time line deck.
image

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
@codecov
Copy link

codecov bot commented Jun 7, 2023

Codecov Report

Merging #1680 (a0d1f95) into master (c8433ea) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1680      +/-   ##
==========================================
+ Coverage   71.00%   71.02%   +0.02%     
==========================================
  Files         336      336              
  Lines       30781    30779       -2     
  Branches     5576     5576              
==========================================
+ Hits        21855    21860       +5     
+ Misses       8379     8373       -6     
+ Partials      547      546       -1     
Impacted Files Coverage Δ
flytekit/core/base_task.py 48.32% <ø> (-0.39%) ⬇️
flytekit/core/data_persistence.py 52.19% <100.00%> (+1.09%) ⬆️

... and 2 files with indirect coverage changes

@pingsutw pingsutw merged commit 06d95b8 into flyteorg:master Jun 8, 2023
124 checks passed
eapolinario pushed a commit that referenced this pull request Jun 29, 2023
* more visualization

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* more visualization

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

---------

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
eapolinario added a commit that referenced this pull request Jun 30, 2023
* FLYTECTL_CONFIG env var higher precedence, config flag respected in pyflyte package (#1662)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Change flytekit Pytorch, TFJob and MPI plugins to use new kubeflow config (#1627)

* upgrade tensorflow plugin to v1

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* minor fix

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* fix tests and lints

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* move models file into task make backward compatible

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* add code example in README

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* bump flyteidl

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* add pytorch

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* add mpi and fix requirements.txt

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* lint and fmt

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* Regenerate requirements files using python 3.8

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: Yubo Wang <yubwang@linkedin.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Yubo Wang <yubwang@linkedin.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Root cert should be byte string when loading from caCertFilePath (#1669)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Explicitly set the content type for flyte deck (#1658)

* Set content type for flyte deck

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Use protos of new kubeflow.pytorch plugin instead of legacy pytorch plugin (#1678)

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>
Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com>

* More time info for time line deck (#1680)

* more visualization

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* more visualization

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

---------

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* Add http_proxy to client & Fix deviceflow (#1611)

* Add http_proxy to client & Fix deviceflow

RB=3890720

Signed-off-by: byhsu <byhsu@linkedin.com>

* nit

Signed-off-by: byhsu <byhsu@linkedin.com>

* lint!

Signed-off-by: byhsu <byhsu@linkedin.com>

---------

Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>

* Pass verify flag to all authenticators (#1641)

Signed-off-by: byhsu <byhsu@linkedin.com>

* feat: Add Auth0/audience support for ClientCredentials flow (#1639)

* feat: Add Auth0/audience support for ClientCredentials flow

Signed-off-by: tnam <tommy.nam@aiq-index.com>

* refactor: Remove unneeded variables & condense code

Signed-off-by: tnam <tommy.nam@aiq-index.com>

* refactor: Reduce verbosity of code

Signed-off-by: tnam <tommy.nam@aiq-index.com>

* refactor(chore): Remove unused commented code

Signed-off-by: tnam <tommy.nam@aiq-index.com>

* fix: Missing comma in input args - authenticator.py 213

Signed-off-by: tnam <tommy.nam@aiq-index.com>

* style: Run pre-commit against all files

Signed-off-by: tnam <tommy.nam@aiq-index.com>

---------

Signed-off-by: tnam <tommy.nam@aiq-index.com>
Co-authored-by: tnam <tommy.nam@aiq-index.com>

* pyflyte run remote file (#1670)

Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Kevin Su <pingsutw@apache.org>

* upload deck.html only with deck enable (#1693)

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* Add dask plugin #patch (#1366)

* Add dummy task type to test backend plugin

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add docs page

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add dask models

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add function to convert resources

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add tests to `dask` task

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Remove namespace

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update setup.py

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add dask to `plugin/README.md`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Add README.md for `dask`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Top level export of `JopPodSpec` and `DaskCluster`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update docs for images

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update README.md

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update models after `flyteidl` change

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update task after `flyteidl` change

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Raise error when less than 1 worker

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update flyteidl to >= 1.3.2

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update doc requirements

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Update doc-requirements.txt

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Re-lock dependencies on linux

Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com>

* Update dask API docs

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Fix documentation links

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Default optional model constructor arguments to `None`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Refactor `convert_resources_to_resource_model` to `core.resources`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Use `convert_resources_to_resource_model` in `core.node`

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

* Incorporate review feedback

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Lint

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Do not bring dask changes

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Remove readthedocs

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Linting

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Force scipy<1.11.0 is on windows

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yubo Wang <yubwang@linkedin.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>
Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
Signed-off-by: byhsu <byhsu@linkedin.com>
Signed-off-by: tnam <tommy.nam@aiq-index.com>
Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com>
Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>
Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Co-authored-by: Yubo Wang <yubowang2019@gmail.com>
Co-authored-by: Yubo Wang <yubwang@linkedin.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Fabio M. Graetz, Ph.D <fabiograetz@googlemail.com>
Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com>
Co-authored-by: Yicheng-Lu-llll <51814063+Yicheng-Lu-llll@users.noreply.github.com>
Co-authored-by: ByronHsu <byronhsu1230@gmail.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
Co-authored-by: TomNam <92072956+PudgyPigeon@users.noreply.github.com>
Co-authored-by: tnam <tommy.nam@aiq-index.com>
Co-authored-by: ChungYujoyce <46030368+ChungYujoyce@users.noreply.github.com>
Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants