Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Faster registration in flytekit #554

Closed
1 of 13 tasks
katrogan opened this issue Oct 14, 2020 · 0 comments
Closed
1 of 13 tasks

[Feature] Faster registration in flytekit #554

katrogan opened this issue Oct 14, 2020 · 0 comments
Assignees
@katrogan
Labels
enhancement New feature or request
Milestone
0.10.0

Comments

@katrogan
Copy link
Contributor

Motivation: Why do you think this is important?
In order to improve iteration time and the end-user experience it would be helpful to speed-up workflow registration. Currently the process for building an image can be slow (at Lyft, on the order of many minutes) which is extremely annoying for flyte workflow authors.

Goal: What should the final outcome look like, ideally?
Registering a new version of a workflow (without updating container dependencies) should take a few seconds.

Describe alternatives you've considered

  • code pickling Instead of rebuilding a container on every task/workflow version change, registration can be modified to pass along pickled code which is then unpickled at execution time (idea/doc courtesy of @kumare3)

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

[Optional] Propose: Link/Inline
https://docs.google.com/document/d/1n7DmKXaOv8dAb9eHD4TKqka6FsdJNs3Tugpf1I6nvMg/edit?usp=sharing

Additional context
Add any other context or screenshots about the feature request here.

Is this a blocker for you to adopt Flyte
no
@katrogan katrogan added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Oct 14, 2020
@katrogan katrogan self-assigned this Oct 14, 2020
@katrogan katrogan removed the untriaged This issues has not yet been looked at by the Maintainers label Oct 14, 2020
@kumare3 kumare3 added this to the 0.9.0 milestone Oct 15, 2020
@katrogan katrogan mentioned this issue Oct 29, 2020
Merged
8 tasks
@katrogan katrogan modified the milestones: 0.9.0, 0.10.0 Nov 4, 2020
@katrogan katrogan mentioned this issue Nov 11, 2020
Closed
This was referenced Nov 16, 2020
Merged
Merged
@katrogan katrogan mentioned this issue Dec 4, 2020
Closed
8 tasks
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Jul 24, 2023
@jeevb
fix makefile to read variables from environment and overrides (flyteo…
b9d1f62 
…rg#554)

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Jul 24, 2023
@jeevb @hamersaw @EngHabu
Implement ability to specify additional/override annotations when usi…
972c0d6 
…ng Vault Secret Manager (flyteorg#556)

* Implement ability to specify additional annotations when using Vault secret manager

Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Infer GOOS and GOARCH from environment (flyteorg#552)

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* fix makefile to read variables from environment and overrides (flyteorg#554)

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Remove BarrierTick (flyteorg#545)

* removed barrier logic

Signed-off-by: Daniel Rammer <daniel@union.ai>

* deprecated TransitionTypeBarrier

Signed-off-by: Daniel Rammer <daniel@union.ai>

* removed barrier tests

Signed-off-by: Daniel Rammer <daniel@union.ai>

* bumping flyteplugins

Signed-off-by: Daniel Rammer <daniel@union.ai>

---------

Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Check for TerminateExecution error and eat Precondition status (flyteorg#553)

* Check for TerminateExecution error and eat Precondition status

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* lint

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Rename to annotation

Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Inline merging annotations

Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

---------

Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Co-authored-by: Jeev B <jeevb@users.noreply.github.com>
Co-authored-by: Dan Rammer <daniel@union.ai>
Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com>
eapolinario added a commit to eapolinario/flyte that referenced this issue Jul 24, 2023
@eapolinario
Add migration to turn parent_id column into bigint only if necess…
00915ec 
…ary (flyteorg#554)

* Add migration to turn `parent_id` column into `bigint` only if necessary

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Migrations to handle the change of `parent_id` column in `node_executions` table

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Add unit test

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* More coverage

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Create index outside transaction

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@jeevb
fix makefile to read variables from environment and overrides (flyteo…
36d84f7 
…rg#554)

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@jeevb @hamersaw @EngHabu
Implement ability to specify additional/override annotations when usi…
7b0243d 
…ng Vault Secret Manager (flyteorg#556)

* Implement ability to specify additional annotations when using Vault secret manager

Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Infer GOOS and GOARCH from environment (flyteorg#552)

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* fix makefile to read variables from environment and overrides (flyteorg#554)

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Remove BarrierTick (flyteorg#545)

* removed barrier logic

Signed-off-by: Daniel Rammer <daniel@union.ai>

* deprecated TransitionTypeBarrier

Signed-off-by: Daniel Rammer <daniel@union.ai>

* removed barrier tests

Signed-off-by: Daniel Rammer <daniel@union.ai>

* bumping flyteplugins

Signed-off-by: Daniel Rammer <daniel@union.ai>

---------

Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Check for TerminateExecution error and eat Precondition status (flyteorg#553)

* Check for TerminateExecution error and eat Precondition status

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* lint

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Rename to annotation

Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

* Inline merging annotations

Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>

---------

Signed-off-by: Pradithya Aria <pradithya.pura@go-jek.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Co-authored-by: Jeev B <jeevb@users.noreply.github.com>
Co-authored-by: Dan Rammer <daniel@union.ai>
Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com>
eapolinario added a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@eapolinario
Add migration to turn parent_id column into bigint only if necess…
2066e7a 
…ary (flyteorg#554)

* Add migration to turn `parent_id` column into `bigint` only if necessary

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Migrations to handle the change of `parent_id` column in `node_executions` table

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Add unit test

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* More coverage

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Create index outside transaction

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@katrogan katrogan

Labels
enhancement New feature or request
Projects
None yet
Milestone
0.10.0
Development

No branches or pull requests
2 participants
@katrogan @kumare3