Flyte v0.5.0 release #374
Assignees
Milestone
Comments
Merged
|
This is now live |
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 6, 2022
* Scale out with propeller manager and workflow sharding (flyteorg#351) * added 'manager' command Signed-off-by: Daniel Rammer <daniel@union.ai> * using go routine and timer for manager loop Signed-off-by: Daniel Rammer <daniel@union.ai> * moved manager loop out of cmd and into pkg directory Signed-off-by: Daniel Rammer <daniel@union.ai> * detecting missing replicas Signed-off-by: Daniel Rammer <daniel@union.ai> * moved extracting replica from pod name to new function Signed-off-by: Daniel Rammer <daniel@union.ai> * creating managed flytepropeller pods Signed-off-by: Daniel Rammer <daniel@union.ai> * refactored configuration Signed-off-by: Daniel Rammer <daniel@union.ai> * removed regex parsing for replica - checking for existance with fully qualified pod name Signed-off-by: Daniel Rammer <daniel@union.ai> * mocked out shard strategy abstraction Signed-off-by: Daniel Rammer <daniel@union.ai> * adding arguments to podspec for ConsistentHashingShardStrategy Signed-off-by: Daniel Rammer <daniel@union.ai> * updated import naming Signed-off-by: Daniel Rammer <daniel@union.ai> * moved manager to a top-level package Signed-off-by: Daniel Rammer <daniel@union.ai> * added shard strategy to manager configuration Signed-off-by: Daniel Rammer <daniel@union.ai> * setting shard key label selector on managed propeller instances Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed random lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * split pod name generate to separate function to ease future auto-scaler implementation Signed-off-by: Daniel Rammer <daniel@union.ai> * cleaned up pod label selector Signed-off-by: Daniel Rammer <daniel@union.ai> * delete pods on shutdown Signed-off-by: Daniel Rammer <daniel@union.ai> * added prometheus metric reporting Signed-off-by: Daniel Rammer <daniel@union.ai> * updated manager run loop to use k8s wait.UntilWithContext Signed-off-by: Daniel Rammer <daniel@union.ai> * moved getKubeConfig into a shared package Signed-off-by: Daniel Rammer <daniel@union.ai> * assigning shard and namespace labels on FlyteWorkflow Signed-off-by: Daniel Rammer <daniel@union.ai> * implement NamespaceShardStrategy Signed-off-by: Daniel Rammer <daniel@union.ai> * implemented NamespaceShardStrategy Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed shard label Signed-off-by: Daniel Rammer <daniel@union.ai> * added comments Signed-off-by: Daniel Rammer <daniel@union.ai> * checking for existing pods on startup Signed-off-by: Daniel Rammer <daniel@union.ai> * handling delete of non-existent pod Signed-off-by: Daniel Rammer <daniel@union.ai> * changes ConsistentHashing name to Random - because that's what it really is Signed-off-by: Daniel Rammer <daniel@union.ai> * implemented EnableUncoveredReplica configuration option Signed-off-by: Daniel Rammer <daniel@union.ai> * added leader election to manager using existing propeller config Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed disable leader election in managed propeller pods Signed-off-by: Daniel Rammer <daniel@union.ai> * removed listPods function Signed-off-by: Daniel Rammer <daniel@union.ai> * added leader election to mitigate concurrent modification issues Signed-off-by: Daniel Rammer <daniel@union.ai> * enabled pprof to profile resource metrics Signed-off-by: Daniel Rammer <daniel@union.ai> * added 'manager' target to Makefile to start manager in development mode (similar to existing server) Signed-off-by: Daniel Rammer <daniel@union.ai> * added shard strategy test for computing key ranges Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed key range computation Signed-off-by: Daniel Rammer <daniel@union.ai> * implemented project and domain shard types Signed-off-by: Daniel Rammer <daniel@union.ai> * returning error on out of range podIndex during UpdatePodSpec call on shard strategy Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed random lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * added manager tests Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * added doc comments on exported types and functions Signed-off-by: Daniel Rammer <daniel@union.ai> * exporting ComputeKeyRange function and changed adding addLabelSelector function name to addLabelSelectorIfExists to better reflect functionality Signed-off-by: Daniel Rammer <daniel@union.ai> * adding pod template resource version and shard config hash annotations to fuel automatic pod management on updates Signed-off-by: Daniel Rammer <daniel@union.ai> * removed pod deletion on manager shutdown Signed-off-by: Daniel Rammer <daniel@union.ai> * cleaned up unit tests and lint Signed-off-by: Daniel Rammer <daniel@union.ai> * updated getContainer function to retrive flytepropeller container from pod spec using container name instead of command Signed-off-by: Daniel Rammer <daniel@union.ai> * removed addLabelSelectorIfExists function call Signed-off-by: Daniel Rammer <daniel@union.ai> * changed bytes.Buffer from a var to declaring with new Signed-off-by: Daniel Rammer <daniel@union.ai> * created a new shardstrategy package Signed-off-by: Daniel Rammer <daniel@union.ai> * generating mocks for ShardStrategy to decouple manager package tests from shardstrategy package tests Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * changed shard configuration defintions and added support for wildcard id in EnvironmentShardStrategy Signed-off-by: Daniel Rammer <daniel@union.ai> * updated documentation Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * setting managed pod owner references Signed-off-by: Daniel Rammer <daniel@union.ai> * updated documentation Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed a few nits Signed-off-by: Daniel Rammer <daniel@union.ai> * delete pods with failed state Signed-off-by: Daniel Rammer <daniel@union.ai> * changed ShardType type to int instead of string Signed-off-by: Daniel Rammer <daniel@union.ai> * removed default values in manager config Signed-off-by: Daniel Rammer <daniel@union.ai> * updated config_flags with pflags generation Signed-off-by: Daniel Rammer <daniel@union.ai> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Create codeql-analysis.yml Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Handle code quality issue Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * check boundaries Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * 0 is ok Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Use ParseUint instead Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * bump for DCO Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Co-authored-by: Dan Rammer <daniel@union.ai>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 20, 2022
…fo (flyteorg#374) * fix: service account is not properly propagated from the execution info Signed-off-by: eugenejahn <eugenejahnjahn@gmail.com>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 20, 2022
## [0.51.2](http://github.com/lyft/flyteconsole/compare/v0.51.1...v0.51.2) (2022-04-13) ### Bug Fixes * service account is not properly propagated from the execution info ([flyteorg#374](http://github.com/lyft/flyteconsole/issues/374)) ([2e250bb](http://github.com/lyft/flyteconsole/commit/2e250bb53159069bc34b8578ac53aa64334837ce))
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Aug 9, 2023
* Scale out with propeller manager and workflow sharding (flyteorg#351) * added 'manager' command Signed-off-by: Daniel Rammer <daniel@union.ai> * using go routine and timer for manager loop Signed-off-by: Daniel Rammer <daniel@union.ai> * moved manager loop out of cmd and into pkg directory Signed-off-by: Daniel Rammer <daniel@union.ai> * detecting missing replicas Signed-off-by: Daniel Rammer <daniel@union.ai> * moved extracting replica from pod name to new function Signed-off-by: Daniel Rammer <daniel@union.ai> * creating managed flytepropeller pods Signed-off-by: Daniel Rammer <daniel@union.ai> * refactored configuration Signed-off-by: Daniel Rammer <daniel@union.ai> * removed regex parsing for replica - checking for existance with fully qualified pod name Signed-off-by: Daniel Rammer <daniel@union.ai> * mocked out shard strategy abstraction Signed-off-by: Daniel Rammer <daniel@union.ai> * adding arguments to podspec for ConsistentHashingShardStrategy Signed-off-by: Daniel Rammer <daniel@union.ai> * updated import naming Signed-off-by: Daniel Rammer <daniel@union.ai> * moved manager to a top-level package Signed-off-by: Daniel Rammer <daniel@union.ai> * added shard strategy to manager configuration Signed-off-by: Daniel Rammer <daniel@union.ai> * setting shard key label selector on managed propeller instances Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed random lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * split pod name generate to separate function to ease future auto-scaler implementation Signed-off-by: Daniel Rammer <daniel@union.ai> * cleaned up pod label selector Signed-off-by: Daniel Rammer <daniel@union.ai> * delete pods on shutdown Signed-off-by: Daniel Rammer <daniel@union.ai> * added prometheus metric reporting Signed-off-by: Daniel Rammer <daniel@union.ai> * updated manager run loop to use k8s wait.UntilWithContext Signed-off-by: Daniel Rammer <daniel@union.ai> * moved getKubeConfig into a shared package Signed-off-by: Daniel Rammer <daniel@union.ai> * assigning shard and namespace labels on FlyteWorkflow Signed-off-by: Daniel Rammer <daniel@union.ai> * implement NamespaceShardStrategy Signed-off-by: Daniel Rammer <daniel@union.ai> * implemented NamespaceShardStrategy Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed shard label Signed-off-by: Daniel Rammer <daniel@union.ai> * added comments Signed-off-by: Daniel Rammer <daniel@union.ai> * checking for existing pods on startup Signed-off-by: Daniel Rammer <daniel@union.ai> * handling delete of non-existent pod Signed-off-by: Daniel Rammer <daniel@union.ai> * changes ConsistentHashing name to Random - because that's what it really is Signed-off-by: Daniel Rammer <daniel@union.ai> * implemented EnableUncoveredReplica configuration option Signed-off-by: Daniel Rammer <daniel@union.ai> * added leader election to manager using existing propeller config Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed disable leader election in managed propeller pods Signed-off-by: Daniel Rammer <daniel@union.ai> * removed listPods function Signed-off-by: Daniel Rammer <daniel@union.ai> * added leader election to mitigate concurrent modification issues Signed-off-by: Daniel Rammer <daniel@union.ai> * enabled pprof to profile resource metrics Signed-off-by: Daniel Rammer <daniel@union.ai> * added 'manager' target to Makefile to start manager in development mode (similar to existing server) Signed-off-by: Daniel Rammer <daniel@union.ai> * added shard strategy test for computing key ranges Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed key range computation Signed-off-by: Daniel Rammer <daniel@union.ai> * implemented project and domain shard types Signed-off-by: Daniel Rammer <daniel@union.ai> * returning error on out of range podIndex during UpdatePodSpec call on shard strategy Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed random lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * added manager tests Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * added doc comments on exported types and functions Signed-off-by: Daniel Rammer <daniel@union.ai> * exporting ComputeKeyRange function and changed adding addLabelSelector function name to addLabelSelectorIfExists to better reflect functionality Signed-off-by: Daniel Rammer <daniel@union.ai> * adding pod template resource version and shard config hash annotations to fuel automatic pod management on updates Signed-off-by: Daniel Rammer <daniel@union.ai> * removed pod deletion on manager shutdown Signed-off-by: Daniel Rammer <daniel@union.ai> * cleaned up unit tests and lint Signed-off-by: Daniel Rammer <daniel@union.ai> * updated getContainer function to retrive flytepropeller container from pod spec using container name instead of command Signed-off-by: Daniel Rammer <daniel@union.ai> * removed addLabelSelectorIfExists function call Signed-off-by: Daniel Rammer <daniel@union.ai> * changed bytes.Buffer from a var to declaring with new Signed-off-by: Daniel Rammer <daniel@union.ai> * created a new shardstrategy package Signed-off-by: Daniel Rammer <daniel@union.ai> * generating mocks for ShardStrategy to decouple manager package tests from shardstrategy package tests Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * changed shard configuration defintions and added support for wildcard id in EnvironmentShardStrategy Signed-off-by: Daniel Rammer <daniel@union.ai> * updated documentation Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * setting managed pod owner references Signed-off-by: Daniel Rammer <daniel@union.ai> * updated documentation Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed a few nits Signed-off-by: Daniel Rammer <daniel@union.ai> * delete pods with failed state Signed-off-by: Daniel Rammer <daniel@union.ai> * changed ShardType type to int instead of string Signed-off-by: Daniel Rammer <daniel@union.ai> * removed default values in manager config Signed-off-by: Daniel Rammer <daniel@union.ai> * updated config_flags with pflags generation Signed-off-by: Daniel Rammer <daniel@union.ai> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Create codeql-analysis.yml Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Handle code quality issue Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * check boundaries Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * 0 is ok Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Use ParseUint instead Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * bump for DCO Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Co-authored-by: Dan Rammer <daniel@union.ai>
eapolinario
pushed a commit
that referenced
this issue
Sep 26, 2023
* Add failing test Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com> * WIP Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com> * Improve test Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com> * Refactor to use `ToK8sPodSpec` Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com> * Fix linting Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com> * Use `Always` restart policy for workers Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com> * Add test which checks whether labels are propagated Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com> * Replace `removeInterruptibleConfig` with `TaskExectuionMetadata` wrapper Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com> --------- Signed-off-by: Bernhard Stadlbauer <11799671+bstadlbauer@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Propeller: v0.2.64
Admin: v0.2.11
Flytekit: v0.10.0
Console: v0.5.3
IDL: 0.17.34
The text was updated successfully, but these errors were encountered: