feat: make Test Workflow processing more stable #5814

rangoo94 · 2024-09-04T12:59:32Z

Pull request description

Make execution signature available in Pod spec as label
Replace Control Server from HTTP to bare faster TCP
Avoid reading all logs into memory
Rewrite watching the Test Workflows execution
Handle EPIPE, connection lost and other edge case errors in the containers and cluster
Reuse Execution Controllers when it is possible (to avoid duplicated Kubernetes watchers)
Retry saving the logs when it is possible
Retry fetching the tarball

Checklist (choose whats happened)

Fixes

TKC-2387

…ies from resources

… received

cmd/tcl/testworkflow-toolkit/spawn/utils.go

cmd/testworkflow-toolkit/commands/tarball.go

pkg/testworkflows/testworkflowcontroller/logs.go

pkg/testworkflows/testworkflowcontroller/watchers/commons.go

pkg/testworkflows/testworkflowcontroller/watchers/executionwatcher.go

… newest logs/events

pkg/testworkflows/testworkflowcontroller/watchinstrumentedpod.go

pkg/testworkflows/testworkflowexecutor/executor.go

cmd/tcl/testworkflow-toolkit/commands/parallel.go

vsukhin

lgtm

* feat: add new individual Events and Job Watchers * feat: add individual pod watcher * fix: resuming concurrency problems in parallel operations * feat: add multiple fallbacks for fetching important execution properties from resources * chore: get rid of Peek(ctx) and Exists() from PodWatcher and JobWatcher * chore: get rid of Stop() from PodWatcher, JobWatcher and EventsWatcher * chore: rename `hook` to `listener`, delete IsStarted() * chore: store watcher's error directly in the context * chore: extract code for reading the job/pod latest timestamp * chore: add boolean to determine if the watcher is actually watching * chore: make pod events watcher immediately available when pod name is received * feat: reimplement watching * fix: handle case when the job is aborted before the pod is started * fix: building unique references on the layer * fix: deadlock while watching the execution * feat: make the control server lightweight and faster (small TCP packets instead of HTTP) * fix: add retrying mechanism for resuming the parallel workflows * chore: delete Merge() from TestWorkflowStepResult * feat: implement refined way to build Test Workflow's result * feat: speed up aborting job * chore: get rid of part of Notifier functionality * fix: make sending the results more stable * chore: optimize getting container error from Pod * chore: optimize reading container name from event * chore: optimize building result state by caching actions and sequence * chore: delete some logs and avoid sending finishedAt without status * fix: handle edge case with node resources issue (EPIPE) * chore: move RecomputeTimestamps implementation to TestWorkflowResult * chore: move more healing functions to TestWorkflowResult * chore: move another set of healing functions to TestWorkflowResult * fix: adjust a bit completion timestamp * fix: handle GOAWAY in logs * fix: handle connection to control plane lost in logs * fix: retry fetching tarball on failure * fix: retry fetching parallel/service logs * fix: retry saving the logs to Minio * fix: reduce number of FS operations in Init Process * fix: avoid negative periods in results * fix: processing tests with massive logs, align completion time to the newest logs/events

rangoo94 added 30 commits August 29, 2024 09:15

feat: add new individual Events and Job Watchers

644a192

feat: add individual pod watcher

3dbbecf

chore: adjust watchers

70523e7

fix: improve watchers stability

267d03e

fix: improve watchers stability

c7f59ca

fix: improve watchers stability

7bfb44e

fix: improve watchers stability

c7694ec

fix: watcher race conditions

060354c

fix: resuming concurrency problems in parallel operations

e6fadef

fix: watchers improvements

45d2f23

feat: initial rewrite of watching the test workflow executions

b8fc4b8

fixup delete part of debug

1948569

feat: add multiple fallbacks for fetching important execution propert…

e1062ba

…ies from resources

chore: delete debug and add todo

f606cde

feat: next iteration of watcher redesign

e47014d

chore: get rid of Peek(ctx) and Exists() from PodWatcher and JobWatcher

a2ef7dd

chore: get rid of Stop() from PodWatcher, JobWatcher and EventsWatcher

7d845d4

chore: rename hook to listener, delete IsStarted()

6eab3af

chore: store watcher's error directly in the context

ae1c589

chore: simplify using startedCh

738e467

chore: simplify

fe9ebd8

chore: extract code for reading the job/pod latest timestamp

e0cb7b5

fix: exposing read events

1d039d7

chore: clean

334f87e

chore: add boolean to determine if the watcher is actually watching

a6deeea

chore: make pod events watcher immediately available when pod name is…

95f71cc

… received

feat: reimplement watching

4894c7f

chore: simplify a bit accessing pod/job data

565a4ee

fixup

cbffa78

fix: handle case when the job is aborted before the pod is started

6679e00

chore: combine resultState and notifier

9808870

rangoo94 marked this pull request as ready for review September 4, 2024 17:01

rangoo94 requested a review from a team as a code owner September 4, 2024 17:01

rangoo94 added 9 commits September 5, 2024 11:05

fixup error messages

dc76f38

fix: handle GOAWAY in logs

e60f340

fix: handle connection to control plane lost in logs

85f5ba9

fix: retry fetching tarball on failure

6bd328a

fix: retry fetching parallel/service logs

307fc32

fix: retry saving the logs to Minio

dc497c0

chore: use context instead of timeout duration for refreshing pod/job

b695d8d

fix: reduce number of FS operations in Init Process

eb88f62

fix: avoid negative periods in results

4e053d1