Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: change the Test Workflow orchestration mechanism to atomic instructions #5676

Merged
merged 87 commits into from
Jul 26, 2024

Conversation

rangoo94
Copy link
Member

@rangoo94 rangoo94 commented Jul 19, 2024

Pull request description

  • Replace the orchestration from building Init Process arguments per container, to build full list of atomic instructions (actions) to run, like start, end, execute, container or declare
    • This way, we are preparing into having multiple operations in a single container
  • Optimize these actions to avoid unnecessary computation and differences
  • Support secret obfuscation
    • All values of environment variables sourced from secret will be masked in the output
  • Clean up the conditions/retries/timeouts logic
    • i.e. nested timeouts will work better now, as the group will properly persist its start time

Checklist (choose whats happened)

  • breaking change! (describe)
  • tested locally
  • tested on cluster
  • added new dependencies
  • updated the docs
  • added a test

@rangoo94 rangoo94 requested a review from a team as a code owner July 19, 2024 17:17
@rangoo94 rangoo94 marked this pull request as draft July 19, 2024 17:17
@rangoo94 rangoo94 force-pushed the dawid/chore/detach-container-name-of-ref branch from 20ccf70 to ba35d0a Compare July 19, 2024 17:17
Copy link

netlify bot commented Jul 19, 2024

Deploy Preview for testkube-docs-preview ready!

Name Link
🔨 Latest commit 20ccf70
🔍 Latest deploy log https://app.netlify.com/sites/testkube-docs-preview/deploys/669a9f912bb9d100081a9ec8
😎 Deploy Preview https://deploy-preview-5676--testkube-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link

netlify bot commented Jul 19, 2024

Deploy Preview for testkube-docs-preview canceled.

Name Link
🔨 Latest commit 9cb876c
🔍 Latest deploy log https://app.netlify.com/sites/testkube-docs-preview/deploys/66a39dc36da842000835d4b7

@rangoo94 rangoo94 marked this pull request as ready for review July 25, 2024 14:51
Copy link
Collaborator

@vsukhin vsukhin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comments
a lot of TODO, not clear which of them are optional or mandatory

@rangoo94 rangoo94 merged commit 220ac74 into develop Jul 26, 2024
11 checks passed
@rangoo94 rangoo94 deleted the dawid/chore/detach-container-name-of-ref branch July 26, 2024 13:49
exu added a commit that referenced this pull request Aug 2, 2024
rangoo94 added a commit that referenced this pull request Aug 14, 2024
…ructions (#5676)

* fix(testworkflows): avoid unnecessary empty data in ContainerConfig
* feat(testworkflows): add basic implementation to build list of operations / grouping them, based on the TestWorkflowProcessor's Stages
* feat(testworkflows): migrate basics of container execution for handling multiple steps in single container
* feat: support conditions with new Init Process kind
* feat: encapsulate init process' container state
* chore: rename container to setup
* feat: support dynamic environment variables
* feat: add mock for pause in the Init Process
* feat: discriminate TestWorkflow instructions with ActionType
* chore: rename Paused to PausedOnStart
* chore: change StepData mutations
* chore: extract action.Action to separate package
* feat: move *Stage to separate package
* chore: reorganize action files
* chore: delete unused
* chore: extract lightweight version of actions to avoid increase in Init Process size
* feat: abstract the executions in the Init Process
* fix: aborting Test Workflow processes
* fix: support statuses properly in the Init Process
* fix: parsing container logs
* chore: delete debug information
* feat: make the root operation constant
* feat: support timeout for Test Workflow steps
* feat: add back retry functionality for the Test Workflows
* feat: support pause/resume for Test Workflows
* chore: delete unused code
* fix: watching services
* fix: some unit tests
* chore: add instructions check in unit test
* chore: refactor action unit tests a bit
* fix: build correct conditions when there are grouped steps
* chore: adjust processor tests
* chore: delete unused function
* feat: streamline the output in the Init Process, to allow obfuscating sensitive words
* feat: mask secret environment variables
* chore: extract function to get last step
* chore: unify internal machine for Init Process
* chore: move step timeout logic to the Init Process main switch
* fix: use default ContainerConfig for setup step container
* feat: use /bin/sh instead of /.tktw/bin/sh in case the Init/Toolkit image is used
* chore: fail property accessor in case of nullish value to access
* feat: ensure that aborted step is considered aborted, not skipped
* fix: machine for retry in the Init Process
* chore: rename `instructions` related to Actions to `actions`
* fix: send again toolkit/init image data along with Pod information from Test Workflow
* chore: simplify StepStatusFromCode
* fix: clean up printing in the Init Process
* chore: clean up getting actions in Processor tests
* chore: rename variable in Control Server
* chore: extract control server options to separate file
* chore: expose a bit configuration of sensitive words minimum length
* chore: delete obsolete todos
* fix: avoid copying binaries from the Init image when it's not necessary
* fix: unit tests
* fix: group conditions
* chore: wrap errors
* chore: reduce the size of Init Process
* fix: detect the Toolkit properly
* chore: rename SensitiveReadWriter to Obfuscator
* feat: show last characters in the obfuscated log output
* feat: make some environment variables sensitive by default
* chore: first set of code review fixes
* chore: make the 00 and 01 special groups constants
* chore: log error for mkdir of working directory
* chore: delete commented out code
* fix: handle errors in the setup step
rangoo94 added a commit that referenced this pull request Aug 14, 2024
…ructions (#5676)

* fix(testworkflows): avoid unnecessary empty data in ContainerConfig
* feat(testworkflows): add basic implementation to build list of operations / grouping them, based on the TestWorkflowProcessor's Stages
* feat(testworkflows): migrate basics of container execution for handling multiple steps in single container
* feat: support conditions with new Init Process kind
* feat: encapsulate init process' container state
* chore: rename container to setup
* feat: support dynamic environment variables
* feat: add mock for pause in the Init Process
* feat: discriminate TestWorkflow instructions with ActionType
* chore: rename Paused to PausedOnStart
* chore: change StepData mutations
* chore: extract action.Action to separate package
* feat: move *Stage to separate package
* chore: reorganize action files
* chore: delete unused
* chore: extract lightweight version of actions to avoid increase in Init Process size
* feat: abstract the executions in the Init Process
* fix: aborting Test Workflow processes
* fix: support statuses properly in the Init Process
* fix: parsing container logs
* chore: delete debug information
* feat: make the root operation constant
* feat: support timeout for Test Workflow steps
* feat: add back retry functionality for the Test Workflows
* feat: support pause/resume for Test Workflows
* chore: delete unused code
* fix: watching services
* fix: some unit tests
* chore: add instructions check in unit test
* chore: refactor action unit tests a bit
* fix: build correct conditions when there are grouped steps
* chore: adjust processor tests
* chore: delete unused function
* feat: streamline the output in the Init Process, to allow obfuscating sensitive words
* feat: mask secret environment variables
* chore: extract function to get last step
* chore: unify internal machine for Init Process
* chore: move step timeout logic to the Init Process main switch
* fix: use default ContainerConfig for setup step container
* feat: use /bin/sh instead of /.tktw/bin/sh in case the Init/Toolkit image is used
* chore: fail property accessor in case of nullish value to access
* feat: ensure that aborted step is considered aborted, not skipped
* fix: machine for retry in the Init Process
* chore: rename `instructions` related to Actions to `actions`
* fix: send again toolkit/init image data along with Pod information from Test Workflow
* chore: simplify StepStatusFromCode
* fix: clean up printing in the Init Process
* chore: clean up getting actions in Processor tests
* chore: rename variable in Control Server
* chore: extract control server options to separate file
* chore: expose a bit configuration of sensitive words minimum length
* chore: delete obsolete todos
* fix: avoid copying binaries from the Init image when it's not necessary
* fix: unit tests
* fix: group conditions
* chore: wrap errors
* chore: reduce the size of Init Process
* fix: detect the Toolkit properly
* chore: rename SensitiveReadWriter to Obfuscator
* feat: show last characters in the obfuscated log output
* feat: make some environment variables sensitive by default
* chore: first set of code review fixes
* chore: make the 00 and 01 special groups constants
* chore: log error for mkdir of working directory
* chore: delete commented out code
* fix: handle errors in the setup step
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants