Skip to content

Conversation

@lexbritvin
Copy link
Collaborator

No description provided.

@lexbritvin
Copy link
Collaborator Author

@davidferlay
To test the kubernetes integration

  1. You don't need kubectl installed, but ensure you have ~/.kube with a configured cluster
  2. Create .launchr/config.yaml. This behavior is described in launchr docs. Content
     container:
       runtime: kubernetes
  3. The local directories are copied to the remote container before start like it worked with --use-volume-wd, but the flag removed now. See --help. This behavior is described in launchr docs.
  4. Actions are executed in Pods with ephemeral containers, see kubernetes docs for more info.
  5. Pods are deleted after action finish
  6. If you produce artifacts during action run, use --remote-copy-back to copy the result directory. Works as --use-volume-wd before but now with explicit flag. This behavior is described in launchr docs.
  7. Image build is not supported yet. The image must exist in the available container registry.

@lexbritvin lexbritvin requested a review from iignatevich May 6, 2025 23:18
@lexbritvin
Copy link
Collaborator Author

@iignatevich @davidferlay
Since there are a lot of changes and if Docker part is not broken, I suggest to merge it to upstream as-is, only with regression tests. And then evolve based on that.

@davidferlay
Copy link
Contributor

Thanks @lexbritvin ,

Roger that, I'm gonna make regression tests and we will merge as is.

About Image build is not supported yet. The image must exist in the available container registry.

Any idea how to support that ?

switch t {
case Docker:
return NewDockerDriver()
return NewDockerRuntime()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it should be named as New%TypeDriver or New%TypeRunner + renaming internal structs, as it's confusing with NewContainerRuntimeKubernetes and NewContainerRuntimeDocker

Copy link
Collaborator Author

@lexbritvin lexbritvin May 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is confusing. And naming must be addressed here.
Driver is a bad naming. So we don't consider it for now.
I suggest that we cover the case when we rename the package driver to container.

A bit info about why so:

  • New%TypeRunner - we don't create a Type and we don't create a runner.
  • We create a client to Runtime. See about Container Runtimes and CRI. Even though kubernetes is not exactly a runtime, for us it is.
  • In the action we also have Runtime, but it's about what is running the action. It can be function, shell, container.
  • We also have runtime package in go that describes the go language and os.

Please, let me know what you think about it.

Copy link
Collaborator

@iignatevich iignatevich May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's rename to container, but as discussed, need to rethink runtime (upper level) naming.

@lexbritvin lexbritvin changed the title K8s Kubernetes May 18, 2025
@davidferlay
Copy link
Contributor

davidferlay commented May 19, 2025

Bellow some feedbacks from first test -> just dropping here to be treated later in follow-ups

Pods get created as expected locally with k3s:

  • env var set with: export KUBECONFIG="/etc/rancher/k3s/k3s.yaml"
  • .plasmactl/config.yaml
    container.runtime value is expected to be at root of config file:
➜ kubernetes runtime defined in `.plasmactl/config.yaml`:
launchrctl:
  actions_naming:
    - search: ".roles."
      replace: "."
    - search: "_"
      replace: "-"
container:
  runtime: kubernetes
Every 1.0s: k3s kubectl get all

NAME                                           READY   STATUS    RESTARTS   AGE
pod/plasmactl-platform-nodes-peaceful-austin   1/1     Running   0          4s
  • Why not have container key inside of launchrctl key for consistency ?

  • Would be great to have a way to define --remote-copy-back true globally, using either an env var or a key in .config file (or both)

Executing an action with container runtime fails with failed to patch ephemeral container to pod: the server could not find the requested resource:

➜ plx platform:nodes -vvvv       
2025-05-19 16:24:04 DEBUG hook DiscoveryPlugin       
                      └ plugins: [{github.com/launchrctl/launchr/plugins/builder.Plugin 0xc000682ae0} {github.com/skilld-labs/plasmactl-release.Plugin 0x41f3a80} {github.com/launchrctl/keyring.Plugin 0xc00033ecc0} {github.com/launchrctl/web.Plugin 0xc000141da0} {github.com/skilld-labs/plasmactl-bump/v2.Plugin 0xc000141e00} {github.com/launchrctl/compose.Plugin 0xc000141d00} {github.com/skilld-labs/plasmactl-update.Plugin 0xc000682ba0} {github.com/skilld-labs/plasmactl-dependencies.Plugin 0x41f3a80} {github.com/skilld-labs/plasmactl-meta.Plugin 0xc00033ef30} {github.com/launchrctl/launchr/plugins/yamldiscovery.Plugin 0xc000141c40}]
2025-05-19 16:24:04 DEBUG init success
                      ├ wd: /home/dferlay/Sources/ski-platform
                      └ actions_dir: /home/dferlay/Sources/ski-platform
2025-05-19 16:24:04 DEBUG executing command 
2025-05-19 16:24:04 DEBUG hook CobraPlugin
                      └ plugins: [{github.com/launchrctl/keyring.Plugin 0xc00033ecc0} {github.com/launchrctl/launchr/plugins/actionscobra.Plugin 0xc00033ec00}]
2025-05-19 16:24:04 DEBUG action was overridden by another declaration
                      ├ action_id: platform.entities.timelog:diff
                      ├ old: /home/dferlay/Sources/ski-platform/platform/entities/roles/timelog/actions/diff/action.yaml
                      └ new: /home/dferlay/Sources/ski-platform/.compose/build/platform/entities/roles/timelog/actions/diff/action.yaml
2025-05-19 16:24:04 DEBUG action was overridden by another declaration
                      ├ action_id: platform:release
                      ├ old: action.yaml
                      └ new: /home/dferlay/Sources/ski-platform/.compose/build/platform/actions/release/action.yaml
2025-05-19 16:24:04 DEBUG hook PersistentPreRunPlugin
                      └ plugins: [{github.com/launchrctl/keyring.Plugin 0xc00033ecc0}]
2025-05-19 16:24:04 DEBUG starting execution of the action
                      ├ run_env: kubernetes
                      └ action_id: platform:nodes
2025-05-19 16:24:04 DEBUG creating a container for an action
                      ├ action_id: platform:nodes
                      ├ image: platform-actions-nodes:latest
                      ├ command: [python3 -B /action/main.py --servers  --clusters  --work-dir /home/dferlay/Sources/ski-platform --base-dir /home/dferlay/Sources/ski-platform/.compose/build]
                      ├ entrypoint: []
                      └ run_env: kubernetes
2025-05-19 16:24:04 DEBUG creating pod
                      ├ namespace: default
                      └ pod: plasmactl-platform-nodes-zealous-gagarin
2025-05-19 16:24:04 DEBUG waiting for pod to start running
                      ├ pod: plasmactl-platform-nodes-zealous-gagarin
                      └ namespace: default
2025-05-19 16:24:07 DEBUG pod is running
                      ├ namespace: default
                      └ pod: plasmactl-platform-nodes-zealous-gagarin
2025-05-19 16:24:07 DEBUG successfully created a container for an action
                      ├ action_id: platform:nodes
                      ├ image: platform-actions-nodes:latest
                      ├ command: [python3 -B /action/main.py --servers  --clusters  --work-dir /home/dferlay/Sources/ski-platform --base-dir /home/dferlay/Sources/ski-platform/.compose/build]
                      ├ entrypoint: []
                      ├ container_id: default/plasmactl-platform-nodes-zealous-gagarin/plasmactl-platform-nodes-zealous-gagarin
                      └ run_env: kubernetes
 INFO  Running in the remote environment. Copying the working directory and action directory inside the container.
2025-05-19 16:25:36 DEBUG starting container
                      ├ command: [python3 -B /action/main.py --servers  --clusters  --work-dir /home/dferlay/Sources/ski-platform --base-dir /home/dferlay/Sources/ski-platform/.compose/build]
                      ├ entrypoint: []
                      ├ container_id: default/plasmactl-platform-nodes-zealous-gagarin/plasmactl-platform-nodes-zealous-gagarin
                      ├ run_env: kubernetes
                      ├ action_id: platform:nodes
                      └ image: platform-actions-nodes:latest
2025-05-19 16:25:36 DEBUG failed starting the container
                      ├ container_id: default/plasmactl-platform-nodes-zealous-gagarin/plasmactl-platform-nodes-zealous-gagarin
                      ├ run_env: kubernetes
                      ├ action_id: platform:nodes
                      ├ image: platform-actions-nodes:latest
                      ├ command: [python3 -B /action/main.py --servers  --clusters  --work-dir /home/dferlay/Sources/ski-platform --base-dir /home/dferlay/Sources/ski-platform/.compose/build]
                      └ entrypoint: []
2025-05-19 16:25:36 DEBUG execution error
                      └ err: failed to patch ephemeral container to pod: the server could not find the requested resource
  ERROR   failed to patch ephemeral container to pod: the server could not find the requested resource
2025-05-19 16:25:36 DEBUG shutdown cleanup 
  • It looks like the image defined in action is not part of pods containers, most likely because action fails:
➜ k3s kubectl get pod/plasmactl-platform-var-get-funny-meninsky -o jsonpath="{.spec.initContainers[*].name} {.spec.containers[*].name}"         
 supervisor%

Also had another kind of issue with another action:

➜ plx image-build sub --keyring-passphrase-file /tmp/kp --firstoption 111 --secondoption 
panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1aac555]

goroutine 1 [running]:
github.com/launchrctl/launchr/pkg/action.(*runtimeContainer).Close(0x0?)
	github.com/launchrctl/launchr@v0.20.3-0.20250519141527-da8481d0a54f/pkg/action/runtime.container.go:356 +0x15
github.com/launchrctl/launchr/pkg/action.(*Action).Execute(0xc00013a8c0, {
0x2cfe970, 0x41f3a80})
	github.com/launchrctl/launchr@v0.20.3-0.20250519141527-da8481d0a54f/pkg/action/action.go:337 +0xb8
github.com/launchrctl/launchr/plugins/actionscobra.CobraImpl.func2(0xc000142908?, {0xc000116600?, 0x1?, 0x6?})
	github.com/launchrctl/launchr@v0.20.3-0.20250519141527-da8481d0a54f/plugins/actionscobra/cobra.go:57 +0x2f
github.com/spf13/cobra.(*Command).execute(0xc000142908, {0xc0001165a0, 0x6, 0x6})
	github.com/spf13/cobra@v1.9.1/command.go:1015 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000142308)
	github.com/spf13/cobra@v1.9.1/command.go:1148 +0x46f
github.com/spf13/cobra.(*Command).Execute(0xc000341500?)
	github.com/spf13/cobra@v1.9.1/command.go:1071 +0x13
github.com/launchrctl/launchr.(*appImpl).exec(0xc000019c20)
	github.com/launchrctl/launchr@v0.20.3-0.20250519141527-da8481d0a54f/app.go:173 +0x36b
github.com/launchrctl/launchr.(*appImpl).Execute(0xc000019c20)
	github.com/launchrctl/launchr@v0.20.3-0.20250519141527-da8481d0a54f/app.go:194 +0x11d
github.com/launchrctl/launchr.Run(...)
	github.com/launchrctl/launchr@v0.20.3-0.20250519141527-da8481d0a54f/app.go:217
github.com/launchrctl/launchr.RunAndExit()
	github.com/launchrctl/launchr@v0.20.3-0.20250519141527-da8481d0a54f/app.go:222 +0x1f
main.main()
	plasmactl/main.go:9 +0xf

here are action files to reproduce: test-action.tar.gz

@davidferlay
Copy link
Contributor

Gonna start testing for regression to get PR merged soon

@davidferlay
Copy link
Contributor

davidferlay commented May 21, 2025

Found some significant issues:

  • 1 - In some cases, terminal output becomes messed-up for current command as well as any following command:

image

It seems to be reproduced everytime command fails or requires user input:

➜  ski-platform git:(master) ll
total 93M
drwxr-xr-x  1 dferlay dferlay  588 May 21 11:30 .
drwxr-xr-x. 1 dferlay dferlay 4.4K May 19 12:16 ..
-rw-r--r--  1 dferlay dferlay  970 Nov  4  2024 ansible.cfg
lrwxrwxrwx  1 dferlay dferlay    1 Aug  8  2024 ansible_collections -> .
drwxr-xr-x  1 dferlay dferlay  196 Apr 22 16:05 cognition
drwxr-xr-x  1 dferlay dferlay   26 May 21 11:31 .compose
drwxr-xr-x  1 dferlay dferlay   54 Feb 25 11:20 conversation
-rw-r--r--  1 dferlay dferlay    0 Nov 27 16:35 file.diff
-rw-r--r--  1 dferlay dferlay    0 Nov 22 17:34 file_from_domain
-rw-r--r--  1 dferlay dferlay  76K Aug  8  2024 flow_map.svg
drwxr-xr-x  1 dferlay dferlay  108 Apr 22 16:05 foundation
drwxr-xr-x  1 dferlay dferlay  226 May 21 14:31 .git
-rw-r--r--  1 dferlay dferlay  233 Apr 22 16:05 .gitignore
drwxr-xr-x  1 dferlay dferlay   30 Jun 27  2024 .gitlab
-rw-r--r--  1 dferlay dferlay 8.1K Apr 15 16:06 .gitlab-ci.yml
-rw-r--r--  1 dferlay dferlay    0 Aug  8  2024 .gitmodules
drwxr-xr-x  1 dferlay dferlay  158 Apr 22 16:05 integration
drwxr-xr-x  1 dferlay dferlay  110 Apr 22 16:05 interaction
drwxr-xr-x  1 dferlay dferlay   22 Aug  8  2024 library
-rw-r--r--  1 dferlay dferlay   17 Aug  8  2024 origin
-rwxr-xr-x  1 dferlay dferlay 3.5M Aug  8  2024 plasma
-rw-r--r--  1 dferlay dferlay 1.8K May 21 11:30 plasma-compose.yaml
drwxr-x---  1 dferlay dferlay   76 May 21 11:27 .plasmactl
-rwxr-xr-x  1 dferlay dferlay  26M Apr 11 15:19 plasmactl
-rwxr-xr-x  1 dferlay dferlay  62M May 20 17:05 plasmactlx
drwxr-xr-x  1 dferlay dferlay  134 Apr 15 16:06 platform
-rw-r--r--  1 dferlay dferlay 117K Nov  8  2024 README.md
-rw-r--r--  1 dferlay dferlay 1.7M Aug  8  2024 README.svg
drwxr-xr-x  1 dferlay dferlay  160 Aug  8  2024 scripts
drwxr-xr-x  1 dferlay dferlay   22 Feb 25 11:21 toolbox
-rw-r--r--  1 dferlay dferlay  58K Jan 26  2024 .welcome.mp3

➜  ski-platform git:(master) plasmactlx platform:vault edit platform/group_vars/platform/vault.yaml --keyring-passphrase-file /tmp/kp
No changes detected in the temp file.

➜  ski-platform git:(master) ll
total 93M
         drwxr-xr-x  1 dferlay dferlay  588 May 21 11:30 .
                                                          drwxr-xr-x. 1 dferlay dferlay 4.4K May 19 12:16 ..
                                                                                                            -rw-r--r--  1 dferlay dferlay  970 Nov  4  2024 ansible.cfg
                                                                                                                                                                       lrwxrwxrwx  1 dferlay dferlay    1 Aug  8  2024 ansible_collections -> .
                                                drwxr-xr-x  1 dferlay dferlay  196 Apr 22 16:05 cognition
                                                                                                         drwxr-xr-x  1 dferlay dferlay   26 May 21 11:31 .compose
                                                                                                                                                                 drwxr-xr-x  1 dferlay dferlay   54 Feb 25 11:20 conversation
                              -rw-r--r--  1 dferlay dferlay    0 Nov 27 16:35 file.diff
                                                                                       -rw-r--r--  1 dferlay dferlay    0 Nov 22 17:34 file_from_domain
                                                                                                                                                       -rw-r--r--  1 dferlay dferlay  76K Aug  8  2024 flow_map.svg
                    drwxr-xr-x  1 dferlay dferlay  108 Apr 22 16:05 foundation
                                                                              drwxr-xr-x  1 dferlay dferlay  226 May 21 14:31 .git
                                                                                                                                  -rw-r--r--  1 dferlay dferlay  233 Apr 22 16:05 .gitignore
                                                                                                                                                                                            drwxr-xr-x  1 dferlay dferlay   30 Jun 27  2024 .gitlab
                                                    -rw-r--r--  1 dferlay dferlay 8.1K Apr 15 16:06 .gitlab-ci.yml
                                                                                                                  -rw-r--r--  1 dferlay dferlay    0 Aug  8  2024 .gitmodules
                                                                                                                                                                             drwxr-xr-x  1 dferlay dferlay  158 Apr 22 16:05 integration
                                         drwxr-xr-x  1 dferlay dferlay  110 Apr 22 16:05 interaction
                                                                                                    drwxr-xr-x  1 dferlay dferlay   22 Aug  8  2024 library
                                                                                                                                                           -rw-r--r--  1 dferlay dferlay   17 Aug  8  2024 origin
                  -rwxr-xr-x  1 dferlay dferlay 3.5M Aug  8  2024 plasma
                                                                        -rw-r--r--  1 dferlay dferlay 1.8K May 21 11:30 plasma-compose.yaml
                                                                                                                                           drwxr-x---  1 dferlay dferlay   76 May 21 11:27 .plasmactl
      -rwxr-xr-x  1 dferlay dferlay  26M Apr 11 15:19 plasmactl
                                                               -rwxr-xr-x  1 dferlay dferlay  62M May 20 17:05 plasmactlx
                                                                                                                         drwxr-xr-x  1 dferlay dferlay  134 Apr 15 16:06 platform
                                                                                                                                                                                 -rw-r--r--  1 dferlay dferlay 117K Nov  8  2024 README.md
                                           -rw-r--r--  1 dferlay dferlay 1.7M Aug  8  2024 README.svg
                                                                                                     drwxr-xr-x  1 dferlay dferlay  160 Aug  8  2024 scripts
                                                                                                                                                            drwxr-xr-x  1 dferlay dferlay   22 Feb 25 11:21 toolbox
                    -rw-r--r--  1 dferlay dferlay  58K Jan 26  2024 .welcome.mp3
➜  ski-platform git:(master) plx platform:release -vvvv
2025-05-21 14:37:41 DEBUG hook DiscoveryPlugin
                      └ plugins: [{github.com/launchrctl/launchr/plugins/builder.Plugin 0xc000693760} {github.com/launchrctl/web.Plugin 0xc00037d400} {github.com/launchrctl/keyring.Plugin 0xc00054c330} {github.com/skilld-labs/plasmactl-release.Plugin 0x41f3a80} {github.com/skilld-labs/plasmactl-bump/v2.Plugin 0xc00037d4c0} {github.com/launchrctl/compose.Plugin 0xc00037d3c0} {github.com/skilld-labs/plasmactl-dependencies.Plugin 0x41f3a80} {github.com/skilld-labs/plasmactl-update.Plugin 0xc000693820} {github.com/skilld-labs/plasmactl-meta.Plugin 0xc00054c5a0} {github.com/launchrctl/launchr/plugins/yamldiscovery.Plugin 0xc00037d280}]
2025-05-21 14:37:41 DEBUG init success
                      ├ wd: /home/dferlay/Sources/ski-platform
                      └ actions_dir: /home/dferlay/Sources/ski-platform
2025-05-21 14:37:41 DEBUG executing command 
2025-05-21 14:37:41 DEBUG hook CobraPlugin
                      └ plugins: [{github.com/launchrctl/keyring.Plugin 0xc00054c330} {github.com/launchrctl/launchr/plugins/actionscobra.Plugin 0xc00054c270}]
2025-05-21 14:37:41 DEBUG hook PersistentPreRunPlugin
                      └ plugins: [{github.com/launchrctl/keyring.Plugin 0xc00054c330}]
2025-05-21 14:37:41 DEBUG starting execution of the action
                      ├ action_id: platform:release
                      └ run_env: docker
2025-05-21 14:37:41 DEBUG creating a container for an action
                      ├ run_env: docker
                      ├ action_id: platform:release
                      ├ image: platform-actions-release:latest
                      ├ command: [/bin/bash /action/action.sh  false]
                      └ entrypoint: []
2025-05-21 14:37:41 DEBUG image exists locally
                      ├ run_env: docker
                      ├ action_id: platform:release
                      ├ image: platform-actions-release:latest
                      ├ command: [/bin/bash /action/action.sh  false]
                      └ entrypoint: []
2025-05-21 14:37:41 DEBUG successfully created a container for an action
                      ├ run_env: docker
                      ├ action_id: platform:release
                      ├ image: platform-actions-release:latest
                      ├ command: [/bin/bash /action/action.sh  false]
                      ├ entrypoint: []
                      └ container_id: 7bbb5a4ef309d436a0ec28acb2b56293a2666677ef28d592e7701489d8d30b1b
2025-05-21 14:37:41 DEBUG starting container
                      ├ image: platform-actions-release:latest
                      ├ command: [/bin/bash /action/action.sh  false]
                      ├ entrypoint: []
                      ├ container_id: 7bbb5a4ef309d436a0ec28acb2b56293a2666677ef28d592e7701489d8d30b1b
                      ├ run_env: docker
                      └ action_id: platform:release
2025-05-21 14:37:41 DEBUG attaching container streams 
2025-05-21 14:37:41 DEBUG watching run status of container 
2025-05-21 14:37:41 DEBUG watching TTY resize 
2025-05-21 14:37:41 DEBUG waiting execution of the container
                      ├ container_id: 7bbb5a4ef309d436a0ec28acb2b56293a2666677ef28d592e7701489d8d30b1b
                      ├ run_env: docker
                      ├ action_id: platform:release
                      ├ image: platform-actions-release:latest
                      ├ command: [/bin/bash /action/action.sh  false]
                      └ entrypoint: []
/bin/bash: /action/action.sh: No such file or directory
2025-05-21 14:37:41 DEBUG end of stdout/stderr 
                                               2025-05-21 14:37:41 DEBUG received run status code exit_code: 127
                                                                                                                2025-05-21 14:37:41 INFO  action finished with the exit code
                                                                                                                                                                                                  ├ command: [/bin/bash /action/action.sh  false]
                                                                        ├ entrypoint: []
                                                                                                              ├ container_id: 7bbb5a4ef309d436a0ec28acb2b56293a2666677ef28d592e7701489d8d30b1b
                                                                                                                                                                                                                    ├ exit_code: 127
                                                           ├ run_env: docker
                                                                                                  ├ action_id: platform:release
                                                                                                                                                     └ image: platform-actions-release:latest
                                                                                                                                                                                             2025-05-21 14:37:41 DEBUG execution error
                                                             └ err: action "platform:release" finished with exit code 127
                                                                                                                           ERROR   action "platform:release" finished with exit code 127
                                                                                                                                                                                        2025-05-21 14:37:41 DEBUG shutdown cleanup 

@lexbritvin
Copy link
Collaborator Author

lexbritvin commented May 21, 2025

2 - Action from plugin with container runtime always fail

Fixed

@lexbritvin
Copy link
Collaborator Author

Why not have container key inside of launchrctl key for consistency ?

I think we should address that in the next refactorings.
The problem is it's strange to have launchrctl as a key because we can have a custom name for an app. Maybe it's better to call it {my_app_name} or action and have configuration there.

@davidferlay
Copy link
Contributor

Maybe it's better to call it {my_app_name} or action and have configuration there.

I'd rather opt for simplicity and either:

  • move all "core" config at root (so need to update actions_naming)
  • use always same root key everywhere: core maybe ?

In any case, let's not bother with it now I'll create a specific task for it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants