[Feature]: server/config.yml load kubeconfig via data key

[svanzoest]

Problem

Currently when using the kubernetes backend the configuration block expects a reference to a kubeconfig filename.

projects:
- name: main
    backends:
    - type: kubernetes
      kubeconfig:
        filename: ~/.kube/config

However, that reference is not something that can be entered via the dstack server backend interface running in a docker container.

Solution

Noted in the code https://github.com/dstackai/dstack/blob/master/src/dstack/_internal/core/backends/kubernetes/models.py#L39-L50 there is a reference to be able to specify a data key instead of filename. It is unclear to me if this already supported, but if it is then an example on that would be great. I am assuming it would work like this, but not clear if that is right.

projects:
- name: main
    backends:
    - type: kubernetes
      kubeconfig:
        data:
         apiVersion: v1
         clusters:
           - cluster:
               certificate-authority-data: <redacted>
               server: https://cluster.k8s.io
             name: kubernetes
         contexts:
          - context:
               cluster: kubernetes
               namespace: development
               user: dstack-server
            name: dstack-server-context
         current-context: dstack-server-context
         kind: Config
         users:
         - name: dstack-server
           user:
            token: <redacted>
      proxy_jump:
        hostname: 204.12.171.137
        port: 32000

It looks like this is somewhat described in https://dstack.ai/docs/reference/server/config.yml/#kubernetes-kubeconfig although that assumes you have compact raw JSON blob of the config, while generally you deal with yaml files.

cat my-service-account-file.json | jq -c | jq -R

Workaround

don't use the dstack server UI or if the data block is supported, use something similar to the following to convert the yaml.

kubectl config --kubeconfig ~/.kube/config  view --raw=true -o json |  jq -c | jq -R

Would you like to help us implement this feature by sending a PR?

Yes

[r4victor]

@svanzoest, data does work and is expected to be a string with kubeconfig contents. So putting kubeconfig content directly under data won't do, you need to convert it to a string as you described in the workaround. The json example was copy-pasted from another backend – we'll fix that. For kubeconfig, I'd do yq -o=json ~/.kube/config | jq -c | jq -R and paste the output under data.

[svanzoest]

@r4victor thank you for your feedback.

It would be nice if you could include the yaml directly in the yaml, but that is more of a usability thing for you to consider not anything that would limit the functionality.

[peterschmidt85]

As far as I understand, it's already possible to pass a YAML as a string:

projects:
  - name: main
    backends:
      - type: kubernetes
        kubeconfig:
          data: |
            apiVersion: v1
            clusters:
              - cluster:
                  certificate-authority-data: <redacted>
                  server: https://cluster.k8s.io
                name: kubernetes
            contexts:
              - context:
                  cluster: kubernetes
                  namespace: development
                  user: dstack-server
                name: dstack-server-context
            current-context: dstack-server-context
            kind: Config
            users:
              - name: dstack-server
                user:
                  token: <redacted>
        proxy_jump:
          hostname: 204.12.171.137
          port: 32000

@svanzoest could you please confirm if this works for you already?

[svanzoest]

@peterschmidt85 adding the pipe makes this work! Thank you!

uv run dstack server
╱╱╭╮╱╱╭╮╱╱╱╱╱╱╭╮
╱╱┃┃╱╭╯╰╮╱╱╱╱╱┃┃
╭━╯┣━┻╮╭╋━━┳━━┫┃╭╮
┃╭╮┃━━┫┃┃╭╮┃╭━┫╰╯╯
┃╰╯┣━━┃╰┫╭╮┃╰━┫╭╮╮
╰━━┻━━┻━┻╯╰┻━━┻╯╰╯
╭━━┳━━┳━┳╮╭┳━━┳━╮
┃━━┫┃━┫╭┫╰╯┃┃━┫╭╯
┣━━┃┃━┫┃╰╮╭┫┃━┫┃
╰━━┻━━┻╯╱╰╯╰━━┻╯

[09:28:03] INFO     Applying ~/.dstack/server/config.yml...                     
[09:28:04] WARNING  dstack._internal.server.services.config:173 Failed to       
                    configure backend kubernetes: Invalid credentials           
[09:28:06] INFO     dstack._internal.server.services.plugins:77 Found not       
                    enabled builtin plugin rest_plugin. Plugin will not be      
                    loaded.                                                     
           INFO     dstack._internal.server.app:163 Job network mode:           
                    HOST_WHEN_POSSIBLE (2)                                      
           INFO     The admin token is <redacted>
           INFO     The dstack server (no version) is running at                
                    http://127.0.0.1:3000                                       
[09:29:06] INFO     dstack._internal.server.services.backends:347 Requesting    
                    instance offers from backends: ['kubernetes']               
[09:29:13] INFO     dstack._internal.server.services.backends:347 Requesting    
                    instance offers from backends: ['kubernetes']               
[09:29:15] INFO     dstack._internal.server.background.tasks.process_submitted_j
                    obs:365 job(976a12)dstack-gpu-cpu-test-0-0: now is          
                    provisioning a new instance                                 
           INFO     dstack._internal.server.background.tasks.process_submitted_j
                    obs:396 The job dstack-gpu-cpu-test-0-0 created the new     
                    instance lazy-wombat-0                                      
           INFO     dstack._internal.server.background.tasks.process_runs:367   
                    run(3bccde)dstack-gpu-cpu-test: run status has changed      
                    SUBMITTED -> PROVISIONING                                   
           INFO     dstack._internal.server.background.tasks.process_runs:376   
                    run(3bccde)dstack-gpu-cpu-test: run took 6.24 seconds from  
                    submission to provisioning.                                 
[09:30:03] INFO     dstack._internal.server.background.tasks.process_runs:367 run(3bccde)dstack-gpu-cpu-test: run status has changed PROVISIONING
                    -> RUNNING                                                                                                                   
[09:30:14] INFO     dstack._internal.server.background.tasks.process_running_jobs:762 job(976a12)dstack-gpu-cpu-test-0-0: now is TERMINATING     
[09:30:16] INFO     dstack._internal.server.services.jobs:324 job(976a12)dstack-gpu-cpu-test-0-0: instance 'lazy-wombat-0' has been released, new
                    status is TERMINATING                                                                                                        
           INFO     dstack._internal.server.services.services:270 job(976a12)dstack-gpu-cpu-test-0-0: service replica unregistered from receiving
                    requests, gateway=False                                                                                                      
           INFO     dstack._internal.server.services.jobs:378 job(976a12)dstack-gpu-cpu-test-0-0: job status is DONE, reason: DONE_BY_RUNNER     
[09:30:17] INFO     dstack._internal.server.background.tasks.process_runs:367 run(3bccde)dstack-gpu-cpu-test: run status has changed RUNNING ->  
                    TERMINATING                                                                                                                  
[09:30:22] INFO     dstack._internal.server.services.runs:1172 run(3bccde)dstack-gpu-cpu-test: run status has changed TERMINATING -> DONE,       
                    reason: ALL_JOBS_DONE                                                                                                        
[09:30:24] INFO     dstack._internal.server.background.tasks.process_instances:967 Instance lazy-wombat-0 terminated                             
[09:30:29] INFO     dstack._internal.server.background.tasks.process_fleets:258 Automatic cleanup of an empty fleet lazy-wombat                  
           INFO     dstack._internal.server.background.tasks.process_fleets:261 Fleet lazy-wombat deleted                                        

I still get this warning

WARNING  dstack._internal.server.services.config:173 Failed to configure backend kubernetes: Invalid credentials   

But is more likely related to #3168 where I am likely not including the needed rbac permissions as part of the service account, because I don't know what is missing.

The | pipe was the key detail I was missing. It would be great if could include that example in the documentation.

[peterschmidt85]

Perfect, @svanzoest! Then if its OK we can close this issue? For other things, we can create a separate issue! ok?