File sync in dev mode fails with segmentation fault when container is running in vcluster

[gustaff-weldon]

What happened?
devspacehelper fails to start when running devspace dev for project running in vcluster on m1.

When we run devspace dev command devspacehelper fails with segmentation fault when trying to start. The pod and container are running in vcluster running in docker-desktop kubernetes, on m1 host.

We have tested the same devspace dev for a pod which did not run in vcluster, but directly in docker-desktop cluster on the same m1 machine. It worked fine.

When run without debug, failure happens when sync starts (or is about to start):

info Using namespace 'm1-test'
info Using kube context 'loft-vcluster_m1_vcluster-m1-ksnlz_loft-cluster'
info Created namespace: m1-test
deploy:app Deploying chart component-chart (app) with helm...
deploy:app Deployed helm chart (Release revision: 1)
deploy:app Successfully deployed app with helm
dev:app Waiting for pod to become ready...
dev:app DevSpace is waiting, because Pod app-devspace-5fd678bf98-h5gr2 has status: ContainerCreating
dev:app Selected pod app-devspace-5fd678bf98-h5gr2
dev:app sync  Sync started on: ./ <-> ./
dev:app sync  Waiting for initial sync to complete
dev:app sync  Initial sync completed
start_dev: initial sync: Sync - connection lost to pod m1-test/app-devspace-5fd678bf98-h5gr2: qemu: uncaught target signal 11 (Segmentation fault) - core dumped
 command terminated with exit code 139
fatal exit status 1

When run in debug mode:

09:45:08 run_dependencies --all
09:45:08 Marked project excluded: m1-test
09:45:08 create_deployments --all
09:45:08 Deploying 1 deployments concurrently...
09:45:08 deploy:app Skipping deployment app
09:45:08 Deploying 0 deployments concurrently
09:45:08 start_dev app
09:45:08 dev:app DevPod Config:
name: app
imageSelector: username/app
devImage: ghcr.io/loft-sh/devspace-containers/alpine:3
sync:
    - path: ./
terminal:
    command: ./devspace_start.sh
ssh:
    enabled: true
proxyCommands:
    - command: devspace
    - command: kubectl
    - command: helm
    - command: git


09:45:08 dev:app Try to find replaced deployment...
09:45:08 dev:app Replaced pod spec:
metadata:
  annotations:
    devspace.sh/container: container-0
    devspace.sh/imageSelector: username/app
    helm.sh/chart: component-chart-0.8.5
  creationTimestamp: null
  labels:
    app.kubernetes.io/component: app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: devspace-app
    devspace.sh/replaced: "true"
spec:
  containers:
  - command:
    - sleep
    - "1000000000"
    image: ghcr.io/loft-sh/devspace-containers/alpine:3
    imagePullPolicy: Always
    name: container-0
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  terminationGracePeriodSeconds: 5


09:45:08 dev:app Update replaced deployment with patch:
 {"metadata":{"annotations":{"devspace.sh/replicas":"1"}}}
09:45:08 dev:app Waiting for pod to become ready...
09:45:08 dev:app Start selecting a single container with selector image selector: username/app
09:45:09 dev:app Selected pod app-devspace-5fd678bf98-h5gr2
09:45:09 dev:app sync  Start selecting a single container with selector pod name: app-devspace-5fd678bf98-h5gr2
09:45:09 dev:app sync  Starting sync...
09:45:10 dev:app sync  Inject devspacehelper...
09:45:10 dev:app sync  Use local devspacehelper found at /Users/edited/.devspace/devspacehelper/v6.1.1/devspacehelper
09:45:11 dev:app sync  Successfully injected devspacehelper into pod m1-test/app-devspace-5fd678bf98-h5gr2
09:45:11 dev:app sync  Start syncing
09:45:11 dev:app sync  Sync started on: ./ <-> ./
09:45:11 dev:app sync  Waiting for initial sync to complete
09:45:11 dev:app sync  Initial Sync - Retrieve Initial State
09:45:11 dev:app sync  Downstream - Start collecting changes
09:45:11 dev:app sync  Helper - qemu: uncaught target signal 11 (Segmentation fault) - core dumped
09:45:11 dev:app sync  Helper - qemu: uncaught target signal 11 (Segmentation fault) - core dumped
09:45:11 dev:app sync  Sync Error on /Users/edited/edited/monorepo/test-m1: Sync - connection lost to pod m1-test/app-devspace-5fd678bf98-h5gr2: qemu: uncaught target signal 11 (Segmentation fault) - core dumped
 command terminated with exit code 139
09:45:11 dev:app sync  Sync stopped
09:45:11 dev:app sync  Downstream - Done collecting changes
09:45:11 dev:app sync  Initial Sync - Done Retrieving Initial State
09:45:11 dev:app sync  Initial sync completed
09:45:11 dev:app term  Start selecting a single container with selector pod name: app-devspace-5fd678bf98-h5gr2
09:45:11 dev:app Stopped dev app
09:45:11 start_dev: initial sync: Sync - connection lost to pod m1-test/app-devspace-5fd678bf98-h5gr2: qemu: uncaught target signal 11 (Segmentation fault) - core dumped
09:45:11  command terminated with exit code 139
09:45:11 fatal exit status 1

It seems when running in vcluster (that runs on m1 host), the architecture is not detected properly and devspacehelper fails with segfault. (probably x86 version is started instead of arm one)

We have checked the .devspace/devspacehelper/v6.1.1/ it contained 2 files, one with arm.

What did you expect to happen instead?

devspace dev to sync files and open terminal when run in both with and without vcluster on m1 host.

How can we reproduce the bug? (as minimally and precisely as possible)
The reproduction is quite simple, we have used the devspace.yml generated by devspace init and cleaned it a bit, included below.

After you have the devspace.yml:

> loft create vcluster m1
> [devspace use vcluster m1]
> devspace dev

My devspace.yaml:

version: v2beta1
name: m1-test

# This is a list of `pipelines` that DevSpace can execute (you can define your own)
pipelines:
  # This is the pipeline for the main command: `devspace dev` (or `devspace run-pipeline dev`)
  dev:
    run: |-
      run_dependencies --all       # 1. Deploy any projects this project needs (see "dependencies")
      create_deployments --all     # 2. Deploy Helm charts and manifests specfied as "deployments"
      start_dev app                # 3. Start dev mode "app" (see "dev" section)
  # You can run this pipeline via `devspace deploy` (or `devspace run-pipeline deploy`)
  deploy:
    run: |-
      run_dependencies --all                            # 1. Deploy any projects this project needs (see "dependencies")
      build_images --all -t $(git describe --always)    # 2. Build, tag (git commit hash) and push all images (see "images")
      create_deployments --all                          # 3. Deploy Helm charts and manifests specfied as "deployments"

# This is a list of `images` that DevSpace can build for this project
# We recommend to skip image building during development (devspace dev) as much as possible
images:
  app:
    image: username/app
    dockerfile: ./Dockerfile

# This is a list of `deployments` that DevSpace can create for this project
deployments:
  app:
    # This deployment uses `helm` but you can also define `kubectl` deployments or kustomizations
    helm:
      # We are deploying this project with the Helm chart you provided
      chart:
        name: component-chart
        repo: https://charts.devspace.sh
      # Under `values` we can define the values for this Helm chart used during `helm install/upgrade`
      # You may also use `valuesFiles` to load values from files, e.g. valuesFiles: ["values.yaml"]
      values:
        containers:
          - image: username/app

# This is a list of `dev` containers that are based on the containers created by your deployments
dev:
  app:
    # Search for the container that runs this image
    imageSelector: username/app
    # Replace the container image with this dev-optimized image (allows to skip image building during development)
    devImage: ghcr.io/loft-sh/devspace-containers/alpine:3
    # Sync files between the local filesystem and the development container
    sync:
      - path: ./
    # Open a terminal and use the following command to start it
    terminal:
      command: ./devspace_start.sh
    # Inject a lightweight SSH server into the container (so your IDE can connect to the remote dev env)
    ssh:
      enabled: true
    # Make the following commands from my local machine available inside the dev container
    proxyCommands:
      - command: devspace
      - command: kubectl
      - command: helm
      - command: git

vars:
  DEVSPACE_FLAGS: --namespace m1-test

Local Environment:

• DevSpace Version: 6.1.1
• Operating System: mac
• ARCH of the OS: ARM64
  Kubernetes Cluster:
• Cloud Provider: local | docker-desktop
• Kubernetes Version: 1.25

Anything else we need to know?

Result of desribing the devspace pod running in vcluster:

Name:         app-devspace-5fd678bf98-h5gr2
Namespace:    m1-test
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Fri, 23 Sep 2022 09:44:24 +0200
Labels:       app.kubernetes.io/component=app
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=devspace-app
              devspace.sh/replaced=true
              pod-template-hash=5fd678bf98
Annotations:  devspace.sh/container: container-0
              devspace.sh/imageSelector: username/app
              helm.sh/chart: component-chart-0.8.5
Status:       Running
IP:           10.1.0.168
IPs:
  IP:           10.1.0.168
Controlled By:  ReplicaSet/app-devspace-5fd678bf98
Containers:
  container-0:
    Container ID:  docker://c504e6931afba6ec065588a618ad2188236a57765a057b04baaf1c423fc8541f
    Image:         ghcr.io/loft-sh/devspace-containers/alpine:3
    Image ID:      docker-pullable://ghcr.io/loft-sh/devspace-containers/alpine@sha256:6464d4b489d9d611c75e5610683d69b281c5415f9e54fe8ed7715f217698c216
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      1000000000
    State:          Running
      Started:      Fri, 23 Sep 2022 09:44:35 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mqlq8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-mqlq8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  116s  default-scheduler  Successfully assigned m1-test/app-devspace-5fd678bf98-h5gr2 to docker-desktop
  Normal  Pulling    116s  kubelet            Pulling image "ghcr.io/loft-sh/devspace-containers/alpine:3"
  Normal  Pulled     106s  kubelet            Successfully pulled image "ghcr.io/loft-sh/devspace-containers/alpine:3" in 10.108621421s
  Normal  Created    106s  kubelet            Created container container-0
  Normal  Started    106s  kubelet            Started container container-0

/kind bug

[FabianKramm]

@gustaff-weldon thanks for creating this issue! The problem occurs when a amd64 binary is injected into a node that runs arm64. I guess the underlying question is is why qemu is not able to emulate the helper properly. A fix for this problem is to use the setting:

sync:
  nodes:
    enabled: true

during vcluster creation inside Loft. This will sync the real node labels which will show the correct arch label to devspace and DevSpace will then inject the arm helper instead. On the DevSpace side you can also force it to inject the correct helper via the arch option as a workaround:

dev:
  my-dev:
    arch: arm64

[gustaff-weldon]

Thanks for getting back.

@gustaff-weldon thanks for creating this issue! The problem occurs when a amd64 binary is injected into a node that runs arm64.

That confirms my suspicion:

It seems when running in vcluster (that runs on m1 host), the architecture is not detected properly and devspacehelper fails with segfault. (probably x86 version is started instead of arm one)

A fix for this problem is to use the setting during vcluster creation inside Loft:

sync:
  nodes:
    enabled: true

👍 I think in UI this is under Helm Values for Virtual Cluster Chart in Advanced Options

On the DevSpace side you can also force it to inject the correct helper via the arch option as a workaround:

dev:
  my-dev:
    arch: arm64

We cannot force arch, as developers use different architectures on their laptops.