[Kubernetes] - missing architecture for amd64 jobs

[angelnu]

I am running a drone pipeline in a cluster with arm and amd64 nodes. The arm job has the right nodeSelector

        "restartPolicy": "Never",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "nodeSelector": {
          "beta.kubernetes.io/arch": "arm"
        },

but the nodeSelector is missing on the amd64 job. This results on the sometimes job (and its pod) being scheduled in the arm node. Even worse: the steps are then executed in the wrong architecture resulting on invalid docker containers.

I run the same .drone.yml successfully with the cloud.drone.io: https://github.com/angelnu/docker-ioBroker

Here the job in the amd64 node:

{
  "kind": "Job",
  "apiVersion": "batch/v1",
  "metadata": {
    "name": "drone-job-85-rgcckorgduk5sf25h",
    "generateName": "drone-job-85-rgcckorgduk5",
    "namespace": "drone",
    "selfLink": "/apis/batch/v1/namespaces/drone/jobs/drone-job-85-rgcckorgduk5sf25h",
    "uid": "1e6f835a-6502-11e9-b835-001e0630fab9",
    "resourceVersion": "2363029",
    "creationTimestamp": "2019-04-22T13:25:38Z",
    "labels": {
      "controller-uid": "1e6f835a-6502-11e9-b835-001e0630fab9",
      "job-name": "drone-job-85-rgcckorgduk5sf25h"
    },
    "annotations": {
      "io.drone": "true",
      "io.drone.build.id": "22",
      "io.drone.repo.id": "5",
      "io.drone.stage.arch": "amd64",
      "io.drone.stage.created": "2019-04-22 13:25:38 +0000 UTC",
      "io.drone.stage.id": "85",
      "io.drone.stage.number": "1",
      "io.drone.stage.os": "linux",
      "io.drone.stage.scheduled": "2019-04-22 13:25:38.649085112 +0000 UTC m=+3816.546213546"
    }
  },
  "spec": {
    "parallelism": 1,
    "completions": 1,
    "backoffLimit": 6,
    "selector": {
      "matchLabels": {
        "controller-uid": "1e6f835a-6502-11e9-b835-001e0630fab9"
      }
    },
    "template": {
      "metadata": {
        "creationTimestamp": null,
        "labels": {
          "controller-uid": "1e6f835a-6502-11e9-b835-001e0630fab9",
          "job-name": "drone-job-85-rgcckorgduk5sf25h"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "drone-controller",
            "image": "drone/controller:1.0.0",
            "env": [
              {
                "name": "DRONE_RPC_SECRET",
                "value": "xxxxxx"
              },
              {
                "name": "DRONE_SECRET_SKIP_VERIFY",
                "value": "false"
              },
              {
                "name": "DRONE_STAGE_ID",
                "value": "85"
              },
              {
                "name": "DRONE_RPC_HOST",
                "value": "drone.angelnu.com"
              },
              {
                "name": "DRONE_LIMIT_CPU",
                "value": "0"
              },
              {
                "name": "DRONE_LOGS_DEBUG",
                "value": "false"
              },
              {
                "name": "DRONE_LOGS_PRETTY",
                "value": "false"
              },
              {
                "name": "DRONE_REGISTRY_SKIP_VERIFY",
                "value": "false"
              },
              {
                "name": "DRONE_LIMIT_MEM",
                "value": "0"
              },
              {
                "name": "DRONE_LOGS_TRACE",
                "value": "false"
              },
              {
                "name": "DRONE_LOGS_TEXT",
                "value": "false"
              },
              {
                "name": "DRONE_RPC_PROTO",
                "value": "https"
              },
              {
                "name": "DRONE_RPC_DEBUG",
                "value": "false"
              },
              {
                "name": "KUBERNETES_NODE",
                "valueFrom": {
                  "fieldRef": {
                    "apiVersion": "v1",
                    "fieldPath": "spec.nodeName"
                  }
                }
              },
              {
                "name": "DRONE_RUNNER_NAME",
                "valueFrom": {
                  "fieldRef": {
                    "apiVersion": "v1",
                    "fieldPath": "spec.nodeName"
                  }
                }
              }
            ],
            "resources": {},
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "Always"
          }
        ],
        "restartPolicy": "Never",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {},
        "schedulerName": "default-scheduler"
      }
    }
  },
  "status": {
    "startTime": "2019-04-22T13:25:38Z",
    "active": 1
  }
}

The main drone server is deployed to a amd64 node - I am going to check how it looks like when I deploy it to the arm node...

Update: same result after deploying drone/drone:1 to arm. The amd64 is still deployed without a nodeSelector annotation.

[ahmedmagdiosman]

Since my issue ( harness/harness#2816 )was closed and this issue has been open for a while. Any ideas what's causing this? I've checked and nodeSelector is not set for amd64 jobs.

[ahmedmagdiosman]

I understand that all the contributors might be busy with other stuff, but I'd appreciate some pointers to where the nodeSelector label gets set. It should be an easy fix, I'll be happy to submit a pull request.

Sidenote: I am aware of the upcoming Kubernetes run-time changes, and I'm excited for the improvements. But I'd like to continue using Drone on Kubernetes until the new run-time/project is ready since it already works great in my environment.

[ahmedmagdiosman]

In case someone stumbles upon this, I've found a workaround using a node block which is passed using the nodeSelector syntax.

Example:

  kind: pipeline
  name: amd64

  platform:
    os: linux
    arch: amd64

+ node:
+   kubernetes.io/arch: amd64
  
  steps:
  ...

[bradrydzewski]

fixed by drone-runners/drone-runner-kube@971c183

new kubernetes runner documentation can be found here:
https://kube-runner.docs.drone.io/