Provision rke2 cluster from 3rd party node driver #34782

danbaie · 2021-09-17T10:04:58Z

Rancher Server Setup

Rancher version: 2.6.0
Installation option (Docker install/Helm Chart): Helm Chart
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): v1.21.4+rke2r3
Proxy/Cert Details: lets-encrypt

Information about the Cluster

Kubernetes version: v1.21.4+rke2r3
Cloud provider: Hetzner Cloud

Describe the bug

Cannot create RKE2 cluster from RancherUI because I can't create cloud credentials for hetzner node driver.

To Reproduce

Add Hetzner node driver to Rancher via UI
Go to Cluster Management -> Cluster -> Create
Get prompted a form where I can create cloud credentials
Add information and click on create
Get prompted a dropdown list where I can select the cloud provider to use
Dropdown list is empty

I tried to add cloud credentials from the cloud credentials menu but there is no option for hetzner or custom or similar.
I tried to activate some of the built-in node drivers in rancher and some have the same problem but some work.

Result
Not able to provision RKE2 cluster with Hetzner node driver because of cloud credentials

Expected Result

I should be able to create cloud credentials in some way. When activating a node driver, the cloud provider should be added to the list of creatable cloud credentials.

Screenshots

Additional context

Negashev · 2021-10-12T20:22:29Z

It's fix UI bug

rancher/tasks#20 (comment)

but next problem on rancher with nodedriver management:

in rancher logs: cannot find a suitable driver, I think the problem is that it searches by name and not by ID

upd:
add annotation to your /v3/nodeDrivers/nd-xxxx (hetzner driver)

annotations": {
"privateCredentialFields": "apiToken"
}

deactivate and activate your hetzner driver -> it's "rebuild" your /v3/schemas/hetznercredentialconfig and add apiToken to resourceFields

Negashev · 2021-10-21T04:58:12Z

Fix on #33392 or #34914?

Negashev · 2021-10-26T16:21:47Z

Add rancher docker with name by CRD in local cluster (with rancher server)

apiVersion: management.cattle.io/v3
kind: NodeDriver
metadata:
  annotations:
    lifecycle.cattle.io/create.node-driver-controller: "true"
    privateCredentialFields: apiToken
  name: hetzner # <- fix my generated ID on rancher
spec:
  active: false
  addCloudCredential: false
  builtin: false
  checksum: ""
  description: ""
  displayName: hetzner
  externalId: ""
  uiUrl: https://storage.googleapis.com/hcloud-rancher-v2-ui-driver/component.js
  url: https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/download/3.2.0/docker-machine-driver-hetzner_3.2.0_linux_amd64.tar.gz
  whitelistDomains:
  - storage.googleapis.com

Now, Ican create/delete node on cloud (hetzner and another custom node driver)

But now i have problem with UserData (cloud-init)
https://github.com/rancher/machine/blob/e51aa220eacad5bd89cfcd05cab620c2131338b9/commands/create.go#L233

Because not any docker-machine drivers work correctly with cloud-init

Negashev · 2021-10-26T16:55:26Z

Hm.....

When I copy node driver job, and start it with --debug key on rancher-machine

user data file replaced with /tmp/modified-user-data557062213

and ssh to my hetzner VM, i find in cat /var/lib/cloud/instances/15488979/user-data.txt

/tmp/modified-user-data557062213

not content like

#cloud-config
runcmd:
- sh /usr/local/custom_script/install.sh
write_files:
- content: H4sIAAAAAAAA/wAALONG-LONG-LONG-LONG-AP//1Hx/d9u2DAAP//K+6J769cAAA=
  encoding: gzip+b64
  path: /usr/local/custom_script/install.sh
  permissions: "0644"

something wrong on https://github.com/rancher/machine/blob/e51aa220eacad5bd89cfcd05cab620c2131338b9/commands/create.go#L233

Fix rancher/rancher#34782

phal0r · 2021-11-14T19:58:09Z

@Negashev
Can you tell, when your fix will be merged? I can provision VMs with your workaround aswell. But initializing rancher gets stuck. The last log entry is Custom install script was sent via userdata, provisioning complete.... So I guess, this is the same problem.

Negashev · 2021-11-15T03:48:11Z

@phal0r
I don’t know when it will be merged, it’s out of my control

stale · 2022-01-14T04:43:43Z

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

Negashev · 2022-01-14T11:39:18Z

/upd )

stale · 2022-03-16T02:28:56Z

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

apo2k7 · 2022-03-24T09:42:58Z

Are there any news to this? I ran into the exact problem. @Negashev did you find a workaround if this dosn't get fixed?

Negashev · 2022-05-09T03:16:32Z

@apo2k7
I have not yet found the right solution, in the pull request, they offer me a fix, but I have not tested it) rancher/machine#153

Work with yandex

apiVersion: management.cattle.io/v3
kind: NodeDriver
metadata:
  annotations:
    privateCredentialFields: saKeyFile
  name: yandex  # fix name
spec:
  active: true
  addCloudCredential: false
  builtin: false
  checksum: ""
  description: ""
  displayName: yandex
  externalId: ""
  uiUrl: ""
  url: https://github.com/yandex-cloud/docker-machine-driver-yandex/releases/download/v0.1.35/docker-machine-driver-yandex_0.1.35_linux_amd64.tar.gz

but problem with ENV name in rancher YANDEX_* in docker-driver YC_*

and we can't hide token to YANDEX_TOKEN in rancher because in yandex docker driver use YC_TOKEN

i'm not use sa-key-file and in creadential it empty)

apiVersion: v1
data:
  yandexcredentialConfig-saKeyFile: ""
kind: Secret
metadata:
  annotations:
    field.cattle.io/creatorId: user-xxxxx
    field.cattle.io/name: My-Team
    provisioning.cattle.io/driver: yandex
  labels:
    cattle.io/creator: norman
  name: cc-xxxxx
  namespace: cattle-global-data
type: Opaque

dcardellino · 2022-05-30T12:58:39Z

I also ran in this issue with Hetzner, is there any update on this issue?

github-actions · 2022-07-30T02:17:15Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

phal0r · 2022-07-31T07:39:07Z

still relevant

github-actions · 2022-09-30T02:16:36Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

hoerup · 2022-09-30T06:48:31Z

still relevant

github-actions · 2022-12-01T02:11:57Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

hoerup · 2022-12-01T06:10:39Z

Still relevant

Audifire · 2022-12-05T13:17:02Z

I also ran in this issue with Hetzner, is there any update on this issue?

Same for me. I also use Hetzner and ran into this problem.
How to generate a log output for this? Which params are delivered to docker-machine?

Negashev · 2022-12-06T02:01:38Z

I think we need to add changes on the hetzner side of the docker driver, a key that would read userdata from a file, I'll try to make a pull request today

Audifire · 2022-12-06T09:28:08Z

I think we need to add changes on the hetzner side of the docker driver, a key that would read userdata from a file, I'll try to make a pull request today

Yep, that's what I also found out. I tried to change it. But sadly I have not much experience with Go and I get other errors when importing the driver. 😅

Negashev · 2022-12-07T02:12:59Z

Ohhhh.... next level problem...

2022/12/07 02:03:47 [ERROR] error syncing 'hetzner': handler node-driver-controller: DynamicSchema.management.cattle.io "hetzner_instrumentedconfig" is invalid: metadata.name: Invalid value: "hetzner_instrumentedconfig": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), requeuing

in hetzner driver, instrumented auto added

rancher generate docker-machine-driver-hetzner_instrumented

Negashev · 2022-12-07T02:49:45Z

After 3.9.2 (hetzner docker machine) tar.gz include 2 binary files, it seems the rancher takes the one with the end _instrumented

Negashev · 2022-12-07T08:43:54Z

Okay, FIX for hetzner after 3.10.0 (in 3.9.2 added *_instrumented)

create proxy download (nginx deploy in local cluster) for tar.gz without binary like docker-machine-driver-hetzner_instrumented

deploy that in you local cluster in ns cattle-system

apiVersion: v1
kind: Service
metadata:
  name: docker-machine-driver-hetzner
  namespace: cattle-system
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    docker-machine-driver: hetzner
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    docker-machine-driver: hetzner
  name: docker-machine-driver-hetzner
  namespace: cattle-system
spec:
  replicas: 1
  selector:
    matchLabels:
      docker-machine-driver: hetzner
  template:
    metadata:
      labels:
        docker-machine-driver: hetzner
    spec:
      containers:
      - image: nginx:stable-alpine
        imagePullPolicy: Always
        name: nginx
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: storage
      dnsPolicy: ClusterFirst
      initContainers:
      - args:
        - -c
        - |
          wget -O /tmp/source.tar.gz "https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/download/${VERSION}/docker-machine-driver-hetzner_${VERSION}_linux_amd64.tar.gz"
          cd /tmp/
          tar -zxvf source.tar.gz
          tar -czvf "docker-machine-driver-hetzner_${VERSION}_linux_amd64.tar.gz" docker-machine-driver-hetzner
        command:
        - /bin/sh
        env:
        - name: VERSION
          value: 3.10.0
        image: busybox
        imagePullPolicy: Always
        name: clean-tar
        volumeMounts:
        - mountPath: /tmp
          name: storage
      volumes:
      - emptyDir: {}
        name: storage

create nodedriver in local cluster yaml deploy (check name, displayName and url - it's download from local)

apiVersion: management.cattle.io/v3
kind: NodeDriver
metadata:
  annotations:
    lifecycle.cattle.io/create.node-driver-controller: "true"
    privateCredentialFields: apiToken
  generation: 13
  name: hetzner
spec:
  active: true
  addCloudCredential: false
  builtin: false
  checksum: ""
  description: ""
  displayName: hetzner
  externalId: ""
  uiUrl: https://storage.googleapis.com/hcloud-rancher-v2-ui-driver/component.js
  url: http://docker-machine-driver-hetzner.cattle-system.svc/docker-machine-driver-hetzner_3.10.0_linux_amd64.tar.gz
  whitelistDomains:
  - storage.googleapis.com

go to UI and activate driver
Add credentials for hetzner
create hetzner cluster, and use User data is file in all pools
Enjoy!

simonostendorf · 2022-12-25T22:07:28Z

Getting "[ERROR] error syncing 'fleet-default/test-pool1-b0030872-r5rgz': handler machine-provision: nodedrivers.management.cattle.io "hetzner" not found, requeuing" when using your instructions @Negashev .
Any idea why third party node drivers (hetzner) not working for me?

Negashev · 2022-12-26T02:23:51Z

@simonostendorf
It's important to add nodedriver in local cluster as yaml deploy with metadata/name: hetzner, not in rancher nodedriver UI

simonostendorf · 2022-12-26T14:23:03Z

@simonostendorf It's important to add nodedriver in local cluster as yaml deploy with metadata/name: hetzner, not in rancher nodedriver UI

Thank you very much, i thought that if i set the name via GUI it is also possible. This fixed my initial problem, hope there will be no more big problems :D

klauserber · 2023-02-06T06:56:49Z

The Solution does not working for me. It fixes the driver filename successfully and there are no related error messages in the log or on screen anymore. But I still cannot create a credential for Hetzner, the key field is not populated.

The Version 3.10.1 is switching back the driver name, so we should not need this workaround anymore. But this Version is also not working for me to create RKE2 clusters. Btw, the creation of RKE clusters works fine.

Negashev · 2023-02-06T07:12:34Z

@klauserber
I create Cloud Credential, before i create cluster

and annotation "privateCredentialFields" required

metadata:
  annotations:
    privateCredentialFields: apiToken

klauserber · 2023-02-06T08:25:04Z

@Negashev
ok, interesting. That solves it. The Hetzner NodeDriver CR has the privateCredentialFields annotation.

I already have tried to create a credential as a Secret like so:

apiVersion: v1
kind: Secret
metadata:
  name: cc-test
  namespace: cattle-global-data
  labels:
    cattle.io/creator: norman
  annotations:
    field.cattle.io/creatorId: user-r8flk
    field.cattle.io/name: hetzner
    provisioning.cattle.io/driver: hetzner
type: Opaque
stringData:
  apiToken: XXXX

But this secret does not appear in the UI. After that I can add Hetzner Cloud Credentials. The resulting secrets have a Field called hetznercredentialConfig-apiToken an the provisioning.cattle.io/driver: hetzner is obviously not needed.

apiVersion: v1
kind: Secret
metadata:
  name: cc-lab2-2
  namespace: cattle-global-data
  labels:
    cattle.io/creator: norman
  annotations:
    field.cattle.io/creatorId: user-r8flk
    field.cattle.io/name: hetzner-lab2-2
type: Opaque
stringData:
  hetznercredentialConfig-apiToken: XXXX

Now I have the Hetzner credentials type and I can create them and get into the UI.

Don't understand it completely, but it works. Thank you.

github-actions · 2023-04-08T03:57:41Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

ItsReddi · 2023-04-28T06:15:46Z

We already have the nodedriver added via UI and also running an rke1 cluster with it.
Is it safe to delete it from UI and deploy it like @Negashev ?
Or would it break the already running cluster...

More than that. I do not not fully understand the issue.
Where do we need fix, . Driver? Rancher? UI-Driver? So anything is working "normally" via rancher UI.

Its a issue with the node Driver installation correct? So what must be fixed in the node driver..

Negashev · 2023-04-28T08:19:09Z

@ItsReddi

I don't remember exactly, it's better to test removing the driver on a test environment)

yes, it is still important to add the driver via yaml in the local cluster

if you want a cluster on rke2 and a hetzner driver, you will have to do installation manipulations through the local cluster

Negashev added a commit to Negashev/machine that referenced this issue Oct 29, 2021

Fix get user data from tmpfile

6e8f920

Fix rancher/rancher#34782

Negashev mentioned this issue Oct 29, 2021

Fix get user data from tmpfile rancher/machine#153

Closed

stale bot added the status/stale label Jan 14, 2022

stale bot removed the status/stale label Jan 14, 2022

stale bot added the status/stale label Mar 16, 2022

stale bot removed the status/stale label Mar 24, 2022

github-actions bot added the status/stale label Jul 30, 2022

github-actions bot removed the status/stale label Aug 1, 2022

github-actions bot added the status/stale label Sep 30, 2022

github-actions bot removed the status/stale label Oct 1, 2022

github-actions bot added the status/stale label Dec 1, 2022

github-actions bot removed the status/stale label Dec 2, 2022

Negashev mentioned this issue Dec 6, 2022

Add Cloud-init based User data as file JonasProgrammer/docker-machine-driver-hetzner#93

Merged

simonostendorf mentioned this issue Dec 26, 2022

I have latest Rancher v2.6.5 and try to install RKE2 and I can not bypass new credential screen mxschmitt/ui-driver-hetzner#132

Open

klauserber mentioned this issue Feb 8, 2023

Cannot create a RKE2 or K3S cluster mxschmitt/ui-driver-hetzner#137

Open

github-actions bot added the status/stale label Apr 8, 2023

github-actions bot closed this as completed Apr 22, 2023

derlinuxer mentioned this issue May 19, 2023

Can't get past credential creation with Rancher 2.7.3 and docker-machine-driver 3.13.0 mxschmitt/ui-driver-hetzner#138

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provision rke2 cluster from 3rd party node driver #34782

Provision rke2 cluster from 3rd party node driver #34782

danbaie commented Sep 17, 2021

Negashev commented Oct 12, 2021 •

edited

Loading

Negashev commented Oct 21, 2021 •

edited

Loading

Negashev commented Oct 26, 2021

Negashev commented Oct 26, 2021

phal0r commented Nov 14, 2021

Negashev commented Nov 15, 2021

stale bot commented Jan 14, 2022

Negashev commented Jan 14, 2022

stale bot commented Mar 16, 2022

apo2k7 commented Mar 24, 2022

Negashev commented May 9, 2022

dcardellino commented May 30, 2022

github-actions bot commented Jul 30, 2022

phal0r commented Jul 31, 2022

github-actions bot commented Sep 30, 2022

hoerup commented Sep 30, 2022

github-actions bot commented Dec 1, 2022

hoerup commented Dec 1, 2022

Audifire commented Dec 5, 2022

Negashev commented Dec 6, 2022

Audifire commented Dec 6, 2022

Negashev commented Dec 7, 2022 •

edited

Loading

Negashev commented Dec 7, 2022

Negashev commented Dec 7, 2022 •

edited

Loading

simonostendorf commented Dec 25, 2022

Negashev commented Dec 26, 2022

simonostendorf commented Dec 26, 2022

klauserber commented Feb 6, 2023

Negashev commented Feb 6, 2023 •

edited

Loading

klauserber commented Feb 6, 2023

github-actions bot commented Apr 8, 2023

ItsReddi commented Apr 28, 2023 •

edited

Loading

Negashev commented Apr 28, 2023 •

edited

Loading

Provision rke2 cluster from 3rd party node driver #34782

Provision rke2 cluster from 3rd party node driver #34782

Comments

danbaie commented Sep 17, 2021

Negashev commented Oct 12, 2021 • edited Loading

Negashev commented Oct 21, 2021 • edited Loading

Negashev commented Oct 26, 2021

Negashev commented Oct 26, 2021

phal0r commented Nov 14, 2021

Negashev commented Nov 15, 2021

stale bot commented Jan 14, 2022

Negashev commented Jan 14, 2022

stale bot commented Mar 16, 2022

apo2k7 commented Mar 24, 2022

Negashev commented May 9, 2022

dcardellino commented May 30, 2022

github-actions bot commented Jul 30, 2022

phal0r commented Jul 31, 2022

github-actions bot commented Sep 30, 2022

hoerup commented Sep 30, 2022

github-actions bot commented Dec 1, 2022

hoerup commented Dec 1, 2022

Audifire commented Dec 5, 2022

Negashev commented Dec 6, 2022

Audifire commented Dec 6, 2022

Negashev commented Dec 7, 2022 • edited Loading

Negashev commented Dec 7, 2022

Negashev commented Dec 7, 2022 • edited Loading

simonostendorf commented Dec 25, 2022

Negashev commented Dec 26, 2022

simonostendorf commented Dec 26, 2022

klauserber commented Feb 6, 2023

Negashev commented Feb 6, 2023 • edited Loading

klauserber commented Feb 6, 2023

github-actions bot commented Apr 8, 2023

ItsReddi commented Apr 28, 2023 • edited Loading

Negashev commented Apr 28, 2023 • edited Loading

Negashev commented Oct 12, 2021 •

edited

Loading

Negashev commented Oct 21, 2021 •

edited

Loading

Negashev commented Dec 7, 2022 •

edited

Loading

Negashev commented Dec 7, 2022 •

edited

Loading

Negashev commented Feb 6, 2023 •

edited

Loading

ItsReddi commented Apr 28, 2023 •

edited

Loading

Negashev commented Apr 28, 2023 •

edited

Loading