Azure cloud-config misses tags value

[aslafy-z]

RKE version: v1.2.9

Docker version: (docker version,docker info preferred) N/A

Operating system and kernel: (cat /etc/os-release, uname -r preferred) N/A

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) Azure

cluster.yml file: N/A

Steps to Reproduce: N/A

Results: N/A

tags allows to specify custom tags to be added to all the resources created by the cloud provider integration.

See https://kubernetes-sigs.github.io/cloud-provider-azure/install/configs/#cluster-config
https://kubernetes-sigs.github.io/cloud-provider-azure/topics/tagging-resources/

[davidnuzik]

@sowmyav27 is this ready to test? I don't see any labels on this issue.

[jiaqiluo]

Root cause

The upstream Azure cloud-config supports the option tags, but REK does not have it in the internal AzureCloudProvider struct, so the value is ignored.

What was fixed, or what changes have occurred

add it to RKE's internal data structure

Areas or cases that should be tested

When creating a cluster with Azure as the cloud provider, we can now set the tags ( see RKE Docs)

And according to the upstream docs, the tags will be added to the cloud provider-managed resources, including lb, public IP, network security group, and route table.

What areas could experience regressions?

Creating a cluster with Azure as the cloud provider with or without setting tags should work.

[jiaqiluo]

Waiting for the next RC - v1.4.0-rc2

[jiaqiluo]

Can be validated in v1.4.0-rc3

[thaneunsoo]

Test Environment:

RKE version: v1.4.0-rc3
Cloud Provider: Azure

---

Testing:

Tested the issue with the following steps:

1. Add tags to cloud_provider cloud-config in cluster.yml file
2. Run rke up --config cluster.yml

Tested the following testcases:

• rke up with no cloud_provider configs
• rke up with basic cloud_provider configs
• rke up with tags` cloud_provider config
• Provision azure cluster on Rancher UI and grab config values to put into cluster.yml

Result
I seem to be able to provision when there are no cloud_provider configs and within the Rancher UI, but when I try to create a cluster with the cloud_provider configs added into the cluster.yml, even with the correct values (based on Rancher documentation AND from pulling values from active rke1 azure cluster from Rancher UI), I seem to always hit a health check failure on kubelet towards the end of the rke up command.

FATA[0199] [workerPlane] Failed to bring up Worker Plane: [Failed to verify healthcheck: Failed to check http://localhost:10248/healthz for service [kubelet] on host [ip_address]: Get "http://localhost:10248/healthz": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: ERROR: Please run 'az login' to setup account.] 

[jiaqiluo]

I confirmed with @thaneunsoo that the same test also failed when using RKE 1.3.15, which means it is an existing bug if it is.

[jiaqiluo]

Move this back to to-test because it is not a bug. I get it working with the following cluster.yml + rke version v1.4.0-rc3

nodes:
  - address: <redacted>
    internal_address: <redacted>
    hostname_override: <redacted> 
    user: ubuntu
    role: [controlplane,worker,etcd]
    ssh_key_path: <redacted>
cloud_provider:
  azureCloudProvider:
    aadClientId: <redacted>
    aadClientSecret: <redacted>
    subscriptionId: <redacted>
    tenantId: <redacted>
    tags: "foo=bar,bar=foo"
    securityGroupResourceGroup: <redacted>
    loadBalancerSku: standard
  name: azure

Something to call out here:

• the hostname_override must match the VM name in Azure
• the user must match what you set when creating the VM in Azure
• the securityGroupResourceGroup must match the resource group of the network security group used for the VM
• the loadBalancerSku must be standard to make it work at least in my test

We can start our testing after the cluster is provisioned. We will create a workload and LB in the cluster.
I import the cluster into Rancher, so I can do it via Rancher UI.
Please refer to the regular tests for the load balancer for steps.

Eventually, I can see the LB is created in Azure with the tags

[thaneunsoo]

Test Environment:

RKE version: v1.4.0-rc3
Cloud Provider: Azure

---

Testing:

Tested the issue with the following steps:

1. Add tags to cloud_provider cloud-config in cluster.yml file
2. Run rke up --config cluster.yml

Result
Tags have been added successfully