ACM certificate x509: certificate is valid for <DOMAIN>, not elb.amazonaws.com #6290

abhyuditjain · 2019-01-03T07:27:52Z

1. What kops version are you running? The command kops version, will display
this information.

1.11.0

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.13.1

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kops create cluster $NAME \
--api-loadbalancer-type public \
--topology private \
--encrypt-etcd-storage=true \
--api-ssl-certificate "CERTIFICATE ARN" \
--zones "ap-south-1b,ap-south-1a" \
--ssh-public-key ~/.ssh/id_rsa \
--networking calico \
--master-count 1 \
--master-size t2.medium \
--master-volume-size 30 \
--node-count 2 \
--node-size t2.small \
--authorization "RBAC" \
--kubernetes-version "1.13.1" \
--image ami-1780a878 \
--node-volume-size 30 \
--yes

kops update cluster --yes

kops validate cluster

5. What happened after the commands executed?
The validate cluster failed. Also, I can't do rolling-update.

Using cluster from kubectl context: cluster.k8s.local

Validating cluster cluster.k8s.local


unexpected error during validation: error listing nodes: Get https://cluster-k8s-local-<something>.<region>.elb.amazonaws.com/api/v1/nodes: x509: certificate is valid for *.domain.com, not cluster-k8s-local-<something>.<region>.elb.amazonaws.com

6. What did you expect to happen?

Successful validation.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2019-01-03T06:32:09Z
  name: cluster.k8s.local
spec:
  api:
    loadBalancer:
      sslCertificate: <certificate-ARN>
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://cluster-state-store/cluster.k8s.local
  etcdClusters:
  - etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-ap-south-1b
      name: b
    name: main
  - etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-ap-south-1b
      name: b
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.13.1
  masterInternalName: api.internal.cluster.k8s.local
  masterPublicName: api.cluster.k8s.local
  networkCIDR: 172.20.0.0/16
  networking:
    calico: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.20.32.0/19
    egress: nat-<id>
    name: ap-south-1a
    type: Private
    zone: ap-south-1a
  - cidr: 172.20.64.0/19
    egress: nat-<id>
    name: ap-south-1b
    type: Private
    zone: ap-south-1b
  - cidr: 172.20.0.0/22
    name: utility-ap-south-1a
    type: Utility
    zone: ap-south-1a
  - cidr: 172.20.4.0/22
    name: utility-ap-south-1b
    type: Utility
    zone: ap-south-1b
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2019-01-03T07:12:43Z
  labels:
    kops.k8s.io/cluster: cluster.k8s.local
  name: bastions
spec:
  image: ami-1780a878
  machineType: t2.micro
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: bastions
  role: Bastion
  subnets:
  - utility-ap-south-1a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2019-01-03T06:32:10Z
  labels:
    kops.k8s.io/cluster: cluster.k8s.local
  name: master-ap-south-1b
spec:
  image: ami-1780a878
  machineType: t2.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-ap-south-1b
  role: Master
  rootVolumeSize: 30
  subnets:
  - ap-south-1b

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2019-01-03T06:32:10Z
  labels:
    kops.k8s.io/cluster: cluster.k8s.local
  name: nodes
spec:
  image: ami-1780a878
  machineType: t2.small
  maxSize: 2
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  rootVolumeSize: 30
  subnets:
  - ap-south-1a
  - ap-south-1b

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

The problem I think is that, the certificate I am using is for my domain, but the AWS issues its own domain for ELB that is used to communicate with the master, so there's SSL mismatch.

The text was updated successfully, but these errors were encountered:

abhyuditjain · 2019-01-08T14:34:17Z

@Raffo Tagging you here in reference to #5414 (comment)

Let me know if you want to know anything else.

abhyuditjain · 2019-01-16T14:34:08Z

@Raffo Any updates on this?

Raffo · 2019-01-16T16:32:59Z

Sorry, didn't have the chance to have a look at this, I'll try to take a look this weekend.

…

On Wed, Jan 16, 2019 at 3:34 PM Abhyudit Jain ***@***.***> wrote: @Raffo <https://github.com/Raffo> Any updates on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6290 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApv1C4dJz3ToxWjyJK-PIxLD2LQbuzeks5vDzhqgaJpZM4ZnqcD> .

-- Raffaele Di Fazio

abhyuditjain · 2019-01-17T10:45:37Z

@Raffo Sure. Let me know if I have to make any changes or if you can share your config with me that should work, I would be grateful.

abhyuditjain · 2019-01-21T05:46:30Z

@Raffo Did you get a chance to look at it?

thetutlage · 2019-02-06T13:30:12Z

Yes, stuck on same

Raffo · 2019-02-06T14:10:47Z

Sorry, didn't have time to look into this issue :-( Of course everyone can work on a fix, but I'll try to give it a shot in the next few days.

…

On Wed, Feb 6, 2019 at 2:30 PM Harminder Virk ***@***.***> wrote: Yes, stuck on same — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6290 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApv1PwXNNqvYOFck9ZWmcTGKCkYGJ2Rks5vKtjugaJpZM4ZnqcD> .

-- Raffaele Di Fazio

Raffo · 2019-02-08T13:19:20Z

Hi @abhyuditjain @thetutlage , I finally had some time to review this issue and try to debug it myself.
The good news is: there is no issue!
In fact, it all depends from the combination of cluster name and certificate that you specified: given a cluster name cluster.example.com you will need to pass the ARN for a domain *.cluster.example.com as kops will create an ELB with the name api.cluster.example.com.

Given that information I was able to reproduce your issue and demonstrate that with the correct combination of cert it does work. In case you believe that this is not clear (which is probably true), feel free to provide a PR with documentation!

Hope this help and you can resume working with kops and real certificates ;-)

abhyuditjain · 2019-02-08T13:34:12Z

@Raffo @thetutlage I found the issue. It's not with kops. You have to manually update the file located at ~/.kube/config and use the DNS name for which you have issued the certificate for. You have to use that DNS for the apiserver. Closing the issue.

arboat · 2019-03-21T07:14:44Z

@Raffo @thetutlage I found the issue. It's not with kops. You have to manually update the file located at ~/.kube/config and use the DNS name for which you have issued the certificate for. You have to use that DNS for the apiserver. Closing the issue.

Can you explain specificly with your example , I meet the same issue and do not follow what you mean,thank you.

abhyuditjain · 2019-05-03T09:59:41Z

@arboat When you create a cluster, an ELB is created whose CNAME is like this <something>.elb.amazonaws.com. This is what is in the ~/.kube/config under clusters. For example:

apiVersion: v1
clusters:
- cluster:
    server: <something>.elb.amazonaws.com
  name: cluster.k8s.local

But the ACM certificate is for your domain. Hence the error. So you should make a CNAME record pointing it to ELB and use the new CNAME in the ~/.kube/config in the server key.

acaire · 2019-07-29T02:12:06Z

For anyone stumbling across this issue, you might still be using private DNS like we were, and this is the block that's causing the mismatch when generating the ~/.kube/config.

naorlivne · 2020-01-30T15:54:37Z

For anyone stumbling across this issue, you might still be using private DNS like we were, and this is the block that's causing the mismatch when generating the ~/.kube/config.

Another option (which I found out the hard way gives the same error) is if you give your kube nodes a hostname naming scheme that's resolvable by a wildcard DNS record but isn't pointed to them (having the node named node.domain.com & a *.domain.com DNS records pointing to a catchall website that isn't the node)

abhyuditjain mentioned this issue Jan 8, 2019

[WIP] Initial implementation of ACM certificate for API server ELB #5414

Merged

abhyuditjain closed this as completed Feb 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ACM certificate x509: certificate is valid for <DOMAIN>, not elb.amazonaws.com #6290

ACM certificate x509: certificate is valid for <DOMAIN>, not elb.amazonaws.com #6290

abhyuditjain commented Jan 3, 2019 •

edited

abhyuditjain commented Jan 8, 2019

abhyuditjain commented Jan 16, 2019

Raffo commented Jan 16, 2019 via email

abhyuditjain commented Jan 17, 2019

abhyuditjain commented Jan 21, 2019

thetutlage commented Feb 6, 2019

Raffo commented Feb 6, 2019 via email

Raffo commented Feb 8, 2019

abhyuditjain commented Feb 8, 2019

arboat commented Mar 21, 2019

abhyuditjain commented May 3, 2019

acaire commented Jul 29, 2019

naorlivne commented Jan 30, 2020 •

edited

ACM certificate x509: certificate is valid for <DOMAIN>, not elb.amazonaws.com #6290

ACM certificate x509: certificate is valid for <DOMAIN>, not elb.amazonaws.com #6290

Comments

abhyuditjain commented Jan 3, 2019 • edited

abhyuditjain commented Jan 8, 2019

abhyuditjain commented Jan 16, 2019

Raffo commented Jan 16, 2019 via email

abhyuditjain commented Jan 17, 2019

abhyuditjain commented Jan 21, 2019

thetutlage commented Feb 6, 2019

Raffo commented Feb 6, 2019 via email

Raffo commented Feb 8, 2019

abhyuditjain commented Feb 8, 2019

arboat commented Mar 21, 2019

abhyuditjain commented May 3, 2019

acaire commented Jul 29, 2019

naorlivne commented Jan 30, 2020 • edited

abhyuditjain commented Jan 3, 2019 •

edited

naorlivne commented Jan 30, 2020 •

edited