[WIP] Drop IAM roles for aws master/workers #697
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
openshift-ci-robot
added
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rajatchopra The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
| @@ -57,7 +57,6 @@ func TFVars(cfg *types.InstallConfig, bootstrapIgn, masterIgn, workerIgn string) | |||
| if m.Platform.AWS != nil { | |||
| config.AWS.Master = aws.Master{ | |||
| EC2Type: m.Platform.AWS.InstanceType, | |||
| IAMRoleName: m.Platform.AWS.IAMRoleName, | |||
There was a problem hiding this comment.
i think this forces terraform to create one for you.
There was a problem hiding this comment.
i think this forces terraform to create one for you.
I haven't tracked that down, but we'll also want to remove/prune some of:
$ git grep iam_role origin/pr/697
origin/pr/697:data/data/aws/bootstrap/main.tf: role = "${var.iam_role == "" ?
origin/pr/697:data/data/aws/bootstrap/main.tf: join("|", aws_iam_role.bootstrap.*.name) :
origin/pr/697:data/data/aws/bootstrap/main.tf: join("|", data.aws_iam_role.bootstrap.*.name)
origin/pr/697:data/data/aws/bootstrap/main.tf:data "aws_iam_role" "bootstrap" {
origin/pr/697:data/data/aws/bootstrap/main.tf: count = "${var.iam_role == "" ? 0 : 1}"
origin/pr/697:data/data/aws/bootstrap/main.tf: name = "${var.iam_role}"
origin/pr/697:data/data/aws/bootstrap/main.tf:resource "aws_iam_role" "bootstrap" {
origin/pr/697:data/data/aws/bootstrap/main.tf: count = "${var.iam_role == "" ? 1 : 0}"
origin/pr/697:data/data/aws/bootstrap/main.tf:resource "aws_iam_role_policy" "bootstrap" {
origin/pr/697:data/data/aws/bootstrap/main.tf: count = "${var.iam_role == "" ? 1 : 0}"
origin/pr/697:data/data/aws/bootstrap/main.tf: role = "${aws_iam_role.bootstrap.id}"
origin/pr/697:data/data/aws/bootstrap/variables.tf:variable "iam_role" {
origin/pr/697:data/data/aws/iam/main.tf: role = "${var.worker_iam_role == "" ?
origin/pr/697:data/data/aws/iam/main.tf: join("|", aws_iam_role.worker_role.*.name) :
origin/pr/697:data/data/aws/iam/main.tf: join("|", data.aws_iam_role.worker_role.*.name)
origin/pr/697:data/data/aws/iam/main.tf:data "aws_iam_role" "worker_role" {
origin/pr/697:data/data/aws/iam/main.tf: count = "${var.worker_iam_role == "" ? 0 : 1}"
origin/pr/697:data/data/aws/iam/main.tf: name = "${var.worker_iam_role}"
origin/pr/697:data/data/aws/iam/main.tf:resource "aws_iam_role" "worker_role" {
origin/pr/697:data/data/aws/iam/main.tf: count = "${var.worker_iam_role == "" ? 1 : 0}"
origin/pr/697:data/data/aws/iam/main.tf:resource "aws_iam_role_policy" "worker_policy" {
origin/pr/697:data/data/aws/iam/main.tf: count = "${var.worker_iam_role == "" ? 1 : 0}"
origin/pr/697:data/data/aws/iam/main.tf: role = "${aws_iam_role.worker_role.id}"
origin/pr/697:data/data/aws/iam/variables.tf:variable "worker_iam_role" {
origin/pr/697:data/data/aws/main.tf: iam_role = "${var.tectonic_aws_master_iam_role_name}"
origin/pr/697:data/data/aws/main.tf: master_iam_role = "${var.tectonic_aws_master_iam_role_name}"
origin/pr/697:data/data/aws/main.tf: worker_iam_role = "${var.tectonic_aws_worker_iam_role_name}"
origin/pr/697:data/data/aws/master/main.tf: role = "${var.master_iam_role == "" ?
origin/pr/697:data/data/aws/master/main.tf: join("|", aws_iam_role.master_role.*.name) :
origin/pr/697:data/data/aws/master/main.tf: join("|", data.aws_iam_role.master_role.*.name)
origin/pr/697:data/data/aws/master/main.tf:data "aws_iam_role" "master_role" {
origin/pr/697:data/data/aws/master/main.tf: count = "${var.master_iam_role == "" ? 0 : 1}"
origin/pr/697:data/data/aws/master/main.tf: name = "${var.master_iam_role}"
origin/pr/697:data/data/aws/master/main.tf:resource "aws_iam_role" "master_role" {
origin/pr/697:data/data/aws/master/main.tf: count = "${var.master_iam_role == "" ? 1 : 0}"
origin/pr/697:data/data/aws/master/main.tf:resource "aws_iam_role_policy" "master_policy" {
origin/pr/697:data/data/aws/master/main.tf: count = "${var.master_iam_role == "" ? 1 : 0}"
origin/pr/697:data/data/aws/master/main.tf: role = "${aws_iam_role.master_role.id}"
origin/pr/697:data/data/aws/master/variables.tf:variable "master_iam_role" {
origin/pr/697:data/data/aws/variables-aws.tf:variable "tectonic_aws_master_iam_role_name" {
origin/pr/697:data/data/aws/variables-aws.tf:variable "tectonic_aws_worker_iam_role_name" {
origin/pr/697:pkg/tfvars/aws/aws.go: IAMRoleName string `json:"tectonic_aws_master_iam_role_name,omitempty" yaml:"iamRoleName,omitempty"`
origin/pr/697:pkg/tfvars/aws/aws.go: IAMRoleName string `json:"tectonic_aws_worker_iam_role_name,omitempty" yaml:"iamRoleName,omitempty"`
as well. I'm not sure about the bootstrap role. Maybe we should keep that?
There was a problem hiding this comment.
Also should prune:
$ git grep -i iamrole origin/pr/697 -- pkg
origin/pr/697:pkg/tfvars/aws/aws.go: IAMRoleName string `json:"tectonic_aws_master_iam_role_name,omitempty" yaml:"iamRoleName,omitempty"`
origin/pr/697:pkg/tfvars/aws/aws.go: IAMRoleName string `json:"tectonic_aws_worker_iam_role_name,omitempty" yaml:"iamRoleName,omitempty"`
origin/pr/697:pkg/types/aws/machinepool.go: // IAMRoleName defines the IAM role associated
origin/pr/697:pkg/types/aws/machinepool.go: IAMRoleName string `json:"iamRoleName"`
origin/pr/697:pkg/types/aws/machinepool.go: if required.IAMRoleName != "" {
origin/pr/697:pkg/types/aws/machinepool.go: a.IAMRoleName = required.IAMRoleNameand related.
rajatchopra
force-pushed
the
iam_roles
branch
from
bbc3b64
to
80edcda
Compare
openshift-ci-robot
added
size/L
openshift-bot
added
the
needs-rebase
rajatchopra
force-pushed
the
iam_roles
branch
from
80edcda
to
a67376f
Compare
openshift-bot
removed
the
needs-rebase
mazzystr
reviewed
Nov 19, 2018
rajatchopra
force-pushed
the
iam_roles
branch
4 times, most recently
from
c85bdb7
to
d5ba752
Compare
rajatchopra
force-pushed
the
iam_roles
branch
from
d5ba752
to
95a9bad
Compare
|
So it looks like $ [core@ip-10-0-0-241 ~]$ journalctl -fu kubelet
-- Logs begin at Mon 2018-11-26 21:50:51 UTC. --
Nov 26 21:57:05 ip-10-0-0-241 hyperkube[4543]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 26 21:57:05 ip-10-0-0-241 hyperkube[4543]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 26 21:57:05 ip-10-0-0-241 hyperkube[4543]: I1126 21:57:05.079437 4543 server.go:418] Version: v1.11.0+d4cacc0
Nov 26 21:57:05 ip-10-0-0-241 hyperkube[4543]: I1126 21:57:05.079878 4543 aws.go:1032] Building AWS cloudprovider
Nov 26 21:57:05 ip-10-0-0-241 hyperkube[4543]: I1126 21:57:05.079917 4543 aws.go:994] Zone not specified in configuration file; querying AWS metadata service
Nov 26 21:57:05 ip-10-0-0-241 hyperkube[4543]: F1126 21:57:05.084483 4543 server.go:262] failed to run Kubelet: could not init cloud provider "aws": error finding instance i-0fc2564fadb34ac2f: "error listing AWS instances: \"NoCredentialProviders: no valid providers in chain. Deprecated.\\n\\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\""
Nov 26 21:57:05 ip-10-0-0-241 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Nov 26 21:57:05 ip-10-0-0-241 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 26 21:57:05 ip-10-0-0-241 systemd[1]: Unit kubelet.service entered failed state.
Nov 26 21:57:05 ip-10-0-0-241 systemd[1]: kubelet.service failed.
Nov 26 21:57:15 ip-10-0-0-241 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Nov 26 21:57:15 ip-10-0-0-241 systemd[1]: Stopped Kubernetes Kubelet.
Nov 26 21:57:15 ip-10-0-0-241 systemd[1]: Starting Kubernetes Kubelet...
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --rotate-certificates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --allow-privileged has been deprecated, will be removed in a future version
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --rotate-certificates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --allow-privileged has been deprecated, will be removed in a future version
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: I1126 21:57:15.301090 4558 server.go:418] Version: v1.11.0+d4cacc0
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: I1126 21:57:15.301374 4558 aws.go:1032] Building AWS cloudprovider
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: I1126 21:57:15.301405 4558 aws.go:994] Zone not specified in configuration file; querying AWS metadata service
Nov 26 21:57:15 ip-10-0-0-241 hyperkube[4558]: F1126 21:57:15.305323 4558 server.go:262] failed to run Kubelet: could not init cloud provider "aws": error finding instance i-0fc2564fadb34ac2f: "error listing AWS instances: \"NoCredentialProviders: no valid providers in chain. Deprecated.\\n\\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\""kubelet on master nodes is not running due to missing credentials for aws. |
Does anyone know off the top of their head why kubelet would care about listing AWS instances? |
The InitCloudProvider wants to verify the instance ID so that it can mark itself ready. This code is supposed to be removed soon in favour of an external controller playing with the taints.. but I am not sure when it will land (or maybe it has landed already but not made it to our snapshot). |
Looks like this feature is still beta in 1.13. So is there a way to get sub-creds in to the kubelet without empowering the entire node? |
|
@rajatchopra: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
needs-rebase
|
@rajatchopra: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
Closing this PR because we are blocked on kubelet needing cloud-config+secrets to work without iam-instance roles. |
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Jan 30, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Jan 30, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Jan 30, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Jan 30, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Jan 30, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Jan 30, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Feb 1, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Feb 1, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Feb 1, 2019
We're planning on dropping instance profiles in favor of the new credentials operator [1], because we want AWS access to have operator/pod/namespace granularity and not instance granularity. Many pods could be running on a given instance, and not all of them should have a given permission. While we're blocked from dropping these at the moment due to kubelet cloud-config+secrets [2], we can drop the user-facing knobs for this feature now. Then pivoting the internal approach once we get the kubelet sorted will be a non-breaking change. [1]: https://github.com/openshift/cloud-credential-operator [2]: openshift#697 (comment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For the first step towards making sure all iam-role creds are granularized and each sufficient one stored within the scope of the component/operator that requires it.
This is a work in progress - the author still needs to understand what all will break.