Add audit log to Nutanix control plane template #7190

adiantum · 2023-12-16T01:17:51Z

Description of changes:
Add audit log to Nutanix control plane template

add extra args for audit logs
add required extra volumes
change test manifests

Testing:
Ubuntu 20.04 K8S 1.27:

$ eksctl anywhere create cluster -f ntnx-audit-log.yaml --bundles-override ./local-bundle-release.yaml
Performing setup and validations
odd number of arguments passed as key-value pairs for logging	{"ignored key": "ntnx-audit-log"}
ValidateClusterSpec for Nutanix datacenter
✅ Nutanix Provider setup is valid
✅ Validate OS is compatible with registry mirror configuration
✅ Validate certificate for registry mirror
✅ Validate authentication for git provider
✅ Validate cluster's eksaVersion matches EKS-A version
...
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
--------------------------------------------------------------------------------------
The Amazon EKS Anywhere Curated Packages are only available to customers with the
Amazon EKS Anywhere Enterprise Subscription
--------------------------------------------------------------------------------------
Enabling curated packages on the cluster
Installing helm chart on cluster	{"chart": "eks-anywhere-packages", "version": "0.0.0-46804348296d08393a2532a11f7acdedfa82659a"}

eksa@ntnx-audit-log-lt85n:~$ ls /etc/kubernetes/audit-policy.yaml
/etc/kubernetes/audit-policy.yaml
eksa@ntnx-audit-log-lt85n:~$ ls /var/log/kubernetes/api-audit.log
/var/log/kubernetes/api-audit.log

Ubuntu 22.04 K8S 1.27:

$ eksctl anywhere create cluster -f ntnx-audit-log.yaml --bundles-override ./local-bundle-release.yaml
Performing setup and validations
odd number of arguments passed as key-value pairs for logging	{"ignored key": "ntnx-audit-log"}
ValidateClusterSpec for Nutanix datacenter
✅ Nutanix Provider setup is valid
✅ Validate OS is compatible with registry mirror configuration
✅ Validate certificate for registry mirror
✅ Validate authentication for git provider
✅ Validate cluster's eksaVersion matches EKS-A version
...
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
--------------------------------------------------------------------------------------
The Amazon EKS Anywhere Curated Packages are only available to customers with the
Amazon EKS Anywhere Enterprise Subscription
--------------------------------------------------------------------------------------
Enabling curated packages on the cluster
Installing helm chart on cluster	{"chart": "eks-anywhere-packages", "version": "0.0.0-46804348296d08393a2532a11f7acdedfa82659a"}

eksa@ntnx-audit-log-cwq84:~$ ls /etc/kubernetes/audit-policy.yaml
/etc/kubernetes/audit-policy.yaml
eksa@ntnx-audit-log-cwq84:~$ ls /var/log/kubernetes/api-audit.log
/var/log/kubernetes/api-audit.log

RHEL 8 K8S 1.28:

$ eksctl anywhere create cluster -f ntnx-audit-log.yaml --bundles-override ./local-bundle-release.yaml
Performing setup and validations
odd number of arguments passed as key-value pairs for logging	{"ignored key": "ntnx-audit-log"}
ValidateClusterSpec for Nutanix datacenter
✅ Nutanix Provider setup is valid
✅ Validate OS is compatible with registry mirror configuration
✅ Validate certificate for registry mirror
✅ Validate authentication for git provider
✅ Validate cluster's eksaVersion matches EKS-A version
...
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
--------------------------------------------------------------------------------------
The Amazon EKS Anywhere Curated Packages are only available to customers with the
Amazon EKS Anywhere Enterprise Subscription
--------------------------------------------------------------------------------------
Enabling curated packages on the cluster
Installing helm chart on cluster	{"chart": "eks-anywhere-packages", "version": "0.0.0-46804348296d08393a2532a11f7acdedfa82659a"}

[eksa@ntnx-audit-log-qw8g8 ~]$ ls /etc/kubernetes/audit-policy.yaml
/etc/kubernetes/audit-policy.yaml
[eksa@ntnx-audit-log-qw8g8 ~]$ ls /var/log/kubernetes/api-audit.log
/var/log/kubernetes/api-audit.log

RHEL 9 K8S 1.28:

$ eksctl anywhere create cluster -f ntnx-audit-log.yaml --bundles-override ./local-bundle-release.yaml
Performing setup and validations
odd number of arguments passed as key-value pairs for logging	{"ignored key": "ntnx-audit-log"}
ValidateClusterSpec for Nutanix datacenter
✅ Nutanix Provider setup is valid
✅ Validate OS is compatible with registry mirror configuration
✅ Validate certificate for registry mirror
✅ Validate authentication for git provider
✅ Validate cluster's eksaVersion matches EKS-A version
...
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
--------------------------------------------------------------------------------------
The Amazon EKS Anywhere Curated Packages are only available to customers with the
Amazon EKS Anywhere Enterprise Subscription
--------------------------------------------------------------------------------------
Enabling curated packages on the cluster
Installing helm chart on cluster	{"chart": "eks-anywhere-packages", "version": "0.0.0-46804348296d08393a2532a11f7acdedfa82659a"}

[eksa@ntnx-audit-log-2hhlb ~]$ ls /etc/kubernetes/audit-policy.yaml
/etc/kubernetes/audit-policy.yaml
[eksa@ntnx-audit-log-2hhlb ~]$ ls /var/log/kubernetes/api-audit.log
/var/log/kubernetes/api-audit.log

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

eks-distro-bot · 2023-12-16T01:18:04Z

Hi @adiantum. Thanks for your PR.

I'm waiting for a aws member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

deepakm-ntnx · 2023-12-18T17:15:03Z

@adiantum could you please post sample audit logs generated for nutanix provider

pkg/providers/nutanix/config/cp-template.yaml

pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_domain_name.yaml

codecov · 2023-12-18T18:07:41Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (444d9ad) 71.59% compared to head (1635daa) 71.59%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7190   +/-   ##
=======================================
  Coverage   71.59%   71.59%           
=======================================
  Files         548      548           
  Lines       42521    42527    +6     
=======================================
+ Hits        30441    30447    +6     
  Misses      10386    10386           
  Partials     1694     1694

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pkg/providers/nutanix/config/cp-template.yaml

pkg/providers/nutanix/template.go

jiayiwang7 · 2023-12-18T19:42:34Z

lets add more tests for the codecov

adiantum · 2023-12-18T21:45:44Z

lets add more tests for the codecov

Done

dkoshkin · 2023-12-19T23:03:43Z

pkg/providers/nutanix/template.go

@@ -158,7 +159,14 @@ func buildTemplateMapCP(
 	kubeletExtraArgs := clusterapi.SecureTlsCipherSuitesExtraArgs().
 		Append(clusterapi.ResolvConfExtraArgs(clusterSpec.Cluster.Spec.ClusterNetwork.DNS.ResolvConf)).
 		Append(clusterapi.ControlPlaneNodeLabelsExtraArgs(clusterSpec.Cluster.Spec.ControlPlaneConfiguration))
+
+	auditPolicy, err := common.GetAuditPolicy(clusterSpec.Cluster.Spec.KubernetesVersion)


Just curious where this policy comes from. Should it be updated with latest from upstream?
https://github.com/kubernetes/kubernetes/blob/v1.29.0/cluster/gce/gci/configure-helper.sh#L1144

Maybe have it be dynamic based on the minor k8s version?

you are right it should be updated with upstream. we haven't done that yet.

jiayiwang7 · 2023-12-20T16:09:11Z

can you please rebase from main? we fixed a CVE and should pass the vulnerability scan.

- add extra args for audit logs - add required extra volumes - change test manifests

adiantum · 2023-12-20T17:10:56Z

can you please rebase from main? we fixed a CVE and should pass the vulnerability scan.

Done. Rebased.

jiayiwang7 · 2023-12-20T17:22:26Z

/approve
/lgtm

eks-distro-bot · 2023-12-20T17:22:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiayiwang7

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jiayiwang7]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jiayiwang7 · 2023-12-20T17:57:25Z

/cherry-pick release-0.18

eks-distro-pr-bot · 2023-12-20T17:58:01Z

@jiayiwang7: #7190 failed to apply on top of branch "release-0.18":

Applying: Add audit log to Nutanix control plane template - add extra args for audit logs - add required extra volumes - change test manifests
Using index info to reconstruct a base tree...
M	pkg/providers/nutanix/config/cp-template.yaml
M	pkg/providers/nutanix/template.go
A	pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_domain_name.yaml
A	pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_ip.yaml
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_ip.yaml deleted in HEAD and modified in Add audit log to Nutanix control plane template - add extra args for audit logs - add required extra volumes - change test manifests. Version Add audit log to Nutanix control plane template - add extra args for audit logs - add required extra volumes - change test manifests of pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_ip.yaml left in tree.
CONFLICT (modify/delete): pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_domain_name.yaml deleted in HEAD and modified in Add audit log to Nutanix control plane template - add extra args for audit logs - add required extra volumes - change test manifests. Version Add audit log to Nutanix control plane template - add extra args for audit logs - add required extra volumes - change test manifests of pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_domain_name.yaml left in tree.
Auto-merging pkg/providers/nutanix/template.go
Auto-merging pkg/providers/nutanix/config/cp-template.yaml
CONFLICT (content): Merge conflict in pkg/providers/nutanix/config/cp-template.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Add audit log to Nutanix control plane template - add extra args for audit logs - add required extra volumes - change test manifests
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-0.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

* Add audit log to Nutanix control plane template - add extra args for audit logs - add required extra volumes - change test manifests * Fixed comments and missed unit test # Conflicts: # pkg/providers/nutanix/config/cp-template.yaml # pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_domain_name.yaml # pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_ip.yaml

eks-distro-bot added the needs-ok-to-test label Dec 16, 2023

eks-distro-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 16, 2023

deepakm-ntnx reviewed Dec 18, 2023

View reviewed changes

pkg/providers/nutanix/config/cp-template.yaml Show resolved Hide resolved

deepakm-ntnx reviewed Dec 18, 2023

View reviewed changes

pkg/providers/nutanix/config/cp-template.yaml Show resolved Hide resolved

deepakm-ntnx reviewed Dec 18, 2023

View reviewed changes

pkg/providers/nutanix/testdata/expected_cluster_api_server_cert_san_domain_name.yaml Show resolved Hide resolved

jiayiwang7 reviewed Dec 18, 2023

View reviewed changes

pkg/providers/nutanix/config/cp-template.yaml Show resolved Hide resolved

pkg/providers/nutanix/template.go Outdated Show resolved Hide resolved

adiantum requested a review from jiayiwang7 December 18, 2023 21:46

dkoshkin reviewed Dec 19, 2023

View reviewed changes

adiantum added 2 commits December 20, 2023 12:09

Add audit log to Nutanix control plane template

f7f4332

- add extra args for audit logs - add required extra volumes - change test manifests

Fixed comments and missed unit test

1635daa

eks-distro-bot assigned jiayiwang7 Dec 20, 2023

eks-distro-bot added the lgtm label Dec 20, 2023

eks-distro-bot added the approved label Dec 20, 2023

eks-distro-bot merged commit a592862 into aws:main Dec 20, 2023
12 checks passed

jiayiwang7 mentioned this pull request Dec 20, 2023

Add audit log to Nutanix control plane template (#7190) #7220

Merged

adiantum deleted the ntnx-audit-log branch December 21, 2023 09:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add audit log to Nutanix control plane template #7190

Add audit log to Nutanix control plane template #7190

adiantum commented Dec 16, 2023 •

edited

eks-distro-bot commented Dec 16, 2023

deepakm-ntnx commented Dec 18, 2023

codecov bot commented Dec 18, 2023 •

edited

jiayiwang7 commented Dec 18, 2023

adiantum commented Dec 18, 2023

dkoshkin Dec 19, 2023

jiayiwang7 Dec 20, 2023

jiayiwang7 commented Dec 20, 2023

adiantum commented Dec 20, 2023

jiayiwang7 commented Dec 20, 2023

eks-distro-bot commented Dec 20, 2023

jiayiwang7 commented Dec 20, 2023

eks-distro-pr-bot commented Dec 20, 2023

Add audit log to Nutanix control plane template #7190

Add audit log to Nutanix control plane template #7190

Conversation

adiantum commented Dec 16, 2023 • edited

eks-distro-bot commented Dec 16, 2023

deepakm-ntnx commented Dec 18, 2023

codecov bot commented Dec 18, 2023 • edited

Codecov Report

jiayiwang7 commented Dec 18, 2023

adiantum commented Dec 18, 2023

dkoshkin Dec 19, 2023

Choose a reason for hiding this comment

jiayiwang7 Dec 20, 2023

Choose a reason for hiding this comment

jiayiwang7 commented Dec 20, 2023

adiantum commented Dec 20, 2023

jiayiwang7 commented Dec 20, 2023

eks-distro-bot commented Dec 20, 2023

jiayiwang7 commented Dec 20, 2023

eks-distro-pr-bot commented Dec 20, 2023

adiantum commented Dec 16, 2023 •

edited

codecov bot commented Dec 18, 2023 •

edited