feat: add auditd as an ubuntu option #1143

jackfrancis · 2019-04-24T21:01:52Z

Reason for Change:

Introduces auditd package and configuration into the VHD + cloud-init implementation. Requires an explicit "auditDEnabled": true profile configuration to enable. If not enabled explicitly, we disable auditd via systemd.

Issue Fixed:

Fixes #1008

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
tested upgrade from previous version

Notes:

acs-bot · 2019-04-24T21:03:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2019-04-24T21:22:28Z

Codecov Report

Merging #1143 into master will increase coverage by 0.13%.
The diff coverage is 98.14%.

@@            Coverage Diff             @@
##           master    #1143      +/-   ##
==========================================
+ Coverage    74.7%   74.84%   +0.13%     
==========================================
  Files         128      128              
  Lines       18236    18337     +101     
==========================================
+ Hits        13623    13724     +101     
  Misses       3829     3829              
  Partials      784      784

jackfrancis · 2019-04-24T21:25:41Z

Current implementation (via "ubuntu" distro) is failing thusly:

# systemctl status auditd.service
● auditd.service - Security Auditing Service
   Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-04-24 20:58:30 UTC; 2s ago
  Process: 13508 ExecStartPost=/sbin/auditctl -R /etc/audit/audit.rules (code=exited, status=0/SUCCESS)
  Process: 13507 ExecStart=/sbin/auditd -n (code=exited, status=6)
 Main PID: 13507 (code=exited, status=6)

Apr 24 20:58:30 k8s-master-16990356-0 auditctl[13508]: pid 0
Apr 24 20:58:30 k8s-master-16990356-0 auditctl[13508]: rate_limit 0
Apr 24 20:58:30 k8s-master-16990356-0 auditctl[13508]: backlog_limit 320
Apr 24 20:58:30 k8s-master-16990356-0 auditctl[13508]: lost 0
Apr 24 20:58:30 k8s-master-16990356-0 auditctl[13508]: backlog 0
Apr 24 20:58:30 k8s-master-16990356-0 auditctl[13508]: backlog_wait_time 15000
Apr 24 20:58:30 k8s-master-16990356-0 systemd[1]: auditd.service: Main process exited, code=exited, status=6/NOTCONFIGURED
Apr 24 20:58:30 k8s-master-16990356-0 systemd[1]: Failed to start Security Auditing Service.
Apr 24 20:58:30 k8s-master-16990356-0 systemd[1]: auditd.service: Unit entered failed state.
Apr 24 20:58:30 k8s-master-16990356-0 systemd[1]: auditd.service: Failed with result 'exit-code'.

jackfrancis · 2019-04-24T23:07:03Z

Verified this disables auditd on the OS if not enabled vi api model:

+ ensureAuditD
+ [[ false == true ]]
+ grep auditd
+ apt list --installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

auditd/xenial-updates,now 1:2.4.5-1ubuntu2.1 amd64 [installed]
+ systemctlDisableAndStop auditd
+ systemctl_stop 100 5 30 auditd
+ retries=100
+ wait_sleep=5
+ timeout=30
+ svcname=auditd
++ seq 1 100
+ for i in '$(seq 1 $retries)'
+ timeout 30 systemctl daemon-reload
+ timeout 30 systemctl stop auditd
+ '[' 0 -eq 0 ']'
+ break
+ '[' 0 -ne 0 ']'
+ retrycmd_if_failure 120 5 25 systemctl disable auditd
+ retries=120
+ wait_sleep=5
+ timeout=25
+ shift
+ shift
+ shift
++ seq 1 120
+ for i in '$(seq 1 $retries)'
+ timeout 25 systemctl disable auditd
Synchronizing state of auditd.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install disable auditd
insserv: warning: current start runlevel(s) (empty) of script `auditd' overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `auditd' overrides LSB defaults (0 1 6).
+ '[' 0 -eq 0 ']'
+ break
+ echo Executed '"systemctl' disable 'auditd"' 1 times

jackfrancis · 2019-04-24T23:08:20Z

Verified auditd is enabled via systemd when enabled via api model:

+ ensureAuditD
+ [[ true == true ]]
+ systemctlEnableAndStart auditd
+ systemctl_restart 100 5 30 auditd
+ retries=100
+ wait_sleep=5
+ timeout=30
+ svcname=auditd
++ seq 1 100
+ for i in '$(seq 1 $retries)'
+ timeout 30 systemctl daemon-reload
+ timeout 30 systemctl restart auditd
+ '[' 0 -eq 0 ']'
+ break
+ RESTART_STATUS=0
+ systemctl status auditd --no-pager -l
+ '[' 0 -ne 0 ']'
+ retrycmd_if_failure 120 5 25 systemctl enable auditd
+ retries=120
+ wait_sleep=5
+ timeout=25
+ shift
+ shift
+ shift
++ seq 1 120
+ for i in '$(seq 1 $retries)'
+ timeout 25 systemctl enable auditd
Synchronizing state of auditd.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable auditd
+ '[' 0 -eq 0 ']'
+ break
+ echo Executed '"systemctl' enable 'auditd"' 1 times
Executed "systemctl enable auditd" 1 times

jackfrancis · 2019-04-24T23:09:07Z

Validated via "ubuntu" distro that an enabled auditd + config produces a cluster that passes E2E.

jackfrancis · 2019-04-25T00:00:14Z

Validated "ubuntu-18.04" distro as well.

parts/k8s/cloud-init/artifacts/cse_config.sh

parts/k8s/cloud-init/artifacts/cse_install.sh

jackfrancis · 2019-04-26T19:28:09Z

Status: debug why auditd is exiting 6 after apt installation via VHD. Unable to repro using "ubuntu" distro creation flow:

$ sudo systemctl status auditd
● auditd.service - Security Auditing Service
   Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-04-26 19:12:23 UTC; 15min ago
 Main PID: 10054 (auditd)
    Tasks: 2
   Memory: 1.1M
      CPU: 63ms
   CGroup: /system.slice/auditd.service
           └─10054 /sbin/auditd -n

Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: No rules
Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: enabled 1
Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: failure 1
Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: pid 10054
Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: rate_limit 0
Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: backlog_limit 320
Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: lost 0
Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: backlog 1
Apr 26 19:12:23 k8s-master-15192496-0 auditctl[10055]: backlog_wait_time 15000
Apr 26 19:12:23 k8s-master-15192496-0 systemd[1]: Started Security Auditing Service.

parts/k8s/cloud-init/artifacts/cis.sh

jackfrancis · 2019-05-01T21:35:54Z

validated new E2E on ubuntu distros

CecileRobertMichon · 2019-05-01T22:52:21Z

parts/k8s/cloud-init/artifacts/cse_config.sh

+  else
+    apt list --installed | grep 'auditd'
+    if [ $? -eq 0 ]; then
+      systemctlDisableAndStop auditd || exit $ERR_SYSTEMCTL_START_FAIL


why not apt-get purge if it's not enabled?

Just trying to avoid more apt operations at runtime for all cluster creates. Strictly speaking it's not doing anything except taking up space on the filesystem

this seems like it's introducing more failure potential since with the existing cleanup functions we simply attempt to purge but without returning a non-zero exit code if the cleanup is unsuccessful (cleaning up an unnecessary package is arguably not critical to cluster deployment success). This breaks that pattern by saying: "return $ERR_SYSTEMCTL_START_FAIL if we can't disable and stop the service" which could fail for reason xyz.

It's very important that we (1) ensure that auditd isn't running and (2) that it won't start after reboot, so we definitely need this validation.

You were right all along:

#2078

:)

pkg/api/types.go

test/e2e/kubernetes/kubernetes_test.go

pkg/api/vlabs/types.go

jackfrancis · 2019-05-03T22:14:33Z

/azp run

azure-pipelines · 2019-05-03T22:14:45Z

Azure Pipelines successfully started running 1 pipeline(s).

CecileRobertMichon

lgtm

acs-bot added do-not-merge/work-in-progress size/L labels Apr 24, 2019

acs-bot added the approved label Apr 24, 2019

jackfrancis changed the title ~~WIP feat: add auditd as an ubuntu option~~ feat: add auditd as an ubuntu option Apr 24, 2019

acs-bot removed the do-not-merge/work-in-progress label Apr 24, 2019

jackfrancis force-pushed the cis-2019-04-29 branch from 7f3f033 to 6a10c29 Compare April 24, 2019 22:16

jackfrancis added this to In progress in CIS spike - Spring 2019 Apr 25, 2019

CecileRobertMichon added the needs-rebase Changes in the target branch require a `git rebase` and `git push -f` label Apr 26, 2019

CecileRobertMichon reviewed Apr 26, 2019

View reviewed changes

parts/k8s/cloud-init/artifacts/cse_config.sh Outdated Show resolved Hide resolved

CecileRobertMichon reviewed Apr 26, 2019

View reviewed changes

parts/k8s/cloud-init/artifacts/cse_install.sh Outdated Show resolved Hide resolved

jackfrancis force-pushed the cis-2019-04-29 branch from f3709e5 to 29f6e92 Compare April 26, 2019 18:30

mboersma reviewed Apr 26, 2019

View reviewed changes

parts/k8s/cloud-init/artifacts/cis.sh Outdated Show resolved Hide resolved

mboersma removed the needs-rebase Changes in the target branch require a `git rebase` and `git push -f` label Apr 26, 2019

jackfrancis force-pushed the cis-2019-04-29 branch from f2c891c to 1fbdad3 Compare April 26, 2019 20:15

mboersma added this to Under Review in backlog Apr 29, 2019

jackfrancis force-pushed the cis-2019-04-29 branch from fed1eb3 to b7451aa Compare May 1, 2019 20:34

CecileRobertMichon reviewed May 1, 2019

View reviewed changes

pkg/api/types.go Outdated Show resolved Hide resolved

CecileRobertMichon reviewed May 1, 2019

View reviewed changes

pkg/api/types.go Outdated Show resolved Hide resolved

CecileRobertMichon reviewed May 2, 2019

View reviewed changes

test/e2e/kubernetes/kubernetes_test.go Outdated Show resolved Hide resolved

CecileRobertMichon reviewed May 2, 2019

View reviewed changes

pkg/api/vlabs/types.go Show resolved Hide resolved

jackfrancis added 18 commits May 3, 2019 09:39

chore: disable auditd via systemctl, always install

49bcdc6

chore: use ensureAuditD

4a78bed

chore: only disable auditd if installed

4295af4

chore: add docs mention

6ebe67b

chore: auditDEnabled for master nodes

9d404bd

chore: we don’t need AUDITD_ENABLED in VHD build script

6efa622

chore: remove (Scored)

6b2b5d5

chore: add newline

d592d82

fix: broken bash logic

042f657

debug: print files

c0e959f

debug: journalctl output

2ab5e09

fix: typo

2a21420

chore: enable per-package debug

dbfd5b7

fix: remove auditd.conf

717976f

test: add basic auditd validation to e2e

1274015

test: generalize to IsInProfile

2b74667

chore: prefer direct to.Bool instead of object method

a21b54c

test: add vlabs types ubuntu unit tests

8627efc

jackfrancis force-pushed the cis-2019-04-29 branch from 75a21cb to 8627efc Compare May 3, 2019 16:46

acs-bot added size/XL and removed size/L labels May 3, 2019

jackfrancis added 2 commits May 3, 2019 09:49

test: add validation unit test for masterProfile auditd option

f992029

chore: fix auditd e2e message

7c7b866

CecileRobertMichon approved these changes May 3, 2019

View reviewed changes

jackfrancis merged commit ed80e84 into Azure:master May 4, 2019

backlog automation moved this from Under Review to Done May 4, 2019

CIS spike - Spring 2019 automation moved this from In progress to Done May 4, 2019

jackfrancis deleted the cis-2019-04-29 branch May 4, 2019 22:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add auditd as an ubuntu option #1143

feat: add auditd as an ubuntu option #1143

jackfrancis commented Apr 24, 2019 •

edited

acs-bot commented Apr 24, 2019

codecov bot commented Apr 24, 2019 •

edited

jackfrancis commented Apr 24, 2019

jackfrancis commented Apr 24, 2019

jackfrancis commented Apr 24, 2019

jackfrancis commented Apr 24, 2019

jackfrancis commented Apr 25, 2019

jackfrancis commented Apr 26, 2019

jackfrancis commented May 1, 2019

CecileRobertMichon May 1, 2019

jackfrancis May 1, 2019

CecileRobertMichon May 1, 2019

jackfrancis May 1, 2019

jackfrancis Oct 1, 2019

jackfrancis commented May 3, 2019

azure-pipelines bot commented May 3, 2019

CecileRobertMichon left a comment

feat: add auditd as an ubuntu option #1143

feat: add auditd as an ubuntu option #1143

Conversation

jackfrancis commented Apr 24, 2019 • edited

acs-bot commented Apr 24, 2019

codecov bot commented Apr 24, 2019 • edited

Codecov Report

jackfrancis commented Apr 24, 2019

jackfrancis commented Apr 24, 2019

jackfrancis commented Apr 24, 2019

jackfrancis commented Apr 24, 2019

jackfrancis commented Apr 25, 2019

jackfrancis commented Apr 26, 2019

jackfrancis commented May 1, 2019

CecileRobertMichon May 1, 2019

Choose a reason for hiding this comment

jackfrancis May 1, 2019

Choose a reason for hiding this comment

CecileRobertMichon May 1, 2019

Choose a reason for hiding this comment

jackfrancis May 1, 2019

Choose a reason for hiding this comment

jackfrancis Oct 1, 2019

Choose a reason for hiding this comment

jackfrancis commented May 3, 2019

azure-pipelines bot commented May 3, 2019

CecileRobertMichon left a comment

Choose a reason for hiding this comment

jackfrancis commented Apr 24, 2019 •

edited

codecov bot commented Apr 24, 2019 •

edited