change: feature flags for image-allow-rules + secure-by-default policy (#1549)

[iwilltry42]

• switch to "secure by default" mode, meaning that with no IAR present,
  all images will be disallowed and having only one passing for a given
  image will allow it
• additional field "scope" on IAR CR to define the images for which the
  rule will apply (prefix or glob matching)

Signed-off-by: Thorsten Klein tk@thklein.io

Ref: #1549

Checklist

• The title of this PR would make a good line in Acorn's Release Note's Changelog
• The title of this PR ends with a link to the main issue being address in parentheses, like: This is a title (#1216). Here's an example
• All relevant issues are referenced in the PR description. NOTE: don't use GitHub keywords that auto-close issues
• Commits follow contributing guidance
• Automated tests added to cover the changes. If tests couldn't be added, an explanation is provided in the Verification and Testing section
• Changes to user-facing functionality, API, CLI, and upgrade impacts are clearly called out in PR description
• PR has at least two approvals before merging (or a reasonable exception, like it's just a docs change)

[iwilltry42]

Update 15.05.2023

Feature Flag

In yesterday's meeting we agreed upon hiding this "deny by default" functionality behind a feature flag for now until we figured out a way of non-intrusively exposing this whole IAR feature to the user.
-> We may use https://openfeature.dev in the future for feature flags in general.

I'll follow-up on a full-fledged customizable feature flagging solution here: #1609

User Prompting

When prompting the user to allow an image in a deny-all scenario, we aligned on creating the IAR at the client-side.

Permissions

Only project admins should be allowed to create/edit IARs

[iwilltry42]

I'm proposing a slightly different way than completely enabling/disabling the feature with an install flag: An install flag that just toggles the deny-by-default (strict mode) behavior:
acorn install --image-allow-rules-strict-mode=true enabled deny-by-default. When off, we only deny images if someone explicitly added an IAR to a project (i.e. a single IAR locks down the project with only the images matching it allowed).

Here's an example

$ acorn install --image harbor.thklein.dev/public/acorn:20230517.01                                           
  ✔  Running Pre-install Checks                                                                                                                                                                                                                                                                                                                                                                                                            
  ✔  Installing ClusterRoles                                                                                                                                                                                                                                                                                                                                                                                                               
  ✔  Installing APIServer and Controller (image harbor.thklein.dev/public/acorn:20230517.01)                                                                                                                                                                                                                                                                                                                                               
  ✔  Waiting for controller deployment to be available                                                                                                                                                                                                                                                                                                                                                                                     
  ✔  Waiting for API server deployment to be available                                                                                                                                                                                                                                                                                                                                                                                     
  ✔  Waiting for registry server deployment to be available                                                                                                                                                                                                                                                                                                                                                                                
  ✔  Running Post-install Checks                                                                                                                                                                                                                                                                                                                                                                                                           
  ✔  Installation done

$ acorn run index.docker.io/sangeetha/myfirstacorn:v0.0.1          
dark-hill
STATUS: ENDPOINTS[] HEALTHY[] UPTODATE[] 
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] pending
STATUS: ENDPOINTS[] HEALTHY[0/1] UPTODATE[1] [containers: myweb1 ContainerCreating]
STATUS: ENDPOINTS[http://myweb1-dark-hill-6f5942e7.local.on-acorn.io => myweb1:80] HEALTHY[0/1] UPTODATE[1] [containers: myweb1 ContainerCreating]
STATUS: ENDPOINTS[http://myweb1-dark-hill-6f5942e7.local.on-acorn.io => myweb1:80] HEALTHY[0/1] UPTODATE[1] [containers: myweb1 is not ready]
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://myweb1-dark-hill-6f5942e7.local.on-acorn.io => myweb1:80] HEALTHY[1] UPTODATE[1] OK |
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://myweb1-dark-hill-6f5942e7.local.on-acorn.io => myweb1:80] HEALTHY[1] UPTODATE[1] OK |
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

$ kaf .local/imageallowrules/iar-scope-only.yaml
imageallowrule.api.acorn.io/iar-scope-only created

$ acorn run ghcr.io/k3d-io/k3d-proxy:5.4.9               
  ✗  ERROR:  invalid image ghcr.io/k3d-io/k3d-proxy:5.4.9: invalid image no Acornfile found

$ acorn run ghcr.io/acorn-io/library/hello-world:latest
patient-brook
STATUS: ENDPOINTS[] HEALTHY[] UPTODATE[] 
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] pending
STATUS: ENDPOINTS[] HEALTHY[0/1] UPTODATE[1] [containers: webapp ContainerCreating]
STATUS: ENDPOINTS[http://webapp-patient-brook-ca96a1ca.local.on-acorn.io => webapp:80] HEALTHY[0/1] UPTODATE[1] [containers: webapp ContainerCreating]

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://webapp-patient-brook-ca96a1ca.local.on-acorn.io => webapp:80] HEALTHY[1] UPTODATE[1] OK |
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

$ acorn run docker.io/iwilltry42/acorn:test.01         
  •  WARNING:  This application would like to use the image 'docker.io/iwilltry42/acorn:test.01'.
                 This could be VERY DANGEROUS to the cluster if you do not trust this
                 application. If you are unsure say no.

? Do you want to allow this app to use this (POTENTIALLY DANGEROUS) image? Yes
  ✗  ERROR:  image  is not allowed by any ImageAllowRule in this project

$ acorn run docker.io/iwilltry42/acorn:test.01
  •  WARNING:  This application would like to use the image 'docker.io/iwilltry42/acorn:test.01'.
                 This could be VERY DANGEROUS to the cluster if you do not trust this
                 application. If you are unsure say no.

? Do you want to allow this app to use this (POTENTIALLY DANGEROUS) image? No
  ✗  ERROR:  image docker.io/iwilltry42/acorn:test.01 is not allowed by any ImageAllowRule in this project

$ kdelf .local/imageallowrules/iar-scope-only.yaml
imageallowrule.api.acorn.io "iar-scope-only" deleted

$ acorn run docker.io/iwilltry42/acorn:test.01    
ancient-waterfall
STATUS: ENDPOINTS[] HEALTHY[] UPTODATE[] 
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] pending
STATUS: ENDPOINTS[] HEALTHY[0/1] UPTODATE[1] [containers: signednginx ContainerCreating]
STATUS: ENDPOINTS[http://signednginx-ancient-waterfall-2719a48b.local.on-acorn.io => signednginx:80] HEALTHY[0/1] UPTODATE[1] [containers: signednginx ContainerCreating]

┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://signednginx-ancient-waterfall-2719a48b.local.on-acorn.io => signednginx:80] HEALTHY[1] UPTODATE[1] OK |
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

$ acorn install --image harbor.thklein.dev/public/acorn:20230517.01 --image-allow-rules-strict-mode=true
  ✔  Running Pre-install Checks                                                                                                                                                                           
  ✔  Installing ClusterRoles                                                                                                                                                                              
  ✔  Installing APIServer and Controller (image harbor.thklein.dev/public/acorn:20230517.01)                                                                                                              
  ✔  Waiting for controller deployment to be available                                                                                                                                                    
  ✔  Waiting for API server deployment to be available                                                                                                                                                    
  ✔  Waiting for registry server deployment to be available                                                                                                                                               
  ✔  Running Post-install Checks                                                                                                                                                                          
  ✔  Installation done

$ acorn run docker.io/iwilltry42/acorn:test.01                                                          
  •  WARNING:  This application would like to use the image 'docker.io/iwilltry42/acorn:test.01'.
                 This could be VERY DANGEROUS to the cluster if you do not trust this
                 application. If you are unsure say no.

? Do you want to allow this app to use this (POTENTIALLY DANGEROUS) image? No
  ✗  ERROR:  image docker.io/iwilltry42/acorn:test.01 is not allowed by any ImageAllowRule in this project

$ acorn run ghcr.io/acorn-io/library/hello-world:latest                                                 
  •  WARNING:  This application would like to use the image 'ghcr.io/acorn-io/library/hello-world:latest'.
                 This could be VERY DANGEROUS to the cluster if you do not trust this
                 application. If you are unsure say no.

? Do you want to allow this app to use this (POTENTIALLY DANGEROUS) image? No
  ✗  ERROR:  image ghcr.io/acorn-io/library/hello-world:latest is not allowed by any ImageAllowRule in this project

$ kaf .local/imageallowrules/iar-scope-only.yaml       
imageallowrule.api.acorn.io/iar-scope-only created

$ acorn run ghcr.io/acorn-io/library/hello-world:latest
muddy-breeze
STATUS: ENDPOINTS[] HEALTHY[] UPTODATE[] 
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] pending
STATUS: ENDPOINTS[] HEALTHY[0/1] UPTODATE[1] [containers: webapp ContainerCreating]
STATUS: ENDPOINTS[http://webapp-muddy-breeze-d66cf766.local.on-acorn.io => webapp:80] HEALTHY[0/1] UPTODATE[1] [containers: webapp ContainerCreating]
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://webapp-muddy-breeze-d66cf766.local.on-acorn.io => webapp:80] HEALTHY[1] UPTODATE[1] OK |
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://webapp-muddy-breeze-d66cf766.local.on-acorn.io => webapp:80] HEALTHY[1] UPTODATE[1] OK |
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[iwilltry42]

It may still be a little rough around the edges and I don't like some smaller parts about it, but I'd love to get some feedback :)

[iwilltry42]

We discussed another change here.

• Generic --features flag similar to Kubernetes feature gates
• All-In or nothing for IARs, on means that strict mode is enabled everywhere
• No Project-Level setting
• Warning if off but IAR created
  • Not sure how to do this. The best place would be the validator for IARs, but there it would error right away. It would work for Acorn internally and for kubectl applications, but I think we don't want it to fail but just to inform the user that it won't have any effect.

[g-linville]

Lgtm, some tests are failing though