Add support for ApplicationGateway #3176

vimorra · 2023-08-02T13:20:09Z

What this PR does / why we need it:
Add support for Application gateway service
Special notes for your reviewer:
Still working on the sample
How does this PR make you feel:

If applicable:

this PR contains documentation
this PR contains tests

v2/api/network/customizations/network_security_group_extension_test.go

matthchr · 2023-08-04T20:17:52Z

v2/api/network/v1api20220701/structure.txt

+│   ├── Owner: github.com/Azure/azure-service-operator/v2/api/resources/v1apiv20191001.ResourceGroup
+│   ├── Spec: Object (37 properties)
+│   │   ├── AuthenticationCertificates: Object (2 properties)[]
+│   │   │   ├── Data: *string


Is this just a public key, or does it contain private details?

It looks like probably just public data based on the documentation, in which case this is fine.

This is the public key certificate that is uploaded to enable mtls on the application gateway, this is used by the app gateway to trust the client. It not contains sensitive data of the users.

v2/api/network/v1api20220701/structure.txt

matthchr · 2023-08-04T20:37:08Z

v2/internal/controllers/crd_networking_applicationgateway_test.go

+		},
+	}
+	publicIPAddress := newPublicIp(tc, testcommon.AsOwner(rg))
+	tc.CreateResourceAndWait(publicIPAddress)


minor, you can combined all of these into a single create+wait

tc.CreateResourcesAndWait(publicIPAddress, vnet, subnet)

v2/samples/network/v1api20220701/v1api20220701_applicationgateway.yaml

matthchr · 2023-08-04T20:50:53Z

v2/samples/network/v1api20220701/v1api20220701_applicationgateway.yaml

+  - name: app-gw-http-listner-1
+    frontendIPConfiguration:
+      reference:  
+        armId: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/<rgName>/providers/Microsoft.Network/applicationGateways/<appgatewayname>/frontendIPConfigurations/<feIpConfigName>


You can replace <rgname> with aso-sample-rg sincethat's the sample rg which the sample is written in the context of.
Same with appgatewayname, it can be aso-sample-application-gateway, the name of this application gateway. Use the actual feIpCOnfigName too.

Same for all the refs below as well.

v2/samples/network/v1api20220701/v1api20220701_applicationgateway.yaml

matthchr · 2023-08-04T20:51:38Z

v2/samples/network/v1api20220701/v1api20220701_applicationgateway.yaml

+        armId: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/<rgName>/providers/Microsoft.Network/applicationGateways/<appgatewayname>/frontendIPConfigurations/<feIpConfigName>
+    frontendPort:
+      reference:  
+        armId: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/<rgName>/providers/Microsoft.Network/applicationGateways/<appgatewayname>/frontendPorts/<fePortName>


@super-harsh, is this going to pass sample validation? Do we deal with swapping subid into raw ARM refs?

v2/internal/controllers/crd_networking_applicationgateway_test.go

theunrepentantgeek

Still a few cleanup items required (see comments by @matthchr) - including more reliance on Go type inference, which will clean up the test in controllers some. I'll have a look at the test and work out what the failure means.

theunrepentantgeek · 2023-09-01T03:25:26Z

v2/azure-arm.yaml

@@ -1561,6 +1561,51 @@ objectModelConfiguration:
        $exportAs: VirtualNetworksVirtualNetworkPeering
        $supportedFrom: v2.0.0-alpha.1
    2022-07-01:
+      ApplicationGateway:
+        $export: true
+        $supportedFrom: v2.3.0


I don't think this will make it for v2.3.0.

Suggested change

$supportedFrom: v2.3.0

$supportedFrom: v2.4.0

fixed in updated commit

v2/azure-arm.yaml

theunrepentantgeek · 2023-09-01T03:32:41Z

v2/internal/controllers/crd_networking_applicationgateway_test.go

+		{
+			//KeyVaultSecretId: to.Ptr[string]("https://keyvaultname.vault.azure.net/secrets/secretname"),
+			Name: to.Ptr[string](subResname),
+			Data: to.Ptr[string](`MIIKEQIBAzCCCdcGCSqGSIb3DQEHAaCCCcgEggnEMIIJwDCCBHcGCSqGSIb3DQEH


If this is an embedded certificate, what's its expiry?

The test should either be self-contained (creating its own certificate, garanteeing that it's not stale), or at the very least there should be a comment here indicating when the certificate will expire so that a future dev knows what needs doing when the test fails.

theunrepentantgeek

We're keen to get this PR merged, it's looking very close to finished.

There are just two items outstanding:

The signature of getFrontendPortsARMID() needs to be modified to avoid the brittle nature of passing around arbitrary maps.
We can't commit secrets to the repo, so we need to tidy up the use of certificates.

I believe we have infrastructure available through the TestContext (tc) to create the required secrets on the fly, so that nothing needs to be hard coded. If you find the capability you need is missing, let us know and we'll fill in the gap for you.

theunrepentantgeek · 2023-09-26T00:51:02Z

v2/internal/controllers/crd_networking_applicationgateway_test.go

+	return subRes, subResARMID
+}
+
+func getFrontendPortsARMID(tc *testcommon.KubePerTestContext, rg *resources.ResourceGroup, params map[string]string) (string, error) {


As previously highlighted by @matthchr this method signature needs to be changed - passing around parameter in a map like this is brittle, creating a landmine for future maintainers where a simple change causes very non-obvious errors. We strongly prefer for problems to be identified by the compiler where possible.

There are two good alternatives.

Option 1: Replace the map with four parameters like this:

func getFrontendPortsARMID( tc *testcommon.KubePerTestContext, rg *resources.ResourceGroup, type1 string, value1 string, type2 string, value2 string, ) (string, error) {

calls would change to this form:

subResARMID, err := getFrontendPortsARMID(tc, rg, "applicationGateways", appGatewayName, "backendAddressPools", subResname)

Option 2: Replace the map with a varidic parameter:

func getFrontendPortsARMID( tc *testcommon.KubePerTestContext, rg *resources.ResourceGroup, params ...string, ) (string, error) {

This would allow any number of parameters, so you might want to include a check to panic if someone passes an odd number.

Calling would be the same:

subResARMID, err := getFrontendPortsARMID(tc, rg, "applicationGateways", appGatewayName, "backendAddressPools", subResname)

fixed with option 1. @theunrepentantgeek please review

theunrepentantgeek · 2023-09-26T00:53:40Z

v2/internal/tests/appgwcert.pfx

We don't want to have secrets committed to the repo, for all the usual policy reasons. I expect this would be promptly flagged as a security risk regardless of whether it's used elsewhere.

Remove this file.

theunrepentantgeek · 2023-09-26T00:54:05Z

v2/internal/tests/appgwcert64.pfx

Remove this file for the same reasons as appgwcert.pfx.

v2/samples/network/v1api20220701/v1api20220701_applicationgateway.yaml

matthchr · 2023-10-02T23:03:17Z

If you're busy @vimorra and don't have the time to finish updating based on our feedback, we can pick this up and get it over the line. No promises that it makes it into 2.4.0 but we'll try.

vimorra · 2023-10-03T14:29:44Z

If you're busy @vimorra and don't have the time to finish updating based on our feedback, we can pick this up and get it over the line. No promises that it makes it into 2.4.0 but we'll try.

yes I will do my best, fixed the comment 1 about the signature of getFrontendPortsARMID. I will try to fix the certificate issue by adding the isSecret

vimorra · 2023-10-03T14:33:17Z

We're keen to get this PR merged, it's looking very close to finished.

There are just two items outstanding:

The signature of getFrontendPortsARMID() needs to be modified to avoid the brittle nature of passing around arbitrary maps.

We can't commit secrets to the repo, so we need to tidy up the use of certificates.

I believe we have infrastructure available through the TestContext (tc) to create the required secrets on the fly, so that nothing needs to be hard coded. If you find the capability you need is missing, let us know and we'll fill in the gap for you.

not clear which option you are suggesting if create the certificate in an automated way or mark the attribute as "isSecret" ? in any case con you provide an example @theunrepentantgeek

theunrepentantgeek · 2023-10-03T19:23:40Z

I went back and had another look, and it's actually both of the issues you've identified.

The issue I was referring to above is about hard coded certificates:

the file appgwcert64.pfx
the file appgwcert.pfx
the sslCertificates.data block in the sample v1api20220701_applicationgateway.yaml

Checking certificates into repos is a security issue even if they only contain public keys. They also expire, which will result in the test failing in the future.

The second issue is the authenticationCertificates.Data property on ApplicationGateway.spec - this should be configured with $isSecret: true so that the value is pulled from the cluster. Apologies for missing this earlier.

This will have two benefits.

When the certificate is rotated, updating the secret will result in ASO updating the ApplicationGateway, much easier (and more automated) than having to redeploy everything from scratch. (Rotating the certificate directly in Azure isn't an option because ASO will proactively restore the configured certificate next time it reconciles the resource.)
The certificate won't be disclosed to someone looking at the YAML for the resource. If the cert contains only a public key, this wouldn't matter - but people do take shortcuts and make mistakes, and if they use a cert containing a private key disclosure is a security risk.

We believe there's infrastructure available on the TestContext used in your test that should allow you to create a certificate on the fly as a part of the test. If this is missing, or if you can't find it, ping us and we'll dig in further.

vimorra · 2023-10-04T07:41:57Z

@matthchr, @super-harsh and @theunrepentantgeek I need your support to understand how to generate an on the fly ssl certificate.
Meanwhile I have fixed the:

$isSecret: true point on all the resources that point to secrets and certificates
fixed the signature of $isSecret: true

theunrepentantgeek · 2023-10-06T03:20:47Z

@vimorra - I'm looking into generating the SSL certificate; hoping to put together a sample for ASO users that will be relevant to your needs too. I'll update back here as that progresses.

Use HTTP protocol for Application Gateway in controller test to avoid certs and fix sample

vimorra · 2023-10-24T06:59:15Z

Hi @theunrepentantgeek and @matthchr with the support of @super-harsh the PR should be ready to be merged
, shall we look together at how to solve the conflicts or I can assume that the incoming version is the one to merge?

super-harsh · 2023-10-24T20:54:59Z

@vimorra An easy way to deal with these conflicts is to merge main into your branch and run task controller:generate-types task and commit the files.

into feature-a

…e-service-operator into feature-app-gateway

…into feature-app-gateway

codecov-commenter · 2023-10-25T16:25:08Z

Codecov Report

Merging #3176 (d1ebad0) into main (db857cf) will decrease coverage by 0.12%.
The diff coverage is 58.04%.

@@            Coverage Diff             @@
##             main    #3176      +/-   ##
==========================================
- Coverage   54.32%   54.21%   -0.12%     
==========================================
  Files        1566     1570       +4     
  Lines      655434   663722    +8288     
==========================================
+ Hits       356073   359844    +3771     
- Misses     241588   245516    +3928     
- Partials    57773    58362     +589

Files	Coverage Δ
...zations/application_gateway_extension_types_gen.go	`100.00% <100.00%> (ø)`
...ork/v1api20220701/application_gateway_types_gen.go	`47.22% <ø> (ø)`
...pi/network/v1api20220701/bastion_host_types_gen.go	`54.08% <ø> (ø)`
...rk/v1api20220701/nat_gateway_spec_arm_types_gen.go	`33.33% <ø> (ø)`
...api/network/v1api20220701/nat_gateway_types_gen.go	`57.76% <ø> (+0.35%)`	⬆️
...rk/v1api20220701/storage/bastion_host_types_gen.go	`52.72% <ø> (ø)`
...ork/v1api20220701/storage/nat_gateway_types_gen.go	`52.72% <ø> (ø)`
...20220701/application_gateway_spec_arm_types_gen.go	`33.33% <33.33%> (ø)`
...i20220701/storage/application_gateway_types_gen.go	`52.72% <52.72%> (ø)`
...2/internal/controllers/controller_resources_gen.go	`83.60% <61.53%> (-0.72%)`	⬇️

... and 35 files with indirect coverage changes

theunrepentantgeek · 2023-10-25T19:34:29Z

/ok-to-test sha=d1ebad0

theunrepentantgeek · 2023-10-25T19:34:53Z

/azp run

azure-pipelines · 2023-10-25T19:35:04Z

Azure Pipelines successfully started running 1 pipeline(s).

first commit app gateway

723285a

vimorra requested review from davefellows, theunrepentantgeek, matthchr, babbageclunk and super-harsh as code owners August 2, 2023 13:20

vimorra added 2 commits August 2, 2023 15:56

more code in the sample

dada4b5

complete sample and fix lint issues

5501e67

matthchr requested changes Aug 4, 2023

View reviewed changes

vimorra added 2 commits August 23, 2023 11:11

fix pr comments

0a94048

add removed files in first commit

5211fd4

matthchr added this to the v2.4.0 milestone Aug 28, 2023

theunrepentantgeek changed the title ~~first commit app gateway~~ Add support for ApplicationGateway Sep 1, 2023

theunrepentantgeek reviewed Sep 1, 2023

View reviewed changes

vimorra added 3 commits September 1, 2023 07:13

fix pr comments and add test recording

e7b0de9

update supported from field

35f0209

adding the self sig. cert for the app gateway test

b7de1db

theunrepentantgeek modified the milestones: v2.5.0, v2.4.0 Sep 7, 2023

theunrepentantgeek requested changes Sep 26, 2023

View reviewed changes

fix getFrontendPortsARMID comment on PR

17a3a13

add isSecrets to appgateway subresources

1c64359

super-harsh added 2 commits October 20, 2023 20:40

Fixups for controller test and add recording

fac16bb

Fix sample; Add sample refs; Record samples test

67f5781

Merge pull request #1 from super-harsh/minor-fixups

d271553

Use HTTP protocol for Application Gateway in controller test to avoid certs and fix sample

vimorra added 16 commits October 24, 2023 21:35

Merge branch 'main' of https://github.com/vimorra/azure-service-operator

5848d2c

into feature-a

merge main

920dc40

fix lint error

efa6838

fix lint error only app-gtw

3051a85

Merge branch 'feature-app-gateway' of https://github.com/vimorra/azur…

d97e5f3

…e-service-operator into feature-app-gateway

Merge branch 'main' of https://github.com/Azure/azure-service-operator …

2ead903

…into feature-app-gateway

test restore genruntime

dcc8264

rever genruntime

33d1df1

revert genruntime

d680168

fixcdn and compute

41e4171

fix container instance

2cbfbb2

fix all genruntime

256ccc1

fix remaining genruntime

c5534fc

test fix deepcopy appgateway

a0650ba

fix genruntime appp gateway

ac14012

fix genruntime app gateway

d1ebad0

super-harsh approved these changes Oct 26, 2023

View reviewed changes

theunrepentantgeek approved these changes Oct 26, 2023

View reviewed changes

theunrepentantgeek enabled auto-merge October 27, 2023 01:45

matthchr approved these changes Oct 27, 2023

View reviewed changes

theunrepentantgeek added this pull request to the merge queue Oct 27, 2023

Merged via the queue into Azure:main with commit 6daa0d3 Oct 27, 2023
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for ApplicationGateway #3176

Add support for ApplicationGateway #3176

vimorra commented Aug 2, 2023 •

edited

matthchr Aug 4, 2023

vimorra Aug 23, 2023 •

edited

matthchr Aug 4, 2023

vimorra Aug 23, 2023

matthchr Aug 4, 2023

vimorra Aug 24, 2023

matthchr Aug 4, 2023

theunrepentantgeek left a comment •

edited

theunrepentantgeek Sep 1, 2023

vimorra Sep 1, 2023

theunrepentantgeek Sep 1, 2023

theunrepentantgeek left a comment

theunrepentantgeek Sep 26, 2023

vimorra Oct 3, 2023

theunrepentantgeek Sep 26, 2023

theunrepentantgeek Sep 26, 2023

matthchr commented Oct 2, 2023

vimorra commented Oct 3, 2023

vimorra commented Oct 3, 2023

theunrepentantgeek commented Oct 3, 2023

vimorra commented Oct 4, 2023

theunrepentantgeek commented Oct 6, 2023

vimorra commented Oct 24, 2023

super-harsh commented Oct 24, 2023

codecov-commenter commented Oct 25, 2023

theunrepentantgeek commented Oct 25, 2023

theunrepentantgeek commented Oct 25, 2023

azure-pipelines bot commented Oct 25, 2023

Add support for ApplicationGateway #3176

Add support for ApplicationGateway #3176

Conversation

vimorra commented Aug 2, 2023 • edited

Choose a reason for hiding this comment

vimorra Aug 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theunrepentantgeek left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theunrepentantgeek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthchr commented Oct 2, 2023

vimorra commented Oct 3, 2023

vimorra commented Oct 3, 2023

theunrepentantgeek commented Oct 3, 2023

vimorra commented Oct 4, 2023

theunrepentantgeek commented Oct 6, 2023

vimorra commented Oct 24, 2023

super-harsh commented Oct 24, 2023

codecov-commenter commented Oct 25, 2023

Codecov Report

theunrepentantgeek commented Oct 25, 2023

theunrepentantgeek commented Oct 25, 2023

azure-pipelines bot commented Oct 25, 2023

vimorra commented Aug 2, 2023 •

edited

vimorra Aug 23, 2023 •

edited

theunrepentantgeek left a comment •

edited