Skip to content

Commit

Permalink
fix(ci.jenkins.io-agents-1) remove AKS subnet NSG to avoid kubelet an…
Browse files Browse the repository at this point in the history
…d pod issues (#698)

Related to jenkins-infra/helpdesk#3954

After a pairing session with @smerle33 where we discovered that the
applications running in the AKS cluster's `kube-system` namespace where
failing due to timeouts, we realized that the NSG in charge fo setting
up rules for inbound agents is not shaped to handle an AKS cluster as it
blocks technical requests within the cluster:

- Pods (`10.100.0.0/14`) to internal DNS (`10.0.0.0`) server
- Kubelet (unknown CIDR) to pods (`10.100.0.0/14`)
- Pods (`10.100.0.0/14`) to Kubernetes services
- Etc.

This PR removes the NSG (for now) until we have deployed the cluster

Signed-off-by: Damien Duportal <damien.duportal@gmail.com>
  • Loading branch information
dduportal committed May 13, 2024
1 parent 48dc0ea commit ddf3f49
Showing 1 changed file with 1 addition and 36 deletions.
37 changes: 1 addition & 36 deletions ci.jenkins.io.tf
Original file line number Diff line number Diff line change
Expand Up @@ -78,27 +78,7 @@ module "ci_jenkins_io_aci_agents_sponsorship" {
controller_service_principal_id = module.ci_jenkins_io_sponsorship.controller_service_principal_id
}

module "ci_jenkins_io_kubernetes_agents_jenkins_sponsorship_1" {
providers = {
azurerm = azurerm.jenkins-sponsorship
}
source = "./.shared-tools/terraform/modules/azure-jenkinsinfra-inbound-agents"

service_fqdn = local.ci_jenkins_io_fqdn
inbound_agents_cidrs = [local.ci_jenkins_io_agents_1_pod_cidr]
inbound_agents_network_rg_name = data.azurerm_resource_group.public_jenkins_sponsorship.name
inbound_agents_network_name = data.azurerm_virtual_network.public_jenkins_sponsorship.name
inbound_agents_subnet_name = data.azurerm_subnet.ci_jenkins_io_kubernetes_sponsorship.name

controller_rg_name = module.ci_jenkins_io_sponsorship.controller_resourcegroup_name
controller_ips = compact([
module.ci_jenkins_io_sponsorship.controller_private_ipv4,
module.ci_jenkins_io_sponsorship.controller_public_ipv4
])
default_tags = local.default_tags
}

## Allow ci.jenkins.io to reach the private AKS cluster API (outbound in controller NSG and inbound in cluster subnet)
## Allow ci.jenkins.io to reach the private AKS cluster API
resource "azurerm_network_security_rule" "allow_outbound_https_from_cijio_to_cijenkinsio_agents_1_api" {
provider = azurerm.jenkins-sponsorship
name = "allow-out-https-from-cijio-to-cijenkinsio_agents-1-api"
Expand All @@ -114,21 +94,6 @@ resource "azurerm_network_security_rule" "allow_outbound_https_from_cijio_to_cij
resource_group_name = module.ci_jenkins_io_sponsorship.controller_resourcegroup_name
network_security_group_name = module.ci_jenkins_io_sponsorship.controller_nsg_name
}
resource "azurerm_network_security_rule" "allow_inbound_https_from_cijio_to_cijenkinsio_agents_1_api" {
provider = azurerm.jenkins-sponsorship
name = "allow-in-https-from-cijio-to-cijenkinsio_agents-1-api"
priority = 4000
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "443"
source_address_prefix = module.ci_jenkins_io_sponsorship.controller_private_ipv4 # Only private IPv4
# All IPs has the endpoint NIC may change inside this subnet
destination_address_prefixes = data.azurerm_subnet.ci_jenkins_io_kubernetes_sponsorship.address_prefixes
resource_group_name = module.ci_jenkins_io_kubernetes_agents_jenkins_sponsorship_1.inbound_agents_nsg_rg_name
network_security_group_name = module.ci_jenkins_io_kubernetes_agents_jenkins_sponsorship_1.inbound_agents_nsg_name
}

## Service DNS records
resource "azurerm_dns_cname_record" "ci_jenkins_io" {
Expand Down

0 comments on commit ddf3f49

Please sign in to comment.