Create a MQ broker in non-default vpc

[chakch]

Describe the bug
I'm not able to create a broker in a different VPC than the default one

Steps to reproduce
Create an MQ broker using the following manifest:
apiVersion: mq.services.k8s.aws/v1alpha1
kind: Broker
metadata:
name: my-broker
spec:
name: my-broker
autoMinorVersionUpgrade: false
deploymentMode: SINGLE_INSTANCE
engineType: RabbitMQ
engineVersion: "3.9.16"
hostInstanceType: "mq.t3.micro"
publiclyAccessible: false

storageType: string

securityGroups:

• SECURITY_GROUP_ID
  subnetIDs:
• SUBNET_ID_GROUP_ID
  users:
  • password:
    namespace: ack-system
    name: mq-password
    key: password
    groups: []
    consoleAccess: true
    username: admin
    =>
    2022-10-21T22:50:06.756Z ERROR controller.broker Reconciler error {"reconciler group": "mq.services.k8s.aws", "reconciler kind": "Broker", "name": "my-broker", "namespace": "ack-system", "error": "BadRequestException: Subnets must belong to the same VPC.\n{\n RespMetadata: {\n StatusCode: 400,\n RequestID: "e32fab71-2318-4a95-bcac-b1fcc27c86df"\n },\n ErrorAttribute: "subnetIds",\n Message_: "Subnets must belong to the same VPC."\n}"}

I think there is need to add VPCId ine the mq broker spec

Expected outcome
create an Broker in VPC different from the default one

Environment

• Kubernetes version:1.23
• Using EKS (yes/no), if so version? eks.2
• AWS service targeted (S3, RDS, etc.) MQ

[a-hilaly]

Hi @chakch are you able reproduce this issue using CDK or aws cli? just trying to understand if this ACK related or an API behaviour

[csantanapr]

I'm able to create a broker in a specific vpc other than the default one. the requirement is that if you specify a security group from a vpc you also need to specify the subnet ids on that vpc.

Here is a yaml that I used to create a broker

apiVersion: mq.services.k8s.aws/v1alpha1
kind: Broker
metadata:
  name: mq-eks-workshop
spec:
  name: mq-eks-workshop
  deploymentMode: SINGLE_INSTANCE
  engineType: ActiveMQ
  engineVersion: "5.15.8"
  hostInstanceType: "mq.t3.micro"
  publiclyAccessible: false
  autoMinorVersionUpgrade: false
  users:
    - password:
        namespace: default
        name: mq-eks-workshop
        key: password
      groups: []
      consoleAccess: true
      username: admin
  subnetIDs:
  - $(VPC_PRIVATE_SUBNET_ID_0)
  securityGroupRefs:
    - from: 
        name: mq-eks-workshop

/close

[ack-bot]

@csantanapr: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

I'm able to create a broker in a specific vpc other than the default one. the requirement is that if you specify a security group from a vpc you also need to specify the subnet ids on that vpc.

Here is a yaml that I used to create a broker

apiVersion: mq.services.k8s.aws/v1alpha1
kind: Broker
metadata:
 name: mq-eks-workshop
spec:
 name: mq-eks-workshop
 deploymentMode: SINGLE_INSTANCE
 engineType: ActiveMQ
 engineVersion: "5.15.8"
 hostInstanceType: "mq.t3.micro"
 publiclyAccessible: false
 autoMinorVersionUpgrade: false
 users:
   - password:
       namespace: default
       name: mq-eks-workshop
       key: password
     groups: []
     consoleAccess: true
     username: admin
 subnetIDs:
 - $(VPC_PRIVATE_SUBNET_ID_0)
 securityGroupRefs:
   - from: 
       name: mq-eks-workshop

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[csantanapr]

@chakch I didn't read closely the first time.
I was able to create a ActiveMQ specifying subnet and security-group from a non default vpc and worked, but now I tried to create a RabbitMQ and I can't create it on a none-default vpc

@a-hilaly here is a yaml with specifying subnet and security-group for rabbitmq

apiVersion: mq.services.k8s.aws/v1alpha1
kind: Broker
metadata:
  name: mq-eks-workshop
spec:
  name: mq-eks-workshop
  deploymentMode: SINGLE_INSTANCE
  engineType: RabbitMQ
  engineVersion: "3.10.10"
  hostInstanceType: "mq.t3.micro"
  publiclyAccessible: false
  autoMinorVersionUpgrade: false
  users:
    - password:
        namespace: default
        name: mq-eks-workshop
        key: password
      groups: []
      consoleAccess: true
      username: admin
  subnetIDs:
    - subnet-0e95abf6cd1647262
  securityGroups:
    - sg-0a97c3ca774be1446

The broker will be created fine, you can see it in the console with the in Running state with correct subnet and security group.

In the status of the broker you will see the error

status:
  ackResourceMetadata:
    ownerAccountID: "123456789"
    region: us-east-1
  conditions:
  - message: |-
      BadRequestException: Subnets and security groups must belong to the default VPC when only one is provided.
      {
        RespMetadata: {
          StatusCode: 400,
          RequestID: "5ffd35ab-3d65-4776-ade4-cf126af0c997"
        },
        ErrorAttribute: "subnetIds",
        Message_: "Subnets and security groups must belong to the default VPC when only one is provided."
      }
    status: "True"
    type: ACK.Recoverable
  - lastTransitionTime: "2022-11-22T01:12:40Z"
    message: Unable to determine if desired resource state matches latest observed
      state
    reason: |-
      BadRequestException: Subnets and security groups must belong to the default VPC when only one is provided.
      {
        RespMetadata: {
          StatusCode: 400,
          RequestID: "5ffd35ab-3d65-4776-ade4-cf126af0c997"
        },
        ErrorAttribute: "subnetIds",
        Message_: "Subnets and security groups must belong to the default VPC when only one is provided."
      }
    status: Unknown
    type: ACK.ResourceSynced

In Cloud Trail you see the go sdk sending an UpdateBroker call after the Broker is Running and includes in the parameters the security group, which is already set to that value.

"eventTime": "2022-11-22T00:37:42Z",
"eventSource": "amazonmq.amazonaws.com",
"eventName": "UpdateBroker",
"awsRegion": "us-east-1",
"sourceIPAddress": "18.235.50.27",
"userAgent": "aws-controllers-k8s/-v0.0.23 (GitCommit/740eedcdf35da5376ee0df354447476ca02d90f1; BuildDate/2022-11-15T17:22; CRDKind/; CRDVersion/) aws-sdk-go/1.44.93 (go1.17.13; linux; amd64)",
"errorCode": "BadRequestException",
"requestParameters": {
    "engineVersion": "3.10.10",
    "broker-id": "b-519d2e3f-c154-41c1-8a61-ffcb94c6b9f9",
    "securityGroups": [
        "sg-0db2eb2c65e795d4b"
    ],
    "autoMinorVersionUpgrade": false,
    "hostInstanceType": "mq.t3.micro"
},
"responseElements": {
    "message": "Changing security groups is not supported for RabbitMQ brokers.",
    "errorAttribute": "securityGroups"
},
"requestID": "2eabed1e-a66d-4613-8192-fedca30002d4",
"eventID": "1f396ee4-df9c-45ce-9079-e22878793a01",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,

In the logs

2022-11-22T01:20:45.606Z        INFO    ackrt   desired resource state has changed      {"account": "123456789", "role": "", "region": "us-east-1", "kind": "Broker", "namespace": "default", "name": "mq-eks-workshop", "is_adopted": false, "generation": 1, "diff": [{"Path":{"Parts":["Spec","AuthenticationStrategy"]},"A":null,"B":"simple"},{"Path":{"Parts":["Spec","EncryptionOptions"]},"A":null,"B":{"useAWSOwnedKey":true}},{"Path":{"Parts":["Spec","Logs"]},"A":null,"B":{"general":false}},{"Path":{"Parts":["Spec","MaintenanceWindowStartTime"]},"A":null,"B":{"dayOfWeek":"FRIDAY","timeOfDay":"22:00","timeZone":"UTC"}},{"Path":{"Parts":["Spec","StorageType"]},"A":null,"B":"ebs"}]}
2022-11-22T01:20:45.838Z        ERROR   controller.broker       Reconciler error        {"reconciler group": "mq.services.k8s.aws", "reconciler kind": "Broker", "name": "mq-eks-workshop", "namespace": "default", "error": "BadRequestException: Changing security groups is not supported for RabbitMQ brokers.\n{\n  RespMetadata: {\n    StatusCode: 400,\n    RequestID: \"a1e5ae66-86be-4624-8dc9-8dc631e6037f\"\n  },\n  ErrorAttribute: \"securityGroups\",\n  Message_: \"Changing security groups is not supported for RabbitMQ brokers.\"\n}"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227

I think having the Broker condition ACK.ResourceSynced not True affects the fieldexporters .

status: Unknown
type: ACK.ResourceSynced

It looks to fix this we need to find a way to avoid to include security group on this API call, or avoid the API call since the values are the same that already set on the Broker.

[a-hilaly]

Looks like we have multiple issues in here:

• Missing late initialization.
• Delta function could be improved.
• We should not try to update field that did not change.

[csantanapr]

@a-hilaly any updates on this issue, I want to be able to show users how to use ACK with rabbitmq and currently is not working

[ack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[ack-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle rotten