Skip to content

Latest commit

 

History

History
338 lines (254 loc) · 12 KB

README.adoc

File metadata and controls

338 lines (254 loc) · 12 KB

AMQ Broker Operator Helm Enhancement

The aim is to automate the creation of the AMQ Broker Custom Resources and at the same time enhance the broker.xml with a configuration that is not yet available in the operator. The helm chart keeps in sync the custom resources and the custom broker.xml.

Prerequisites

  1. AMQ Broker Operator must running in your target namespace.

Install AMQ Broker Helm Charts

helm upgrade --install mycluster amq-broker/

Test it

helm test mycluster

Custom Certificate Authority

The helm chart provides out of the box self signed certificate for each acceptor defined, but in case you want provide your custom certificate you need to create in advance the secret.

mkdir -p tls/{ca,broker}

Certificate Autority

This section is to simulate a private certificate authority.

  1. Certificate Authority (CA)

    openssl req -new -newkey rsa:2048 -x509 -keyout tls/ca/ca.key -out tls/ca/ca.crt -days 365 -subj "/CN=mycompany.com"
  2. Create a Truststore

        keytool -import -storepass password -file tls/ca/ca.crt -alias mycompany.com -keystore tls/ca/client.ts

Broker Certificate

  1. Broker Key

    keytool -genkeypair -storepass password -keyalg RSA -keysize 2048 -dname "CN=mycluster-amq-broker-*-svc-rte-brk.apps-crc.testing" -ext san="dns:mycluster-amq-broker-all-0-svc-rte-brk.apps-crc.testing,dns:mycluster-amq-broker-all-0-svc.brk.svc" -alias broker -keystore tls/broker/broker.ks
  2. Certificate Signing Request (CSR)

    keytool -certreq -storepass password -keyalg rsa -alias broker -keystore tls/broker/broker.ks -file tls/broker/tls.csr
  3. Certificate Authority Sign

    openssl x509 -req -CA tls/ca/ca.crt -CAkey tls/ca/ca.key -in tls/broker/tls.csr -out tls/broker/tls.crt -days 365 -CAcreateserial
    
    keytool -import -v -trustcacerts -alias root -file tls/ca/ca.crt -keystore tls/broker/broker.ks
    keytool -import -v -trustcacerts -alias broker -file tls/broker/tls.crt -keystore tls/broker/broker.ks
    1. Verify the chain

      keytool -list -v -keystore tls/broker/broker.ks
  4. Import to truststore

    keytool -import -storepass password -file tls/broker/tls.crt -alias server -keystore tls/ca/client.ts

Create Secret

Note
Naming convention: <CustomResourceName>-<AcceptorName>-secret
oc create secret generic mycluster-amq-broker-all-secret \
  --from-file=tls/broker \
  --from-file=tls/ca/client.ts \
  --from-file=tls/ca/ca.crt \
  --from-literal=keyStorePassword=password \
  --from-literal=trustStorePassword=password
Important
Adopt with Helm
oc annotate secret/mycluster-amq-broker-all-secret meta.helm.sh/release-name=mycluster --overwrite
oc annotate secret/mycluster-amq-broker-all-secret meta.helm.sh/release-namespace=brk --overwrite
oc label secret/mycluster-amq-broker-all-secret app.kubernetes.io/managed-by=Helm --overwrite

Install with Custom Certificate

helm upgrade --install mycluster --set pki.self=false

Test External Client Communication

  1. Extract Client TrustStore

    oc extract secret/mycluster-amq-broker-all-secret --keys=client.ts

  2. Extract AMQ Endpoint

    export AMQ_ENDPOINT=$(oc get route mycluster-amq-broker-all-0-svc-rte --no-headers -o custom-columns=:.spec.host)

  3. Launch Artemis Producer Command

    ./artemis producer --user amq --password amq --message-count 10 --url="tcp://${AMQ_ENDPOINT}:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password"
    
    Connection brokerURL = tcp://mycluster-amq-broker-all-0-svc-rte-brk.apps-crc.testing:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password
    Producer ActiveMQQueue[TEST], thread=0 Started to calculate elapsed time ...
    
    Producer ActiveMQQueue[TEST], thread=0 Produced: 10 messages
    Producer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 0 s
    Producer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second : 79 milli seconds
  4. Launch Artemis Consumer Command

./artemis consumer --user amq --password amq --message-count 10 --url="tcp://${AMQ_ENDPOINT}:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password"

Connection brokerURL = tcp://mycluster-amq-broker-all-0-svc-rte-brk.apps-crc.testing:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password
Consumer:: filter = null
Consumer ActiveMQQueue[TEST], thread=0 wait until 10 messages are consumed
Consumer ActiveMQQueue[TEST], thread=0 Consumed: 10 messages
Consumer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 0 s
Consumer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second : 17 milli seconds
Consumer ActiveMQQueue[TEST], thread=0 Consumed: 10 messages
Consumer ActiveMQQueue[TEST], thread=0 Consumer thread finished

Bridge Scenario

  1. Create two OpenShift project.

    oc new-project brk1
    oc new-project brk2
  2. Install AMQ Broker Operator for both the projects.

  3. Create a NetworkPolicy to allow connection from brk1 to brk2.

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: mycluster1-amq-broker-all
      labels:
        app.kubernetes.io/name: amq-broker
        app.kubernetes.io/instance: mycluster2
    spec:
      podSelector:
        matchLabels:
          app.kubernetes.io/name: amq-broker
          app.kubernetes.io/instance: mycluster2
      ingress:
        - from:
            - namespaceSelector:
                matchLabels:
                  app.kubernetes.io/instance: mycluster
          ports:
          - port: 61617
            protocol: TCP
  4. Label brk1 namespace with app.kubernetes.io/instance: mycluster

    oc label namespace brk1 app.kubernetes.io/instance=mycluster

  5. Install AMQ Broker Helm Chart on brk2

    helm upgrade --install mycluster2 amq-broker -n brk2 -f examples/bridge/values.brk2.yaml

  6. Test the installation on brk2

    helm test mycluster2 -n brk2

  7. In case of SSL Connector to brk2, create a secret on brk1 to able to connect to brk2.

    mkdir -p examples/bridge/brk2-certs
    
    oc extract secret/mycluster2-amq-broker-all-secret --to=examples/bridge/brk2-certs -n brk2
    
    oc create secret generic mycluster2-amq-broker-all-brk2-secret --from-file=examples/bridge/brk2-certs/ -n brk1
  8. Install AMQ Broker Helm Chart on brk1

    helm upgrade --install mycluster amq-broker -n brk1 -f examples/bridge/values.brk1.yaml

  9. Test the installation on brk1

    helm test mycluster -n brk1

Test the bridge connection

Note
Check Test External Client Communication to launch producer and consumer command.
  1. Produce message on brk1

./artemis producer --user amq --password amq --message-count 10 --url="tcp://${AMQ_ENDPOINT}:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password" --destination=bridge.test.address
  1. Consume message on brk2

./artemis consumer --user amq --password amq --message-count 10 --url="tcp://${AMQ_ENDPOINT}:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password" --destination=bridge.test.forwarding

Uninstall it

helm uninstall mycluster --no-hooks

Failed Status

In case of failed installation due to missing role rights, AMQ Broker operator, tests failed or others, please make sure to clean up all the pending resources.

oc delete all -lapp.kubernetes.io/name=amq-broker

Progress

Table 1. Progress
Task Status

Add Address CRD logic

DONE

Patch AMQ Broker with custom dynamic configuration

DONE

Test Internal Connection Implementation

DONE

SSL Selfsigned Implementation

DONE

Test with External Client Implementation

DONE

Keystore and Truststore Password

DONE

SSL Custom CA Implementation

DONE #2

User Management

DONE #3

Diverts Implementation

DONE #1

Bridge Implementation

#1

Network Policy

#4

Make sure that the operator creates all the Addresses

TODO

High Availability and How scale down controller actives without the operator.

TODO

E2E Test- Migrate one standalone broker

TODO

Pseudo Code

  1. Install Operator to specific namespace on the cluster

    1. This can be provided by the cluster-admin as namespaced installation and should give the right role to manage the AMQ Broker CRD.

      Note
      You cannot create more than one broker deployment in a given OpenShift project by deploying multiple broker Custom Resource (CR) instances. However, when you have created a broker deployment in a project, you can deploy multiple CR instances for addresses. Reference.
  2. [pre-install hook] Restore AMQ Broker Operator

  3. [install/upgrade] Create kind: ActiveMQArtemis and ActiveMQArtemisAddress

  4. [install/upgrade] Create Config Map with custom broker xml.

  5. [post-install hook] Shutdown the AMQ Broker Operator

  6. [post-install hook] Adopt the AMQ Broker resource to Helm:

    annotations:
      meta.helm.sh/release-name: release-name
      meta.helm.sh/release-namespace: namespace-name
    labels:
      app.kubernetes.io/managed-by: Helm
  7. [post install] Adjust AMQ Broker Stateful set to use the custom broker xml.

    1. Set BROKER_XML environment variable with your custom broker.xml.

  8. [test] Verify the installation is correct.

Note
A *-hook install image requires oc client quay.io/openshift/origin-cli:4.6 and running with edit role on the specific namespace.

Important

  • In AMQ Broker 7.7, if you want to configure any of the following items, you must add the appropriate configuration to the main CR instance before deploying the CR for the first time.

    • Address settings

    • The size of the Persistent Volume Claim (PVC) required by each broker in a deployment for persistent storage

    • Limits and requests for memory and CPU for each broker in a deployment

  • During an active scaling event, any further changes that you apply are queued by the Operator and executed only when scaling is complete. For example, suppose that you scale the size of your deployment down from four brokers to one. Then, while scaledown is taking place, you also change the values of the broker administrator user name and password. In this case, the Operator queues the user name and password changes until the deployment is running with one active broker.

  • All CR changes – apart from changing the size of your deployment, or changing the value of the expose attribute for acceptors, connectors, or the console – cause existing brokers to be restarted. If you have multiple brokers in your deployment, only one broker restarts at a time.

  • To configure address and queue settings for broker deployments on OpenShift Container Platform, you add configuration to an addressSettings section of the main Custom Resource (CR) instance for the broker deployment. This contrasts with standalone deployments on Linux or Windows, for which you add configuration to an address-settings element in the broker.xml configuration file.

  • The format used for the names of configuration items differs between OpenShift Container Platform and standalone broker deployments. For OpenShift Container Platform deployments, configuration item names are in camel case, for example, defaultQueueRoutingType. By contrast, configuration item names for standalone deployments are in lower case and use a dash (-) separator, for example, default-queue-routing-type.

  • Addresses are created by the AMQ Broker Operator using Artemis Jolokia and MBean.