Add Broker APB source to project #939

djzager · 2018-05-09T15:12:18Z

Currently, the broker-apb and the broker are loosely coupled. They are not specifically tied together, except the broker-apb using latest by default. In the future, we want a couple of things for our broker and broker-apb:

Test the broker using the broker-apb in CI (also testing the broker-apb provision/deprovision)
Have a means to build a broker-apb that can reliably provision/deprovision a specific version of the broker

Allowing the broker-apb source to live with the broker project gives us more flexibility going forward as changes are introduced to either 1) the way the broker is deployed (CRDs as a recent example) 2) the broker itself.

Fixes: #912 #931 #969

dymurray

Worked successfully for me on first try using install.yml. Great work!

jmrodri · 2018-05-12T02:32:14Z

Here is the run after doing make deploy. I don't think this EXIT_CODE line is correct:

EXIT_CODE=$(${CMD} get pod -n ${APB_NAME} "${APB_NAME}" -o go-template="{{ range .status.containerStatuses }}{{.state.terminated.exitCode}}{{ end }}")

Here is the output of make deploy

[jesusr@speed3 ansible-service-broker{djzager-add-apb}]$ make deploy
env GOOS=linux go build -i -ldflags="-s -s" -o "/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/build"/broker ./cmd/broker
env GOOS=linux go build -i -ldflags="-s -s" -o "/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/build"/migration ./cmd/migration
env GOOS=linux go build -i -ldflags="-s -s" -o "/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/build"/dashboard-redirector ./cmd/dashboard-redirector
docker build -f "/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/build"/Dockerfile-localdev -t docker.io/ansibleplaybookbundle/origin-ansible-service-broker:latest "/home/jesusr/dev/src/github.c
om/openshift/ansible-service-broker/build"
Sending build context to Docker daemon 90.01 MB
Step 1/15 : FROM centos:7
 ---> e934aafc2206
Step 2/15 : MAINTAINER Ansible Service Broker Community
 ---> Using cache
 ---> 2de87dd187a6
Step 3/15 : ENV USER_NAME ansibleservicebroker USER_UID 1001 BASE_DIR /opt/ansibleservicebroker
 ---> Using cache
 ---> 138ad5341aa3
Step 4/15 : ENV HOME ${BASE_DIR}
 ---> Using cache
 ---> d2e884286852
Step 5/15 : RUN mkdir -p ${BASE_DIR} ${BASE_DIR}/etc  && useradd -u ${USER_UID} -r -g 0 -M -d ${BASE_DIR} -b ${BASE_DIR} -s /sbin/nologin -c "ansibleservicebroker user" ${USER_NAME}  && chown -R ${USER_NAME}:0 $
{BASE_DIR}  && chmod -R g+rw ${BASE_DIR} /etc/passwd
 ---> Using cache
 ---> ccd3a4270e2f
Step 6/15 : RUN yum -y update  && yum -y install epel-release centos-release-openshift-origin  && yum -y install origin-clients net-tools bind-utils  && yum clean all
 ---> Using cache
 ---> 45d5a615ab45
Step 7/15 : RUN mkdir /var/log/ansible-service-broker     && touch /var/log/ansible-service-broker/asb.log     && mkdir /etc/ansible-service-broker
 ---> Using cache
 ---> 72a86a407603
Step 8/15 : COPY entrypoint.sh /usr/bin/
 ---> Using cache
 ---> 78ad4d458a1c
Step 9/15 : COPY broker /usr/bin/asbd
 ---> Using cache
 ---> fbca146f90b3
Step 10/15 : COPY migration /usr/bin/migration
 ---> Using cache
 ---> 23514e020eac
Step 11/15 : COPY dashboard-redirector /usr/bin/dashboard-redirector
 ---> Using cache
 ---> 262eefedbad2
Step 12/15 : RUN chown -R ${USER_NAME}:0 /var/log/ansible-service-broker  && chown -R ${USER_NAME}:0 /etc/ansible-service-broker  && chmod -R g+rw /var/log/ansible-service-broker /etc/ansible-service-broker
 ---> Using cache
 ---> 135c05ee30ae
Step 13/15 : USER ${USER_UID}
 ---> Using cache
 ---> e16e0c7b2a8b
Step 14/15 : RUN sed "s@${USER_NAME}:x:${USER_UID}:@${USER_NAME}:x:\${USER_ID}:@g" /etc/passwd > ${BASE_DIR}/etc/passwd.template
 ---> Using cache
 ---> 4b1468a790dd
Step 15/15 : ENTRYPOINT entrypoint.sh
 ---> Using cache
 ---> d0aef343fdbe
Successfully built d0aef343fdbe

Remember you need to push your image before calling make deploy
    docker push docker.io/ansibleplaybookbundle/origin-ansible-service-broker:latest
docker build -f apb/Dockerfile -t docker.io/ansibleplaybookbundle/automation-broker-apb:latest apb
Sending build context to Docker daemon 69.63 kB
Step 1/7 : FROM ansibleplaybookbundle/apb-base:canary
 ---> fba19a3f7e67
Step 2/7 : LABEL "com.redhat.apb.spec" "LS0tCnZlcnNpb246IDEuMApuYW1lOiBhdXRvbWF0aW9uLWJyb2tlci1hcGIKZGVzY3JpcHRpb246IGRlcGxveXMgYXV0b21hdGlvbiBicm9rZXIgZm9yIHRoZSBzZXJ2aWNlLWNhdGFsb2cKYmluZGFibGU6ICJGYWxzZSIKYXN
5bmM6IG9wdGlvbmFsCm1ldGFkYXRhOgogIGRpc3BsYXlOYW1lOiBBdXRvbWF0aW9uIEJyb2tlciAoQVBCKQogIGxvbmdEZXNjcmlwdGlvbjoKICAgIEFuIEFQQiBmb3IgbWFuYWdpbmcgdGhlIGF1dG9tYXRpb24gYnJva2VyIGluIGEgY2x1c3RlcgogIGRlcGVuZGVuY2llczogW1
0KICBwcm92aWRlckRpc3BsYXlOYW1lOiAiUmVkIEhhdCwgSW5jLiIKcGxhbnM6CiAgLSBuYW1lOiBtYWluCiAgICBkZXNjcmlwdGlvbjogTWFuYWdlIHRoZSBhdXRvbWF0aW9uIGJyb2tlciBpbiBhIGNsdXN0ZXIKICAgIGZyZWU6ICJUcnVlIgogICAgbWV0YWRhdGE6CiAgICAgI
GRpc3BsYXlOYW1lOiBEZWZhdWx0CiAgICAgIGxvbmdEZXNjcmlwdGlvbjoKICAgICAgICBUaGlzIHBsYW4gZGVwbG95cyB0aGUgYXV0b21hdGlvbiBicm9rZXIKICAgICAgY29zdDogJDAuMDAKICAgIHBhcmFtZXRlcnM6CiAgICAgIC0gbmFtZTogYnJva2VyX25hbWUKICAgICAg
ICB0aXRsZTogTmFtZSBvZiB0aGUgYnJva2VyCiAgICAgICAgdHlwZTogc3RyaW5nCiAgICAgICAgZGVmYXVsdDogYXV0b21hdGlvbi1icm9rZXIKICAgICAgLSBuYW1lOiBicm9rZXJfbmFtZXNwYWNlCiAgICAgICAgdGl0bGU6IE5hbWVzcGFjZSBvZiB0aGUgYnJva2VyCiAgICA
gICAgdHlwZTogc3RyaW5nCiAgICAgICAgZGVmYXVsdDogYXV0b21hdGlvbi1icm9rZXIKICAgICAgLSBuYW1lOiBicm9rZXJfaW1hZ2UKICAgICAgICB0aXRsZTogQ29udGFpbmVyIGltYWdlIGZvciB0aGUgYnJva2VyCiAgICAgICAgdHlwZTogc3RyaW5nCiAgICAgICAgZGVmYX
VsdDogYW5zaWJsZXBsYXlib29rYnVuZGxlL29yaWdpbi1hbnNpYmxlLXNlcnZpY2UtYnJva2VyOmxhdGVzdAo="
 ---> Using cache
 ---> dacf8ae83496
Step 3/7 : ADD playbooks /opt/apb/actions
 ---> Using cache
 ---> 6b769c22b717
Step 4/7 : ADD . /opt/ansible/roles/automation-broker-apb
 ---> Using cache
 ---> d57095915eb6
Step 5/7 : RUN yum -y install epel-release openssl && yum clean all
 ---> Using cache
 ---> 4585c2ee7d77
Step 6/7 : RUN chmod -R g=u /opt/{ansible,apb}
 ---> Using cache
 ---> 8c91047a8478
Step 7/7 : USER apb
 ---> Using cache
 ---> 5c5f20c9830f
Successfully built 5c5f20c9830f
APB_IMAGE=docker.io/ansibleplaybookbundle/automation-broker-apb:latest BROKER_IMAGE=docker.io/ansibleplaybookbundle/origin-ansible-service-broker:latest ACTION="provision" ./scripts/deploy.sh
/home/jesusr/bin/kubectl
---
apiVersion: v1
kind: Namespace
metadata:
  name: automation-broker-apb

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: automation-broker-apb
  namespace: automation-broker-apb

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: automation-broker-apb
roleRef:
  name: cluster-admin
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: automation-broker-apb
  namespace: automation-broker-apb

---
apiVersion: v1
kind: Pod
metadata:
  name: automation-broker-apb
  namespace: automation-broker-apb
spec:
  serviceAccount: automation-broker-apb
  containers:
    - name: apb
      image: docker.io/ansibleplaybookbundle/automation-broker-apb:latest
      args: [ "provision", "-e create_broker_namespace=true", "-e wait_for_broker=true", "-e broker_image=docker.io/ansibleplaybookbundle/origin-ansible-service-broker:latest" ]
      imagePullPolicy: IfNotPresent
  restartPolicy: Never
namespace "automation-broker-apb" created
serviceaccount "automation-broker-apb" created
clusterrolebinding "automation-broker-apb" created
pod "automation-broker-apb" created
PLAY [automation-broker-apb provision] *****************************************
TASK [automation-broker-apb : Determine cluster type] **************************
changed: [localhost]
TASK [automation-broker-apb : Set facts] ***************************************
ok: [localhost]
TASK [automation-broker-apb : Check for service catalog] ***********************
changed: [localhost]
TASK [automation-broker-apb : Fail if service-catalog not installed] ***********
skipping: [localhost]
TASK [automation-broker-apb : Set broker namespace state=present] **************
changed: [localhost]
TASK [automation-broker-apb : Get the broker namespace] ************************
ok: [localhost]
TASK [automation-broker-apb : Set broker objects state=present] ****************
changed: [localhost] => (item=broker.service.yaml)
changed: [localhost] => (item=broker.serviceaccount.yaml)
changed: [localhost] => (item=broker.clusterrolebinding.yaml)
changed: [localhost] => (item=broker.configmap.yaml)
changed: [localhost] => (item=broker-auth.clusterrole.yaml)
changed: [localhost] => (item=broker-auth.clusterrolebinding.yaml)
changed: [localhost] => (item=broker-client.serviceaccount.yaml)
changed: [localhost] => (item=broker-client.secret.yaml)
changed: [localhost] => (item=broker-client.clusterrolebinding.yaml)
changed: [localhost] => (item=broker-access.clusterrole.yaml)
TASK [automation-broker-apb : Broker auth secret state=present] ****************
skipping: [localhost]
TASK [automation-broker-apb : include_tasks] ***********************************
included: /opt/ansible/roles/automation-broker-apb/tasks/dao_crd.yaml for localhost
TASK [automation-broker-apb : Use kubectl to create cluster resource definitions] ***
changed: [localhost] => (item=bundle.crd.yaml)
changed: [localhost] => (item=bundlebindings.crd.yaml)
changed: [localhost] => (item=bundleinstances.crd.yaml)
TASK [automation-broker-apb : include_tasks] ***********************************
included: /opt/ansible/roles/automation-broker-apb/tasks/generate_certificate.yaml for localhost
TASK [automation-broker-apb : Create directory for cert] ***********************
skipping: [localhost]
TASK [automation-broker-apb : Create cert] *************************************
skipping: [localhost]
TASK [automation-broker-apb : Create tls secret] *******************************
skipping: [localhost]
TASK [automation-broker-apb : Register certificate related facts for broker] ***
skipping: [localhost]
TASK [automation-broker-apb : Get the cluster ca] ******************************
changed: [localhost]
TASK [automation-broker-apb : Register certificate related facts for broker] ***
ok: [localhost]
TASK [automation-broker-apb : Set broker_client_secret fact] *******************
ok: [localhost]
TASK [automation-broker-apb : Set broker deployment object state=present] ******
changed: [localhost]
TASK [automation-broker-apb : Set route state=present] *************************
changed: [localhost]
TASK [automation-broker-apb : Template out cluster service broker object] ******
changed: [localhost]
TASK [automation-broker-apb : Use kubectl to create cluster service broker object] ***
changed: [localhost]
TASK [automation-broker-apb : Wait for clusterservicebroker to become available] ***
FAILED - RETRYING: Wait for clusterservicebroker to become available (60 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (59 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (58 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (57 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (56 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (55 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (54 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (53 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (52 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (51 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (50 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (49 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (48 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (47 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (46 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (45 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (44 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (43 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (42 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (41 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (40 retries left).
FAILED - RETRYING: Wait for clusterservicebroker to become available (39 retries left).
changed: [localhost]
PLAY RECAP *********************************************************************
localhost                  : ok=17   changed=11   unreachable=0    failed=0
namespace "automation-broker-apb" deleted
serviceaccount "automation-broker-apb" deleted
clusterrolebinding "automation-broker-apb" deleted
pod "automation-broker-apb" deleted
./scripts/deploy.sh: line 28: exit: <no: numeric argument required
make: *** [Makefile:101: deploy] Error 2

jmrodri · 2018-05-12T02:33:17Z

This is how I brought up my cluster using these commands in a simpleup script:

#oc cluster up  --enable=service-catalog,router,registry,web-console,persistent-volumes
oc cluster up --routing-suffix=172.17.0.1.nip.io --public-hostname=172.17.0.1 --base-dir=/tmp/openshift.local.clusterup --tag=latest --image=docker.io/openshift/origin-\${component}:\${version} --enable=service-catalog,template-service-broker,router,registry,web-console,persistent-volumes,sample-templates,rhel-imagestreams
oc login -u system:admin
oc new-project ansible-service-broker 
#cat template-sample-broker.yaml | oc process -n samplebroker -f - | oc create -f -

jmrodri

Didn't work for me.

djzager · 2018-05-14T13:21:41Z

@jmrodri I ran into this problem initially with my scripts/deploy.sh because, on successful execution, the pod does not have a .state.terminated.exitCode in the .status.containerStatuses. I had fixed this by adding:

if [ -n "${EXIT_CODE}" ]; then
    exit ${EXIT_CODE}
else
    exit 0
fi

Can you confirm that you have that deploy script? Based on the line number bash is reporting, I think you are using the right one. I am just struggling to recreate.

djzager · 2018-05-14T13:54:00Z

Nevermind. I was able to recreate. working on a fix.

djzager · 2018-05-25T20:00:10Z

Update broker-apb with dashboard-redirector

eriknelson

VISIACK, tried to scan most of this and it looks sane to me. I have tested locally with a make deploy and make undeploy with an oc cluster up based cluster. I also used this in a vanilla k8s-dind cluster I was using for my namespaced broker demo, so I have reasonable confidence in it working. Nice job @djzager!

shawn-hurley · 2018-05-31T18:47:52Z

apb/.travis.yml

@@ -0,0 +1,31 @@
+---


Do we still need this file after this is merged into the broker? is there a way to run travis only when files under apb get changed?

It's useless once this is merged. I just didn't want to lose it because I'm hoping to merge it into ${PROJECT_ROOT}/.travis.yml in some future PR

cben · 2019-02-28T22:50:53Z

apb/files/bundlebindings.crd.yaml

+  scope: Namespaced
+  names:
+    plural: bundlebindings
+    singular: bundleebinding


Is double ee here a typo?

(now exists in couple more yamls: https://github.com/openshift/ansible-service-broker/search?q=bundleebinding
and in OperatorHub.io: https://github.com/operator-framework/community-operators/blob/master/community-operators/automationbroker/bundlebindings.crd.yaml
and some other repos: https://github.com/search?q=bundleebinding&type=Code)

bundleebinding <- double ee is probably typo, compare to kind: BundleBinding Reported on openshift/ansible-service-broker#939 (comment)

openshift-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 9, 2018

djzager mentioned this pull request May 9, 2018

Merge run_latest_k8s_build into run_latest_build script #920

Closed

djzager changed the title ~~Add Broker APB source to project~~ [WIP] Add Broker APB source to project May 9, 2018

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 9, 2018

djzager force-pushed the add-apb branch from 3d2bca7 to d97c644 Compare May 10, 2018 13:32

djzager changed the title ~~[WIP] Add Broker APB source to project~~ Add Broker APB source to project May 10, 2018

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 10, 2018

djzager force-pushed the add-apb branch from 75b30b7 to 0421814 Compare May 10, 2018 16:14

dymurray approved these changes May 10, 2018

View reviewed changes

djzager added the needs-review label May 11, 2018

jmrodri self-requested a review May 11, 2018 17:56

jmrodri suggested changes May 12, 2018

View reviewed changes

This was referenced May 14, 2018

pass ROUTING_SUFFIX parameter #887

Closed

[wip] Use run_latest_build with existing clusters #908

Closed

shawn-hurley added the 3.11 | release-1.3 Kubernetes 1.11 | Openshift 3.11 | Broker release-1.3 label May 15, 2018

This was referenced May 16, 2018

run latest script fails #957

Closed

Support building a debuggable image for development #926

Merged

djzager force-pushed the add-apb branch from e42ed38 to eec8440 Compare May 23, 2018 14:15

djzager added 7 commits May 24, 2018 11:18

Add apb source to broker project

49a7139

Cleanup scripts directory

c9eff9a

Update testing/makefile

4d4e243

Check for no value exitCode

8a790fa

Add useful debug information in tasks

0e86461

Move role after yum for better caching

f1f1312

Update apb to use ansible 2.6 and dynamic client

63521e7

djzager force-pushed the add-apb branch from eec8440 to 63521e7 Compare May 24, 2018 15:18

djzager added 4 commits May 24, 2018 14:37

What for pod running before following logs

38da281

Use broker name when setting fact

bc9eb98

Use projectrequest in openshift

e517f5c

Delete the project on deprovision

d8b0428

Add dashboard-redirector

e00eec5

djzager force-pushed the add-apb branch from 11bad9d to e00eec5 Compare May 29, 2018 12:45

Condition on state (instead of apb action)

ccfc0d9

djzager mentioned this pull request May 30, 2018

Instructions in README doesn't work with OpenShift v3.9 #969

Closed

eriknelson approved these changes May 31, 2018

View reviewed changes

jmrodri approved these changes May 31, 2018

View reviewed changes

shawn-hurley approved these changes May 31, 2018

View reviewed changes

djzager merged commit c00dfd3 into openshift:master May 31, 2018

cben reviewed Feb 28, 2019

View reviewed changes

Add Broker APB source to project #939

Add Broker APB source to project #939

djzager commented May 9, 2018 •

edited

dymurray left a comment

jmrodri commented May 12, 2018

jmrodri commented May 12, 2018

jmrodri left a comment

djzager commented May 14, 2018

djzager commented May 14, 2018

djzager commented May 25, 2018

eriknelson left a comment

shawn-hurley May 31, 2018

djzager May 31, 2018

cben Feb 28, 2019

Add Broker APB source to project #939

Add Broker APB source to project #939

Conversation

djzager commented May 9, 2018 • edited

dymurray left a comment

Choose a reason for hiding this comment

jmrodri commented May 12, 2018

jmrodri commented May 12, 2018

jmrodri left a comment

Choose a reason for hiding this comment

djzager commented May 14, 2018

djzager commented May 14, 2018

djzager commented May 25, 2018

eriknelson left a comment

Choose a reason for hiding this comment

shawn-hurley May 31, 2018

Choose a reason for hiding this comment

djzager May 31, 2018

Choose a reason for hiding this comment

cben Feb 28, 2019

Choose a reason for hiding this comment

djzager commented May 9, 2018 •

edited