Test the designed deployment solution in production-like environment

[juhtornr]

Issue by juhtornr
Friday Sep 21, 2018 at 07:43 GMT
Originally opened as NBISweden/LocalEGA#361

---

Description

Test the deployment solution designed in #314 (still open) in production-like environment at CSC.

DoD (Definition of Done)

Two OpenShift clusters up and running in selected machines.

Testing

Peer review and file can be ingested (test using the sftp script).

[juhtornr]

Comment by blankdots
Tuesday Sep 25, 2018 at 07:13 GMT

---

We decided against using minishift in replicating a production-like environment and running an Openshift cluster locally.

General instructions:

• https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md\
• https://docs.okd.io/latest/install/index.html
• https://github.com/isnuryusuf/openshift-install/blob/master/openshift-origin-quickstart.md

Installation [WIP] - to be updated

Install kubectl

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Installing OpenShift origin

Instructions: https://github.com/openshift/origin#installation

wget https://github.com/openshift/origin/releases/download/v3.10.0/openshift-origin-client-tools-v3.10.0-dd10d17-linux-64bit.tar.gz
tar -xf openshift-origin-client-tools-v3.10.0-dd10d17-linux-64bit.tar.gz
mv openshift-origin-client-tools-v3.10.0-dd10d17-linux-64bit/oc /usr/local/bin/oc

[juhtornr]

Comment by blankdots
Tuesday Sep 25, 2018 at 07:14 GMT

---

After following prerequisites and trying to start the cluster with oc cluster up run into error:

Error: invalid configuration: Missing or incomplete configuration info.  Please login or point to an existing, complete config file:

  1. Via the command-line flag --config
  2. Via the KUBECONFIG environment variable
  3. In your home directory as ~/.kube/config

seems to be caused: E0925 <date pid> interface.go:34] Failed to install "openshift-web-console-operator": timed out waiting for the condition might be caused by: https://bugzilla.redhat.com/show_bug.cgi?id=1618311

[juhtornr]

Comment by blankdots
Monday Oct 08, 2018 at 05:53 GMT

---

oc cluster up currently hangs with:

Error: Error response from daemon: {"message": "no space left on device"}

Possible bugs:

• oc adm diagnostics produces: https://bugzilla.redhat.com/show_bug.cgi?id=1368865
• Error message "no space left on device" after executing command "oc cluster status" openshift/origin#20346
  cleaned images docker rmi -f $(docker images --quiet --filter "dangling=true") did not help
• Ubuntu 16.04 error on starting openshift cluster openshift/origin#20184
• https://bugzilla.redhat.com/show_bug.cgi?id=1510167

[juhtornr]

Comment by juhtornr
Thursday Dec 20, 2018 at 08:24 GMT

---

The definition of production-like environment has changed during last two months. At CSC we ended up using plain Kubernetes instead Openshift and we have now deployed that (see #360).

[juhtornr]

Time estimate: 10 days

[blankdots]

Docker Images used for M4:

minio/minio:RELEASE.2018-11-30T03-56-59Z with Image ID: sha256:87aa03385853a268ad14eaa02b6a824c2f88c9f3d30fe7ab6e07bb12c6c9bb45
postgres:10-alpine with Image ID: sha256:787004db496a65b949515e809683667ed7c4b566fd27b3dc6ed46b322575f278
rabbitmq:3.6-management-alpine with Image ID: sha256:b99e5acec5dedcb046083df1c6383b7980b00c497c3a021a3f760c56151c6b70

---

nbisweden/ega-base:m4 with Image ID: sha256:628134e10e6354b5f9275dcfbed2e51798a536d3edcdf2e2dc35ec8cb66f1245
nbisweden/ega-inbox:m4 with Image ID: sha256:4768084c4850799dd03ae1ecbe8d73d77e3715452c8a36228dcd85fab04699d5

---

cscfi/ega-dataedge:m4 with Image ID: sha256:52eef763f29e9733f80737dc0956bd1485a1d734de228d20b39573feb429591c
cscfi/ega-res:m4 with Image ID: sha256:18a4cf0070cbb547cda96af0ab3e789a17613fd210a62a948108a2944859781f
cscfi/ega-keyserver:m4 with Image ID: sha256:8d184bd28ff52333847b7470eb8eb27364ef7b911ceef492bb8e9c469b8e22ff
cscfi/ega-filedatabase:m4 with Image ID: sha256:c4569db13827854326537e7e2daa0eda30c4a4bfe4147ebc9bd361941f65e81d