Skip to content

Commit

Permalink
Merge pull request #2141 from muawiakh/update-dns-server
Browse files Browse the repository at this point in the history
Problem: Cluster DNS and Health check terminology is incorrect
  • Loading branch information
ttmc committed Mar 26, 2018
2 parents ff62675 + a8e68c3 commit dea76f7
Show file tree
Hide file tree
Showing 16 changed files with 72 additions and 53 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ First create a directory for the CA and cd into it:

.. code:: bash
mkdir bdb-cluster-ca
mkdir bdb-node-ca
cd bdb-cluster-ca
cd bdb-node-ca
Then :ref:`install and configure Easy-RSA in that directory <how-to-install-and-configure-easyrsa>`.

Expand All @@ -27,7 +27,7 @@ Step 2: Create a Self-Signed CA
-------------------------------

You can create a self-signed CA
by going to the ``bdb-cluster-ca/easy-rsa-3.0.1/easyrsa3`` directory and using:
by going to the ``bdb-node-ca/easy-rsa-3.0.1/easyrsa3`` directory and using:

.. code:: bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ to sign the request.

If you are the admin of the managing organization's self-signed CA,
then you can import the CSR and use Easy-RSA to sign it.
Go to your ``bdb-cluster-ca/easy-rsa-3.0.1/easyrsa3/``
Go to your ``bdb-node-ca/easy-rsa-3.0.1/easyrsa3/``
directory and do something like:

.. code:: bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,19 @@ These parameters are shared across the cluster. More information about the gener
of these parameters can be found at :ref:`generate-the-blockchain-id-and-genesis-time`.


vars.NODE_DNS_SERVER
^^^^^^^^^^^^^^^^^^^^
IP of Kubernetes service(kube-dns), can be retrieved using
using CLI(kubectl) or k8s dashboard. This parameter is used by the Nginx gateway instance
to resolve the hostnames of all the services running in the k8s cluster.
The value defaults to `10.0.0.1`.

.. code::
# retrieval via commandline.
$ kubectl get services
.. _generate-config:

Generate configuration
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ to the above command (i.e. the path to the private key).
will get a list of the pods in the Kubernetes cluster associated
with the context named ``k8s-bdb-test-cluster-0``.

Step 2: Connect to Your Cluster's Web UI (Optional)
---------------------------------------------------
Step 2: Connect to Your Kubernetes Cluster's Web UI (Optional)
---------------------------------------------------------------

You can connect to your cluster's
`Kubernetes Dashboard <https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/>`_
Expand Down Expand Up @@ -157,9 +157,9 @@ Step 5: Assign DNS Name to the NGINX Public IP
* Once a public IP is assigned, you can map it to
a DNS name.
We usually assign ``bdb-test-cluster-0``, ``bdb-test-cluster-1`` and
We usually assign ``bdb-test-node-0``, ``bdb-test-node-1`` and
so on in our documentation.
Let's assume that we assign the unique name of ``bdb-test-cluster-0`` here.
Let's assume that we assign the unique name of ``bdb-test-node-0`` here.


**Set up DNS mapping in Azure.**
Expand All @@ -171,7 +171,7 @@ changes to be reflected.
Select the ``Public IP`` resource that is attached to your service (it should
have the Azure DNS prefix name along with a long random string, without the
``master-ip`` string), select ``Configuration``, add the DNS assigned above
(for example, ``bdb-test-cluster-0``), click ``Save``, and wait for the
(for example, ``bdb-test-node-0``), click ``Save``, and wait for the
changes to be applied.

To verify the DNS setting is operational, you can run ``nslookup <DNS
Expand Down Expand Up @@ -244,7 +244,7 @@ Step 10: Start the NGINX Kubernetes Deployment
----------------------------------------------

* NGINX is used as a proxy to the BigchainDB, Tendermint and MongoDB instances in
the node. It proxies HTTP/HTTPS requests on the ``cluster-frontend-port``
the node. It proxies HTTP/HTTPS requests on the ``node-frontend-port``
to the corresponding OpenResty(if 3scale enabled) or BigchainDB backend, TCP connections
on ``mongodb-frontend-port``, ``tm-p2p-port`` and ``tm-pub-key-access``
to MongoDB and Tendermint respectively.
Expand Down Expand Up @@ -580,7 +580,7 @@ Step 20(Optional): Start a Kubernetes Deployment for OpenResty

* The configuration uses the following values set in the ConfigMap:

- ``cluster-dns-server-ip``
- ``node-dns-server-ip``
- ``openresty-backend-port``
- ``ngx-bdb-instance-name``
- ``bigchaindb-api-port``
Expand Down Expand Up @@ -736,7 +736,7 @@ To test the vanilla NGINX instance:
$ nslookup ngx-http-instance-0
$ dig +noall +answer _public-cluster-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-node-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-health-check-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV
Expand All @@ -755,15 +755,15 @@ To test the NGINX instance with HTTPS and 3scale integration:
$ nslookup ngx-instance-0
$ dig +noall +answer _public-secure-cluster-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-secure-node-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-mdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-insecure-cluster-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-insecure-node-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
$ wsc -er wss://<cluster-fqdn>/api/v1/streams/valid_transactions
$ wsc -er wss://<node-fqdn>/api/v1/streams/valid_transactions
$ curl -X GET http://<cluster-fqdn>:27017
$ curl -X GET http://<node-fqdn>:27017
The above curl command should result in the response
``It looks like you are trying to access MongoDB over HTTP on the native driver port.``
Expand All @@ -776,7 +776,7 @@ Check the MongoDB monitoring agent on the MongoDB Cloud Manager
portal to verify they are working fine.

If you are using the NGINX with HTTP support, accessing the URL
``http://<DNS/IP of your exposed BigchainDB service endpoint>:cluster-frontend-port``
``http://<DNS/IP of your exposed BigchainDB service endpoint>:node-frontend-port``
on your browser should result in a JSON response that shows the BigchainDB
server version, among other things.
If you are using the NGINX with HTTPS support, use ``https`` instead of
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Since we used Easy-RSA version 3 to
we use it to revoke certificates too.

Go to the following directory (associated with the self-signed CA):
``.../bdb-cluster-ca/easy-rsa-3.0.1/easyrsa3``.
``.../bdb-node-ca/easy-rsa-3.0.1/easyrsa3``.
You need to be aware of the file name used to import the certificate using the
``./easyrsa import-req`` before. Run the following command to revoke a
certificate:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ to sign the request.

If you are the admin of the managing organization's self-signed CA,
then you can import the CSR and use Easy-RSA to sign it.
Go to your ``bdb-cluster-ca/easy-rsa-3.0.1/easyrsa3/``
Go to your ``bdb-node-ca/easy-rsa-3.0.1/easyrsa3/``
directory and do something like:

.. code:: bash
Expand Down
20 changes: 10 additions & 10 deletions k8s/configuration/config-map.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,31 +14,31 @@ data:
# are available to external clients.
node-frontend-port: "443"

# cluster-health-check-port is the port number on which an external load
# node-health-check-port is the port number on which an external load
# balancer can check the status/liveness of the external/public server.
# In our deployment, Kubernetes sends 'livenessProbes' to this port and
# interprets a successful response as a 'healthy' service.
cluster-health-check-port: "8888"
node-health-check-port: "8888"

# cluster-dns-server-ip is the IP of the DNS server. A Kubernetes deployment
# always has a DNS server (kube-dns) running at 10.0.0.10
cluster-dns-server-ip: "10.0.0.10"
# node-dns-server-ip is the IP of the DNS server. A Kubernetes deployment
# always has a DNS server (kube-dns).
node-dns-server-ip: "<IP address of node's DNS server, e.g. 10.0.0.10>"

# mdb-instance-name is the name of the MongoDB instance in this cluster.
# mdb-instance-name is the name of the MongoDB instance in this Kubernetes cluster.
mdb-instance-name: "<name of the mdb instance>"

# ngx-instance-name is the name of the NGINX instance in this cluster.
# ngx-instance-name is the name of the NGINX instance in this Kubernetes cluster.
ngx-instance-name: "<name of the nginx instance>"

# openresty-instance-name is the name of the OpenResty instance in this
# cluster.
# Kubernetes cluster.
openresty-instance-name: "<name of the openresty instance>"

# bdb-instance-name is the name of the BigchainDB instance in this cluster.
# bdb-instance-name is the name of the BigchainDB instance in this Kubernetes cluster.
bdb-instance-name: "<name of the bdb instance>"

# mdb-mon-instance-name is the name of the MongoDB Monitoring Agent instance
# in this cluster.
# in this Kubernetes cluster.
mdb-mon-instance-name: "<name of the mdb monitoring agent instance>"

# ngx-mdb-instance-name is the FQDN of the MongoDB instance in this
Expand Down
4 changes: 2 additions & 2 deletions k8s/dev-setup/nginx-https.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,11 @@ spec:
ports:
- port: 443
targetPort: 443
name: public-secure-cluster-port
name: public-secure-node-port
protocol: TCP
- port: 80
targetPort: 80
name: public-insecure-cluster-port
name: public-insecure-node-port
protocol: TCP
- port: 27017
targetPort: 27017
Expand Down
6 changes: 3 additions & 3 deletions k8s/nginx-http/nginx-http-dep.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@ spec:
valueFrom:
configMapKeyRef:
name: vars
key: cluster-health-check-port
key: node-health-check-port
- name: DNS_SERVER
valueFrom:
configMapKeyRef:
name: vars
key: cluster-dns-server-ip
key: node-dns-server-ip
- name: MONGODB_BACKEND_HOST
valueFrom:
configMapKeyRef:
Expand Down Expand Up @@ -71,7 +71,7 @@ spec:
name: tendermint-config
key: tm-p2p-port
ports:
- containerPort: "<cluster-health-check-port from ConfigMap>"
- containerPort: "<node-health-check-port from ConfigMap>"
protocol: TCP
name: ngx-health
- containerPort: "<node-frontend-port from ConfigMap>"
Expand Down
6 changes: 3 additions & 3 deletions k8s/nginx-https-web-proxy/nginx-https-web-proxy-dep.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,12 +54,12 @@ spec:
valueFrom:
configMapKeyRef:
name: vars
key: cluster-dns-server-ip
key: node-dns-server-ip
- name: HEALTH_CHECK_PORT
valueFrom:
configMapKeyRef:
name: vars
key: cluster-health-check-port
key: node-health-check-port
- name: BIGCHAINDB_BACKEND_HOST
valueFrom:
configMapKeyRef:
Expand All @@ -76,7 +76,7 @@ spec:
livenessProbe:
httpGet:
path: /health
port: <cluster-health-check-port from the ConfigMap>
port: <node-health-check-port from the ConfigMap>
initialDelaySeconds: 15
periodSeconds: 15
failureThreshold: 3
Expand Down
4 changes: 2 additions & 2 deletions k8s/nginx-https/nginx-https-dep.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ spec:
valueFrom:
configMapKeyRef:
name: vars
key: cluster-health-check-port
key: node-health-check-port
- name: NODE_FQDN
valueFrom:
configMapKeyRef:
Expand All @@ -34,7 +34,7 @@ spec:
valueFrom:
configMapKeyRef:
name: vars
key: cluster-dns-server-ip
key: node-dns-server-ip
- name: MONGODB_BACKEND_HOST
valueFrom:
configMapKeyRef:
Expand Down
4 changes: 2 additions & 2 deletions k8s/nginx-https/nginx-https-svc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ spec:
ports:
- port: 443
targetPort: 443
name: public-secure-cluster-port
name: public-secure-node-port
protocol: TCP
- port: 27017
targetPort: 27017
Expand All @@ -31,6 +31,6 @@ spec:
name: tm-p2p-port
- port: 80
targetPort: 80
name: public-insecure-cluster-port
name: public-insecure-node-port
protocol: TCP
type: LoadBalancer
2 changes: 1 addition & 1 deletion k8s/nginx-openresty/nginx-openresty-dep.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ spec:
valueFrom:
configMapKeyRef:
name: vars
key: cluster-dns-server-ip
key: node-dns-server-ip
- name: OPENRESTY_FRONTEND_PORT
valueFrom:
configMapKeyRef:
Expand Down
19 changes: 10 additions & 9 deletions k8s/scripts/functions
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,7 @@ function generate_config_map(){
tm_genesis_time=$7
tm_chain_id=$8
tm_instance_name=$9
dns_resolver_k8s=$10

cat > config-map.yaml << EOF
apiVersion: v1
Expand All @@ -242,27 +243,27 @@ data:
# are available to external clients.
node-frontend-port: "443"
# cluster-health-check-port is the port number on which an external load
# node-health-check-port is the port number on which an external load
# balancer can check the status/liveness of the external/public server.
# In our deployment, Kubernetes sends 'livenessProbes' to this port and
# interprets a successful response as a 'healthy' service.
cluster-health-check-port: "8888"
node-health-check-port: "8888"
# cluster-dns-server-ip is the IP of the DNS server. A Kubernetes deployment
# always has a DNS server (kube-dns) running at 10.0.0.10
cluster-dns-server-ip: "10.0.0.10"
# node-dns-server-ip is the IP of the DNS server. A Kubernetes deployment
# always has a DNS server (kube-dns).
node-dns-server-ip: "${dns_resolver_k8s}"
# mdb-instance-name is the name of the MongoDB instance in this cluster.
# mdb-instance-name is the name of the MongoDB instance in this Kubernetes cluster.
mdb-instance-name: "${mdb_instance_name}"
# ngx-instance-name is the name of the NGINX instance in this cluster.
# ngx-instance-name is the name of the NGINX instance in this Kubernetes cluster.
ngx-instance-name: "${ngx_instance_name}"
# bdb-instance-name is the name of the BigchainDB instance in this cluster.
# bdb-instance-name is the name of the BigchainDB instance in this Kubernetes cluster.
bdb-instance-name: "${bdb_instance_name}"
# openresty-instance-name is the name of the OpenResty instance in this
# cluster.
# Kubernetes cluster.
openresty-instance-name: "openresty-instance-0"
# ngx-mdb-instance-name is the FQDN of the MongoDB instance in this
Expand Down
2 changes: 1 addition & 1 deletion k8s/scripts/generate_configs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ fi
BASE_DIR="$(pwd)/${CERT_DIR}"
mkdir -p "${BASE_DIR}"

BASE_CA_DIR="${BASE_DIR}"/bdb-cluster-ca
BASE_CA_DIR="${BASE_DIR}"/bdb-node-ca
BASE_MEMBER_CERT_DIR="${BASE_DIR}"/member-cert
BASE_CLIENT_CERT_DIR="${BASE_DIR}"/client-cert
BASE_EASY_RSA_PATH='easy-rsa-3.0.1/easyrsa3'
Expand Down
7 changes: 6 additions & 1 deletion k8s/scripts/vars
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,9 @@ TM_GENESIS_TIME='0001-01-01T00:00:00Z'

# Blockchain ID must be unique for
# every blockchain
TM_CHAIN_ID='test-chain-rwcPML'
TM_CHAIN_ID='test-chain-rwcPML'

# IP Address of the resolver(DNS server).
# i.e. IP of `kubernetes` service(kube-dns)
# Can be retrieved using `$ kubectl get svc`
NODE_DNS_SERVER='10.0.0.10'

0 comments on commit dea76f7

Please sign in to comment.