Refactored the fluentd-es addon files, moved the fluentd configuratio…

…n to ConfigMap

cluster/addons/fluentd-elasticsearch/README.md

@@ -1,45 +1,82 @@
 # Elasticsearch Add-On
-This add-on consists of a combination of
-[Elasticsearch](https://www.elastic.co/products/elasticsearch), [Fluentd](http://www.fluentd.org/)
-and [Kibana](https://www.elastic.co/products/elasticsearch). Elasticsearch is a search engine
-that is responsible for storing our logs and allowing for them to be queried. Fluentd sends
-log messages from Kubernetes to Elasticsearch, whereas Kibana is a graphical interface for
-viewing and querying the logs stored in Elasticsearch.
+
+This add-on consists of a combination of [Elasticsearch][elasticsearch],
+[Fluentd][fluentd] and [Kibana][kibana]. Elasticsearch is a search engine
+that is responsible for storing our logs and allowing for them to be queried.
+Fluentd sends log messages from Kubernetes to Elasticsearch, whereas Kibana
+is a graphical interface for viewing and querying the logs stored in
+Elasticsearch.
+
+**Note:** this addon should **not** be used as-is in production. This is
+an example and you should treat is as such. Please see at least the
+[Security](#security) and the [Storage](#storage) sections for more
+information.
 
 ## Elasticsearch
-Elasticsearch is deployed as a
-[StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/), which
-is like a Deployment, but allows for maintaining state on storage volumes. 
 
-### Authentication
-Elasticsearch has basic authentication enabled by default, in our configuration the credentials
-are at their default values, i.e. username 'elastic' and password 'changeme'. In order to change
-them, please read up on [the official documentation](https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html#reset-built-in-user-passwords).
+Elasticsearch is deployed as a [StatefulSet][statefulSet], which is like
+a Deployment, but allows for maintaining state on storage volumes. 
+
+### Security
+
+Elasticsearch has capabilities to enable authorization using
+[X-Pack plugin][xPack]. See configuration parameter `xpack.security.enabled`
+in Elasticsearch and Kibana configurations. It can also be set via
+`XPACK_SECURITY_ENABLED` env variable. After enabling the feature,
+follow [official documentation][setupCreds] to set up credentials in
+Elasticsearch and Kibana. Don't forget to propagate those credentials also to
+Fluentd in its [configuration][fluentdCreds], using for example
+[environment variables][fluentdEnvVar]. You can utilize [ConfigMaps][configMap]
+and [Secrets][secret] to store credentials in the Kubernetes apiserver.
 
 ### Initialization
+
 The Elasticsearch Statefulset manifest specifies that there shall be an
-[init container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) executing
-before Elasticsearch containers themselves, in order to ensure that the kernel state variable
-`vm.max_map_count` is at least 262144, since this is a requirement of Elasticsearch.
-You may remove the init container if you know that your host OS meets this requirement.
+[init container][initContainer] executing before Elasticsearch containers
+themselves, in order to ensure that the kernel state variable
+`vm.max_map_count` is at least 262144, since this is a requirement of
+Elasticsearch. You may remove the init container if you know that your host
+OS meets this requirement.
 
 ### Storage
-The Elasticsearch StatefulSet will claim a storage volume 'elasticsearch-logging',
-of the standard
-[StorageClass](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses),
-that by default will be 100 Gi per replica. Please adjust this to your needs (including
-possibly choosing a more suitable StorageClass).
+
+The Elasticsearch StatefulSet will use the [EmptyDir][emptyDir] volume to
+store data. EmptyDir is erased when the pod terminates, here it is used only
+for testing purposes. **Important:** please change the storage to persistent
+volume claim before actually using this StatefulSet in your setup!
 
 ## Fluentd
-Fluentd is deployed as a
-[DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) which spawns a
-pod on each node that reads logs, generated by kubelet, container runtime and containers and
-sends them to Elasticsearch.
 
-*Please note that for Fluentd to work, every Kubernetes node must be labeled*
-`beta.kubernetes.io/fluentd-ds-ready=true`, as otherwise Fluentd will ignore them.
+Fluentd is deployed as a [DaemonSet][daemonSet] which spawns a pod on each
+node that reads logs, generated by kubelet, container runtime and containers
+and sends them to Elasticsearch.
 
-Learn more at: https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana
+**Note:** in order for Fluentd to work, every Kubernetes node must be labeled
+with `beta.kubernetes.io/fluentd-ds-ready=true`, as otherwise the Fluentd
+DaemonSet will ignore them.
 
-[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/cluster/addons/fluentd-elasticsearch/README.md?pixel)]()
+Learn more in the [official Kubernetes documentation][k8sElasticsearchDocs].
+
+### Known problems
+
+Since Fluentd talks to the Elasticsearch service inside the cluster, instances
+on masters won't work, because masters have no kube-proxy. Don't mark masters
+with a label mentioned in the previous paragraph or add a taint on them to
+avoid Fluentd pods scheduling there.
 
+[fluentd]: http://www.fluentd.org/
+[elasticsearch]: https://www.elastic.co/products/elasticsearch
+[kibana]: https://www.elastic.co/products/kibana
+[xPack]: https://www.elastic.co/products/x-pack
+[setupCreds]: https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html#reset-built-in-user-passwords
+[fluentdCreds]: https://github.com/uken/fluent-plugin-elasticsearch#user-password-path-scheme-ssl_verify
+[fluentdEnvVar]: https://docs.fluentd.org/v0.12/articles/faq#how-can-i-use-environment-variables-to-configure-parameters-dynamically
+[configMap]: https://kubernetes.io/docs/tasks/configure-pod-container/configmap/
+[secret]: https://kubernetes.io/docs/concepts/configuration/secret/
+[statefulSet]: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset
+[initContainer]: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
+[emptyDir]: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
+[daemonSet]: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
+[k8sElasticsearchDocs]: https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana
+
+[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/cluster/addons/fluentd-elasticsearch/README.md?pixel)]()

cluster/addons/fluentd-elasticsearch/es-image/Dockerfile

@@ -14,22 +14,12 @@
 
 FROM docker.elastic.co/elasticsearch/elasticsearch:5.5.1
 
-USER root
-
-RUN mkdir /data
-RUN chown -R elasticsearch:elasticsearch /data
-
-WORKDIR /usr/share/elasticsearch
-
 VOLUME ["/data"]
 EXPOSE 9200 9300
 
-USER elasticsearch
-COPY elasticsearch_logging_discovery bin/
-COPY config/elasticsearch.yml config/
-COPY config/log4j2.properties config/
-COPY run.sh bin/
+COPY elasticsearch_logging_discovery run.sh bin/
+COPY config/elasticsearch.yml config/log4j2.properties config/
 
 USER root
-RUN chown -R elasticsearch:elasticsearch config
+RUN chown -R elasticsearch:elasticsearch ./
 CMD ["bin/run.sh"]

cluster/addons/fluentd-elasticsearch/es-image/Makefile

@@ -12,19 +12,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-.PHONY:	elasticsearch_logging_discovery build push
+.PHONY:	binary build push
 
-# The current value of the tag to be used for building and
-# pushing an image to gcr.io
-TAG = v5.5.1
+PREFIX = gcr.io/google-containers
+IMAGE = elasticsearch
+TAG = v5.5.1-1
 
-build:	elasticsearch_logging_discovery
-	docker build --pull -t gcr.io/google_containers/elasticsearch:$(TAG) .
+build:
+	docker build --pull -t $(PREFIX)/$(IMAGE):$(TAG) .
 
 push:
-	gcloud docker -- push gcr.io/google_containers/elasticsearch:$(TAG)
+	gcloud docker -- push $(PREFIX)/$(IMAGE):$(TAG)
 
-elasticsearch_logging_discovery:
+binary:
 	CGO_ENABLED=0 GOOS=linux go build -a -ldflags "-w" elasticsearch_logging_discovery.go
 
 clean:

cluster/addons/fluentd-elasticsearch/es-image/config/elasticsearch.yml

@@ -12,3 +12,6 @@ path.data: /data
 network.host: 0.0.0.0
 
 discovery.zen.minimum_master_nodes: ${MINIMUM_MASTER_NODES}
+
+xpack.security.enabled: false
+xpack.monitoring.enabled: false

cluster/addons/fluentd-elasticsearch/es-statefulset.yaml

@@ -1,11 +1,60 @@
+# RBAC authn and authz
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: elasticsearch-logging
+  namespace: kube-system
+  labels:
+    k8s-app: elasticsearch-logging
+    kubernetes.io/cluster-service: "true"
+    addonmanager.kubernetes.io/mode: Reconcile
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: elasticsearch-logging
+  labels:
+    k8s-app: elasticsearch-logging
+    kubernetes.io/cluster-service: "true"
+    addonmanager.kubernetes.io/mode: Reconcile
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - "services"
+  - "namespaces"
+  - "endpoints"
+  verbs:
+  - "get"
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  namespace: kube-system
+  name: elasticsearch-logging
+  labels:
+    k8s-app: elasticsearch-logging
+    kubernetes.io/cluster-service: "true"
+    addonmanager.kubernetes.io/mode: Reconcile
+subjects:
+- kind: ServiceAccount
+  name: elasticsearch-logging
+  namespace: kube-system
+  apiGroup: ""
+roleRef:
+  kind: ClusterRole
+  name: elasticsearch-logging
+  apiGroup: ""
+---
+# Elasticsearch deployment itself
 apiVersion: apps/v1beta1
 kind: StatefulSet
 metadata:
-  name: elasticsearch-logging-v1
+  name: elasticsearch-logging
   namespace: kube-system
   labels:
     k8s-app: elasticsearch-logging
-    version: v1
+    version: v5.5.1
     kubernetes.io/cluster-service: "true"
     addonmanager.kubernetes.io/mode: Reconcile
 spec:
@@ -14,17 +63,17 @@ spec:
   selector:
     matchLabels:
       k8s-app: elasticsearch-logging
-      version: v1
+      version: v5.5.1
   template:
     metadata:
       labels:
         k8s-app: elasticsearch-logging
-        version: v1
+        version: v5.5.1
         kubernetes.io/cluster-service: "true"
     spec:
       serviceAccountName: elasticsearch-logging
       containers:
-      - image: gcr.io/google_containers/elasticsearch:v5.5.1
+      - image: gcr.io/google-containers/elasticsearch:v5.5.1-1
         name: elasticsearch-logging
         resources:
           # need more cpu upon initialization, therefore burstable class
@@ -47,17 +96,15 @@ spec:
           valueFrom:
             fieldRef:
               fieldPath: metadata.namespace
+      volumes:
+      - name: elasticsearch-logging
+        emptyDir: {}
+      # Elasticsearch requires vm.max_map_count to be at least 262144.
+      # If your OS already sets up this number to a higher value, feel free
+      # to remove this init container.
       initContainers:
       - image: alpine:3.6
         command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
         name: elasticsearch-logging-init
         securityContext:
           privileged: true
-  volumeClaimTemplates:
-  - metadata:
-      name: elasticsearch-logging
-    spec:
-      accessModes: ["ReadWriteOnce"]
-      resources:
-        requests:
-          storage: 100Gi