-
Notifications
You must be signed in to change notification settings - Fork 99
/
README.md
1202 lines (935 loc) · 48.8 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# kube-fluentd-operator (KFO) [![Build Status](https://travis-ci.org/vmware/kube-fluentd-operator.svg?branch=master)](https://travis-ci.org/vmware/kube-fluentd-operator)
[![Go Report Card](https://goreportcard.com/badge/github.com/vmware/kube-fluentd-operator)](https://goreportcard.com/report/github.com/vmware/kube-fluentd-operator)
## Overview
Kubernetes Fluentd Operator (KFO) is a Fluentd config manager with batteries included, config validation, no needs to restart, with sensible defaults and best practices built-in. Use Kubernetes labels to filter/route logs per namespace!
*kube-fluentd-operator* configures Fluentd in a Kubernetes environment. It compiles a Fluentd configuration from configmaps (one per namespace) - similar to how an Ingress controller would compile nginx configuration from several Ingress resources. This way only one instance of Fluentd can handle all log shipping for an entire cluster and the cluster admin does NOT need to coordinate with namespace admins.
Cluster administrators set up Fluentd only once and namespace owners can configure log routing as they wish. *KFO* will re-configure Fluentd accordingly and make sure logs originating from a namespace will not be accessible by other tenants/namespaces.
*KFO* also extends the Fluentd configuration language making it possible to refer to pods based on their labels and the container name pattern. This enables for very fined-grained targeting of log streams for the purpose of pre-processing before shipping. Writing a custom processor, adding a new Fluentd plugin, or writing a custom Fluentd plugin allow KFO to be extendable for any use case and any external logging ingestion system.
Finally, it is possible to ingest logs from a file on the container filesystem. While this is not recommended, there are still legacy or misconfigured apps that insist on logging to the local filesystem.
## Try it out
The easiest way to get started is using the Helm chart. Official images are not published yet, so you need to pass the image.repository and image.tag manually:
```bash
git clone git@github.com:vmware/kube-fluentd-operator.git
helm install kfo ./kube-fluentd-operator/charts/log-router \
--set rbac.create=true \
--set image.tag=v1.16.8 \
--set image.repository=vmware/kube-fluentd-operator
```
Alternatively, deploy the Helm chart from a Github release:
```bash
CHART_URL='https://github.com/vmware/kube-fluentd-operator/releases/download/v1.16.8/log-router-0.4.0.tgz'
helm install kfo ${CHART_URL} \
--set rbac.create=true \
--set image.tag=v1.16.8 \
--set image.repository=vmware/kube-fluentd-operator
```
Then create a namespace `demo` and a configmap describing where all logs from `demo` should go to. The configmap must contain an entry called "fluent.conf". Finally, point the kube-fluentd-operator to this configmap using annotations.
```bash
kubectl create ns demo
cat > fluent.conf << EOF
<match **>
@type null
</match>
EOF
# Create the configmap with a single entry "fluent.conf"
kubectl create configmap fluentd-config --namespace demo --from-file=fluent.conf=fluent.conf
# The following step is optional: the fluentd-config is the default configmap name.
# kubectl annotate namespace demo logging.csp.vmware.com/fluentd-configmap=fluentd-config
```
In a minute, this configuration would be translated to something like this:
```xml
<match demo.**>
@type null
</match>
```
Even though the tag `**` was used in the `<match>` directive, the kube-fluentd-operator correctly expands this to `demo.**`. Indeed, if another tag which does not start with `demo.` was used, it would have failed validation. Namespace admins can safely assume that they has a dedicated Fluentd for themselves.
All configuration errors are stored in the annotation `logging.csp.vmware.com/fluentd-status`. Try replacing `**` with an invalid tag like 'hello-world'. After a minute, verify that the error message looks like this:
```bash
# extract just the value of logging.csp.vmware.com/fluentd-status
kubectl get ns demo -o jsonpath='{.metadata.annotations.logging\.csp\.vmware\.com/fluentd-status}'
bad tag for <match>: hello-world. Tag must start with **, $thisns or demo
```
When the configuration is made valid again the `fluentd-status` is set to "".
To see kube-fluentd-operator in action you need a cloud log collector like logz.io, loggly, papertrail or ELK accessible from the K8S cluster. A simple loggly configuration looks like this (replace TOKEN with your customer token):
```xml
<match **>
@type loggly
loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/fluentd
</match>
```
## Build
Get the code using `go get` or git clone this repo:
```bash
go get -u github.com/vmware/kube-fluentd-operator/config-reloader
cd $GOPATH/src/github.com/vmware/kube-fluentd-operator
# build a base-image
cd base-image && make build-image
# build helm chart
cd charts/log-router && make helm-package
# build the daemon
cd config-reloader
make install
make build-image
# run with mock data (loaded from the examples/ folder)
make run-once-fs
# run with mock data in a loop (may need to ctrl+z to exit)
make run-loop-fs
# inspect what is generated from the above command
ls -l tmp/
```
### Project structure
* `charts/log-router`: Builds the Helm chart
* `base-image`: Builds a Fluentd 1.2.x image with a curated list of plugins
* `config-reloader`: Builds the daemon that generates fluentd configuration files
### Config-reloader
This is where interesting work happens. The [dependency graph](config-reloader/godepgraph.png) shows the high-level package interaction and general dataflow.
* `config`: handles startup configuration, reading and validation
* `datasource`: fetches Pods, Namespaces, ConfigMaps from Kubernetes
* `fluentd`: parses Fluentd config files into an object graph
* `processors`: walks this object graph doing validations and modifications. All features are implemented as chained `Processor` subtypes
* `generator`: serializes the processed object graph to the filesystem for Fluentd to read
* `controller`: orchestrates the high-level `datasource` -> `processor` -> `generator` pipeline.
### How does it work
It works be rewriting the user-provided configuration. This is possible because *kube-fluentd-operator* knows about the kubernetes cluster, the current namespace and
also has some sensible defaults built in. To get a quick idea what happens behind the scenes consider this configuration deployed in a namespace called `monitoring`:
```xml
<filter $labels(server=apache)>
@type parser
<parse>
@type apache2
</parse>
</filter>
<filter $labels(app=django)>
@type detect_exceptions
language python
</filter>
<match **>
@type es
</match>
```
It gets processed into the following configuration which is then fed to Fluentd:
```xml
<filter kube.monitoring.*.*>
@type record_transformer
enable_ruby true
<record>
kubernetes_pod_label_values ${record["kubernetes"]["labels"]["app"]&.gsub(/[.-]/, '_') || '_'}.${record["kubernetes"]["labels"]["server"]&.gsub(/[.-]/, '_') || '_'}
</record>
</filter>
<match kube.monitoring.*.*>
@type rewrite_tag_filter
<rule>
key kubernetes_pod_label_values
pattern ^(.+)$
tag ${tag}._labels.$1
</rule>
</match>
<filter kube.monitoring.*.*.**>
@type record_transformer
remove_keys kubernetes_pod_label_values
</filter>
<filter kube.monitoring.*.*._labels.*.apache _proc.kube.monitoring.*.*._labels.*.apache>
@type parser
<parse>
@type apache2
</parse>
</filter>
<match kube.monitoring.*.*._labels.django.*>
@type rewrite_tag_filter
<rule>
invert true
key _dummy
pattern /ZZ/
tag 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee._proc.${tag}
</rule>
</match>
<match 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee._proc.kube.monitoring.*.*._labels.django.*>
@type detect_exceptions
remove_tag_prefix 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee
stream container_info
</match>
<match kube.monitoring.*.*._labels.*.* _proc.kube.monitoring.*.*._labels.*.*>
@type es
</match>
```
## Configuration
### Basic usage
To give the illusion that every namespace runs a dedicated Fluentd the user-provided configuration is post-processed. In general, expressions starting with `$` are macros that are expanded. These two directives are equivalent: `<match **>`, `<match $thisns>`. Almost always, using the `**` is the preferred way to match logs: this way you can reuse the same configuration for multiple namespaces.
### The admin namespace
Kube-fluentd-operator defines one namespace to be the *admin* namespace. By default this is set to `kube-system`. The *admin* namespace is treated differently. Its configuration is not processed further as it is assumed only the cluster admin can manipulate resources in this namespace. If you don't plan to use any of the advanced features described bellow, you can just route all logs from all namespaces using this snippet in the *admin* namespace:
```xml
<match **>
@type ...
# destination configuration omitted
</match>
```
`**` in this context is not processed and it means *literally* everything.
Fluentd assumes it is running in a distro with systemd and generates logs with these Fluentd tags:
* `systemd.{unit}`: the journal of a systemd unit, for example `systemd.docker.service`
* `docker`: all docker logs, not containers. If systemd is used, the docker logs are in `systemd.docker.service`
* `k8s.{component}`: logs from a K8S component, for example `k8s.kube-apiserver`
* `kube.{namespace}.{pod_name}.{container_name}`: a log originating from (namespace, pod, container)
As the *admin* namespace is processed first, a match-all directive would consume all logs and any other namespace configuration will become irrelevant (unless `<copy>` is used).
A recommended configuration for the *admin* namespace is this one (assuming it is set to `kube-system`) - it captures all but the user namespaces' logs:
```xml
<match systemd.** kube.kube-system.** k8s.** docker>
# all k8s-internal and OS-level logs
# destination config omitted...
</match>
```
Note the `<match systemd.**` syntax. A single `*` would not work as the tag is the full name - including the unit type, for example *systemd.nginx.service*
### Using the $labels macro
A very useful feature is the `<filter>` and the `$labels` macro to define parsing at the namespace level. For example, the config-reloader container uses the `logfmt` format. This makes it easy to use structured logging and ingest json data into a remote log ingestion service.
```xml
<filter $labels(app=log-router, _container=reloader)>
@type parser
reserve_data true
<parse>
@type logfmt
</parse>
</filter>
<match **>
@type loggly
# destination config omitted
</match>
```
The above config will pipe all logs from the pods labelled with `app=log-router` through a [logfmt](https://github.com/vmware/kube-fluentd-operator/blob/master/base-image/plugins/parser_logfmt.rb) parser before sending them to loggly. Again, this configuration is valid in any namespace. If the namespace doesn't contain any `log-router` components then the `<filter>` directive is never activated. The `_container` is sort of a "meta" label and it allows for targeting the log stream of a specific container in a multi-container pod.
If you use [Kubernetes recommended labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) for the pods and deployments, then KFO will rewrite `.` characters into `_`.
For example, let's assume the following labels exist in the fluentd-config in the `testing` namespace:
This label `$labels(_container=nginx-ingress-controller)` will filter by container name pattern. The label will convert to this for example: `kube.testing.*.nginx-ingress-controller._labels.*.*.`
This label `$labels(app.kubernetes.io/name=nginx-ingress, _container=nginx-ingress-controller)` converts to this `kube.testing.*.nginx-ingress-controller._labels.*.nginx_ingress`.
This label `$labels(app.kubernetes.io/name=nginx-ingress)` converts to this `$labels(kube.testing.*.*._labels.*.nginx_ingress)`.
This fluentd configmap in the `testing` namespace:
```xml
<filter **>
@type concat
timeout_label @DISTILLERY_TYPES
key message
stream_identity_key cont_id
multiline_start_regexp /^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}|\[\w+\]\s|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b|=\w+ REPORT====|\d{2}\:\d{2}\:\d{2}\.\d{3})/
flush_interval 10
</filter>
<match **>
@type relabel
@label @DISTILLERY_TYPES
</match>
<label @DISTILLERY_TYPES>
<filter $labels(app_kubernetes_io/name=kafka)>
@type parser
key_name log
format json
reserve_data true
suppress_parse_error_log true
</filter>
<filter $labels(app.kubernetes.io/name=nginx-ingress, _container=controller)>
@type parser
key_name log
<parse>
@type json
reserve_data true
time_format %FT%T%:z
emit_invalid_record_to_error false
</parse>
</filter>
<match $labels(tag=noisy)>
@type null
</match>
</label>
```
will be rewritten inside of KFO pods as this:
```xml
<filter kube.testing.**>
@type concat
flush_interval 10
key message
multiline_start_regexp /^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}|\[\w+\]\s|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b|=\w+ REPORT====|\d{2}\:\d{2}\:\d{2}\.\d{3})/
stream_identity_key cont_id
timeout_label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba
</filter>
<match kube.testing.**>
@label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba
@type relabel
</match>
<match kube.testing.**>
@label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba
@type null
</match>
<label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba>
<filter kube.testing.*.*._labels.*.kafka.*>
@type parser
format json
key_name log
reserve_data true
suppress_parse_error_log true
</filter>
<filter kube.testing.*.controller._labels.nginx_ingress.*.*>
@type parser
key_name log
<parse>
@type json
emit_invalid_record_to_error false
reserve_data true
time_format %FT%T%:z
</parse>
</filter>
<match kube.testing.*.*._labels.*.*.noisy>
@type null
</match>
</label>
```
All plugins that change the fluentd tag are disabled for security reasons. Otherwise a rogue configuration may divert other namespace's logs to itself by prepending its name to the tag.
### Ingest logs from a file in the container
The only allowed `<source>` directive is of type `mounted-file`. It is used to ingest a log file from a container on an `emptyDir`-mounted volume:
```xml
<source>
@type mounted-file
path /var/log/welcome.log
labels app=grafana, _container=test-container
<parse>
@type none
</parse>
</source>
```
The `labels` parameter is similar to the `$labels` macro and helps the daemon locate all pods that might log to the given file path. The `<parse>` directive is optional and if omitted the default `@type none` will be used. If you know the format of the log file you can explicitly specify it, for example `@type apache2` or `@type json`.
The above configuration would translate at runtime to something similar to this:
```xml
<source>
@type tail
path /var/lib/kubelet/pods/723dd34a-4ac0-11e8-8a81-0a930dd884b0/volumes/kubernetes.io~empty-dir/logs/welcome.log
pos_file /var/log/kfotail-7020a0b821b0d230d89283ba47d9088d9b58f97d.pos
read_from_head true
tag kube.kfo-test.welcome-logger.test-container
<parse>
@type none
</parse>
</source>
```
### Dealing with multi-line exception stacktraces (since v1.3.0)
Most log streams are line-oriented. However, stacktraces always span multiple lines. *kube-fluentd-operator* integrates stacktrace processing using the [fluent-plugin-detect-exceptions](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions). If a Java-based pod produces stacktraces in the logs, then the stacktraces can be collapsed in a single log event like this:
```xml
<filter $labels(app=jpetstore)>
@type detect_exceptions
# you can skip language in which case all possible languages will be tried: go, java, python, ruby, etc...
language java
</filter>
# The rest of the configuration stays the same even though quite a lot of tag rewriting takes place
<match **>
@type es
</match>
```
Notice how `filter` is used instead of `match` as described in [fluent-plugin-detect-exceptions](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions). Internally, this filter is translated into several `match` directives so that the end user doesn't need to bother with rewriting the Fluentd tag.
Also, users don't need to bother with setting the correct `stream` parameter. *kube-fluentd-operator* generates one internally based on the container id and the stream.
### Reusing output plugin definitions (since v1.6.0)
Sometimes you only have a few valid options for log sinks: a dedicated S3 bucket, the ELK stack you manage, etc. The only flexibility you're after is letting namespace owners filter and parse their logs. In such cases you can abstract over an output plugin configuration - basically reducing it to a simple name which can be referenced from any namespace. For example, let's assume you have an S3 bucket for a "test" environment and you use loggly for a "staging" environment. The first thing you do is define these two output in the *admin* namespace:
```xml
admin-ns.conf:
<match systemd.** docker kube.kube-system.** k8s.**>
@type loggly
loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/fluentd
</match>
<plugin test>
@type s3
aws_key_id YOUR_AWS_KEY_ID
aws_sec_key YOUR_AWS_SECRET_KEY
s3_bucket YOUR_S3_BUCKET_NAME
s3_region AWS_REGION
</plugin>
<plugin staging>
@type loggly
loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/fluentd
</plugin>
```
In the above example for the admin configuration, the `match` directive is first defined to direct where to send logs for the `systemd`, `docker`, `kube-system`, and kubernetes control plane components. Below the `match` directive we have defined the `plugin` directives which define the log sinks that can be reused by namespace configurations.
A namespace can refer to the `staging` and `test` plugins oblivious to the fact where exactly the logs end up:
```xml
acme-test.conf
<match **>
@type test
</match>
acme-staging.conf
<match **>
@type staging
</match>
```
kube-fluentd-operator will insert the content of the `plugin` directive in the `match` directive. From then on, regular validation and postprocessing takes place.
### Retagging based on log contents (since v1.12.0)
Sometimes you might need to split a single log stream to perform different processing based on the contents of one of the fields. To achieve this you can use the `retag` plugin that allows to specify a set of rules that match regular expressions against the specified fields. If one of the rules matches, the log is re-emitted with a new namespace-unique tag based on the specified tag.
Logs that are emitted by this plugin can be consequently filtered and processed by using the `$tag` macro when specifiying the tag:
```xml
<match $labels(app=apache)>
@type retag
<rule>
key message
pattern /^(ERROR) .*$/
tag notifications.$1 # refer to a capturing group using $number
</rule>
<rule>
key message
pattern /^(FATAL) .*$/
tag notifications.$1
</rule>
<rule>
key message
pattern /^(ERROR)|(FATAL) .*$/
tag notifications.other
invert true # rewrite tag when unmatch pattern
</rule>
</match>
<filter $tag(notifications.ERROR)>
# perform some extra processing
</filter>
<filter $tag(notifications.FATAL)>
# perform different processing
</filter>
<match $tag(notifications.**)>
# send to common output plugin
</match>
```
*kube-fluentd-operator* ensures that tags specified using the `$tag` macro never conflict with tags from other namespaces, even if the tag itself is equivalent.
### Sharing logs between namespaces
By default, you can consume logs only from your namespaces. Often it is useful for multiple namespaces (tenants) to get access to the logs streams of a shared resource (pod, namespace). *kube-fluentd-operator* makes it possible using two constructs: the source namespace expresses its intent to share logs with a destination namespace and the destination namespace expresses its desire to consume logs from a source. As a result logs are streamed only when both sides agree.
A source namespace can share with another namespace using the `@type share` macro:
producer namespace configuration:
```xml
<match $labels(msg=nginx-ingress)>
@type copy
<store>
@type share
# share all logs matching the labels with the namespace "consumer"
with_namespace consumer
</store>
</match>
```
consumer namespace configuration:
```xml
# use $from(producer) to get all shared logs from a namespace called "producer"
<label @$from(producer)>
<match **>
# process all shared logs here as usual
</match>
</match>
```
The consuming namespace can use the usual syntax inside the `<label @$from...>` directive. The fluentd tag is being rewritten as if the logs originated from the same namespace.
The producing namespace need to wrap `@type share` within a `<store>` directive. This is done on purpose as it is very easy to just redirect the logs to the destination namespace and lose them. The `@type copy` clones the whole stream.
### Log metadata
Often you run mulitple Kubernetes clusters but you need to aggregate all logs to a single destination. To distinguish between different sources, `kube-fluentd-operator` can attach arbitrary metadata to every log event.
The metadata is nested under a key chosen with `--meta-key`. Using the helm chart, metadata can be enabled like this:
```bash
helm install ... \
--set meta.key=metadata \
--set meta.values.region=us-east-1 \
--set meta.values.env=staging \
--set meta.values.cluster=legacy
```
Every log event, be it from a pod, mounted-file or a systemd unit, will now carry this metadata:
```json
{
"metadata": {
"region": "us-east-1",
"env": "staging",
"cluster": "legacy",
}
}
```
All logs originating from a file look exactly as all other Kubernetes logs. However, their `stream` field is not set to `stdout` but to the path to the source file:
```json
{
"message": "Some message from the welcome-logger pod",
"stream": "/var/log/welcome.log",
"kubernetes": {
"container_name": "test-container",
"host": "ip-11-11-11-11.us-east-2.compute.internal",
"namespace_name": "kfo-test",
"pod_id": "723dd34a-4ac0-11e8-8a81-0a930dd884b0",
"pod_name": "welcome-logger",
"labels": {
"msg": "welcome",
"test-case": "b"
},
"namespace_labels": {}
},
"metadata": {
"region": "us-east-2",
"cluster": "legacy",
"env": "staging"
}
}
```
### Custom resource definition(CRD) support (since v1.13.0)
Custom resources are introduced from v1.13.0 release onwards. It allows to have a dedicated resource for fluentd configurations, which enables to manage them in a more consistent way and move away from the generic ConfigMaps.
It is possible to create configs for a new application simply by attaching a FluentdConfig resource to the application manifests, rather than using a more generic ConfigMap with specific names and/or labels.
```xml
apiVersion: logs.vdp.vmware.com/v1beta1
kind: FluentdConfig
metadata:
name: fd-config
spec:
fluentconf: |
<match kube.ns.**>
@type relabel
@label @NOTIFICATIONS
</match>
<label @NOTIFICATIONS>
<match **>
@type null
</match>
</label>
```
The "crd" has been introduced as a new datasource, configurable through the helm chart values, to allow users that are currently set up with ConfigMaps and do not want to perform the switchover to FluentdConfigs, to be able to keep on using them. The config-reloader has been equipped with the capability of installing the CRD at startup if requested, so no manual actions to enable it on the cluster are needed.
The existing configurations though ConfigMaps can be migrated to CRDs through the following migration flow
* A new user, who is installing kube-fluentd-operator for the first time, should set the datasource: crd option in the chart. This enables the crd support
* A user who is already using kube-fluentd-operator with either datasource: default or datasource: multimap will have update to the new chart and set the 'crdMigrationMode' property to 'true'. This enables the config-reloader to launch with the crd datasource and the legacy datasource (either default or multimap depending on what was configured in the datasource property). The user can slowly migrate one by one all configmap resources to the corresponding fluentdconfig resources. When the migration is complete, the Helm release can be upgraded by changing the 'crdMigrationMode' property to 'false' and switching the datasource property to 'crd'. This will effectively disable the legacy datasource and set the config-reloader to only watch fluentdconfig resources.
## Tracking Fluentd version
This projects tries to keep up with major releases for [Fluentd docker image](https://github.com/fluent/fluentd-docker-image/).
| Fluentd version | Operator version |
|----------------------------|-------------------------|
| 0.12.x | 1.0.0 |
| 1.1.0 | 1.2.0 |
| 1.1.3 | 1.3.0 |
| 1.2.6 | 1.8.0 |
| 1.5.2 | 1.10.0 |
| 1.9.1 | 1.12.0 |
| 1.12.2 | 1.14.0 |
| 1.12.3 | 1.14.1 |
| 1.13.0 | 1.15.0 |
| 1.13.1 | 1.15.1 |
| 1.13.3 | 1.15.2 |
| 1.14.0 | 1.15.3 |
| 1.14.1 | 1.16.0 |
| 1.14.2 | 1.16.1 |
| 1.14.2 | 1.16.2 |
| 1.14.4 | 1.16.3 |
| 1.14.4 | 1.16.4 |
| 1.14.4 | 1.16.5 |
| 1.14.6 | 1.16.6 |
| 1.14.6 | 1.16.7 |
| 1.15.3 | 1.16.8 |
## Plugins in latest release (1.16.7)
`kube-fluentd-operator` aims to be easy to use and flexible. It also favors sending logs to multiple destinations using `<copy>` and as such comes with many plugins pre-installed:
* fluentd (1.15.3)
* fluent-config-regexp-type (1.0.0)
* fluent-mixin-config-placeholders (0.4.0)
* fluent-plugin-amqp (0.14.0)
* fluent-plugin-azure-loganalytics (0.7.0)
* fluent-plugin-cloudwatch-logs (0.14.3)
* fluent-plugin-concat (2.5.0)
* fluent-plugin-datadog (0.14.2)
* fluent-plugin-detect-exceptions (0.0.14) - forked to allow fluentd v1 plugin api
* fluent-plugin-elasticsearch (5.2.4)
* fluent-plugin-opensearch (1.0.9)
* fluent-plugin-gelf-hs (1.0.8)
* fluent-plugin-google-cloud (0.13.0) - forked to allow fluentd v1.14.x
* fluent-plugin-grafana-loki (1.2.19)
* fluent-plugin-grok-parser (2.6.2)
* fluent-plugin-json-in-json-2 (1.0.2)
* fluent-plugin-kafka (0.18.1)
* fluent-plugin-kinesis (3.4.2)
* fluent-plugin-kubernetes (0.3.1)
* fluent-plugin-kubernetes_metadata_filter (2.13.0)
* fluent-plugin-kubernetes_sumologic (2.4.2)
* fluent-plugin-logentries (0.2.10)
* fluent-plugin-loggly (1.0.0) - forked to fix for new fluentd api
* fluent-plugin-logzio (0.0.22)
* fluent-plugin-mail (0.3.0)
* fluent-plugin-mongo (1.5.0)
* fluent-plugin-multi-format-parser (1.0.0)
* fluent-plugin-out-http (1.3.3)
* fluent-plugin-papertrail (0.2.8)
* fluent-plugin-prometheus (2.0.2)
* fluent-plugin-record-modifier (2.1.0)
* fluent-plugin-record-reformer (0.9.1)
* fluent-plugin-redis (0.3.5)
* fluent-plugin-remote_syslog (1.0.0)
* fluent-plugin-rewrite-tag-filter (2.4.0)
* fluent-plugin-route (1.0.0)
* fluent-plugin-s3 (1.7.2)
* fluent-plugin-secure-forward (0.4.5)
* fluent-plugin-splunk-hec (1.3.1)
* fluent-plugin-splunkhec (2.3)
* fluent-plugin-sumologic_output (1.7.2)
* fluent-plugin-systemd (1.0.5)
* fluent-plugin-uri-parser (0.3.0)
* fluent-plugin-verticajson (0.0.6)
* fluent-plugin-vmware-loginsight (1.3.1)
* fluent-plugin-vmware-log-intelligence (2.0.6)
* fluent-plugin-mysqlslowquery (0.0.9)
* fluent-plugin-throttle (0.0.5)
* fluent-plugin-webhdfs (1.5.0)
When customizing the image be careful not to uninstall plugins that are used internally to implement the macros.
If you need other destination plugins you are welcome to contribute a patch or just create an issue.
## Synopsis
The config-reloader binary is the one that listens to changes in K8S and generates Fluentd files. It runs as a daemonset and is not intended to interact with directly. The synopsis is useful when trying to understand the Helm chart or just hacking.
```txt
usage: config-reloader [<flags>]
Regenerates Fluentd configs based Kubernetes namespace annotations against templates, reloading
Fluentd if necessary
Flags:
--help Show context-sensitive help (also try --help-long and
--help-man).
--version Show application version.
--master="" The Kubernetes API server to connect to (default: auto-detect)
--kubeconfig="" Retrieve target cluster configuration from a Kubernetes
configuration file (default: auto-detect)
--datasource=default Datasource to use (default|fake|fs|multimap|crd)
--crd-migration-mode Enable the crd datasource together with the current datasource to facilitate the migration (used only with --datasource=default|multimap)
--fs-dir=FS-DIR If datasource=fs is used, configure the dir hosting the files
--interval=60 Run every x seconds
--allow-file Allow @type file for namespace configuration
--id="default" The id of this deployment. It is used internally so that two
deployments don't overwrite each other's data
--fluentd-rpc-port=24444 RPC port of Fluentd
--log-level="info" Control verbosity of config-reloader logs
--fluentd-loglevel="info" Control verbosity of fluentd logs
--buffer-mount-folder="" Folder in /var/log/{} where to create all fluentd buffers
--annotation="logging.csp.vmware.com/fluentd-configmap"
Which annotation on the namespace stores the configmap name?
--default-configmap="fluentd-config"
Read the configmap by this name if namespace is not annotated.
Use empty string to suppress the default.
--status-annotation="logging.csp.vmware.com/fluentd-status"
Store configuration errors in this annotation, leave empty to
turn off
--kubelet-root="/var/lib/kubelet/"
Kubelet root dir, configured using --root-dir on the kubelet
service
--namespaces=NAMESPACES ... List of namespaces to process. If empty, processes all namespaces
--templates-dir="/templates" Where to find templates
--output-dir="/fluentd/etc" Where to output config files
--meta-key=META-KEY Attach metadat under this key
--meta-values=META-VALUES Metadata in the k=v,k2=v2 format
--fluentd-binary=FLUENTD-BINARY
Path to fluentd binary used to validate configuration
--prometheus-enabled Prometheus metrics enabled (default: false)
--admin-namespace="kube-system"
The namespace to be treated as admin namespace
```
## Helm chart
| Parameter | Description | Default |
|------------------------------------------|-------------------------------------|---------------------------------------------------|
| `rbac.create` | Create a serviceaccount+role, use if K8s is using RBAC | `false` |
| `serviceAccountName` | Reuse an existing service account | `""` |
| `defaultConfigmap` | Read the configmap by this name if the namespace is not annotated | `"fluentd-config"` |
| `image.repositiry` | Repository | `vmware/kube-fluentd-operator` |
| `image.tag` | Image tag | `latest` |
| `image.pullPolicy` | Pull policy | `Always` |
| `image.pullSecret` | Optional pull secret name | `""` |
| `logLevel` | Default log level for config-reloader | `info` |
| `fluentdLogLevel` | Default log level for fluentd | `info` |
| `bufferMountFolder` | Folder in /var/log/{} where to create all fluentd buffers | `""` |
| `kubeletRoot` | The home dir of the kubelet, usually set using `--root-dir` on the kubelet | `/var/lib/kubelet` |
| `namespaces` | List of namespaces to operate on. Empty means all namespaces | `[]` |
| `interval` | How often to check for config changes (seconds) | `45` |
| `meta.key` | The metadata key (optional) | `""` |
| `meta.values` | Metadata to use for the key | `{}`
| `extraVolumes` | Extra volumes | |
| `fluentd.extraVolumeMounts` | Mount extra volumes for the fluentd container, required to mount ssl certificates when elasticsearch has tls enabled | |
| `fluentd.resources` | Resource definitions for the fluentd container | `{}`|
| `fluentd.extraEnv` | Extra env vars to pass to the fluentd container | `{}` |
| `reloader.extraVolumeMounts` | Mount extra volumes for the reloader container | |
| `reloader.resources` | Resource definitions for the reloader container | `{}` |
| `reloader.extraEnv` | Extra env vars to pass to the reloader container | `{}` |
| `tolerations` | Pod tolerations | `[]` |
| `updateStrategy` | UpdateStrategy for the daemonset. Leave empty to get the K8S' default (probably the safest choice) | `{}` |
| `podAnnotations` | Pod annotations for the daemonset | |
| `adminNamespace` | The namespace to be treated as admin namespace | `kube-system` |
## Cookbook
### I want to use one destination for everything
Simple, define configuration only for the *admin* namespace (by default `kube-system`):
```bash
kube-system.conf:
<match **>
# configure destination here
</match>
```
### I dont't care for systemd and docker logs
Simple, exclude them at the *admin* namespace level (by default `kube-system`):
```bash
kube-system.conf:
<match systemd.** docker>
@type null
</match>
<match **>
# all but systemd.** is still around
# configure destination
</match>
```
### I want to use one destination but also want to just exclude a few pods
It is not possible to handle this globally. Instead, provide this config for the noisy namespace and configure other namespaces at the cost of some code duplication:
```xml
noisy-namespace.conf:
<match $labels(app=verbose-logger)>
@type null
</match>
# all other logs are captured here
<match **>
@type ...
</match>
```
On the bright side, the configuration of `noisy-namespace` contains nothing specific to noisy-namespace and the same content can be used for all namespaces whose logs we need collected.
### I am getting errors "namespaces is forbidden: ... cannot list namespaces at the cluster scope"
Your cluster is running under RBAC. You need to enable a serviceaccount for the log-router pods. It's easy when using the Helm chart:
```bash
helm install ./charts/log-router --set rbac.create=true ...
```
### I have a legacy container that logs to /var/log/httpd/access.log
First you need version 1.1.0 or later. At the namespace level you need to add a `source` directive of type `mounted-file`:
```xml
<source>
@type mounted-file
path /var/log/httpd/access.log
labels app=apache2
<parse>
@type apache2
</parse>
</source>
<match **>
# destination config omitted
</match>
```
The type `mounted-file` is again a macro that is expanded to a `tail` plugin. The `<parse>` directive is optional and if not set a `@type none` will be used instead.
In order for this to work the pod must define a mount of type `emptyDir` at `/var/log/httpd` or any of it parent folders. For example, this pod definition is part of the test suite (it logs to /var/log/hello.log):
```yaml
apiVersion: v1
kind: Pod
metadata:
name: hello-logger
namespace: kfo-test
labels:
msg: hello
spec:
containers:
- image: ubuntu
name: greeter
command:
- bash
- -c
- while true; do echo `date -R` [INFO] "Random hello number $((var++)) to file"; sleep 2; [[ $(($var % 100)) == 0 ]] && :> /var/log/hello.log ;done > /var/log/hello.log
volumeMounts:
- mountPath: /var/log
name: logs
volumes:
- name: logs
emptyDir: {}
```
To get the hello.log ingested by Fluentd you need at least this in the configuration for `kfo-test` namespace:
```xml
<source>
@type mounted-file
# need to specify the path on the container filesystem
path /var/log/hello.log
# only look at pods labeled this way
labels msg=hello
<parse>
@type none
</parse>
</source>
<match $labels(msg=hello)>
# store the hello.log somewhere
@type ...
</match>
```
### I want to push logs from namespace `demo` to logz.io
```xml
demo.conf:
<match **>
@type logzio_buffered
endpoint_url https://listener.logz.io:8071?token=TOKEN&type=log-router
output_include_time true
output_include_tags true
<buffer>
@type memory
flush_thread_count 4
flush_interval 3s
queue_limit_length 4096
</buffer>
</match>
```
For details you should consult the plugin documentation.
### I want to push logs to a remote syslog server
The built-in `remote_syslog` plugin cannot be used as the fluentd tag may be longer than 32 bytes. For this reason there is a `truncating_remote_syslog` plugin that shortens the tag to the allowed limit. If you are currently using the `remote_syslog` output plugin you only need to change a single line:
```xml
<match **>
# instead of "remote_syslog"
@type truncating_remote_syslog
# the usual config for remote_syslog
</match>
```
To get the general idea how truncation works, consider this table:
| Original Tag | Truncated tag |
|-------------|----------------|
| `kube.demo.test.test` | `demo.test.test` |
| `kube.demo.nginx-65899c769f-5zj6d.nginx` | `demo.nginx-65899c769f-5zj*.nginx` |
| `kube.demo.test.nginx11111111._lablels.hello` | `demo.test.nginx11111111` |
### I want to push logs to Humio
Humio speaks the elasticsearh protocol so configuration is pretty similar to Elasticsearch. The example bellow is based on https://github.com/humio/kubernetes2humio/blob/master/fluentd/docker-image/fluent.conf.
```xml
<match **>
@type elasticsearch
include_tag_key false
host "YOUR_HOST"
path "/api/v1/dataspaces/YOUR_NAMESPACE/ingest/elasticsearch/"
scheme "https"
port "443"
user "YOUR_KEY"
password ""
logstash_format true
reload_connections "true"
logstash_prefix "fluentd:kubernetes2humio"
buffer_chunk_limit 1M
buffer_queue_limit 32
flush_interval 1s
max_retry_wait 30
disable_retry_limit
num_threads 8
</match>
```
### I want to push logs to papertrail
```xml
test.conf:
<match **>
@type papertrail
papertrail_host YOUR_HOST.papertrailapp.com
papertrail_port YOUR_PORT
flush_interval 30