Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove fluent-bit rawConfig mentions after upgrading to new fluent-bit chart #1264

Merged
merged 1 commit into from
Dec 28, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions deploy/docs/Best_Practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,16 +246,23 @@ you will need to ensure the logs get mounted to the host so fluent-bit can be
configured to capture from the host.

Example:
In `values.yaml` in the `fluent-bit.rawConfig` section, you have to add
a new input specifying the file path, eg.
In `values.yaml` in the `fluent-bit.config.input` section, you have to add
a new `INPUT` specifying the file path (retaining the remaining part of `input`
config), e.g.:

```bash
[INPUT]
Name tail
Path /var/log/syslog
```yaml
fluent-bit:
config:
# ...
inputs: |-
# Copy original fluent-bit.config.inputs here
# ...
[INPUT]
Name tail
Path /var/log/syslog
```

Reference: https://fluentbit.io/documentation/0.12/input/tail.html
Reference: https://docs.fluentbit.io/manual/pipeline/inputs/tail#configuration-file

## Filtering Prometheus Metrics by Namespace

Expand Down
5 changes: 4 additions & 1 deletion deploy/helm/sumologic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,10 @@ Parameter | Description | Default
`metrics-server.args` | Arguments for metric server. | `["--kubelet-insecure-tls","--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname"]`
`fluent-bit.resources` | Resources for Fluent-bit daemonsets. | `{}`
`fluent-bit.enabled` | Flag to control deploying Fluent-bit Helm sub-chart. | `true`
`fluent-bit.config.service` | Configure Fluent-bit Helm sub-chart service. | [fluent-bit.config.service in values.yaml](https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/v2.0.0-beta.0/deploy/helm/sumologic/values.yaml#L816-L824)
`fluent-bit.config.inputs` | Configure Fluent-bit Helm sub-chart inputs. | [fluent-bit.config.inputs in values.yaml](https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/v2.0.0-beta.0/deploy/helm/sumologic/values.yaml#L826-L891)
`fluent-bit.config.outputs` | Configure Fluent-bit Helm sub-chart outputs. | [fluent-bit.config.outputs in values.yaml](https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/v2.0.0-beta.0/deploy/helm/sumologic/values.yaml#L894-L902)
`fluent-bit.config.customParsers` | Configure Fluent-bit Helm sub-chart customParsers. | [fluent-bit.config.customParsers in values.yaml](https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/v2.0.0-beta.0/deploy/helm/sumologic/values.yaml#L904-L913)
`fluent-bit.service.labels` | Labels for fluent-bit service. | `{sumologic.com/scrape: "true"}`
`fluent-bit.podLabels` | Additional labels for fluent-bit pods. | `{}`
`fluent-bit.podAnnotations` | Additional annotations for fluent-bit pods. | `{}`
Expand All @@ -182,7 +186,6 @@ Parameter | Description | Default
`fluent-bit.input.systemd.enabled` | Enable systemd input. | `true`
`fluent-bit.parsers.enabled` | Enable custom parsers. | `true`
`fluent-bit.parsers.regex` | List of regex parsers. | `[{"name":"multi_line","regex":"(?\u003clog\u003e^{\"log\":\"\\d{4}-\\d{1,2}-\\d{1,2}.\\d{2}:\\d{2}:\\d{2}.*)"}]`
`fluent-bit.rawConfig` | DESCRIPTION | `@INCLUDE fluent-bit-service.conf [INPUT] Name tail Path /var/log/containers/*.log Multiline On Parser_Firstline multi_line Tag containers.* Refresh_Interval 1 Rotate_Wait 60 Mem_Buf_Limit 5MB Skip_Long_Lines On DB /tail-db/tail-containers-state-sumo.db DB.Sync Normal [INPUT] Name systemd Tag host.* Systemd_Filter _SYSTEMD_UNIT=addon-config.service Systemd_Filter _SYSTEMD_UNIT=addon-run.service Systemd_Filter _SYSTEMD_UNIT=cfn-etcd-environment.service Systemd_Filter _SYSTEMD_UNIT=cfn-signal.service Systemd_Filter _SYSTEMD_UNIT=clean-ca-certificates.service Systemd_Filter _SYSTEMD_UNIT=containerd.service Systemd_Filter _SYSTEMD_UNIT=coreos-metadata.service Systemd_Filter _SYSTEMD_UNIT=coreos-setup-environment.service Systemd_Filter _SYSTEMD_UNIT=coreos-tmpfiles.service Systemd_Filter _SYSTEMD_UNIT=dbus.service Systemd_Filter _SYSTEMD_UNIT=docker.service Systemd_Filter _SYSTEMD_UNIT=efs.service Systemd_Filter _SYSTEMD_UNIT=etcd-member.service Systemd_Filter _SYSTEMD_UNIT=etcd.service Systemd_Filter _SYSTEMD_UNIT=etcd2.service Systemd_Filter _SYSTEMD_UNIT=etcd3.service Systemd_Filter _SYSTEMD_UNIT=etcdadm-check.service Systemd_Filter _SYSTEMD_UNIT=etcdadm-reconfigure.service Systemd_Filter _SYSTEMD_UNIT=etcdadm-save.service Systemd_Filter _SYSTEMD_UNIT=etcdadm-update-status.service Systemd_Filter _SYSTEMD_UNIT=flanneld.service Systemd_Filter _SYSTEMD_UNIT=format-etcd2-volume.service Systemd_Filter _SYSTEMD_UNIT=kube-node-taint-and-uncordon.service Systemd_Filter _SYSTEMD_UNIT=kubelet.service Systemd_Filter _SYSTEMD_UNIT=ldconfig.service Systemd_Filter _SYSTEMD_UNIT=locksmithd.service Systemd_Filter _SYSTEMD_UNIT=logrotate.service Systemd_Filter _SYSTEMD_UNIT=lvm2-monitor.service Systemd_Filter _SYSTEMD_UNIT=mdmon.service Systemd_Filter _SYSTEMD_UNIT=nfs-idmapd.service Systemd_Filter _SYSTEMD_UNIT=nfs-mountd.service Systemd_Filter _SYSTEMD_UNIT=nfs-server.service Systemd_Filter _SYSTEMD_UNIT=nfs-utils.service Systemd_Filter _SYSTEMD_UNIT=node-problem-detector.service Systemd_Filter _SYSTEMD_UNIT=ntp.service Systemd_Filter _SYSTEMD_UNIT=oem-cloudinit.service Systemd_Filter _SYSTEMD_UNIT=rkt-gc.service Systemd_Filter _SYSTEMD_UNIT=rkt-metadata.service Systemd_Filter _SYSTEMD_UNIT=rpc-idmapd.service Systemd_Filter _SYSTEMD_UNIT=rpc-mountd.service Systemd_Filter _SYSTEMD_UNIT=rpc-statd.service Systemd_Filter _SYSTEMD_UNIT=rpcbind.service Systemd_Filter _SYSTEMD_UNIT=set-aws-environment.service Systemd_Filter _SYSTEMD_UNIT=system-cloudinit.service Systemd_Filter _SYSTEMD_UNIT=systemd-timesyncd.service Systemd_Filter _SYSTEMD_UNIT=update-ca-certificates.service Systemd_Filter _SYSTEMD_UNIT=user-cloudinit.service Systemd_Filter _SYSTEMD_UNIT=var-lib-etcd2.service Max_Entries 1000 Read_From_Tail true @INCLUDE fluent-bit-output.conf`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we add fluent-bit.config.input etc instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could (should?) but how do you suggest to put the multiline yaml code snippet in the table?

inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
Docker_Mode On
Docker_Mode_Parser multi_line
Tag containers.*
Refresh_Interval 1
Rotate_Wait 60
Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB /tail-db/tail-containers-state-sumo.db
DB.Sync Normal
[INPUT]
Name systemd
Tag host.*
DB /tail-db/systemd-state-sumo.db
Systemd_Filter _SYSTEMD_UNIT=addon-config.service
Systemd_Filter _SYSTEMD_UNIT=addon-run.service
Systemd_Filter _SYSTEMD_UNIT=cfn-etcd-environment.service
Systemd_Filter _SYSTEMD_UNIT=cfn-signal.service
Systemd_Filter _SYSTEMD_UNIT=clean-ca-certificates.service
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Systemd_Filter _SYSTEMD_UNIT=coreos-metadata.service
Systemd_Filter _SYSTEMD_UNIT=coreos-setup-environment.service
Systemd_Filter _SYSTEMD_UNIT=coreos-tmpfiles.service
Systemd_Filter _SYSTEMD_UNIT=dbus.service
Systemd_Filter _SYSTEMD_UNIT=docker.service
Systemd_Filter _SYSTEMD_UNIT=efs.service
Systemd_Filter _SYSTEMD_UNIT=etcd-member.service
Systemd_Filter _SYSTEMD_UNIT=etcd.service
Systemd_Filter _SYSTEMD_UNIT=etcd2.service
Systemd_Filter _SYSTEMD_UNIT=etcd3.service
Systemd_Filter _SYSTEMD_UNIT=etcdadm-check.service
Systemd_Filter _SYSTEMD_UNIT=etcdadm-reconfigure.service
Systemd_Filter _SYSTEMD_UNIT=etcdadm-save.service
Systemd_Filter _SYSTEMD_UNIT=etcdadm-update-status.service
Systemd_Filter _SYSTEMD_UNIT=flanneld.service
Systemd_Filter _SYSTEMD_UNIT=format-etcd2-volume.service
Systemd_Filter _SYSTEMD_UNIT=kube-node-taint-and-uncordon.service
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _SYSTEMD_UNIT=ldconfig.service
Systemd_Filter _SYSTEMD_UNIT=locksmithd.service
Systemd_Filter _SYSTEMD_UNIT=logrotate.service
Systemd_Filter _SYSTEMD_UNIT=lvm2-monitor.service
Systemd_Filter _SYSTEMD_UNIT=mdmon.service
Systemd_Filter _SYSTEMD_UNIT=nfs-idmapd.service
Systemd_Filter _SYSTEMD_UNIT=nfs-mountd.service
Systemd_Filter _SYSTEMD_UNIT=nfs-server.service
Systemd_Filter _SYSTEMD_UNIT=nfs-utils.service
Systemd_Filter _SYSTEMD_UNIT=node-problem-detector.service
Systemd_Filter _SYSTEMD_UNIT=ntp.service
Systemd_Filter _SYSTEMD_UNIT=oem-cloudinit.service
Systemd_Filter _SYSTEMD_UNIT=rkt-gc.service
Systemd_Filter _SYSTEMD_UNIT=rkt-metadata.service
Systemd_Filter _SYSTEMD_UNIT=rpc-idmapd.service
Systemd_Filter _SYSTEMD_UNIT=rpc-mountd.service
Systemd_Filter _SYSTEMD_UNIT=rpc-statd.service
Systemd_Filter _SYSTEMD_UNIT=rpcbind.service
Systemd_Filter _SYSTEMD_UNIT=set-aws-environment.service
Systemd_Filter _SYSTEMD_UNIT=system-cloudinit.service
Systemd_Filter _SYSTEMD_UNIT=systemd-timesyncd.service
Systemd_Filter _SYSTEMD_UNIT=update-ca-certificates.service
Systemd_Filter _SYSTEMD_UNIT=user-cloudinit.service
Systemd_Filter _SYSTEMD_UNIT=var-lib-etcd2.service
Max_Entries 1000
Read_From_Tail true

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This answer on SO says that it's not possible to add multiline code snippet in a table cell in markdown:

https://stackoverflow.com/a/28514979

`kube-prometheus-stack.kubeTargetVersionOverride` | Provide a target gitVersion of K8S, in case .Capabilites.KubeVersion is not available (e.g. helm template). Changing this may break Sumo Logic apps. | `1.13.0-0`
`kube-prometheus-stack.enabled` | Flag to control deploying Prometheus Operator Helm sub-chart. | `true`
`kube-prometheus-stack.alertmanager.enabled` | Deploy alertmanager. | `false`
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/sumologic/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -824,6 +824,7 @@ fluent-bit:
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Expand Down