The interceptors cannot remove fields such as stream and time

[xiaojiayu404]

环境：kubernetes1.21 + docker
logconfig
apiVersion: loggie.io/v1beta1 kind: LogConfig metadata: name: nginx namespace: default spec: pipeline: interceptorsRef: nginx-interce sinkRef: nginx-sink sources: | - type: file name: mylog containerName: nginx fields: topic: "nginx-access" matchFields: labelKey: [app] paths: - stdout selector: labelSelector: app: nginx type: pod
Interceptor
apiVersion: loggie.io/v1beta1 kind: Interceptor metadata: name: nginx-interce spec: interceptors: | - type: normalize name: stdproc belongTo: ["mylog"] processors: - jsonDecode: target: body - drop: targets: ["stream", "time", "body"] - rename: convert: - from: "log" to: "message"
sink
apiVersion: loggie.io/v1beta1 kind: Sink metadata: name: nginx-sink spec: sink: | type: dev printEvents: true

部署完成请求nginx，查看loggie pod日志为：
{ "fields": { "namespace": "default", "nodename": "10.0.20.28", "podname": "nginx-6799fc88d8-td4sc", "containername": "nginx", "logconfig": "nginx", "topic": "nginx-access" }, "body": "{\"log\":\"10.203.2.0 - - [21/Mar/2022:14:47:44 +0000] \\\"GET / HTTP/1.1\\\" 200 615 \\\"-\\\" \\\"curl/7.29.0\\\" \\\"-\\\"\\n\",\"stream\":\"stdout\",\"time\":\"2022-03-21T14:47:44.246358969Z\"}" }
日志中log字段没有被替换，且stream、time字段未被删除

[ethfoo]

麻烦先仅配置jsonDecode，然后配置dev sink，打印一下输出看下

[xiaojiayu404]

麻烦先仅配置jsonDecode，然后配置dev sink，打印一下输出看下

好的，在loggie pod看到日志如下：

{"level":"info","time":"2022-03-22T02:35:43Z","caller":"/go/src/loggie.io/loggie/pkg/sink/dev/sink.go:98","message":"event: {"fields":{"app":"nginx","containername":"nginx","logconfig":"nginx","namespace":"default","nodename":"10.10.20.8","podname":"nginx-6799fc88d8-td4sc","topic":"nginx-access"},"body":"{"log":"10.2.2.0 - - [22/Mar/2022:02:35:41 +0000] \"GET / HTTP/1.1\" 200 615 \"-\" \"curl/7.29.0\" \"-\"\n","stream":"stdout","time":"2022-03-22T02:35:41.641575846Z"}"}"}

[ethfoo]

我有点怀疑你的容器里实际的interceptors配置没有生效，麻烦进到loggie容器里看下：

cat /opt/loggie/pipelines/kube-loggie.yml

现在修改interceptor cr，是不会直接触发被引用的logConfig重新渲染配置，这个后面考虑会优化一下

[xiaojiayu404]

我有点怀疑你的容器里实际的interceptors配置没有生效，麻烦进到loggie容器里看下：

cat /opt/loggie/pipelines/kube-loggie.yml

现在修改interceptor cr，是不会直接触发被引用的logConfig重新渲染配置，这个后面考虑会优化一下

/opt/loggie/pipeline # cat kube-loggie.yml
pipelines:

• name: default/nginx
  sources:
  • fields:
    app: nginx
    containername: nginx
    logconfig: nginx
    namespace: default
    nodename: 10.10.20.8
    podname: nginx-6799fc88d8-td4sc
    topic: nginx-access
    name: nginx-6799fc88d8-td4sc/nginx/mylog
    paths:
    • /var/lib/docker/containers/a56d54a98e79a4e59e0b7c78e321c29d6436195d616d48facf20b562efd165d1/a56d54a98e79a4e59e0b7c78e321c29d6436195d616d48facf20b562efd165d1-json.log
      type: file
      sink:
      printEvents: true
      type: dev

配置interceptor 之后，有delete loggie pod，因为之前发现修改logconfig后也有配置不生效的情况。 现在如何触发重新渲染配置呢？

[ethfoo]

应该是interceptorRef，这个我们在文档里确实有的地方写成interceptorsRef，多写了一个s，我们修改一下文档，算是文档bug。

[xiaojiayu404]

应该是interceptorRef，这个我们在文档里确实有的地方写成interceptorsRef，多写了一个s，我们修改一下文档，算是文档bug。

测试OK，tks