[BUG]: ilogtail 将日志输入到 Loki 异常 #1020

ixiaoyi93 · 2023-07-18T08:01:09Z

Describe the bug
A clear and concise description of what the bug is, ideally within 20 words.

iLogtail Running Environment
Please provide the following information:
通过ilogtail 采集容器标准输出日志，在输出到 loki 时通过 DynamicLabels 和 Convert.TagFieldsRename 实现 tags 内的字段重命名。出现 server returned HTTP status 400 Bad Request (400): 1:6: parse error: unexpected character inside braces: '.' 异常错误。如果去掉该功能采用静态表情是能够正常输出到 loki 的。

ilogtail version:
v1.7.0
Yaml configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ilogtail-configmap
  namespace: ilogtail
data:
  default_pod_stdout.yaml: |
    enable: true
    inputs:
      - Type: service_docker_stdout
        Stderr: true
        Stdout: true
        BeginLineCheckLength: 10
        BeginLineRegex: "\\d+-\\d+-\\d+.*"
    processors:
      - Type: processor_add_fields
        Fields:
          cluster: aliyun-public-test
        IgnoreIfExist: false
    flushers:
      - Type: flusher_loki
        URL: http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push
        DynamicLabels:
          - tag.host.name
        Convert:
          TagFieldsRename:
            host.name: my.host.name

ilogtail.LOG:

[2023-07-18 07:49:54.704539]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:156              AppConfigBase AppConfigBase:success
[2023-07-18 07:49:54.704556]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:320              load env tags from env key:_node_name_|_node_ip_
[2023-07-18 07:49:54.704574]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:328              load env, key:_node_name_       value:cn-beijing.192.168.254.132
[2023-07-18 07:49:54.704576]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:328              load env, key:_node_ip_ value:192.168.254.132
[2023-07-18 07:49:54.704583]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:385              purage container mode:true
[2023-07-18 07:49:54.704601]    [info]  [000011]        /src/core/logtail.cpp:158               change working dir:/usr/local/ilogtail/1.7.0    fail, reason:No such file or directory
[2023-07-18 07:49:54.704613]    [info]  [000011]        /src/core/logtail.cpp:160               change working dir:/usr/local/ilogtail/ result:0
[2023-07-18 07:49:54.704680]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:1224             set logtail sys conf dir:/usr/local/ilogtail/./ user local config path:/usr/local/ilogtail/./user_local_config.json     user local config dir path:/usr/local/ilogtail/./user_config.d/ user local yaml config dir path:/usr/local/ilogtail/./user_yaml_config.d/   user remote yaml config dir path:/usr/local/ilogtail/./remote_yaml_config.d/
[2023-07-18 07:49:54.704702]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:288              load logtail config file, path:ilogtail_config.json
[2023-07-18 07:49:54.704708]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:289              load logtail config file, detail:{}

[2023-07-18 07:49:54.704755]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:601              logtail checkpoint path:/usr/local/ilogtail/checkpoint/logtail_check_point
[2023-07-18 07:49:54.704762]    [info]  [000011]        /src/core/common/JsonUtil.cpp:138               load config from env:docker_file_cache_path     value:checkpoint/docker_path_config.json
[2023-07-18 07:49:54.704764]    [info]  [000011]        /src/core/common/JsonUtil.cpp:171               load parameter from env:docker_file_cache_path  value:checkpoint/docker_path_config.json
[2023-07-18 07:49:54.704786]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:346              set cpu_usage_limit from env, value:1
[2023-07-18 07:49:54.704794]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:346              set mem_usage_limit from env, value:512
[2023-07-18 07:49:54.704797]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:1147             default cache size, polling total max stat:250000       per dir max stat:25000  per config max stat:25000       cache config max size:250000    modify max:25000        watch dir count max:25000       max open files limit:32765      max reader open files limit:25000
[2023-07-18 07:49:54.704799]    [info]  [000011]        /src/core/app_config/AppConfigBase.cpp:1151             send byte per second limit:26214400     batch send interval:3   batch send size:262144
[2023-07-18 07:49:54.704844]    [info]  [000011]        /src/core/common/TimeUtil.cpp:234               get system boot time from /proc/uptime:1659519526
[2023-07-18 07:49:54.704883]    [info]  [000011]        /src/core/config_manager/ConfigManagerBase.cpp:2168             invalid aliuid conf dir:/usr/local/ilogtail/./users     error:No such file or directory
[2023-07-18 07:49:54.704916]    [info]  [000011]        /src/core/config_manager/ConfigManagerBase.cpp:2175             recreate aliuid conf dir success:/usr/local/ilogtail/./users
[2023-07-18 07:49:54.715753]    [info]  [000011]        /src/core/config_manager/ConfigManagerBase.cpp:2462             user yaml config file loaded:/usr/local/ilogtail/./user_yaml_config.d/default_pod_stdout.yaml
[2023-07-18 07:49:54.715766]    [info]  [000011]        /src/core/config_manager/ConfigManagerBase.cpp:2471             user yaml config removed or added, last:0       now:1
[2023-07-18 07:49:54.715797]    [info]  [000011]        /src/core/config_manager/ConfigManagerBase.cpp:2043             load user defined id from env:default
[2023-07-18 07:49:54.715984]    [info]  [000011]        /src/core/config_manager/ConfigYamlToJson.cpp:255               Trans yaml to json:success      config_name:/usr/local/ilogtail/./user_yaml_config.d/default_pod_stdout.yaml    is_file_mode:false      input_plugin_type:service_docker_stdout has_accelerate_processor:false  accelerate_processor_plugin_type:  log_split_processor:     logtype:plugin  user_local_json_config:{
        "metrics" : 
        {
                "config#/usr/local/ilogtail/./user_yaml_config.d/default_pod_stdout.yaml" : 
                {
                        "enable" : true,
                        "log_type" : "plugin",
                        "plugin" : 
                        {
                                "flushers" : 
                                [
                                        {
                                                "detail" : 
                                                {
                                                        "Convert" : 
                                                        {
                                                                "TagFieldsRename" : 
                                                                {
                                                                        "host.name" : "my.host.name"
                                                                }
                                                        },
                                                        "DynamicLabels" : 
                                                        [
                                                                "tag.host.name"
                                                        ],
                                                        "URL" : "http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push"
                                                },
                                                "type" : "flusher_loki"
                                        }
                                ],
                                "inputs" : 
                                [
                                        {
                                                "detail" : 
                                                {
                                                        "BeginLineCheckLength" : 10,
                                                        "BeginLineRegex" : "\\d+-\\d+-\\d+.*",
                                                        "Stderr" : true,
                                                        "Stdout" : true
                                                },
                                                "type" : "service_docker_stdout"
                                        }
                                ],
                                "processors" : 
                                [
                                        {
                                                "detail" : 
                                                {
                                                        "Fields" : 
                                                        {
                                                                "cluster" : "aliyun-public-test"
                                                        },
                                                        "IgnoreIfExist" : false
                                                },
                                                "type" : "processor_add_fields"
                                        }
                                ],
                                "version" : "v1"
                        }
                }
        }
}

[2023-07-18 07:49:54.716229]    [info]  [000011]        /src/core/config_manager/ConfigManagerBase.cpp:1099             load user_yaml_config.d config:true     file:/usr/local/ilogtail/./user_yaml_config.d/default_pod_stdout.yaml   now config count:1
[2023-07-18 07:49:54.716361]    [info]  [000011]        /src/core/monitor/Monitor.cpp:542               machine cpu cores:16
[2023-07-18 07:49:54.716486]    [info]  [000011]        /src/core/sls_control/SLSControl.cpp:72         user agent:ilogtail/1.7.0 (Linux; 3.10.0; x86_64) ip/192.168.254.132 env/K8S-Daemonset
[2023-07-18 07:49:54.716498]    [info]  [000011]        /src/core/sender/Sender.cpp:467         Set sender queue param depend value:10
[2023-07-18 07:49:54.716632]    [info]  [000019]        /src/core/sender/Sender.cpp:1319                SendThread:start
[2023-07-18 07:49:54.716667]    [info]  [000011]        /src/core/plugin/LogtailPlugin.cpp:321          load plugin base, config count:1        docker env config:false dl file:/usr/local/ilogtail/libPluginAdapter.so
[2023-07-18 07:49:54.716753]    [info]  [000011]        /src/core/plugin/LogtailPlugin.cpp:342          check plugin adapter version success, version:300
[2023-07-18 07:49:54.763358]    [info]  [000011]        /src/core/plugin/LogtailPlugin.cpp:453          init plugin base:success
[2023-07-18 07:49:54.898877]    [info]  [000011]        /src/core/plugin/LogtailPlugin.cpp:121          logtail plugin Resume:start
[2023-07-18 07:49:54.898962]    [info]  [000011]        /src/core/plugin/LogtailPlugin.cpp:123          logtail plugin Resume:success
[2023-07-18 07:49:54.898982]    [info]  [000011]        /src/core/common/DynamicLibHelper.cpp:104               load glibc dynamic library:begin
[2023-07-18 07:49:54.899001]    [info]  [000011]        /src/core/common/DynamicLibHelper.cpp:118               load glibc dynamic library:success
[2023-07-18 07:49:54.899055]    [info]  [000011]        /src/core/checkpoint/CheckPointManager.cpp:150          load checkpoint, version:200    file check point:0      dir check point:0
[2023-07-18 07:49:54.899087]    [info]  [000011]        /src/core/profiler/LogIntegrity.cpp:507         no integrity file to load:/usr/local/ilogtail/logtail_integrity_snapshot.json
[2023-07-18 07:49:54.899103]    [info]  [000011]        /src/core/profiler/LogLineCount.cpp:265         no line count file to load:/usr/local/ilogtail/logtail_line_count_snapshot.json
[2023-07-18 07:49:54.899251]    [info]  [000011]        /src/core/logtail.cpp:284               Logtail started, appInfo:{
        "UUID" : "AF2AE122-253F-11EE-B76D-00163E14EF71",
        "build_date" : "20230702",
        "compiler" : "GCC 4.8.5",
        "git_hash" : "2c26cc6e521e22d22cff29eec1280be13ffa2682",
        "hostname" : "generation-dev_test-worker05-254_132",
        "instance_id" : "AF2ADCE0-253F-11EE-8850-00163E14EF71_192.168.254.132_1689666594",
        "ip" : "192.168.254.132",
        "logtail_version" : "1.7.0 Community Edition",
        "os" : "Linux; 3.10.0-1160.71.1.el7.x86_64; #1 SMP Tue Jun 28 15:37:28 UTC 2022; x86_64",
        "update_time" : "2023-07-18 07:49:54"
}

[2023-07-18 07:49:54.899311]    [info]  [000011]        /src/core/controller/EventDispatcherBase.cpp:607                start to verify existed checkpoints, total checkpoint count:0
[2023-07-18 07:49:54.899318]    [info]  [000011]        /src/core/controller/EventDispatcherBase.cpp:621                checkpoint verification ends, generated event count:0   checkpoint deletion count:0
[2023-07-18 07:49:54.899347]    [info]  [000011]        /src/core/polling/PollingModify.cpp:51          PollingModify:start
[2023-07-18 07:49:54.899369]    [info]  [000041]        /src/core/polling/PollingModify.cpp:238         PollingModify::Polling:start
[2023-07-18 07:49:54.899372]    [info]  [000011]        /src/core/polling/PollingDirFile.cpp:71         PollingDirFile:start
[2023-07-18 07:49:54.899443]    [info]  [000011]        /src/core/event_handler/LogInput.cpp:88         LogInput:start
[2023-07-18 07:49:54.899521]    [info]  [000044]        /src/core/processor/LogProcess.cpp:208          local timezone offset second:0

logtail_plugin.LOG:

2023-07-18 07:53:54 [INF] [input_docker_stdout.go:386] [ClearUselessCheckpoint] [config#/usr/local/ilogtail/./user_yaml_config.d/default_pod_stdout.yaml,]      delete checkpoint, id:b17a30ce73af3728ab1cab0819907f1e3701fead26fef258a5deec0ca6711e11
2023-07-18 07:53:54 [INF] [input_docker_stdout.go:325] [FlushAll] [config#/usr/local/ilogtail/./user_yaml_config.d/default_pod_stdout.yaml,]    update match list, firstStart: false, new: 0, delete: 1
2023-07-18 07:53:54 [INF] [input_docker_stdout.go:345] [FlushAll] [config#/usr/local/ilogtail/./user_yaml_config.d/default_pod_stdout.yaml,]    docker stdout:deleted   id:8490ece7b940 name:sop-manager
2023-07-18 07:53:55 [INF] [logger.go:374] [func3] stdout:level=error component=client host=loki-gateway.monitoring.svc.cluster.local msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): 1:6: parse error: unexpected character inside braces: '.'"

2023-07-18 07:53:58 [INF] [logger.go:374] [func3] stdout:level=error component=client host=loki-gateway.monitoring.svc.cluster.local msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): 1:6: parse error: unexpected character inside braces: '.'"

2023-07-18 07:54:01 [INF] [logger.go:374] [func3] stdout:level=error component=client host=loki-gateway.monitoring.svc.cluster.local msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): 1:6: parse error: unexpected character inside braces: '.'"

2023-07-18 07:54:04 [INF] [logger.go:374] [func3] stdout:level=error component=client host=loki-gateway.monitoring.svc.cluster.local msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): 1:6: parse error: unexpected character inside braces: '.'"

2023-07-18 07:54:07 [INF] [logger.go:374] [func3] stdout:level=error component=client host=loki-gateway.monitoring.svc.cluster.local msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): 1:6: parse error: unexpected character inside braces: '.'"

2023-07-18 07:54:10 [INF] [logger.go:374] [func3] stdout:level=error component=client host=loki-gateway.monitoring.svc.cluster.local msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): 1:6: parse error: unexpected character inside braces: '.'"

2023-07-18 07:54:13 [INF] [logger.go:374] [func3] stdout:level=error component=client host=loki-gateway.monitoring.svc.cluster.local msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): 1:6: parse error: unexpected character inside braces: '.'"

The text was updated successfully, but these errors were encountered:

Abingcbc · 2023-07-19T07:02:52Z

@ixiaoyi93 这个报错是因为 Loki 不支持 label 中包含 . 。https://grafana.com/docs/loki/latest/fundamentals/labels/

ilogtail文档上举的例子有点问题，你可以来修复一下，或者贡献一个更好的例子。

Abingcbc · 2023-07-19T07:23:16Z

@ixiaoyi93 文档中这个例子应当为

flushers:
  - Type: flusher_loki
    URL: http://<loki 服务的地址与端口>/loki/api/v1/push
    DynamicLabels:
      - tag.host_name
    Convert:
      TagFieldsRename:
        host.name: host_name

DynamicLabels 应该为重命名后的 label。然后，通过 {host_name="master"} 可以在 loki 中进行查询。
可以来提个 PR 更新一下文档 :)

ixiaoyi93 · 2023-07-20T02:31:55Z

好的，我先测试一下。

ixiaoyi93 · 2023-07-20T11:11:38Z

@Abingcbc 现在可以重命令了，我想在请教一个问题。采用 docker 类型的日志采集，他提供类似于 _container_name_ 的标签。如果是这样的话，我理解如果这样直接传递给 loki 的话，默认就有类似的标签了吧。
但是我看文档说明，loki 输出的格式应该是重写了这些标签，使其放入到了 tags 的 json 串中。如下：

但是我感觉这种日志结构反而增加了 loki 的展示。这块该如何优化？因为我本身的日志内容只是 content 字段。我在想如果loki插件如果不修改日志的格式，是不是就可以和 promtail 采集过来的日志展示一样。

还请大佬解答。

Abingcbc · 2023-07-21T02:38:45Z

@Abingcbc 现在可以重命令了，我想在请教一个问题。采用 docker 类型的日志采集，他提供类似于 _container_name_ 的标签。如果是这样的话，我理解如果这样直接传递给 loki 的话，默认就有类似的标签了吧。但是我看文档说明，loki 输出的格式应该是重写了这些标签，使其放入到了 tags 的 json 串中。如下：

但是我感觉这种日志结构反而增加了 loki 的展示。这块该如何优化？因为我本身的日志内容只是 content 字段。我在想如果loki插件如果不修改日志的格式，是不是就可以和 promtail 采集过来的日志展示一样。

还请大佬解答。

目前 ilogtail 还不支持只发送 content 字段。cc @yyuuttaaoo @henryzhx8

ixiaoyi93 · 2023-07-24T01:42:39Z

@henryzhx8 @yyuuttaaoo @Abingcbc 大佬们，安排一波？

Abingcbc · 2023-08-21T07:46:51Z

@ixiaoyi93 目前可以支持的是通过 convert 把 tag 里的字段删掉

flushers:
  - Type: flusher_loki
    URL: http://localhost:3100/loki/api/v1/push
    DynamicLabels:
      - content.test
    Convert:
      TagFieldsRename:
        host.name: ""
        host.ip: ""
        log.file.path: ""

可以达到下面这样的效果

ixiaoyi93 added the bug Something isn't working label Jul 18, 2023

yyuuttaaoo assigned Abingcbc Jul 19, 2023

lujiajing1126 mentioned this issue Dec 5, 2023

Support contents only for loki flusher #1256

Merged

yyuuttaaoo closed this as completed in #1256 Dec 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]: ilogtail 将日志输入到 Loki 异常 #1020

[BUG]: ilogtail 将日志输入到 Loki 异常 #1020

ixiaoyi93 commented Jul 18, 2023

Abingcbc commented Jul 19, 2023 •

edited

Abingcbc commented Jul 19, 2023

ixiaoyi93 commented Jul 20, 2023

ixiaoyi93 commented Jul 20, 2023 •

edited

Abingcbc commented Jul 21, 2023

ixiaoyi93 commented Jul 24, 2023

Abingcbc commented Aug 21, 2023

[BUG]: ilogtail 将日志输入到 Loki 异常 #1020

[BUG]: ilogtail 将日志输入到 Loki 异常 #1020

Comments

ixiaoyi93 commented Jul 18, 2023

Abingcbc commented Jul 19, 2023 • edited

Abingcbc commented Jul 19, 2023

ixiaoyi93 commented Jul 20, 2023

ixiaoyi93 commented Jul 20, 2023 • edited

Abingcbc commented Jul 21, 2023

ixiaoyi93 commented Jul 24, 2023

Abingcbc commented Aug 21, 2023

Abingcbc commented Jul 19, 2023 •

edited

ixiaoyi93 commented Jul 20, 2023 •

edited