Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

一直报错，看起来是 fluentd-1.1.0 的 super 类没有 format 方法 #150

Closed
buhe opened this issue Oct 8, 2018 · 6 comments
Closed

一直报错，看起来是 fluentd-1.1.0 的 super 类没有 format 方法 #150

buhe opened this issue Oct 8, 2018 · 6 comments

Comments

@buhe
Copy link

buhe commented Oct 8, 2018

最新版本，执行后

docker run -e FLUENTD_OUTPUT=aliyun_sls -e ALIYUNSLS_PROJECT=xxx -e ALIYUNSLS_NEED_CREATE_LOGSTORE=true -e ALIYUNSLS_REGION_ENDPOINT=cn-hangzhou.log.aliyuncs.com -e ALIYUNSLS_ACCESS_KEY_ID=xxx -e ALIYUNSLS_ACCESS_KEY_SECRET=xxx -e ALIYUNSLS_CREATE_LOGSTORE_TTL=7 --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /:/host registry.cn-hangzhou.aliyuncs.com/acs-sample/log-pilot:0.9.5-fluentd

一直报错，看起来是 fluentd-1.1.0 的 super 类没有 format 方法

2018-10-08 12:56:19 +0000 [error]: #0 unexpected error error_class=NotImplementedError error="BUG: output plugins MUST implement this method"
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:124:in `format'
  2018-10-08 12:56:19 +0000 [error]: #0 /etc/fluentd/plugins/out_aliyun_sls.rb:39:in `format'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/compat/output.rb:372:in `block in handle_stream_simple'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/event.rb:193:in `block in each'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/event.rb:192:in `each'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/event.rb:192:in `each'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/compat/output.rb:372:in `map'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/compat/output.rb:372:in `handle_stream_simple'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:841:in `execute_chunking'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:771:in `emit_buffered'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:159:in `emit_events'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:96:in `emit_stream'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:385:in `receive_lines'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:503:in `wrap_receive_lines'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:707:in `block in on_notify'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:751:in `with_io'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:685:in `on_notify'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:534:in `on_notify'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:507:in `attach'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:264:in `setup_watcher'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:292:in `block in start_watchers'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:278:in `each'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:278:in `start_watchers'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:258:in `refresh_watchers'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_tail.rb:193:in `start'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:165:in `block in start'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:153:in `each'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:153:in `block in lifecycle'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:140:in `each'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:140:in `lifecycle'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:164:in `start'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/engine.rb:274:in `start'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/engine.rb:219:in `run'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:774:in `run_engine'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:523:in `block in run_worker'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:699:in `main_process'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:518:in `run_worker'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/bin/fluentd:8:in `<top (required)>'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/bin/fluentd:23:in `load'
  2018-10-08 12:56:19 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>'
@buhe
Copy link
Author

buhe commented Oct 8, 2018

1.1.0 代码如下

def format(tag, time, record)
        # standard msgpack_event_stream chunk will be used if this method is not implemented in plugin subclass
        raise NotImplementedError, "BUG: output plugins MUST implement this method"
      end

log-pilot

def format(tag, time, record)
      if record["@target"]
        [tag, time, record].to_msgpack
      else
        super
      end
    end

那么走到下面的分支会 crash 掉？

@chenquanzhao
Copy link
Member

@buhe 生成的fluentd配置文件里是否有target配置？

@buhe
Copy link
Author

buhe commented Oct 16, 2018 •
edited
Loading 

<source>
    @type tail
    tag "docker.02dc4faee4f7cc69620a10b75c618f0e5b6295fcb387281b4a759729c9717c3f.edgeapp"
    path "/host/var/lib/docker/containers/02dc4faee4f7cc69620a10b75c618f0e5b6295fcb387281b4a759729c9717c3f/02dc4faee4f7cc69620a10b75c618f0e5b6295fcb387281b4a759729c9717c3f-json.log"
    read_from_head true
    pos_file "/pilot/pos/02dc4faee4f7cc69620a10b75c618f0e5b6295fcb387281b4a759729c9717c3f.edgeapp.pos"
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      keep_time_key true
      time_type string
    </parse>
  </source>
  <filter docker.02dc4faee4f7cc69620a10b75c618f0e5b6295fcb387281b4a759729c9717c3f.edgeapp>
    @type record_transformer
    enable_ruby true
    <record>
      host 6922974b13e3
      index edgeapp
      topic edgeapp
      _target edgeapp
      docker_app app-edge-lambda
      docker_container app-edge-lambda_lambda.2.hii0ussunsms6ybjbv01n0org
      docker_service app-edge-lambda_lambda
    </record>
  </filter>
  <match docker.**>
    @type aliyun_sls
    project "white-logs"
    region_endpoint "cn-hangzhou.log.aliyuncs.com"
    access_key_id "LTAIdIgc3ScCo8Xy"
    access_key_secret "efeIceFOCANGpfFAypte7UnGe1VkY8"
    ssl_verify false
    need_create_logstore true
    create_logstore_ttl 7
    create_logstore_shard_count 2
    <buffer>
      flush_mode interval
      retry_type exponential_backoff
    </buffer>
  </match>
  <system>
  </system>
</ROOT>

只剩下一个之后的配置，查看 white-logs 没有生成任何 log-store ，报错信息依旧。

启动日志：

2018-10-16 11:49:52 +0000 [info]: starting fluentd-1.1.0 pid=1339 ruby="2.4.4"
2018-10-16 11:49:52 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/etc/fluentd/fluentd.conf", "-p", "/etc/fluentd/plugins", "--under-supervisor"]
2018-10-16 11:49:52 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.2'
2018-10-16 11:49:52 +0000 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0'
2018-10-16 11:49:52 +0000 [info]: gem 'fluent-plugin-kafka' version '0.7.4'
2018-10-16 11:49:52 +0000 [info]: gem 'fluent-plugin-remote_syslog' version '1.0.0'
2018-10-16 11:49:52 +0000 [info]: gem 'fluentd' version '1.1.0'
2018-10-16 11:49:52 +0000 [info]: adding filter pattern="docker.02dc4faee4f7cc69620a10b75c618f0e5b6295fcb387281b4a759729c9717c3f.edgeapp" type="record_transformer"
2018-10-16 11:49:52 +0000 [info]: adding match pattern="docker.**" type="aliyun_sls"
2018-10-16 11:49:52 +0000 [info]: adding source type="tail"
2018-10-16 11:49:52 +0000 [info]: #0 Oj is not installed, and failing back to Yajl for json parser
2018-10-16 11:49:52 +0000 [info]: #0 Oj is not installed, and failing back to Yajl for json parser
2018-10-16 11:49:52 +0000 [info]: #0 starting fluentd worker pid=1343 ppid=1339 worker=0

没有看到相关的插件

启动命令：

docker run -e PILOT_TYPE=fluentd -e FLUENTD_OUTPUT=aliyun_sls -e ALIYUNSLS_PROJECT=white-logs -e ALIYUNSLS_NEED_CREATE_LOGSTORE=true -e ALIYUNSLS_REGION_ENDPOINT=cn-hangzhou.log.aliyuncs.com -e ALIYUNSLS_ACCESS_KEY_ID=xx -e ALIYUNSLS_ACCESS_KEY_SECRET=xx -e ALIYUNSLS_CREATE_LOGSTORE_TTL=7 --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /:/host:ro registry.cn-hangzhou.aliyuncs.com/acs-sample/log-pilot:0.9.5-fluentd

stack

version: "3.3"
services:
  lambda:
    image: "registry.cn-hangzhou.aliyuncs.com/white/xxx:latest"
    user: "node"
    ports:
      - 8545:3000
    environment:
      - ENV=edge
    deploy:
      mode: replicated
      replicas: 2
#      resources:
#        limits:
#          cpus: '0.30'
#          memory: 500M
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 20
    labels:
      aliyun.logs.edgeapp: stdout

@chenquanzhao 麻烦抽空帮忙看看

@buhe
Copy link
Author

buhe commented Oct 16, 2018 •
edited
Loading

用 file 模式，也没有日志输出到目录，什么都不配置可以在控制台看到内容

docker run --rm -it --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /:/host:ro -e PILOT_TYPE=fluentd -e LOGGING_OUTPUT=file -e FILE_PATH=/root/logs -e FLUENTD_BUFFER_TYPE=file registry.cn-hangzhou.aliyuncs.com/acs-sample/log-pilot:0.9.5-fluentd

@chenquanzhao
Copy link
Member

@buhe 这个我们处理下，这两天更新一个新版本，感谢反馈！

chenquanzhao pushed a commit that referenced this issue Oct 19, 2018
fix issue #150 and update fluentd version to 1.2.6
74431b9
@chenquanzhao
Copy link
Member

@buhe 可以升级到新版本已解决该问题，谢谢！
https://github.com/AliyunContainerService/log-pilot/releases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@buhe @chenquanzhao