Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When try to append to file more often then default timekey, exception happens #68

Open
bobykus31 opened this issue Nov 19, 2018 · 0 comments

Comments

@bobykus31
Copy link

Here is my config

<match iotop.0 >
    @type copy
    deep_copy true

    <store>
        @type webhdfs
        namenode          master1.hdoo.com:50070 
        standby_namenode  master2.hdoo.com:50070
        path  "/raw/iostats/%Y%m%d/%H/#{Socket.gethostname}.log"
        compress gzip  # or 'bzip2', 'snappy', 'lzo_command'
        retry_known_errors yes
        retry_interval 65
        open_timeout 180
        read_timeout 180
        append yes
        username fluent
        ignore_start_check_error true
        <buffer>
           @type   file
           flush_interval 1m
           timekey  10m
           path  /var/spool/td-agent/fluentd_buffer_fwebhdfs/
        </buffer>
        <format>
           @type csv
           fields UNIXTIME,USER,DISKREAD,DISKWRITE
           force_quotes false 
        </format>
        <secondary>
          @type secondary_file
          directory /var/spool/td-agent/webhdfs_failed/
        </secondary>
    </store>


</match>

The goal is to sync data down to hadoop every 10 min even if file is %H based. Every time the flush process start, the following issue appears

2018-11-19 13:40:31 +0000 [warn]: #19 failed to flush the buffer. retry_time=0 next_retry_seconds=2018-11-19 13:40:32 +0000 chunk="57b048883d347b0ca22650dac40e1961" error_class=WebHDFS::IOError error="{\"RemoteException\":{\"exception\":\"AlreadyBeingCreatedException\",\"javaClassName\":\"org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException\",\"message\":\"Failed to create file [/raw/iostats/20181119/13/fluentd1.log.gz] for [DFSClient_NONMAPREDUCE_1591351286_116] for client [10.64.11.19], because this file is already being created by [DFSClient_NONMAPREDUCE_1898391037_115] on [10.64.11.19]\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:3276)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:3056)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3341)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3305)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:621)\\n\\tat org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.append(AuthorizationProviderProxyClientProtocol.java:129)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:433)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)\\n\"}}"
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:401:in `request'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:270:in `operate_requests'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:84:in `append'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.3/lib/fluent/plugin/out_webhdfs.rb:272:in `send_data'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.3/lib/fluent/plugin/out_webhdfs.rb:389:in `block in write'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.3/lib/fluent/plugin/out_webhdfs.rb:335:in `compress_context'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.3/lib/fluent/plugin/out_webhdfs.rb:388:in `write'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1111:in `try_flush'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1394:in `flush_thread_run'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:445:in `block (2 levels) in start'
  2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Then next second retry with success

2018-11-19 13:40:32 +0000 [warn]: #19 retry succeeded. chunk_id="57b048883d347b0ca22650dac40e1961"

Is it expected behavior?
We use Cloudera Manager 5.13.0-1.cm5130.p0.55~wheezy-cm5 as hadoop implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant