We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Here is my config
<match iotop.0 > @type copy deep_copy true <store> @type webhdfs namenode master1.hdoo.com:50070 standby_namenode master2.hdoo.com:50070 path "/raw/iostats/%Y%m%d/%H/#{Socket.gethostname}.log" compress gzip # or 'bzip2', 'snappy', 'lzo_command' retry_known_errors yes retry_interval 65 open_timeout 180 read_timeout 180 append yes username fluent ignore_start_check_error true <buffer> @type file flush_interval 1m timekey 10m path /var/spool/td-agent/fluentd_buffer_fwebhdfs/ </buffer> <format> @type csv fields UNIXTIME,USER,DISKREAD,DISKWRITE force_quotes false </format> <secondary> @type secondary_file directory /var/spool/td-agent/webhdfs_failed/ </secondary> </store> </match>
The goal is to sync data down to hadoop every 10 min even if file is %H based. Every time the flush process start, the following issue appears
2018-11-19 13:40:31 +0000 [warn]: #19 failed to flush the buffer. retry_time=0 next_retry_seconds=2018-11-19 13:40:32 +0000 chunk="57b048883d347b0ca22650dac40e1961" error_class=WebHDFS::IOError error="{\"RemoteException\":{\"exception\":\"AlreadyBeingCreatedException\",\"javaClassName\":\"org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException\",\"message\":\"Failed to create file [/raw/iostats/20181119/13/fluentd1.log.gz] for [DFSClient_NONMAPREDUCE_1591351286_116] for client [10.64.11.19], because this file is already being created by [DFSClient_NONMAPREDUCE_1898391037_115] on [10.64.11.19]\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:3276)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:3056)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3341)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3305)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:621)\\n\\tat org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.append(AuthorizationProviderProxyClientProtocol.java:129)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:433)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)\\n\"}}" 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:401:in `request' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:270:in `operate_requests' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:84:in `append' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.3/lib/fluent/plugin/out_webhdfs.rb:272:in `send_data' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.3/lib/fluent/plugin/out_webhdfs.rb:389:in `block in write' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.3/lib/fluent/plugin/out_webhdfs.rb:335:in `compress_context' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.2.3/lib/fluent/plugin/out_webhdfs.rb:388:in `write' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1111:in `try_flush' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1394:in `flush_thread_run' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:445:in `block (2 levels) in start' 2018-11-19 13:40:31 +0000 [warn]: #19 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Then next second retry with success
2018-11-19 13:40:32 +0000 [warn]: #19 retry succeeded. chunk_id="57b048883d347b0ca22650dac40e1961"
Is it expected behavior? We use Cloudera Manager 5.13.0-1.cm5130.p0.55~wheezy-cm5 as hadoop implementation.
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Here is my config
The goal is to sync data down to hadoop every 10 min even if file is %H based. Every time the flush process start, the following issue appears
Then next second retry with success
Is it expected behavior?
We use Cloudera Manager 5.13.0-1.cm5130.p0.55~wheezy-cm5 as hadoop implementation.
The text was updated successfully, but these errors were encountered: