Fluentd always "Created new chunk" for every single event with armv7l environment #3088
Assignees
Labels
Comments
|
I don't have armv7l environment for now. |
|
I tested this problem with Arm chroot environment using qemu+binfmt and I could reproduced it. I can't find any online services easy to use that provide the armv7l environment. |
Using
|
|
I built Raspberry Pi OS armv7l qemu chroot environment like following steps, and it was also reproducible. build Raspberry Pi OS armv7l on Ubuntu 18.04 x86_64 with AWS EC2
binfmt-support qemu-user-static
mount Raspberry Pi OS
chroot to Raspberry Pi OS
test with Raspberry Pi OS armv7l
|
|
I'm confirming this issue on Jetson TX2 and its docker runtime. |
test with Ubuntu 18.04 on Jetson TX2Pull armhf imageCreate configuration fileRun FluentdResultFrom outside armhf container: Inside running Fluentd container: I could reproduce this issue with armhf container. |
|
Using fluent/fluentd:v1.11.1-debian-arm64-1.0 image does not create flood of buffer files:
|
|
The root cause of this issue is: timekey should be bigger than 2**30-1: But, Ruby on Windows and 32bit Linux environment does not handle such big integer's object_id as stable ids: That is why this issue is caused. |
cosmo0920
added a commit
that referenced
this issue
Jul 31, 2020
Object#object_id should return values the following rule:
```
On Windows:
irb(main):140:0> RUBY_VERSION
=> "2.7.1"
irb(main):141:0> a = 2**30 - 1
irb(main):142:0> a.object_id
=> 2147483647
irb(main):143:0> a = 2**30 - 1
irb(main):144:0> a.object_id
=> 2147483647
irb(main):145:0> a = 2**30
irb(main):146:0> a.object_id
=> 640
irb(main):147:0> a = 2**30
irb(main):148:0> a.object_id
=> 660
```
For Windows, less than equal `2^30 - 1` should be stable and be able to use with #hash.
---
```
On GNU/Linux x86_64:
irb(main):001:0> RUBY_VERSION
=> "2.7.0"
irb(main):002:0> a = 2**30 - 1
irb(main):003:0> a.object_id
=> 2147483647
irb(main):004:0> a = 2**30 - 1
irb(main):005:0> a.object_id
=> 2147483647
irb(main):006:0> a = 2**30
irb(main):007:0> a.object_id
=> 2147483649
irb(main):008:0> a = 2**30
irb(main):009:0> a.object_id
=> 2147483649
irb(main):010:0> a = 2**62 - 1
irb(main):011:0> a.object_id
=> 9223372036854775807
irb(main):012:0> a = 2**62 - 1
irb(main):013:0> a.object_id
=> 9223372036854775807
irb(main):014:0> a = 2**62
irb(main):015:0> a.object_id
=> 180
irb(main):016:0> a = 2**62
irb(main):017:0> a.object_id
=> 200
```
For GNU/Linux x86_64, less than equal `2^62 - 1` should be stable and be able to use with #hash.
---
```
On GNU/Linux aarch64:
irb(main):001:0> RUBY_VERSION
=> "2.7.1"
irb(main):002:0> a = 2**30 - 1
irb(main):003:0> a.object_id
=> 2147483647
irb(main):004:0> a = 2**30 - 1
irb(main):005:0> a.object_id
=> 2147483647
irb(main):006:0> a = 2**30
irb(main):007:0> a.object_id
=> 2147483649
irb(main):008:0> a = 2**30
irb(main):009:0> a.object_id
=> 2147483649
irb(main):010:0> a = 2**62 -1
irb(main):011:0> a.object_id
=> 9223372036854775807
irb(main):012:0> a = 2**62 -1
irb(main):013:0> a.object_id
=> 9223372036854775807
irb(main):014:0> a = 2**62
irb(main):015:0> a.object_id
=> 180
irb(main):016:0> a = 2**62
irb(main):017:0> a.object_id
=> 200
```
For GNU/Linux aarch64, less than equal `2^62 - 1` should be stable and be able to use with #hash.
---
```
On GNU/Linux armv7l
irb(main):001:0> RUBY_VERSION
=> "2.6.6"
irb(main):002:0> a = 2**30 -1
=> 1073741823
irb(main):003:0> a.object_id
=> 2147483647
irb(main):004:0> a = 2**30 -1
=> 1073741823
irb(main):005:0> a.object_id
=> 2147483647
irb(main):006:0> a = 2**30
=> 1073741824
irb(main):007:0> a.object_id
=> -209995496
irb(main):008:0> a = 2**30
=> 1073741824
irb(main):009:0> a.object_id
=> -210001856
irb(main):010:0> a = 2**62 -1
=> 4611686018427387903
irb(main):011:0> a.object_id
=> -209951576
irb(main):012:0> a = 2**62 -1
=> 4611686018427387903
irb(main):013:0> a.object_id
=> -209925764
irb(main):014:0> a = 2**62
=> 4611686018427387904
irb(main):015:0> a.object_id
=> -209907800
irb(main):016:0> a = 2**62
=> 4611686018427387904
irb(main):017:0> a.object_id
=> -209891900
```
For GNU/Linux armv7l than equal `2^30 - 1` should be stable and be able to use with #hash.
Nowadays, unixtime should be bigger than `2^30 -1`:
irb> Time.parse("2020/07/31 18:30:00+09:00").to_i > 2**30 - 1
=> true
So, we should check to #hash method optimization validity with the following method:
```ruby
def self.enable_optimize?
a1 = 2**30 - 1
a2 = 2**30 - 1
b1 = 2**62 - 1
b2 = 2**62 - 1
(a1.object_id == a2.object_id) && (b1.object_id == b2.object_id)
end
```
Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
repeatedly
added a commit
that referenced
this issue
Aug 4, 2020
…with-hash-method buffer: Do more precise timekey optimization handling. Fix #3088
|
Many thanks! |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I'm trying to use fluentd on Raspberry Pi 3B+ with Arch Linux ARM (armv7l).
With armv7l environment, fluentd always creates new chunk for every single event.
This issue does not happen in aarch64 environment.
To Reproduce
Install fluentd to armv7l environment, and send some event messages.
Fluentd creates many small chunk files, just 1 log line each.
Expected behavior
Fluentd should not create a small chunk file for an each single event.
Your Environment
Your Configuration
Your Error Log
$ for i in {1..100}; do echo '{"status":"OK"}' | fluent-cat test; doneproduces following outputs.Additional context
I found similar issue #2713 and commented out
hashmethod like following:And the problem was gone. (only one chunk file was created as expected)
I also tested fluentd in the aarch64 environments (AWS EC2 m6g.medium instance and qemu-arm emulation with version 5.0.0) and the problem did not happen.
hashmethod was introduced at v1.7.0 so I tested fluentd version 1.6.0 and that is not also affected.The text was updated successfully, but these errors were encountered: