in_mem: add MemAvailable, Buffers and Cached from /proc/meminfo #3092

fopina · 2021-02-19T00:00:03Z

in_mem plugin modified to include MemAvailable (from /proc/meminfo)

the sysinfo syscall exposes si_meminfo from the kernel but MemAvailable is calculated by si_mem_available and not exposed in any syscall, only in /proc/meminfo (kernel meminfo src

The free command, part of procps, parses /proc/meminfo - free calls meminfo function that parses it

To avoid adding libprocps as dependency, I chose to copy&paste busybox implementation - also parses /proc/meminfo but the piece of code that does the parsing is small and easy to port.

Note I am currently using this as standalone plugin without any issues, so tested and available in case anyone wants to use it (before merge)

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

[N/A] Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#467

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Filipe Pina <fopina@gmail.com>

fopina · 2021-02-19T00:58:11Z

debug log for fluent-bit -i mem -o stdout -vvv:

Fluent Bit v1.8.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/02/19 00:56:24] [ info] Configuration:
[2021/02/19 00:56:24] [ info]  flush time     | 5.000000 seconds
[2021/02/19 00:56:24] [ info]  grace          | 5 seconds
[2021/02/19 00:56:24] [ info]  daemon         | 0
[2021/02/19 00:56:24] [ info] ___________
[2021/02/19 00:56:24] [ info]  inputs:
[2021/02/19 00:56:24] [ info]      mem
[2021/02/19 00:56:24] [ info] ___________
[2021/02/19 00:56:24] [ info]  filters:
[2021/02/19 00:56:24] [ info] ___________
[2021/02/19 00:56:24] [ info]  outputs:
[2021/02/19 00:56:24] [ info]      stdout.0
[2021/02/19 00:56:24] [ info] ___________
[2021/02/19 00:56:24] [ info]  collectors:
[2021/02/19 00:56:24] [ info] [engine] started (pid=3923)
[2021/02/19 00:56:24] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/02/19 00:56:24] [debug] [storage] [cio stream] new stream registered: mem.0
[2021/02/19 00:56:24] [ info] [storage] version=1.1.0, initializing...
[2021/02/19 00:56:24] [ info] [storage] in-memory
[2021/02/19 00:56:24] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/02/19 00:56:24] [debug] [stdout:stdout.0] created event channels: read=18 write=19
[2021/02/19 00:56:24] [debug] [router] default match rule mem.0:stdout.0
[2021/02/19 00:56:24] [ info] [sp] stream processor started
[2021/02/19 00:56:25] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:25] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[2021/02/19 00:56:26] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:26] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[2021/02/19 00:56:27] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:27] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[2021/02/19 00:56:28] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:28] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[0] mem.0: [1613696185.540699831, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[1] mem.0: [1613696185.540699831, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[2] mem.0: [1613696185.540699831, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[3] mem.0: [1613696185.540699831, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[2021/02/19 00:56:29] [debug] [task] created task=0x7f701be371e0 id=0 OK
[4] mem.0: [1613696185.540699831, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[5] mem.0: [1613696185.540699831, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[6] mem.0: [1613696185.540699831, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[7] mem.0: [1613696186.542531758, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[8] mem.0: [1613696186.542531758, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[9] mem.0: [1613696186.542531758, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[10] mem.0: [1613696186.542531758, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[11] mem.0: [1613696186.542531758, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[12] mem.0: [1613696186.542531758, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[13] mem.0: [1613696186.542531758, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[14] mem.0: [1613696187.540477776, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[15] mem.0: [1613696187.540477776, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[16] mem.0: [1613696187.540477776, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[17] mem.0: [1613696187.540477776, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[18] mem.0: [1613696187.540477776, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[19] mem.0: [1613696187.540477776, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[20] mem.0: [1613696187.540477776, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[21] mem.0: [1613696188.538160274, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[22] mem.0: [1613696188.538160274, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[23] mem.0: [1613696188.538160274, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[24] mem.0: [1613696188.538160274, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[25] mem.0: [1613696188.538160274, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[26] mem.0: [1613696188.538160274, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[27] mem.0: [1613696188.538160274, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[2021/02/19 00:56:29] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:29] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[2021/02/19 00:56:29] [debug] [out coro] cb_destroy coro_id=0
[2021/02/19 00:56:29] [debug] [task] destroy task=0x7f701be371e0 (task_id=0)
[2021/02/19 00:56:30] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:30] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[2021/02/19 00:56:31] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:31] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[2021/02/19 00:56:32] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:32] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[2021/02/19 00:56:33] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:276] memory total=2037988 kb, used=1823572 kb, free=214416 kb
[2021/02/19 00:56:33] [trace] [input:mem:mem.0 at plugins/in_mem/mem.c:278] swap total=1048572 kb, used=207884 kb, free=840688 kb
[0] mem.0: [1613696189.543626561, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
[1] mem.0: [1613696189.543626561, {"Mem.total"=>2037988, "Mem.used"=>1823572, "Mem.free"=>214416, "Mem.available"=>1232128, "Mem.buffer"=>85228, "Mem.cache"=>1276556}]
...

Valgrind log attached
valgrind.log

github-actions · 2021-03-21T02:12:06Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

fopina · 2021-03-21T02:21:23Z

@edsiper anything missing to make this of interest or should I let it close?
Cheers

nokute78 · 2021-04-12T11:52:34Z

I chose to copy&paste busybox implementation

busybox is GPLv2.
Fluent-bit is Apache License v2.0.

I think they are not compatible.
https://www.apache.org/licenses/GPL-compatibility.html

Despite our best efforts, the FSF has never considered the Apache License to be compatible with GPL version 2, citing the patent termination and indemnification provisions as restrictions not present in the older GPL license. The Apache Software Foundation believes that you should always try to obey the constraints expressed by the copyright holder when redistributing their work.

fopina · 2021-04-12T13:31:24Z

busybox is GPLv2.
Fluent-bit is Apache License v2.0.

thanks @nokute78 that makes sense. I'm bit off opensource licenses and common practices though in this case it's like 20 lines in bbfree.h.
Not really about the code itself, just the idea how to get the data. Does it make a difference to rewrite those lines?

nokute78 · 2021-04-14T10:15:33Z

It is licensed under GPLv2 even if only 20lines.

If we merge such code, the license of Fluent-bit will become GPLv2.
According to the license, user needs to publish the code and some companies may hesitate to use Fluent-bit.

FYI
Recently, Ruby on Rails is affected license issue.
https://www.theregister.com/2021/03/25/ruby_rails_code/

fopina · 2021-04-14T11:56:43Z

ok, I’ll rewrite it. Apart from licensing, does the PR look good to you (and does it make sense)?

nokute78 · 2021-04-14T12:20:59Z

Thank you.
First, could you check commit message rule and modify it like in_mem: XXXX
https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#commit-changes

I remember similar pull request #1617
I commented these point for the pull request.

Adding new option for compatibility
How do we treat old kernel ? MemAvailable is supported from v3.14.

If no option, when user update fluent-bit, new values are appended.
It may be a problem for schemaful database.

Ubuntu 14.04 is supported until 2022 and its kernel version is v3.13. ref
(I don't test Ubuntu 14.04. Perhaps the patch is backporting and it may support MemAvailable)

fopina · 2021-04-14T12:43:41Z

I must have missed that PR but I guess this is still useful as the author gave up on it.

I’ll rephrase the commit message.
About compatibility I agree and that’s why behavior of existing fields wasn’t changed, just Available was introduced.
Yet making it optional (when it’s probably the value that most people expect) will hide it and increase configuration complexity.
I would expect any schemaful database to take the field it wants (which have not changed schema themselves, nor meaning) rather than trying to fit all fields into their fields.
Very much like anyone consuming a JSON API into a SQL database, as long as the field you use don’t change, it’s fine. New fields can be added..
I’ll see if I can find anything on the Ubuntu 14 comment

github-actions · 2021-05-16T02:01:23Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

edsiper · 2021-12-13T00:01:06Z

@nokute78 thoughts on this ?

fopina · 2021-12-13T00:57:23Z

I completely forgot about the PR as I’ve been using it as separate module.
Is it still of interest if I make the changes? Can’t really test in Ubuntu 14 if that’s a requirement

edsiper · 2021-12-13T02:47:11Z

just rebase on top of GIT Master, for Ubuntu >= 16.04 or centos >= 7 should be fine

github-actions · 2022-04-04T02:10:15Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

Signed-off-by: Filipe Pina <fopina@gmail.com>

fopina · 2022-04-14T00:19:34Z

just rebase on top of GIT Master, for Ubuntu >= 16.04 or centos >= 7 should be fine

@edsiper ✅

rebased and moved the code around a little bit, looking forward to have this merged and dropping my external plugin :)

fopina · 2023-02-24T22:23:06Z

@niedbalski was there activity as it is meant to be merged?

fopina · 2023-05-27T02:02:00Z

@niedbalski worth pushing conflict resolution? I already did in the standalone copy of the module

fopina added a commit to fopina/fluent-bit-docs that referenced this pull request Feb 19, 2021

documentation changes for fluent/fluent-bit#3092

aa9156a

fopina mentioned this pull request Feb 19, 2021

add MemAvailable, Buffers and Cached from /proc/meminfo to in_mem plugin fluent/fluent-bit-docs#467

Open

fopina added a commit to fopina/fluent-bit-docs that referenced this pull request Feb 19, 2021

documentation changes for fluent/fluent-bit#3092

f81fb94

Signed-off-by: Filipe Pina <fopina@gmail.com>

fopina force-pushed the in_mem_available branch from 7ee63b2 to 8c0e5fe Compare February 19, 2021 00:40

fopina marked this pull request as ready for review February 19, 2021 01:03

fopina requested review from edsiper, fujimotos and koleini as code owners February 19, 2021 01:03

github-actions bot added the Stale label Mar 21, 2021

github-actions bot removed the Stale label Mar 22, 2021

github-actions bot added the Stale label May 16, 2021

github-actions bot removed the Stale label Jan 4, 2022

github-actions bot added the Stale label Apr 4, 2022

fopina force-pushed the in_mem_available branch from 8c0e5fe to 24de32f Compare April 13, 2022 23:53

fopina requested a review from leonardo-albertovich as a code owner April 13, 2022 23:53

fopina force-pushed the in_mem_available branch from 24de32f to 8de980b Compare April 13, 2022 23:56

in_mem: add MemAvailable, Buffers and Cached from /proc/meminfo

514ebb7

Signed-off-by: Filipe Pina <fopina@gmail.com>

fopina force-pushed the in_mem_available branch from 8de980b to 514ebb7 Compare April 14, 2022 00:11

fopina changed the title ~~add MemAvailable, Buffers and Cached from /proc/meminfo to in_mem plugin~~ in_mem: add MemAvailable, Buffers and Cached from /proc/meminfo Apr 14, 2022

github-actions bot removed the Stale label Apr 14, 2022

fopina mentioned this pull request Apr 19, 2022

in_mem: Read mem_free metric from /proc/meminfo instead of sysinfo() if available #5237

Open

cveilleux mentioned this pull request Apr 21, 2022

mem input: Treat buffer/cache memory as available, rather than used #1617

Closed

niedbalski added ok-to-test ok-package-test Run PR packaging tests labels May 31, 2022

niedbalski self-assigned this May 31, 2022

niedbalski temporarily deployed to integration May 31, 2022 11:19 Inactive

niedbalski temporarily deployed to integration May 31, 2022 11:24 Inactive

niedbalski temporarily deployed to integration May 31, 2022 11:25 Inactive

github-actions bot added ci/integration-docker-ok ci/integration-test-ok labels May 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

in_mem: add MemAvailable, Buffers and Cached from /proc/meminfo #3092

in_mem: add MemAvailable, Buffers and Cached from /proc/meminfo #3092

fopina commented Feb 19, 2021 •

edited

fopina commented Feb 19, 2021 •

edited

github-actions bot commented Mar 21, 2021

fopina commented Mar 21, 2021

nokute78 commented Apr 12, 2021

fopina commented Apr 12, 2021 •

edited

nokute78 commented Apr 14, 2021

fopina commented Apr 14, 2021

nokute78 commented Apr 14, 2021

fopina commented Apr 14, 2021

github-actions bot commented May 16, 2021

edsiper commented Dec 13, 2021

fopina commented Dec 13, 2021

edsiper commented Dec 13, 2021

github-actions bot commented Apr 4, 2022

fopina commented Apr 14, 2022

fopina commented Feb 24, 2023

fopina commented May 27, 2023

in_mem: add MemAvailable, Buffers and Cached from /proc/meminfo #3092

Are you sure you want to change the base?

in_mem: add MemAvailable, Buffers and Cached from /proc/meminfo #3092

Conversation

fopina commented Feb 19, 2021 • edited

fopina commented Feb 19, 2021 • edited

github-actions bot commented Mar 21, 2021

fopina commented Mar 21, 2021

nokute78 commented Apr 12, 2021

fopina commented Apr 12, 2021 • edited

nokute78 commented Apr 14, 2021

fopina commented Apr 14, 2021

nokute78 commented Apr 14, 2021

fopina commented Apr 14, 2021

github-actions bot commented May 16, 2021

edsiper commented Dec 13, 2021

fopina commented Dec 13, 2021

edsiper commented Dec 13, 2021

github-actions bot commented Apr 4, 2022

fopina commented Apr 14, 2022

fopina commented Feb 24, 2023

fopina commented May 27, 2023

fopina commented Feb 19, 2021 •

edited

fopina commented Feb 19, 2021 •

edited

fopina commented Apr 12, 2021 •

edited