Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lua: support metatable to save table type #7891

Merged
merged 2 commits into from Sep 23, 2023

Conversation

nokute78
Copy link
Collaborator

@nokute78 nokute78 commented Sep 2, 2023

Fixes #7708

Issue

Fluent-bit converts empty array into empty map.
Lua table represents array and map and we can't determine when a table is empty.

Solution

Lua supports metatable and this patch is to append metatable to save data type.
https://www.lua.org/manual/5.1/manual.html#2.8

  1. Fluent-bit converts msgpack into Lua data type and appends metatable if the type is array or map.
  2. Fluent-bit processes Lua script.
  3. Fluent-bit converts Lua data type into msgpack using metatable if it is saved.

Note:
metatable is for Lua table and userdata since other types share a single metatable for the type.

Tables and full userdata have individual metatables (although multiple tables and userdata can share their metatables). Values of all other types share one single metatable per type; that is, there is one single metatable for all numbers, one for all strings, etc.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • [N/A] Backport to latest stable release.

Configuration

[INPUT]
    Name dummy
    Dummy {"int":10, "double":23.4, "bool":true, "array":[], "map":{}, "nest":{"n_k":"n_v", "n_a":[], "n_m":{}}}
    Samples 1


[FILTER]
    Name lua
    Match *
    Script b.lua
    Call test

[OUTPUT]
    Name stdout
    Match *
function test(tag, timestamp, record)
         new_record = record
         new_record["lua_k"] = "lua_v"

         mt = getmetatable(new_record)

         for key, val in pairs(mt) do
            print(key, val)
         end

         return 1, timestamp, record
end

Valgrind output

$ valgrind --leak-check=full  bin/fluent-bit -c b.conf 
==10710== Memcheck, a memory error detector
==10710== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==10710== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==10710== Command: bin/fluent-bit -c b.conf
==10710== 
Fluent Bit v2.1.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/03 08:32:42] [ info] [fluent bit] version=2.1.9, commit=dd3c63644b, pid=10710
[2023/09/03 08:32:42] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/03 08:32:42] [ info] [cmetrics] version=0.6.3
[2023/09/03 08:32:42] [ info] [ctraces ] version=0.3.1
[2023/09/03 08:32:42] [ info] [input:dummy:dummy.0] initializing
[2023/09/03 08:32:42] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/03 08:32:43] [ info] [output:stdout:stdout.0] worker #0 started
[2023/09/03 08:32:43] [ info] [sp] stream processor started
type	2
[0] dummy.0: [[1693697564.086604595, {}], {"map"=>{}, "nest"=>{"n_a"=>[], "n_m"=>{}, "n_k"=>"n_v"}, "lua_k"=>"lua_v", "int"=>10, "double"=>23.400000, "bool"=>true, "array"=>[]}]
^C[2023/09/03 08:32:47] [engine] caught signal (SIGINT)
[2023/09/03 08:32:47] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/03 08:32:47] [ info] [input] pausing dummy.0
[2023/09/03 08:32:48] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/03 08:32:48] [ info] [input] pausing dummy.0
[2023/09/03 08:32:48] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/09/03 08:32:48] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==10710== 
==10710== HEAP SUMMARY:
==10710==     in use at exit: 0 bytes in 0 blocks
==10710==   total heap usage: 1,746 allocs, 1,746 frees, 890,669 bytes allocated
==10710== 
==10710== All heap blocks were freed -- no leaks are possible
==10710== 
==10710== For lists of detected and suppressed errors, rerun with: -s
==10710== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>
Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>
@nokute78 nokute78 temporarily deployed to pr September 2, 2023 23:48 — with GitHub Actions Inactive
@nokute78 nokute78 temporarily deployed to pr September 2, 2023 23:48 — with GitHub Actions Inactive
@nokute78 nokute78 temporarily deployed to pr September 2, 2023 23:48 — with GitHub Actions Inactive
@nokute78 nokute78 temporarily deployed to pr September 3, 2023 00:18 — with GitHub Actions Inactive
@edsiper edsiper merged commit 7ebe7a2 into fluent:master Sep 23, 2023
42 of 43 checks passed
@nokute78 nokute78 deleted the lua_metatable branch September 24, 2023 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

filter_lua changes the JSON schema
2 participants