You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
filter_lua: Document record split and many minor improvements (#436)
Add description about that the Lua callback can return an array of
records, i.e., the third return value of the callback function can be an
array of tables. This feature enables this Lua filter to split an input
record into multiple records. Document the record split feature as a
new subsection and add an example about it.
See also fluent/fluent-bit#811 .
While there, variosu minor grammar and format fixes.
Signed-off-by: Weitian LI <liweitianux@live.com>
Lua Filter allows you to modify the incoming records using custom [Lua](https://www.lua.org/)Scripts.
3
+
The **Lua** filter allows you to modify the incoming records (even split one record into multiple records) using custom [Lua](https://www.lua.org/)scripts.
4
4
5
-
Due to the necessity to have a flexible filtering mechanism, now is possible to extend Fluent Bit capabilities writing simple filters using Lua programming language. A Luabased filter takes two steps:
5
+
Due to the necessity to have a flexible filtering mechanism, it is now possible to extend Fluent Bit capabilities by writing custom filters using Lua programming language. A Lua-based filter takes two steps:
6
6
7
-
* Configure the Filter in the main configuration
8
-
* Prepare a Lua script that will be used by the Filter
7
+
1. Configure the Filter in the main configuration
8
+
2. Prepare a Lua script that will be used by the Filter
9
9
10
10
## Configuration Parameters <aid="config"></a>
11
11
12
12
The plugin supports the following configuration parameters:
13
13
14
14
| Key | Description |
15
15
| :--- | :--- |
16
-
| script | Path to the Lua script that will be used. |
17
-
| call | Lua function name that will be triggered to do filtering. It's assumed that the function is declared inside the Script defined above. |
16
+
| script | Path to the Lua script that will be used. This can be a relative path against the main configuration file. |
17
+
| call | Lua function name that will be triggered to do filtering. It's assumed that the function is declared inside the **script** parameter defined above. |
18
18
| type\_int\_key | If these keys are matched, the fields are converted to integer. If more than one key, delimit by space. Note that starting from Fluent Bit v1.6 integer data types are preserved and not converted to double as in previous versions. |
19
19
| type\_array\_key| If these keys are matched, the fields are handled as array. If more than one key, delimit by space. It is useful the array can be empty. |
20
-
| protected\_mode | If enabled, Lua script will be executed in protected mode. It prevents to crash when invalid Lua script is executed. Default is true. |
21
-
| time\_as\_table | By default when the Lua script is invoked, the record timestamp is passed as a Floating number which might lead to loss precision when the data is converted back. If you desire timestamp precision enabling this option will pass the timestamp as a Lua table with keys `sec` for seconds since epoch and `nsec` for nanoseconds. |
20
+
| protected\_mode | If enabled, Lua script will be executed in protected mode. It prevents Fluent Bit from crashing when invalid Lua script is executed or the triggered Lua function throws exceptions. Default is true. |
21
+
| time\_as\_table | By default when the Lua script is invoked, the record timestamp is passed as a *floating number* which might lead to precision loss when it is converted back. If you desire timestamp precision, enabling this option will pass the timestamp as a Lua table with keys `sec` for seconds since epoch and `nsec` for nanoseconds. |
22
22
23
23
## Getting Started <aid="getting_started"></a>
24
24
25
-
In order to test the filter, you can run the plugin from the command line or through the configuration file. The following examples uses the [dummy](../inputs/dummy.md) input plugin for data ingestion, invoke Lua filter using the [test.lua](https://github.com/fluent/fluent-bit/blob/master/scripts/test.lua) script and calls the [cb\_print\(\)](https://github.com/fluent/fluent-bit/blob/master/scripts/test.lua#L29) function which only print the same information to the standard output:
25
+
In order to test the filter, you can run the plugin from the command line or through the configuration file. The following examples use the [dummy](../inputs/dummy.md) input plugin for data ingestion, invoke Lua filter using the [test.lua](https://github.com/fluent/fluent-bit/blob/master/scripts/test.lua) script and call the [cb\_print\(\)](https://github.com/fluent/fluent-bit/blob/master/scripts/test.lua#L29) function which only prints the same information to the standard output:
26
26
27
27
### Command Line
28
28
@@ -38,7 +38,7 @@ In your main configuration file append the following _Input_, _Filter_ & _Output
38
38
39
39
```python
40
40
[INPUT]
41
-
Name dummy
41
+
Name dummy
42
42
43
43
[FILTER]
44
44
Name lua
@@ -47,26 +47,27 @@ In your main configuration file append the following _Input_, _Filter_ & _Output
47
47
call cb_print
48
48
49
49
[OUTPUT]
50
-
Name null
51
-
Match *
50
+
Name null
51
+
Match *
52
52
```
53
53
54
54
## Lua Script Filter API <aid="lua_script"></a>
55
55
56
56
The life cycle of a filter have the following steps:
57
57
58
-
* Upon Tag matching by filter\_lua, it may process or bypass the record.
59
-
* If filter\_lua accepts the record, it will invoke the function defined in the _call_ property which basically is the name of a function defined in the Lua _script_.
60
-
* Invoke Lua function passing each record in JSON format.
61
-
* Upon return, validate return value and take some action \(described above\)
58
+
1. Upon Tag matching by this filter, it may process or bypass the record.
59
+
2. If tag matched, it will accept the record and invoke the function defined in the `call` property which basically is the name of a function defined in the Lua `script`.
60
+
3. Invoke Lua function and pass each record in JSON format.
61
+
4. Upon return, validate return value and continue the pipeline.
62
62
63
63
## Callback Prototype
64
64
65
-
The Lua script can have one or multiple callbacks that can be used by filter\_lua, it prototype is as follows:
65
+
The Lua script can have one or multiple callbacks that can be used by this filter. The function prototype is as follows:
66
66
67
67
```lua
68
68
functioncb_print(tag, timestamp, record)
69
-
returncode, timestamp, record
69
+
...
70
+
returncode, timestamp, record
70
71
end
71
72
```
72
73
@@ -75,7 +76,7 @@ end
75
76
| name | description |
76
77
| :--- | :--- |
77
78
| tag | Name of the tag associated with the incoming record. |
78
-
| timestamp | Unix timestamp with nanoseconds associated with the incoming record. The original format is a double \(seconds.nanoseconds\)|
79
+
| timestamp | Unix timestamp with nanoseconds associated with the incoming record. The original format is a double (seconds.nanoseconds) |
79
80
| record | Lua table with the record content |
80
81
81
82
#### Return Values
@@ -84,9 +85,9 @@ Each callback **must** return three values:
84
85
85
86
| name | data type | description |
86
87
| :--- | :--- | :--- |
87
-
| code | integer | The code return value represents the result and further action that may follows. If _code_ equals -1, means that filter\_lua must drop the record. If _code_ equals 0 the record will not be modified, otherwise if _code_ equals 1, means the original timestamp and record have been modified so it must be replaced by the returned values from _timestamp_\(second return value\) and _record_\(third return value\). If _code_ equals 2, means the original timestamp is not modified and the record has been modified so it must be replaced by the returned values from _record_\(third return value\). The _code_ 2 is supported from v1.4.3. |
88
+
| code | integer | The code return value represents the result and further action that may follows. If _code_ equals -1, means that the record will be dropped. If _code_ equals 0, the record will not be modified, otherwise if _code_ equals 1, means the original timestamp and record have been modified so it must be replaced by the returned values from _timestamp_ (second return value) and _record_ (third return value). If _code_ equals 2, means the original timestamp is not modified and the record has been modified so it must be replaced by the returned values from _record_ (third return value). The _code_ 2 is supported from v1.4.3. |
88
89
| timestamp | double | If code equals 1, the original record timestamp will be replaced with this new value. |
89
-
| record | table |if code equals 1, the original record information will be replaced with this new value. Note that the format of this value **must** be a valid Lua table. |
90
+
| record | table |If code equals 1, the original record information will be replaced with this new value. Note that the _record_value **must** be a valid Lua table. This value can be an array of tables (i.e., array of objects in JSON format), and in that case the input record is effectively split into multiple records. (see below for more details)|
90
91
91
92
### Code Examples
92
93
@@ -96,9 +97,65 @@ For functional examples of this interface, please refer to the code samples prov
96
97
97
98
### Number Type
98
99
99
-
In Lua, Fluent Bit treats number as double. It means an integer field \(e.g. IDs, log levels\) will be converted double. To avoid type conversion, **Type\_int\_key** property is available.
100
+
+Luatreats number as double. It means an integer field (e.g. IDs, log levels) will be converted double. To avoid type conversion, The `type_int_key` property is available.
100
101
101
102
### Protected Mode
102
103
103
104
Fluent Bit supports protected mode to prevent crash when executes invalid Lua script. See also [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html).
104
105
106
+
### Record Split
107
+
108
+
The Lua callback function can return an array of tables (i.e., array of records) in its third _record_ return value. With this feature, the Lua filter can split one input record into multiple records according to custom logic.
0 commit comments