Allow for configuration of OJ options #3315

conorevans · 2021-04-04T12:54:35Z

Signed-off-by: Conor Evans coevans@tcd.ie

Which issue(s) this PR fixes:
It relates to #3311 but does not close it

What this PR does / why we need it:

is there a way to change DEFAULT_OJ_OPTIONS variable ?

It seems there is no way to do it (pull request is welcome 😄)

from @ashie

Docs Changes:

TODO: add docs for this

Release Note:

Hi @ashie, I saw the discussion on #3311 and figured I could take this on. I just wanted to loop you in first:

TODO: add documentation for these env vars
TODO: add testing to types explicitly (it is somewhat tested via env_utils)
TODO: should I add support for all of https://github.com/ohler55/oj/blob/develop/pages/Options.md? I didn't want to create a huge PR before consulting, so I just set the current options (+ max_nesting as requested on Max nesting level for json parser #3311 by @bazzilio)

lib/fluent/config/types.rb

lib/fluent/env_utils.rb

lib/fluent/config/types.rb

conorevans · 2021-04-04T13:08:32Z

I'm also happy to extend the max_nesting functionality beyond oj (i.e. to yajl which is actually used) but I wasn't sure if these two things should be done in the same PR. Let me know - thanks.

ashie

Sorry for my late response.
It's basically O.K. for me but some changes are required (see inline comments).

I'm also happy to extend the max_nesting functionality beyond oj (i.e. to yajl which is actually used) but I wasn't sure if these two things should be done in the same PR. Let me know - thanks.

I think it should be another PR.

lib/fluent/config/types.rb

lib/fluent/env_utils.rb

lib/fluent/config/types.rb

lib/fluent/env_utils.rb

kenhys · 2021-05-19T03:14:56Z

a6623bc
020693a
0d7c980
It is better to squash these commits in one commit.

kenhys · 2021-05-19T03:25:39Z

lib/fluent/env_utils.rb

+        OJ_OPTIONS_DEFAULTS.each { |key, value| @options[key] = value }
+      end
+
+      def get_options


get_options may be a redundant name.
It seems that options is enough.

Would that not clash with @options above?

It seems that it does not conflict with it.

conorevans · 2021-05-19T18:58:27Z

Thanks for feedback. Aim to get back to this soon.

conorevans · 2021-05-30T18:15:28Z

Will aim to get back to this at some point in 7-14 days, cannot before then unfortunately.

kenhys · 2021-06-18T04:31:43Z

@conorevans we are looking forward to make this PR forward!

conorevans · 2021-06-20T16:05:25Z

Hi @kenhys sorry this slipped off my mind, thanks for the reminder!

I've addressed the feedback left on the MR. I think squashing the commits is fine (isn't that an option on merge? or shall I clean that up manually?).

kenhys · 2021-06-21T02:48:59Z

I think squashing the commits is fine (isn't that an option on merge? or shall I clean that up manually?).

Yes, PR should be done with a fineness of commit effectively (supporting Config.symbol_value and test. Oj options and test case so on)

ashie · 2021-07-05T08:18:48Z

Sorry for my late response.
I confirmed that the things I pointed out are fixed. Thanks for your work!

But CI was failed due to errors like following:

https://pipelines.actions.githubusercontent.com/ZYtSfNSOzFZbJsnI7WpW4orkGZmyuPUw7mkWiXZKPSFdBSfK7l/_apis/pipelines/1/runs/2212/signedlogcontent/3?urlExpires=2021-07-05T07%3A56%3A28.4329674Z&urlSigningMethod=HMACV1&urlSignature=29l%2Bas9fKsaFJ1%2BPikZ4CahzSImmK5Dg%2BsLwOPrZ25Q%3D

2021-07-05T07:19:30.6421804Z Failure: test: all secret parameters in worker section is sealed(TestFluentdCommand::config dump):
2021-07-05T07:19:30.6423828Z   unexpected error in launching fluentd: #<Errno::ENOENT: No such file or directory - bundle>.
2021-07-05T07:19:30.6424612Z   <"">
2021-07-05T07:19:30.6425163Z   was expected to include:
2021-07-05T07:19:30.6425743Z   <["secret xxxxxx"]>
2021-07-05T07:19:30.6426291Z   and not include:
2021-07-05T07:19:30.6426895Z   <["secret secret0", "secret secret1"]>
2021-07-05T07:19:30.6428005Z /home/runner/work/fluentd/fluentd/test/command/test_fluentd.rb:162:in `assert_log_matches'
2021-07-05T07:19:30.6429405Z /home/runner/work/fluentd/fluentd/test/command/test_fluentd.rb:1072:in `block (2 levels) in <class:TestFluentdCommand>'

It's caused on loading lib/fluent/config.rb and its dependencies.
But it seems that it's caused by unnecessary dependencies.
So I add a commit to remove such dependency.

ashie · 2021-07-05T08:46:12Z

So I add a commit to remove such dependency.

Hmm, although it solves the issue on my environment, not yet solved on CI environment 🤔 : https://github.com/fluent/fluentd/pull/3315/checks?check_run_id=2988122775

ashie · 2021-07-05T09:08:12Z

So I add a commit to remove such dependency.

Hmm, although it solves the issue on my environment, not yet solved on CI environment thinking : https://github.com/fluent/fluentd/pull/3315/checks?check_run_id=2988122775

I've removed this commit since it's doesn't take effect on it.

test/test_oj_options.rb

ashie · 2021-07-06T00:51:34Z

I'll fix broken tests and add trivial fixes.
After finishing them, I'll merge this.

Signed-off-by: Conor Evans <coevans@tcd.ie>

…Options directly under Fluent, 2) use strict=true in reformatted value and return the int parsing back to normal as that is a breaking change Signed-off-by: Conor Evans <coevans@tcd.ie>

Signed-off-by: Conor Evans <coevans@tcd.ie>

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

It shouldn't be set by each plugins, because later loaded plugins will overwrites it. Signed-off-by: Takuro Ashie <ashie@clear-code.com>

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

ashie · 2021-07-08T04:34:18Z

@kenhys Could you review my additional commit especially 3217fe4 ?
Because Oj.default_options shouldn't be set by each plugins, I modified to set it only once at starting up of fluentd.

kenhys · 2021-07-08T05:42:01Z

Ok, I'll do it.

kenhys

It seems that load_env works as expected. 👍

ashie · 2021-07-09T04:46:59Z

Thanks for your contribution!

conorevans · 2021-07-09T11:44:33Z

No problem @ashie - thanks for helping close it out!

…-2.9.x) (#10822) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [fluentd](https://www.fluentd.org/) ([source](https://togithub.com/fluent/fluentd)) | `'1.9.0'` -> `'1.14.2'` | [![age](https://developer.mend.io/api/mc/badges/age/rubygems/fluentd/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/rubygems/fluentd/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/rubygems/fluentd/'1.9.0'/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/rubygems/fluentd/'1.9.0'/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | ### GitHub Vulnerability Alerts #### [CVE-2021-41186](https://togithub.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662) ### Impact parser_apache2 plugin in Fluentd v0.14.14 to v1.14.1 suffers from a regular expression denial of service (ReDoS) vulnerability. A broken apache log with a certain pattern of string can spend too much time in a regular expression, resulting in the potential for a DoS attack. ### Patches v1.14.2 ### Workarounds Either of the following: * Don't use parser_apache2 for parsing logs which cannot guarantee generated by Apache. * Put patched version of parser_apache2.rb into /etc/fluent/plugin directory (or any other directories specified by the environment variable `FLUENT_PLUGIN` or `--plugin` option of fluentd). ### References * [CVE-2021-41186](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41186) * [GHSA-hwhf-64mh-r662](https://togithub.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662) * [GHSL-2021-102](https://securitylab.github.com/advisories/GHSL-2021-102-fluent-fluentd/) * https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1142 --- ### ReDoS vulnerability in parser_apache2 [CVE-2021-41186](https://nvd.nist.gov/vuln/detail/CVE-2021-41186) / [GHSA-hwhf-64mh-r662](https://togithub.com/advisories/GHSA-hwhf-64mh-r662) <details> <summary>More information</summary> #### Details ##### Impact parser_apache2 plugin in Fluentd v0.14.14 to v1.14.1 suffers from a regular expression denial of service (ReDoS) vulnerability. A broken apache log with a certain pattern of string can spend too much time in a regular expression, resulting in the potential for a DoS attack. ##### Patches v1.14.2 ##### Workarounds Either of the following: * Don't use parser_apache2 for parsing logs which cannot guarantee generated by Apache. * Put patched version of parser_apache2.rb into /etc/fluent/plugin directory (or any other directories specified by the environment variable `FLUENT_PLUGIN` or `--plugin` option of fluentd). ##### References * [CVE-2021-41186](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41186) * [GHSA-hwhf-64mh-r662](https://togithub.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662) * [GHSL-2021-102](https://securitylab.github.com/advisories/GHSL-2021-102-fluent-fluentd/) * https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1142 #### Severity - CVSS Score: 5.9 / 10 (Medium) - Vector String: `CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:N/A:H` #### References - [https://github.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662](https://togithub.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662) - [https://nvd.nist.gov/vuln/detail/CVE-2021-41186](https://nvd.nist.gov/vuln/detail/CVE-2021-41186) - [https://github.com/fluent/fluentd/commit/5482a3d049dab351de0be68f4b4bc562319d8511](https://togithub.com/fluent/fluentd/commit/5482a3d049dab351de0be68f4b4bc562319d8511) - [https://github.com/fluent/fluentd](https://togithub.com/fluent/fluentd) - [https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1142](https://togithub.com/fluent/fluentd/blob/master/CHANGELOG.md#v1142) - [https://github.com/github/securitylab-vulnerabilities/blob/52dc4a2a828c6dc24231967c2937ad92038184a9/vendor_reports/GHSL-2021-102-fluent-fluentd.md](https://togithub.com/github/securitylab-vulnerabilities/blob/52dc4a2a828c6dc24231967c2937ad92038184a9/vendor_reports/GHSL-2021-102-fluent-fluentd.md) - [https://github.com/rubysec/ruby-advisory-db/blob/master/gems/fluentd/CVE-2021-41186.yml](https://togithub.com/rubysec/ruby-advisory-db/blob/master/gems/fluentd/CVE-2021-41186.yml) This data is provided by [OSV](https://osv.dev/vulnerability/GHSA-hwhf-64mh-r662) and the [GitHub Advisory Database](https://togithub.com/github/advisory-database) ([CC-BY 4.0](https://togithub.com/github/advisory-database/blob/main/LICENSE.md)). </details> --- ### Release Notes <details> <summary>fluent/fluentd (fluentd)</summary> ### [`v1.14.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1142---20211029) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.14.1...v1.14.2) IMPORTANT: This release contain the fix for CVE-2021-41186 - ReDoS vulnerability in `parser_apache2`. This vulnerability is affected from Fluentd v0.14.14 to v1.14.1. We recommend to upgrade Fluentd to v1.14.2 or use patched version of `parser_apache2` plugin. ##### Enhancement - fluent-cat: Add `--event-time` option to send specified event time for testing. [fluent/fluentd#3528 ##### Bug fix - Fixed to generate correct epoch timestamp even after switching Daylight Saving Time [fluent/fluentd#3524 - Fixed ReDoS vulnerability in parser_apache2. This vulnerability is caused by a certain pattern of a broken apache log. ### [`v1.14.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1141---20210929) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.14.0...v1.14.1) ##### Enhancement - in_tail: Added file related metrics. These metrics should be collected same as fluent-bit's in_ta[fluent/fluentd#3504 - out_forward: Changed to use metrics mechanism for node statistics [fluent/fluentd#3506 ##### Bug fix - in_tail: Fixed a crash bug that it raise undefined method of eof? error. This error may happen only when `read_bytes_limit_per_second` was specified. [fluent/fluentd#3500 - out_forward: Fixed a bug that node statistics information is not included correctly. [fluent/fluentd#3503 - Fixed a error when using `@include` directive It was occurred when http/https scheme URI is used in `@include` directive with Ruby 3. [fluent/fluentd#3517 - out_copy: Fixed to suppress a wrong warning for `ignore_if_prev_success` It didn't work even if a user set [fluent/fluentd#3515 - Fixed not to output nanoseconds field of next retry time in warning log Then, inappropriate labels in log are also fixed. (retry_time -> retry_times, next_retry_seconds -> next_retr[fluent/fluentd#3518 ### [`v1.14.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1140---20210830) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.3...v1.14.0) ##### Enhancement - Added `enable_input_metrics`, `enable_size_metrics` system configuration parameter This feature might need to pay higher CPU cost, so input event metrics features are disabled by default. These features are also enabled by `--enable-input-metrics`,`--enable-size-metrics` command line opti[fluent/fluentd#3440 - Added reserved word `@ROOT` for getting root router. This is incompatible change. Do not use `@ROOT` for label name. [fluent/fluentd#3358 - in_syslog: Added `send_keepalive_packet` option [fluent/fluentd#3474 - in_http: Added `cors_allow_credentials` option. This option tells browsers whether to expose the response to frontend when the credentials mode is "in[fluent/fluentd#3481 ##### Bug fix - in_tail: Fixed a bug that deleted paths are not removed from pos file by file compaction at start[fluent/fluentd#3467 - in_tail: Revived a warning message of retrying unaccessible file [fluent/fluentd#3478 - TLSServer: Fixed a crash bug on logging peer host name errors [fluent/fluentd#3483 ##### Misc - Added metrics plugin mechanism The implementations is changed to use metrics plugin. In the future, 3rd party plugin will be able to handle these m[fluent/fluentd#3471 ### [`v1.13.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1133---20210727) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.2...v1.13.3) ##### Bug fix - in_tail: Care DeletePending state on Windows [fluent/fluentd#3457 - in_tail: Fix some pos_file bugs. Avoid deleting pos_file entries unexpectedly when both `pos_file_compaction_interval` and `follow_inode` are enabled. Use `bytesize` instead of `size` for path length. [fluent/fluentd#3459 - in_tail: Fix detecting rotation twice on `follow_inode`. [fluent/fluentd#3466 ##### Misc - Remove needless spaces in a sample config file [fluent/fluentd#3456 ### [`v1.13.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1132---20210712) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.1...v1.13.2) ##### Enhancement - fluent-plugin-generate: Storage plugin was supported. [fluent/fluentd#3426 - parser_json: Added support to customize configuration of oj options. Use `FLUENT_OJ_OPTION_BIGDECIMAL_LOAD`, `FLUENT_OJ_OPTION_MAX_NESTING`, `FLUENT_OJ_OPTION_MODE`, and `FLUENT_OJ_OPTION_USE_TO_JSON` environment variable to configure [fluent/fluentd#3315 ##### Bug fix - binlog_reader: Fixed a crash bug by missing "fluent/env" dependency. [fluent/fluentd#3443 - Fixed a crash bug on outputting log at the early stage when parsing config file. This is a regression since v1.13.0. If you use invalid '@' prefix parameter, remove it as [fluent/fluentd#3451 - in_tail: Fixed a bug that when rotation is occurred, remaining lines will be discarded if the throttling feature is enabl[fluent/fluentd#3390 - fluent-plugin-generate: Fixed a crash bug during gemspec generation. It was unexpectedly introduced by [#3305](https://togithub.com/fluent/fluentd/issues/3305), thus this bug was a regression since 1.12[fluent/fluentd#3444 ##### Misc - Fixed the runtime dependency version of http_parse.rb to 0.7.0. It was fixed because false positive detection is occurred frequently by security scanning[fluent/fluentd#3450 ### [`v1.13.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1131---20210625) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.0...v1.13.1) ##### Bug fix - out_forward: Fixed a race condition on handshake It's caused by using a same unpacker from multiple threa[fluent/fluentd#3405 - in_tail: Fixed to remove too much verbose debugging logs It was unexpectedly introduced by [#3185](https://togithub.com/fluent/fluentd/issues/3185) log throttling feature. [fluent/fluentd#3418 - Fixed not to echo back the provides path as is on a 404 error There was a potential cross-site scripting vector even though it is quite difficult to e[fluent/fluentd#3427 ##### Misc - Pretty print for Fluent::Config::Section has been supported for debugg[fluent/fluentd#3398 - CI: Dropped to run CI for Ruby 2.5 [fluent/fluentd#3412 ### [`v1.13.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1130---20210529) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.4...v1.13.0) ##### Enhancement - in_tail: Handle log throttling per file feature [fluent/fluentd#3185 - Extend to support service discovery manager in simpler way [fluent/fluentd#3299 - in_http: HTTP GET requests has been supported [fluent/fluentd#3373 - The log rotate settings in system configuration has been supported [fluent/fluentd#3352 ##### Bug fix - Fix to disable `trace_instruction` when `RubyVM::InstructionSequence` is available. It improves compatibility with `truffleruby` some extent. [fluent/fluentd#3376 - in_tail: Safely skip files which are used by another process on Windows. It improves exception handling about `ERROR_SHARING_VIOLATION` on Windows. [fluent/fluentd#3378 - fluent-cat: the issue resending secondary file in specific format has been fi[fluent/fluentd#3368 - in_tail: Shutdown immediately & safely even if reading huge files Note that `skip_refresh_on_startup` must be enabled. [fluent/fluentd#3380 ##### Misc - example: Change a path to backup_path in counter_server correctly [fluent/fluentd#3359 - README: Update link to community forum to discuss.fluentd.org [fluent/fluentd#3360 ### [`v1.12.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1124---20210526) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.3...v1.12.4) ##### Bug fix - in_tail: Fix a bug that refresh_watcher fails to handle file rotations [fluent/fluentd#3393 ### [`v1.12.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1123---20210423) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.2...v1.12.3) ##### Enhancement - plugin_helper: Allow TLS to use keep-alive socket option [fluent/fluentd#3308 ##### Bug fix - parser_csv, parser_syslog: Fix a naming conflict on parser_type [fluent/fluentd#3302 - in_tail: Fix incorrect error code & message on Windows [fluent/fluentd#3325 - in_tail: Fix a crash bug on catching a short-lived log [fluent/fluentd#3328 - storage_local: Fix position file corruption issue on concurrent gracefulReloads [fluent/fluentd#3335 - Fix incorrect warnings about ${chunk_id} with out_s3 [fluent/fluentd#3339 - TLS Server: Add peer information to error log message [fluent/fluentd#3330 ##### Misc - fluent-plugin-generate: add note about plugin name [fluent/fluentd#3303 - fluent-plugin-generate: Use same depended gem version with fluentd [fluent/fluentd#3305 - Fix some broken unit tests and improve CI's stability [fluent/fluentd#3304 - Permit to install with win32-service 2.2.0 on Windows [fluent/fluentd#3343 ### [`v1.12.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1122---20210329) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.1...v1.12.2) ##### Enhancement - out_copy: Add ignore_if_prev_successes [fluent/fluentd#3190 - Support multiple kind of timestamp format [fluent/fluentd#3252 - formatter_ltsv: suppress delimiters in output [fluent/fluentd#1666 ##### Bug fix - in_tail: Expect ENOENT during stat [fluent/fluentd#3275 - out_forward: Prevent transferring duplicate logs on restart [fluent/fluentd#3267 - in_tail: Handle to send rotated logs when mv is used for rotating [fluent/fluentd#3294 - fluent-plugin-config-format: Fill an uninitialized instance variable [fluent/fluentd#3297 - Fix MessagePackEventStream issue with Enumerable methods [fluent/fluentd#2116 ##### Misc - Add webrick to support Ruby 3.0 [fluent/fluentd#3257 - Suggest Discource instead of Google Groups [fluent/fluentd#3261 - Update MAINTAINERS.md [fluent/fluentd#3282 - Introduce DeepSource to check code quality [fluent/fluentd#3286 - Migrate to GitHub Actions and stabilize tests [fluent/fluentd#3266 ### [`v1.12.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1121---20210218) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.0...v1.12.1) ##### Enhancement - out_http: Add `headers_from_placeholders` parameter [fluent/fluentd#3241 - fluent-plugin-config-format: Add `--table` option to use markdown table [fluent/fluentd#3240 - Add `--disable-shared-socket`/`disable_shared_socket` to disable ServerEngine's shared socket setup [fluent/fluentd#3250 ##### Bug fix - ca_generate: Fix creating TLS certification files which include broken extensions [fluent/fluentd#3246 - test: Drop TLS 1.1 tests [fluent/fluentd#3256 - Remove old gem constraints to support Ruby 3 ##### Misc - Use GitHub Actions [fluent/fluentd#3233 ### [`v1.12.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1120---20210105) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.5...v1.12.0) ##### New feature - in_tail: Add `follow_inode` to support log rotation with wild card [fluent/fluentd#3182 - in_tail: Handle linux capability [fluent/fluentd#3155 - windows: Add win32 events alternative to unix signals [fluent/fluentd#3131 ##### Enhancement - buffer: Enable metadata comparison optimization on all platforms [fluent/fluentd#3095 - fluent-plugin-config-formatter: Handle `service_discovery` type [fluent/fluentd#3178 - in_http: Add `add_query_params` parameter to add query params to event record [fluent/fluentd#3197 - inject: Support `unixtime_micros` and `unixtime_nanos` in `time_type` [fluent/fluentd#3220 - Refactoring code [fluent/fluentd#3167 ##### Bug fix - output: Prevent retry.step from being called too many times in a short time [fluent/fluentd#3203 ### [`v1.11.5`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1115---20201106) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.4...v1.11.5) ##### Enhancement - formatter: Provide `newline` parameter to support `CRLF` [fluent/fluentd#3152 - out_http: adding support for intermediate certificates [fluent/fluentd#3146 - Update serverengine dependency to 2.2.2 or later ##### Bug fix - Fix a bug that windows service isn't stopped gracefuly [fluent/fluentd#3156 ### [`v1.11.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1114---20201013) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.3...v1.11.4) ##### Enhancement - inject: Support `unixtime_millis` in `time_type` parameter [fluent/fluentd#3145 ##### Bug fix - out_http: Fix broken data with `json_array true` [fluent/fluentd#3144 - output: Fix wrong logging issue for `${chunk_id}` [fluent/fluentd#3134 ### [`v1.11.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1113---20200930) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.2...v1.11.3) ##### Enhancement - in_exec: Add `connect_mode` parameter to read stderr [fluent/fluentd#3108 - parser_json: Improve the performance [fluent/fluentd#3109 - log: Add `ignore_same_log_interval` parameter [fluent/fluentd#3119 - Upgrade win32 gems [fluent/fluentd#3100 - Refactoring code [fluent/fluentd#3094 ##### Bug fix - buffer: Fix calculation of timekey stats [fluent/fluentd#3018 - buffer: fix binmode usage for prevent gc [fluent/fluentd#3138 ### [`v1.11.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1112---20200804) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.1...v1.11.2) ##### Enhancement - `in_dummy` renamed to `in_sample` [fluent/fluentd#3065 - Allow regular expression in filter/match directive [fluent/fluentd#3071 - Refactoring code [fluent/fluentd#3051 ##### Bug fix - buffer: Fix log message for `chunk_limit_records` case [fluent/fluentd#3079 - buffer: Fix timekey optimization for non-windows platform [fluent/fluentd#3092 - cert: Raise an error for broken certificate file [fluent/fluentd#3086 - cert: Set TLS ciphers list correcty on older OpenSSL [fluent/fluentd#3093 ### [`v1.11.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1111---20200622) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.0...v1.11.1) ##### Enhancement - in_http: Add `dump_error_log` parameter [fluent/fluentd#3035 - in_http: Improve time field handling [fluent/fluentd#3046 - Refactoring code [fluent/fluentd#3047 ##### Bug fix - in_tail: Use actual path instead of based pattern for ignore list [fluent/fluentd#3042 - child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout [fluent/fluentd#3044 ### [`v1.11.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1110---20200604) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.4...v1.11.0) ##### New feature - in_unix: Use v1 API [fluent/fluentd#2992 ##### Enhancement - parser_syslog: Support any `time_format` for RFC3164 string parser [fluent/fluentd#3014 - parser_syslog: Add new parser for RFC5424 [fluent/fluentd#3015 - Refactoring code [fluent/fluentd#3019 ##### Bug fix - in_gc_stat: Add `use_symbol_keys` parameter to emit string key record [fluent/fluentd#3008 ### [`v1.10.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1104---20200512) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.3...v1.10.4) ##### Enhancement - out_http: Support single json array payload [fluent/fluentd#2973 - Refactoring [fluent/fluentd#2988 ##### Bug fix - supervisor: Call `File.umask(0)` for standalone worker [fluent/fluentd#2987 - out_forward: Fix ZeroDivisionError issue with `weight 0` [fluent/fluentd#2989 ### [`v1.10.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1103---20200501) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.2...v1.10.3) ##### Enhancement - record_accessor: Add `set` method [fluent/fluentd#2977 - config: Ruby DSL format is deprecated [fluent/fluentd#2958 - Refactor code [fluent/fluentd#2961 ##### Bug fix - out_forward: Disable `linger_timeout` setting on Windows [fluent/fluentd#2959 - out_forward: Fix warning of service discovery manager when fluentd stops [fluent/fluentd#2974 ### [`v1.10.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1102---20200415) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.1...v1.10.2) ##### Enhancement - out_copy: Add plugin_id to log message [fluent/fluentd#2934 - socket: Allow cert chains in mutual auth [fluent/fluentd#2930 - system: Add ignore_repeated_log_interval parameter [fluent/fluentd#2937 - windows: Allow to launch fluentd from whitespace included path [fluent/fluentd#2920 - Refactor code [fluent/fluentd#2935 ##### Bug fix - in_syslog: Fix octet-counting mode bug [fluent/fluentd#2942 - out_forward: Create timer for purging obsolete sockets when keepalive_timeout is not set [fluent/fluentd#2943 - out_forward: Need authentication when sending tcp heartbeat with keepalive [fluent/fluentd#2945 - command: Fix fluent-debug start failure [fluent/fluentd#2948 - command: Fix regression of supervisor's worker and `--daemon` combo [fluent/fluentd#2950 ### [`v1.10.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1101---20200402) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.0...v1.10.1) ##### Enhancement - command: `--daemon` and `--no-supervisor` now work together [fluent/fluentd#2912 - Refactor code [fluent/fluentd#2913 ##### Bug fix - in_tail: `Fix pos_file_compaction_interval` parameter type [fluent/fluentd#2921 - in_tail: Fix seek position update after compaction [fluent/fluentd#2922 - parser_syslog: Fix regression in the `with_priority` and RFC5424 case [fluent/fluentd#2923 ##### Misc - Add document for security audit [fluent/fluentd#2911 ### [`v1.10.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1100---20200324) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.3...v1.10.0) ##### New feature - sd plugin: Add SRV record plugin [fluent/fluentd#2876 ##### Enhancement - server: Add `cert_verifier` parameter for TLS transport [fluent/fluentd#2888 - parser_syslog: Support customized time format [fluent/fluentd#2886 - in_dummy: Delete `suspend` parameter [fluent/fluentd#2897 - Refactor code [fluent/fluentd#2858 [fluent/fluentd#2899 [fluent/fluentd#2900 fluent/fluentd#2906 ##### Bug fix - out_forward: windows: Permit to specify `linger_timeout` [fluent/fluentd#2868 - parser_syslog: Fix syslog format detection [fluent/fluentd#2879 - buffer: Fix `available_buffer_space_ratio` calculation [fluent/fluentd#2882 - tls: Support CRLF based X.509 certificates [fluent/fluentd#2890 - msgpack_factory mixin: Fix performance penalty for deprecation log [fluent/fluentd#2903 ### [`v1.9.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v193---20200305) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.2...v1.9.3) ##### Enhancement - in_tail: Emit buffered lines as `unmatched_line` at shutdown phase when `emit_unmatched_lines true` [fluent/fluentd#2837 - Specify directory mode explicitly [fluent/fluentd#2827 - server helper: Change SSLError log level to warn in accept [fluent/fluentd#2861 - Refactor code [fluent/fluentd#2829 ##### Bug fix - buffer: Add seq to metadata that it can be unique [fluent/fluentd#2824 - buffer: Use `Tempfile` as binmode for decompression [fluent/fluentd#2847 ##### Misc - Add `.idea` to git ignore file [fluent/fluentd#2834 - appveyor: Fix tests [fluent/fluentd#2853 - Update pem for test [fluent/fluentd#2839 ### [`v1.9.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v192---20200213) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.1...v1.9.2) ##### Enhancement - in_tail: Add `pos_file_compaction_interval` parameter for auto compaction [fluent/fluentd#2805 - command: Use given encoding when RUBYOPT has `-E` [fluent/fluentd#2814 ##### Bug fix - command: Accept RUBYOPT with two or more options [fluent/fluentd#2807 - command: Fix infinite loop bug when RUBYOPT is invalid [fluent/fluentd#2813 - log: serverengine's log should be formatted with the same format of fluentd [fluent/fluentd#2812 - in_http: Fix `NoMethodError` when `OPTIONS` request doesn't have 'Origin' header [fluent/fluentd#2823 - parser_syslog: Improved for parsing RFC5424 structured data in `parser_syslog` [fluent/fluentd#2816 ### [`v1.9.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v191---20200131) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.0...v1.9.1) ##### Enhancement - http_server helper: Support HTTPS [fluent/fluentd#2787 - in_tail: Add `path_delimiter` to split with any char [fluent/fluentd#2796 - in_tail: Remove an entry from PositionaFile when it is unwatched [fluent/fluentd#2803 - out_http: Add warning for `retryable_response_code` [fluent/fluentd#2809 - parser_syslog: Add multiline RFC5424 support [fluent/fluentd#2767 - Add TLS module to unify TLS related code [fluent/fluentd#2802 ##### Bug fix - output: Add `EncodingError` to unrecoverable errors [fluent/fluentd#2808 - tls: Fix TLS version handling in secure mode [fluent/fluentd#2802 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…10818) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [fluentd](https://www.fluentd.org/) ([source](https://togithub.com/fluent/fluentd)) | `'1.9.0'` -> `'1.14.2'` | [![age](https://developer.mend.io/api/mc/badges/age/rubygems/fluentd/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/rubygems/fluentd/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/rubygems/fluentd/'1.9.0'/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/rubygems/fluentd/'1.9.0'/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. ### GitHub Vulnerability Alerts #### [CVE-2021-41186](https://togithub.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662) ### Impact parser_apache2 plugin in Fluentd v0.14.14 to v1.14.1 suffers from a regular expression denial of service (ReDoS) vulnerability. A broken apache log with a certain pattern of string can spend too much time in a regular expression, resulting in the potential for a DoS attack. ### Patches v1.14.2 ### Workarounds Either of the following: * Don't use parser_apache2 for parsing logs which cannot guarantee generated by Apache. * Put patched version of parser_apache2.rb into /etc/fluent/plugin directory (or any other directories specified by the environment variable `FLUENT_PLUGIN` or `--plugin` option of fluentd). ### References * [CVE-2021-41186](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41186) * [GHSA-hwhf-64mh-r662](https://togithub.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662) * [GHSL-2021-102](https://securitylab.github.com/advisories/GHSL-2021-102-fluent-fluentd/) * https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1142 --- ### Release Notes <details> <summary>fluent/fluentd (fluentd)</summary> ### [`v1.14.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1142---20211029) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.14.1...v1.14.2) IMPORTANT: This release contain the fix for CVE-2021-41186 - ReDoS vulnerability in `parser_apache2`. This vulnerability is affected from Fluentd v0.14.14 to v1.14.1. We recommend to upgrade Fluentd to v1.14.2 or use patched version of `parser_apache2` plugin. ##### Enhancement - fluent-cat: Add `--event-time` option to send specified event time for testing. [fluent/fluentd#3528 ##### Bug fix - Fixed to generate correct epoch timestamp even after switching Daylight Saving Time [fluent/fluentd#3524 - Fixed ReDoS vulnerability in parser_apache2. This vulnerability is caused by a certain pattern of a broken apache log. ### [`v1.14.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1141---20210929) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.14.0...v1.14.1) ##### Enhancement - in_tail: Added file related metrics. These metrics should be collected same as fluent-bit's in_ta[fluent/fluentd#3504 - out_forward: Changed to use metrics mechanism for node statistics [fluent/fluentd#3506 ##### Bug fix - in_tail: Fixed a crash bug that it raise undefined method of eof? error. This error may happen only when `read_bytes_limit_per_second` was specified. [fluent/fluentd#3500 - out_forward: Fixed a bug that node statistics information is not included correctly. [fluent/fluentd#3503 - Fixed a error when using `@include` directive It was occurred when http/https scheme URI is used in `@include` directive with Ruby 3. [fluent/fluentd#3517 - out_copy: Fixed to suppress a wrong warning for `ignore_if_prev_success` It didn't work even if a user set [fluent/fluentd#3515 - Fixed not to output nanoseconds field of next retry time in warning log Then, inappropriate labels in log are also fixed. (retry_time -> retry_times, next_retry_seconds -> next_retr[fluent/fluentd#3518 ### [`v1.14.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1140---20210830) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.3...v1.14.0) ##### Enhancement - Added `enable_input_metrics`, `enable_size_metrics` system configuration parameter This feature might need to pay higher CPU cost, so input event metrics features are disabled by default. These features are also enabled by `--enable-input-metrics`,`--enable-size-metrics` command line opti[fluent/fluentd#3440 - Added reserved word `@ROOT` for getting root router. This is incompatible change. Do not use `@ROOT` for label name. [fluent/fluentd#3358 - in_syslog: Added `send_keepalive_packet` option [fluent/fluentd#3474 - in_http: Added `cors_allow_credentials` option. This option tells browsers whether to expose the response to frontend when the credentials mode is "in[fluent/fluentd#3481 ##### Bug fix - in_tail: Fixed a bug that deleted paths are not removed from pos file by file compaction at start[fluent/fluentd#3467 - in_tail: Revived a warning message of retrying unaccessible file [fluent/fluentd#3478 - TLSServer: Fixed a crash bug on logging peer host name errors [fluent/fluentd#3483 ##### Misc - Added metrics plugin mechanism The implementations is changed to use metrics plugin. In the future, 3rd party plugin will be able to handle these m[fluent/fluentd#3471 ### [`v1.13.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1133---20210727) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.2...v1.13.3) ##### Bug fix - in_tail: Care DeletePending state on Windows [fluent/fluentd#3457 - in_tail: Fix some pos_file bugs. Avoid deleting pos_file entries unexpectedly when both `pos_file_compaction_interval` and `follow_inode` are enabled. Use `bytesize` instead of `size` for path length. [fluent/fluentd#3459 - in_tail: Fix detecting rotation twice on `follow_inode`. [fluent/fluentd#3466 ##### Misc - Remove needless spaces in a sample config file [fluent/fluentd#3456 ### [`v1.13.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1132---20210712) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.1...v1.13.2) ##### Enhancement - fluent-plugin-generate: Storage plugin was supported. [fluent/fluentd#3426 - parser_json: Added support to customize configuration of oj options. Use `FLUENT_OJ_OPTION_BIGDECIMAL_LOAD`, `FLUENT_OJ_OPTION_MAX_NESTING`, `FLUENT_OJ_OPTION_MODE`, and `FLUENT_OJ_OPTION_USE_TO_JSON` environment variable to configure [fluent/fluentd#3315 ##### Bug fix - binlog_reader: Fixed a crash bug by missing "fluent/env" dependency. [fluent/fluentd#3443 - Fixed a crash bug on outputting log at the early stage when parsing config file. This is a regression since v1.13.0. If you use invalid '@' prefix parameter, remove it as [fluent/fluentd#3451 - in_tail: Fixed a bug that when rotation is occurred, remaining lines will be discarded if the throttling feature is enabl[fluent/fluentd#3390 - fluent-plugin-generate: Fixed a crash bug during gemspec generation. It was unexpectedly introduced by [#3305](https://togithub.com/fluent/fluentd/issues/3305), thus this bug was a regression since 1.12[fluent/fluentd#3444 ##### Misc - Fixed the runtime dependency version of http_parse.rb to 0.7.0. It was fixed because false positive detection is occurred frequently by security scanning[fluent/fluentd#3450 ### [`v1.13.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1131---20210625) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.0...v1.13.1) ##### Bug fix - out_forward: Fixed a race condition on handshake It's caused by using a same unpacker from multiple threa[fluent/fluentd#3405 - in_tail: Fixed to remove too much verbose debugging logs It was unexpectedly introduced by [#3185](https://togithub.com/fluent/fluentd/issues/3185) log throttling feature. [fluent/fluentd#3418 - Fixed not to echo back the provides path as is on a 404 error There was a potential cross-site scripting vector even though it is quite difficult to e[fluent/fluentd#3427 ##### Misc - Pretty print for Fluent::Config::Section has been supported for debugg[fluent/fluentd#3398 - CI: Dropped to run CI for Ruby 2.5 [fluent/fluentd#3412 ### [`v1.13.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1130---20210529) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.4...v1.13.0) ##### Enhancement - in_tail: Handle log throttling per file feature [fluent/fluentd#3185 - Extend to support service discovery manager in simpler way [fluent/fluentd#3299 - in_http: HTTP GET requests has been supported [fluent/fluentd#3373 - The log rotate settings in system configuration has been supported [fluent/fluentd#3352 ##### Bug fix - Fix to disable `trace_instruction` when `RubyVM::InstructionSequence` is available. It improves compatibility with `truffleruby` some extent. [fluent/fluentd#3376 - in_tail: Safely skip files which are used by another process on Windows. It improves exception handling about `ERROR_SHARING_VIOLATION` on Windows. [fluent/fluentd#3378 - fluent-cat: the issue resending secondary file in specific format has been fi[fluent/fluentd#3368 - in_tail: Shutdown immediately & safely even if reading huge files Note that `skip_refresh_on_startup` must be enabled. [fluent/fluentd#3380 ##### Misc - example: Change a path to backup_path in counter_server correctly [fluent/fluentd#3359 - README: Update link to community forum to discuss.fluentd.org [fluent/fluentd#3360 ### [`v1.12.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1124---20210526) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.3...v1.12.4) ##### Bug fix - in_tail: Fix a bug that refresh_watcher fails to handle file rotations [fluent/fluentd#3393 ### [`v1.12.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1123---20210423) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.2...v1.12.3) ##### Enhancement - plugin_helper: Allow TLS to use keep-alive socket option [fluent/fluentd#3308 ##### Bug fix - parser_csv, parser_syslog: Fix a naming conflict on parser_type [fluent/fluentd#3302 - in_tail: Fix incorrect error code & message on Windows [fluent/fluentd#3325 - in_tail: Fix a crash bug on catching a short-lived log [fluent/fluentd#3328 - storage_local: Fix position file corruption issue on concurrent gracefulReloads [fluent/fluentd#3335 - Fix incorrect warnings about ${chunk_id} with out_s3 [fluent/fluentd#3339 - TLS Server: Add peer information to error log message [fluent/fluentd#3330 ##### Misc - fluent-plugin-generate: add note about plugin name [fluent/fluentd#3303 - fluent-plugin-generate: Use same depended gem version with fluentd [fluent/fluentd#3305 - Fix some broken unit tests and improve CI's stability [fluent/fluentd#3304 - Permit to install with win32-service 2.2.0 on Windows [fluent/fluentd#3343 ### [`v1.12.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1122---20210329) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.1...v1.12.2) ##### Enhancement - out_copy: Add ignore_if_prev_successes [fluent/fluentd#3190 - Support multiple kind of timestamp format [fluent/fluentd#3252 - formatter_ltsv: suppress delimiters in output [fluent/fluentd#1666 ##### Bug fix - in_tail: Expect ENOENT during stat [fluent/fluentd#3275 - out_forward: Prevent transferring duplicate logs on restart [fluent/fluentd#3267 - in_tail: Handle to send rotated logs when mv is used for rotating [fluent/fluentd#3294 - fluent-plugin-config-format: Fill an uninitialized instance variable [fluent/fluentd#3297 - Fix MessagePackEventStream issue with Enumerable methods [fluent/fluentd#2116 ##### Misc - Add webrick to support Ruby 3.0 [fluent/fluentd#3257 - Suggest Discource instead of Google Groups [fluent/fluentd#3261 - Update MAINTAINERS.md [fluent/fluentd#3282 - Introduce DeepSource to check code quality [fluent/fluentd#3286 - Migrate to GitHub Actions and stabilize tests [fluent/fluentd#3266 ### [`v1.12.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1121---20210218) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.0...v1.12.1) ##### Enhancement - out_http: Add `headers_from_placeholders` parameter [fluent/fluentd#3241 - fluent-plugin-config-format: Add `--table` option to use markdown table [fluent/fluentd#3240 - Add `--disable-shared-socket`/`disable_shared_socket` to disable ServerEngine's shared socket setup [fluent/fluentd#3250 ##### Bug fix - ca_generate: Fix creating TLS certification files which include broken extensions [fluent/fluentd#3246 - test: Drop TLS 1.1 tests [fluent/fluentd#3256 - Remove old gem constraints to support Ruby 3 ##### Misc - Use GitHub Actions [fluent/fluentd#3233 ### [`v1.12.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1120---20210105) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.5...v1.12.0) ##### New feature - in_tail: Add `follow_inode` to support log rotation with wild card [fluent/fluentd#3182 - in_tail: Handle linux capability [fluent/fluentd#3155 - windows: Add win32 events alternative to unix signals [fluent/fluentd#3131 ##### Enhancement - buffer: Enable metadata comparison optimization on all platforms [fluent/fluentd#3095 - fluent-plugin-config-formatter: Handle `service_discovery` type [fluent/fluentd#3178 - in_http: Add `add_query_params` parameter to add query params to event record [fluent/fluentd#3197 - inject: Support `unixtime_micros` and `unixtime_nanos` in `time_type` [fluent/fluentd#3220 - Refactoring code [fluent/fluentd#3167 ##### Bug fix - output: Prevent retry.step from being called too many times in a short time [fluent/fluentd#3203 ### [`v1.11.5`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1115---20201106) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.4...v1.11.5) ##### Enhancement - formatter: Provide `newline` parameter to support `CRLF` [fluent/fluentd#3152 - out_http: adding support for intermediate certificates [fluent/fluentd#3146 - Update serverengine dependency to 2.2.2 or later ##### Bug fix - Fix a bug that windows service isn't stopped gracefuly [fluent/fluentd#3156 ### [`v1.11.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1114---20201013) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.3...v1.11.4) ##### Enhancement - inject: Support `unixtime_millis` in `time_type` parameter [fluent/fluentd#3145 ##### Bug fix - out_http: Fix broken data with `json_array true` [fluent/fluentd#3144 - output: Fix wrong logging issue for `${chunk_id}` [fluent/fluentd#3134 ### [`v1.11.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1113---20200930) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.2...v1.11.3) ##### Enhancement - in_exec: Add `connect_mode` parameter to read stderr [fluent/fluentd#3108 - parser_json: Improve the performance [fluent/fluentd#3109 - log: Add `ignore_same_log_interval` parameter [fluent/fluentd#3119 - Upgrade win32 gems [fluent/fluentd#3100 - Refactoring code [fluent/fluentd#3094 ##### Bug fix - buffer: Fix calculation of timekey stats [fluent/fluentd#3018 - buffer: fix binmode usage for prevent gc [fluent/fluentd#3138 ### [`v1.11.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1112---20200804) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.1...v1.11.2) ##### Enhancement - `in_dummy` renamed to `in_sample` [fluent/fluentd#3065 - Allow regular expression in filter/match directive [fluent/fluentd#3071 - Refactoring code [fluent/fluentd#3051 ##### Bug fix - buffer: Fix log message for `chunk_limit_records` case [fluent/fluentd#3079 - buffer: Fix timekey optimization for non-windows platform [fluent/fluentd#3092 - cert: Raise an error for broken certificate file [fluent/fluentd#3086 - cert: Set TLS ciphers list correcty on older OpenSSL [fluent/fluentd#3093 ### [`v1.11.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1111---20200622) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.0...v1.11.1) ##### Enhancement - in_http: Add `dump_error_log` parameter [fluent/fluentd#3035 - in_http: Improve time field handling [fluent/fluentd#3046 - Refactoring code [fluent/fluentd#3047 ##### Bug fix - in_tail: Use actual path instead of based pattern for ignore list [fluent/fluentd#3042 - child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout [fluent/fluentd#3044 ### [`v1.11.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1110---20200604) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.4...v1.11.0) ##### New feature - in_unix: Use v1 API [fluent/fluentd#2992 ##### Enhancement - parser_syslog: Support any `time_format` for RFC3164 string parser [fluent/fluentd#3014 - parser_syslog: Add new parser for RFC5424 [fluent/fluentd#3015 - Refactoring code [fluent/fluentd#3019 ##### Bug fix - in_gc_stat: Add `use_symbol_keys` parameter to emit string key record [fluent/fluentd#3008 ### [`v1.10.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1104---20200512) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.3...v1.10.4) ##### Enhancement - out_http: Support single json array payload [fluent/fluentd#2973 - Refactoring [fluent/fluentd#2988 ##### Bug fix - supervisor: Call `File.umask(0)` for standalone worker [fluent/fluentd#2987 - out_forward: Fix ZeroDivisionError issue with `weight 0` [fluent/fluentd#2989 ### [`v1.10.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1103---20200501) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.2...v1.10.3) ##### Enhancement - record_accessor: Add `set` method [fluent/fluentd#2977 - config: Ruby DSL format is deprecated [fluent/fluentd#2958 - Refactor code [fluent/fluentd#2961 ##### Bug fix - out_forward: Disable `linger_timeout` setting on Windows [fluent/fluentd#2959 - out_forward: Fix warning of service discovery manager when fluentd stops [fluent/fluentd#2974 ### [`v1.10.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1102---20200415) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.1...v1.10.2) ##### Enhancement - out_copy: Add plugin_id to log message [fluent/fluentd#2934 - socket: Allow cert chains in mutual auth [fluent/fluentd#2930 - system: Add ignore_repeated_log_interval parameter [fluent/fluentd#2937 - windows: Allow to launch fluentd from whitespace included path [fluent/fluentd#2920 - Refactor code [fluent/fluentd#2935 ##### Bug fix - in_syslog: Fix octet-counting mode bug [fluent/fluentd#2942 - out_forward: Create timer for purging obsolete sockets when keepalive_timeout is not set [fluent/fluentd#2943 - out_forward: Need authentication when sending tcp heartbeat with keepalive [fluent/fluentd#2945 - command: Fix fluent-debug start failure [fluent/fluentd#2948 - command: Fix regression of supervisor's worker and `--daemon` combo [fluent/fluentd#2950 ### [`v1.10.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1101---20200402) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.0...v1.10.1) ##### Enhancement - command: `--daemon` and `--no-supervisor` now work together [fluent/fluentd#2912 - Refactor code [fluent/fluentd#2913 ##### Bug fix - in_tail: `Fix pos_file_compaction_interval` parameter type [fluent/fluentd#2921 - in_tail: Fix seek position update after compaction [fluent/fluentd#2922 - parser_syslog: Fix regression in the `with_priority` and RFC5424 case [fluent/fluentd#2923 ##### Misc - Add document for security audit [fluent/fluentd#2911 ### [`v1.10.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1100---20200324) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.3...v1.10.0) ##### New feature - sd plugin: Add SRV record plugin [fluent/fluentd#2876 ##### Enhancement - server: Add `cert_verifier` parameter for TLS transport [fluent/fluentd#2888 - parser_syslog: Support customized time format [fluent/fluentd#2886 - in_dummy: Delete `suspend` parameter [fluent/fluentd#2897 - Refactor code [fluent/fluentd#2858 [fluent/fluentd#2899 [fluent/fluentd#2900 fluent/fluentd#2906 ##### Bug fix - out_forward: windows: Permit to specify `linger_timeout` [fluent/fluentd#2868 - parser_syslog: Fix syslog format detection [fluent/fluentd#2879 - buffer: Fix `available_buffer_space_ratio` calculation [fluent/fluentd#2882 - tls: Support CRLF based X.509 certificates [fluent/fluentd#2890 - msgpack_factory mixin: Fix performance penalty for deprecation log [fluent/fluentd#2903 ### [`v1.9.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v193---20200305) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.2...v1.9.3) ##### Enhancement - in_tail: Emit buffered lines as `unmatched_line` at shutdown phase when `emit_unmatched_lines true` [fluent/fluentd#2837 - Specify directory mode explicitly [fluent/fluentd#2827 - server helper: Change SSLError log level to warn in accept [fluent/fluentd#2861 - Refactor code [fluent/fluentd#2829 ##### Bug fix - buffer: Add seq to metadata that it can be unique [fluent/fluentd#2824 - buffer: Use `Tempfile` as binmode for decompression [fluent/fluentd#2847 ##### Misc - Add `.idea` to git ignore file [fluent/fluentd#2834 - appveyor: Fix tests [fluent/fluentd#2853 - Update pem for test [fluent/fluentd#2839 ### [`v1.9.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v192---20200213) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.1...v1.9.2) ##### Enhancement - in_tail: Add `pos_file_compaction_interval` parameter for auto compaction [fluent/fluentd#2805 - command: Use given encoding when RUBYOPT has `-E` [fluent/fluentd#2814 ##### Bug fix - command: Accept RUBYOPT with two or more options [fluent/fluentd#2807 - command: Fix infinite loop bug when RUBYOPT is invalid [fluent/fluentd#2813 - log: serverengine's log should be formatted with the same format of fluentd [fluent/fluentd#2812 - in_http: Fix `NoMethodError` when `OPTIONS` request doesn't have 'Origin' header [fluent/fluentd#2823 - parser_syslog: Improved for parsing RFC5424 structured data in `parser_syslog` [fluent/fluentd#2816 ### [`v1.9.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v191---20200131) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.0...v1.9.1) ##### Enhancement - http_server helper: Support HTTPS [fluent/fluentd#2787 - in_tail: Add `path_delimiter` to split with any char [fluent/fluentd#2796 - in_tail: Remove an entry from PositionaFile when it is unwatched [fluent/fluentd#2803 - out_http: Add warning for `retryable_response_code` [fluent/fluentd#2809 - parser_syslog: Add multiline RFC5424 support [fluent/fluentd#2767 - Add TLS module to unify TLS related code [fluent/fluentd#2802 ##### Bug fix - output: Add `EncodingError` to unrecoverable errors [fluent/fluentd#2808 - tls: Fix TLS version handling in secure mode [fluent/fluentd#2802 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…rafana#10818) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [fluentd](https://www.fluentd.org/) ([source](https://togithub.com/fluent/fluentd)) | `'1.9.0'` -> `'1.14.2'` | [![age](https://developer.mend.io/api/mc/badges/age/rubygems/fluentd/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/rubygems/fluentd/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/rubygems/fluentd/'1.9.0'/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/rubygems/fluentd/'1.9.0'/1.14.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. ### GitHub Vulnerability Alerts #### [CVE-2021-41186](https://togithub.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662) ### Impact parser_apache2 plugin in Fluentd v0.14.14 to v1.14.1 suffers from a regular expression denial of service (ReDoS) vulnerability. A broken apache log with a certain pattern of string can spend too much time in a regular expression, resulting in the potential for a DoS attack. ### Patches v1.14.2 ### Workarounds Either of the following: * Don't use parser_apache2 for parsing logs which cannot guarantee generated by Apache. * Put patched version of parser_apache2.rb into /etc/fluent/plugin directory (or any other directories specified by the environment variable `FLUENT_PLUGIN` or `--plugin` option of fluentd). ### References * [CVE-2021-41186](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41186) * [GHSA-hwhf-64mh-r662](https://togithub.com/fluent/fluentd/security/advisories/GHSA-hwhf-64mh-r662) * [GHSL-2021-102](https://securitylab.github.com/advisories/GHSL-2021-102-fluent-fluentd/) * https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1142 --- ### Release Notes <details> <summary>fluent/fluentd (fluentd)</summary> ### [`v1.14.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1142---20211029) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.14.1...v1.14.2) IMPORTANT: This release contain the fix for CVE-2021-41186 - ReDoS vulnerability in `parser_apache2`. This vulnerability is affected from Fluentd v0.14.14 to v1.14.1. We recommend to upgrade Fluentd to v1.14.2 or use patched version of `parser_apache2` plugin. ##### Enhancement - fluent-cat: Add `--event-time` option to send specified event time for testing. [fluent/fluentd#3528 ##### Bug fix - Fixed to generate correct epoch timestamp even after switching Daylight Saving Time [fluent/fluentd#3524 - Fixed ReDoS vulnerability in parser_apache2. This vulnerability is caused by a certain pattern of a broken apache log. ### [`v1.14.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1141---20210929) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.14.0...v1.14.1) ##### Enhancement - in_tail: Added file related metrics. These metrics should be collected same as fluent-bit's in_ta[fluent/fluentd#3504 - out_forward: Changed to use metrics mechanism for node statistics [fluent/fluentd#3506 ##### Bug fix - in_tail: Fixed a crash bug that it raise undefined method of eof? error. This error may happen only when `read_bytes_limit_per_second` was specified. [fluent/fluentd#3500 - out_forward: Fixed a bug that node statistics information is not included correctly. [fluent/fluentd#3503 - Fixed a error when using `@include` directive It was occurred when http/https scheme URI is used in `@include` directive with Ruby 3. [fluent/fluentd#3517 - out_copy: Fixed to suppress a wrong warning for `ignore_if_prev_success` It didn't work even if a user set [fluent/fluentd#3515 - Fixed not to output nanoseconds field of next retry time in warning log Then, inappropriate labels in log are also fixed. (retry_time -> retry_times, next_retry_seconds -> next_retr[fluent/fluentd#3518 ### [`v1.14.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1140---20210830) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.3...v1.14.0) ##### Enhancement - Added `enable_input_metrics`, `enable_size_metrics` system configuration parameter This feature might need to pay higher CPU cost, so input event metrics features are disabled by default. These features are also enabled by `--enable-input-metrics`,`--enable-size-metrics` command line opti[fluent/fluentd#3440 - Added reserved word `@ROOT` for getting root router. This is incompatible change. Do not use `@ROOT` for label name. [fluent/fluentd#3358 - in_syslog: Added `send_keepalive_packet` option [fluent/fluentd#3474 - in_http: Added `cors_allow_credentials` option. This option tells browsers whether to expose the response to frontend when the credentials mode is "in[fluent/fluentd#3481 ##### Bug fix - in_tail: Fixed a bug that deleted paths are not removed from pos file by file compaction at start[fluent/fluentd#3467 - in_tail: Revived a warning message of retrying unaccessible file [fluent/fluentd#3478 - TLSServer: Fixed a crash bug on logging peer host name errors [fluent/fluentd#3483 ##### Misc - Added metrics plugin mechanism The implementations is changed to use metrics plugin. In the future, 3rd party plugin will be able to handle these m[fluent/fluentd#3471 ### [`v1.13.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1133---20210727) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.2...v1.13.3) ##### Bug fix - in_tail: Care DeletePending state on Windows [fluent/fluentd#3457 - in_tail: Fix some pos_file bugs. Avoid deleting pos_file entries unexpectedly when both `pos_file_compaction_interval` and `follow_inode` are enabled. Use `bytesize` instead of `size` for path length. [fluent/fluentd#3459 - in_tail: Fix detecting rotation twice on `follow_inode`. [fluent/fluentd#3466 ##### Misc - Remove needless spaces in a sample config file [fluent/fluentd#3456 ### [`v1.13.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1132---20210712) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.1...v1.13.2) ##### Enhancement - fluent-plugin-generate: Storage plugin was supported. [fluent/fluentd#3426 - parser_json: Added support to customize configuration of oj options. Use `FLUENT_OJ_OPTION_BIGDECIMAL_LOAD`, `FLUENT_OJ_OPTION_MAX_NESTING`, `FLUENT_OJ_OPTION_MODE`, and `FLUENT_OJ_OPTION_USE_TO_JSON` environment variable to configure [fluent/fluentd#3315 ##### Bug fix - binlog_reader: Fixed a crash bug by missing "fluent/env" dependency. [fluent/fluentd#3443 - Fixed a crash bug on outputting log at the early stage when parsing config file. This is a regression since v1.13.0. If you use invalid '@&grafana#8203;' prefix parameter, remove it as [fluent/fluentd#3451 - in_tail: Fixed a bug that when rotation is occurred, remaining lines will be discarded if the throttling feature is enabl[fluent/fluentd#3390 - fluent-plugin-generate: Fixed a crash bug during gemspec generation. It was unexpectedly introduced by [#&grafana#8203;3305](https://togithub.com/fluent/fluentd/issues/3305), thus this bug was a regression since 1.12[fluent/fluentd#3444 ##### Misc - Fixed the runtime dependency version of http_parse.rb to 0.7.0. It was fixed because false positive detection is occurred frequently by security scanning[fluent/fluentd#3450 ### [`v1.13.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1131---20210625) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.13.0...v1.13.1) ##### Bug fix - out_forward: Fixed a race condition on handshake It's caused by using a same unpacker from multiple threa[fluent/fluentd#3405 - in_tail: Fixed to remove too much verbose debugging logs It was unexpectedly introduced by [#&grafana#8203;3185](https://togithub.com/fluent/fluentd/issues/3185) log throttling feature. [fluent/fluentd#3418 - Fixed not to echo back the provides path as is on a 404 error There was a potential cross-site scripting vector even though it is quite difficult to e[fluent/fluentd#3427 ##### Misc - Pretty print for Fluent::Config::Section has been supported for debugg[fluent/fluentd#3398 - CI: Dropped to run CI for Ruby 2.5 [fluent/fluentd#3412 ### [`v1.13.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1130---20210529) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.4...v1.13.0) ##### Enhancement - in_tail: Handle log throttling per file feature [fluent/fluentd#3185 - Extend to support service discovery manager in simpler way [fluent/fluentd#3299 - in_http: HTTP GET requests has been supported [fluent/fluentd#3373 - The log rotate settings in system configuration has been supported [fluent/fluentd#3352 ##### Bug fix - Fix to disable `trace_instruction` when `RubyVM::InstructionSequence` is available. It improves compatibility with `truffleruby` some extent. [fluent/fluentd#3376 - in_tail: Safely skip files which are used by another process on Windows. It improves exception handling about `ERROR_SHARING_VIOLATION` on Windows. [fluent/fluentd#3378 - fluent-cat: the issue resending secondary file in specific format has been fi[fluent/fluentd#3368 - in_tail: Shutdown immediately & safely even if reading huge files Note that `skip_refresh_on_startup` must be enabled. [fluent/fluentd#3380 ##### Misc - example: Change a path to backup_path in counter_server correctly [fluent/fluentd#3359 - README: Update link to community forum to discuss.fluentd.org [fluent/fluentd#3360 ### [`v1.12.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1124---20210526) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.3...v1.12.4) ##### Bug fix - in_tail: Fix a bug that refresh_watcher fails to handle file rotations [fluent/fluentd#3393 ### [`v1.12.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1123---20210423) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.2...v1.12.3) ##### Enhancement - plugin_helper: Allow TLS to use keep-alive socket option [fluent/fluentd#3308 ##### Bug fix - parser_csv, parser_syslog: Fix a naming conflict on parser_type [fluent/fluentd#3302 - in_tail: Fix incorrect error code & message on Windows [fluent/fluentd#3325 - in_tail: Fix a crash bug on catching a short-lived log [fluent/fluentd#3328 - storage_local: Fix position file corruption issue on concurrent gracefulReloads [fluent/fluentd#3335 - Fix incorrect warnings about ${chunk_id} with out_s3 [fluent/fluentd#3339 - TLS Server: Add peer information to error log message [fluent/fluentd#3330 ##### Misc - fluent-plugin-generate: add note about plugin name [fluent/fluentd#3303 - fluent-plugin-generate: Use same depended gem version with fluentd [fluent/fluentd#3305 - Fix some broken unit tests and improve CI's stability [fluent/fluentd#3304 - Permit to install with win32-service 2.2.0 on Windows [fluent/fluentd#3343 ### [`v1.12.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1122---20210329) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.1...v1.12.2) ##### Enhancement - out_copy: Add ignore_if_prev_successes [fluent/fluentd#3190 - Support multiple kind of timestamp format [fluent/fluentd#3252 - formatter_ltsv: suppress delimiters in output [fluent/fluentd#1666 ##### Bug fix - in_tail: Expect ENOENT during stat [fluent/fluentd#3275 - out_forward: Prevent transferring duplicate logs on restart [fluent/fluentd#3267 - in_tail: Handle to send rotated logs when mv is used for rotating [fluent/fluentd#3294 - fluent-plugin-config-format: Fill an uninitialized instance variable [fluent/fluentd#3297 - Fix MessagePackEventStream issue with Enumerable methods [fluent/fluentd#2116 ##### Misc - Add webrick to support Ruby 3.0 [fluent/fluentd#3257 - Suggest Discource instead of Google Groups [fluent/fluentd#3261 - Update MAINTAINERS.md [fluent/fluentd#3282 - Introduce DeepSource to check code quality [fluent/fluentd#3286 - Migrate to GitHub Actions and stabilize tests [fluent/fluentd#3266 ### [`v1.12.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1121---20210218) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.12.0...v1.12.1) ##### Enhancement - out_http: Add `headers_from_placeholders` parameter [fluent/fluentd#3241 - fluent-plugin-config-format: Add `--table` option to use markdown table [fluent/fluentd#3240 - Add `--disable-shared-socket`/`disable_shared_socket` to disable ServerEngine's shared socket setup [fluent/fluentd#3250 ##### Bug fix - ca_generate: Fix creating TLS certification files which include broken extensions [fluent/fluentd#3246 - test: Drop TLS 1.1 tests [fluent/fluentd#3256 - Remove old gem constraints to support Ruby 3 ##### Misc - Use GitHub Actions [fluent/fluentd#3233 ### [`v1.12.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1120---20210105) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.5...v1.12.0) ##### New feature - in_tail: Add `follow_inode` to support log rotation with wild card [fluent/fluentd#3182 - in_tail: Handle linux capability [fluent/fluentd#3155 - windows: Add win32 events alternative to unix signals [fluent/fluentd#3131 ##### Enhancement - buffer: Enable metadata comparison optimization on all platforms [fluent/fluentd#3095 - fluent-plugin-config-formatter: Handle `service_discovery` type [fluent/fluentd#3178 - in_http: Add `add_query_params` parameter to add query params to event record [fluent/fluentd#3197 - inject: Support `unixtime_micros` and `unixtime_nanos` in `time_type` [fluent/fluentd#3220 - Refactoring code [fluent/fluentd#3167 ##### Bug fix - output: Prevent retry.step from being called too many times in a short time [fluent/fluentd#3203 ### [`v1.11.5`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1115---20201106) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.4...v1.11.5) ##### Enhancement - formatter: Provide `newline` parameter to support `CRLF` [fluent/fluentd#3152 - out_http: adding support for intermediate certificates [fluent/fluentd#3146 - Update serverengine dependency to 2.2.2 or later ##### Bug fix - Fix a bug that windows service isn't stopped gracefuly [fluent/fluentd#3156 ### [`v1.11.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1114---20201013) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.3...v1.11.4) ##### Enhancement - inject: Support `unixtime_millis` in `time_type` parameter [fluent/fluentd#3145 ##### Bug fix - out_http: Fix broken data with `json_array true` [fluent/fluentd#3144 - output: Fix wrong logging issue for `${chunk_id}` [fluent/fluentd#3134 ### [`v1.11.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1113---20200930) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.2...v1.11.3) ##### Enhancement - in_exec: Add `connect_mode` parameter to read stderr [fluent/fluentd#3108 - parser_json: Improve the performance [fluent/fluentd#3109 - log: Add `ignore_same_log_interval` parameter [fluent/fluentd#3119 - Upgrade win32 gems [fluent/fluentd#3100 - Refactoring code [fluent/fluentd#3094 ##### Bug fix - buffer: Fix calculation of timekey stats [fluent/fluentd#3018 - buffer: fix binmode usage for prevent gc [fluent/fluentd#3138 ### [`v1.11.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1112---20200804) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.1...v1.11.2) ##### Enhancement - `in_dummy` renamed to `in_sample` [fluent/fluentd#3065 - Allow regular expression in filter/match directive [fluent/fluentd#3071 - Refactoring code [fluent/fluentd#3051 ##### Bug fix - buffer: Fix log message for `chunk_limit_records` case [fluent/fluentd#3079 - buffer: Fix timekey optimization for non-windows platform [fluent/fluentd#3092 - cert: Raise an error for broken certificate file [fluent/fluentd#3086 - cert: Set TLS ciphers list correcty on older OpenSSL [fluent/fluentd#3093 ### [`v1.11.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1111---20200622) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.11.0...v1.11.1) ##### Enhancement - in_http: Add `dump_error_log` parameter [fluent/fluentd#3035 - in_http: Improve time field handling [fluent/fluentd#3046 - Refactoring code [fluent/fluentd#3047 ##### Bug fix - in_tail: Use actual path instead of based pattern for ignore list [fluent/fluentd#3042 - child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout [fluent/fluentd#3044 ### [`v1.11.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1110---20200604) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.4...v1.11.0) ##### New feature - in_unix: Use v1 API [fluent/fluentd#2992 ##### Enhancement - parser_syslog: Support any `time_format` for RFC3164 string parser [fluent/fluentd#3014 - parser_syslog: Add new parser for RFC5424 [fluent/fluentd#3015 - Refactoring code [fluent/fluentd#3019 ##### Bug fix - in_gc_stat: Add `use_symbol_keys` parameter to emit string key record [fluent/fluentd#3008 ### [`v1.10.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1104---20200512) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.3...v1.10.4) ##### Enhancement - out_http: Support single json array payload [fluent/fluentd#2973 - Refactoring [fluent/fluentd#2988 ##### Bug fix - supervisor: Call `File.umask(0)` for standalone worker [fluent/fluentd#2987 - out_forward: Fix ZeroDivisionError issue with `weight 0` [fluent/fluentd#2989 ### [`v1.10.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1103---20200501) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.2...v1.10.3) ##### Enhancement - record_accessor: Add `set` method [fluent/fluentd#2977 - config: Ruby DSL format is deprecated [fluent/fluentd#2958 - Refactor code [fluent/fluentd#2961 ##### Bug fix - out_forward: Disable `linger_timeout` setting on Windows [fluent/fluentd#2959 - out_forward: Fix warning of service discovery manager when fluentd stops [fluent/fluentd#2974 ### [`v1.10.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1102---20200415) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.1...v1.10.2) ##### Enhancement - out_copy: Add plugin_id to log message [fluent/fluentd#2934 - socket: Allow cert chains in mutual auth [fluent/fluentd#2930 - system: Add ignore_repeated_log_interval parameter [fluent/fluentd#2937 - windows: Allow to launch fluentd from whitespace included path [fluent/fluentd#2920 - Refactor code [fluent/fluentd#2935 ##### Bug fix - in_syslog: Fix octet-counting mode bug [fluent/fluentd#2942 - out_forward: Create timer for purging obsolete sockets when keepalive_timeout is not set [fluent/fluentd#2943 - out_forward: Need authentication when sending tcp heartbeat with keepalive [fluent/fluentd#2945 - command: Fix fluent-debug start failure [fluent/fluentd#2948 - command: Fix regression of supervisor's worker and `--daemon` combo [fluent/fluentd#2950 ### [`v1.10.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1101---20200402) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.10.0...v1.10.1) ##### Enhancement - command: `--daemon` and `--no-supervisor` now work together [fluent/fluentd#2912 - Refactor code [fluent/fluentd#2913 ##### Bug fix - in_tail: `Fix pos_file_compaction_interval` parameter type [fluent/fluentd#2921 - in_tail: Fix seek position update after compaction [fluent/fluentd#2922 - parser_syslog: Fix regression in the `with_priority` and RFC5424 case [fluent/fluentd#2923 ##### Misc - Add document for security audit [fluent/fluentd#2911 ### [`v1.10.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1100---20200324) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.3...v1.10.0) ##### New feature - sd plugin: Add SRV record plugin [fluent/fluentd#2876 ##### Enhancement - server: Add `cert_verifier` parameter for TLS transport [fluent/fluentd#2888 - parser_syslog: Support customized time format [fluent/fluentd#2886 - in_dummy: Delete `suspend` parameter [fluent/fluentd#2897 - Refactor code [fluent/fluentd#2858 [fluent/fluentd#2899 [fluent/fluentd#2900 fluent/fluentd#2906 ##### Bug fix - out_forward: windows: Permit to specify `linger_timeout` [fluent/fluentd#2868 - parser_syslog: Fix syslog format detection [fluent/fluentd#2879 - buffer: Fix `available_buffer_space_ratio` calculation [fluent/fluentd#2882 - tls: Support CRLF based X.509 certificates [fluent/fluentd#2890 - msgpack_factory mixin: Fix performance penalty for deprecation log [fluent/fluentd#2903 ### [`v1.9.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v193---20200305) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.2...v1.9.3) ##### Enhancement - in_tail: Emit buffered lines as `unmatched_line` at shutdown phase when `emit_unmatched_lines true` [fluent/fluentd#2837 - Specify directory mode explicitly [fluent/fluentd#2827 - server helper: Change SSLError log level to warn in accept [fluent/fluentd#2861 - Refactor code [fluent/fluentd#2829 ##### Bug fix - buffer: Add seq to metadata that it can be unique [fluent/fluentd#2824 - buffer: Use `Tempfile` as binmode for decompression [fluent/fluentd#2847 ##### Misc - Add `.idea` to git ignore file [fluent/fluentd#2834 - appveyor: Fix tests [fluent/fluentd#2853 - Update pem for test [fluent/fluentd#2839 ### [`v1.9.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v192---20200213) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.1...v1.9.2) ##### Enhancement - in_tail: Add `pos_file_compaction_interval` parameter for auto compaction [fluent/fluentd#2805 - command: Use given encoding when RUBYOPT has `-E` [fluent/fluentd#2814 ##### Bug fix - command: Accept RUBYOPT with two or more options [fluent/fluentd#2807 - command: Fix infinite loop bug when RUBYOPT is invalid [fluent/fluentd#2813 - log: serverengine's log should be formatted with the same format of fluentd [fluent/fluentd#2812 - in_http: Fix `NoMethodError` when `OPTIONS` request doesn't have 'Origin' header [fluent/fluentd#2823 - parser_syslog: Improved for parsing RFC5424 structured data in `parser_syslog` [fluent/fluentd#2816 ### [`v1.9.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v191---20200131) [Compare Source](https://togithub.com/fluent/fluentd/compare/v1.9.0...v1.9.1) ##### Enhancement - http_server helper: Support HTTPS [fluent/fluentd#2787 - in_tail: Add `path_delimiter` to split with any char [fluent/fluentd#2796 - in_tail: Remove an entry from PositionaFile when it is unwatched [fluent/fluentd#2803 - out_http: Add warning for `retryable_response_code` [fluent/fluentd#2809 - parser_syslog: Add multiline RFC5424 support [fluent/fluentd#2767 - Add TLS module to unify TLS related code [fluent/fluentd#2802 ##### Bug fix - output: Add `EncodingError` to unrecoverable errors [fluent/fluentd#2808 - tls: Fix TLS version handling in secure mode [fluent/fluentd#2802 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

conorevans marked this pull request as draft April 4, 2021 12:55

conorevans commented Apr 4, 2021

View reviewed changes

lib/fluent/config/types.rb Outdated Show resolved Hide resolved

lib/fluent/config/types.rb Outdated Show resolved Hide resolved

lib/fluent/env_utils.rb Outdated Show resolved Hide resolved

lib/fluent/env_utils.rb Outdated Show resolved Hide resolved

conorevans commented Apr 4, 2021

View reviewed changes

lib/fluent/config/types.rb Outdated Show resolved Hide resolved

ashie added this to the v1.13 milestone Apr 23, 2021

ashie requested changes May 13, 2021

View reviewed changes

kenhys reviewed May 19, 2021

View reviewed changes

kenhys removed this from the v1.13 milestone May 31, 2021

conorevans changed the title ~~Draft: Allow for configuration of OJ options~~ Allow for configuration of OJ options Jun 20, 2021

conorevans requested review from ashie, cosmo0920 and kenhys June 20, 2021 16:02

ashie marked this pull request as ready for review July 5, 2021 07:06

ashie reviewed Jul 6, 2021

View reviewed changes

test/test_oj_options.rb Outdated Show resolved Hide resolved

conorevans added 6 commits July 8, 2021 11:29

Allow for setting of OJ Options

7f8afda

Signed-off-by: Conor Evans <coevans@tcd.ie>

It is compat, and not concat

5e35c60

Signed-off-by: Conor Evans <coevans@tcd.ie>

newline

2b16490

Signed-off-by: Conor Evans <coevans@tcd.ie>

refactor: address PR comments.. 1) remove EnvUtils module and move Oj…

68ed3b5

…Options directly under Fluent, 2) use strict=true in reformatted value and return the int parsing back to normal as that is a breaking change Signed-off-by: Conor Evans <coevans@tcd.ie>

add types tests, update symbol type to return nil for '' too

fb096df

Signed-off-by: Conor Evans <coevans@tcd.ie>

fix typo: concat should be compat

30044f4

Signed-off-by: Conor Evans <coevans@tcd.ie>

ashie added 5 commits July 8, 2021 11:31

test: Restore ENV after testing OjOptions

d9968a3

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

OjOptions: Remove needless prefix of constants

54e0cc2

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

OjOptions: Remove needless instantiation

beae9f7

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

Set Oj.default_options only once when fluentd is started

3217fe4

It shouldn't be set by each plugins, because later loaded plugins will overwrites it. Signed-off-by: Takuro Ashie <ashie@clear-code.com>

OjOptions: Don't require needless files

05c3cf2

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

ashie approved these changes Jul 8, 2021

View reviewed changes

kenhys approved these changes Jul 8, 2021

View reviewed changes

kenhys added this to the v1.13.2 milestone Jul 9, 2021

kenhys mentioned this pull request Jul 9, 2021

v1.13.2 #3447

Merged

ashie merged commit dee18ea into fluent:master Jul 9, 2021

ashie mentioned this pull request Jul 14, 2021

Max nesting level for json parser #3311

Open

ashie mentioned this pull request Oct 27, 2022

Remove meaningless oj options #3929

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow for configuration of OJ options #3315

Allow for configuration of OJ options #3315

conorevans commented Apr 4, 2021

conorevans commented Apr 4, 2021

ashie left a comment

kenhys commented May 19, 2021

kenhys May 19, 2021

conorevans Jun 20, 2021

kenhys Jun 21, 2021

conorevans commented May 19, 2021

conorevans commented May 30, 2021

kenhys commented Jun 18, 2021

conorevans commented Jun 20, 2021

kenhys commented Jun 21, 2021

ashie commented Jul 5, 2021 •

edited

ashie commented Jul 5, 2021

ashie commented Jul 5, 2021

ashie commented Jul 6, 2021

ashie commented Jul 8, 2021

kenhys commented Jul 8, 2021

kenhys left a comment

ashie commented Jul 9, 2021

conorevans commented Jul 9, 2021

Allow for configuration of OJ options #3315

Allow for configuration of OJ options #3315

Conversation

conorevans commented Apr 4, 2021

conorevans commented Apr 4, 2021

ashie left a comment

Choose a reason for hiding this comment

kenhys commented May 19, 2021

kenhys May 19, 2021

Choose a reason for hiding this comment

conorevans Jun 20, 2021

Choose a reason for hiding this comment

kenhys Jun 21, 2021

Choose a reason for hiding this comment

conorevans commented May 19, 2021

conorevans commented May 30, 2021

kenhys commented Jun 18, 2021

conorevans commented Jun 20, 2021

kenhys commented Jun 21, 2021

ashie commented Jul 5, 2021 • edited

ashie commented Jul 5, 2021

ashie commented Jul 5, 2021

ashie commented Jul 6, 2021

ashie commented Jul 8, 2021

kenhys commented Jul 8, 2021

kenhys left a comment

Choose a reason for hiding this comment

ashie commented Jul 9, 2021

conorevans commented Jul 9, 2021

ashie commented Jul 5, 2021 •

edited