Skip to content

Add support of json logs for k8s containers #328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
repeatedly merged 4 commits into from
Jul 3, 2019
Merged

Add support of json logs for k8s containers #328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
028d2e6
Add support of json logs for k8s containers
alikhil Jun 17, 2019
22ae70d
Change version of json-in-json plugin
alikhil Jun 17, 2019
7aec1a7
Set container tail parser type from env var
alikhil Jun 24, 2019
623d3e9
Change default format for non v1
alikhil Jul 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docker-image/v1.4/debian-cloudwatch/Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.3.0"
gem 'fluent-plugin-json-in-json-2', ">= 1.0.2"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.1.0"
gem "aws-sdk-cloudwatchlogs", "~> 1.0"
gem "fluent-plugin-cloudwatch-logs", "~> 0.7.3"
Expand Down
16 changes: 10 additions & 6 deletions docker-image/v1.4/debian-cloudwatch/Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ GEM
addressable (2.6.0)
public_suffix (>= 2.0.2, < 4.0)
aws-eventstream (1.0.3)
aws-partitions (1.155.0)
aws-sdk-cloudwatchlogs (1.17.0)
aws-sdk-core (~> 3, >= 3.48.2)
aws-partitions (1.175.0)
aws-sdk-cloudwatchlogs (1.21.0)
aws-sdk-core (~> 3, >= 3.53.0)
aws-sigv4 (~> 1.1)
aws-sdk-core (3.48.6)
aws-sdk-core (3.55.0)
aws-eventstream (~> 1.0, >= 1.0.2)
aws-partitions (~> 1.0)
aws-sigv4 (~> 1.1)
Expand All @@ -25,7 +25,7 @@ GEM
dig_rb (1.0.1)
domain_name (0.5.20180417)
unf (>= 0.0.5, < 1.0.0)
ffi (1.10.0)
ffi (1.11.1)
fluent-config-regexp-type (1.0.0)
fluentd (> 1.0.0, < 2)
fluent-plugin-cloudwatch-logs (0.7.3)
Expand All @@ -35,6 +35,9 @@ GEM
fluentd (>= 0.14.0, < 2)
fluent-plugin-grok-parser (2.5.1)
fluentd (>= 0.14.6, < 2)
fluent-plugin-json-in-json-2 (1.0.2)
fluentd (>= 0.14.0, < 2)
yajl-ruby (~> 1.0)
fluent-plugin-kubernetes_metadata_filter (2.1.6)
fluentd (>= 0.14.0, < 2)
kubeclient (~> 1.1.4)
Expand Down Expand Up @@ -88,7 +91,7 @@ GEM
oj (3.5.1)
prometheus-client (0.9.0)
quantile (~> 0.2.1)
public_suffix (3.0.3)
public_suffix (3.1.0)
quantile (0.2.1)
recursive-open-struct (1.0.0)
rest-client (2.0.2)
Expand Down Expand Up @@ -120,6 +123,7 @@ DEPENDENCIES
fluent-plugin-cloudwatch-logs (~> 0.7.3)
fluent-plugin-concat (~> 2.3.0)
fluent-plugin-grok-parser (~> 2.5.0)
fluent-plugin-json-in-json-2 (>= 1.0.2)
fluent-plugin-kubernetes_metadata_filter (~> 2.1.4)
fluent-plugin-multi-format-parser (~> 1.0.0)
fluent-plugin-prometheus (~> 1.3.0)
Expand Down
2 changes: 1 addition & 1 deletion docker-image/v1.4/debian-cloudwatch/conf/kubernetes.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
tag kubernetes.*
read_from_head true
<parse>
@type json
@type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
Expand Down
1 change: 1 addition & 0 deletions docker-image/v1.4/debian-elasticsearch/Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.3.0"
gem 'fluent-plugin-json-in-json-2', ">= 1.0.2"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.1.0"
gem "fluent-plugin-elasticsearch", "~> 3.4.2"
gem "fluent-plugin-kubernetes_metadata_filter", "~> 2.1.4"
Expand Down
20 changes: 12 additions & 8 deletions docker-image/v1.4/debian-elasticsearch/Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,18 @@ GEM
dig_rb (1.0.1)
domain_name (0.5.20180417)
unf (>= 0.0.5, < 1.0.0)
elasticsearch (7.0.0)
elasticsearch-api (= 7.0.0)
elasticsearch-transport (= 7.0.0)
elasticsearch-api (7.0.0)
elasticsearch (7.1.0)
elasticsearch-api (= 7.1.0)
elasticsearch-transport (= 7.1.0)
elasticsearch-api (7.1.0)
multi_json
elasticsearch-transport (7.0.0)
elasticsearch-transport (7.1.0)
faraday
multi_json
excon (0.64.0)
faraday (0.15.4)
multipart-post (>= 1.2, < 3)
ffi (1.10.0)
ffi (1.11.1)
fluent-config-regexp-type (1.0.0)
fluentd (> 1.0.0, < 2)
fluent-plugin-concat (2.3.0)
Expand All @@ -35,6 +35,9 @@ GEM
fluentd (>= 0.14.22)
fluent-plugin-grok-parser (2.5.1)
fluentd (>= 0.14.6, < 2)
fluent-plugin-json-in-json-2 (1.0.2)
fluentd (>= 0.14.0, < 2)
yajl-ruby (~> 1.0)
fluent-plugin-kubernetes_metadata_filter (2.1.6)
fluentd (>= 0.14.0, < 2)
kubeclient (~> 1.1.4)
Expand Down Expand Up @@ -84,12 +87,12 @@ GEM
minitest (5.11.3)
msgpack (1.2.10)
multi_json (1.13.1)
multipart-post (2.0.0)
multipart-post (2.1.1)
netrc (0.11.0)
oj (3.5.1)
prometheus-client (0.9.0)
quantile (~> 0.2.1)
public_suffix (3.0.3)
public_suffix (3.1.0)
quantile (0.2.1)
recursive-open-struct (1.0.0)
rest-client (2.0.2)
Expand Down Expand Up @@ -120,6 +123,7 @@ DEPENDENCIES
fluent-plugin-concat (~> 2.3.0)
fluent-plugin-elasticsearch (~> 3.4.2)
fluent-plugin-grok-parser (~> 2.5.0)
fluent-plugin-json-in-json-2 (>= 1.0.2)
fluent-plugin-kubernetes_metadata_filter (~> 2.1.4)
fluent-plugin-multi-format-parser (~> 1.0.0)
fluent-plugin-prometheus (~> 1.3.0)
Expand Down
2 changes: 1 addition & 1 deletion docker-image/v1.4/debian-elasticsearch/conf/kubernetes.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
tag kubernetes.*
read_from_head true
<parse>
@type json
@type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
Expand Down
1 change: 1 addition & 0 deletions docker-image/v1.4/debian-forward/Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.3.0"
gem 'fluent-plugin-json-in-json-2', ">= 1.0.2"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.1.0"
gem "fluent-plugin-kubernetes_metadata_filter", "~> 2.1.4"
gem "ffi"
Expand Down
8 changes: 6 additions & 2 deletions docker-image/v1.4/debian-forward/Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,16 @@ GEM
dig_rb (1.0.1)
domain_name (0.5.20180417)
unf (>= 0.0.5, < 1.0.0)
ffi (1.10.0)
ffi (1.11.1)
fluent-config-regexp-type (1.0.0)
fluentd (> 1.0.0, < 2)
fluent-plugin-concat (2.3.0)
fluentd (>= 0.14.0, < 2)
fluent-plugin-grok-parser (2.5.1)
fluentd (>= 0.14.6, < 2)
fluent-plugin-json-in-json-2 (1.0.2)
fluentd (>= 0.14.0, < 2)
yajl-ruby (~> 1.0)
fluent-plugin-kubernetes_metadata_filter (2.1.6)
fluentd (>= 0.14.0, < 2)
kubeclient (~> 1.1.4)
Expand Down Expand Up @@ -72,7 +75,7 @@ GEM
oj (3.5.1)
prometheus-client (0.9.0)
quantile (~> 0.2.1)
public_suffix (3.0.3)
public_suffix (3.1.0)
quantile (0.2.1)
recursive-open-struct (1.0.0)
rest-client (2.0.2)
Expand Down Expand Up @@ -102,6 +105,7 @@ DEPENDENCIES
ffi
fluent-plugin-concat (~> 2.3.0)
fluent-plugin-grok-parser (~> 2.5.0)
fluent-plugin-json-in-json-2 (>= 1.0.2)
fluent-plugin-kubernetes_metadata_filter (~> 2.1.4)
fluent-plugin-multi-format-parser (~> 1.0.0)
fluent-plugin-prometheus (~> 1.3.0)
Expand Down
2 changes: 1 addition & 1 deletion docker-image/v1.4/debian-forward/conf/kubernetes.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
tag kubernetes.*
read_from_head true
<parse>
@type json
@type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
Expand Down
1 change: 1 addition & 0 deletions docker-image/v1.4/debian-gcs/Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.3.0"
gem 'fluent-plugin-json-in-json-2', ">= 1.0.2"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.1.0"
gem "fluent-plugin-gcs", "0.4.0.beta1"
gem "fluent-plugin-kubernetes_metadata_filter", "~> 2.1.4"
Expand Down
14 changes: 9 additions & 5 deletions docker-image/v1.4/debian-gcs/Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ GEM
unf (>= 0.0.5, < 1.0.0)
faraday (0.15.4)
multipart-post (>= 1.2, < 3)
ffi (1.10.0)
ffi (1.11.1)
fluent-config-regexp-type (1.0.0)
fluentd (> 1.0.0, < 2)
fluent-plugin-concat (2.3.0)
Expand All @@ -28,6 +28,9 @@ GEM
google-cloud-storage (~> 1.1.0)
fluent-plugin-grok-parser (2.5.1)
fluentd (>= 0.14.6, < 2)
fluent-plugin-json-in-json-2 (1.0.2)
fluentd (>= 0.14.0, < 2)
yajl-ruby (~> 1.0)
fluent-plugin-kubernetes_metadata_filter (2.1.6)
fluentd (>= 0.14.0, < 2)
kubeclient (~> 1.1.4)
Expand Down Expand Up @@ -63,7 +66,7 @@ GEM
retriable (>= 2.0, < 4.0)
google-cloud-core (1.3.0)
google-cloud-env (~> 1.0)
google-cloud-env (1.0.5)
google-cloud-env (1.1.0)
faraday (~> 0.11)
google-cloud-storage (1.1.0)
digest-crc (~> 0.4)
Expand All @@ -88,7 +91,7 @@ GEM
httpclient (2.8.3)
i18n (1.6.0)
concurrent-ruby (~> 1.0)
jwt (2.1.0)
jwt (2.2.1)
kubeclient (1.1.4)
activesupport
http (= 0.9.8)
Expand All @@ -102,13 +105,13 @@ GEM
minitest (5.11.3)
msgpack (1.2.10)
multi_json (1.13.1)
multipart-post (2.0.0)
multipart-post (2.1.1)
netrc (0.11.0)
oj (3.5.1)
os (1.0.1)
prometheus-client (0.9.0)
quantile (~> 0.2.1)
public_suffix (3.0.3)
public_suffix (3.1.0)
quantile (0.2.1)
recursive-open-struct (1.0.0)
representable (3.0.4)
Expand Down Expand Up @@ -150,6 +153,7 @@ DEPENDENCIES
fluent-plugin-concat (~> 2.3.0)
fluent-plugin-gcs (= 0.4.0.beta1)
fluent-plugin-grok-parser (~> 2.5.0)
fluent-plugin-json-in-json-2 (>= 1.0.2)
fluent-plugin-kubernetes_metadata_filter (~> 2.1.4)
fluent-plugin-multi-format-parser (~> 1.0.0)
fluent-plugin-prometheus (~> 1.3.0)
Expand Down
2 changes: 1 addition & 1 deletion docker-image/v1.4/debian-gcs/conf/kubernetes.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
tag kubernetes.*
read_from_head true
<parse>
@type json
@type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
Expand Down
1 change: 1 addition & 0 deletions docker-image/v1.4/debian-graylog/Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.3.0"
gem 'fluent-plugin-json-in-json-2', ">= 1.0.2"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.1.0"
gem "gelf", "3.0.0"
gem "fluent-plugin-gelf-hs", "~> 1.0.7"
Expand Down
8 changes: 6 additions & 2 deletions docker-image/v1.4/debian-graylog/Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ GEM
dig_rb (1.0.1)
domain_name (0.5.20180417)
unf (>= 0.0.5, < 1.0.0)
ffi (1.10.0)
ffi (1.11.1)
fluent-config-regexp-type (1.0.0)
fluentd (> 1.0.0, < 2)
fluent-plugin-concat (2.3.0)
Expand All @@ -23,6 +23,9 @@ GEM
gelf (>= 2.0.0)
fluent-plugin-grok-parser (2.5.1)
fluentd (>= 0.14.6, < 2)
fluent-plugin-json-in-json-2 (1.0.2)
fluentd (>= 0.14.0, < 2)
yajl-ruby (~> 1.0)
fluent-plugin-kubernetes_metadata_filter (2.1.6)
fluentd (>= 0.14.0, < 2)
kubeclient (~> 1.1.4)
Expand Down Expand Up @@ -80,7 +83,7 @@ GEM
oj (3.5.1)
prometheus-client (0.9.0)
quantile (~> 0.2.1)
public_suffix (3.0.3)
public_suffix (3.1.0)
quantile (0.2.1)
recursive-open-struct (1.0.0)
rest-client (2.0.2)
Expand Down Expand Up @@ -111,6 +114,7 @@ DEPENDENCIES
fluent-plugin-concat (~> 2.3.0)
fluent-plugin-gelf-hs (~> 1.0.7)
fluent-plugin-grok-parser (~> 2.5.0)
fluent-plugin-json-in-json-2 (>= 1.0.2)
fluent-plugin-kubernetes_metadata_filter (~> 2.1.4)
fluent-plugin-multi-format-parser (~> 1.0.0)
fluent-plugin-prometheus (~> 1.3.0)
Expand Down
2 changes: 1 addition & 1 deletion docker-image/v1.4/debian-graylog/conf/kubernetes.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
tag kubernetes.*
read_from_head true
<parse>
@type json
@type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
Expand Down
1 change: 1 addition & 0 deletions docker-image/v1.4/debian-kafka/Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.3.0"
gem 'fluent-plugin-json-in-json-2', ">= 1.0.2"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.1.0"
gem "fluent-plugin-kafka", "~> 0.7.9"
gem "zookeeper", "~> 1.4.11"
Expand Down
8 changes: 6 additions & 2 deletions docker-image/v1.4/debian-kafka/Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,16 @@ GEM
dig_rb (1.0.1)
domain_name (0.5.20180417)
unf (>= 0.0.5, < 1.0.0)
ffi (1.10.0)
ffi (1.11.1)
fluent-config-regexp-type (1.0.0)
fluentd (> 1.0.0, < 2)
fluent-plugin-concat (2.3.0)
fluentd (>= 0.14.0, < 2)
fluent-plugin-grok-parser (2.5.1)
fluentd (>= 0.14.6, < 2)
fluent-plugin-json-in-json-2 (1.0.2)
fluentd (>= 0.14.0, < 2)
yajl-ruby (~> 1.0)
fluent-plugin-kafka (0.7.9)
fluentd (>= 0.10.58, < 2)
ltsv
Expand Down Expand Up @@ -77,7 +80,7 @@ GEM
oj (3.5.1)
prometheus-client (0.9.0)
quantile (~> 0.2.1)
public_suffix (3.0.3)
public_suffix (3.1.0)
quantile (0.2.1)
recursive-open-struct (1.0.0)
rest-client (2.0.2)
Expand Down Expand Up @@ -110,6 +113,7 @@ DEPENDENCIES
ffi
fluent-plugin-concat (~> 2.3.0)
fluent-plugin-grok-parser (~> 2.5.0)
fluent-plugin-json-in-json-2 (>= 1.0.2)
fluent-plugin-kafka (~> 0.7.9)
fluent-plugin-kubernetes_metadata_filter (~> 2.1.4)
fluent-plugin-multi-format-parser (~> 1.0.0)
Expand Down
2 changes: 1 addition & 1 deletion docker-image/v1.4/debian-kafka/conf/kubernetes.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
tag kubernetes.*
read_from_head true
<parse>
@type json
@type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
Expand Down
1 change: 1 addition & 0 deletions docker-image/v1.4/debian-kinesis/Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.3.0"
gem 'fluent-plugin-json-in-json-2', ">= 1.0.2"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.1.0"
gem "fluent-plugin-kinesis", "~> 2.1.1"
gem "fluent-plugin-kubernetes_metadata_filter", "~> 2.1.4"
Expand Down
Loading