Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS Elasticsearch image #317

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ IMAGE_NAME := fluent/fluentd-kubernetes
ALL_IMAGES := \
v1.7/debian-elasticsearch7:v1.7.3-debian-elasticsearch7-1.0,v1.7-debian-elasticsearch7-1,v1-debian-elasticsearch \
v1.7/debian-elasticsearch6:v1.7.3-debian-elasticsearch6-1.0,v1.7-debian-elasticsearch6-1 \
v1.7/debian-awselasticsearch:v1.7.2-debian-awselasticsearch-1.0,v1.7-debian-awselasticsearch-1 \
v1.7/debian-loggly:v1.7.3-debian-loggly-1.0,v1.7-debian-loggly-1 \
v1.7/debian-logentries:v1.7.3-debian-logentries-1.0,v1.7-debian-logentries-1 \
v1.7/debian-cloudwatch:v1.7.3-debian-cloudwatch-1.0,v1.7-debian-cloudwatch-1 \
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ See also dockerhub tags page: https://hub.docker.com/r/fluent/fluentd-kubernetes

- `v1.7.3-debian-elasticsearch7-1.0,v1.7-debian-elasticsearch7-1,v1-debian-elasticsearch` [docker-image/v1.7/debian-elasticsearch7/Dockerfile](docker-image/v1.7/debian-elasticsearch7/Dockerfile)
- `v1.7.3-debian-elasticsearch6-1.0,v1.7-debian-elasticsearch6-1` [docker-image/v1.7/debian-elasticsearch6/Dockerfile](docker-image/v1.7/debian-elasticsearch6/Dockerfile)
- `v1.7.2-debian-awselasticsearch-1.0,v1.7-debian-awselasticsearch-1` [docker-image/v1.7/debian-awselasticsearch/Dockerfile](docker-image/v1.7/debian-awselasticsearch/Dockerfile)
- `v1.7.3-debian-loggly-1.0,v1.7-debian-loggly-1` [docker-image/v1.7/debian-loggly/Dockerfile](docker-image/v1.7/debian-loggly/Dockerfile)
- `v1.7.3-debian-logentries-1.0,v1.7-debian-logentries-1` [docker-image/v1.7/debian-logentries/Dockerfile](docker-image/v1.7/debian-logentries/Dockerfile)
- `v1.7.3-debian-cloudwatch-1.0,v1.7-debian-cloudwatch-1` [docker-image/v1.7/debian-cloudwatch/Dockerfile](docker-image/v1.7/debian-cloudwatch/Dockerfile)
Expand Down
1 change: 1 addition & 0 deletions docker-image/v1.4/debian-awselasticsearch/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**/*.gitkeep
49 changes: 49 additions & 0 deletions docker-image/v1.4/debian-awselasticsearch/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/Dockerfile.erb

FROM fluent/fluentd:v1.4.2-debian-2.0

LABEL maintainer="Eduardo Silva <eduardo@treasure-data.com>"
USER root
WORKDIR /home/fluent
ENV PATH /fluentd/vendor/bundle/ruby/2.6.0/bin:$PATH
ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.6.0
ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.6.0
# skip runtime bundler installation
ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1

COPY Gemfile* /fluentd/
RUN buildDeps="sudo make gcc g++ libc-dev libffi-dev" \
&& apt-get update \
&& apt-get upgrade -y \
&& apt-get install \
-y --no-install-recommends \
$buildDeps net-tools \
&& gem install bundler --version 1.16.2 \
&& bundle config silence_root_warning true \
&& bundle install --gemfile=/fluentd/Gemfile --path=/fluentd/vendor/bundle \
&& SUDO_FORCE_REMOVE=yes \
apt-get purge -y --auto-remove \
-o APT::AutoRemove::RecommendsImportant=false \
$buildDeps \
&& rm -rf /var/lib/apt/lists/* \
&& gem sources --clear-all \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

# Copy configuration files
COPY ./conf/fluent.conf /fluentd/etc/
COPY ./conf/systemd.conf /fluentd/etc/
COPY ./conf/kubernetes.conf /fluentd/etc/
COPY ./conf/prometheus.conf /fluentd/etc/
RUN touch /fluentd/etc/disable.conf

# Copy plugins
COPY plugins /fluentd/plugins/
COPY entrypoint.sh /fluentd/entrypoint.sh

# Environment variables
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"

# Overwrite ENTRYPOINT to run fluentd as root for /var/log / /var/lib
ENTRYPOINT ["tini", "--", "/fluentd/entrypoint.sh"]
16 changes: 16 additions & 0 deletions docker-image/v1.4/debian-awselasticsearch/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/Gemfile.erb

source "https://rubygems.org"

gem "fluentd", "1.4.2"
gem "oj", "3.5.1"
gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.3.0"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.1.0"
gem "fluent-plugin-aws-elasticsearch-service"
gem "fluent-plugin-kubernetes_metadata_filter", "~> 2.1.4"
gem "ffi"
gem "fluent-plugin-systemd", "~> 1.0.1"
150 changes: 150 additions & 0 deletions docker-image/v1.4/debian-awselasticsearch/Gemfile.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
GEM
remote: https://rubygems.org/
specs:
activesupport (5.2.3)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (>= 0.7, < 2)
minitest (~> 5.1)
tzinfo (~> 1.1)
addressable (2.6.0)
public_suffix (>= 2.0.2, < 4.0)
aws-eventstream (1.0.3)
aws-partitions (1.167.0)
aws-sdk-core (3.53.1)
aws-eventstream (~> 1.0, >= 1.0.2)
aws-partitions (~> 1.0)
aws-sigv4 (~> 1.1)
jmespath (~> 1.0)
aws-sigv4 (1.1.0)
aws-eventstream (~> 1.0, >= 1.0.2)
concurrent-ruby (1.1.5)
cool.io (1.5.4)
dig_rb (1.0.1)
domain_name (0.5.20180417)
unf (>= 0.0.5, < 1.0.0)
elasticsearch (7.1.0)
elasticsearch-api (= 7.1.0)
elasticsearch-transport (= 7.1.0)
elasticsearch-api (7.1.0)
multi_json
elasticsearch-transport (7.1.0)
faraday
multi_json
excon (0.64.0)
faraday (0.15.4)
multipart-post (>= 1.2, < 3)
faraday_middleware-aws-sigv4 (0.2.4)
aws-sigv4 (~> 1.0)
faraday (>= 0.9)
ffi (1.11.1)
fluent-config-regexp-type (1.0.0)
fluentd (> 1.0.0, < 2)
fluent-plugin-aws-elasticsearch-service (2.1.0)
aws-sdk-core (~> 3)
faraday_middleware-aws-sigv4 (>= 0.2.4, < 0.3.0)
fluent-plugin-elasticsearch (>= 3.3.0, < 4)
fluentd (>= 0.14.15, < 2)
fluent-plugin-concat (2.3.0)
fluentd (>= 0.14.0, < 2)
fluent-plugin-elasticsearch (3.5.2)
elasticsearch
excon
fluentd (>= 0.14.22)
fluent-plugin-grok-parser (2.5.1)
fluentd (>= 0.14.6, < 2)
fluent-plugin-kubernetes_metadata_filter (2.1.6)
fluentd (>= 0.14.0, < 2)
kubeclient (~> 1.1.4)
lru_redux
fluent-plugin-multi-format-parser (1.0.0)
fluentd (>= 0.14.0, < 2)
fluent-plugin-prometheus (1.3.0)
fluentd (>= 0.14.20, < 2)
prometheus-client
fluent-plugin-rewrite-tag-filter (2.1.1)
fluent-config-regexp-type
fluentd (>= 0.14.2, < 2)
fluent-plugin-systemd (1.0.2)
fluentd (>= 0.14.11, < 2)
systemd-journal (~> 1.3.2)
fluentd (1.4.2)
cool.io (>= 1.4.5, < 2.0.0)
dig_rb (~> 1.0.0)
http_parser.rb (>= 0.5.1, < 0.7.0)
msgpack (>= 0.7.0, < 2.0.0)
serverengine (>= 2.0.4, < 3.0.0)
sigdump (~> 0.2.2)
strptime (>= 0.2.2, < 1.0.0)
tzinfo (~> 1.0)
tzinfo-data (~> 1.0)
yajl-ruby (~> 1.0)
http (0.9.8)
addressable (~> 2.3)
http-cookie (~> 1.0)
http-form_data (~> 1.0.1)
http_parser.rb (~> 0.6.0)
http-cookie (1.0.3)
domain_name (~> 0.5)
http-form_data (1.0.3)
http_parser.rb (0.6.0)
i18n (1.6.0)
concurrent-ruby (~> 1.0)
jmespath (1.4.0)
kubeclient (1.1.4)
activesupport
http (= 0.9.8)
recursive-open-struct (= 1.0.0)
rest-client
lru_redux (1.1.0)
mime-types (3.2.2)
mime-types-data (~> 3.2015)
mime-types-data (3.2019.0331)
minitest (5.11.3)
msgpack (1.2.10)
multi_json (1.13.1)
multipart-post (2.1.1)
netrc (0.11.0)
oj (3.5.1)
prometheus-client (0.9.0)
quantile (~> 0.2.1)
public_suffix (3.0.3)
quantile (0.2.1)
recursive-open-struct (1.0.0)
rest-client (2.0.2)
http-cookie (>= 1.0.2, < 2.0)
mime-types (>= 1.16, < 4.0)
netrc (~> 0.8)
serverengine (2.1.1)
sigdump (~> 0.2.2)
sigdump (0.2.4)
strptime (0.2.3)
systemd-journal (1.3.3)
ffi (~> 1.9)
thread_safe (0.3.6)
tzinfo (1.2.5)
thread_safe (~> 0.1)
tzinfo-data (1.2019.1)
tzinfo (>= 1.0.0)
unf (0.1.4)
unf_ext
unf_ext (0.0.7.6)
yajl-ruby (1.4.1)

PLATFORMS
ruby

DEPENDENCIES
ffi
fluent-plugin-aws-elasticsearch-service
fluent-plugin-concat (~> 2.3.0)
fluent-plugin-grok-parser (~> 2.5.0)
fluent-plugin-kubernetes_metadata_filter (~> 2.1.4)
fluent-plugin-multi-format-parser (~> 1.0.0)
fluent-plugin-prometheus (~> 1.3.0)
fluent-plugin-rewrite-tag-filter (~> 2.1.0)
fluent-plugin-systemd (~> 1.0.1)
fluentd (= 1.4.2)
oj (= 3.5.1)

BUNDLED WITH
1.17.2
34 changes: 34 additions & 0 deletions docker-image/v1.4/debian-awselasticsearch/conf/fluent.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@

# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/fluent.conf.erb

@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"
@include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"
@include kubernetes.conf
@include conf.d/*.conf

<match **>
@type aws-elasticsearch-service
@id out_aws_es
@log_level info
include_tag_key true
<endpoint>
url "#{ENV['FLUENT_AWS_ELASTICSEARCH_ENDPOINT']}"
region "#{ENV['FLUENT_AWS_ELASTICSEARCH_REGION']}"
</endpoint>
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"
reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"
reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"
logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"
index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}"
type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"
<buffer>
flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"
queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
retry_forever true
</buffer>
</match>
Loading