New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fluentd-elasticsearch - update fluentd to 1.3.3 & added filter_parser for json fields in es #71180

Merged
merged 8 commits into from Jan 12, 2019

Conversation

@monotek
Copy link
Member

monotek commented Nov 18, 2018

What type of PR is this?

Uncomment only one, leave it on its own line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes # #69603

Special notes for your reviewer:
Please be aware that the 2.4.0 image has to be build and pushed to gcr.io to get the edited deamonset work.

Does this PR introduce a user-facing change?:

Updates Fluentd to 1.3.2 & added filter_parser 
- updated fluentd to 1.3.0
- added filter_parse_json_field to restore removed functionality of kubernetes_metadata (fabric8io/fluent-plugin-kubernetes_metadata_filter#145)
- updated all gems to newest version
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 18, 2018

Hi @monotek. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Nov 18, 2018

/assign @coffeepac

@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Nov 19, 2018

/ok-to-test

@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Nov 19, 2018

/approve

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Nov 19, 2018

@coffeepac
Tests are looking good.
Can you please build the 2.4.0 image and push it to gcr.io before the merge?

@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Nov 20, 2018

no. I will try and ping people who can.

@monotek monotek referenced this pull request Dec 5, 2018

Closed

REQUEST: New membership for monotek #288

6 of 6 tasks complete
@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Dec 21, 2018

/retest

1 similar comment
@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Dec 21, 2018

/retest

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Dec 21, 2018

/test pull-kubernetes-e2e-kops-aws

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Dec 21, 2018

Could somebody trigger the 2.4.0 image build?
I'd like to update the helm chart too, to use this version.

Can we move the image to somewhere else?
Maybe hub.docker.com or quay.io?
So we wouldn't need the manual build step by a google employee.
I'm also fine with any other kind of automatic build on gcr.io ;-)

@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Dec 21, 2018

@monotek moving the images to a location where a non-googler can push them is the plan. I need to write up a KEP for this but the general plan is:

  • move entire addon out of k/k and into kubernetes-sigs/ org
  • change official build process for images to use either public CI (travis, circle, etc) or tie it to a new flow to k/k prow
  • move the image target

I need to file a KEP for this and include stuff like roll forward/backwards over the break, level of effort to keep current image updated, etc.

@monotek monotek changed the title fluentd-elasticsearch - update fluentd to 1.3.2 & added filter_parse_json_field filter fluentd-elasticsearch - update fluentd to 1.3.2 & added filter_parser for json fields in es Dec 22, 2018

@sn-x

This comment has been minimized.

Copy link

sn-x commented Jan 8, 2019

Any updates on when can we expect the image to be available?

@monotek monotek changed the title fluentd-elasticsearch - update fluentd to 1.3.2 & added filter_parser for json fields in es fluentd-elasticsearch - update fluentd to 1.3.3 & added filter_parser for json fields in es Jan 8, 2019

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Jan 8, 2019

updated fluentd to 1.3.3 using newest versions of all gems.

unfortunately i cant tell when the image is build.

if you like to try it out you can use: monotek/fluentd-elasticsearch:26

source: https://cloud.docker.com/repository/docker/monotek/fluentd-elasticsearch/tags

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Jan 8, 2019

/test pull-kubernetes-e2e-kops-aws

@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Jan 9, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Jan 9, 2019

@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Jan 9, 2019

I have pinged the relevant engineers at google. Hopefully they can respond quickly.

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Jan 9, 2019

/test pull-kubernetes-e2e-kops-aws

@sn-x

This comment has been minimized.

Copy link

sn-x commented Jan 10, 2019

Hi @monotek! Thanks for doing great work.

I have tried the new fluentd image from your repo, but I am having some issues with it.

2019-01-10 07:19:32 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" 
\"message\"=&gt;\"\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"}\\n\"}\n"}

Did I forget to update or set something?

@sn-x

This comment has been minimized.

Copy link

sn-x commented Jan 10, 2019

Looks like json parsing started working on a new index for default kibana logs. I guess I have issues with nested json in json logs from node bunyan library. I'll try to debug it and let you know what I have discovered.

@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Jan 11, 2019

/test pull-kubernetes-e2e-kops-aws

1 similar comment
@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Jan 11, 2019

/test pull-kubernetes-e2e-kops-aws

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Jan 12, 2019

/retest

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 12, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 7319ace into kubernetes:master Jan 12, 2019

19 checks passed

cla/linuxfoundation monotek authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Job succeeded.
Details
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-godeps Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Jan 12, 2019

@coffeepac

Oops. Seems we have merged without the image build? Should I add another pr with your new image repo url?

@coffeepac

This comment has been minimized.

Copy link
Member

coffeepac commented Jan 14, 2019

no, a googler has built this image so it should work just fine.

changing the repo will be done in a follow-up PR

@monotek

This comment has been minimized.

Copy link
Member Author

monotek commented Jan 15, 2019

Saw it some hours ago. Already updated the helm chart: kiwigrid/helm-charts#17 😎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment