add elasticsearch sink for heapster #733

huangyuqi · 2015-11-26T02:33:32Z

This sink supports monitoring metrics and events.
To use the Elasticsearch sink add the following flag:

--sink=elasticsearch:<ES_SERVER_URL>[?<OPTIONS>]

Normally, elasticsearch cluster has multi nodes or a proxy, so nodes' list or proxy
server need be configured for elasticsearch sink.
Thus, we can set ES_SERVER_URL to a dummy value, and provide nodes list or proxy server
in url's query string.
Besides, the following options can be set in query string:

index - ES's index for metrics&events. Default: heapster

if the ES cluster needs authentication, we should provide the username and secret.
esUserName - ES's user name
esUserSecret - ES's user secret

Like this:

--sink="elasticsearch:?nodes=0.0.0.0:9200&index=testMetric"

or

--sink="elasticsearch:?nodes=0.0.0.0:9200&index=testEvent"

cadvisorJenkinsBot · 2015-11-26T02:33:32Z

Can one of the admins verify this patch?

vishh · 2015-11-30T20:37:07Z

ok to test

On Wed, Nov 25, 2015 at 6:33 PM, cadvisorJenkinsBot <
notifications@github.com> wrote:

Can one of the admins verify this patch?

—
Reply to this email directly or view it on GitHub
#733 (comment).

vishh · 2015-11-30T20:39:11Z

cc @kubernetes/heapster-maintainers

vishh · 2015-12-03T00:13:23Z

needs a rebase

huangyuqi · 2015-12-03T02:53:51Z

Hi, @vishh ,Is there something wrong about the CI? I got the error in checking process,

I1203 02:14:44.359848   15679 heapster_api_test.go:79] Failed to create ns: namespaces "heapster-e2e-tests" already exists
--- FAIL: TestHeapster (51.98s)
    assertions.go:154: 

    Location:   heapster_api_test.go:784

    Error:      No error is expected but got namespaces "heapster-e2e-tests" already exists

vishh · 2015-12-04T00:22:43Z

Can you rebase to HEAD and try again?

On Wed, Dec 2, 2015 at 6:53 PM, yuqi huang notifications@github.com wrote:

Hi, @vishh https://github.com/vishh ,Is there something wrong about the
CI? I got the error in checking process,

I1203 02:14:44.359848 15679 heapster_api_test.go:79] Failed to create ns: namespaces "heapster-e2e-tests" already exists
--- FAIL: TestHeapster (51.98s)
assertions.go:154:
Location:   heapster_api_test.go:784

Error:      No error is expected but got namespaces "heapster-e2e-tests" already exists
—
Reply to this email directly or view it on GitHub
#733 (comment).

huangyuqi · 2015-12-04T00:45:13Z

OK, I will try. Thanks @vishh

cadvisorJenkinsBot · 2015-12-04T00:51:43Z

Can one of the admins verify this patch?

huangyuqi · 2015-12-04T01:16:58Z

Hi, @vishh, I have rebased, but the checks were still not successful.

vishh · 2015-12-04T20:44:05Z

cc @piosz. Is our test infra broken?

On Thu, Dec 3, 2015 at 5:17 PM, yuqi huang notifications@github.com wrote:

Hi, @vishh https://github.com/vishh, I have rebased, but the checks
were still not successful.

—
Reply to this email directly or view it on GitHub
#733 (comment).

piosz · 2015-12-06T20:33:07Z

It seems to be broken. How can I get there to fix it?

cadvisorJenkinsBot · 2015-12-11T20:11:41Z

Can one of the admins verify this patch?

vishh · 2015-12-11T20:13:35Z

ok to test

On Fri, Dec 11, 2015 at 12:11 PM, cadvisorJenkinsBot <
notifications@github.com> wrote:

Can one of the admins verify this patch?

—
Reply to this email directly or view it on GitHub
#733 (comment).

huangyuqi · 2015-12-14T01:13:21Z

Hi, @vishh , how can I resolve the issue about the above checks? Thanks.

k8s-bot · 2015-12-14T21:01:07Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

vishh · 2015-12-17T19:21:54Z

@huangyuqi: @mwielgus might be able to assist you with that.

huangyuqi · 2015-12-18T01:41:01Z

thanks @vishh
Hi, @mwielgus , how can I resolve the issue about the above checks? Thanks.

cadvisorJenkinsBot · 2015-12-22T07:48:19Z

Can one of the admins verify this patch?

vishh · 2015-12-22T17:53:44Z

ok to test

On Mon, Dec 21, 2015 at 11:48 PM, cadvisorJenkinsBot <
notifications@github.com> wrote:

Can one of the admins verify this patch?

—
Reply to this email directly or view it on GitHub
#733 (comment).

huangyuqi · 2016-01-06T08:52:16Z

Hi, @vishh
I have taken a rebase with master.would you please help to review ? Thanks.

piosz · 2016-05-10T19:24:00Z

Any progress on this? It would be great to include it in Heapster 1.1/Kubernetes 1.3. The code freeze is in 2 weeks from now.

k8s-bot · 2016-05-12T13:25:13Z

Jenkins GCE e2e

Build/test failed for commit a212a9d.

Build Log

huangyuqi · 2016-05-12T13:27:52Z

Hi, @piosz I have finished this ES sink, please give me some review comments. Thanks.

k8s-bot · 2016-05-12T13:35:44Z

Jenkins GCE e2e

Build/test failed for commit 4eb0664.

Build Log

huangyuqi · 2016-05-12T13:45:22Z

I got an error in CI:

ok      k8s.io/heapster/events/manager  5.528s
--- FAIL: TestAllExportsInTime (2.00s)
    <autogenerated>:31: 

    Location:   manager_test.go:55

    Error:      Not equal: 3 (expected)
                    != 2 (actual)

W0512 06:35:16.299389   29333 manager.go:108] Failed to events data to sink: s2
W0512 06:35:19.309638   29333 manager.go:108] Failed to events data to sink: s2
W0512 06:35:22.310790   29333 manager.go:108] Failed to events data to sink: s2
W0512 06:35:22.310817   29333 manager.go:108] Failed to events data to sink: s1
W0512 06:35:25.311504   29333 manager.go:108] Failed to events data to sink: s2
W0512 06:35:25.311505   29333 manager.go:108] Failed to events data to sink: s1
FAIL
FAIL    k8s.io/heapster/events/sinks    15.047s

piosz · 2016-05-12T13:49:56Z

this is a known flake

piosz · 2016-05-12T13:50:03Z

@k8s-bot test this

k8s-bot · 2016-05-12T14:24:11Z

Jenkins GCE e2e

Build/test failed for commit 4eb0664.

Build Log

huangyuqi · 2016-05-16T01:05:19Z

@piosz , some issues still exist in CI

[0;33mWaiting for 2 ready nodes. 0 ready nodes, 0 registered. Retrying.�[0m
�[0;33mWaiting for 2 ready nodes. 0 ready nodes, 0 registered. Retrying.�[0m
�[0;31mDetected 0 ready nodes, found 0 nodes out of expected 2. Your cluster may not be working.�[0m
I0512 07:20:20.354669   22591 framework.go:285] Retry validation after 10ns seconds.
I0512 07:20:30.354878   22591 framework.go:212] validating existing cluster
I0512 07:20:30.354934   22591 framework.go:133] about to run &{/tmp/heapster_test/1.0.7/kubernetes/cluster/validate-cluster.sh [/tmp/heapster_test/1.0.7/kubernetes/cluster/validate-cluster.sh] []  <nil> <nil> <nil> [] <nil> <nil> <nil> <nil> false [] [] [] [] <nil>}
panic: test timed out after 30m0s

goroutine 35 [running]:
testing.startAlarm.func1()
    /usr/local/go/src/testing/testing.go:703 +0x132
created by time.goFunc
    /usr/local/go/src/time/sleep.go:129 +0x3a

piosz · 2016-05-16T05:26:40Z

@k8s-bot test this

piosz · 2016-05-16T05:27:25Z

@huangyuqi I fixed it yesterday #1159

huangyuqi · 2016-05-16T05:31:06Z

thanks so much @piosz

ixdy · 2016-05-16T06:20:07Z

@k8s-bot test this please

k8s-bot · 2016-05-16T07:54:57Z

Jenkins GCE e2e

Build/test passed for commit 4eb0664.

Build Log

k8s-bot · 2016-05-16T08:20:27Z

Jenkins GCE e2e

Build/test passed for commit a775491.

Build Log

mwielgus · 2016-05-16T11:55:57Z

LGTM

huangyuqi · 2016-05-16T11:57:30Z

thanks so much @mwielgus

mwielgus · 2016-05-16T12:17:20Z

Thank you too :)

googlebot added the cla: yes label Nov 26, 2015

huangyuqi mentioned this pull request Nov 26, 2015

kafka sink #648

Closed

piosz assigned mwielgus Nov 30, 2015

huangyuqi mentioned this pull request Nov 30, 2015

add another three types of sink in overview.md #739

Merged

huangyuqi closed this Dec 4, 2015

huangyuqi reopened this Dec 4, 2015

vishh closed this Dec 11, 2015

vishh reopened this Dec 11, 2015

huangyuqi closed this Dec 22, 2015

huangyuqi reopened this Dec 22, 2015

huangyuqi closed this May 13, 2016

huangyuqi reopened this May 13, 2016

huangyuqi added 4 commits May 16, 2016 16:10

import the ES client library

00649df

implement ES sink

df9d6f0

go fmt files

48dfce4

pick up the common part of elasticsearch sink

a775491

mwielgus added the lgtm Indicates that a PR is ready to be merged. label May 16, 2016

mwielgus merged commit c11300a into kubernetes-retired:master May 16, 2016

piosz mentioned this pull request May 17, 2016

Fixed broken godeps #1165

Merged

Thermi mentioned this pull request Aug 3, 2016

elasticsearch integration heapster basic auth broken #1238

Closed

pieterlange mentioned this pull request Oct 29, 2016

Production Quality Deployment kubernetes-retired/kube-aws#9

Closed

22 tasks

cblecker unassigned mwielgus Nov 30, 2018

add elasticsearch sink for heapster #733

add elasticsearch sink for heapster #733

Conversation

huangyuqi commented Nov 26, 2015 • edited

cadvisorJenkinsBot commented Nov 26, 2015

vishh commented Nov 30, 2015

vishh commented Nov 30, 2015

vishh commented Dec 3, 2015

huangyuqi commented Dec 3, 2015

vishh commented Dec 4, 2015

huangyuqi commented Dec 4, 2015

cadvisorJenkinsBot commented Dec 4, 2015

huangyuqi commented Dec 4, 2015

vishh commented Dec 4, 2015

piosz commented Dec 6, 2015

cadvisorJenkinsBot commented Dec 11, 2015

vishh commented Dec 11, 2015

huangyuqi commented Dec 14, 2015

k8s-bot commented Dec 14, 2015

vishh commented Dec 17, 2015

huangyuqi commented Dec 18, 2015

cadvisorJenkinsBot commented Dec 22, 2015

vishh commented Dec 22, 2015

huangyuqi commented Jan 6, 2016

piosz commented May 10, 2016

k8s-bot commented May 12, 2016

huangyuqi commented May 12, 2016

k8s-bot commented May 12, 2016

huangyuqi commented May 12, 2016

piosz commented May 12, 2016 • edited

piosz commented May 12, 2016

k8s-bot commented May 12, 2016

huangyuqi commented May 16, 2016

piosz commented May 16, 2016

piosz commented May 16, 2016

huangyuqi commented May 16, 2016

ixdy commented May 16, 2016

k8s-bot commented May 16, 2016

k8s-bot commented May 16, 2016

mwielgus commented May 16, 2016

huangyuqi commented May 16, 2016

mwielgus commented May 16, 2016

huangyuqi commented Nov 26, 2015 •

edited

piosz commented May 12, 2016 •

edited