Create a logging cookbook

[brendandburns]

We need to have a cookbook that describes how to set up open source logging and monitoring in Kubernetes.

This should both describe the state of tools, as well as how to integrate with OSS logging solutions like fluentd or logstash.

[bgrant0607]

Perhaps @satnam6502 could take a look

[satnam6502]

Will do. I've looked a little into fluentd and once I have understood enough about what is going on I can write something. I assume we want to talk about two kinds of logging: (a) for the Kubernetes system itself and (b) for the user code in Docker containers. Right? Or should we just focus on (b)?

[bgrant0607]

@satnam6502 This issue is about (b). We also want a solution for (a).

[satnam6502]

Right. I was thinking of producing a super-simple degenerate example i.e.
just one container running a simple hello world services as a baby-step
example. I could add some fluentd logging to this example and write it up.
How about that?

On Fri, Sep 12, 2014 at 12:38 PM, bgrant0607 notifications@github.com
wrote:

@satnam6502 https://github.com/satnam6502 This issue is about (b). We
also want a solution for (a).

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

[vishh]

Monitoring is already taken care of by heapster.

On Fri, Sep 12, 2014 at 12:41 PM, Satnam Singh notifications@github.com
wrote:

Right. I was thinking of producing a super-simple degenerate example i.e.
just one container running a simple hello world services as a baby-step
example. I could add some fluentd logging to this example and write it up.
How about that?

On Fri, Sep 12, 2014 at 12:38 PM, bgrant0607 notifications@github.com
wrote:

@satnam6502 https://github.com/satnam6502 This issue is about (b). We
also want a solution for (a).

—
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/1236#issuecomment-55450961>

.

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

[brendandburns]

I've done some initial work on this, and I'm looking at adding fluentd.

Satnam, if you wanted to integrate heapster into cite kubernetes, I think
that's awesome.

Brendan
On Sep 12, 2014 12:55 PM, "Vish Kannan" notifications@github.com wrote:

Monitoring is already taken care of by heapster.

On Fri, Sep 12, 2014 at 12:41 PM, Satnam Singh notifications@github.com
wrote:

Right. I was thinking of producing a super-simple degenerate example
i.e.
just one container running a simple hello world services as a baby-step
example. I could add some fluentd logging to this example and write it
up.
How about that?

On Fri, Sep 12, 2014 at 12:38 PM, bgrant0607 notifications@github.com
wrote:

@satnam6502 https://github.com/satnam6502 This issue is about (b).
We
also want a solution for (a).

—
Reply to this email directly or view it on GitHub
<

https://github.com/GoogleCloudPlatform/kubernetes/issues/1236#issuecomment-55450961>

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/1236#issuecomment-55451246>

.

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

[bgrant0607]

This issue is about logging. Do we want to broaden it to including monitoring as well?

heapster doesn't address application monitoring at all. It's only resource monitoring. I'm not saying it's not useful, but it's only a small part of the story.

[satnam6502]

Let's focus on logging for now.

[brendandburns]

Ok, I think what you'll want to do is adapt a container to run fluent, and mount in /var/lib/docker/containers into that container, then you should be able to use fluent to monitor the logs there.

Let's sync up on monday based on the experimentation I've done so far.

[satnam6502]

Yesterday I performed an experiment where I wrote a simple Go that program that served some static HTML and also logged interactions (using the Go log package to stderr). I put that in a Docker image which I ran on an ubuntu wheezy GCE VM. I configured fluentd on this VM to collect logs from the log file that corresponds to the container with the Go program. Here are the relevant lines from the fluentd config file /etc/td-agent/td-agent.conf and of course b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb is the full container ID.

type tail format json time_key time path /var/lib/docker/containers/b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb/b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb-json.log pos_file /var/lib/docker/containers/b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb/b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb-json.log.pos tag docker.container.b155666e5690 rotate_wait 5

This does not work because fluentd does not have access permissions to read this file since the log files are rightly accessed controlled and /var/lib/docker has root drwx------ permissions. By default fluentd is run as user td-agent which has ground td-agent.

For a demo we can of course circumvent this but perhaps we should give some thought to how we want to respect access control to log files -- both from Docker as well as the files gathered by flutend which for its log file /var/log/td-agent/td-agent.log has access default permissions -rw-r--r--

Now, we could adopt a sledgehammer solution e.g. add td-agent to the root group but this seems bad. Another option might be to create a new group that allows both Docker and fluentd to read/write to /var/lib/docker/*.

I shall contemplate today how we should handle permissions but I'd welcome any input from wiser folks.

[thockin]

Does docker have flags for log permissions? I don't see them - probably
docker doesn't want people reaching around and accessing the files. Might
be worth poking around docker community on it to see what other people have
asked and answered.

Worst case, why not run fluentd as root?

On Wed, Sep 17, 2014 at 9:32 AM, Satnam Singh notifications@github.com
wrote:

Yesterday I performed an experiment where I wrote a simple Go that program
that served some static HTML and also logged interactions (using the Go log
package to stderr). I put that in a Docker image which I ran on an ubuntu
wheezy GCE VM. I configured fluentd on this VM to collect logs from the log
file that corresponds to the container with the Go program. Here are the
relevant lines from the fluentd config file /etc/td-agent/td-agent.conf and
of course b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb
is the full container ID.

type tail
format json
time_key time
path
/var/lib/docker/containers/b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb/b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb-json.log
pos_file
/var/lib/docker/containers/b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb/b155666e56904127fa7d1f1ee4dcef873754211c1e3dd4eaf76142c156190cdb-json.log.pos
tag docker.container.b155666e5690
rotate_wait 5

This does not work because fluentd does not have access permissions to
read this file since the log files are rightly accessed controlled and
/var/lib/docker has root drwx------ permissions. By default fluentd is run
as user td-agent which has ground td-agent.

For a demo we can of course circumvent this but perhaps we should give
some thought to how we want to respect access control to log files -- both
from Docker as well as the files gathered by flutend which for its log file
/var/log/td-agent/td-agent.log has access default permissions -rw-r--r--

Now, we could adopt a sledgehammer solution e.g. add td-agent to the root
group but this seems bad. Another option might be to create a new group
that allows both Docker and fluentd to read/write to /var/lib/docker/*.

I shall contemplate today how we should handle permissions but I'd welcome
any input from wiser folks.

Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

Yes, I thought about running fluentd as root (easily done in the config file).
Right now I've just manually adjusted the permissions to the container directory under /var/lib/docker

Another option could be to change the setup to log messages to syslog which could then be collected without needing root permssion on fluentd (I think).

[thockin]

I think we want a syslog answer, too, but I think stdout/stderr is more
common and more tractable

On Wed, Sep 17, 2014 at 10:44 AM, Satnam Singh notifications@github.com
wrote:

Yes, I thought about running fluentd as root (easily done in the config
file).
Right now I've just manually adjusted the permissions to the container
directory under /var/lib/docker

Another option could be to change the setup to log messages to syslog
which could then be collected without needing root permssion on fluentd (I
think).

Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

From http://stackoverflow.com/questions/25312710/how-to-monitor-docker-containers-log-from-non-root-user I note "Downside of this approach is that if the syslog daemon on the host is restarted, the container will lose it's socket since the daemon recreates the socket at restart".

[dchen1107]

I think we should run fluentd as root, just like how we run cadvisor.

[thockin]

Or a) petition docker for a uid/gid/mode flag to control logs access and
then b) run as root until docker responds

On Wed, Sep 17, 2014 at 10:57 AM, Dawn Chen notifications@github.com
wrote:

I think we should run fluentd as root, just like how we run cadvisor.

Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

Amazingly using path wildcards in the fluentd config:

<source>
  type tail
  format json
  time_key time
  path /var/lib/docker/containers/*/*-json.log
  tag docker.container.all
</source>

<match docker.container.*>
  type stdout
</match>

collects from a wee container I made with a Go program that serves static HTML and logs interactions. I'll now try multiple containers to make sure that works.

[satnam6502]

Premature excitement: the wildcard only matches against one file.

[thockin]

We can do gross things like HUP fluentd, but we should also investigate
making it support live updates of config. Adding Salty since he's probably
as close to a fluentd expert as we have.

On Wed, Sep 17, 2014 at 11:48 AM, Satnam Singh notifications@github.com
wrote:

Premature excitement: the wildcard only matches against one file.

Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

Is there scope for contributing to fluentd to add the functionality we require? Then we could solve the problem for ourselves as well as others in the wider world who will no doubt face similar issues?

[thockin]

There's certainly scope - whether it is practical is another question, and
also whether the maintainers would want such changes. So we start with
gross wrappers and we proveit works, and we push on "doing it right" in the
background.

cf. everything we do with docker :)

On Wed, Sep 17, 2014 at 8:48 PM, Satnam Singh notifications@github.com
wrote:

Is there scope for actually contributing to fluentd to add the
functionality we require? The we could solve the problem for ourselves as
well as others in the wider world who will no doubt face similar issues?

Reply to this email directly or view it on GitHub
#1236 (comment)
.

[brendandburns]

I think there is scope, but I'd rather not block on upstreaming a patch,
and I'd rather not maintain a patched fluentd. So perhaps we can get it
going with HUP or otherwise for now, and in parallel, look at how hard it
would be to upstream a patch that is better at handling wildcards?

Its possible that this is architectural, each config describes a single log
stream, and thus a wildcard that matches multiple files would conceptually
be a single stream, which we definitely don't want.

On Wed, Sep 17, 2014 at 8:48 PM, Satnam Singh notifications@github.com
wrote:

Is there scope for actually contributing to fluentd to add the
functionality we require? The we could solve the problem for ourselves as
well as others in the wider world who will no doubt face similar issues?

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

[brendandburns]

I just saw:

https://github.com/tagomoris/fluent-plugin-forest

Which appears to do the things we need.

Take a look at this thread:

https://groups.google.com/forum/#!topic/fluentd/LhMLwOttC4A

Where someone is basically discussing exactly our use case.

--brendan

On Wed, Sep 17, 2014 at 8:53 PM, Tim Hockin notifications@github.com
wrote:

There's certainly scope - whether it is practical is another question, and
also whether the maintainers would want such changes. So we start with
gross wrappers and we proveit works, and we push on "doing it right" in the
background.

cf. everything we do with docker :)

On Wed, Sep 17, 2014 at 8:48 PM, Satnam Singh notifications@github.com
wrote:

Is there scope for actually contributing to fluentd to add the
functionality we require? The we could solve the problem for ourselves as
well as others in the wider world who will no doubt face similar issues?

Reply to this email directly or view it on GitHub
<
#1236 (comment)

.

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

[mr-salty]

I think the default in_tail plugin does what we need (tail_ex merged in some time ago). As I said to Satnam in email, by default it only checks for new files every 60s (this can be reduced by specifying i.e. 'refresh_interval 5' in the config), but a single wildcard can match multiple files.

I spent some time playing around with this (using the config above) and it works for me. As configured, all logs will go to a single tag, but if you specify '*' in the 'tag' name it will expand to the path (with some substitutions) if that's what you want.

I used 'tag docker.container.all.*' and was just manually writing to some files using "echo", like:
/var/lib/docker/containers/c2/4-json.log

which turns into the tag 'docker.container.all.var.lib.docker.containers.c2.4-json.log'

I have 4 files in 2 directories and it is definitely watching all of them simultaneously (I can write updates to all the files and see them pop out the output end)

[satnam6502]

Excellent, thanks. I must have got something wrong in my experiments --
perhaps a missing permission under the /var/lib/docker directory.

Satnam

On Fri, Sep 19, 2014 at 12:46 PM, Todd Derr notifications@github.com
wrote:

I think the default in_tail plugin does what we need (tail_ex merged in
some time ago). As I said to Satnam in email, by default it only checks for
new files every 60s (this can be reduced by specifying i.e.
'refresh_interval 5' in the config), but a single wildcard can match
multiple files.

I spent some time playing around with this (using the config above) and it
works for me. As configured, all logs will go to a single tag, but if you
specify '*' in the 'tag' name it will expand to the path (with some
substitutions) if that's what you want.

I used 'tag docker.container.all.*' and was just manually writing to some
files using "echo", like:
/var/lib/docker/containers/c2/4-json.log

which turns into the tag
'docker.container.all.var.lib.docker.containers.c2.4-json.log'

I have 4 files in 2 directories and it is definitely watching all of them
simultaneously (I can write updates to all the files and see them pop out
the output end)

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

Indeed, it was a permission problem. Since then I have been running fluentd as root which fixes the problem.

2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/5029a90ea3afaf6fc7f3015c92f04bde77e45c1439fb22fe334489d72b3248e7/5029a90ea3afaf6fc7f3015c92f04bde77e45c1439fb22fe334489d72b3248e7-json.log
2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/ef596816b2606d578e064a6384b6bbb5e16d9c5b3bfae492b2eab87859771acc/ef596816b2606d578e064a6384b6bbb5e16d9c5b3bfae492b2eab87859771acc-json.log
2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/0194d00fb2ca6ba7e165defefbfd394e058e2825a8f9e8855c6d61cc3828922b/0194d00fb2ca6ba7e165defefbfd394e058e2825a8f9e8855c6d61cc3828922b-json.log
2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/a0c9e577422535f138da0544261264bd938df256da06373518449c938fd4c070/a0c9e577422535f138da0544261264bd938df256da06373518449c938fd4c070-json.log
2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/56818036078c1fdf3332dab3cbafec7b01377230a8e4c73d7288ef90c692680f/56818036078c1fdf3332dab3cbafec7b01377230a8e4c73d7288ef90c692680f-json.log
2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/bebf91e5533735a20638aec824fd9d2de3d690921e8bc83eaf68425d54041633/bebf91e5533735a20638aec824fd9d2de3d690921e8bc83eaf68425d54041633-json.log
2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/da1ef65ffc08a876d25b15afbdc232b38d43280df4fb39965104054f0832a6e1/da1ef65ffc08a876d25b15afbdc232b38d43280df4fb39965104054f0832a6e1-json.log
2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/25772acc84f18c4f8d25cec3c37e61892e67d690f9b056469226d73f63670c60/25772acc84f18c4f8d25cec3c37e61892e67d690f9b056469226d73f63670c60-json.log
2014-09-19 20:38:44 +0000 [info]: following tail of /var/lib/docker/containers/a3bfe7c7026f77a793a2fff6f46600cfd30173460b87b3fd3266379d9603c81e/a3bfe7c7026f77a793a2fff6f46600cfd30173460b87b3fd3266379d9603c81e-json.log

[satnam6502]

So when I run fluentd in a container and I export the Docker log files into the container:

  run -t -i -v /var/lib/docker/containers/:/containers satnam6502/go-hello-fluentd  /bin/bash

I can't seem to get the fluentd (td-agent) inside the container to recognize the mapped log files using the config:

<source>
  type tail
  format json
  time_key time
  path /containers/*/*-json.log
  tag docker.container.all
</source>

<match docker.container.*>
  type stdout
</match>

I also can't get the contained fluentd to listen for JSON messages on port 8888 -- but this works fine for the fluentd running directly on the GCE instance.

Any ideas about what I might do doing wrong? It looks pretty much identical to what works on the GCE instance itself. Thank you kindly.

[brendandburns]

I would manually start the fluentd container and run a shell inside the
container, and explore what is going on by running td-agent yourself.

Brendan
On Sep 19, 2014 4:56 PM, "Satnam Singh" notifications@github.com wrote:

So when I run fluentd in a container and I export the Docker log files
into the container:

run -t -i -v /var/lib/docker/containers/:/containers satnam6502/go-hello-fluentd /bin/bash

I can't seem to get the fluentd (td-agent) inside the container to
recognize the mapped log files using the config:

type tail format json time_key time path /containers/_/_-json.log tag docker.container.all

<match docker.container.*>
type stdout

``
I also can't get the contained fluentd to listen for JSON messages on port 8888 -- but this works fine for the fluentd running directly on the GCE instance.

Any ideas about what I might do doing wrong? It looks pretty much identical to what works on the GCE instance itself. Thank you kindly.

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

[thockin]

This - when it doesn't work, start at the inside and work your way out. If
it's still not coming together Monday AM, let me know.

On Fri, Sep 19, 2014 at 6:54 PM, Brendan Burns notifications@github.com
wrote:

I would manually start the fluentd container and run a shell inside the
container, and explore what is going on by running td-agent yourself.

Brendan
On Sep 19, 2014 4:56 PM, "Satnam Singh" notifications@github.com wrote:

So when I run fluentd in a container and I export the Docker log files
into the container:

run -t -i -v /var/lib/docker/containers/:/containers
satnam6502/go-hello-fluentd /bin/bash

I can't seem to get the fluentd (td-agent) inside the container to
recognize the mapped log files using the config:

type tail format json time_key time path /containers/_/_-json.log tag docker.container.all

<match docker.container.*>
type stdout

``
I also can't get the contained fluentd to listen for JSON messages on
port 8888 -- but this works fine for the fluentd running directly on the
GCE instance.

Any ideas about what I might do doing wrong? It looks pretty much
identical to what works on the GCE instance itself. Thank you kindly.

Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/1236#issuecomment-56249759>

.

Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

Yes, I did that. I notice that within the container things work fine i.e.
if I POST to 8888 then the JSON message ends up in the flutend log, but
when I do the same thing from outside the container (with 8888 => 8888) it
does not work.

I will bludgeon the problem to death this weekend (because is my wife is
away camping).

Cheers,

Satnam

On 19 September 2014 18:54, Brendan Burns notifications@github.com wrote:

I would manually start the fluentd container and run a shell inside the
container, and explore what is going on by running td-agent yourself.

Brendan
On Sep 19, 2014 4:56 PM, "Satnam Singh" notifications@github.com wrote:

So when I run fluentd in a container and I export the Docker log files
into the container:

run -t -i -v /var/lib/docker/containers/:/containers
satnam6502/go-hello-fluentd /bin/bash

I can't seem to get the fluentd (td-agent) inside the container to
recognize the mapped log files using the config:

type tail format json time_key time path /containers/_/_-json.log tag docker.container.all

<match docker.container.*>
type stdout

``
I also can't get the contained fluentd to listen for JSON messages on
port 8888 -- but this works fine for the fluentd running directly on the
GCE instance.

Any ideas about what I might do doing wrong? It looks pretty much
identical to what works on the GCE instance itself. Thank you kindly.

—
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/1236#issuecomment-56249759>

.

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

Satnam Singh
Email: satnam@raintown.org satnam6502@gmail.com
Cell (USA): +1 408 656 4590
Office: +1 650 214 4488
Mobile (UK): +44 7979 648412

[thockin]

docker inspect the target container (the 'pause' one associated with yours)
and make sure you have the right IP and port :)

On Fri, Sep 19, 2014 at 9:13 PM, Satnam Singh notifications@github.com
wrote:

Yes, I did that. I notice that within the container things work fine i.e.
if I POST to 8888 then the JSON message ends up in the flutend log, but
when I do the same thing from outside the container (with 8888 => 8888) it
does not work.

I will bludgeon the problem to death this weekend (because is my wife is
away camping).

Cheers,

Satnam

On 19 September 2014 18:54, Brendan Burns notifications@github.com
wrote:

I would manually start the fluentd container and run a shell inside the
container, and explore what is going on by running td-agent yourself.

Brendan
On Sep 19, 2014 4:56 PM, "Satnam Singh" notifications@github.com
wrote:

So when I run fluentd in a container and I export the Docker log files
into the container:

run -t -i -v /var/lib/docker/containers/:/containers
satnam6502/go-hello-fluentd /bin/bash

I can't seem to get the fluentd (td-agent) inside the container to
recognize the mapped log files using the config:

type tail format json time_key time path /containers/_/_-json.log tag docker.container.all

<match docker.container.*>
type stdout

``
I also can't get the contained fluentd to listen for JSON messages on
port 8888 -- but this works fine for the fluentd running directly on the
GCE instance.

Any ideas about what I might do doing wrong? It looks pretty much
identical to what works on the GCE instance itself. Thank you kindly.

Reply to this email directly or view it on GitHub
<

https://github.com/GoogleCloudPlatform/kubernetes/issues/1236#issuecomment-56249759>

.

Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/1236#issuecomment-56253724>

.

Satnam Singh
Email: satnam@raintown.org satnam6502@gmail.com
Cell (USA): +1 408 656 4590
Office: +1 650 214 4488
Mobile (UK): +44 7979 648412

Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

I've now got a fluentd running in a Docker container and it can observe logging message from a different container. Thank you! Next step: prepare a Dockerfile for this fluentd config and adjust the minion/node creation process to instantiate such a node. Initially just log to a file and then later adjust to target ELK etc.

[thockin]

Do you really need a dockerfile, or just a pod config?

On Mon, Sep 22, 2014 at 1:28 PM, Satnam Singh notifications@github.com
wrote:

I've now got a fluentd running in a Docker container and it can observe
logging message from a different container. Thank you! Next step: prepare a
Dockerfile for this fluentd config and adjust the minion/node creation
process to instantiate such a node. Initially just log to a file and then
later adjust to target ELK etc.

Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

Thanks. I'll take a look at this in detail when I have emerged from the
world of Salt configuration.

Cheers,

Satnam

On Wed, Sep 17, 2014 at 8:59 PM, Brendan Burns notifications@github.com
wrote:

I just saw:

https://github.com/tagomoris/fluent-plugin-forest

Which appears to do the things we need.

Take a look at this thread:

https://groups.google.com/forum/#!topic/fluentd/LhMLwOttC4A

Where someone is basically discussing exactly our use case.

--brendan

On Wed, Sep 17, 2014 at 8:53 PM, Tim Hockin notifications@github.com
wrote:

There's certainly scope - whether it is practical is another question,
and
also whether the maintainers would want such changes. So we start with
gross wrappers and we proveit works, and we push on "doing it right" in
the
background.

cf. everything we do with docker :)

On Wed, Sep 17, 2014 at 8:48 PM, Satnam Singh notifications@github.com

wrote:

Is there scope for actually contributing to fluentd to add the
functionality we require? The we could solve the problem for ourselves
as
well as others in the wider world who will no doubt face similar
issues?

Reply to this email directly or view it on GitHub
<

#1236 (comment)

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/1236#issuecomment-55992137>

.

—
Reply to this email directly or view it on GitHub
#1236 (comment)
.

[satnam6502]

I think the README.md file I wrote the Elasticsearch/Kibana directory contrib/logging/flutend-eslasticsearch acts as kind of Cookbook. Issue closed or do we need more work?