New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify and Automate maintenance of multiple logging targets #4

Merged
merged 10 commits into from Mar 21, 2017

Conversation

Projects
None yet
3 participants
@so0k
Contributor

so0k commented Mar 14, 2017

This PR is to automate the creation of specific images for multiple logging targets.

Each target may have different Dockerfile, fluent.conf or plugins requirements.
Plugins may be added as gem install instructions in the Dockerfile, or provided as
ruby file in the respective plugins folder.

The aproach is copied from the fluentd/fluentd-docker-image repository, using
a modified Makefile and Ruby templates.

Checklist:

  • Create directory structure for multiple logging targets

  • Create Dockerfile template for sample logging targets

  • Create fluent.conf template for sample logging targets
    - [ ] Create Tests to validate generated docker images
    - [ ] Create Travis pipeline to run Tests

  • Add Documentation on available tags and maintenance

  • Create Automated Docker hub repo for this repository
    Somebody from treasure data should do this @repeatedly ?

I suggest to use the image name fluent/fluentd-kubernetes, Please advise.

@so0k so0k referenced this pull request Mar 14, 2017

Closed

List of forks #3

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 14, 2017

Contributor

@tyranron I love your Makefile, please help review this PR
@edsiper as discussed yesterday.

Contributor

so0k commented Mar 14, 2017

@tyranron I love your Makefile, please help review this PR
@edsiper as discussed yesterday.

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 14, 2017

Contributor

PS: I'm not using -onbuild, so I can remove the dummy fluent.conf that was at the root of the docker-image folder, I do however have to add the command to copy the plugins from the plugin folder.

This causes .gitkeep to be copied into the plugins folder as well, I haven't tested if this causes an error or not.

Contributor

so0k commented Mar 14, 2017

PS: I'm not using -onbuild, so I can remove the dummy fluent.conf that was at the root of the docker-image folder, I do however have to add the command to copy the plugins from the plugin folder.

This causes .gitkeep to be copied into the plugins folder as well, I haven't tested if this causes an error or not.

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 14, 2017

Contributor

If you would like to test the images built by this script, I have pushed them all here:
https://hub.docker.com/r/so0k/fluentd-kubernetes/tags/

Contributor

so0k commented Mar 14, 2017

If you would like to test the images built by this script, I have pushed them all here:
https://hub.docker.com/r/so0k/fluentd-kubernetes/tags/

@c-knowles

This comment has been minimized.

Show comment
Hide comment
@c-knowles

c-knowles Mar 14, 2017

Contributor

.dockerignore may be your friend for the gitkeep.

Contributor

c-knowles commented Mar 14, 2017

.dockerignore may be your friend for the gitkeep.

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 14, 2017

Contributor

I thought about adding .dockerignore but... I was worried about another make target & template to worry about... but I think this can just be a simple copy everywhere thing ~ I'll add it.

Contributor

so0k commented Mar 14, 2017

I thought about adding .dockerignore but... I was worried about another make target & template to worry about... but I think this can just be a simple copy everywhere thing ~ I'll add it.

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 14, 2017

Contributor

Done, dockerignore added which will ignore .gitkeep at any level

Contributor

so0k commented Mar 14, 2017

Done, dockerignore added which will ignore .gitkeep at any level

@so0k

This comment has been minimized.

Show comment
Hide comment
Contributor

so0k commented Mar 15, 2017

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 15, 2017

Contributor

@c-knowles - what does the stackdriver setup look like?

Contributor

so0k commented Mar 15, 2017

@c-knowles - what does the stackdriver setup look like?

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 15, 2017

Contributor

@repeatedly - can you please advise on what is missing to get this merged?

Contributor

so0k commented Mar 15, 2017

@repeatedly - can you please advise on what is missing to get this merged?

@c-knowles

This comment has been minimized.

Show comment
Hide comment
@c-knowles

c-knowles Mar 15, 2017

Contributor

@so0k looking good! Stackdriver setup is very similar to your work. It adds fluent-plugin-google-cloud gem as per the gcp contrib image and then the outputs taken from k8s addons. Judging from the work that's gone on there recently it looks like stackdriver team will take ownership of that image.

My implementation is a sort of merge of the work there and here. For example they don't have the kubernetes custom parser. I'm running it outside of GCP with no stackdriver agent present so I also need to add credentials in. There are a few other useful bits contained there which we don't have here, either we could attempt to add those here or get the basics in and I can submit another PR once this one is done.

Contributor

c-knowles commented Mar 15, 2017

@so0k looking good! Stackdriver setup is very similar to your work. It adds fluent-plugin-google-cloud gem as per the gcp contrib image and then the outputs taken from k8s addons. Judging from the work that's gone on there recently it looks like stackdriver team will take ownership of that image.

My implementation is a sort of merge of the work there and here. For example they don't have the kubernetes custom parser. I'm running it outside of GCP with no stackdriver agent present so I also need to add credentials in. There are a few other useful bits contained there which we don't have here, either we could attempt to add those here or get the basics in and I can submit another PR once this one is done.

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 15, 2017

Contributor

I think a 2nd PR from you once this is merged is the best way forward.. may take a while though

I also wonder if it's ok to merge this without the automated testing at first..

Contributor

so0k commented Mar 15, 2017

I think a 2nd PR from you once this is merged is the best way forward.. may take a while though

I also wonder if it's ok to merge this without the automated testing at first..

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 16, 2017

Contributor

@frsyuki - need review and advise on how to get this merged

Contributor

so0k commented Mar 16, 2017

@frsyuki - need review and advise on how to get this merged

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 16, 2017

Contributor

@c-knowles - if you want to add your changes you can add it to https://github.com/honestbee/fluentd-kubernetes, I'll cherry pick to this branch

I set up the fully automated docker build: https://hub.docker.com/r/honestbee/fluentd-kubernetes/

Contributor

so0k commented Mar 16, 2017

@c-knowles - if you want to add your changes you can add it to https://github.com/honestbee/fluentd-kubernetes, I'll cherry pick to this branch

I set up the fully automated docker build: https://hub.docker.com/r/honestbee/fluentd-kubernetes/

@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Mar 16, 2017

Contributor

fyi: I will make sure to review this today

Contributor

edsiper commented Mar 16, 2017

fyi: I will make sure to review this today

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 21, 2017

Contributor

any update?

Contributor

so0k commented Mar 21, 2017

any update?

@edsiper edsiper merged commit e79807f into fluent:master Mar 21, 2017

@c-knowles

This comment has been minimized.

Show comment
Hide comment
@c-knowles

c-knowles Mar 27, 2017

Contributor

Based on comments in #6 I'm no longer sure there is a want to have Stackdriver/GCP logging added. The one thing I would add is that the official image is tailored a bit more towards those running on GCP whereas we're running on AWS. Plus they don't even add some things like the Kubernetes metadata filter which seems almost essential here. Maybe they will make it more agnostic but in the meantime I wouldn't mind adding support. @edsiper @so0k any particular thoughts?

Contributor

c-knowles commented Mar 27, 2017

Based on comments in #6 I'm no longer sure there is a want to have Stackdriver/GCP logging added. The one thing I would add is that the official image is tailored a bit more towards those running on GCP whereas we're running on AWS. Plus they don't even add some things like the Kubernetes metadata filter which seems almost essential here. Maybe they will make it more agnostic but in the meantime I wouldn't mind adding support. @edsiper @so0k any particular thoughts?

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 27, 2017

Contributor

@c-knowles I'm not at all familiar with the stackdriver logging and if anyone else is using it while not on GCP, although it might be good for anyone if you are willing to add your work into this central location so anyone can built on top of it?

Contributor

so0k commented Mar 27, 2017

@c-knowles I'm not at all familiar with the stackdriver logging and if anyone else is using it while not on GCP, although it might be good for anyone if you are willing to add your work into this central location so anyone can built on top of it?

@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Mar 27, 2017

Contributor

@c-knowles

My general perception when talking with different people from different organizations, is that Kubernetes metadata is a fundamental piece for them, not just Google.

Anyways I would like to hear more about more if the filter is generating some issues, so we can consider a way to disable it in the templates generation.

Contributor

edsiper commented Mar 27, 2017

@c-knowles

My general perception when talking with different people from different organizations, is that Kubernetes metadata is a fundamental piece for them, not just Google.

Anyways I would like to hear more about more if the filter is generating some issues, so we can consider a way to disable it in the templates generation.

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Mar 27, 2017

Contributor

Right now the metadata gem is in a shared gems folder so it's included for all builds and it shouldn't be hard to make exceptions to that rule

However I don't think that is what @c-knowles means as he said k8s metadata gem is essential to a fluent shipping container actually

Contributor

so0k commented Mar 27, 2017

Right now the metadata gem is in a shared gems folder so it's included for all builds and it shouldn't be hard to make exceptions to that rule

However I don't think that is what @c-knowles means as he said k8s metadata gem is essential to a fluent shipping container actually

@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Mar 27, 2017

Contributor

you are right, I misunderstood.

Contributor

edsiper commented Mar 27, 2017

you are right, I misunderstood.

@c-knowles

This comment has been minimized.

Show comment
Hide comment
@c-knowles

c-knowles Mar 28, 2017

Contributor

@edsiper, yes sorry if my comment was confusing. I meant the official image does not container the k8s metadata filter but from what I have seen it is pretty much essential. It is one reason to consider a stackdriver build here as well.

Sounds like we're good to add it then, will submit a PR.

Contributor

c-knowles commented Mar 28, 2017

@edsiper, yes sorry if my comment was confusing. I meant the official image does not container the k8s metadata filter but from what I have seen it is pretty much essential. It is one reason to consider a stackdriver build here as well.

Sounds like we're good to add it then, will submit a PR.

@edsiper edsiper referenced this pull request Apr 26, 2017

Merged

Add debian based images #12

@joerx joerx deleted the honestbee:unify_logging_targets branch Mar 14, 2018

ParticleDecay pushed a commit to ParticleDecay/fluentd-kubernetes-daemonset that referenced this pull request Sep 25, 2018

Merge pull request #4 from beamb/fluentd01243
Bumping debian-kafka to 0.12.43 to fix syslog_in bug
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment