Skip to content

remove conf from image and add logwriter requirements#15

Merged
leonlnj merged 3 commits intocaraml-dev:mainfrom
leonlnj:log_writer_image
Jan 31, 2023
Merged

remove conf from image and add logwriter requirements#15
leonlnj merged 3 commits intocaraml-dev:mainfrom
leonlnj:log_writer_image

Conversation

@leonlnj
Copy link
Copy Markdown
Contributor

@leonlnj leonlnj commented Jan 30, 2023

What this PR does / why we need it:
This PR is targeted the fluentd image to make it more generic for all use-cases within Timber.
Currently it is used by

  • Observation Service , fluentd producer to BQ
  • Logwriter, to read from kafka and persist to BQ

The following changes are made

  • Adding bigquery and upi-proto-parser to fluentd image
  • BQ plugin version bump to support json field marshalling
  • Removing fluent.conf

Fluent config will be supplied during helm install, upi parser will be available as well due to the imports.

@leonlnj leonlnj marked this pull request as ready for review January 30, 2023 08:28
@pradithya
Copy link
Copy Markdown
Member

What's the docker image size after including bigquery and upi-proto-parser is there any significant increase?

Fluent config will be supplied during helm install

How will it be achieved?

@leonlnj
Copy link
Copy Markdown
Contributor Author

leonlnj commented Jan 30, 2023

its 18mb bigger (405 to 423mb)
image

Im still WIP for a couple of stuff for caraml helm chart in this draft pr.

Sample of how it looks like in the values.yaml

The fluentd is isolated as a standalone, observation service will use it as a subchart and likewise logwriter will can supply it own values.yaml.

The idea is to supply the fluentd.conf in helms values.yaml, then the fluentd chart will create a configMap and volumeMount onto the fluentd pod at the default conf location.

Copy link
Copy Markdown
Member

@pradithya pradithya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@leonlnj leonlnj merged commit 01ac82e into caraml-dev:main Jan 31, 2023
@leonlnj leonlnj added the enhancement New feature or request label Jan 31, 2023
@leonlnj leonlnj self-assigned this Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants