Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: firehose plugin support #178

Merged
merged 2 commits into from Dec 6, 2021
Merged

Feature: firehose plugin support #178

merged 2 commits into from Dec 6, 2021

Conversation

peter-svensson
Copy link
Contributor

No description provided.

@wenchajun
Copy link
Member

Thanks for your contribution, but I didn't find firehose output plug-in in the official document, could you please provide me with a document or link?

@peter-svensson
Copy link
Contributor Author

peter-svensson commented Dec 3, 2021

Thanks for your contribution, but I didn't find firehose output plug-in in the official document, could you please provide me with a document or link?

Sure, it's part of the AWS for fluentbit, specifically kinesis-firehose.
So it's probably not an official plugin in that sense...

Found some docs here as well 😊

manifests/setup/setup.yaml Outdated Show resolved Hide resolved
manifests/setup/fluentbit-operator-deployment.yaml Outdated Show resolved Hide resolved
@@ -6,7 +6,7 @@ metadata:
labels:
app.kubernetes.io/name: fluent-bit
spec:
image: kubesphere/fluent-bit:v1.8.3
image: peter-fluentbit:v1.8.10
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

V1.8.3 does not support this plugin, or there is a bug in v1.8.3? If true, we should build a fluent bit image of v1.8.10.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, local testing.


// The Firehose output plugin, allows to ingest your records into AWS Firehose.
// https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit
type Firehose struct {
Copy link
Collaborator

@wanjunlei wanjunlei Dec 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable naming needs to be adjusted. Firstly, it is not recommended to use _ in the definition of crd, for example, delivery_stream should be changed to deliveryStream. Secondly, for acronyms, use uppercase, such as Arn => ARN, Sts => STS.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great - I'll fix!

@peter-svensson
Copy link
Contributor Author

Updated a bit, seems like I confused the old and new version of the firehose plugin.
The kinesis_firehose plugin seems to be supported in the kubesphere/fluent-bit:v1.8.3 docker image so should work just fine.
Sorry for the confusion 😊

@wanjunlei
Copy link
Collaborator

wanjunlei commented Dec 6, 2021

By the way, please add the sign-off for the commit.

Download dependencies and binary in Makefile target

Signed-off-by: Peter Svensson <peter@sparetimecoders.com>
This plugin add support for the new kinesis_firehose plugin
https://docs.fluentbit.io/manual/pipeline/outputs/firehose
not the old firehose plugin.

Signed-off-by: Peter Svensson <peter@sparetimecoders.com>
@wanjunlei wanjunlei merged commit 2b4181f into fluent:master Dec 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants