-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use out_http to send logs to in_http #2909
Comments
You can send/receive data when you set
I think It's a reasonable idea. WDYT? @repeatedly , @cosmo0920 |
Looks good. Adding link to |
Almost looks good. |
I will look into using
I am fine with your decision, just adding my viewpoint: For direct connections, I agree. In cases with a mandatory (reverse-) proxy in between, HTTP has far better support in my experience. Raw TCP is more of an edge-case. HTTP also only needs 1 open port for multiple services, since host-based routing can be used to forward to different backends. |
Thanks for the suggestion of using This particular configuration of
|
related issue #1936 |
Hello. I have a related challenge: my Fluentd should send logs to third-party ELK. The endpoint is Logstash http with json codec. But instead of Logstash endpoint, it could be Fluentd in_http with the same result. So speaking about json format I have these issues:
Possible workarounds:
The last one is more convenient for me of course. For example, in the FluentBit we may choose from json, json_stream, and json_lines. The last two options we already have in Fluentd with an |
Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.
Is your feature request related to a problem? Please describe.
I have 2 managed k8s clusters with individual fluentd services and want to forward some logs from cluster_1 to cluster_2. I can't use the
forward
protocol because the ingress does not allow raw TCP forwarding.Instead I tried to use
out_http
in cluster_1 andin_http
in cluster_2, since HTTP based ingress configurations are easy to do. But I couldn't find a working configuration, all failed with one of:400 Bad Request\nReceived event is not json
400 Bad Request\n'json' or 'msgpack' parameter is required
incoming event is invalid: path=/some.tag params={"json":"{\"hello\":\"world\",\"num\":18}\n{\"hello\":\"world\",\"num\":19}\n",...}
Describe the solution you'd like
There should be an easy configuration to allow using http to forward events between 2 fluentd instances.
I see 2 possible solutions:
out_http
to optionally wrap (json) events in an (json) arrayin_http
/parse_json
to be able to handle the json lines format used byout_http
Describe alternatives you've considered
Declare the
out_http -> in_http
case to be out of scope. In that case there should be some mention of it in the docs, to save the next person time. Two plugins which communicate over HTTP but can't talk to one another is very counter-intuitive in my opinion.Additional context
Configuration for reproducing the problem.
The text was updated successfully, but these errors were encountered: