Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle multiline from nested JSON strings #337

Open
edsiper opened this issue Jul 26, 2017 · 50 comments

Comments

Projects
None yet
@edsiper
Copy link
Contributor

commented Jul 26, 2017

there is a specific use case where an application running under Docker and generating multiline log messages ends up with logs as follows:

{"log":"2017-07-26 07:54:42.130  WARN parse organization id exception\n","stream":"stdout","time":"2017-07-25T07:54:42.131621351Z"}
{"log":"\n","stream":"stdout","time":"2017-07-25T07:54:42.13171348Z"}
{"log":"java.lang.NumberFormatException: For input string: \"PodQueryBykind\"\n","stream":"stdout","time":"2017-07-25T07:54:42.131723859Z"}

there are 3 JSON log entries, but the contained messages are multiline. We likely need to implement a specific feature in our parsers to reduce the pain.

@djotanov

This comment has been minimized.

Copy link

commented Dec 25, 2017

Any news on when could this be implemented?

@monotek

This comment has been minimized.

Copy link

commented Jan 10, 2018

You could also try to use the detect-exceptions plugin mentioned in #476

https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions

Edit: ahh... damn... the fluentd plugins don't seem compatible :-(

@gavrie

This comment has been minimized.

Copy link
Contributor

commented May 2, 2018

It seems this issue is not being addressed. This is a showstopper for me, so I guess it's back to plain old fluentd...

@abhishek-buragadda

This comment has been minimized.

Copy link

commented Jun 25, 2018

@gavrie Can we handle the same issue in fluentd?

@gavrie

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2018

@abhishek-buragadda

This comment has been minimized.

Copy link

commented Jun 27, 2018

@gavrie This is a paid plugin rgt? are there any open source plugins that does the same?

@gavrie

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2018

@abhishek-buragadda

This comment has been minimized.

Copy link

commented Jun 29, 2018

@gavrie thanks .

@flypenguin

This comment has been minimized.

Copy link

commented Jul 10, 2018

@edsiper just a small request for information - is this something which is being developed now (or soon)? if yes I could wait, if not I would have to look for alternatives. I think this use case is rather straightforward, so my (maybe naive ;) hope is that it is at least on the map.

@edsiper edsiper self-assigned this Jul 11, 2018

@Markbnj

This comment has been minimized.

Copy link

commented Jul 26, 2018

This is an issue for us as well. I spun up a fluent-bit daemonset and was happy with the performance and footprint, but I have not been able to figure out a workaround for the issue with multiline logs. It's not difficult to create a parser for the additional lines which drops the docker cruft and captures the message content. The problem is that the named field gets added to the record for each line, which creates a json record with duplicate keys. One idea comes to mind which would solve this problem neatly: an option to allow the input plugin to append to an existing field when it encounters a new field of the same name. For now we're going back to fluentd because the detect_exceptions plugin helps with some of these cases.

@nikolay

This comment has been minimized.

Copy link

commented Jul 27, 2018

@Markbnj I think, for the time being, Fluent Bit is just an experiment, a POC, as it lacks rudimentary features and cannot be used in the real world.

@breeze7086

This comment has been minimized.

Copy link

commented Jul 27, 2018

Same issue for me.
The cause of this is default docker logging-driver of json-file.

@Markbnj

This comment has been minimized.

Copy link

commented Jul 27, 2018

@nikolay Just FYI we're using it in the real world, as the leaf node collector for over 300 GCP instances, logging over 150 million events per day. The issues on K8S are due to docker's pretty silly logging format, which annotates each line with cruft that ranges from useless to semi-useless. Dealing with that format in a multiline logging scenario is probably beyond fluent-bit's charter, but unfortunately there are not really any easier places to deal with it when you're running on a hosted cluster.

@rawkode

This comment has been minimized.

Copy link

commented Jul 27, 2018

@Markbnj Fluentd doesn't have this problem with Docker logs, so why does fluentbit? As "silly" or "useless" as its output may be, this appears to be a solved problem; but not with fluent-bit

@Markbnj

This comment has been minimized.

Copy link

commented Jul 27, 2018

@rawkode Actually fluentd has pretty much the same issue, except that it has the detect exceptions plugin, which does a pretty good job of detecting multiline exceptions. It doesn't handle all cases of multi-line container logs however.

@rawkode

This comment has been minimized.

Copy link

commented Jul 27, 2018

So if I use a different CRI implementation, this problem goes away?

Has anyone ported the plugin to fluentbit?

@Markbnj

This comment has been minimized.

Copy link

commented Jul 27, 2018

@rawkode good question... I don't have experience actually running anything on cri0, but from looking at google's fluentd config for stackdriver logging it seems like you could also expect per line additions of at least timestamp and implied severity (based on stream) so probably the same issue in a slightly different format.

@michiel

This comment has been minimized.

Copy link
Contributor

commented Jul 27, 2018

@Markbnj @nikolay same here, real world deployment. We are using fluent-bit as k8s node collector across various environments at a similar scale and event volume

@nikolay

This comment has been minimized.

Copy link

commented Jul 30, 2018

@Markbnj Look, your seemingly big numbers are meaningless when even their example setup does not work on a single node. It kinda works for you with tons of hacks and compromises, but Fluent Bit, unlike Fluentd, is targeting Kubernetes, and, yet, it is totally defunct with it. So, if Fluentd needs a plugin - it's understandable and acceptable, but Fluent Bit needs this basic use case out of the box without the requirement for a plugin from the future... as there's no such plugin at this point. So, I repeat what I said - Fluent Bit is possibly the future, but definitely not the present! At this point, it's just a POC, which hopefully will be shaped to something workable around v1.0... but it's still just a v0.13. My point was if I wasn't clear that it needs a big warning sign so that people don't spin wheels!

@nikolay

This comment has been minimized.

Copy link

commented Jul 31, 2018

@michiel As explained, the number of nodes is irrelevant when even the "hello world" equivalent fails with a single node! Provide versions of Fluent Bit and types of apps running in the cluster, which would be something substation other than just bragging!

@Markbnj

This comment has been minimized.

Copy link

commented Jul 31, 2018

@nikolay our numbers aren't really big. I was just giving you a data point to consider. It doesn't seem accurate to me to suggest that fluent-bit is "targeting kubernetes" and is thus insufficient for its primary use case, although the authors can address that better than I can. Kubernetes is mentioned on one line of the readme, in the filters section. In other words, it is one potential source of logs that fluent-bit can be used to collect.

@nikolay

This comment has been minimized.

Copy link

commented Jul 31, 2018

@Markbnj That's what Eduardo said himself during KubeCon 2017, which I attended.

@flypenguin

This comment has been minimized.

Copy link

commented Jul 31, 2018

IMHO pretty much the whole discussion is pointless. I really don't care if fluentbit is production or not, 0.x or not, supercool or not - it's useful to me. And getting this fixed makes it even more useful to me. what more is there to say? why even bother "warning" people who are happy with their choice so far?!

so if @nikolay wants to jump in here and troll an opinion, I personally choose to ignore him because I don't see him contributing anything remotely useful, just some strongly worded opinion about which label to attach to fluentbit, which does not help me at all and I franky don't care about that.

EDIT: changed subject ;) - I only speak for myself.

@edsiper

This comment has been minimized.

Copy link
Contributor Author

commented Jul 31, 2018

Our focus is cloud native in general which includes Kubernetes, and yes, this is a missing feature. It's good to have different opinions.

The ticket continue being an enhancement request; if this missing feature is a blocker for your environment you should go with Fluentd instead.

@rchench

This comment has been minimized.

Copy link

commented Aug 21, 2018

+1 for this feature.
my use case is to collect logs for Spinnaker running in Kubernetes.
as majority of the Spinnaker services are written in java and I do have the needs to parse multiple lines of java exception and feed them into ES to trigger some follow-up actions.

and I like fluent-bit over fluentd as well :)

thanks.

@iwilltry42

This comment has been minimized.

Copy link

commented Sep 7, 2018

@breeze7086 , does your comment mean, that the problem won't exist with any other logging driver from docker?
I guess it definitely won't exist with the fluentd driver?

@IssueHuntBot

This comment has been minimized.

Copy link

commented Sep 15, 2018

@rogerstorm funded this issue with $50. Visit this issue on Issuehunt

@IssueHuntBot

This comment has been minimized.

Copy link

commented Sep 17, 2018

@andrewnazarov

This comment has been minimized.

Copy link

commented Oct 25, 2018

@edsiper, any news on this?

@xpflying

This comment has been minimized.

Copy link

commented Nov 30, 2018

@edsiper filebeat 6.x can deals with multiline from nested JSON strings,https://www.elastic.co/guide/en/beats/filebeat/6.x/filebeat-input-docker.html , is it possible to implement the operation refer to the above content.
Thanks

@edsiper

This comment has been minimized.

Copy link
Contributor Author

commented Nov 30, 2018

note the are two different cases associated with multiline in Docker logs:

  1. Docker split long lines in multiple lines.
  2. Application generate "multiline logs" and Docker put each one as separate entities.

Case #1 is already solved by the new Docker mode of in_tail plugin. Case #2 is still a missing feature.

@protium-dev

This comment has been minimized.

Copy link

commented Jan 14, 2019

I would like to help on this when I have free time the next week. How is the input and the expected output?
Is there some guide to configure a testing docker images in order to test the log outputs?

@shahbour

This comment has been minimized.

Copy link

commented Jan 18, 2019

@ioprotium you can run any java application and throw an error in it it will generate multi line

If you want i can provide a docker image for simulation

{"log":"2019-01-18 07:46:22.634 [               ]  INFO 1 --- [vent-bus.prod-1] c.t.listener.CommonListener              : warehousing Dailywarehousing.daily\n","stream":"stdout","time":"2019-01-18T07:46:22.63525751Z"}
{"log":"2019-01-18 07:49:27.370 [ali            ] ERROR 1 --- [nio-8083-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.Exception: This is a test exception] with root cause\n","stream":"stdout","time":"2019-01-18T07:49:27.373656664Z"}
{"log":"java.lang.Exception: This is a test exception\n","stream":"stdout","time":"2019-01-18T07:49:27.373776864Z"}
{"log":"\u0009at com.xxxxxxxx.controller.ExceptionTestController.exceptionTest(ExceptionTestController.java:15)\n","stream":"stdout","time":"2019-01-18T07:49:27.373799011Z"}
{"log":"\u0009at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n","stream":"stdout","time":"2019-01-18T07:49:27.373815758Z"}
{"log":"\u0009at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n","stream":"stdout","time":"2019-01-18T07:49:27.373831075Z"}
{"log":"\u0009at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n","stream":"stdout","time":"2019-01-18T07:49:27.373847381Z"}
{"log":"\u0009at java.lang.reflect.Method.invoke(Method.java:498)\n","stream":"stdout","time":"2019-01-18T07:49:27.373862835Z"}
{"log":"\u0009at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)\n","stream":"stdout","time":"2019-01-18T07:49:27.373877918Z"}
{"log":"\u0009at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)\n","stream":"stdout","time":"2019-01-18T07:49:27.373893255Z"}
{"log":"\u0009at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)\n","stream":"stdout","time":"2019-01-18T07:49:27.373933561Z"}
{"log":"\u0009at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:891)\n","stream":"stdout","time":"2019-01-18T07:49:27.373950281Z"}
{"log":"\u0009at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)\n","stream":"stdout","time":"2019-01-18T07:49:27.373966221Z"}
{"log":"\u0009at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n","stream":"stdout","time":"2019-01-18T07:49:27.373981291Z"}
{"log":"\u0009at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)\n","stream":"stdout","time":"2019-01-18T07:49:27.373995698Z"}
{"log":"\u0009at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)\n","stream":"stdout","time":"2019-01-18T07:49:27.374009749Z"}
{"log":"\u0009at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:981)\n","stream":"stdout","time":"2019-01-18T07:49:27.374023972Z"}
{"log":"\u0009at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:873)\n","stream":"stdout","time":"2019-01-18T07:49:27.374038132Z"}
{"log":"\u0009at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)\n","stream":"stdout","time":"2019-01-18T07:49:27.374052439Z"}
{"log":"\u0009at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:858)\n","stream":"stdout","time":"2019-01-18T07:49:27.374069639Z"}
{"log":"\u0009at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)\n","stream":"stdout","time":"2019-01-18T07:49:27.374084189Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n","stream":"stdout","time":"2019-01-18T07:49:27.374097982Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374111936Z"}
{"log":"\u0009at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n","stream":"stdout","time":"2019-01-18T07:49:27.374125656Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374139159Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374152839Z"}
{"log":"\u0009at com.xxxxxxxx.filter.LoggingFilter.doFilter(LoggingFilter.java:27)\n","stream":"stdout","time":"2019-01-18T07:49:27.374166452Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374180012Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374193836Z"}
{"log":"\u0009at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)\n","stream":"stdout","time":"2019-01-18T07:49:27.374207526Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374221337Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374234977Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.37424904Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)\n","stream":"stdout","time":"2019-01-18T07:49:27.374295243Z"}
{"log":"\u0009at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n","stream":"stdout","time":"2019-01-18T07:49:27.374329427Z"}
{"log":"\u0009at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n","stream":"stdout","time":"2019-01-18T07:49:27.37434519Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.37435971Z"}
{"log":"\u0009at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)\n","stream":"stdout","time":"2019-01-18T07:49:27.37437389Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374388197Z"}
{"log":"\u0009at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)\n","stream":"stdout","time":"2019-01-18T07:49:27.374402333Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374418393Z"}
{"log":"\u0009at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n","stream":"stdout","time":"2019-01-18T07:49:27.374432798Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374447341Z"}
{"log":"\u0009at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)\n","stream":"stdout","time":"2019-01-18T07:49:27.374461598Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374476194Z"}
{"log":"\u0009at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)\n","stream":"stdout","time":"2019-01-18T07:49:27.374490461Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374504388Z"}
{"log":"\u0009at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)\n","stream":"stdout","time":"2019-01-18T07:49:27.374518434Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374533138Z"}
{"log":"\u0009at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)\n","stream":"stdout","time":"2019-01-18T07:49:27.374546824Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374560551Z"}
{"log":"\u0009at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:74)\n","stream":"stdout","time":"2019-01-18T07:49:27.374574684Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374588648Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374602644Z"}
{"log":"\u0009at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)\n","stream":"stdout","time":"2019-01-18T07:49:27.374616758Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374640288Z"}
{"log":"\u0009at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)\n","stream":"stdout","time":"2019-01-18T07:49:27.374655505Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374671955Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374690312Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)\n","stream":"stdout","time":"2019-01-18T07:49:27.374704522Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)\n","stream":"stdout","time":"2019-01-18T07:49:27.374718459Z"}
{"log":"\u0009at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)\n","stream":"stdout","time":"2019-01-18T07:49:27.374732919Z"}
{"log":"\u0009at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)\n","stream":"stdout","time":"2019-01-18T07:49:27.374750799Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374764819Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374778682Z"}
{"log":"\u0009at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)\n","stream":"stdout","time":"2019-01-18T07:49:27.374792429Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374805985Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374819625Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374833335Z"}
{"log":"\u0009at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)\n","stream":"stdout","time":"2019-01-18T07:49:27.374847845Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374861925Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.37487589Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374890043Z"}
{"log":"\u0009at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)\n","stream":"stdout","time":"2019-01-18T07:49:27.374903813Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374917793Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374931586Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374946006Z"}
{"log":"\u0009at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:117)\n","stream":"stdout","time":"2019-01-18T07:49:27.37496104Z"}
{"log":"\u0009at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:106)\n","stream":"stdout","time":"2019-01-18T07:49:27.37498773Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.375003113Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.375017063Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.37503086Z"}
{"log":"\u0009at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)\n","stream":"stdout","time":"2019-01-18T07:49:27.3750454Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.37505928Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.37507306Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.375086726Z"}
{"log":"\u0009at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)\n","stream":"stdout","time":"2019-01-18T07:49:27.375100817Z"}
{"log":"\u0009at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)\n","stream":"stdout","time":"2019-01-18T07:49:27.375115354Z"}
{"log":"\u0009at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)\n","stream":"stdout","time":"2019-01-18T07:49:27.375129454Z"}
{"log":"\u0009at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)\n","stream":"stdout","time":"2019-01-18T07:49:27.375144001Z"}
{"log":"\u0009at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)\n","stream":"stdout","time":"2019-01-18T07:49:27.375157464Z"}
{"log":"\u0009at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)\n","stream":"stdout","time":"2019-01-18T07:49:27.375170981Z"}
{"log":"\u0009at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)\n","stream":"stdout","time":"2019-01-18T07:49:27.375184417Z"}
{"log":"\u0009at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)\n","stream":"stdout","time":"2019-01-18T07:49:27.375198024Z"}
{"log":"\u0009at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)\n","stream":"stdout","time":"2019-01-18T07:49:27.375211594Z"}
{"log":"\u0009at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)\n","stream":"stdout","time":"2019-01-18T07:49:27.375225237Z"}
{"log":"\u0009at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)\n","stream":"stdout","time":"2019-01-18T07:49:27.375239487Z"}
{"log":"\u0009at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n","stream":"stdout","time":"2019-01-18T07:49:27.375253464Z"}
{"log":"\u0009at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n","stream":"stdout","time":"2019-01-18T07:49:27.375323255Z"}
{"log":"\u0009at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n","stream":"stdout","time":"2019-01-18T07:49:27.375345642Z"}
{"log":"\u0009at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n","stream":"stdout","time":"2019-01-18T07:49:27.375363208Z"}
{"log":"\u0009at java.lang.Thread.run(Thread.java:748)\n","stream":"stdout","time":"2019-01-18T07:49:27.375377695Z"}
{"log":"\n","stream":"stdout","time":"2019-01-18T07:49:27.375391335Z"}
{"log":"\n","stream":"stdout","time":"2019-01-18T07:49:27.375416915Z"}
{"log":"2019-01-18 07:53:06.419 [               ]  INFO 1 --- [vent-bus.prod-1] c.t.listener.CommonListener              : warehousing Dailywarehousing.daily\n","stream":"stdout","time":"2019-01-18T07:53:06.420527437Z"}
@shahbour

This comment has been minimized.

Copy link

commented Jan 18, 2019

i did create a simple image that will produce multiline every 10 second , please use shahbour/java-error:0.0.1

@protium-dev

This comment has been minimized.

Copy link

commented Jan 18, 2019

@shahbour Great. So the idea is that it should generate only one log for multiline messages like a java error with its call stack, right?

@shahbour

This comment has been minimized.

Copy link

commented Jan 18, 2019

@JPM84

This comment has been minimized.

Copy link

commented Jan 28, 2019

We really liked to switch to fluent-bit but as we have many Java Applications this is holding us back...

@shahbour

This comment has been minimized.

Copy link

commented Jan 31, 2019

@ioprotium did you have time to work on this , if you need any testing i can help in that

@stang

This comment has been minimized.

Copy link

commented Feb 21, 2019

+1! This would make our life so much easier!

Ideally, if we could have docker mode and multiline work together, and if both are enabled: all parsers defined for the multiline should be applied against the value of the field named after Key rather than the whole event.


Meanwhile, our work-around was to:

  • use Parser_Firstline to match multi-line logs together based on a simple regex (eg: if the payload starts with a whitespace char, it should be appended to the previous line)
  • the side effect is that any lines that are not matching, are appended to the payload of the field named after Key, but in their docker format
  • use a lua filter to parse these lines and re-construct the content of the field named after Key

It does the job but we have to deal with a whole bunch of things that are already handled by fluent-bit in other places (eg: string escape, utf encoding etc...)

@kiich

This comment has been minimized.

Copy link

commented Mar 31, 2019

Super interested in how you implemetend the above @stang - would you be ok to share the lua filter config and code perhaps?
I'm running into the same issue and have been struggling loads with fluent-bit and multi-line and splunk!

@Kirchen99

This comment has been minimized.

Copy link

commented Apr 3, 2019

Our application generate multiline logs and we also have this issue. Any update for this feature?

@stang

This comment has been minimized.

Copy link

commented Apr 8, 2019

Hi @kiich,

Super interested in how you implemetend the above @stang - would you be ok to share the lua filter config and code perhaps?

Here you go:

  • gist of the helpers.lua file (called from your lua filter in fluent-bit configuration)
  • gist of the JSON.lua file which a slightly modified version of a lua JSON library (original code is linked so you can see what we added)
  • and hereafter, an extract of our fluent-bit configuration:
[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Parser           docker
    Tag              kube.*
    Refresh_Interval 5
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state.db
    DB.Sync          Normal
    Ignore_Older 2d
    Multiline On
    Multiline_Flush 5
    Parser_Firstline first_line
...
[FILTER]
    Name lua
    Match kube.*
    script /fluent-bit/etc/helpers.lua
    call process
...
[PARSER]
    Name        first_line
    Format      regex
    Regex       ^{"log":"(?!\\u0009)(?<log>\S(?:(\\")|[^"]){9}(?:(\\")|[^"])*)"

You might want to fine tune the Regex of the parser for your specific use case.

I'm running into the same issue and have been struggling loads with fluent-bit and multi-line and splunk!

In our case, fluent-bit was the only component used to collect and ship log straight to an elasticsearch instance and we didn't want to add more components to the stack, but you might be able to handle such a thing on the Splunk side.

@kiich

This comment has been minimized.

Copy link

commented May 9, 2019

@stang Awesome! thanks so much for this. We are still struggling with fluent-bit and how it handles multiline json logs - we'll try this out! thanks again.

@shahbour

This comment has been minimized.

Copy link

commented May 21, 2019

@stang are you running this inside Kubernetes or stand alone ?

@stang

This comment has been minimized.

Copy link

commented May 22, 2019

@stang are you running this inside Kubernetes or stand alone ?

In kubernetes.

@rafaelmagu

This comment has been minimized.

Copy link

commented May 23, 2019

It'd be awesome if that helper was present in the official fluent-bit Docker image by default. I'll be able to easily switch that on using the official Helm chart.

@shahbour

This comment has been minimized.

Copy link

commented May 23, 2019

@stang are you running this inside Kubernetes or stand alone ?

In kubernetes.

I asked because i did not see any Kubernetes filter in the example above . Would you mind sharing the full daemonset yaml and configmap you used . Just to have full idea .

@mcrapet

This comment has been minimized.

Copy link

commented Jun 17, 2019

@loadbalance-sudachi-kun, @rogerstorm : I saw this issue on Issuehunt.
Is this still relevant for you?

@shahbour

This comment has been minimized.

Copy link

commented Jun 18, 2019

I just tested @stang solution and it worked perfectly for me , the only thing i had to change is regex to set first line , for me all lines should start with date so below is what i used

    [PARSER]
        Name        first_line
        Format      regex
        Regex       ^{"log":"(?<log>(?:[12]\d{3}-(?:0[1-9]|1[0-2])-(?:0[1-9]|[12]\d|3[01]))(?:(\\")|[^"]){9}(?:(\\")|[^"])*)"
@ZMMWMY

This comment has been minimized.

Copy link

commented Jun 24, 2019

@shahbour it is work , but the format so ..... , you know , anyway , thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.