-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Filebeat 6.4.0 chokes on docker continuation lines, "parsing CRI timestamp" errors #8203
Comments
I'm still experiencing this with 6.4.1
|
Hi @SharpEdgeMarshall, did you remove the existing registry file? Try again after removing |
Removed the file and restarted k8s pods but the problem still exist. |
I did a quick test now and it seems to be working for me, you will need to follow these steps:
|
Worked for me too. We had filebeat as daemonset and mounted host filesystem. |
I'm experiencing similar behavior on 6.4.2 (also saw same behavior on 6.4.1) when restarting Filebeat. I'm using docker autodiscover and setting module inputs on container streams. On first run (no registry file exists), harvesters start up without a problem. On service restart (registry file exists), a few containers exhibit the CRI timestamp problem. To workaround, I have to stop Filebeat, remove the registry, and start Filebeat. This means I lose a few messages because of Error message:
The docker json log message in question:
The docker autodiscover inputs:
|
I confirm the last comment's behavior on filebeat version 6.4.2. I too use docker autodiscover. The second case when the issue comes up is when I restart a container. |
The first version of filebeat was 6.4.2 on that server. And while experimenting I also cleaned registry file. |
Thanks, @a-abella , will check this week if update works for me. |
@genuss Scratch that, just checked right now and one of our servers has started erroring again over the weekend. Once again, it's a docker autodiscover entry that starts up the nginx module:
The current registry entry :
Typical log content:
|
I appear to have this issue as well, on 6.5.1. See my post https://discuss.elastic.co/t/filebeat-6-4-2-and-6-5-1-read-line-error-parsing-cri-timestamp-and-invalid-cri-log-format/159383/2. |
FWIW we went back to this and disabled autodicover. I think it's just more reliable:
|
@towolf Did you have to add any other configuration, other than you showed above, after disabling autodiscover? I'd like to do this too, as nobody from Elastic seems to care about this error and/or know how to fix it, but I would like to understand what a minimal but somewhat equivalent to autodiscover configuration would be. |
The above fix for cri worked for filebeat 6.5.3. |
Similar to @rocketraman I've encountered the same issues on 6.5.3. I've documented my rough findings so far here: https://discuss.elastic.co/t/filebeat-6-4-2-and-6-5-1-read-line-error-parsing-cri-timestamp-and-invalid-cri-log-format/159383/6?u=evesy |
6.5.4 same problem.
Problem appears with nginx container
|
@exekias, can we reopen issue? Since some still have problems... |
@BulatSaif a fix for this will be released in 6.6.1 (#10211) if problems persist by then we'll reopen. |
Do you have ETA on the relase date of 6.6.1? If that's too far away, does the 7.0 beta have that fix? |
@oliveiraVG it has just been released https://www.elastic.co/blog/elastic-stack-6-6-1-and-5-6-15-released 🙂 7.0.0-beta1 also includes #10211 |
Thank you! |
I'm still seeing these errors with 6.6.1 |
This is the kind of nginx log line it's erroring on: {"log":"10.50.131.251 - - [27/Feb/2019:05:00:28 +0000] "GET /status HTTP/1.1" 200 8 "-" "kube-probe/1.12"\n","stream":"stdout","time":"2019-02-27T05:00:28.617850486Z"} |
#10951 probably helps with the timestamp issues. |
We upgraded to Filebeat 6.4.0 running in Kubernetes with regular Docker engine with
json-file
logging.Now it begun to choke on some messages, trying to parse dates for CRI format. We do not have CRI format.
The actual file on disk looks like this:
The errors printed are these:
And so on and so forth.
The text was updated successfully, but these errors were encountered: