Restart of collector sidecar causes Graylog server to reprocess all messages

[edmundoa]

From graylog-labs/graylog2-web-interface#1734.

@hryzec wrote:

After change of collector configuration input using options "Read since start" & "Save read position" Gaylog server seems to receive all the log messages at once causing Graylog to fill in disk journal and hang the processing until messages are processed. This however should not happen as it is expected from collector and nxLog to not process messages older than collector restart and older than collector read position.

[mariussturm]

@hryzec: Which sidecar version do you use?
Please check the generated NXLog file for the options SavePos and ReadFromLast, they should be there and set to true. Furthermore the CacheDir should be accessable by the NXLog user to store the log file position.

[hryzec]

@mariussturm Currently using 2x 0.0.9 (sidecar) and one 0.5.0 (deprecated collector)

[mariussturm]

Ok, there was a similar issue please check if it's the same problem for you: #40

Especially the configcache.dat file in the CacheDir is important, if that file is missing NXlog has no idea where to start and ingest the whole file on every run.

[hryzec]

@mariussturm I am using Windows VMs for collectors, nxLog creates log file in the same dir .../data as CacheDir points to but I cannot see any configcache.dat there. nxLog version nxlog-ce-2.9.1716 and collector-sidecar download from 27/7/2016. I have not disabled tthe snippet and distribute to collector hosts using tag 'default'.
This is default snippet Windows part

{{if .Windows}}
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>
{{end}}

[hryzec]

@mariussturm I played a little with CahceDir setting, used "" and '' t enclose got these errors in nxLog:

2016-08-18 16:33:00 INFO configuration OK
2016-08-18 16:33:00 ERROR couldn't open config cache ''C:\Program Files (x86)\nxlog\data'\configcache.dat'; The filename, directory name, or volume label syntax is incorrect.
2016-08-18 16:33:00 ERROR exiting...
2016-08-18 16:33:05 ERROR couldn't open config cache ''C:\Program Files (x86)\nxlog\data'\configcache.dat'; The filename, directory name, or volume label syntax is incorrect.
2016-08-18 16:33:05 ERROR exiting...
2016-08-18 16:33:10 ERROR couldn't open config cache ''C:\Program Files (x86)\nxlog\data'\configcache.dat'; The filename, directory name, or volume label syntax is incorrect.
2016-08-18 16:33:10 ERROR exiting...
2016-08-18 16:33:15 ERROR couldn't open config cache ''C:\Program Files (x86)\nxlog\data'\configcache.dat'; The filename, directory name, or volume label syntax is incorrect.
2016-08-18 16:33:15 ERROR exiting...
2016-08-18 16:36:21 INFO configuration OK
2016-08-18 16:36:21 INFO connecting to 172.28.63.85:12304
2016-08-18 16:36:21 INFO nxlog-ce-2.9.1716 started

I believe the path is ok, also permission should be fine only nxLog does not create the configcache.dat file.

[mariussturm]

I think this is kind of a bug in NXLog. The Sidecar is executing NXlog in foreground mode but on a Windows system NXLog never saves the file position when it runs in foreground. Not sure how to solve this :/

[hryzec]

Yeah, the same behavior when I remove collector-sidecar from the equation and just run nxLog with generated configuration. But what now - their forum sucks :\ As soon as this is only a Windows issue it should not matter that much once I deploy this into production as we have only Linux machines (SLES). But still it is a little awkward. When would new release of GL Server and Collector Sidecar v. 1 eventually be?

[mariussturm]

I looked a little bit more into this. To work around this NXlog limitation I have to do a bigger change in the way the collector is started and executed. For now you can switch to Filebeat instead, the binary is included in the Sidecar packages. I will fix the issue around 0.10 I guess.

[hryzec]

Ok, thank you for your prompt help @mariussturm! I will have a look on that.

[hryzec]

Hi, just to update the actual cause... I have reply from nxLog support:

When you start with nxlog.exe from the command line this invokes the service start and this should work as usual. If you are running it in the foreground with "nxlog.exe -f" this is known to behave the way you have observed because the process cannot do graceful shutdown this way.

So just to clarify... The .dat and .pid files are created only when nxLog is running as service. As soon as nxLog runs with -f it does not creates neither .dat or .pid file?

The pid file is only on used on unix. The configcache.dat is written by NXLog CE on shutdown only. When you terminate it on windows with CTRL-C it is equivalent to "kill -9" and it cannot write the .dat file.

[mariussturm]

Thanks for the update!
Thats actually a little sad that NXlog is not fully functional in foreground mode on Windows :/
I am trying to build a workaround for this.

[hryzec]

@mariussturm Sorry, just one more thing little bit outside of this actual nxLog issue... I cannot really find any filebeat Collector Configuration Input/Output examples or description on web. I have downloaded 2.1.0 Beta 4-1 Server now but ave no idea how to create working configuration that collector sidecar would download and create local filebeat.conf. Currently it just creates file with empty prospector and says:

[filebeat] Multiline match can either be 'after' or 'before', but not ''

[mariussturm]

It works basically like the nxlog configuration but on the 'Beats'-tab. Here is a step-by-step guide how to create a configuration: http://docs.graylog.org/en/2.1/pages/collector_sidecar.html#step-by-step-guide
Make sure to enable beats support in the sidecar configuration: https://github.com/Graylog2/collector-sidecar/blob/master/collector_sidecar.yml#L19

[hryzec]

Thanks @mariussturm ! I will have a look on it.

[tennaen]

@mariussturm i'm trying to configure filebeat, but when i'm creating new collector configuration, i cannot see 'Beats" tab.

[mariussturm]

@tennaen you have to upgrade to Graylog 2.1 for that feature.

[tennaen]

@mariussturm i have updated Graylog to 2.1, but still can't see any tabs.

[mariussturm]

@tennaen which OS and browser version do you use? Do you see a javascript error in the developer console of your browser?

[tennaen]

Now it is working. Problem was with firefox cache. Thanks.

[mariussturm]

@hryzec in 0.1.0-alpha.1 is a fix for this issue, could you please test that version?

[mariussturm]

Closing this, please open a new ticket if there are more problems with nxlog on Windows hosts.