Permission denied Error: Permission denied - /var/log/logstash.9844.4050.994333 #3311
Comments
|
With debug output: |
|
Think I might have found the problem. I had sincedb_path configured in my logstash.conf pointed at /var/log/logstash. Commenting it out has fixed logstash so far.. |
|
I'm having the same issue. If we comment out the since_db statement where is the logstash.log located? I don't see anything in the $home dir |
|
I think this is a real issue with Logstash 1.5.0. Here's a test configuration, with more details: Here's the creation of the folder: Now I run Logstash with the same user: And I put something on the target file: Logstash breaks with the following error: Funny enough, the target folder seems to be broken. Please notice the lack of execution permissions on the folder, which prevents the user getting into the folder: Although something is there already: Now with sudo, to make things clear: This corruption of the permissions of the target folder explains the |
|
Hello, I have the same issue in my case it's "Error: Permission denied - /var/lib/logstash/" that cause problem. I updated Logstash to 1.5.1 and the issue still here. I did chmod 777 to this folder but after re-running Logstash, it's get back to his original rights i.e. rw-rw-rw-. I will try Logstash 1.4.3 to see if issue will be still here. |
|
Pulling my hair out over the same issue. Just tried changing the sincedb path to /tmp/sincedb which I gave logstash ownership of and perms of 777: Not sure if it needs execute perms to function, if it does it's breaking itself..... sigh |
|
As I mentioned before, looks like a legitimate bug. As a workaround, don't use |
|
This is a problem of poor configuration documentation, naming and expectations.. To check that it works, setting |
|
That's fair enough, but even then I would argue that Logstash configuration validation should fail if we're trying to use |
|
Totally agree, I'm opening an issue in logstash-plugins/logstash-input-file to properly give out an error if the destination is a directory. |
|
This might be a consequence on inconsistent File.open behavior on MRI and JRuby: Digging deeper.. |
|
Thanks @Sodki, setting the SINCEDB_DIR environment variable workaround is working for me so far! =) |
|
I am having the exact same issue, permissions are definitely being changed by logstash I can if i review the settings the |
|
@natemccoy as mentioned, the config setting was meant to be a file path, not a directory path. if you set it to /var/cache/sincedbs/mysince.db, no corruption of permissions will occur |
|
I have resolved the issue by adding temp/sincedb/.sincedb path. |
|
Hi, I am having the same issue on CentOS 6.7 w/ Logstash 1.5.4 It has occurred w/o setting the 'sincedb_path' and also with setting it to a specific file {:timestamp=>"2015-10-21T07:42:50.868000-0700", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::File type=>"GA_riskengine_1", path=>["/mnt/dc1filer02//current//logs/riskengine/*/riskengine.log"], start_position=>"beginning", sincedb_path=>"/var/lib/logstash/sincedb_riskengine", codec=><LogStash::Codecs::Multiline patterns_dir=>["/etc/logstash/conf.d/patterns"], pattern=>"^%{GATIMESTAMP}|%{IP} ", negate=>true, what=>"previous", max_lines=>2500, charset=>"UTF-8", multiline_tag=>"multiline", max_bytes=>10485760>, debug=>false, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">\n Error: Permission denied - /var/lib/logstash/sincedb_riskengine.25256.21254.235099", :level=>:error} {:timestamp=>"2015-10-21T08:07:23.659000-0700", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::File type=>"GA_collector_1", path=>["/mnt/dc1filer02//current//logs/collector/collector.log"], start_position=>"beginning", sincedb_path=>"/var/lib/logstash/sincedb_collector", codec=><LogStash::Codecs::Multiline patterns_dir=>["/etc/logstash/conf.d/patterns"], pattern=>"^%{GATIMESTAMP}|%{IP} ", negate=>true, what=>"previous", max_lines=>2500, charset=>"UTF-8", multiline_tag=>"multiline", max_bytes=>10485760>, debug=>false, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">\n Error: Permission denied - /var/lib/logstash/sincedb_collector.20212.23238.470530", :level=>:error} file { [root@mon-esc-2 ~]# ls -alt /var/lib/logstash/ Suggestions? |
|
Hey there I don't remember well but I think the issue was that I needed to specify a I had another issue once where the file was corrupt and I just deleted it Try both of those, keep me updated, I will try to help if I can
|
|
Still having the same problem. Time frame varies, I have tried both removing the files and changing permissions. After a period of time I still get the same permission denied error and logstash crashes. |
|
I don't have a system setup with this to test, but I do remember a few
|
|
@natemccoy Yes, here is the final log entry at the the time of the crash: {:timestamp=>"2015-10-22T06:24:54.351000-0700", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::File type=>"GA_riskengine_1", path=>["/mnt/dc1filer02//current/prod03/logs/riskengine//riskengine.log"], start_position=>"beginning", sincedb_path=>"/var/lib/logstash/sincedb_riskengine_prod03", codec=><LogStash::Codecs::Multiline patterns_dir=>["/etc/logstash/conf.d/patterns"], pattern=>"^%{GATIMESTAMP}|%{IP} ", negate=>true, what=>"previous", max_lines=>2500, charset=>"UTF-8", multiline_tag=>"multiline", max_bytes=>10485760>, debug=>false, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">\n Error: Permission denied - /var/lib/logstash/sincedb_riskengine_prod03.20168.11673.157291", :level=>:error} And the input config section: |
|
Is sincedb_riskengine_prod03 a file or folder?
|
|
@natemccoy and a listing of the directory..... [root@mon-esc-2 ~]# ls -al /var/lib/logstash/ |
|
Well I am sorry to day I am not sure, I had an issue in the past where I am sorry to day I don't think I can be of much more help, I do not have a You may want to run strace on the daemon if you know how to make it crash You may also want to check if the file permissions are changing fire some Good luck
|
|
Thanks for your help. |
|
Haha wow i made so many typos! Good luck!
|
|
If I want to dig deeper, can you point me to a place to understand how to trace this in better detail? Thanks. |
|
closing this in favor of #4072 |
Hi guys
Having a weird problem with logstash 1.5.0-1 on CentOS 7. When I start logstash it crashes after just under two minutes. Only error in logs is the following repeated over and over:
{:timestamp=>"2015-05-25T19:43:08.410000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::File path=>["/var/log/network.log"], sincedb_path=>"/var/log/logstash", start_position=>"beginning", type=>"syslog", tags=>["asa_log"], delimiter=>"\n">\n Error: Permission denied - /var/log/logstash.9844.4050.994333 or /var/log/logstash", :level=>:error}
I've verified all permissions:
After giving logstash rwx to /var/log I noticed the logstash.xxxx.yyyy.zzzzzz files being created - is that expected behaviour? This problem does exist as well if I run logstash as root...
Best regards,
Daniel
The text was updated successfully, but these errors were encountered: