logstash stop failed; still running. - logstash-2.3.2

[robertd]

Ubuntu: 14.04

robertd@orion:/~$ sudo service logstash stop
Killing logstash (pid 9597) with SIGTERM
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
/etc/init.d/logstash: 100: [: 0: unexpected operator
logstash stop failed; still running.

robertd@orion:/~$ cat /etc/init.d/logstash | grep _TIMEOUT
KILL_ON_STOP_TIMEOUT=0
      if [ "$KILL_ON_STOP_TIMEOUT" == 1 ] ; then

Patch landed in #5201 and #5202 but this is still an issue. My logstash instance keeps running...

Related to #4991.

[robertd]

I've also tried this but no luck

robertd@orion:/~$ echo $KILL_ON_STOP_TIMEOUT
1
robertd@orion:/~$ sudo service logstash stop
Killing logstash (pid 9597) with SIGTERM
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
Waiting logstash (pid 9597) to die...
/etc/init.d/logstash: 100: [: 0: unexpected operator
logstash stop failed; still running.

[suyograo]

Can you add more details here:

Version:
Operating System:
Config File (if you have sensitive info, please remove it):
Sample Data:
Steps to Reproduce:

[robertd]

Version: logstash_2.3.2-1
Operating System: Ubuntu 14.04
Config Files:
/etc/logstash/conf.d/02-beats-input.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

/etc/logstash/conf.d/10-syslog-filter.conf

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

/etc/logstash/conf.d/30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Sample data: Beats is sending syslog data from other VM to logstash.
Steps to reproduce: sudo service logstash stop
Tutorial followed

@suyograo Let me know if you need anything else.

[jjm]

I'm seeing this too, the following has fixed the failing to stop when KILL_ON_STOP_TIMEOUT is set to 1:

--- /etc/init.d/logstash        2016-06-06 11:51:08.763889001 +0100
+++ /etc/init.d/logstash~       2016-04-26 15:35:35.000000000 +0100
@@ -97,7 +97,7 @@
       sleep 1
     done
     if status ; then
-      if [ "$KILL_ON_STOP_TIMEOUT" = "1" ] ; then
+      if [ "$KILL_ON_STOP_TIMEOUT" == 1 ] ; then
         echo "Timeout reached. Killing $name (pid $pid) with SIGKILL. This may                                 result in data loss."
         kill -KILL $pid
         echo "$name killed with SIGKILL."

[EslamElHusseiny]

I have the same issue
Logstash version : logstash-2.3.2-1.noarch
OS : CentOS Linux release 7.2.1511 (Core)
Config file :

input {
  file {
    path => [ "/var/log/**/*" ]
  }
}

output {

  gelf {
    host => "graylog.example.in"
  }

  elasticsearch {
    hosts => [ "elk.example.in" ]
  }
  }

[robertd]

@jjm In OP I've grepped /etc/init.d/logstash file and it shows use of == instead of = (#5202). However... even with KILL_ON_STOP_TIMEOUT set to 1 I'm experiencing the same issue (see second post)

[purbon]

There are a couple of things here to see, one is /etc/init.d/logstash: 100: [: 0: unexpected operator this is due to the the way the comparison is done and the fact for ubuntu's bin/sh => bin/dash with his funny things.

Then there is a second error here and is the usage of KILL_ON_STOP_TIMEOUT that is always set to zero in the init.d script (what is wrong and should be removed), because of this this var is always more or less ignored. See https://github.com/elastic/logstash/blob/2.x/pkg/logstash.sysv#L38 source of it.

This problem affects all versions <= 2.x but not the new 5.0 versions where the package system has been rewritten.

[suyograo]

Fixed in #5479