Mysteriously stops processing UDP JSON input

[epchris]

I have logstash listening on port 9999/udp for JSON input and outputting to elasticsearch (also running in docker) via the following config:

input {
  udp {
    port => 9999
    codec => json
    buffer_size => 65536
  }

}

filter {
  if [message] == "" {
    drop { }
  }
}

output {
  elasticsearch {
    hosts => ['elasticsearch:9200']
  }                   
}

Recently it has begun to completely stop processing new input events after a certain set of logs are sent to it. There's nothing specifically strange about the logs, it's Rails controller logs, but it's always a certain controller/action. As long as I don't hit that controller and action it will continue processing just fine.

Strangely, the last log that it does process is some generated SQL statement logging from Rails:

{
  "_index": "logstash-2016.08.12",
  "_type": "rails",
  "_id": "AVZ_VBEVkvTVIgjBD87_",
  "_score": null,
  "_source": {
    "request_id": "f8c8257d-a7bc-46ac-806e-b8af33034070",
    "foo_user_id": "fdb1f325-9ab1-400f-ad7d-e4bf588f01ae",
    "session": "d219ec146d5081b5858add5535cc361e",
    "application": "APP",
    "app_host": "host",
    "level": "DEBUG",
    "type": "rails",
    "tags": [
      "app",
      "sql"
    ],
    "message": "  \u001b[1m\u001b[35mCACHE (0.0ms)\u001b[0m  SELECT \"flipper_gates\".* FROM \"flipper_gates\" WHERE \"flipper_gates\".\"feature_key\" = $1  [[\"feature_key\", \"feature_name\"]]",
    "@timestamp": "2016-08-12T15:25:49.689Z",
    "@version": "1",
    "host": "172.18.0.1"
  },
  "fields": {
    "@timestamp": [
      1471015549689
    ]
  },
  "sort": [
    1471015549689
  ]
}

Sometimes this is the last one:

{
  "_index": "logstash-2016.08.12",
  "_type": "rails",
  "_id": "AVZ_VBEVkvTVIgjBD879",
  "_score": null,
  "_source": {
    "request_id": "f8c8257d-a7bc-46ac-806e-b8af33034070",
    "application": "APP",
    "app_host": "host",
    "level": "DEBUG",
    "type": "rails",
    "tags": [
      "app",
      "sql"
    ],
    "message": "  \u001b[1m\u001b[35m (0.9ms)\u001b[0m  COMMIT",
    "@timestamp": "2016-08-12T15:25:49.674Z",
    "@version": "1",
    "host": "172.18.0.1"
  },
  "fields": {
    "@timestamp": [
      1471015549674
    ]
  },
  "sort": [
    1471015549674
  ]
}

I've been using logstash in docker (on mac) for a long time now and have only started encountering this problem very recently. I recently applied OSX automatic updates and also updated to the latest docker for mac.

I have tried:

• Rolling back application version to one from months ago
• Changing logstash docker image versions (2.1, 2.2 and 2.3)
• downgrading to docker for mac 1.12.0 (latest is 1.12.0-a)
• completely uninstalling docker and reinstalling

None of these things have fixed the problem. What does work to fix it is to restart the logstash container which fixes it until I encounter the same controller/action log again.

I have verified that running logstash locally (on mac, via homebrew, writing to the same elasticsearch instance in docker) with the same config works fine through all cases that cause the dockerized version to fail.

I have also watched traffic to localhost:9999/udp via wireshark and confirmed that the rails application is still sending log events to logstash.

Turning on debug output in logstash (in docker) does not indicate any errors happening. Prior to the failure I can see periodic flush events to elasticsearch with data and then all of a sudden every flush message indicates no data being sent to ES. There haven't been any errors in the logs.

So as far as I know:

• logstash is still running ok and periodically flushing it's empty list of messages to elasticsearch
• UDP packets are being sent to localhost:9999/udp with log events
• No log messages show up in ES after the "event" that triggers failure until I restart logstash container

I really have no idea what could be going on and am hoping someone can help me triage this "issue".

Thanks
Chris

[epchris]

FWIW, I just tried switching to the Docker for Mac Beta channel and am experiencing the same issue :-(

[tianon]

This image is officially deprecated in favor of upstream's images (see https://www.elastic.co/guide/en/logstash/current/docker.html).

For more information, please see docker-library/elasticsearch#160, docker-library/docs#842, and docker-library/docs#945.

This image has been deprecated in favor of the official logstash image provided and maintained by elastic.co. The upstream images are available to pull via docker.elastic.co/logstash/logstash:[version] like 5.4.2. The images found here will receive no further updates once the 5.6.0 release is available upstream. Please adjust your usage accordingly.

Elastic provides open-source support for Logstash via the elastic/logstash GitHub repository and the Docker image via the elastic/logstash-docker GitHub repository, as well as community support via its forums.

Thanks!