Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dead Letter Queue feature should have metrics available via the monitoring API #7287

Closed
robbavey opened this issue Jun 1, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@robbavey
Copy link
Contributor

commented Jun 1, 2017

The Dead Letter Queue feature should expose metrics for each queue, including

  • Queue path
  • Number of segments stored
  • Max segment size (?)
  • Number of items in the queue (is total segments sufficient?)
  • Disk space used by the queue
  • Last event written

work in progress

@robbavey

This comment has been minimized.

Copy link
Contributor Author

commented Jun 5, 2017

Initial thought - similar to how queues are exposed in the pipeline, the same could be done for the dead letter queue:

eg:

     "dead_letter_queue": {
       "path": "/Users/robbavey/code/logstash/data/dead_letter_queue/main",
       "current_queue_size_in_bytes": 684,
       "current_segment_count": 27,
       "last_entry_timestamp": "2017-06-05T00:28:57.394Z",
       "max_segment_size_in_bytes": 10485760
     }
  • Do we need to mark if dlq is not enabled for a pipeline?
  • Getting current number of entries in the queue with the current DLQ implementation might be somewhat 'expensive' for queues with many segments (bulk of this expense most likely on startup/first stats call?)

@suyograo Thoughts?

@suyograo

This comment has been minimized.

Copy link
Member

commented Jun 5, 2017

@robbavey 👍

last_entry_timestamp: not sure we need this. Turns out this was not very useful.

Do we need to mark if dlq is not enabled for a pipeline?

Yes, that would be useful. we could do:

"dead_letter_queue": {
    "enabled": true|false
}

And skip stats if it is false.

Getting current number of entries in the queue with the current DLQ implementation might be somewhat 'expensive' for queues with many segments (bulk of this expense most likely on startup/first stats call?)

Agreed. I do think the number of entries in the DLQ will be small, so this should be ok for now. I would do that on the first API call, not at LS startup.

I think the actionable field here would be current_queue_size_in_bytes so users can alert on a DLQ which is filling and hasn't been purged.

robbavey added a commit to robbavey/logstash that referenced this issue Jun 6, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "current_queue_size_in_bytes": ...,
        "segment_count": ...,
        "max_segment_size_in_bytes": ...
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 6, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "current_queue_size_in_bytes": ...,
        "segment_count": ...,
        "max_segment_size_in_bytes": ...
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 6, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "current_queue_size_in_bytes": ...,
        "segment_count": ...,
        "max_segment_size_in_bytes": ...
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 6, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "current_queue_size_in_bytes": ...,
        "segment_count": ...,
        "max_segment_size_in_bytes": ...
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 6, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "current_queue_size_in_bytes": ...,
        "segment_count": ...,
        "max_segment_size_in_bytes": ...
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 7, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "queue_size_in_bytes": ...,
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 7, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "queue_size_in_bytes": ...,
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 8, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "queue_size_in_bytes": ...,
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 8, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "queue_size_in_bytes": ...,
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 8, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "path": ...,
        "enabled": ...,
        "queue_size_in_bytes": ...,
      }

Note that if the dead letter queue is not enabled, then only the
enabled property will be displayed, with a value of false.

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 8, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 9, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 9, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 9, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

Fixes elastic#7338

robbavey added a commit to robbavey/logstash that referenced this issue Jun 9, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

Fixes elastic#7338

robbavey added a commit to robbavey/logstash that referenced this issue Jun 9, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

Fixes elastic#7338

robbavey added a commit to robbavey/logstash that referenced this issue Jun 9, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

Fixes elastic#7338

elasticsearch-bot pushed a commit that referenced this issue Jun 9, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves #7287

Fixes #7338

Fixes #7406

robbavey added a commit to robbavey/logstash that referenced this issue Jun 14, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 15, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jun 28, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Jul 21, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287

robbavey added a commit to robbavey/logstash that referenced this issue Aug 14, 2017

Add Metrics for dead letter queue
Add an initial set of metrics for the dead letter queue.

Metrics are supplied under the pipeline in the following format:

"dead_letter_queue": {
        "queue_size_in_bytes": ...,
      }

Metrics are populated via a PeriodicPoller

Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.

Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint

Resolves elastic#7287
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.