Allow users to add additional metadata to ingestion metrics#13760
Allow users to add additional metadata to ingestion metrics#13760suneet-s merged 8 commits intoapache:masterfrom
Conversation
When submitting an ingestion spec, users may pass a map of metadata in the ingestion spec config that will be added to ingestion metrics. This will make it possible for operators to tag metrics with other metadata that doesn't necessarily line up with the existing tags like taskId. Druid clusters that ingest these metrics can take advantage of the nested data columns feature to process this additional metadata.
|
Docs + tests are not yet written. I pushed this change up early to get some feedback on the naming of the fields being used. EDIT: Docs and tests are complete. Please review |
| DruidMetrics.TASK_TYPE, new String[]{task.getType()} | ||
| ) | ||
| ), | ||
| (Map<String, Object>) task.getContext().get("metricMetadata") |
There was a problem hiding this comment.
this pattern is used in other places in the code, but if metricMetadata is not a map, I think this will crash. Should we introduce a new pattern where this just ignores the metricMetadata object if it is not a map or is failing loudly the desired behavior?
There was a problem hiding this comment.
metricMetadata is the key users would need to pass in the context map to add additional labels to the ingestion metadata. Any thoughts on a better name?
There was a problem hiding this comment.
"tags". On your blowing up comment, write defensive code please.
processing/src/main/java/org/apache/druid/query/DruidMetrics.java
Outdated
Show resolved
Hide resolved
| DruidMetrics.TASK_TYPE, new String[]{task.getType()} | ||
| ) | ||
| ), | ||
| (Map<String, Object>) task.getContext().get("metricMetadata") |
There was a problem hiding this comment.
"tags". On your blowing up comment, write defensive code please.
...service/src/main/java/org/apache/druid/indexing/common/stats/TaskRealtimeMetricsMonitor.java
Outdated
Show resolved
Hide resolved
| return context; | ||
| } | ||
|
|
||
| @JsonIgnore |
There was a problem hiding this comment.
Why JsonIgnore? Jackson should only be using tagged fields anyway, so just not tagging it should be sufficient.
4 participants
Description
When submitting an ingestion spec, users may pass a map of metadata in the ingestion spec config that will be added to ingestion metrics.
This will make it possible for operators to tag metrics with other metadata that doesn't necessarily line up with the existing tags like taskId.
Druid clusters that ingest these metrics can take advantage of the nested data columns feature to process this additional metadata.
Release note
New: You can now add additional metadata to the ingestion metrics emitted from the Druid cluster.
This PR has: