Enrich stats executing policies empty despite running policy executions

[ddolcimascolo]

Elasticsearch Version

7.11.1

Installed Plugins

No response

Java Version

bundled

OS Version

Ubuntu 18.04

Problem Description

I'm scheduling the execution of multiple enrich policies at once using

POST /_enrich/policy/<policy>/_execute?wait_for_completion=false

Right after I run

GET /_enrich/_stats

and I get an empty executing_policies:

{
  "executing_policies" : [ ],
  "coordinator_stats" : [
    {
      "node_id" : "jQsvQu84TgK_wU6Tv7WzFQ",
      "queue_size" : 0,
      "remote_requests_current" : 0,
      "remote_requests_total" : 621305,
      "executed_searches_total" : 626757
    },
    {
      "node_id" : "nfaUfK3uST-UxMCKQBDYAQ",
      "queue_size" : 0,
      "remote_requests_current" : 0,
      "remote_requests_total" : 616464,
      "executed_searches_total" : 622267
    },
    {
      "node_id" : "oUHCWUxVSlarbQgtG08Xqg",
      "queue_size" : 0,
      "remote_requests_current" : 0,
      "remote_requests_total" : 617850,
      "executed_searches_total" : 621446
    }
  ],
  "cache_stats" : [
    {
      "node_id" : "jQsvQu84TgK_wU6Tv7WzFQ",
      "count" : 1000,
      "hits" : 2639828,
      "misses" : 626759,
      "evictions" : 625757
    },
    {
      "node_id" : "nfaUfK3uST-UxMCKQBDYAQ",
      "count" : 1000,
      "hits" : 2679847,
      "misses" : 622273,
      "evictions" : 621267
    },
    {
      "node_id" : "oUHCWUxVSlarbQgtG08Xqg",
      "count" : 1000,
      "hits" : 2625570,
      "misses" : 621450,
      "evictions" : 620446
    }
  ]
}

while immediately querying the tasks with

GET /_cat/tasks?actions=*policy*

yields all tasks as expected

policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952266 jQsvQu84TgK_wU6Tv7WzFQ:235705700 enrich 1666007116838 11:45:16 2.1s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743409 jQsvQu84TgK_wU6Tv7WzFQ:235705705 enrich 1666007116856 11:45:16 2.1s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952272 jQsvQu84TgK_wU6Tv7WzFQ:235705710 enrich 1666007116864 11:45:16 2.1s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743415 jQsvQu84TgK_wU6Tv7WzFQ:235705713 enrich 1666007116871 11:45:16 2.1s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952277 jQsvQu84TgK_wU6Tv7WzFQ:235705718 enrich 1666007116873 11:45:16 2.1s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743421 jQsvQu84TgK_wU6Tv7WzFQ:235705723 enrich 1666007116882 11:45:16 2.1s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952283 jQsvQu84TgK_wU6Tv7WzFQ:235705727 enrich 1666007116894 11:45:16 2.1s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743428 jQsvQu84TgK_wU6Tv7WzFQ:235705736 enrich 1666007116917 11:45:16 2.1s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952292 jQsvQu84TgK_wU6Tv7WzFQ:235705744 enrich 1666007116920 11:45:16 2.1s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743434 jQsvQu84TgK_wU6Tv7WzFQ:235705749 enrich 1666007116927 11:45:16 2.1s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952298 jQsvQu84TgK_wU6Tv7WzFQ:235705754 enrich 1666007116928 11:45:16 2s   10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743440 jQsvQu84TgK_wU6Tv7WzFQ:235705759 enrich 1666007116936 11:45:16 2s   10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952304 jQsvQu84TgK_wU6Tv7WzFQ:235705764 enrich 1666007116956 11:45:16 2s   10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743449 jQsvQu84TgK_wU6Tv7WzFQ:235705775 enrich 1666007116983 11:45:16 2s   10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952317 jQsvQu84TgK_wU6Tv7WzFQ:235705788 enrich 1666007117024 11:45:17 2s   10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743459 jQsvQu84TgK_wU6Tv7WzFQ:235705793 enrich 1666007117037 11:45:17 1.9s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952323 jQsvQu84TgK_wU6Tv7WzFQ:235705798 enrich 1666007117040 11:45:17 1.9s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743465 jQsvQu84TgK_wU6Tv7WzFQ:235705803 enrich 1666007117059 11:45:17 1.9s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952328 jQsvQu84TgK_wU6Tv7WzFQ:235705807 enrich 1666007117067 11:45:17 1.9s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743474 jQsvQu84TgK_wU6Tv7WzFQ:235705813 enrich 1666007117084 11:45:17 1.9s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952336 jQsvQu84TgK_wU6Tv7WzFQ:235705822 enrich 1666007117091 11:45:17 1.9s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743486 jQsvQu84TgK_wU6Tv7WzFQ:235705827 enrich 1666007117099 11:45:17 1.9s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952345 jQsvQu84TgK_wU6Tv7WzFQ:235705833 enrich 1666007117102 11:45:17 1.9s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743495 jQsvQu84TgK_wU6Tv7WzFQ:235705838 enrich 1666007117111 11:45:17 1.9s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952350 jQsvQu84TgK_wU6Tv7WzFQ:235705843 enrich 1666007117112 11:45:17 1.9s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743501 jQsvQu84TgK_wU6Tv7WzFQ:235705853 enrich 1666007117121 11:45:17 1.9s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952357 jQsvQu84TgK_wU6Tv7WzFQ:235705858 enrich 1666007117121 11:45:17 1.9s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743507 jQsvQu84TgK_wU6Tv7WzFQ:235705863 enrich 1666007117129 11:45:17 1.9s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952367 jQsvQu84TgK_wU6Tv7WzFQ:235705884 enrich 1666007117133 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743517 jQsvQu84TgK_wU6Tv7WzFQ:235705891 enrich 1666007117139 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952373 jQsvQu84TgK_wU6Tv7WzFQ:235705898 enrich 1666007117139 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743522 jQsvQu84TgK_wU6Tv7WzFQ:235705903 enrich 1666007117145 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952379 jQsvQu84TgK_wU6Tv7WzFQ:235705908 enrich 1666007117145 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743529 jQsvQu84TgK_wU6Tv7WzFQ:235705915 enrich 1666007117151 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952392 jQsvQu84TgK_wU6Tv7WzFQ:235705930 enrich 1666007117160 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743540 jQsvQu84TgK_wU6Tv7WzFQ:235705935 enrich 1666007117168 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952397 jQsvQu84TgK_wU6Tv7WzFQ:235705940 enrich 1666007117174 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743545 jQsvQu84TgK_wU6Tv7WzFQ:235705945 enrich 1666007117182 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952404 jQsvQu84TgK_wU6Tv7WzFQ:235705950 enrich 1666007117183 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743551 jQsvQu84TgK_wU6Tv7WzFQ:235705955 enrich 1666007117189 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952411 jQsvQu84TgK_wU6Tv7WzFQ:235705964 enrich 1666007117192 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952417 jQsvQu84TgK_wU6Tv7WzFQ:235705974 enrich 1666007117197 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743558 jQsvQu84TgK_wU6Tv7WzFQ:235705969 enrich 1666007117198 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743565 jQsvQu84TgK_wU6Tv7WzFQ:235705979 enrich 1666007117204 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952422 jQsvQu84TgK_wU6Tv7WzFQ:235705984 enrich 1666007117211 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743570 jQsvQu84TgK_wU6Tv7WzFQ:235705989 enrich 1666007117218 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952428 jQsvQu84TgK_wU6Tv7WzFQ:235705994 enrich 1666007117218 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution nfaUfK3uST-UxMCKQBDYAQ:219952434 jQsvQu84TgK_wU6Tv7WzFQ:235706004 enrich 1666007117224 11:45:17 1.8s 10.78.7.228 es-002.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743576 jQsvQu84TgK_wU6Tv7WzFQ:235705999 enrich 1666007117224 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev
policy_execution oUHCWUxVSlarbQgtG08Xqg:178743582 jQsvQu84TgK_wU6Tv7WzFQ:235706009 enrich 1666007117229 11:45:17 1.8s 10.78.7.229 es-003.stordata.dev

The thing is, we have integration tests for this behavior and the enrich stats are returning executing policies in these tests. It's only on the production cluster that the endpoint always return an empty list.

I've resorted to using the tasks list for now, but I'd like to hear from you on this issue. Am I doing something wrong?

Cheers,
David

Steps to Reproduce

See above

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[kossde]

This is also happening for me on 7.17.6

[joegallo]

I think this is a consequence of #76881 -- the code we're using for the enrich stats is driven by the policy tasks that are executing on the master node (only). So it works great on single node clusters, but not in clusters with dedicated ingest nodes.