[Transform] Healthy transforms are shown errors on kibana UI: "Found task for transform [...], but no configuration for it. To delete this transform use DELETE with force=true."

[wwang500]

In some cases, when running GET _transform/_all or through kibana UI (management/data/transform), it shows some transforms are thrown errors:
Found task for transform [...], but no configuration for it. To delete this transform use DELETE with force=true.,

However, it could be wrong and false positive, when you run GET _transform/<transform id> or GET _transform/<transform id>/_stats for the problematic transform, the stats was in green health, and the documents_processed counts are increasing.

We need to figure out what could cause this problem.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[prwhelan]

Can repro:

    public void testGetTransformsDoesNotErrorOnPageSize() throws Exception {
        var transformId1 = "multiple-transforms-1";
        var transformSrc = "reviews_multiple_transforms_test";
        createReviewsIndex(transformSrc);
        createAndStartTransform(transformId1, transformSrc, transformId1 + "_idx");

        var transformId2 = "multiple-transforms-2";
        createAndStartTransform(transformId2, transformSrc, transformId2 + "_idx");

        // get a single transform from `_all`
        var request = new Request("GET", getTransformEndpoint() + "_all?size=1");
        var response = adminClient().performRequest(request);
        var transforms = entityAsMap(response);
        var transformConfigs = (List<Map<String, Object>>) XContentMapValues.extractValue("transforms", transforms);
        var errors = (List<Map<String, String>>) XContentMapValues.extractValue("errors", transforms);
        assertThat(errors, is(nullValue()));
        assertThat(transformConfigs, hasSize(2));

        var actualIds = transformConfigs.stream().map(config -> (String) config.get("id")).toList();
        assertThat(actualIds, containsInAnyOrder(transformId1, transformId2));
    }

This happens if we have N number of Transforms running, but we call GET _transform/_all with a size < N. So if we have 2 transforms and query _transform/_all?size=1, we will say the second transform is running without a config (which is incorrect):
https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/transform/src/main/java/org/elasticsearch/xpack/transform/action/TransportGetTransformAction.java#L83

[jasonrhodes]

Thanks for finding this! Do we have any sense of how often this could happen for customers, or what the conditions are that cause the ?size mismatch that triggers this message and how common those conditions might be? Because it encourages folks to delete transforms that will break their SLOs, it would be good for us to understand the risk profile for incoming SDHs and to watch for this, and to probably publish a "known issue" too.

[jasonrhodes]

@wwang500 @prwhelan can one of you (or someone else) answer the above question, and also give us a rough timeline of when this fix will land? This is a critical bug in our view.

In addition, we need to communicate via the "Known Issues" and knowledge base tools so that customers are aware of this bug, which will continue to exist in some number of versions. Can you help me identify which versions this affects, i.e. when it was introduced in addition to when the fix will land?

[prwhelan]

Do we have any sense of how often this could happen for customers, or what the conditions are that cause the ?size mismatch that triggers this message and how common those conditions might be?

@jasonrhodes this happens when the ?size is any value smaller than the current number of running transforms, so if there are 2 transforms and you request ?size=1 then the error message will display.

also give us a rough timeline of when this fix will land?

I'll work on it sometime this week. I apologize for the delay, I was on vacation last week and the first part of this week. The fix will be in 8.x and 9.x.

In addition, we need to communicate via the "Known Issues" and knowledge base tools so that customers are aware of this bug, which will continue to exist in some number of versions. Can you help me identify which versions this affects, i.e. when it was introduced in addition to when the fix will land?

I can work on the KB article after I work on the code change (or someone else can work on it in parallel). This affects versions 8.1+

[jasonrhodes]

Thanks, @prwhelan!

this happens when the ?size is any value smaller than the current number of running transforms, so if there are 2 transforms and you request ?size=1 then the error message will display.

What controls what the size value is on the transforms UI page?

[prwhelan]

What controls what the size value is on the transforms UI page?

That is a question better asked of the @elastic/kibana-management team. As I understand it, it is fixed to 1000, but I could be wrong