[ML] Datafeed preview request rejects job configuration containing unknown `job_type` field

[KodeRad]

Elasticsearch Version

Latest

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin Konrad-elastics-MacBook-Pro 24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:40 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6041 arm64

Problem Description

I initially created an issue in Kibana which describes what is happening. Upon further investigation it looks like it might be an elasticsearch problem with schemas and expected fields.

Fields that are causing issues:

job_type,
job_version,
create_time,
finished_time,
model_snapshot_id

Steps to Reproduce

1. Run this API call in Dev Tools:

GET _ml/datafeeds/_preview
{
  "job_config":{
    "job_id": "",
    "description": "",
    "job_type": "anomaly_detector",
    "groups": [],
    "analysis_config": {
      "bucket_span": "15m",
      "detectors": [
        {
          "function": "count"
        }
      ],
      "influencers": []
    },
    "data_description": {
      "time_field": "@timestamp"
    },
    "custom_settings": {
      "created_by": "advanced-wizard"
    },
    "analysis_limits": {
      "model_memory_limit": "11MB"
    }
  },
  "datafeed_config":{
    "datafeed_id": "",
    "job_id": "",
    "indices": [
      "fq"
    ],
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "runtime_mappings": {}
  }
}

2. Observe the error

Logs (if relevant)

{
  "error": {
    "root_cause": [
      {
        "type": "x_content_parse_exception",
        "reason": "[1:61] [job_details] unknown field [job_type]"
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[1:73] [preview_datafeed_action] failed to parse field [job_config]",
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "[1:61] [job_details] unknown field [job_type]"
    }
  },
  "status": 400
}

[jgowdyelastic]

Some additional context..
Some of these properties I'd expect to not be supported, e.g. create_time, finished_time or model_snapshot_id. However according to the docs job_type should be supported.

We can just omit all of these properties in kibana when calling the api, as we're doing in this new PR. If this is an acceptable solution, I'd be fine with closing this elasticsearch PR and just working round the problem in kibana.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[jan-elastic]

Related PRs:

• Revert "Revert "Exclude internal fields from job APIs. (#106115)" (#106277)
• Revert "Exclude internal fields from job APIs. (#106115)"
• Exclude internal fields from job APIs.

[jan-elastic]

I think the field job_type and the other fields that are causing issues are considered internal fields, and should not be sent to the API. The documentation about job_type seems wrong.

[jan-elastic]

Not sure how this request was created. For an existing job, you can obtain the config without the internal fields with:

GET _ml/anomaly_detectors/my-test-job/?exclude_generated

[jan-elastic]

According to this PR:

I've confirmed with @jgowdyelastic that these fields are not used in the production code. They are never explicitly set and when cloning a job, the get job API is called with the query param exclude_generated=true.