Skip to content

Add CPS ML datafeed troubleshooting docs for serverless - #7403

Open
valeriy42 wants to merge 30 commits into
elastic:mainfrom
valeriy42:docs/cps-ml-datafeed-troubleshooting
Open

Add CPS ML datafeed troubleshooting docs for serverless#7403
valeriy42 wants to merge 30 commits into
elastic:mainfrom
valeriy42:docs/cps-ml-datafeed-troubleshooting

Conversation

@valeriy42

@valeriy42 valeriy42 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

CPS ML datafeed troubleshooting set for Elastic Cloud Serverless:

  • Project scope problems — routing that matches nothing, too wide, stale aliases, or inherited clone scope
  • Linked project unavailable — skipped linked projects fail the extraction cycle; stats and recovery
  • Project scope changes — rejected updates, bulk partial failures, model adaptation after scope stabilizes
  • Cloud credential problems — validate-before-mint, runtime authz, cleared or never-minted internal keys
  • Field mapping conflicts — optional-field warnings, time-field fail-fast, and schema drift across projects

Hub: troubleshoot/elasticsearch/machine-learning-cps.md routes symptoms to the leaf pages and documents project_routing quick reference.

Publish user-facing diagnose/resolve pages for cross-project anomaly detection datafeeds, with a hub under Elasticsearch troubleshoot and links from CPS and ML run-jobs topics.

Co-authored-by: Cursor <cursoragent@cursor.com>
@valeriy42
valeriy42 requested review from a team as code owners July 16, 2026 10:49
@github-actions

Copy link
Copy Markdown
Contributor

Elastic Docs AI PR menu

Check the box to run an AI review for this pull request.

  • Review docs changes (docs-review). Status: not started.

Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team.

Co-authored-by: Cursor <cursoragent@cursor.com>
@valeriy42
valeriy42 marked this pull request as draft July 16, 2026 10:51
The CPS user guide lives under explore-analyze, not deploy-manage.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown
Contributor

Elastic Docs Style Checker (Vale)

Summary: 1 warning, 29 suggestions found

⚠️ Warnings (1): Fix when the suggestion improves clarity or correctness.
File Line Rule Message
troubleshoot/elasticsearch/machine-learning/cps-datafeed-stale-project-reference.md 38 Elastic.Spelling 'misrouted' is a possible misspelling.
💡 Suggestions (29): Optional style improvements. Apply when helpful.
File Line Rule Message
troubleshoot/elasticsearch/machine-learning.md 41 Elastic.Clone Use Clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 2 Elastic.Clone Use Clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 12 Elastic.Clone Use Clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 14 Elastic.Clone Use Cloning only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 14 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 20 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 24 Elastic.Clone Use cloned only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 26 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 30 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 36 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 36 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 38 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 46 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 46 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-clone-project-routing.md 56 Elastic.Clone Use clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-cloud-token-mint-failure.md 40 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-cloud-token-runtime-failure.md 14 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-cloud-token-runtime-failure.md 83 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-field-mapping-mismatch.md 26 Elastic.Semicolons Use semicolons judiciously.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-field-mapping-mismatch.md 26 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-field-mapping-mismatch.md 55 Elastic.Semicolons Use semicolons judiciously.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-routing-no-match.md 14 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-routing-no-match.md 40 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-scope-too-broad.md 43 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-search-scope-changed.md 30 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-search-scope-changed.md 75 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-stale-project-reference.md 14 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-stale-project-reference.md 62 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
troubleshoot/elasticsearch/machine-learning/cps-datafeed-stale-project-reference.md 62 Elastic.Semicolons Use semicolons judiciously.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

valeriy42 added 17 commits July 28, 2026 14:57
Extract reusable diagnostic sources and datafeed update precondition blocks for the consolidated CPS troubleshooting pages.
First-time _alias:_origin assignment bypasses the entire rollback gate, not only the snapshot check. Qualify closed-job and snapshot preconditions accordingly.
Merge four leaf pages into cps-datafeed-project-scope.md with
source-verified routing messages, Kibana Project scope UI labels,
and shared diagnostic/precondition snippets.
Split no-match routing rows by flat-world vs qualified indices per
CredentialTransitions deferral logic; note create uses update wording;
drop redundant stop/close line after preconditions snippet.
Replace cps-datafeed-linked-project-skipped.md with source-verified
page that states fail-the-cycle semantics, snake_case stats fields, and
correct recovery guidance.
Clarify that Job messages are authoritative during extraction errors
and remote_cluster_stats reflects the last completed cycle only.
Merge search-scope-changed and bulk-migration-partial into one page
covering rejected updates, partial bulk updates, and post-change model
impact with source-verified ES messages and Kibana UI labels.
Name the bulk flyout Project scope picker, quote the confirmation dialog
body verbatim, and remove restartRunningJobs jargon for end-user audience.
Replace cloud-token mint and runtime failure pages with a single
credentials page derived from CredentialTransitions and probe
diagnostics, documenting cleared-credential and no-op re-key behavior.
Drop dead fragment links to bold sub-labels, distinguish cleared
vs never-minted when cloud_api_key.id is absent, merge probe-fix
guidance into the re-key section.
Merge field-mapping-mismatch and schema-drift pages into one page
derived from DatafeedFieldConflictDiagnostics and Messages.java.
Tie optional-field warnings to post-scope-change recheck only;
correct mapping-drift symptoms and report-only warning wording.
Rename the hub to machine-learning-cps.md, replace the eleven-item list
with symptom routing to five consolidated leaf pages, fix project_routing
quick reference, update toc.yml, inbound links, and CPS entry points.
Align _alias:* with flat-world resolver behavior, note origin inclusion
on wildcard subsets, and qualify extraction-error symptom routing.
Correct recovery stats example, bulk flyout confirmation order, credential
path count, exact-alias routing row, and model snapshot id example format.
ml-cpp generates snapshot ids via typeToString(CTimeUtils::now()), where
now() returns time(nullptr) — seconds, not milliseconds.
Reorder bulk scope-change confirm vs result icons, add credential
routing for origin-only results, correct Kibana parsed/total count,
split bulk entry points, and remove stray blank line.
- **New projects only:** During technical preview, only newly created projects can function as origin projects.
- **{{anomaly-detect-cap}} and transforms:** During technical preview, ML {{anomaly-jobs}} and transforms are not supported with {{cps-init}}. They continue to run on origin project data only.
- **{{anomaly-detect-cap}}:** {{anomaly-jobs}} can search linked projects. Set the {{dfeed}} scope with `project_routing`; jobs migrated from before {{cps-init}} keep origin-only scope (`_alias:_origin`). Refer to [Troubleshoot {{cps}} {{dfeeds}}](/troubleshoot/elasticsearch/machine-learning-cps.md).
- **Transforms:** Transforms are not supported with {{cps-init}}. They continue to run on origin project data only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll let this be changed as part of the transform related update.

Add symptom and message routing tables on the hub, clarify rollback snapshot examples and bulk scope-change UI warnings, and consolidate shared diagnostic guidance in snippets.
Assume the datafeed update API is available: force re-key without surface
changes, fix frontmatter, and clarify key-id logging in runtime failures.
@valeriy42
valeriy42 requested a review from shainaraskas July 29, 2026 15:31
@valeriy42

Copy link
Copy Markdown
Contributor Author

@shainaraskas this is ML troubleshooting docs that I mentioned. They need to be published when CPS for ML is available on serverless. I am not sure how this works technically.

@shainaraskas

Copy link
Copy Markdown
Member

@shainaraskas this is ML troubleshooting docs that I mentioned. They need to be published when CPS for ML is available on serverless. I am not sure how this works technically.

I'll try to review this today. Re: process, for serverless content, we need to time the PRs with prod availability. Within an hour or so of when something makes it to prod is what we aim for. If you can give me an ETA I can make sure that this PR is prepped in time.

@shainaraskas shainaraskas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi, sorry I'm late coming back to this.

I can see a lot of value in these troubleshooting pages, but I think they're a little broadly scoped because we can't rely on core conceptual docs to do some of the heavy lifting. my next job is trying to add some conceptual detail about CPS to the ML docs, then we can come back to this and perhaps xlink out rather than describe certain things in detail here.

Overall, I think the troubleshooting pages could use a little restructure so we don't have only one symptom block and one resolution block - instead, symptoms and resolutions should be paired where possible. we should also lean on heading levels instead of bolded sections. there are also some style nits but I think we can leave these alone for now.

let me know when you have an idea of when this feature will be shipping. it's my focus for now, but a date will make it easier to make sure everything is ready in time

@@ -0,0 +1,143 @@
§§§§§§§§§§§§§§§§§§§§§§§---

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
§§§§§§§§§§§§§§§§§§§§§§§---
---

- **System indices:** Indices such as `.security` and `.fleet-*` are excluded from {{cps}} results by design.
- **New projects only:** During technical preview, only newly created projects can function as origin projects.
- **{{anomaly-detect-cap}} and transforms:** During technical preview, ML {{anomaly-jobs}} and transforms are not supported with {{cps-init}}. They continue to run on origin project data only.
- **{{anomaly-detect-cap}}:** {{anomaly-jobs}} can search linked projects. Set the {{dfeed}} scope with `project_routing`. Jobs migrated from before {{cps-init}} keep origin-only scope (`_alias:_origin`). Refer to [Troubleshoot {{cps}} {{dfeeds}}](/troubleshoot/elasticsearch/machine-learning-cps.md).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to focus on the limitation in this context

Suggested change
- **{{anomaly-detect-cap}}:** {{anomaly-jobs}} can search linked projects. Set the {{dfeed}} scope with `project_routing`. Jobs migrated from before {{cps-init}} keep origin-only scope (`_alias:_origin`). Refer to [Troubleshoot {{cps}} {{dfeeds}}](/troubleshoot/elasticsearch/machine-learning-cps.md).
- **{{anomaly-detect-cap}}:** {{anomaly-jobs}} created before {{cps-init}} was enabled default to origin-only scope (`_alias:_origin`) and must be manually updated to search linked projects. Refer to [Troubleshoot {{cps}} {{dfeeds}}](/troubleshoot/elasticsearch/machine-learning-cps.md).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one might also need to be refactored when we have best practices and considerations closer to the anomaly detection content - consider removing for now


# Troubleshoot cross-project search {{dfeeds}} [cps-ml-datafeed-troubleshooting]

Anomaly detection {{dfeeds}} on {{serverless-full}} can search data across linked projects when {{cps}} is configured. These topics help you diagnose and resolve problems with **Project scope** (`project_routing`), internal cloud credentials, linked-project availability, and field mappings.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Anomaly detection {{dfeeds}} on {{serverless-full}} can search data across linked projects when {{cps}} is configured. These topics help you diagnose and resolve problems with **Project scope** (`project_routing`), internal cloud credentials, linked-project availability, and field mappings.
Anomaly detection {{dfeeds}} on {{serverless-full}} can search data across linked projects when {{cps}} is configured. These topics help you diagnose and resolve problems with project scope (`project_routing`), internal cloud credentials, linked-project availability, and field mappings.

@@ -0,0 +1,12 @@
**Where to look**

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Where to look**
Use these sources to gather diagnostic information:

@@ -0,0 +1,12 @@
**Where to look**

* **{{anomaly-job}} job messages in {{kib}}**: Open **Machine Learning → Anomaly Detection**, select the job, and review the **Job messages** tab for audit entries and warnings about linked projects, credentials, or scope changes. On the **Datafeed** tab, **View datafeed counts** opens the datafeed chart flyout for extraction timing.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need a new var to use this phrasing - need to create in docset.yml

Suggested change
* **{{anomaly-job}} job messages in {{kib}}**: Open **Machine Learning → Anomaly Detection**, select the job, and review the **Job messages** tab for audit entries and warnings about linked projects, credentials, or scope changes. On the **Datafeed** tab, **View datafeed counts** opens the datafeed chart flyout for extraction timing.
* **{{anomaly-job-cap}} job messages in {{kib}}**: Open **Machine Learning → Anomaly Detection**, select the job, and review the **Job messages** tab for audit entries and warnings about linked projects, credentials, or scope changes. On the **Datafeed** tab, **View datafeed counts** opens the datafeed chart flyout for extraction timing.

Comment on lines +778 to +780
## Related pages

* [Troubleshoot cross-project search {{dfeeds}}](/troubleshoot/elasticsearch/machine-learning-cps.md): Diagnose and resolve anomaly detection {{dfeeds}} that search across linked projects.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Related pages
* [Troubleshoot cross-project search {{dfeeds}}](/troubleshoot/elasticsearch/machine-learning-cps.md): Diagnose and resolve anomaly detection {{dfeeds}} that search across linked projects.

- id: machine-learning
---

# Project scope problems [cps-datafeed-project-scope]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this title needs to be more specific

no need for anchors on titles (will fix the helpers to stop doing that)

Suggested change
# Project scope problems [cps-datafeed-project-scope]
# Troubleshoot anomaly detection datafeed project scope

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This page covers four distinct problems but groups them under two mega-sections (Diagnose, Resolve). Readers have to mentally map which fix applies to which scenario. I suggest that you restructure so each problem is self-contained (what you see, fix, verify):

  • Routing matches no project
  • Scope is wider than intended
  • Stale project reference
  • Cloned job inherits wrong scope

Consider moving the shared preconditions (stop datafeed, rollback gate) into a "Before you update" section at the top because they apply to all four problems.

Comment on lines +20 to +21

**Check the effective scope in {{kib}}**

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer proper headings over bold

Suggested change
**Check the effective scope in {{kib}}**
### Check the effective scope in {{kib}}


**Check the effective scope in {{kib}}**

Open **Machine Learning → Anomaly Detection** and review the **Project scope** column in the {{anomaly-jobs}} list. Each cell shows a parsed/total count (for example, `2/5`) derived from the stored routing expression and the total project count (origin plus linked projects), not from live resolution of which aliases match. {{kib}} parses the segment after `_alias:`: omitted or `null` routing shows `1`. `_alias:*` shows the origin-plus-linked total. A single expression such as `_alias:production-*` or `_alias:_origin` shows `1`. Because the count comes from parsing rather than resolution, a wildcard that matches several projects still shows `1`. Select the count to open a popover titled **Project scope** that displays the routing expression stored on the {{dfeed}}. A legacy {{anomaly-job}} with no stored routing shows `_alias:_origin` in the popover.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we optimize this to focus on the gotcha? wonder if this info should move to the conceptual docs because this is documenting a field in the UI

Suggested change
Open **Machine Learning Anomaly Detection** and review the **Project scope** column in the {{anomaly-jobs}} list. Each cell shows a parsed/total count (for example, `2/5`) derived from the stored routing expression and the total project count (origin plus linked projects), not from live resolution of which aliases match. {{kib}} parses the segment after `_alias:`: omitted or `null` routing shows `1`. `_alias:*` shows the origin-plus-linked total. A single expression such as `_alias:production-*` or `_alias:_origin` shows `1`. Because the count comes from parsing rather than resolution, a wildcard that matches several projects still shows `1`. Select the count to open a popover titled **Project scope** that displays the routing expression stored on the {{dfeed}}. A legacy {{anomaly-job}} with no stored routing shows `_alias:_origin` in the popover.
Open **Machine Learning > Anomaly Detection** and review the **Project scope** column. Each cell shows a parsed count out of the total project count (origin plus linked projects). For example, `2/5` means the expression targets 2 projects out of 5 available.
The parsed count comes from the routing expression text, not from resolving which aliases actually match at runtime:
- Omitted or `null` routing shows `1` (origin only).
- `_alias:*` shows the full origin-plus-linked total.
- `_alias:_origin` shows `1`.
- A single wildcard expression like `_alias:production-*` shows `1`, even if it matches several linked projects at runtime.
Select the count to open a popover that displays the routing expression stored on the datafeed. A legacy job with no stored routing shows `_alias:_origin` in the popover.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants