[ResponseOps] Granting kibana_system role access to the cases analytics indices
#129414
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adcoelho
added
:Security/Authorization
|
Pinging @elastic/es-security (Team:Security) |
cnasikas
reviewed
Jun 13, 2025
...va/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java
Outdated
Show resolved
Hide resolved
adcoelho
force-pushed
the
cases-analytics-index
branch
from
404f1a9 to
c0fcec1
Compare
adcoelho
force-pushed
the
cases-analytics-index
branch
from
c0fcec1 to
bc4a819
Compare
adcoelho
added a commit
to elastic/kibana
that referenced
this pull request
Jun 17, 2025
**Merging into a feature branch** ## Summary This PR adds the `.internal.cases-activity` analytics index. ## How to test 1. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` 2. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 3. Start Elastic Search with `yarn es source`. 4. Add a bunch of activity to your cases. Edit tags, category, etc. 5. Go to the dev tools and confirm this activity shows up in the index - `GET /.internal.cases-activity/_search`.
|
@elasticmachine update branch |
Create constants for indexes and aliases. Update tests.
cnasikas
changed the title
kibana_system reserved role access to the cases analytics indices
cnasikas
changed the title
kibana_system reserved role access to the cases analytics indiceskibana_system role access to the cases analytics indices
azasypkin
approved these changes
Jun 18, 2025
There was a problem hiding this comment.
The change is aligned with what has been discussed in the relevant RFC and hence LGTM.
I see the scripts we store are static/const, do we plan to compose these scripts dynamically in the near future?
If no, I have a question for @elastic/es-security: is there any precedent where we define a painless script used by some Elastic Stack component (e.g., Kibana) directly in ES so that we don't need to grant cluster:admin/script/put to a service system account to store such a "predefined" script?
The scripts are created statically, but our logic relies on the capacity to create new scripts to handle different Kibana nodes writing to these indexes concurrently.
If I get your suggestion correctly, this would also complicate our deployment process whenever we want to update mappings and scripts. |
Assuming it's possible in principle to have predefined scripts in ES, it'd make changes a bit more involved (would require a PR against the ES repo), yes, but I'm not sure if it complicates anything, to be honest, as you won't need to worry about this at Kibana startup and can assume that any given ES version already has the script with the content and ID you need for this version. That being said, not saying it's better or worse, just exploring alternatives. |
|
Thanks @azasypkin for the review and the feedback! To be honest, I am not sure if we will compose the scripts dynamically in the future. We may need to. Aside from that, the sync tasks will have some recovery mechanism to recreate the scripts in case of a catastrophic failure (user tampered with them, for example). We would need the scripts to be part of Kibana to do that, so they can recreate them unless it is implemented on the ES side. Also, when a Kibana node updates the mappings, it also updates the scripts. There is a one-to-one correspondence between the mapping and the scripts. Coordinating ES PRs with Kibana PRs may be challenging. Lastly, I am not sure how ownership works in ES and what the review process is. Do you mind if we move forward with this PR (unless there are security concerns) and explore alternatives after FF? I am open to discussing and seeing what approach makes more sense. |
slobodanadamovic
requested review from
slobodanadamovic
and removed request for
a team
Same as in Kibana :) For example, AppEx Security co-owns a role definition for the Kibana System user in Elasticsearch and handles reviews, and you'd co-own script definitions (again, if it's possible to do something like this at all - i.e., having ability to define a "built-in/system" script that users cannot mess up with).
Definitely, I've approved the PR, meaning that's not a blocking suggestion, but more of a thing we could potentially explore now or in the future (assuming it has enough benefits from functional, maintenance, and security standpoints). |
|
Thanks @azasypkin! Yes, let's explore the alternatives. I would love that. |
Comment on lines
+103
to
+104
| "cluster:admin/script/put", | ||
| "cluster:admin/script/get", |
There was a problem hiding this comment.
nit: You can reference actual action constants here.
Suggested change
| "cluster:admin/script/put", | |
| "cluster:admin/script/get", | |
| TransportPutStoredScriptAction.TYPE.name(), | |
| GetStoredScriptAction.NAME, |
slobodanadamovic
approved these changes
Jun 19, 2025
kderusso
pushed a commit
to kderusso/elasticsearch
that referenced
this pull request
Jun 23, 2025
…tics indices (elastic#129414) * Update kibana system user permissions for cases analytics index. * [CI] Auto commit changes from spotless * Add permissions for aliases. Create constants for indexes and aliases. Update tests. * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
adcoelho
added a commit
to elastic/kibana
that referenced
this pull request
Jun 24, 2025
This PR is for a feature branch that is being merged into main. The relevant PRs are: - #219211 - #222820 - #223241 - #224388 - #224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
adcoelho
added a commit
to adcoelho/kibana
that referenced
this pull request
Jun 24, 2025
This PR is for a feature branch that is being merged into main. The relevant PRs are: - elastic#219211 - elastic#222820 - elastic#223241 - elastic#224388 - elastic#224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co> (cherry picked from commit e566fec) # Conflicts: # x-pack/platform/plugins/shared/task_manager/server/mocks.ts
adcoelho
added a commit
to elastic/kibana
that referenced
this pull request
Jun 24, 2025
# Backport This will backport the following commits from `main` to `8.19`: - [[ResponseOps] Cases analytics index (#223405)](#223405) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Antonio","email":"antonio.coelho@elastic.co"},"sourceCommit":{"committedDate":"2025-06-24T06:46:32Z","message":"[ResponseOps] Cases analytics index (#223405)\n\nThis PR is for a feature branch that is being merged into main.\n\nThe relevant PRs are:\n- https://github.com/elastic/kibana/pull/219211\n- https://github.com/elastic/kibana/pull/222820\n- https://github.com/elastic/kibana/pull/223241\n- https://github.com/elastic/kibana/pull/224388\n- https://github.com/elastic/kibana/pull/224682\n\n## Summary\n\nThis PR adds 4 new indexes with case analytics data, which are created\nwhen the cases plugin starts.\n\n - `.internal.cases`\n - `.internal.cases-comments`\n - `.internal.cases-attachments`\n - `.internal.cases-activity`\n\nAfter the indexes are created, a backfill task for each of them is\nscheduled to run 1 minute after creation. This task populates the\nindexes with relevant data from `.kibana_alerting_cases`.\n\nA second type of task is registered, the index synchronization task.\nFour of these tasks, one for each index, are scheduled to run every 5\nminutes. The synchronization tasks populated the indexes with data from\n`.kibana_alerting_cases` that was created or updated in the last five\nminutes.\n\n## How to test\n\nYou might want to start Kibana with `--verbose` to see relevant index\nmessages in the console.\n\nAlternatively(what I normally do), is go to `analytics_index.ts`,\n`backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and\nchange the `logDebug` function to call `this.logger.info` instead. This\nway, you will have less spam in the console.\n\nEvery log message starts with the index name between square brackets, so\nyou can look for `[.internal.cases-` and follow what is happening.\n\n1. You should have some existing case data, so before anything else,\nplease create some activity, attachments, etc.\n2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml`\nand restart Kibana.\n3. Check out [this\nbranch](elastic/elasticsearch#129414) from the\nES project.\n4. Start Elastic Search with `yarn es source`. This will use the above\nversion of Elasticsearch.\n5. Wait a bit for the indexes to be created and populated(backfilled).\n6. Using the dev tools:\n - Confirm the indexes exist.\n- Check the index mapping. Does it match the one in the code? Is the\n`_meta` field correct?\n-\n`x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts`\n - Check that the painless scripts match the code.\n-\n`x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts`\n- Confirm your existing case data is in the indexes. (See **Queries**\nsection below.)\n7. Play around with cases. Some examples:\n - Create a case\n - Change status/severity\n - Attach alerts\n - Add files\n - Change category/tags\n - Add comments\n - etc\n8. Go to the dev tools again and confirm all this shows up in the\nrelevant indexes. (See **Queries** section below.)\n\n## Queries\n\nIn addition to the ones, below I have a few more. Things like reindexing\nwith specific scripts or fetching relevant data from\n`.kibana_alerting_cases`. Ping me if you want those queries.\n\n### Checking index content\n```\nGET /.internal.cases/_search\nGET /.internal.cases-comments/_search\nGET /.internal.cases-attachments/_search\nGET /.internal.cases-activity/_search\n```\n\n### Checking index mappings\n```\nGET /.internal.cases\nGET /.internal.cases-comments\nGET /.internal.cases-attachments\nGET /.internal.cases-activity\n```\n\n### Fetching the painless scripts\n```\nGET /_scripts/cai_cases_script_1\nGET /_scripts/cai_attachments_script_1\nGET /_scripts/cai_comments_script_1\nGET /_scripts/cai_activity_script_1\n```\n\n### Emptying the indexes\n\nIt is sometimes useful for testing.\n```\nPOST /.internal.cases/_delete_by_query\nPOST /.internal.cases-comments/_delete_by_query\nPOST /.internal.cases-attachments/_delete_by_query\nPOST /.internal.cases-activity/_delete_by_query\n```\n\n### Deleting the indexes\n\nIt is sometimes useful for testing.\n```\nDELETE /.internal.cases\nDELETE /.internal.cases-comments\nDELETE /.internal.cases-attachments\nDELETE /.internal.cases-activity\n```\n\n## Release notes\n\nFour dedicated case analytics indexes were created, allowing users to\nbuild dashboards and metrics over case data. These indexes are created\non Kibana startup and updated periodically with cases, comments,\nattachments, and activity data.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\nCo-authored-by: Christos Nasikas <christos.nasikas@elastic.co>","sha":"e566fec14b824f2e3c28a16dbfbcfa622b721b4a","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:ResponseOps","Feature:Cases","release_note:feature","ci:cloud-deploy","backport:version","v9.1.0","v8.19.0"],"title":"[ResponseOps] Cases analytics index","number":223405,"url":"https://github.com/elastic/kibana/pull/223405","mergeCommit":{"message":"[ResponseOps] Cases analytics index (#223405)\n\nThis PR is for a feature branch that is being merged into main.\n\nThe relevant PRs are:\n- https://github.com/elastic/kibana/pull/219211\n- https://github.com/elastic/kibana/pull/222820\n- https://github.com/elastic/kibana/pull/223241\n- https://github.com/elastic/kibana/pull/224388\n- https://github.com/elastic/kibana/pull/224682\n\n## Summary\n\nThis PR adds 4 new indexes with case analytics data, which are created\nwhen the cases plugin starts.\n\n - `.internal.cases`\n - `.internal.cases-comments`\n - `.internal.cases-attachments`\n - `.internal.cases-activity`\n\nAfter the indexes are created, a backfill task for each of them is\nscheduled to run 1 minute after creation. This task populates the\nindexes with relevant data from `.kibana_alerting_cases`.\n\nA second type of task is registered, the index synchronization task.\nFour of these tasks, one for each index, are scheduled to run every 5\nminutes. The synchronization tasks populated the indexes with data from\n`.kibana_alerting_cases` that was created or updated in the last five\nminutes.\n\n## How to test\n\nYou might want to start Kibana with `--verbose` to see relevant index\nmessages in the console.\n\nAlternatively(what I normally do), is go to `analytics_index.ts`,\n`backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and\nchange the `logDebug` function to call `this.logger.info` instead. This\nway, you will have less spam in the console.\n\nEvery log message starts with the index name between square brackets, so\nyou can look for `[.internal.cases-` and follow what is happening.\n\n1. You should have some existing case data, so before anything else,\nplease create some activity, attachments, etc.\n2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml`\nand restart Kibana.\n3. Check out [this\nbranch](elastic/elasticsearch#129414) from the\nES project.\n4. Start Elastic Search with `yarn es source`. This will use the above\nversion of Elasticsearch.\n5. Wait a bit for the indexes to be created and populated(backfilled).\n6. Using the dev tools:\n - Confirm the indexes exist.\n- Check the index mapping. Does it match the one in the code? Is the\n`_meta` field correct?\n-\n`x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts`\n - Check that the painless scripts match the code.\n-\n`x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts`\n- Confirm your existing case data is in the indexes. (See **Queries**\nsection below.)\n7. Play around with cases. Some examples:\n - Create a case\n - Change status/severity\n - Attach alerts\n - Add files\n - Change category/tags\n - Add comments\n - etc\n8. Go to the dev tools again and confirm all this shows up in the\nrelevant indexes. (See **Queries** section below.)\n\n## Queries\n\nIn addition to the ones, below I have a few more. Things like reindexing\nwith specific scripts or fetching relevant data from\n`.kibana_alerting_cases`. Ping me if you want those queries.\n\n### Checking index content\n```\nGET /.internal.cases/_search\nGET /.internal.cases-comments/_search\nGET /.internal.cases-attachments/_search\nGET /.internal.cases-activity/_search\n```\n\n### Checking index mappings\n```\nGET /.internal.cases\nGET /.internal.cases-comments\nGET /.internal.cases-attachments\nGET /.internal.cases-activity\n```\n\n### Fetching the painless scripts\n```\nGET /_scripts/cai_cases_script_1\nGET /_scripts/cai_attachments_script_1\nGET /_scripts/cai_comments_script_1\nGET /_scripts/cai_activity_script_1\n```\n\n### Emptying the indexes\n\nIt is sometimes useful for testing.\n```\nPOST /.internal.cases/_delete_by_query\nPOST /.internal.cases-comments/_delete_by_query\nPOST /.internal.cases-attachments/_delete_by_query\nPOST /.internal.cases-activity/_delete_by_query\n```\n\n### Deleting the indexes\n\nIt is sometimes useful for testing.\n```\nDELETE /.internal.cases\nDELETE /.internal.cases-comments\nDELETE /.internal.cases-attachments\nDELETE /.internal.cases-activity\n```\n\n## Release notes\n\nFour dedicated case analytics indexes were created, allowing users to\nbuild dashboards and metrics over case data. These indexes are created\non Kibana startup and updated periodically with cases, comments,\nattachments, and activity data.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\nCo-authored-by: Christos Nasikas <christos.nasikas@elastic.co>","sha":"e566fec14b824f2e3c28a16dbfbcfa622b721b4a"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/223405","number":223405,"mergeCommit":{"message":"[ResponseOps] Cases analytics index (#223405)\n\nThis PR is for a feature branch that is being merged into main.\n\nThe relevant PRs are:\n- https://github.com/elastic/kibana/pull/219211\n- https://github.com/elastic/kibana/pull/222820\n- https://github.com/elastic/kibana/pull/223241\n- https://github.com/elastic/kibana/pull/224388\n- https://github.com/elastic/kibana/pull/224682\n\n## Summary\n\nThis PR adds 4 new indexes with case analytics data, which are created\nwhen the cases plugin starts.\n\n - `.internal.cases`\n - `.internal.cases-comments`\n - `.internal.cases-attachments`\n - `.internal.cases-activity`\n\nAfter the indexes are created, a backfill task for each of them is\nscheduled to run 1 minute after creation. This task populates the\nindexes with relevant data from `.kibana_alerting_cases`.\n\nA second type of task is registered, the index synchronization task.\nFour of these tasks, one for each index, are scheduled to run every 5\nminutes. The synchronization tasks populated the indexes with data from\n`.kibana_alerting_cases` that was created or updated in the last five\nminutes.\n\n## How to test\n\nYou might want to start Kibana with `--verbose` to see relevant index\nmessages in the console.\n\nAlternatively(what I normally do), is go to `analytics_index.ts`,\n`backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and\nchange the `logDebug` function to call `this.logger.info` instead. This\nway, you will have less spam in the console.\n\nEvery log message starts with the index name between square brackets, so\nyou can look for `[.internal.cases-` and follow what is happening.\n\n1. You should have some existing case data, so before anything else,\nplease create some activity, attachments, etc.\n2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml`\nand restart Kibana.\n3. Check out [this\nbranch](elastic/elasticsearch#129414) from the\nES project.\n4. Start Elastic Search with `yarn es source`. This will use the above\nversion of Elasticsearch.\n5. Wait a bit for the indexes to be created and populated(backfilled).\n6. Using the dev tools:\n - Confirm the indexes exist.\n- Check the index mapping. Does it match the one in the code? Is the\n`_meta` field correct?\n-\n`x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts`\n - Check that the painless scripts match the code.\n-\n`x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts`\n- Confirm your existing case data is in the indexes. (See **Queries**\nsection below.)\n7. Play around with cases. Some examples:\n - Create a case\n - Change status/severity\n - Attach alerts\n - Add files\n - Change category/tags\n - Add comments\n - etc\n8. Go to the dev tools again and confirm all this shows up in the\nrelevant indexes. (See **Queries** section below.)\n\n## Queries\n\nIn addition to the ones, below I have a few more. Things like reindexing\nwith specific scripts or fetching relevant data from\n`.kibana_alerting_cases`. Ping me if you want those queries.\n\n### Checking index content\n```\nGET /.internal.cases/_search\nGET /.internal.cases-comments/_search\nGET /.internal.cases-attachments/_search\nGET /.internal.cases-activity/_search\n```\n\n### Checking index mappings\n```\nGET /.internal.cases\nGET /.internal.cases-comments\nGET /.internal.cases-attachments\nGET /.internal.cases-activity\n```\n\n### Fetching the painless scripts\n```\nGET /_scripts/cai_cases_script_1\nGET /_scripts/cai_attachments_script_1\nGET /_scripts/cai_comments_script_1\nGET /_scripts/cai_activity_script_1\n```\n\n### Emptying the indexes\n\nIt is sometimes useful for testing.\n```\nPOST /.internal.cases/_delete_by_query\nPOST /.internal.cases-comments/_delete_by_query\nPOST /.internal.cases-attachments/_delete_by_query\nPOST /.internal.cases-activity/_delete_by_query\n```\n\n### Deleting the indexes\n\nIt is sometimes useful for testing.\n```\nDELETE /.internal.cases\nDELETE /.internal.cases-comments\nDELETE /.internal.cases-attachments\nDELETE /.internal.cases-activity\n```\n\n## Release notes\n\nFour dedicated case analytics indexes were created, allowing users to\nbuild dashboards and metrics over case data. These indexes are created\non Kibana startup and updated periodically with cases, comments,\nattachments, and activity data.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\nCo-authored-by: Christos Nasikas <christos.nasikas@elastic.co>","sha":"e566fec14b824f2e3c28a16dbfbcfa622b721b4a"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
mridula-s109
pushed a commit
to mridula-s109/elasticsearch
that referenced
this pull request
Jun 25, 2025
…tics indices (elastic#129414) * Update kibana system user permissions for cases analytics index. * [CI] Auto commit changes from spotless * Add permissions for aliases. Create constants for indexes and aliases. Update tests. * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
Jun 25, 2025
This PR is for a feature branch that is being merged into main. The relevant PRs are: - elastic#219211 - elastic#222820 - elastic#223241 - elastic#224388 - elastic#224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
|
Manual backport PR for 8.19 #131288. |
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 15, 2025
…tics indices (#129414) (#131288) * Update kibana system user permissions for cases analytics index. * [CI] Auto commit changes from spotless * Add permissions for aliases. Create constants for indexes and aliases. Update tests. * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
The ResponseOps team is currently working on creating(and populating) dedicated cases analytics indexes during Kibana startup.
This is the feature branch draft PR for reference.
The Kibana System role did not have the privileges needed for this, so in this PR, I added the following:
.internal.cases*: all- so that the cases analytics indexes can be created/updated/deletedcluster:admin/script/put + cluster:admin/script/get- our logic for synchronizing the indexes with the cases' index requires the usage of painless scripts that need to be fetched and updated.