Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove fields from UA mappings which don't need to be searchable #64547

Closed
cjcenizal opened this issue Apr 27, 2020 · 3 comments · Fixed by #151116
Closed

Remove fields from UA mappings which don't need to be searchable #64547

cjcenizal opened this issue Apr 27, 2020 · 3 comments · Fixed by #151116
Assignees
@yuliacech
Labels
chore Feature:Upgrade Assistant Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more

Comments

@cjcenizal
Copy link
Contributor

cjcenizal commented Apr 27, 2020 •
edited
Loading

Per convo with @joshdover, Upgrade Assistant's saved object mappings currently define (and therefore index) a number of fields which don't need to be mapped (see #64332).

We can set dynamic: false and then remove these fields from the mapping to avoid hitting the soft limit on the number of indexed fields in the .kibana index. These fields won't be indexed (searchable) but they'll still be persisted in _source.

It looks like the only fields that need to be indexed are indexName and status based on the fields originally defined in the mappings in the pre-NP code, though everything was being mapped due to the original use of dynamic: true, so there's a possibility that code was depending on those dynamically mapped fields somewhere. When we make this change we should do some testing with Josh's help to make sure we haven't broken anything.
@cjcenizal cjcenizal added chore Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more Feature:Upgrade Assistant labels Apr 27, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/es-ui (Team:Elasticsearch UI)

@cjcenizal cjcenizal mentioned this issue Apr 27, 2020
Merged
@cjcenizal
Copy link
Contributor Author

This will address #43673.

@timroes timroes mentioned this issue Jun 29, 2020
Closed
27 tasks
@cjcenizal cjcenizal mentioned this issue Jul 15, 2020
Merged
@rudolf
Copy link
Contributor

rudolf commented Aug 12, 2022

I wonder if we can prioritise this work, UA is currently using 18 fields for the upgrade-assistant-reindex-operation type and only using two of these?

This was referenced Feb 10, 2023
Merged
Merged
Merged
yuliacech added a commit that referenced this issue Feb 21, 2023
@yuliacech
[Upgrade Assistant] Removes unnecessary mappings for reindexing (#150878
eaa0884 
)

## Summary
Partially addresses #64547

This PR removes unnecessary mappings from the saved objects type
`upgrade-assistant-reindex-operation` that holds information about
reindex processes.
Currently, the reindexing is disabled via a config for Upgrade Assistant
plugin `xpack.upgrade_assistant.featureSet.reindexCorrectiveActions`. ES
also doesn't log any deprecations for indices created since 7.0 and
indices created before 7.0 are incompatible with versions 8.0+. That
means that when reindexing will be enabled in UA, it needs to be
configured for the specific upgrade scenario, for example for 9.0.

### How to test
Even though reindexing is currently disabled, I was able to do some
testing for the changes. Here is how to test reindexing locally:
1. Add `xpack.upgrade_assistant.featureSet.reindexCorrectiveActions:
true` to your `kibana.dev.yml` file
2. Start ES and Kibana with `yarn es snapshot --license=trial` and `yarn
start`
3. Create a test user with the role `system_indices_superuser` (in Stack
Management -> Security -> Users). Log in with that user for further
steps.
4. Create an index via Dev Tools 
```
POST /old_index/_doc
{
  "test": "test"
}
```
5. replace `deprecations` on line 22 in the file
[`/server/lib/es_deprecations_status.ts`](https://github.com/elastic/kibana/blob/main/x-pack/plugins/upgrade_assistant/server/lib/es_deprecations_status.ts#L22)
with
```
  const deprecations = {
    cluster_settings: [],
    node_settings: [],
    ml_settings: [],
    index_settings: {
      old_index: [
        {
          level: 'critical',
          message: 'Index created before 7.0',
          url: 'https: //www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-8.0.html',
          details: 'This index was created using version: 6.8.13',
          resolve_during_rolling_upgrade: false,
        },
      ],
    },
  };
```
6. open UA and test the reindexing of `old_index`

### Follow up work
A follow up PR if needed will be opened for saved objects types
`upgrade-assistant-ml-upgrade-operation` and
`upgrade-assistant-telemetry`
yuliacech added a commit that referenced this issue Feb 21, 2023
@yuliacech
[Upgrade Assistant] Remove unnecessary mappings for ml snapshots (#15…
b132935 
…1014)

## Summary
Partially addresses #64547
Follow up to #150878

This PR removes unnecessary mappings from the saved objects type
`upgrade-assistant-ml-upgrade-operation. This saved objects are used to
store the resolution progress of ml snapshot deprecations.

Currently, the ml deprecations are disabled via a config for Upgrade
Assistant plugin `xpack.upgrade_assistant.featureSet.mlSnapshots`. And
ES will probably not let us use any old ml snapshots in the 8.x version.

### How to test
1. Add `xpack.upgrade_assistant.featureSet.mlSnapshots: true` to your
`kibana.dev.yml` file
2. Start ES and Kibana with `yarn es snapshot --license=trial` and `yarn
start`
3. Add a sample data set, for example `kibana_sample_data_flights`.
4. Navigate to Analytics -> Machine Learning -> Anomaly Detection ->
Jobs.
5. Create a job for a single metric, chose a time range with some data
in it. Memorize the value you input for job ID.
6. Start and stop the job's datafeed a couple of times, so that you have
more than 1 snapshot. Memorize the value of the snapshot ID.
7. replace `deprecations` on line 22 in the file
[`/server/lib/es_deprecations_status.ts`](https://github.com/elastic/kibana/blob/main/x-pack/plugins/upgrade_assistant/server/lib/es_deprecations_status.ts#L22)
with
```
  const deprecations = {
    cluster_settings: [],
    node_settings: [],
    ml_settings: [
      {
        level: 'critical',
        message:
          'model snapshot [1676291073] for job [test_123] needs to be deleted or upgraded',
        url: '',
        details: 'details',
        _meta: { snapshot_id: '1676291073', job_id: 'test_123' },
        resolve_during_rolling_upgrade: false,
      },
    ],
    index_settings: {},
  };
```
where `job_id` and `snapshot_id` are values from step 5 and 6.
8. open UA and test the resolution of the ml snapshot deprecation.
yuliacech added a commit that referenced this issue Feb 27, 2023
@yuliacech @kibanamachine
[Upgrade Assistant] Deletes telemetry saved object type (#151116)
848b387 
## Summary
Fixes #64547 
Follow up to #150878 and
#151014

This PR removes the saved object type `upgrade-assistant-telemetry` from
the Upgrade Assistant plugin because it seems to not be used for
telemetry or UA functionality. There is a telemetry object to track UA
deprecation logging in this
[file](https://github.com/elastic/kibana/blob/main/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json#L13628),
I don't think that a saved object is needed for that.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
wwang500 pushed a commit to wwang500/kibana that referenced this issue Feb 27, 2023
@yuliacech @kibanamachine @wwang500
[Upgrade Assistant] Deletes telemetry saved object type (elastic#151116)
2de1553 
## Summary
Fixes elastic#64547 
Follow up to elastic#150878 and
elastic#151014

This PR removes the saved object type `upgrade-assistant-telemetry` from
the Upgrade Assistant plugin because it seems to not be used for
telemetry or UA functionality. There is a telemetry object to track UA
deprecation logging in this
[file](https://github.com/elastic/kibana/blob/main/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json#L13628),
I don't think that a saved object is needed for that.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
bmorelli25 pushed a commit to bmorelli25/kibana that referenced this issue Mar 10, 2023
@yuliacech @kibanamachine @bmorelli25
[Upgrade Assistant] Deletes telemetry saved object type (elastic#151116)
1c738c9 
## Summary
Fixes elastic#64547 
Follow up to elastic#150878 and
elastic#151014

This PR removes the saved object type `upgrade-assistant-telemetry` from
the Upgrade Assistant plugin because it seems to not be used for
telemetry or UA functionality. There is a telemetry object to track UA
deprecation logging in this
[file](https://github.com/elastic/kibana/blob/main/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json#L13628),
I don't think that a saved object is needed for that.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuliacech yuliacech

Labels
chore Feature:Upgrade Assistant Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Upgrade Assistant] Deletes telemetry saved object type yuliacech/kibana
5 participants
@sabarasaba @cjcenizal @rudolf @yuliacech @elasticmachine