[Security Solution] All hidden fields in Alert table become visible when a new field is added

[aarju]

Describe the bug:

When a new field is added to the Alerts table in the security solution it causes all hidden fields to become visible again.

Kibana/Elasticsearch Stack version:

7.15

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Alerts Table

Steps to reproduce:

1. Hide some columns in the alerts view
2. Add a new column to the alert view
3. observe that all of the hidden columns are now visible

Current behavior:
When a new field is added to the Alerts table in the security solution it causes all hidden fields to become visible again.

Expected behavior:
When a new field is added to the Alerts table in the security solution the hidden fields should stay hidden

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[andrew-goldstein]

The team agreed to make the following short-term changes, targeting 7.16, detailed below:

• Configure the Columns popover to be consistent with Discover
• Change the Hide column action to Remove column, to be consistent with Discover
• Persist updates to the Columns popover order in local storage
• Restore the feature to persist column widths in local storage

Configure the Columns popover to be consistent with Discover

• We will pass false to the allowHide EuiDataGrid API

This will make all EuiDataGrid-based views in the Security Solution consistent with Discover's use of the EuiDataGrid Columns popover.

In 7.15, the Columns popover includes the hide column toggle, as shown in the screenshot below:

Above: The Columns popover in the 7.15 Alerts table

The Columns popover in Discover's EuiDataGrid-based table does not display the hide column toggle, as shown the screenshot below:

Above: The EuiDataGrid Columns popover in Discover, in master

Passing false to the allowHide EuiDataGrid API API makes the Columns popover in all EuiDataGrid-based views in the Security Solution consistent with Discover, as illustrated by the screenshot below:

Above: The Columns popover will be consistent with Discover

Change the Hide column action to Remove column, to be consistent with Discover

• The Hide column action shown in the 7.15 alerts table will be changed to Remove column, making it consistent with Discover's use of EuiDataGrid

In 7.15, the Alerts table has a Hide column action, as shown in the screenshot below:

Above: The Hide Column action in the 7.15 Alerts table

In 7.15, clicking the Hide Column action shown in the screenshot above hides the column, but does not remove it.

In 7.15, columns may only be removed by un-checking them in the Fields browser, or by un-toggling them in the Alerts / Events details popover. Both of those methods require multiple clicks, and require uses to re-find the field in the modal or popover before it may be toggled for removal.

In Discover, users don't hide columns.

In Discover, users directly remove columns by clicking the Remove column action, shown in the screenshot below:

Above: The Remove column action in Discover's use of EuiDataGrid in master

All EuiDataGrid-based views in the Security Solution will be made consistent with Discover by replacing Hide column with Remove column

Persist updates to the Columns popover order in local storage

• Persist column order updates to local storage when users update the order of columns via the Columns popover

The following PR #110685 restored partial support for persisting columns across page refreshes via local storage, but the Redux store is not updated when users sort columns via the Columns popover, an shown in the animated gif below:

Above: Ordering via the Columns popover is not persisted to local storage in 7.15

Going forward, we will utilize the setVisibleColumns EuiDataGrid API API as a callback to update Redux when the columns are sorted, which will in-turn update local storage to persist the new order across page refreshes:

Restore the feature to persist column widths in local storage

In previous releases, resized column widths were peristed in local storage to persist across page refreshes, as documented in #110524 :

{
   "detections-page":{
      "id":"detections-page",
      "activeTab":"query",
      "prevActiveTab":"query",
      "columns":[
         {
            "category":"base",
            "columnHeaderType":"not-filtered",
            "description":"Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events.",
            "example":"2016-05-23T08:05:34.853Z",
            "id":"@timestamp",
            "type":"date",
            "aggregatable":true,
            "width":190
         },
         {
            "category":"cloud",
            "columnHeaderType":"not-filtered",
            "description":"The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.",
            "example":"666777888999",
            "id":"cloud.account.id",
            "type":"string",
            "aggregatable":true,
            "width":180
         },
         {
            "category":"cloud",
            "columnHeaderType":"not-filtered",
            "description":"Availability zone in which this host is running.",
            "example":"us-east-1c",
            "id":"cloud.availability_zone",
            "type":"string",
            "aggregatable":true,
            "width":180
         },
         // ...
         }
      ],
      // ...
   }
}

Above: column widths were persisted to local storage in previous release, (going at least back to 7.12)

Going forward, we will utilize the onColumnResize EuiDataGrid API API as a callback to update Redux when the columns are sorted via the Columns popover. Updating Redux will in-turn update local storage, so resized columns widths will persist across page refreshes:

CC: @mdefazio @monina-n @paulewing @michaelolo24

[monina-n]

The team agreed to make the following short-term changes, targeting 7.16, detailed below:

Configure the Columns popover to be consistent with Discover
Change the Hide column action to Remove column, to be consistent with Discover
Persist updates to the Columns popover order in local storage
Restore the feature to persist column widths in local storage

@andrew-goldstein thank you for describing all the changes!

Summarizing the discussions the Product/UX sync on 10/06/21, Product and UX is in all agreement with this direction for 7.16. Please let us know when the PR is up to implement these changes so we can review.

cc: @adamwdraper @paulewing @mdefazio