Skip to content

Commit

Permalink
Merge branch 'master' into improve-ui-doc-viewer
Browse files Browse the repository at this point in the history
  • Loading branch information
kibanamachine committed Jun 25, 2021
2 parents 3ab15f3 + dfc70bd commit 96b70e5
Show file tree
Hide file tree
Showing 639 changed files with 7,245 additions and 5,192 deletions.
22 changes: 13 additions & 9 deletions NOTICE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -149,17 +149,17 @@ SOFTWARE.

---
Detection Rules
Copyright 2020 Elasticsearch B.V.
Copyright 2021 Elasticsearch B.V.

---
This product bundles rules based on https://github.com/BlueTeamLabs/sentinel-attack
which is available under a "MIT" license. The files based on this license are:
which is available under a "MIT" license. The rules based on this license are:

- defense_evasion_via_filter_manager
- discovery_process_discovery_via_tasklist_command
- persistence_priv_escalation_via_accessibility_features
- persistence_via_application_shimming
- defense_evasion_execution_via_trusted_developer_utilities
- "Potential Evasion via Filter Manager" (06dceabf-adca-48af-ac79-ffdf4c3b1e9a)
- "Process Discovery via Tasklist" (cc16f774-59f9-462d-8b98-d27ccd4519ec)
- "Potential Modification of Accessibility Binaries" (7405ddf1-6c8e-41ce-818f-48bea6bcaed8)
- "Potential Application Shimming via Sdbinst" (fd4a992d-6130-4802-9ff8-829b89ae801f)
- "Trusted Developer Application Usage" (9d110cb3-5f4b-4c9a-b9f5-53f0a1707ae1)

MIT License

Expand All @@ -185,9 +185,9 @@ SOFTWARE.

---
This product bundles rules based on https://github.com/FSecureLABS/leonidas
which is available under a "MIT" license. The files based on this license are:
which is available under a "MIT" license. The rules based on this license are:

- credential_access_secretsmanager_getsecretvalue.toml
- "AWS Access Secret in Secrets Manager" (a00681e3-9ed6-447c-ab2c-be648821c622)

MIT License

Expand Down Expand Up @@ -235,6 +235,10 @@ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

---
Portions of this code are licensed under the following license:
For license information please see https://edge.fullstory.com/s/fs.js.LICENSE.txt

---
This product bundles bootstrap@3.3.6 which is available under a
"MIT" license.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
| [isSavedObjectEmbeddableInput(input)](./kibana-plugin-plugins-embeddable-public.issavedobjectembeddableinput.md) | |
| [openAddPanelFlyout(options)](./kibana-plugin-plugins-embeddable-public.openaddpanelflyout.md) | |
| [plugin(initializerContext)](./kibana-plugin-plugins-embeddable-public.plugin.md) | |
| [useEmbeddableFactory({ input, factory, onInputUpdated, })](./kibana-plugin-plugins-embeddable-public.useembeddablefactory.md) | |

## Interfaces

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-plugins-embeddable-public](./kibana-plugin-plugins-embeddable-public.md) &gt; [useEmbeddableFactory](./kibana-plugin-plugins-embeddable-public.useembeddablefactory.md)

## useEmbeddableFactory() function

<b>Signature:</b>

```typescript
export declare function useEmbeddableFactory<I extends EmbeddableInput>({ input, factory, onInputUpdated, }: EmbeddableRendererWithFactory<I>): readonly [ErrorEmbeddable | IEmbeddable<I, import("./i_embeddable").EmbeddableOutput> | undefined, boolean, string | undefined];
```

## Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| { input, factory, onInputUpdated, } | <code>EmbeddableRendererWithFactory&lt;I&gt;</code> | |

<b>Returns:</b>

`readonly [ErrorEmbeddable | IEmbeddable<I, import("./i_embeddable").EmbeddableOutput> | undefined, boolean, string | undefined]`

8 changes: 7 additions & 1 deletion docs/settings/task-manager-settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,13 @@ Task Manager runs background tasks by polling for work on an interval. You can
| The maximum number of tasks that this Kibana instance will run simultaneously. Defaults to 10.
Starting in 8.0, it will not be possible to set the value greater than 100.

| `xpack.task_manager.monitored_stats_warn_delayed_task_start_in_seconds`
| `xpack.task_manager.`
`monitored_stats_health_verbose_log.enabled`
| This flag will enable automatic warn and error logging if task manager self detects a performance issue, such as the time between when a task is scheduled to execute and when it actually executes. Defaults to false.

| `xpack.task_manager.`
`monitored_stats_health_verbose_log.`
`warn_delayed_task_start_in_seconds`
| The amount of seconds we allow a task to delay before printing a warning server log. Defaults to 60.
|===

Expand Down
49 changes: 41 additions & 8 deletions docs/setup/upgrade/upgrade-migrations.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -55,22 +55,55 @@ This section highlights common causes of {kib} upgrade failures and how to preve
There is a known issue in v7.12.0 for users who tried the fleet beta. Upgrade migrations fail because of a large number of documents in the `.kibana` index.

This can cause Kibana to log errors like:
> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]

[source,sh]
--------------------------------------------
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
--------------------------------------------

See https://github.com/elastic/kibana/issues/95321 for instructions to work around this issue.

[float]
===== Corrupt saved objects
We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.
We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment.

Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.

For example, given the following error message:
> Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.

The following steps must be followed to allow the upgrade migration to succeed.
Please be aware the Dashboard having ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` belonging to the space `marketing_space` will no more be available:
1. Delete the corrupt document with `DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275`
2. Restart {kib}
[source,sh]
--------------------------------------------
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
--------------------------------------------

The following steps must be followed to delete the document that is causing the migration to fail:

. Remove the write block which the migration system has placed on the previous index:
+
[source,sh]
--------------------------------------------
PUT .kibana_7.12.1_001/_settings
{
"index": {
"blocks.write": false
}
}
--------------------------------------------

. Delete the corrupt document:
+
[source,sh]
--------------------------------------------
DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275
--------------------------------------------

. Restart {kib}.

In this example, the Dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **will no longer be available**.

Be sure you have a snapshot before you delete the corrupt document. If restoring from a snapshot is not an option, it is recommended to also delete the `temp` and `target` indices the migration created before restarting {kib} and retrying.

[float]
===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings
Expand Down
198 changes: 151 additions & 47 deletions docs/siem/index.asciidoc
Original file line number Diff line number Diff line change
@@ -1,60 +1,164 @@
[chapter]
[role="xpack"]
[[xpack-siem]]
= Elastic Security
= Elastic Security overview
++++
<titleabbrev>Security</titleabbrev>
++++

[partintro]
--
https://www.elastic.co/security[Elastic Security] combines SIEM threat detection features with endpoint
prevention and response capabilities in one solution. These analytical and
protection capabilities, leveraged by the speed and extensibility of
Elasticsearch, enable analysts to defend their organization from threats before
damage and loss occur.

Elastic Security combines SIEM threat detection features with endpoint
prevention and response capabilities in one solution, including:
Elastic Security provides the following security benefits and capabilities:

* A detection engine to identify attacks and system misconfiguration
* A detection engine to identify attacks and system misconfigurations
* A workspace for event triage and investigations
* Interactive visualizations to investigate process relationships
* Embedded case management and automated actions
* Detection of signatureless attacks with prebuilt {ml} anomaly jobs and
detection rules
* Inbuilt case management with automated actions
* Detection of signatureless attacks with prebuilt machine learning anomaly jobs
and detection rules

[role="screenshot"]
image::siem/images/overview-ui.png[Elastic Security in Kibana]

[float]
== Add data

Kibana provides step-by-step instructions to help you add data. The
{security-guide}[Security Guide] is a good source for more
detailed information and instructions.

[float]
=== {Beats}

https://www.elastic.co/products/beats/auditbeat[{auditbeat}],
https://www.elastic.co/products/beats/filebeat[{filebeat}],
https://www.elastic.co/products/beats/winlogbeat[{winlogbeat}], and
https://www.elastic.co/products/beats/packetbeat[{packetbeat}]
send security events and other data to Elasticsearch.
[discrete]
== Elastic Security components and workflow

The default index patterns for Elastic Security events are `auditbeat-*`, `winlogbeat-*`,
`filebeat-*`, `packetbeat-*`, `endgame-*`, `logs-*`, and `apm-*-transaction*`. To change the default pattern patterns, go to *Stack Management > Advanced Settings > securitySolution:defaultIndex*.
The following diagram provides a comprehensive illustration of the Elastic Security workflow.

[float]
=== Elastic Security endpoint agent

The agent detects and protects against malware, and ships host and network
events directly to Elastic Security.

[float]
=== Elastic Common Schema (ECS) for normalizing data

The {ecs-ref}[Elastic Common Schema (ECS)] defines a common set of fields to be
used for storing event data in Elasticsearch. ECS helps users normalize their
event data to better analyze, visualize, and correlate the data represented in
their events.

Elastic Security can ingest and normalize events from ECS-compatible data sources.
[role="screenshot"]
image::../siem/images/workflow.png[]

Here's an overview of the flow and its components:

* Data is shipped from your hosts to {es} via beat modules and the Elastic https://www.elastic.co/endpoint-security/[Endpoint Security agent integration]. This integration provides capabilities such as collecting events, detecting and preventing {security-guide}/detection-engine-overview.html#malware-prevention[malicious activity], and artifact delivery. The {fleet-guide}/fleet-overview.html[{fleet}] app is used to
install and manage agents and integrations on your hosts.
+
The Endpoint Security integration ships the following data sets:
+
*** *Windows*: Process, network, file, DNS, registry, DLL and driver loads,
malware security detections
*** *Linux/macOS*: Process, network, file
+
* https://www.elastic.co/integrations?solution=security[Beat modules]: {beats}
are lightweight data shippers. Beat modules provide a way of collecting and
parsing specific data sets from common sources, such as cloud and OS events,
logs, and metrics. Common security-related modules are listed {security-guide}/ingest-data.html#enable-beat-modules[here].
* The {security-app} in {kib} is used to manage the *Detection engine*,
*Cases*, and *Timeline*, as well as administer hosts running Endpoint Security:
** Detection engine: Automatically searches for suspicious host and network
activity via the following:
*** {security-guide}/detection-engine-overview.html#detection-engine-overview[Detection rules]: Periodically search the data
({es} indices) sent from your hosts for suspicious events. When a suspicious
event is discovered, a detection alert is generated. External systems, such as
Slack and email, can be used to send notifications when alerts are generated.
You can create your own rules and make use of our {security-guide}/prebuilt-rules.html[prebuilt ones].
*** {security-guide}/detections-ui-exceptions.html[Exceptions]: Reduce noise and the number of
false positives. Exceptions are associated with rules and prevent alerts when
an exception's conditions are met. *Value lists* contain source event
values that can be used as part of an exception's conditions. When
Elastic {endpoint-sec} is installed on your hosts, you can add malware exceptions
directly to the endpoint from the Security app.
*** {security-guide}/machine-learning.html#included-jobs[{ml-cap} jobs]: Automatic anomaly detection of host and
network events. Anomaly scores are provided per host and can be used with
detection rules.
** {security-guide}/timelines-ui.html[Timeline]: Workspace for investigating alerts and events.
Timelines use queries and filters to drill down into events related to
a specific incident. Timeline templates are attached to rules and use predefined
queries when alerts are investigated. Timelines can be saved and shared with
others, as well as attached to Cases.
** {security-guide}/cases-overview.html[Cases]: An internal system for opening, tracking, and sharing
security issues directly in the Security app. Cases can be integrated with
external ticketing systems.
** {security-guide}/admin-page-ov.html[Administration]: View and manage hosts running {endpoint-sec}.

{security-guide}/ingest-data.html[Ingest data to Elastic Security] and {security-guide}/install-endpoint.html[Configure and install the Elastic Endpoint integration] describe how to ship security-related
data to {es}.


For more background information, see:

* https://www.elastic.co/products/elasticsearch[{es}]: A real-time,
distributed storage, search, and analytics engine. {es} excels at indexing
streams of semi-structured data, such as logs or metrics.
* https://www.elastic.co/products/kibana[{kib}]: An open-source analytics and
visualization platform designed to work with {es}. You use {kib} to search,
view, and interact with data stored in {es} indices. You can easily compile
advanced data analysis and visualize your data in a variety of charts, tables,
and maps.

[discrete]
=== Compatibility with cold tier nodes

Cold tier is a {ref}/data-tiers.html[data tier] that holds time-series data that is accessed only occasionally. In {stack} version >=7.11.0, {elastic-sec} supports cold tier data for the following {es} indices:

* Index patterns specified in `securitySolution:defaultIndex`
* Index patterns specified in the definitions of detection rules, except for indicator match rules
* Index patterns specified in the data sources selector on various {security-app} pages

{elastic-sec} does NOT support cold tier data for the following {es} indices:

* Index patterns controlled by {elastic-sec}, including signals and list indices
* Index patterns specified in indicator match rules

Using cold tier data for unsupported indices may result in detection rule timeouts and overall performance degradation.

[discrete]
[[self-protection]]
==== Elastic Endpoint self-protection

Self-protection means that {elastic-endpoint} has guards against users and attackers that may try to interfere with its functionality. This protection feature is consistently enhanced to prevent attackers who may attempt to use newer, more sophisticated tactics to interfere with the {elastic-endpoint}. Self-protection is enabled by default when {elastic-endpoint} installs on supported platforms, listed below.

Self-protection is enabled on the following 64-bit Windows versions:

* Windows 8.1
* Windows 10
* Windows Server 2012 R2
* Windows Server 2016
* Windows Server 2019

And on the following macOS versions:

* macOS 10.15 (Catalina)
* macOS 11 (Big Sur)

NOTE: Other Windows and macOS variants (and all Linux distributions) do not have self-protection.

For {stack} version >= 7.11.0, self-protection defines the following permissions:

* Users -- even Administrator/root -- *cannot* delete {elastic-endpoint} files (located at `c:\Program Files\Elastic\Endpoint` on Windows, and `/Library/Elastic/Endpoint` on macOS).
* Users *cannot* terminate the {elastic-endpoint} program or service.
* Administrator/root users *can* read the endpoint's files. On Windows, the easiest way to read Endpoint files is to start an Administrator `cmd.exe` prompt. On macOS, an Administrator can use the `sudo` command.
* Administrator/root users *can* stop the {elastic-agent}'s service. On Windows, run the `sc stop "Elastic Agent"` command. On macOS, run the `sudo launchctl stop elastic-agent` command.


[discrete]
[[siem-integration]]
=== Integration with other Elastic products

You can use {elastic-sec} with other Elastic products and features to help you
identify and investigate suspicious activity:

* https://www.elastic.co/products/stack/machine-learning[{ml-cap}]
* https://www.elastic.co/products/stack/alerting[Alerting]
* https://www.elastic.co/products/stack/canvas[Canvas]

[discrete]
[[data-sources]]
=== APM transaction data sources

By default, {elastic-sec} monitors {apm-app-ref}/apm-getting-started.html[APM]
`apm-*-transaction*` indices. To add additional APM indices, update the
index patterns in the `securitySolution:defaultIndex` setting ({kib} -> Stack Management -> Advanced Settings -> `securitySolution:defaultIndex`).

--
[discrete]
[[ecs-compliant-reqs]]
=== ECS compliance data requirements

The {ecs-ref}[Elastic Common Schema (ECS)] defines a common set of fields used for
storing event data in Elasticsearch. ECS helps users normalize their event data
to better analyze, visualize, and correlate the data represented in their
events. {elastic-sec} supports events and indicator index data from any ECS-compliant data source.

include::siem-ui.asciidoc[]
include::machine-learning.asciidoc[]
IMPORTANT: {elastic-sec} requires {ecs-ref}[ECS-compliant data]. If you use third-party data collectors to ship data to {es}, the data must be mapped to ECS.
{security-guide}/siem-field-reference.html[Elastic Security ECS field reference] lists ECS fields used in {elastic-sec}.
Loading

0 comments on commit 96b70e5

Please sign in to comment.