diff --git a/docs/user/images/alerting-rule.png b/docs/user/images/alerting-rule.png new file mode 100644 index 00000000000000..f3588ef48f7059 Binary files /dev/null and b/docs/user/images/alerting-rule.png differ diff --git a/docs/user/images/annotation-editing.png b/docs/user/images/annotation-editing.png new file mode 100644 index 00000000000000..4e6325c31c7814 Binary files /dev/null and b/docs/user/images/annotation-editing.png differ diff --git a/docs/user/images/ccs-query-inspector.png b/docs/user/images/ccs-query-inspector.png new file mode 100644 index 00000000000000..1814f4d047d7ff Binary files /dev/null and b/docs/user/images/ccs-query-inspector.png differ diff --git a/docs/user/images/color-mapping.png b/docs/user/images/color-mapping.png new file mode 100644 index 00000000000000..d84d11d625c6c9 Binary files /dev/null and b/docs/user/images/color-mapping.png differ diff --git a/docs/user/images/custom-field.gif b/docs/user/images/custom-field.gif new file mode 100644 index 00000000000000..4cf788d0022e20 Binary files /dev/null and b/docs/user/images/custom-field.gif differ diff --git a/docs/user/images/data-drift.png b/docs/user/images/data-drift.png new file mode 100644 index 00000000000000..52360e00961626 Binary files /dev/null and b/docs/user/images/data-drift.png differ diff --git a/docs/user/images/edit-links-panel.png b/docs/user/images/edit-links-panel.png new file mode 100644 index 00000000000000..2e24bef5d282d0 Binary files /dev/null and b/docs/user/images/edit-links-panel.png differ diff --git a/docs/user/images/edit-links.png b/docs/user/images/edit-links.png new file mode 100644 index 00000000000000..920383a5d2bf75 Binary files /dev/null and b/docs/user/images/edit-links.png differ diff --git a/docs/user/images/elser-model-2.png b/docs/user/images/elser-model-2.png new file mode 100644 index 00000000000000..d261d1491cee3a Binary files /dev/null and b/docs/user/images/elser-model-2.png differ diff --git a/docs/user/images/endpoints-example.png b/docs/user/images/endpoints-example.png new file mode 100644 index 00000000000000..948af7d50fe141 Binary files /dev/null and b/docs/user/images/endpoints-example.png differ diff --git a/docs/user/images/endpoints.png b/docs/user/images/endpoints.png new file mode 100644 index 00000000000000..8afa6054200ffb Binary files /dev/null and b/docs/user/images/endpoints.png differ diff --git a/docs/user/images/enrich-index-management.png b/docs/user/images/enrich-index-management.png new file mode 100644 index 00000000000000..b7e9592987027a Binary files /dev/null and b/docs/user/images/enrich-index-management.png differ diff --git a/docs/user/images/enrich-ui.png b/docs/user/images/enrich-ui.png new file mode 100644 index 00000000000000..b9c2554d82bd40 Binary files /dev/null and b/docs/user/images/enrich-ui.png differ diff --git a/docs/user/images/esql-demo.gif b/docs/user/images/esql-demo.gif new file mode 100644 index 00000000000000..ebb79ec8d30fbb Binary files /dev/null and b/docs/user/images/esql-demo.gif differ diff --git a/docs/user/images/esql-enrich.png b/docs/user/images/esql-enrich.png new file mode 100644 index 00000000000000..a84952d06fa0ff Binary files /dev/null and b/docs/user/images/esql-enrich.png differ diff --git a/docs/user/images/esql-rule-type.gif b/docs/user/images/esql-rule-type.gif new file mode 100644 index 00000000000000..1dea827f17d505 Binary files /dev/null and b/docs/user/images/esql-rule-type.gif differ diff --git a/docs/user/images/grok-use-case.gif b/docs/user/images/grok-use-case.gif new file mode 100644 index 00000000000000..d20bf48eb6ea84 Binary files /dev/null and b/docs/user/images/grok-use-case.gif differ diff --git a/docs/user/images/grok.png b/docs/user/images/grok.png new file mode 100644 index 00000000000000..cc85a4cd46c6e7 Binary files /dev/null and b/docs/user/images/grok.png differ diff --git a/docs/user/images/lens-inline-editing.gif b/docs/user/images/lens-inline-editing.gif new file mode 100644 index 00000000000000..4266eeaec1fd6b Binary files /dev/null and b/docs/user/images/lens-inline-editing.gif differ diff --git a/docs/user/images/links-panel.gif b/docs/user/images/links-panel.gif new file mode 100644 index 00000000000000..23143a2edef5a9 Binary files /dev/null and b/docs/user/images/links-panel.gif differ diff --git a/docs/user/images/obv-use-case.png b/docs/user/images/obv-use-case.png new file mode 100644 index 00000000000000..2341eb8fe92125 Binary files /dev/null and b/docs/user/images/obv-use-case.png differ diff --git a/docs/user/images/obvs.png b/docs/user/images/obvs.png new file mode 100644 index 00000000000000..424152e605fc78 Binary files /dev/null and b/docs/user/images/obvs.png differ diff --git a/docs/user/images/start-esql-in-discover.gif b/docs/user/images/start-esql-in-discover.gif new file mode 100644 index 00000000000000..33df4f52dbb77f Binary files /dev/null and b/docs/user/images/start-esql-in-discover.gif differ diff --git a/docs/user/images/term-aggs.png b/docs/user/images/term-aggs.png new file mode 100644 index 00000000000000..2928426686fdc5 Binary files /dev/null and b/docs/user/images/term-aggs.png differ diff --git a/docs/user/images/text-field-support.png b/docs/user/images/text-field-support.png new file mode 100644 index 00000000000000..a64f360b602ddc Binary files /dev/null and b/docs/user/images/text-field-support.png differ diff --git a/docs/user/whats-new.asciidoc b/docs/user/whats-new.asciidoc index e9d1faf845c6f9..4dacf5ce2842dc 100644 --- a/docs/user/whats-new.asciidoc +++ b/docs/user/whats-new.asciidoc @@ -5,12 +5,242 @@ Here are the highlights of what's new and improved in {minor-version}. For detailed information about this release, check the <>. -Previous versions: {kibana-ref-all}/8.6/whats-new.html[8.6] | {kibana-ref-all}/8.5/whats-new.html[8.5] | {kibana-ref-all}/8.4/whats-new.html[8.4] | {kibana-ref-all}/8.3/whats-new.html[8.3] | {kibana-ref-all}/8.2/whats-new.html[8.2] +Previous versions: {kibana-ref-all}/8.10/whats-new.html[8.10] | {kibana-ref-all}/8.9/whats-new.html[8.9] | {kibana-ref-all}/8.8/whats-new.html[8.8] | {kibana-ref-all}/8.7/whats-new.html[8.7] | {kibana-ref-all}/8.6/whats-new.html[8.6] | {kibana-ref-all}/8.5/whats-new.html[8.5] | {kibana-ref-all}/8.4/whats-new.html[8.4] | {kibana-ref-all}/8.3/whats-new.html[8.3] | {kibana-ref-all}/8.2/whats-new.html[8.2] | {kibana-ref-all}/8.1/whats-new.html[8.1] | {kibana-ref-all}/8.0/whats-new.html[8.0] -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide +[discrete] +=== Discover -// tag::notable-highlights[] +[discrete] +==== {esql} in Discover -// end::notable-highlights[] +preview:[] In 8.11 we are introducing link:https://www.elastic.co/guide/en/elasticsearch/reference/master/esql.html[{es} Query Language ({esql})], Elastic’s new piped language for data exploration and investigation. {esql} transforms, enriches, and simplifies your data exploration process. + +[role="screenshot"] +image::images/esql-demo.gif[A short video demo of {esql}] + +Here is what you can expect: + +* *Easy start:* To begin using {esql} in **Discover**, select **Try {esql}** from the data view menu. +* *Efficient and easy query building:* {esql} in Discover offers auto-complete and in-app documentation, making it easy to craft powerful queries right from the query bar. +* *Comprehensive and powerful data exploration:* Conduct ad-hoc data exploration within **Discover**. Create aggregations, transform data, enrich datasets, and more directly from the query builder. Results are presented in a tabular format or as visualizations. It depends on the query you are executing. +* *Contextual visualizations:* When writing {esql} queries in **Discover**, you’ll receive visual representations powered by the **Lens** suggestion engine. Your query’s nature determines the type of visualization you get, such as a metric, histogram, heatmap, and so on. +* *Enrichment:* Use the enrich command to enhance your query dataset with fields from another dataset, complete with in-context suggestions for the selected policy, such as hinting the matching field and enriched columns. +* *In-line visualization editing:* Edit {esql} visualizations directly within **Discover** and dashboards. No need to navigate to **Lens** for quick edits, so you can make changes seamlessly. +* *Dashboard integration:* Save your {esql} visualizations to a dashboard directly from **Discover** once you’re satisfied with the results. +* *Alerting:* Use {esql} for Observability and Security alerts, setting aggregated values as thresholds. Enhance detection accuracy and receive actionable notifications by emphasizing meaningful trends over isolated incidents, reducing false positives. + +{esql} in **Discover** brings efficiency and power to your data investigations, streamlining your path to insights. + +Learn more about {esql}’s capabilities in the link:{ref-bare}/master/esql.html[{es} {esql} documentation]. + + +[role="screenshot"] +image::images/obvs.png[An example of {esql}] + +[role="screenshot"] +image::images/obv-use-case.png[An example using {esql} in an {observability} use case] + + +[discrete] +=== Dashboard + +[discrete] +==== Lens inline editing in dashboards + +You can now edit a **Lens** visualization without leaving the dashboard instead of navigating back and forth to the **Lens** editor. Open the panel menu and select **Edit visualization**. A flyout will be open in the dashboard where you can perform any edits to your **Lens** panels. Once happy with your edits, click **Apply and close**. This new editing experience is more convenient and will save you time since the dashboard will not need to reload when saving your changes. + +[role="screenshot"] +image::images/lens-inline-editing.gif[An example of inline editing in Lens] + +[discrete] +==== Links panel + +You can now easily navigate from one dashboard to another using the links panel. Better organize your dashboards and make them more performant by chunking them in multiple dashboards with fewer visualizations and linking them together. You can carry over your filters, query, and time range when navigating to other related dashboards. Display your links horizontally or vertically as it better suits your dashboard layout. You can also use the links panel to include external links in your dashboards, such as to your wiki page or other applications. Decide whether you want to open the links in the same browser tab or in a new one. + +[role="screenshot"] +image::images/links-panel.gif[An example of the new links panel] + +[role="screenshot"] +image::images/edit-links-panel.png[An example of the edit links panel flyout] + +[role="screenshot"] +image::images/edit-links.png[An example of how to edit links using the panel flyout] + +[discrete] +==== Color mapping for enhanced data visualization + +Color is a fundamental visual element, alongside position and shape, that plays a crucial role in conveying information effectively. + +[role="screenshot"] +image::images/color-mapping.png[An example of the new color mapping feature] + +Here’s what our new color mapping feature offers: + +* *Effortless categorization:* Easily assign one or more field categories to specific colors. This makes it simpler than ever to organize and understand your data. +* *Guided color selection:* Our intuitive color chooser provides you with predefined palettes that ensure your charts not only look great, but also align seamlessly with different {kib} themes. +* *Enhanced color palettes:* We’ve reintroduced a clear concept of color palettes, making it easier for you to select and apply gradients, improving the overall aesthetics of your visualizations. + +Whether you’re working with cartesian, partition, or tag clouds charts, these enhancements are designed to help you make the most of your data. With this feature, you can expect an improved ability to categorize, differentiate, and emphasize data points on your charts, ultimately leading to better insights and more visually appealing dashboards. + +[discrete] +==== {ccs-cap} ({ccs-init}) query inspector + +Customers querying data from multiple clusters link:{ref}/modules-cross-cluster-search.html[({ccs-init} queries)] will get more information about each of the cluster's responses. For each of the visualizations in a dashboard as well as in **Discover**, you can look at response times per cluster and shard failures. This is especially important when one or more clusters are not able to provide all the data, so you know that you are looking at partial results and why they fail. + +[role="screenshot"] +image::images/ccs-query-inspector.png[An example of the new {ccs} query inspector, width=50%] + +[discrete] +==== Individual annotation editing from the library + +We've introduced full annotation group editing in the **Visualize Library**. Now, you can easily edit shared annotation groups without leaving Lens. No more searching for consuming visualizations. + +[role="screenshot"] +image::images/annotation-editing.png[An example of annotation editing] + +[discrete] +=== Machine Learning + +[discrete] +==== ELSER is improved and is now generally available + +In 8.8, we introduced Elastic Learned Sparse Encoder in technical preview. ELSER is Elastic’s text expansion language model for AI search. It offers superior relevance out of the box, without the need for retraining on in-domain data or any other ML or MLOps effort. Deploy it with a couple of clicks from Elastic’s UI and start leveraging the power of AI with your search. + +In 8.11, we're releasing a generally available second version. ELSER model-2 comes in two versions: + +* The optimized model, which runs on the linux-x86_64 platform +* The cross-platform model + +Both ELSER model-2 versions, platform-optimized and cross-platform, show improved relevance compared to the original ELSER release, as measured against the BEIR benchmarks. Importantly, the optimized version also shows significantly improved performance (reduced inference latency). The {ecloud} supports the optimized version and so {ecloud} users will benefit from the materially improved performance of the optimized ELSER model-2. + +[role="screenshot"] +image::images/elser-model-2.png[An example of ELSER model 2, width=60%] + +NOTE: The original version of ELSER (the model available prior to 8.11) will remain in technical preview. + +[discrete] +==== Inference APIs + +preview:[] We are working to introduce a unified inference API that abstracts away the complexity of performing inference on different models that are trained for different tasks. The API introduces a simple, intuitive syntax of the form: + +[source,bash] +---- +POST /_inference// +---- + +In 8.11, we're'releasing a contained first link:{ref}/inference-apis.html[MVP iteration of this framework], which initially only supports ELSER. This link:{ref}/semantic-search-elser.html#inference-ingest-pipeline[greatly simplifies the syntax] for creating an inference pipeline. + +More importantly, in the future the new inference API will support both internal and external models and will integrate with the LLM ecosystem for our users to have the most powerful AI effortlessly and seamlessly at their fingertips, through a unified, self-explanatory API. + +[discrete] +==== AIOps: Log rate analysis supports text fields + +preview:[] Continuing enhancing log analysis capabilities with smart AIOps tools for drastically shorter mean time to repair. We now support detection of log rate changes that are due to text fields, for example the common **message** log field. Previously log rate change was limited to detecting spikes and dips caused by keyword fields. By adding text fields, we've incorporated pattern analysis into log rate analysis. That has significantly reduced the time it takes to detect and diagnose events that used to go unnoticed for periods of time. + +[role="screenshot"] +image::images/text-field-support.png[An example of log rate analysis supporting text fields] + +[discrete] +==== Data drift workflows + +preview:[] In 8.10, we introduced the **Data comparison** view to help you detect data drift. In 8.11, we have renamed it to **Data drift** and we have enhanced it to include workflows that help you visualize changes in the model input data and detect potential model performance degradation over time. + +[role="screenshot"] +image::images/data-drift.png[An example of data drift workflows] + +[discrete] +==== Improvements in anomaly detection embeddability and data frame analytics pipelines + +We have improved the UX for attaching anomaly swim lanes and anomaly charts to dashboards. It is now more friendly and consistent with the same functionality from Lens. In addition, you can now attach these ML charts to new dashboards (previously this was only able to be done for existing ones). + +In **Data Frame Analytics**, we previously added the ability to link directly to **Discover** and **Dashboards** from the results data grid filtering for the row’s field/values for all visible columns. For improved usability, you can now do this during the job creation as well. We have also made UX improvements for the deployment of trained models from Data Frame Analytics jobs, including an option to reindex your data at the end of the ingest pipeline creation. + +[discrete] +=== ResponseOps + +[discrete] +==== New {esql} rule type + +A new {esql} alerting rule type is now available under the existing {es} rule type. This rule type brings all the new functionalities that are available within the new and powerful language, {esql}, to {kib} Alerting to allow and unlock new alerting use cases. + +With the new type, users will be able to generate a single alert based on defined {esql} query and preview the query result before saving the rule. When the query returns an empty result no alerts will be generated. + +[role="screenshot"] +image::images/esql-rule-type.gif[An example of creating a rule type using {esql}] + +[discrete] +==== {kib} cases custom field + +A new functionality is now available in {kib} cases. Users will be able to add custom fields to the case structure, so they can use it for better classification and case enrichment. As a first step, those fields will be available in the case view only. In the next step, we’re planning to have more field types, dedicated privileges, support filters, and search capabilities in the case table. + +[role="screenshot"] +image::images/custom-field.gif[An example of creating a custom case field in {kib}] + +[discrete] +==== Supporting multi levels of term aggregations in {es} rule type + +The existing {es} alerting rule (KQL-based) is now supported by multiple selection when grouping by alert fields, which allows you to define multiple layers of term aggregations. + +[role="screenshot"] +image::images/term-aggs.png[An example of creating multiple layers of term aggregations] + +[discrete] +==== Slack connector - allow List + +The Slack connector supports a new allow list, so customers will be able to manage the available Slack channels within the alert actions. + +[discrete] +=== {observability} + +[discrete] +==== {es} alerting rule now available in {observability} + +The {es} alerting rule is now available in {observability}. Before this update, {observability} customers who leverage the {es} rule were required to move between **{stack-manage-app}** and **{observability}** to manage their alerts. +To manage the {observability} roles properly, we added a new field to determine which role can maintain the created rule and its generated alerts. + +[role="screenshot"] +image::images/alerting-rule.png[An example of creating an altering rule in {observability}] + +[discrete] +=== Global Experience + +[discrete] +==== Create and manage {esql} enrichment index policies + +In support of our new {esql} capabilities, we have added link:{ref}/ingest-enriching-data.html[enrich policies] to our **Index Management** experience. Users can now create their enrichment policies right from here and get started using it right away. Once configured, all enrich policies are available in the **Enrich Policies** tab. + +Example of an ES|QL Query using an enrich policy with enrich: + +[role="screenshot"] +image::images/esql-enrich.png[An example of an {esql} query using an enrich policy] + +Example of the enrich policies shown in the UI: + +[role="screenshot"] +image::images/enrich-ui.png[An example of where to add an enrich policy in the UI] + +[role="screenshot"] +image::images/enrich-index-management.png[An example of where an enrich policy lives in the UI] + + +[discrete] +==== Start Discover {esql} from global search + +**Discover** searches are powered by KQL by default. You can easily switch to {esql} mode within **Discover** and search your data with this new query language. You can also access {esql} in **Discover** from the global search bar within Elastic. With just a few keystrokes, simply type in “ESQL” and you can access **Discover** with {esql} enabled for you. + +[role="screenshot"] +image::images/start-esql-in-discover.gif[An example of using ES|QL in Discover] + +[discrete] +==== Access {es} connection details + +You can now view your {es} endpoint, link:https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html[Cloud ID], and even manage your API keys from many areas in Elastic such as integrations. You can also access the connection details on any page in Elastic using the help menu from the header bar. + +[role="screenshot"] +image::images/endpoints.png[An example of where to find the endpoints in the UI, width=40%] + +[discrete] +==== AWS CloudFormation template updates + +When subscribing to Elastic using the link:https://aws.amazon.com/marketplace/pp/prodview-voru33wi6xs7k[AWS Marketplace], users have the option to quickly get set up with an AWS CloudFormation Template. This step allows you to create an Elastic deployment in the AWS region of your choice. We’ve updated the AWS CloudFormation Template with bug fixes and stability improvements to better help you get started. \ No newline at end of file