From 2552e6d39012aeaa707df71a7ae71672ed607dcb Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 4 Apr 2025 15:42:34 -0500 Subject: [PATCH 1/9] add get started serverless page, remove dupe troubleshoot pages --- .../apps/apm-server-performance-diagnostic.md | 49 -- .../apps/apm-server-response-codes.md | 36 -- .../enable-apm-server-binary-debugging.md | 42 -- .../apps/get-started-apm-serverless.md | 448 ++++++++++++++++++ .../apps/processing-performance.md | 47 -- ...s-when-apm-server-elasticsearch-is-down.md | 25 - solutions/toc.yml | 6 +- 7 files changed, 449 insertions(+), 204 deletions(-) delete mode 100644 solutions/observability/apps/apm-server-performance-diagnostic.md delete mode 100644 solutions/observability/apps/apm-server-response-codes.md delete mode 100644 solutions/observability/apps/enable-apm-server-binary-debugging.md create mode 100644 solutions/observability/apps/get-started-apm-serverless.md delete mode 100644 solutions/observability/apps/processing-performance.md delete mode 100644 solutions/observability/apps/what-happens-when-apm-server-elasticsearch-is-down.md diff --git a/solutions/observability/apps/apm-server-performance-diagnostic.md b/solutions/observability/apps/apm-server-performance-diagnostic.md deleted file mode 100644 index 003d555e47..0000000000 --- a/solutions/observability/apps/apm-server-performance-diagnostic.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/apm-performance-diagnostic.html -applies_to: - stack: ---- - -# APM Server performance diagnostic [apm-performance-diagnostic] - - -## Diagnosing backpressure from {{es}} [apm-es-backpressure] - -When {{es}} is under excessive load or indexing pressure, APM Server could experience the downstream backpressure when indexing new documents into {{es}}. Most commonly, backpressure from {{es}} will manifest itself in the form of higher indexing latency and/or rejected requests, which in return could lead APM Server to deny incoming requests. As a result, APM agents connected to the affected APM Server will suffer from throttling and/or request timeout when shipping APM events. - -To quickly identify possible issues try looking for similar error logs lines in APM Server logs: - -```json -... -{"log.level":"error","@timestamp":"2024-07-27T23:46:28.529Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (429): [429 Too Many Requests]"},"ecs.version":"1.6.0"} -{"log.level":"error","@timestamp":"2024-07-27T23:55:38.612Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (503): [503 Service Unavailable]"},"ecs.version":"1.6.0"} -... -``` - -To gain better insight into APM Server health and performance, consider enabling the monitoring feature by following the steps in [Monitor APM Server](monitor-apm-server.md). When enabled, APM Server will additionally report a set of vital metrics to help you identify any performance degradation. - -Pay careful attention to the next metric fields: - -* `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting to be ingested; (*if this value is increasing rapidly it may indicate {{es}} backpressure*) -* `beats_stats.metrics.libbeat.output.events.acked` that represents the total number of documents that have been ingested successfully; -* `beats_stats.metrics.libbeat.output.events.failed` that represents the total number of documents that failed to ingest; (*if this value is increasing rapidly it may indicate {{es}} backpressure*) -* `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of documents that failed to ingest due to {{es}} responding with 429 Too many Requests; (*if this value is increasing rapidly it may indicate {{es}} backpressure*) -* `beats_stats.output.elasticsearch.bulk_requests.available` that represents the number of bulk indexers available for making bulk index requests; (*if this value is equal to 0 it may indicate {{es}} backpressure*) -* `beats_stats.output.elasticsearch.bulk_requests.completed` that represents the number of already completed bulk requests; -* `beats_stats.metrics.output.elasticsearch.indexers.active` that represents the number of active bulk indexers that are concurrently processing batches; - -See [{{metricbeat}} documentation](beats://reference/metricbeat/exported-fields-beat.md) for the full list of exported metric fields. - -One likely cause of excessive indexing pressure or rejected requests is undersized {{es}}. To mitigate this, follow the guidance in [Rejected requests](../../../troubleshoot/elasticsearch/rejected-requests.md). - -(Not recommended) If scaling {{es}} resources up is not an option, you can adjust the `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in [Configure the Elasticsearch output](configure-elasticsearch-output.md) to reduce APM Server indexing pressure. However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. Down below a custom configuration example is listed where the number of default buffered documents is roughly doubled while {{es}} indexing retries are decreased simultaneously. This configuration provides a generic example and might not be applicable to your situation. Try adjusting the settings further to see what works for you. - -```yaml -output.elasticsearch: - flush_bytes: "2MB" # double the default value - flush_interval: "2s" # double the default value - max_retries: 1 # decrease the default value - timeout: 60 # decrease the default value -``` - diff --git a/solutions/observability/apps/apm-server-response-codes.md b/solutions/observability/apps/apm-server-response-codes.md deleted file mode 100644 index ecabffb9e1..0000000000 --- a/solutions/observability/apps/apm-server-response-codes.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/apm-common-response-codes.html -applies_to: - stack: ---- - -# APM Server response codes [apm-common-response-codes] - - -## HTTP 400: Data decoding error / Data validation error [apm-bad-request] - -The most likely cause for this error is using incompatible versions of {{apm-agent}} and APM Server. See the [agent/server compatibility matrix](apm-agent-compatibility.md) to verify compatibility. - - -## HTTP 400: Event too large [apm-event-too-large] - -APM agents communicate with the APM server by sending events in an HTTP request. Each event is sent as its own line in the HTTP request body. If events are too large, you should consider increasing the [Max event size](general-configuration-options.md#apm-max_event_size) setting in the APM integration, and adjusting relevant settings in the agent. - - -## HTTP 401: Invalid token [apm-unauthorized] - -Either the [Secret token](secret-token.md) in the request header doesn’t match the secret token configured in the APM integration, or the [API keys](api-keys.md) is invalid. - - -## HTTP 403: Forbidden request [apm-forbidden] - -Either you are sending requests to a [RUM](real-user-monitoring-rum.md) endpoint without RUM enabled, or a request is coming from an origin not specified in the APM integration settings. See the [Allowed origins](configure-real-user-monitoring-rum.md#apm-rum-allow-origins) setting for more information. - - -## HTTP 503: Request timed out waiting to be processed [apm-request-timed-out] - -This happens when APM Server exceeds the maximum number of requests that it can process concurrently. To alleviate this problem, you can try to: reduce the sample rate and/or reduce the collected stack trace information. See [Reduce storage](reduce-storage.md) for more information. - -Another option is to increase processing power. This can be done by either migrating your {{agent}} to a more powerful machine or adding more APM Server instances. - diff --git a/solutions/observability/apps/enable-apm-server-binary-debugging.md b/solutions/observability/apps/enable-apm-server-binary-debugging.md deleted file mode 100644 index d9f4d63e03..0000000000 --- a/solutions/observability/apps/enable-apm-server-binary-debugging.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -navigation_title: "APM Server binary debugging" -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/apm-enable-apm-server-debugging.html -applies_to: - stack: ---- - - - -# Enable APM Server binary debugging [apm-enable-apm-server-debugging] - - -::::{note} -Fleet-managed users should see [View {{agent}} logs](/reference/fleet/monitor-elastic-agent.md) to learn how to view logs and change the logging level of {{agent}}. -:::: - - -By default, APM Server sends all its output to syslog. When you run APM Server in the foreground, you can use the `-e` command line flag to redirect the output to standard error instead. For example: - -```sh -apm-server -e -``` - -The default configuration file is apm-server.yml (the location of the file varies by platform). You can use a different configuration file by specifying the `-c` flag. For example: - -```sh -apm-server -e -c myapm-serverconfig.yml -``` - -You can increase the verbosity of debug messages by enabling one or more debug selectors. For example, to view publisher-related messages, start APM Server with the `publisher` selector: - -```sh -apm-server -e -d "publisher" -``` - -If you want all the debugging output (fair warning, it’s quite a lot), you can use `*`, like this: - -```sh -apm-server -e -d "*" -``` - diff --git a/solutions/observability/apps/get-started-apm-serverless.md b/solutions/observability/apps/get-started-apm-serverless.md new file mode 100644 index 0000000000..43cef02f12 --- /dev/null +++ b/solutions/observability/apps/get-started-apm-serverless.md @@ -0,0 +1,448 @@ +--- +navigation_title: "Elastic Cloud Serverless" +mapped_pages: + - https://www.elastic.co/guide/en/serverless/current/observability-apm-get-started.html +applies_to: + serverless: +--- + +# Get started with traces and APM [observability-apm-get-started] + +::::{admonition} Required role +:class: note + +The **Admin** role or higher is required to send APM data to Elastic. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md). +:::: + +In this guide you’ll learn how to collect and send Application Performance Monitoring (APM) data to Elastic, then explore and visualize the data in real time. + +## Step 1: Add data [add-apm-integration-agents] + +You’ll use APM agents to send APM data from your application to Elastic. Elastic offers APM agents written in several languages and supports OpenTelemetry. Which agent you’ll use depends on the language used in your service. + +To send APM data to Elastic, you must install an APM agent and configure it to send data to your project: + +1. [Create a new {{obs-serverless}} project](/solutions/observability/get-started/create-an-observability-project.md), or open an existing one. +2. To install and configure one or more APM agents, do one of following: + + * In your Observability project, go to **Add data** → **Monitor my application performance** → **Elastic APM** and follow the prompts. + * Use the following instructions: + + ::::::{tab-set} + :::::{tab-item} Go + **1. Install the agent** + + Install the {{apm-agent}} package using `go get`: + + ```go + go get -u go.elastic.co/apm/v2 + ``` + + **2. Configure the agent** + + To simplify development and testing, the agent defaults to sending data to Elastic at `http://localhost:8200`. To send data to an alternative location, you must configure `ELASTIC_APM_SERVER_URL`. + + ```go + # The APM integration host and port + export ELASTIC_APM_SERVER_URL= + + # If you do not specify `ELASTIC_APM_SERVICE_NAME`, the Go agent will use the + # executable name. For example, if your executable is called "my-app.exe", then your + # service will be identified as "my-app". + export ELASTIC_APM_SERVICE_NAME= + + # API keys are used to authorize requests to the APM integration + export ELASTIC_APM_API_KEY= + ``` + + **3. Instrument your application** + + Instrumentation is the process of extending your application’s code to report trace data to Elastic APM. Go applications must be instrumented manually at the source code level. To instrument your applications, use one of the following approaches: + + * [Built-in instrumentation modules](apm-agent-go://reference/builtin-modules.md). + * [Custom instrumentation](apm-agent-go://reference/custom-instrumentation.md) and context propagation with the Go Agent API. + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-go://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-go://reference/configuration.md) + * [Detailed guide to instrumenting Go source code](apm-agent-go://reference/set-up-apm-go-agent.md) + ::::: + + :::::{tab-item} Java + Manually set up and configure the agent with the `-javaagent` JVM option. No application code change is required, but this requires an application restart. See below for more information on this setup method. + + **1. Download the {{apm-agent}}** + + The first step in getting started with the Elastic APM Java agent is to retrieve a copy of the agent JAR. Java agent releases are published to [Maven central](https://repo.maven.apache.org/maven2/). In order to get a copy you can either: + + * download the [latest agent](https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST) or a [previous release](https://mvnrepository.com/artifact/co.elastic.apm/elastic-apm-agent) from Maven central. + * download with `curl`: + + ```bash + curl -o 'elastic-apm-agent.jar' -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST' + ``` + + **2. Add `-javaagent` flag** + + When starting your application, add the JVM flag: `-javaagent:/path/to/elastic-apm-agent-.jar`. + + **3. Configure** + + Different application servers have different ways of setting the `-javaagent` flag and system properties. Start your application (for example a Spring Boot application or other embedded servers) and add the `-javaagent` JVM flag. Use the `-D` prefix to configure the agent using system properties: + + ```bash + java -javaagent:/path/to/elastic-apm-agent-.jar -Delastic.apm.service_name=my-cool-service -Delastic.apm.application_packages=org.example,org.another.example -Delastic.apm.server_url=http://127.0.0.1:8200 -jar my-application.jar + ``` + + Refer to [Manual setup with `-javaagent` flag](apm-agent-java://reference/setup-javaagent.md) to learn more. + + **Alternate setup methods** + + * **Automatic setup with `apm-agent-attach-cli.jar`** Automatically set up the agent without needing to alter the configuration of your JVM or application server. This method requires no changes to application code or JVM options, and allows attaching to a running JVM. Refer to the [Java agent documentation](apm-agent-java://reference/setup-attach-cli.md) for more information on this setup method. + * **Programmatic API setup to self-attach** Set up the agent with a one-line code change and an extra `apm-agent-attach` dependency. This method requires no changes to JVM options, and the agent artifact is embedded within the packaged application binary. Refer to the [Java agent documentation](apm-agent-java://reference/setup-attach-api.md) for more information on this setup method. + ::::: + + :::::{tab-item} .NET + **Set up the {{apm-agent}}** + + * **Profiler runtime instrumentation**: The agent supports auto instrumentation without any code change and without any recompilation of your projects. See [Profiler auto instrumentation](apm-agent-dotnet://reference/setup-auto-instrumentation.md). + * **NuGet packages**: The agent ships as a set of [NuGet packages](apm-agent-dotnet://reference/nuget-packages.md) available on [nuget.org](https://nuget.org). You can add the Agent and specific instrumentations to a .NET application by referencing one or more of these packages and following the package documentation. + * **Host startup hook**: On .NET Core 3.0+ or .NET 5+, the agent supports auto instrumentation without any code change and without any recompilation of your projects. See [Zero code change setup on .NET Core](apm-agent-dotnet://reference/setup-dotnet-net-core.md) for more details. + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-dotnet://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-dotnet://reference/configuration.md) + ::::: + + :::::{tab-item} Node.js + **1. Install the {{apm-agent}}** + + Install the {{apm-agent}} for Node.js as a dependency to your application. + + ```js + npm install elastic-apm-node --save + ``` + + **2. Initialization** + + It’s important that the agent is started before you require *any* other modules in your Node.js application - i.e. before `http` and before your router etc. + + This means that you should probably require and start the agent in your application’s main file (usually `index.js`, `server.js` or `app.js`). + + Here’s a simple example of how Elastic APM is normally required and started: + + ```js + // Add this to the VERY top of the first file loaded in your app + var apm = require('elastic-apm-node').start({ + // Override service name from package.json + // Allowed characters: a-z, A-Z, 0-9, -, _, and space + serviceName: '', + + // API keys are used to authorize requests to the APM integration + apiKey: '', + + // Set custom APM integration host and port (default: http://127.0.0.1:8200) + serverUrl: '', + }) + ``` + + The agent will now monitor the performance of your application and record any uncaught exceptions. + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-nodejs://reference/supported-technologies.md) + * [Babel/ES Modules](apm-agent-nodejs://reference/advanced-setup.md) + * [Advanced configuration](apm-agent-nodejs://reference/configuring-agent.md) + ::::: + + :::::{tab-item} PHP + **1. Install the agent** + + Install the PHP agent using one of the [published packages](https://github.com/elastic/apm-agent-php/releases/latest). + + To use the RPM Package (RHEL/CentOS and Fedora): + + ```php + rpm -ivh .rpm + ``` + + To use the DEB package (Debian and Ubuntu): + + ```php + dpkg -i .deb + ``` + + To use the APK package (Alpine): + + ```php + apk add --allow-untrusted .apk + ``` + + If you can’t find your distribution, you can install the agent by building it from the source. The following instructions will build the APM agent using the same docker environment that Elastic uses to build our official packages. + + ::::{note} + The agent is currently only available for Linux operating system. + + :::: + + 1. Download the [agent source](https://github.com/elastic/apm-agent-php/). + 2. Execute the following commands to build the agent and install it: + + ```bash + cd apm-agent-php + # for linux glibc - libc distributions (Ubuntu, Redhat, etc) + export BUILD_ARCHITECTURE=linux-x86-64 + # for linux with musl - libc distributions (Alpine) + export BUILD_ARCHITECTURE=linuxmusl-x86-64 + # provide a path to php-config tool + export PHP_CONFIG=php-config + + # build extensions + make -f .ci/Makefile build + + # run extension tests + PHP_VERSION=`$PHP_CONFIG --version | cut -d'.' -f 1,2` make -f .ci/Makefile run-phpt-tests + + # install agent extensions + sudo cp agent/native/_build/${BUILD_ARCHITECTURE}-release/ext/elastic_apm-*.so `$PHP_CONFIG --extension-dir` + + # install automatic loader + sudo cp agent/native/_build/${BUILD_ARCHITECTURE}-release/loader/code/elastic_apm_loader.so `$PHP_CONFIG --extension-dir` + ``` + + **2. Enable and configure the APM agent** + + Enable and configure your agent inside of the `php.ini` file: + + ```ini + extension=elastic_apm_loader.so + elastic_apm.bootstrap_php_part_file=/agent/php/bootstrap_php_part.php + ``` + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-php://reference/supported-technologies.md) + * [Configuration](apm-agent-php://reference/configuration.md) + ::::: + + :::::{tab-item} Python + Django and Flask are two of several frameworks that the Elastic APM Python Agent supports. For a complete list of supported technologies, refer to the [Elastic APM Python Agent documentation](apm-agent-python://reference/supported-technologies.md). + + *Django* + + ```python + $ pip install elastic-apm + ``` + + **1. Install the {{apm-agent}}** + + Install the {{apm-agent}} for Python as a dependency. + + ```python + $ pip install elastic-apm + ``` + + **2. Configure the {{apm-agent}}** + + Agents are libraries that run inside of your application process. APM services are created programmatically based on the `SERVICE_NAME`. + + ```python + # Add the agent to the installed apps + INSTALLED_APPS = ( + 'elasticapm.contrib.django', + # ... + ) + + ELASTIC_APM = { + # Set required service name. Allowed characters: + # a-z, A-Z, 0-9, -, _, and space + 'SERVICE_NAME': '', + + # API keys are used to authorize requests to the APM integration + 'API_KEY': '', + + # Set custom APM integration host and port (default: http://localhost:8200) + 'SERVER_URL': '', + } + + # To send performance metrics, add our tracing middleware: + MIDDLEWARE = ( + 'elasticapm.contrib.django.middleware.TracingMiddleware', + #... + ) + ``` + + *Flask* + + **1. Install the {{apm-agent}}** + + Install the {{apm-agent}} for Python as a dependency. + + ```python + $ pip install elastic-apm[flask] + ``` + + **2. Configure the {{apm-agent}}** + + Agents are libraries that run inside of your application process. APM services are created programmatically based on the `SERVICE_NAME`. + + ```python + # initialize using environment variables + from elasticapm.contrib.flask import ElasticAPM + app = Flask(__name__) + apm = ElasticAPM(app) + + # or configure to use ELASTIC_APM in your application settings + from elasticapm.contrib.flask import ElasticAPM + app.config['ELASTIC_APM'] = { + # Set required service name. Allowed characters: + # a-z, A-Z, 0-9, -, _, and space + 'SERVICE_NAME': '', + + # API keys are used to authorize requests to the APM integration + 'API_KEY': '', + + # Set custom APM integration host and port (default: http://localhost:8200) + 'SERVER_URL': '', + } + + apm = ElasticAPM(app) + ``` + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-python://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-python://reference/configuration.md) + ::::: + + :::::{tab-item} Ruby + **1. Install the {{apm-agent}}** + + Add the agent to your Gemfile. + + ```ruby + gem 'elastic-apm' + ``` + + **2. Configure the agent** + + *Ruby on Rails* + + APM is automatically started when your app boots. Configure the agent by creating the config file `config/elastic_apm.yml`: + + ```ruby + # config/elastic_apm.yml: + + # Set service name - allowed characters: a-z, A-Z, 0-9, -, _ and space + # Defaults to the name of your Rails app + service_name: 'my-service' + + # API keys are used to authorize requests to the APM integration + api_key: '' + + # Set custom APM integration host and port (default: http://localhost:8200) + server_url: 'http://localhost:8200' + ``` + + *Rack* + + For Rack or a compatible framework, like Sinatra, include the middleware in your app and start the agent. + + ```ruby + # config.ru + + app = lambda do |env| + [200, {'Content-Type' => 'text/plain'}, ['ok']] + end + + # Wraps all requests in transactions and reports exceptions + use ElasticAPM::Middleware + + # Start an instance of the Agent + ElasticAPM.start(service_name: 'NothingButRack') + + run app + + # Gracefully stop the agent when process exits. + # Makes sure any pending transactions are sent. + at_exit { ElasticAPM.stop } + ``` + + Create a config file `config/elastic_apm.yml`: + + ```ruby + # config/elastic_apm.yml: + + # Set service name - allowed characters: a-z, A-Z, 0-9, -, _ and space + # Defaults to the name of your Rack app's class. + service_name: 'my-service' + + # API keys are used to authorize requests to the APM integration + api_key: '' + + # Set custom APM integration host and port (default: http://localhost:8200) + server_url: 'http://localhost:8200' + ``` + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-ruby://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-ruby://reference/configuration.md) + ::::: + + :::::{tab-item} OpenTelemetry + Elastic integrates with OpenTelemetry, allowing you to reuse your existing instrumentation to easily send observability data to Elastic. + + For more information on how to combine Elastic and OpenTelemetry, refer to [OpenTelemetry](/solutions/observability/apps/use-opentelemetry-with-apm.md). + ::::: + :::::: + + While there are many configuration options, all APM agents require: + + * **Service name**: The APM integration maps an instrumented service’s name — defined in each {{apm-agent}}'s configuration — to the index where its data is stored. Service names are case-insensitive and must be unique. + + For example, you cannot have a service named `Foo` and another named `foo`. Special characters will be removed from service names and replaced with underscores (`_`). + + * **Server URL**: The host and port that the managed intake service listens for events on. + + To find the URL for your project: + + 1. Go to the [Cloud console](https://cloud.elastic.co/). + 2. Next to your project, select **Manage**. + 3. Next to *Endpoints*, select **View**. + 4. Copy the *APM endpoint*. + + * **API key**: Authentication method for communication between {{apm-agent}} and the managed intake service. + + You can create and delete API keys in Applications Settings: + + 1. Go to any page in the *Applications* section of the main menu. + 2. Click **Settings** in the top bar. + 3. Go to the **Agent keys** tab. + + * **Environment**: The name of the environment this service is deployed in, for example "production" or "staging". + + Environments allow you to easily filter data on a global level in the UI. It’s important to be consistent when naming environments across agents. + +3. If you’re using the step-by-step instructions in the UI, after you’ve installed and configured an agent, you can click **Check Agent Status** to verify that the agent is sending data. + +To learn more about APM agents, including how to fine-tune how agents send traces to Elastic, refer to [Collect application data](/solutions/observability/apps/collect-application-data.md). + +## Step 2: View your data [view-apm-integration-data] + +After one or more APM agents are installed and successfully sending data, you can view application performance monitoring data in the UI. + +In the *Applications* section of the main menu, select **Service Inventory**. This will show a high-level overview of the health and general performance of all your services. + +Learn more about visualizing APM data in [View and analyze data](/solutions/observability/apps/view-analyze-data.md). + +::::{tip} +Not seeing any data? Find helpful tips in [Troubleshooting](/troubleshoot/observability/apm.md). + +:::: + +## Next steps [observability-apm-get-started-next-steps] + +Now that data is streaming into your project, take your investigation to a deeper level. Learn how to use [Elastic’s built-in visualizations for APM data](/solutions/observability/apps/view-analyze-data.md), [alert on APM data](/solutions/observability/incident-management/alerting.md), or [fine-tune how agents send traces to Elastic](/solutions/observability/apps/collect-application-data.md). diff --git a/solutions/observability/apps/processing-performance.md b/solutions/observability/apps/processing-performance.md deleted file mode 100644 index 978f77a9ac..0000000000 --- a/solutions/observability/apps/processing-performance.md +++ /dev/null @@ -1,47 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/apm-processing-and-performance.html -applies_to: - stack: ---- - -# Processing and performance [apm-processing-and-performance] - -APM Server performance depends on a number of factors: memory and CPU available, network latency, transaction sizes, workload patterns, agent and server settings, versions, and protocol. - -We tested several scenarios to help you understand how to size the APM Server so that it can keep up with the load that your Elastic APM agents are sending: - -* Using the *CPU Optimized* hardware template on AWS, GCP and Azure on {{ecloud}} with the following instances (for more details see [Hardware Profiles](../../../deploy-manage/deploy/elastic-cloud/ec-change-hardware-profile.md)): - - * AWS: c6gd - * Azure: fsv2 - * GCP: n2.68x32x45 - -* For each hardware template, testing with several sizes: 1 GB, 4 GB, 8 GB, and 32 GB. -* For each size, using a fixed number of APM agents: 10 agents for 1 GB, 30 agents for 4 GB, 60 agents for 8 GB, and 240 agents for 32 GB. -* In all scenarios, using medium sized events. Events include [transactions](transactions.md) and [spans](spans.md). - -::::{note} -You will also need to scale up {{es}} accordingly, potentially with an increased number of shards configured. For more details on scaling {{es}}, refer to the [{{es}} documentation](../../../deploy-manage/index.md). -:::: - - -The results below include numbers for a synthetic workload. You can use the results of our tests to guide your sizing decisions, however, **performance will vary based on factors unique to your use case** like your specific setup, the size of APM event data, and the exact number of agents. - -| Profile / Cloud | AWS | Azure | GCP | -| --- | --- | --- | --- | -| **1 GB**
(10 agents) | 15,000
events/second | 14,000
events/second | 17,000
events/second | -| **4 GB**
(30 agents) | 29,000
events/second | 26,000
events/second | 35,000
events/second | -| **8 GB**
(60 agents) | 50,000
events/second | 34,000
events/second | 48,000
events/second | -| **16 GB**
(120 agents) | 96,000
events/second | 57,000
events/second | 90,000
events/second | -| **32 GB**
(240 agents) | 133,000
events/second | 89,000
events/second | 143,000
events/second | - -Don’t forget that the APM Server is stateless. Several instances running do not need to know about each other. This means that with a properly sized {{es}} instance, APM Server scales out linearly. - -::::{note} -RUM deserves special consideration. The RUM agent runs in browsers, and there can be many thousands reporting to an APM Server with very variable network latency. -:::: - - -Alternatively or in addition to scaling the APM Server, consider decreasing the ingestion volume. Read more in [Reduce storage](reduce-storage.md). - diff --git a/solutions/observability/apps/what-happens-when-apm-server-elasticsearch-is-down.md b/solutions/observability/apps/what-happens-when-apm-server-elasticsearch-is-down.md deleted file mode 100644 index cdd3d748e2..0000000000 --- a/solutions/observability/apps/what-happens-when-apm-server-elasticsearch-is-down.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/apm-server-es-down.html -applies_to: - stack: ---- - -# What happens when APM Server or Elasticsearch is down? [apm-server-es-down] - -**If {{es}} is down** - -APM Server does not have an internal queue to buffer requests, but instead leverages an HTTP request timeout to act as back-pressure. If {{es}} goes down, the APM Server will eventually deny incoming requests. Both the APM Server and {{apm-agent}}(s) will issue logs accordingly. - -**If APM Server is down** - -Some agents have internal queues or buffers that will temporarily store data if the APM Server goes down. As a general rule of thumb, queues fill up quickly. Assume data will be lost if APM Server goes down. Adjusting these queues/buffers can increase the agent’s overhead, so use caution when updating default values. - -* **Go agent** - Circular buffer with configurable size: [`ELASTIC_APM_BUFFER_SIZE`](apm-agent-go://reference/configuration.md#config-api-buffer-size). -* **Java agent** - Internal buffer with configurable size: [`max_queue_size`](apm-agent-java://reference/config-reporter.md#config-max-queue-size). -* **Node.js agent** - No internal queue. Data is lost. -* **PHP agent** - No internal queue. Data is lost. -* **Python agent** - Internal [Transaction queue](apm-agent-python://reference/performance-tuning.md#tuning-queue) with configurable size and time between flushes. -* **Ruby agent** - Internal queue with configurable size: [`api_buffer_size`](apm-agent-ruby://reference/configuration.md#config-api-buffer-size). -* **RUM agent** - No internal queue. Data is lost. -* **.NET agent** - No internal queue. Data is lost. diff --git a/solutions/toc.yml b/solutions/toc.yml index b9d5531848..983ce0eb52 100644 --- a/solutions/toc.yml +++ b/solutions/toc.yml @@ -104,6 +104,7 @@ toc: children: - file: observability/apps/get-started-with-apm.md children: + - file: observability/apps/get-started-apm-serverless.md - file: observability/apps/fleet-managed-apm-server.md - file: observability/apps/apm-server-binary.md - file: observability/apps/learn-about-application-data-types.md @@ -257,11 +258,6 @@ toc: - file: observability/apps/elastic-apm-agent-configuration-api.md - file: observability/apps/opentelemetry-intake-api.md - file: observability/apps/jaeger-event-intake.md - - file: observability/apps/what-happens-when-apm-server-elasticsearch-is-down.md - - file: observability/apps/apm-server-response-codes.md - - file: observability/apps/processing-performance.md - - file: observability/apps/enable-apm-server-binary-debugging.md - - file: observability/apps/apm-server-performance-diagnostic.md - file: observability/apps/upgrade.md children: - file: observability/apps/apm-agent-compatibility.md From c41c8b827e63d321d6dbc720082a7fe353c1314a Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 4 Apr 2025 15:44:57 -0500 Subject: [PATCH 2/9] add note about apm server vs. mis --- .../apps/_snippets/apm-server-vs-mis.md | 3 ++ solutions/observability/apps/metrics.md | 32 ++----------------- solutions/observability/apps/spans.md | 13 ++------ .../apps/transaction-sampling.md | 31 ++---------------- ...-opentelemetry-collectors-language-sdks.md | 16 ++-------- .../observability/apps/use-apm-securely.md | 3 ++ .../apps/use-opentelemetry-with-apm.md | 10 ++---- 7 files changed, 20 insertions(+), 88 deletions(-) create mode 100644 solutions/observability/apps/_snippets/apm-server-vs-mis.md diff --git a/solutions/observability/apps/_snippets/apm-server-vs-mis.md b/solutions/observability/apps/_snippets/apm-server-vs-mis.md new file mode 100644 index 0000000000..5225feee4e --- /dev/null +++ b/solutions/observability/apps/_snippets/apm-server-vs-mis.md @@ -0,0 +1,3 @@ +:::{admonition} APM Server vs managed intake service +In {{ech}}, the _APM Server_ receives data from Elastic APM agents and transforms it into Elasticsearch documents. In {{serverless-full}} there is in fact no APM Server running, instead the _managed intake service_ receives and transforms data. +::: \ No newline at end of file diff --git a/solutions/observability/apps/metrics.md b/solutions/observability/apps/metrics.md index 0f33429c3d..72ed999060 100644 --- a/solutions/observability/apps/metrics.md +++ b/solutions/observability/apps/metrics.md @@ -8,12 +8,14 @@ applies_to: # Metrics [apm-data-model-metrics] +:::{include} _snippets/apm-server-vs-mis.md +::: + **Metrics** measure the state of a system by gathering information on a regular interval. There are two types of APM metrics: * **System metrics**: Basic infrastructure and application metrics. * **Calculated metrics**: Aggregated trace event metrics used to power visualizations in the Applications UI. - ## System metrics [_system_metrics] APM agents automatically pick up basic host-level metrics, including system and process-level CPU and memory metrics. Agent specific metrics are also available, like [JVM metrics](apm-agent-java://reference/metrics.md) in the Java Agent, and [Go runtime](apm-agent-go://reference/metrics.md) metrics in the Go Agent. @@ -24,7 +26,6 @@ Infrastructure and application metrics are important sources of information when Most agents limit keyword fields to 1024 characters, non-keyword fields (e.g. `system.memory.total`) to 10,000 characters. :::: - Metrics are stored in metric indices. For a full list of tracked metrics, see the relevant agent documentation: @@ -35,7 +36,6 @@ For a full list of tracked metrics, see the relevant agent documentation: * [Python](apm-agent-python://reference/metrics.md) * [Ruby](apm-agent-ruby://reference/metrics.md) - ### Example system metric document [_example_system_metric_document] This example shows what system metric documents can look like when indexed in {{es}}. @@ -117,8 +117,6 @@ This example contains JVM metrics produced by the {{apm-java-agent}}. and contai :::: - - ## Calculated metrics [_calculated_metrics] APM agents and APM Server calculate metrics from trace events to power visualizations in the Applications UI. @@ -127,7 +125,6 @@ Calculated metrics are an implementation detail and while we aim for stability f These metrics are described below. - ### Breakdown metrics [_breakdown_metrics] To power the [Time spent by span type](transactions-2.md) graph, agents collect summarized metrics about the timings of spans and transactions, broken down by span type. @@ -144,8 +141,6 @@ You can filter and group by these dimensions: * `span.type`: The type of the span, for example `app`, `template` or `db` * `span.subtype`: The sub-type of the span, for example `mysql` (optional) - - ### Example breakdown metric document [_example_breakdown_metric_document] This example shows what breakdown metric documents can look like when indexed in {{es}}. @@ -238,8 +233,6 @@ This example shows what breakdown metric documents can look like when indexed in :::: - - ### Transaction metrics [_transaction_metrics] To power [{{kib}} Applications UI](overviews.md) visualizations, APM Server aggregates transaction events into latency distribution metrics. @@ -288,10 +281,8 @@ You can filter and group by these dimensions (some of which are optional, for ex * `transaction.root`: A boolean flag indicating whether the transaction is the root of a trace * `transaction.type`: The type of the transaction, for example `request` - The `@timestamp` field of these documents holds the start of the aggregation interval. - ### Example transaction document [_example_transaction_document_2] This example shows what transaction documents can look like when indexed in {{es}}. @@ -428,8 +419,6 @@ This example shows what transaction documents can look like when indexed in {{es :::: - - ### Service-transaction metrics [_service_transaction_metrics] To power [{{kib}} Applications UI](overviews.md) visualizations, APM Server aggregates transaction events into service-transaction metrics. Service-transaction metrics are similar to transaction metrics, but they usually have a much lower cardinality as they have significantly fewer dimensions. The UI uses them when fewer details of the transactions are needed. @@ -450,10 +439,8 @@ You can filter and group by these dimensions: * `service.name`: The name of the service that made the request * `transaction.type`: The type of the enclosing transaction, for example `request` - The `@timestamp` field of these documents holds the start of the aggregation interval. - ### Example service-transaction document [_example_service_transaction_document] This example shows what service-transaction documents can look like when indexed in {{es}}. @@ -966,8 +953,6 @@ This example shows what service-transaction documents can look like when indexed :::: - - ### Service-destination metrics [_service_destination_metrics] To power [{{kib}} Applications UI](overviews.md) visualizations, APM Server aggregates span events into service-destination metrics. @@ -992,10 +977,8 @@ You can filter and group by these dimensions: * `span.destination.service.resource`: The destination service resource, for example `mysql` * `span.name`: The name of the operation, for example `SELECT FROM table_name`. - The `@timestamp` field of these documents holds the start of the aggregation interval. - ### Example service-destination document [_example_service_destination_document] This example shows what service-destination documents can look like when indexed in {{es}}. @@ -1063,8 +1046,6 @@ This example shows what service-destination documents can look like when indexed :::: - - ### Service-summary metrics [_service_summary_metrics] To power [{{kib}} Applications UI](overviews.md) visualizations, APM Server aggregates transaction, error, log, and metric events into service-summary metrics. @@ -1083,7 +1064,6 @@ You can filter and group by these dimensions: The `@timestamp` field of these documents holds the start of the aggregation interval. - ### Example service-summary document [_example_service_summary_document] This example shows what service-summary documents can look like when indexed in {{es}}. @@ -1132,8 +1112,6 @@ This example shows what service-summary documents can look like when indexed in :::: - - ## Data streams [_data_streams_4] Metrics are stored in the following data streams: @@ -1147,12 +1125,10 @@ Metrics are stored in the following data streams: See [Data streams](data-streams.md) to learn more. - ## Aggregated metrics: limits and overflows [_aggregated_metrics_limits_and_overflows] For all aggregated metrics, namely transaction, service-transaction, service-destination, and service-summary metrics, there are limits on the number of unique groups tracked at any given time. - ### Limits [_limits] Note that all the below limits may change in the future with further improvements. @@ -1164,10 +1140,8 @@ Note that all the below limits may change in the future with further improvement * For service-destination metrics, there is an additional limit of 5000 total service destination groups per GB of APM Server starting with 10000 service destination groups for 1 GB APM Server, and each service may only consume up to 10% of the service destination groups, which is 1000 service destination groups for 1GB APM Server with 500 increment per GB of APM Server. * For service-summary metrics, there is no additional limit. - In the previous metrics, a service is defined as a combination of `service.name`, `service.environment`, `service.language.name` and `agent.name`. - ### Overflows [_overflows] When a dimension has a high cardinality and exceeds the limit, the metrics will be aggregated under a dedicated overflow bucket. diff --git a/solutions/observability/apps/spans.md b/solutions/observability/apps/spans.md index aa4f5640ee..97fc589046 100644 --- a/solutions/observability/apps/spans.md +++ b/solutions/observability/apps/spans.md @@ -9,6 +9,9 @@ applies_to: # Spans [apm-data-model-spans] +:::{include} _snippets/apm-server-vs-mis.md +::: + % Stateful only until Span Compressions **Spans** contain information about the execution of a specific code path. They measure from the start to the end of an activity, and they can have a parent/child relationship with other spans. @@ -27,8 +30,6 @@ Among other things, spans can contain: Most agents limit keyword fields, like `span.id`, to 1024 characters, and non-keyword fields, like `span.start.us`, to 10,000 characters. :::: - - ## Dropped spans [apm-data-model-dropped-spans] For performance reasons, APM agents can choose to sample or omit spans purposefully. This can be useful in preventing edge cases, like long-running transactions with over 100 spans, that would otherwise overload both the Agent and the APM Server. When this occurs, the Applications UI will display the number of spans dropped. @@ -45,12 +46,10 @@ To configure the number of spans recorded per transaction, see the relevant Agen * Python: [`transaction_max_spans`](apm-agent-python://reference/configuration.md#config-transaction-max-spans) * Ruby: [`transaction_max_spans`](apm-agent-ruby://reference/configuration.md#config-transaction-max-spans) - ## Missing spans [apm-data-model-missing-spans] Agents stream spans to the APM Server separately from their transactions. Because of this, unforeseen errors may cause spans to go missing. Agents know how many spans a transaction should have; if the number of expected spans does not equal the number of spans received by the APM Server, the Applications UI will calculate the difference and display a message. - ## Data streams [_data_streams] Spans are stored with transactions in the following data streams: @@ -60,7 +59,6 @@ Spans are stored with transactions in the following data streams: See [Data streams](../../../solutions/observability/apps/data-streams.md) to learn more. - ## Example span document [_example_span_document] This example shows what span documents can look like when indexed in {{es}}. @@ -424,7 +422,6 @@ This example shows what span documents can look like when indexed in {{es}}. :::: - ## Span compression [apm-spans-span-compression] In some cases, APM agents may collect large amounts of very similar or identical spans in a transaction. For example, this can happen if spans are captured inside a loop or in unoptimized SQL queries that use multiple queries instead of joins to fetch related data. @@ -441,12 +438,10 @@ Regardless of the compression strategy, a span is eligible for compression if: * It is an *exit* span (such as database query spans). * Its outcome is not `"failure"`. - ### Compression strategies [apm-span-compression-strategy] The {{apm-agent}} selects between two strategies to decide if adjacent spans can be compressed. In both strategies, only one previous span needs to be kept in memory. This ensures that the agent doesn’t require large amounts of memory to enable span compression. - #### Same-Kind strategy [apm-span-compression-same] The agent uses the same-kind strategy if two adjacent spans have the same: @@ -464,14 +459,12 @@ The agent uses the exact-match strategy if two adjacent spans have the same: * span subtype * `destination.service.resource` (e.g. database name) - ### Settings [apm-span-compression-settings] You can specify the maximum span duration in the agent’s configuration settings. Spans with a duration longer than the specified value will not be compressed. For the "Same-Kind" strategy, the default maximum span duration is 0 milliseconds, which means that the "Same-Kind" strategy is disabled by default. For the "Exact-Match" strategy, the default limit is 50 milliseconds. - ### Agent support [apm-span-compression-support] Support for span compression is available in the following agents and can be configured using the options listed below: diff --git a/solutions/observability/apps/transaction-sampling.md b/solutions/observability/apps/transaction-sampling.md index e7497a0141..2772ffa475 100644 --- a/solutions/observability/apps/transaction-sampling.md +++ b/solutions/observability/apps/transaction-sampling.md @@ -9,6 +9,9 @@ applies_to: # Transaction sampling [apm-sampling] +:::{include} _snippets/apm-server-vs-mis.md +::: + [Distributed tracing](../../../solutions/observability/apps/traces.md) can generate a substantial amount of data. More data can mean higher costs and more noise. Sampling aims to lower the amount of data ingested and the effort required to analyze that data — all while still making it easy to find anomalous patterns in your applications, detect outages, track errors, and lower mean time to recovery (MTTR). Elastic APM supports two types of sampling: @@ -18,7 +21,6 @@ Elastic APM supports two types of sampling: * [Head-based sampling](../../../solutions/observability/apps/transaction-sampling.md#apm-head-based-sampling) * [Tail-based sampling](../../../solutions/observability/apps/transaction-sampling.md#apm-tail-based-sampling) - ## Head-based sampling [apm-head-based-sampling] In head-based sampling, the sampling decision for each trace is made when the trace is initiated. Each trace has a defined and equal probability of being sampled. @@ -29,7 +31,6 @@ Head-based sampling is quick and easy to set up. Its downside is that it’s ent See [Configure head-based sampling](../../../solutions/observability/apps/transaction-sampling.md#apm-configure-head-based-sampling) to get started. - ### Distributed tracing [distributed-tracing-examples] In a distributed trace, the sampling decision is still made when the trace is initiated. Each subsequent service respects the initial service’s sampling decision, regardless of its configured sample rate; the result is a sampling percentage that matches the initiating service. @@ -52,7 +53,6 @@ In the example in *Figure 2*, `Service A` initiates four transactions and has a :screenshot: ::: - #### Trace continuation strategies with distributed tracing [_trace_continuation_strategies_with_distributed_tracing] In addition to setting the sample rate, you can also specify which *trace continuation strategy* to use. There are three trace continuation strategies: `continue`, `restart`, and `restart_external`. @@ -79,7 +79,6 @@ In the example in *Figure 4*, `Service A` and `Service B` are Elastic-monitored :title: Using the `restart` trace continuation strategy ::: - ### OpenTelemetry [_opentelemetry] Head-based sampling is implemented directly in the APM agents and SDKs. The sample rate must be propagated between services and the managed intake service in order to produce accurate metrics. @@ -109,7 +108,6 @@ Tail-based sampling is only supported when writing to {{es}}. If you are using a :::: - In tail-based sampling, the sampling decision for each trace is made after the trace has completed. This means all traces will be analyzed against a set of rules, or policies, which will determine the rate at which they are sampled. Unlike head-based sampling, each trace does not have an equal probability of being sampled. Because slower traces are more interesting than faster ones, tail-based sampling uses weighted random sampling — so traces with a longer root transaction duration are more likely to be sampled than traces with a fast root transaction duration. @@ -118,7 +116,6 @@ A downside of tail-based sampling is that it results in more data being sent fro See [Configure tail-based sampling](../../../solutions/observability/apps/transaction-sampling.md#apm-configure-tail-based-sampling) to get started. - ### Distributed tracing with tail-based sampling [_distributed_tracing_with_tail_based_sampling] With tail-based sampling, all traces are observed and a sampling decision is only made once a trace completes. @@ -129,7 +126,6 @@ In this example, `Service A` initiates four transactions. If our sample rate is :alt: Distributed tracing and tail based sampling example one ::: - ### OpenTelemetry with tail-based sampling [_opentelemetry_with_tail_based_sampling] Tail-based sampling is implemented entirely in APM Server, and will work with traces sent by either Elastic APM agents or OpenTelemetry SDKs. @@ -201,7 +197,6 @@ These calculation methods ensure that the APM app provides the most accurate met 1 Real User Monitoring (RUM) traces are an exception to this rule. The {{kib}} apps that utilize RUM data depend on transaction events, so non-sampled RUM traces retain transaction data — only span data is dropped. - ## Sample rates [_sample_rates] What’s the best sampling rate? Unfortunately, there isn’t one. Sampling is dependent on your data, the throughput of your application, data retention policies, and other factors. A sampling rate from `.1%` to `100%` would all be considered normal. You’ll likely decide on a unique sample rate for different scenarios. Here are some examples: @@ -213,22 +208,18 @@ What’s the best sampling rate? Unfortunately, there isn’t one. Sampling is d Regardless of the above, cost conscious customers are likely to be fine with a lower sample rate. - ## Configure head-based sampling [apm-configure-head-based-sampling] There are three ways to adjust the head-based sampling rate of your APM agents: - ### Dynamic configuration [_dynamic_configuration] The transaction sample rate can be changed dynamically (no redeployment necessary) on a per-service and per-environment basis with [{{apm-agent}} Configuration](../../../solutions/observability/apps/apm-agent-central-configuration.md) in {{kib}}. - ### {{kib}} API configuration [_kib_api_configuration] {{apm-agent}} configuration exposes an API that can be used to programmatically change your agents' sampling rate. For examples, refer to the [Agent configuration API reference](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-agent-configuration). - ### {{apm-agent}} configuration [_apm_agent_configuration] Each agent provides a configuration value used to set the transaction sample rate. See the relevant agent’s documentation for more details: @@ -241,7 +232,6 @@ Each agent provides a configuration value used to set the transaction sample rat * Python: [`transaction_sample_rate`](apm-agent-python://reference/configuration.md#config-transaction-sample-rate) * Ruby: [`transaction_sample_rate`](apm-agent-ruby://reference/configuration.md#config-transaction-sample-rate) - ## Configure tail-based sampling [apm-configure-tail-based-sampling] Enable tail-based sampling with [Enable tail-based sampling](../../../solutions/observability/apps/tail-based-sampling.md#sampling-tail-enabled-ref). When enabled, trace events are mapped to sampling policies. Each sampling policy must specify a sample rate, and can optionally specify other conditions. All of the policy conditions must be true for a trace event to match it. @@ -252,8 +242,6 @@ Trace events are matched to policies in the order specified. Each policy list mu Please note that from version `9.0.0` APM Server has an unlimited storage limit, but will stop writing when the disk where the database resides reaches 80% usage. Due to how the limit is calculated and enforced, the actual disk space may still grow slightly over this disk usage based limit, or any configured storage limit. :::: - - ### Example configuration [_example_configuration] This example defines three tail-based sampling polices: @@ -272,14 +260,10 @@ This example defines three tail-based sampling polices: 2. Samples 1% of traces in `production` with the trace name `"GET /not_important_route"` 3. Default policy to sample all remaining traces at 10%, e.g. traces in a different environment, like `dev`, or traces with any other name - - ### Configuration reference [_configuration_reference] - #### Top-level tail-based sampling settings [_top_level_tail_based_sampling_settings] - ##### Enable tail-based sampling [sampling-tail-enabled] Set to `true` to enable tail based sampling. Disabled by default. (bool) @@ -289,7 +273,6 @@ Set to `true` to enable tail based sampling. Disabled by default. (bool) | APM Server binary | `sampling.tail.enabled` | | Fleet-managed | `Enable tail-based sampling` | - ##### Interval [sampling-tail-interval] Synchronization interval for multiple APM Servers. Should be in the order of tens of seconds or low minutes. Default: `1m` (1 minute). (duration) @@ -299,7 +282,6 @@ Synchronization interval for multiple APM Servers. Should be in the order of ten | APM Server binary | `sampling.tail.interval` | | Fleet-managed | `Interval` | - ##### Policies [sampling-tail-policies] Criteria used to match a root transaction to a sample rate. @@ -311,7 +293,6 @@ Policies map trace events to a sample rate. Each policy must specify a sample ra | APM Server binary | `sampling.tail.policies` | | Fleet-managed | `Policies` | - ##### Storage limit [sampling-tail-storage_limit] The amount of storage space allocated for trace events matching tail sampling policies. Caution: Setting this limit higher than the allowed space may cause APM Server to become unhealthy. @@ -329,32 +310,26 @@ Default: `0GB`. (text) | APM Server binary | `sampling.tail.storage_limit` | | Fleet-managed | `Storage limit` | - #### Policy settings [_policy_settings] - ##### **`sample_rate`** [sampling-tail-sample-rate] The sample rate to apply to trace events matching this policy. Required in each policy. The sample rate must be greater than or equal to `0` and less than or equal to `1`. For example, a `sample_rate` of `0.01` means that 1% of trace events matching the policy will be sampled. A `sample_rate` of `1` means that 100% of trace events matching the policy will be sampled. (int) - ##### **`trace.name`** [sampling-tail-trace-name] The trace name for events to match a policy. A match occurs when the configured `trace.name` matches the `transaction.name` of the root transaction of a trace. A root transaction is any transaction without a `parent.id`. (string) - ##### **`trace.outcome`** [sampling-tail-trace-outcome] The trace outcome for events to match a policy. A match occurs when the configured `trace.outcome` matches a trace’s `event.outcome` field. Trace outcome can be `success`, `failure`, or `unknown`. (string) - ##### **`service.name`** [sampling-tail-service-name] The service name for events to match a policy. (string) - ##### **`service.environment`** [sampling-tail-service-environment] The service environment for events to match a policy. (string) diff --git a/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md b/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md index 91bd4a0a9f..63ef34a446 100644 --- a/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md +++ b/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md @@ -9,18 +9,18 @@ applies_to: # Upstream OpenTelemetry Collectors and language SDKs [apm-open-telemetry-direct] +:::{include} _snippets/apm-server-vs-mis.md +::: + ::::{note} This is one of several approaches you can use to integrate Elastic with OpenTelemetry. **To compare approaches and choose the best approach for your use case, refer to [OpenTelemetry](../../../solutions/observability/apps/use-opentelemetry-with-apm.md).** - :::: - The {{stack}} natively supports the OpenTelemetry protocol (OTLP). This means trace data and metrics collected from your applications and infrastructure can be sent directly to the {{stack}}. * Send data to Elastic from an upstream [OpenTelemetry Collector](../../../solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md#apm-connect-open-telemetry-collector) * Send data to Elastic from an upstream [OpenTelemetry language SDK](../../../solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md#apm-instrument-apps-otel) - ## Send data from an upstream OpenTelemetry Collector [apm-connect-open-telemetry-collector] Connect your OpenTelemetry Collector instances to Elastic {{observability}} or {{obs-serverless}} using the OTLP exporter: @@ -31,7 +31,6 @@ Connect your OpenTelemetry Collector instances to Elastic {{observability}} or { :::{tab-item} Elastic Stack :sync: stack - ```yaml receivers: <1> # ... @@ -139,14 +138,12 @@ service: :::: - You’re now ready to export traces and metrics from your services and applications. ::::{tip} When using the OpenTelemetry Collector, you should always prefer sending data via the [`OTLP` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter). Using other methods, like the [`elasticsearch` exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter), will bypass all of the validation and data processing that Elastic performs. In addition, your data will not be viewable in your Observability project if you use the `elasticsearch` exporter. :::: - ## Send data from an upstream OpenTelemetry SDK [apm-instrument-apps-otel] ::::{note} @@ -154,7 +151,6 @@ This document outlines how to send data directly from an upstream OpenTelemetry :::: - To export traces and metrics to Elastic, instrument your services and applications with the OpenTelemetry API, SDK, or both. For example, if you are a Java developer, you need to instrument your Java app with the [OpenTelemetry agent for Java](https://github.com/open-telemetry/opentelemetry-java-instrumentation). See the [OpenTelemetry Instrumentation guides](https://opentelemetry.io/docs/instrumentation/) to download the OpenTelemetry agent or SDK for your language. Define environment variables to configure the OpenTelemetry agent or SDK and enable communication with Elastic APM. For example, if you are instrumenting a Java app, define the following environment variables: @@ -195,7 +191,6 @@ java -javaagent:/path/to/opentelemetry-javaagent-all.jar \ If you are using a version of the Python OpenTelemetry agent *before* 1.27.0, the content of the header *must* be URL-encoded. You can use the Python standard library’s `urllib.parse.quote` function to encode the content of the header. :::: - `OTEL_METRICS_EXPORTER` : Metrics exporter to use. See [exporter selection](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection) for more information. @@ -236,22 +231,18 @@ java -javaagent:/path/to/opentelemetry-javaagent-all.jar \ :::: - `OTEL_METRICS_EXPORTER` : Metrics exporter to use. See [exporter selection](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection) for more information. `OTEL_LOGS_EXPORTER` : Logs exporter to use. See [exporter selection](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection) for more information. - ::: :::: - You are now ready to collect traces and [metrics](../../../solutions/observability/apps/collect-metrics.md) before [verifying metrics](../../../solutions/observability/apps/collect-metrics.md#apm-open-telemetry-verify-metrics) and [visualizing metrics](../../../solutions/observability/apps/collect-metrics.md#apm-open-telemetry-visualize). - ## Proxy requests to APM Server [apm-open-telemetry-proxy-apm] APM Server supports both the [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc) and [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp) protocol on the same port as Elastic APM agent requests. For ease of setup, we recommend using OTLP/HTTP when proxying or load balancing requests to Elastic. @@ -271,7 +262,6 @@ For more information on how to configure an AWS ALB to support gRPC, see this AW For more information on how APM Server services gRPC requests, see [Muxing gRPC and HTTP/1.1](https://github.com/elastic/apm-server/blob/main/dev_docs/otel.md#muxing-grpc-and-http11). - ## Next steps [apm-open-telemetry-direct-next] * [Collect metrics](../../../solutions/observability/apps/collect-metrics.md) diff --git a/solutions/observability/apps/use-apm-securely.md b/solutions/observability/apps/use-apm-securely.md index 9d4b158976..448be39370 100644 --- a/solutions/observability/apps/use-apm-securely.md +++ b/solutions/observability/apps/use-apm-securely.md @@ -9,6 +9,9 @@ applies_to: # Use APM securely [apm-securing-apm-server] +:::{include} _snippets/apm-server-vs-mis.md +::: + When setting up Elastic APM, it’s critical to ensure that application data is secure from start to finish. You should approach securing your application data from different perspectives: | | | diff --git a/solutions/observability/apps/use-opentelemetry-with-apm.md b/solutions/observability/apps/use-opentelemetry-with-apm.md index 9f06dc5179..1a01d2fecc 100644 --- a/solutions/observability/apps/use-opentelemetry-with-apm.md +++ b/solutions/observability/apps/use-opentelemetry-with-apm.md @@ -10,13 +10,13 @@ applies_to: # Use OpenTelemetry with APM [apm-open-telemetry] +:::{include} _snippets/apm-server-vs-mis.md +::: ::::{note} For a complete overview of using OpenTelemetry with Elastic, explore [**Elastic Distributions of OpenTelemetry**](https://github.com/elastic/opentelemetry). - :::: - [OpenTelemetry](https://opentelemetry.io/docs/concepts/what-is-opentelemetry/) is a set of APIs, SDKs, tooling, and integrations that enable the capture and management of telemetry data from your services and applications. Elastic integrates with OpenTelemetry, allowing you to reuse your existing instrumentation to easily send observability data to the {{stack}}. There are several ways to integrate OpenTelemetry with the {{stack}}: @@ -26,14 +26,12 @@ Elastic integrates with OpenTelemetry, allowing you to reuse your existing instr * [Upstream OpenTelemetry Collector and language SDKs](../../../solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-upstream) * [AWS Lambda collector exporter](../../../solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-lambda) - ## Elastic Distributions of OpenTelemetry language SDKs [apm-otel-elastic-distros] ::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: - Elastic offers several distributions of OpenTelemetry language SDKs. A *distribution* is a customized version of an upstream OpenTelemetry repository. Each Elastic Distribution of OpenTelemetry is a customized version of an [OpenTelemetry language SDK](https://opentelemetry.io/docs/languages/). :::{image} /solutions/images/observability-apm-otel-distro.png @@ -59,7 +57,6 @@ For more details about OpenTelemetry distributions in general, visit the [OpenTe :::: - ## Upstream OpenTelemetry API/SDK + Elastic APM agent [apm-otel-api-sdk-elastic-agent] Use the OpenTelemetry API/SDKs with [Elastic APM agents](../../../solutions/observability/apps/fleet-managed-apm-server.md#_step_3_install_apm_agents) to translate OpenTelemetry API calls to Elastic APM API calls. @@ -80,7 +77,6 @@ Find more details about how to use an OpenTelemetry API or SDK with an Elastic A * [**APM Node.js agent →**](apm-agent-nodejs://reference/opentelemetry-bridge.md) * [**APM Python agent →**](apm-agent-python://reference/opentelemetry-api-bridge.md) - ## Upstream OpenTelemetry Collector and language SDKs [apm-otel-upstream] The {{stack}} natively supports the OpenTelemetry protocol (OTLP). This means trace data and metrics collected from your applications and infrastructure by an OpenTelemetry Collector or OpenTelemetry language SDK can be sent to the {{stack}}. @@ -97,7 +93,6 @@ It’s also possible to send data directly to APM Server from an upstream OpenTe :::: - This approach works well when you need to instrument a technology that Elastic doesn’t provide a solution for. For example, if you want to instrument C or C++ you could use the [OpenTelemetry C++ client](https://github.com/open-telemetry/opentelemetry-cpp). However, there are some limitations when using collectors and language SDKs built and maintained by OpenTelemetry, including: @@ -110,7 +105,6 @@ For more on the limitations associated with using upstream OpenTelemetry tools, [**Get started with upstream OpenTelemetry Collectors and language SDKs →**](../../../solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md) - ## AWS Lambda collector exporter [apm-otel-lambda] AWS Lambda functions can be instrumented with OpenTelemetry and monitored with Elastic {{observability}} or {{obs-serverless}}. From a440af3a4ecd13d8c2e9e298a816efbb08c3f952 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 4 Apr 2025 15:47:07 -0500 Subject: [PATCH 3/9] more applies_to --- solutions/observability/apps.md | 18 +++++----- .../apps/get-started-with-apm.md | 35 ++++++++++++------- 2 files changed, 32 insertions(+), 21 deletions(-) diff --git a/solutions/observability/apps.md b/solutions/observability/apps.md index e1247f4aae..bc04e85e10 100644 --- a/solutions/observability/apps.md +++ b/solutions/observability/apps.md @@ -1,9 +1,11 @@ --- +navigation_title: "Applications and services" mapped_pages: - https://www.elastic.co/guide/en/serverless/current/application-and-service-monitoring.html - https://www.elastic.co/guide/en/observability/current/application-and-service-monitoring.html - -navigation_title: "Applications and services" +applies_to: + stack: + serverless: --- # Application and service monitoring [application-and-service-monitoring] @@ -15,9 +17,9 @@ Explore the topics in this section to learn how to observe and monitor software | | | | --- | --- | -| [LLM Observability](../../solutions/observability/apps/llm-observability.md) | Monitor LLM-powered applications to keep them reliable, efficient, cost-effective, and easy to troubleshoot.| -| [Application performance monitoring (APM)](../../solutions/observability/apps/application-performance-monitoring-apm.md) | Monitor software services and applications in real time, by collecting detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, and more. | -| [Synthetic monitoring](../../solutions/observability/apps/synthetic-monitoring.md) | Monitor the availability of network endpoints and services. | -| [Real user monitoring](../../solutions/observability/apps/real-user-monitoring-user-experience.md) | Quantify and analyze the perceived performance of your web application using real-world user experiences. | -| [Uptime monitoring (deprecated)](../../solutions/observability/apps/uptime-monitoring-deprecated.md) | Periodically check the status of your services and applications. | -| [Tutorial: Monitor a Java application](../../solutions/observability/apps/tutorial-monitor-java-application.md) | Monitor a Java application using Elastic Observability: Logs, Infrastructure metrics, APM, and Uptime. | +| [LLM Observability](/solutions/observability/apps/llm-observability.md) | Monitor LLM-powered applications to keep them reliable, efficient, cost-effective, and easy to troubleshoot.| +| [Application performance monitoring (APM)](/solutions/observability/apps/application-performance-monitoring-apm.md) | Monitor software services and applications in real time, by collecting detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, and more. | +| [Synthetic monitoring](/solutions/observability/apps/synthetic-monitoring.md) | Monitor the availability of network endpoints and services. | +| [Real user monitoring](/solutions/observability/apps/real-user-monitoring-user-experience.md) | Quantify and analyze the perceived performance of your web application using real-world user experiences. | +| [Uptime monitoring (deprecated)](/solutions/observability/apps/uptime-monitoring-deprecated.md) | Periodically check the status of your services and applications. | +| [Tutorial: Monitor a Java application](/solutions/observability/apps/tutorial-monitor-java-application.md) | Monitor a Java application using Elastic Observability: Logs, Infrastructure metrics, APM, and Uptime. | diff --git a/solutions/observability/apps/get-started-with-apm.md b/solutions/observability/apps/get-started-with-apm.md index f89d5c8214..f3f751d9ec 100644 --- a/solutions/observability/apps/get-started-with-apm.md +++ b/solutions/observability/apps/get-started-with-apm.md @@ -10,20 +10,30 @@ applies_to: # Get started with APM [apm-getting-started-apm-server] - ::::{note} Starting in version 8.15.0, the {{es}} apm-data plugin manages APM index templates, lifecycle policies, and ingest pipelines. :::: - The APM Server receives performance data from your APM agents, validates and processes it, and then transforms the data into {{es}} documents. If you’re on this page, then you’ve chosen to self-manage the Elastic Stack, and you now must decide how to run and configure the APM Server. There are two options, and the components required are different for each: -* **[Fleet-managed APM Server](../../../solutions/observability/apps/get-started-with-apm.md#apm-setup-fleet-managed-apm)** -* **[APM Server binary](../../../solutions/observability/apps/get-started-with-apm.md#apm-setup-apm-server-binary)** +* **[Elastic Cloud Serverless](/solutions/observability/apps/get-started-with-apm.md#get-started-apm-serverless)** +* **[Fleet-managed APM Server](/solutions/observability/apps/get-started-with-apm.md#apm-setup-fleet-managed-apm)** +* **[APM Server binary](/solutions/observability/apps/get-started-with-apm.md#apm-setup-apm-server-binary)** + +## Elastic Cloud Serverless [get-started-apm-serverless] + +```{applies_to} +serverless: +``` +Elastic Cloud Serverless is a fully managed solution that allows you to deploy and use Elastic for your use cases without managing the underlying infrastructure. ## Fleet-managed APM Server [apm-setup-fleet-managed-apm] +```{applies_to} +stack: +``` + Fleet is a web-based UI in {{kib}} that is used to centrally manage {{agent}}s. In this deployment model, use {{agent}} to spin up APM Server instances that can be centrally-managed in a custom-curated user interface. :::{image} /solutions/images/observability-fm-ov.png @@ -44,7 +54,6 @@ Fleet is a web-based UI in {{kib}} that is used to centrally manage {{agent}}s. Fleet-managed APM Server does *not* support all the outputs that are supported by the APM Server binary method of running Elastic APM. :::: - **Required components**: * APM agents @@ -52,9 +61,12 @@ Fleet-managed APM Server does *not* support all the outputs that are supported b **Configuration method**: {{kib}} UI - ## APM Server binary [apm-setup-apm-server-binary] +```{applies_to} +stack: +``` + Install, configure, and run the APM Server binary wherever you need it. :::{image} /solutions/images/observability-bin-ov.png @@ -85,18 +97,15 @@ Install, configure, and run the APM Server binary wherever you need it. **Configuration method**: YAML - ## Help me decide [_help_me_decide] +```{applies_to} +stack: +``` + This decision tree highlights key factors to help you make an informed decision about implementing Elastic APM. It provides practical guidance and is not intended to serve as a comprehensive reference of all possible implementations and capabilities. :::{image} /solutions/images/observability-apm-help-me-decide.svg :alt: APM decision tree :screenshot: ::: - -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [ ] ./raw-migrated-files/docs-content/serverless/observability-apm-get-started.md \ No newline at end of file From 1cf94bf4feb76a102a3c756ea4371c21204e4808 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 4 Apr 2025 16:33:08 -0500 Subject: [PATCH 4/9] clean up whitespace --- solutions/observability/apps/act-on-data.md | 1 - .../analyze-data-from-synthetic-monitors.md | 16 ----- .../observability/apps/analyze-monitors.md | 4 -- solutions/observability/apps/analyze.md | 3 - .../apps/anonymous-authentication.md | 3 - solutions/observability/apps/api-keys.md | 15 ----- .../apps/apm-agent-authorization.md | 24 -------- .../apps/apm-agent-central-configuration.md | 5 -- .../apps/apm-agent-compatibility.md | 12 ---- .../observability/apps/apm-agent-explorer.md | 2 - .../apps/apm-agent-tls-communication.md | 4 -- solutions/observability/apps/apm-apis.md | 2 - .../observability/apps/apm-k8s-attacher.md | 3 - .../observability/apps/apm-reader-user.md | 8 --- .../apps/apm-server-advanced-setup.md | 9 --- .../observability/apps/apm-server-api.md | 5 -- .../observability/apps/apm-server-binary.md | 32 ---------- .../apps/apm-server-command-reference.md | 18 ------ .../apps/apm-server-information-api.md | 3 - .../observability/apps/apm-server-systemd.md | 8 --- solutions/observability/apps/apm-ui-api.md | 8 --- .../apps/application-data-security.md | 6 -- .../application-performance-monitoring-apm.md | 1 - .../apps/applications-ui-annotation-user.md | 4 -- .../apps/applications-ui-api-user.md | 7 --- .../applications-ui-central-config-user.md | 6 -- .../apps/applications-ui-settings.md | 3 - .../applications-ui-storage-explorer-user.md | 2 - .../apps/built-in-data-filters.md | 6 -- .../apps/collect-application-data.md | 4 -- .../observability/apps/collect-metrics.md | 3 - .../configure-anonymous-authentication.md | 13 ---- ...nfigure-apm-agent-central-configuration.md | 9 --- .../apps/configure-apm-instrumentation.md | 8 --- .../apps/configure-apm-server.md | 11 ---- .../apps/configure-console-output.md | 9 --- .../apps/configure-elasticsearch-output.md | 35 ----------- .../configure-individual-browser-monitors.md | 2 - .../apps/configure-kafka-output.md | 38 ------------ .../apps/configure-kibana-endpoint.md | 12 ---- .../apps/configure-lightweight-monitors.md | 58 ------------------ .../observability/apps/configure-logging.md | 25 -------- .../apps/configure-logstash-output.md | 35 ----------- ...-elasticsearch-service-on-elastic-cloud.md | 5 -- .../observability/apps/configure-output.md | 8 --- .../apps/configure-project-paths.md | 9 --- .../configure-real-user-monitoring-rum.md | 17 ------ .../apps/configure-redis-output.md | 23 -------- .../observability/apps/configure-settings.md | 5 -- .../apps/configure-synthetics-projects.md | 17 ------ .../apps/configure-synthetics-settings.md | 8 --- .../apps/control-access-to-apm-data.md | 26 -------- .../apps/create-apm-rules-alerts.md | 7 --- ...ssign-feature-roles-to-apm-server-users.md | 29 --------- .../observability/apps/create-custom-links.md | 13 ---- .../apps/create-monitors-in-synthetics-app.md | 7 --- .../create-monitors-with-project-monitors.md | 12 ---- .../apps/create-upload-source-maps-rum.md | 7 --- ...ss-cluster-search-with-application-data.md | 2 - .../observability/apps/custom-filters.md | 16 ----- solutions/observability/apps/data-streams.md | 10 ---- .../apps/delete-sensitive-data.md | 4 -- solutions/observability/apps/dependencies.md | 1 - .../apps/drill-down-into-data.md | 1 - .../elastic-apm-agent-configuration-api.md | 6 -- .../observability/apps/elastic-apm-agents.md | 2 - .../apps/elastic-apm-events-intake-api.md | 21 ------- solutions/observability/apps/errors.md | 5 -- .../apps/explore-data-in-elasticsearch.md | 1 - .../explore-mobile-sessions-with-discover.md | 1 - .../apps/filter-application-data.md | 4 -- .../apps/filter-search-application-data.md | 1 - ...ransaction-latency-failure-correlations.md | 3 - .../apps/fleet-managed-apm-server.md | 28 --------- .../apps/general-configuration-options.md | 16 ----- .../apps/get-started-with-uptime.md | 8 --- solutions/observability/apps/get-started.md | 2 - .../apps/grant-access-using-api-keys.md | 10 ---- ...grant-users-access-to-secured-resources.md | 1 - .../observability/apps/high-availability.md | 1 - .../apps/index-lifecycle-management.md | 13 ---- .../observability/apps/infrastructure.md | 2 - .../apps/inspect-uptime-duration-anomalies.md | 1 - .../observability/apps/installation-layout.md | 1 - .../apps/integrate-with-jaeger-deprecated.md | 14 ----- .../apps/integrate-with-machine-learning.md | 3 - .../apps/interpret-application-data.md | 1 - solutions/observability/apps/inventory.md | 8 --- .../observability/apps/jaeger-event-intake.md | 1 - solutions/observability/apps/limitations.md | 4 -- .../apps/manage-data-retention.md | 2 - .../observability/apps/manage-monitors.md | 5 -- .../observability/apps/manage-storage.md | 8 --- .../apps/managed-intake-service-event-api.md | 34 ----------- solutions/observability/apps/metadata.md | 8 --- ...ate-from-elastic-synthetics-integration.md | 9 --- .../apps/mobile-service-overview.md | 11 ---- .../apps/monitor-apm-server-binary.md | 1 - .../observability/apps/monitor-apm-server.md | 1 - .../apps/monitor-fleet-managed-apm-server.md | 11 ---- .../monitor-resources-on-private-networks.md | 9 --- .../apps/monitoring-aws-lambda-functions.md | 1 - ...authentication-mfa-for-browser-monitors.md | 3 - .../apps/observe-lambda-functions.md | 4 -- .../apps/opentelemetry-intake-api.md | 3 - solutions/observability/apps/overviews.md | 2 - .../apps/parse-data-using-ingest-pipelines.md | 3 - solutions/observability/apps/reader-role.md | 2 - .../real-user-monitoring-user-experience.md | 16 ----- .../observability/apps/reduce-storage.md | 7 --- .../scale-architect-synthetics-deployment.md | 2 - solutions/observability/apps/secret-token.md | 4 -- .../secrets-keystore-for-secure-settings.md | 9 --- .../apps/secure-access-to-applications-ui.md | 1 - .../secure-communication-with-apm-agents.md | 4 -- ...secure-communication-with-elastic-stack.md | 7 --- solutions/observability/apps/service-map.md | 6 -- .../observability/apps/service-overview.md | 10 ---- solutions/observability/apps/services.md | 3 - solutions/observability/apps/setup-role.md | 2 - .../apps/ssltls-input-settings.md | 9 --- .../apps/ssltls-output-settings.md | 25 -------- .../observability/apps/storage-explorer.md | 9 --- .../apps/storage-sizing-guide.md | 2 - ...lastic-cloud-cluster-to-apm-integration.md | 8 --- ...ch-self-installation-to-apm-integration.md | 11 ---- .../apps/switch-to-elastic-apm-integration.md | 3 - .../apps/synthetic-monitoring.md | 3 - .../apps/synthetics-encryption-security.md | 3 - .../apps/synthetics-support-matrix.md | 14 ----- .../observability/apps/tail-based-sampling.md | 12 ---- .../apps/trace-sample-timeline.md | 3 - solutions/observability/apps/traces-2.md | 3 - solutions/observability/apps/traces.md | 10 ---- .../observability/apps/transactions-2.md | 10 ---- solutions/observability/apps/transactions.md | 5 -- .../observability/apps/tune-data-ingestion.md | 5 -- .../apps/tutorial-monitor-java-application.md | 59 ------------------- ...astic-cloud-apm-server-standalone-to-90.md | 3 - ...lastic-cloud-with-apm-integration-to-90.md | 4 -- ...f-installation-of-apm-integration-to-90.md | 5 -- ...allation-of-apm-server-standalone-to-90.md | 6 -- .../apps/upgrade-to-version-90.md | 2 - .../apps/uptime-monitoring-deprecated.md | 4 -- ...se-advanced-queries-on-application-data.md | 4 -- ...-environment-variables-in-configuration.md | 6 -- ...rnal-collection-to-send-monitoring-data.md | 23 -------- .../use-metricbeat-to-send-monitoring-data.md | 6 -- ...-emitted-directly-to-monitoring-cluster.md | 1 - .../observability/apps/use-synthetics-cli.md | 28 --------- .../apps/use-synthetics-recorder.md | 13 ---- .../use-synthetics-with-traffic-filters.md | 5 -- .../apps/view-elasticsearch-index-template.md | 7 --- .../observability/apps/view-monitor-status.md | 5 -- .../apps/work-with-params-secrets.md | 11 ---- .../apps/write-synthetic-test.md | 21 ------- solutions/observability/apps/writer-role.md | 7 --- 157 files changed, 1410 deletions(-) diff --git a/solutions/observability/apps/act-on-data.md b/solutions/observability/apps/act-on-data.md index 850ac51ff3..9a47b487a0 100644 --- a/solutions/observability/apps/act-on-data.md +++ b/solutions/observability/apps/act-on-data.md @@ -10,7 +10,6 @@ applies_to: # Act on application data - In addition to exploring visualizations in the Applications UI in {{kib}}, you can make your application data more actionable with: | | | diff --git a/solutions/observability/apps/analyze-data-from-synthetic-monitors.md b/solutions/observability/apps/analyze-data-from-synthetic-monitors.md index 441f4eb684..0fefb88896 100644 --- a/solutions/observability/apps/analyze-data-from-synthetic-monitors.md +++ b/solutions/observability/apps/analyze-data-from-synthetic-monitors.md @@ -25,7 +25,6 @@ When you use a single monitor configuration to create monitors in multiple locat :::: - :::{image} /solutions/images/observability-synthetics-monitor-page.png :alt: Synthetics UI :screenshot: @@ -35,7 +34,6 @@ To get started with your analysis in the Overview tab, you can search for monito Then click an individual monitor to see some details in a flyout. From there, you can click **Go to monitor** to go to an individual monitor’s page to see more details (as described below). - ## All monitor types [synthetics-analyze-individual-monitors] When you go to an individual monitor’s page, you’ll see much more detail about the monitor’s performance over time. The details vary by monitor type, but for every monitor at the top of the page you’ll see: @@ -59,7 +57,6 @@ When you go to an individual monitor’s page, you’ll see much more detail abo Each individual monitor’s page has three tabs: Overview, History, and Errors. - ### Overview [synthetics-analyze-individual-monitors-overview] The **Overview** tab has information about the monitor availability, duration, and any errors that have occurred since the monitor was created. The *Duration trends* chart displays the timing for each check that was performed in the last 30 days. This visualization helps you to gain insights into how quickly requests resolve by the targeted endpoint and gives you a sense of how frequently a host or endpoint was down. @@ -69,7 +66,6 @@ The **Overview** tab has information about the monitor availability, duration, a :screenshot: ::: - ### History [synthetics-analyze-individual-monitors-history] The **History** tab has information on every time the monitor has run. It includes some high-level stats and a complete list of all test runs. Use the calendar icon (![Calendar icon](/solutions/images/observability-calendar.svg "")) and search bar to filter for runs that occurred in a specific time period. @@ -88,7 +84,6 @@ If the monitor is configured to [retest on failure](../../../solutions/observabi :screenshot: ::: - ### Errors [synthetics-analyze-individual-monitors-errors] The **Errors** tab has information on failed runs. If the monitor is configured to [retest on failure](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor), failed runs will only result in an error if both the initial run and the rerun fail. This can reduce noise related to transient problems. @@ -102,7 +97,6 @@ For browser monitors, you can click on any run in the **Error** list to open an :screenshot: ::: - ## Browser monitors [synthetics-analyze-journeys] For browser monitors, you can look at results at various levels of granularity: @@ -111,7 +105,6 @@ For browser monitors, you can look at results at various levels of granularity: * Drill down into the details of a single run. * Drill down further into the details of a single *step* within a journey. - ### Journey runs over time [journey_runs_over_time] The journey page on the Overview tab includes: @@ -131,7 +124,6 @@ From here, you can either drill down into: * The latest run of the full journey by clicking **![Inspect icon](/solutions/images/observability-inspect.svg "") View test run** or a past run in the list of **Last 10 test runs**. This will take you to the view described below in [Details for one run](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run). * An individual step in this run by clicking the performance breakdown icon (![Performance breakdown icon](/solutions/images/observability-apmTrace.svg "")) next to one of the steps. This will take you to the view described below in [Details for one step](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step). - ### Details for one run [synthetics-analyze-one-run] The page detailing one run for a journey includes more information on each step in the current run and opportunities to compare each step to the same step in previous runs. @@ -153,12 +145,10 @@ This is particularly useful to compare the metrics for a failed step to the last Drill down to see even more details for an individual step by clicking the performance breakdown icon (![Performance breakdown icon](/solutions/images/observability-apmTrace.svg "")) next to one of the steps. This will take you to the view described below in [Details for one step](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step). - ### Details for one step [synthetics-analyze-one-step] After clicking the performance breakdown icon (![Performance breakdown icon](/solutions/images/observability-apmTrace.svg "")) you’ll see more detail for an individual step. - #### Screenshot [synthetics-analyze-one-step-screenshot] By default the synthetics library will capture a screenshot for each step regardless of whether the step completed or failed. @@ -168,7 +158,6 @@ Customize screenshot behavior for all monitors in the [configuration file](../.. :::: - Screenshots can be particularly helpful to identify what went wrong when a step fails because of a change to the UI. You can compare the failed step to the last time the step successfully completed. :::{image} /solutions/images/observability-synthetics-analyze-one-step-screenshot.png @@ -176,7 +165,6 @@ Screenshots can be particularly helpful to identify what went wrong when a step :screenshot: ::: - #### Timing [synthetics-analyze-one-step-timing] The **Timing** visualization shows a breakdown of the time spent in each part of the resource loading process for the step including: @@ -198,7 +186,6 @@ This gives you an overview of how much time is spent (and how that time is spent :screenshot: ::: - #### Metrics [synthetics-analyze-one-step-metrics] The **Metrics** visualization gives you insight into the performance of the web page visited in the step and what a user would experience when going through the current step. Metrics include: @@ -214,7 +201,6 @@ Largest contentful paint and Cumulative layout shift are part of Google’s [Cor :::: - Next to each metric, there’s an icon that indicates whether the value is higher (![Value is higher icon](/solutions/images/observability-sortUp.svg "")), lower (![Value is lower icon](/solutions/images/observability-sortDown.svg "")), or the same (![Value is the same](/solutions/images/observability-minus.svg "")) compared to all runs over the last 24 hours. Hover over the icon to see more details in a tooltip. :::{image} /solutions/images/observability-synthetics-analyze-one-step-metrics.png @@ -222,7 +208,6 @@ Next to each metric, there’s an icon that indicates whether the value is highe :screenshot: ::: - #### Object weight and count [synthetics-analyze-one-step-object] The **Object weight** visualization shows the cumulative size of downloaded resources by type, and **Object count** shows the number of individual resources by type. @@ -234,7 +219,6 @@ This provides a different kind of analysis. For example, you might have a large :screenshot: ::: - #### Network requests [synthetics-analyze-one-step-network] The **Network requests** visualization is a waterfall chart that shows every request the page made when a user executed it. Each line in the chart represents an HTTP network request and helps you quickly identify what resources are taking the longest to load and in what order they are loading. diff --git a/solutions/observability/apps/analyze-monitors.md b/solutions/observability/apps/analyze-monitors.md index 386871fca8..ba5bea51c7 100644 --- a/solutions/observability/apps/analyze-monitors.md +++ b/solutions/observability/apps/analyze-monitors.md @@ -18,7 +18,6 @@ To access this page, go to **{{observability}} > Uptime > Monitors**. Click on a The monitor detail screen displays several panels of information. - ## Status panel [uptime-status-panel] The **Status** panel displays a summary of the latest information regarding your monitor. You can view its availability, monitor ID, type, and any assigned tags. You can click a link to visit the targeted URL, view when the TLS certificate expires, and determine the amount of time elapsed since the last check. @@ -32,7 +31,6 @@ The **Monitoring from** list displays service availability per monitoring locati To display a map with each location as a pinpoint, you can toggle the availability view from list view to map view. - ## Monitor duration [uptime-monitor-duration] The **Monitor duration** chart displays the timing for each check that was performed. The visualization helps you to gain insights into how quickly requests resolve by the targeted endpoint and give you a sense of how frequently a host or endpoint was down in your selected time span. @@ -44,7 +42,6 @@ Included on this chart is the {{anomaly-detect}} ({{ml}}) integration. For more :screenshot: ::: - ## Pings over time [uptime-pings-chart] The **Pings over time** chart is a graphical representation of the check statuses over time. Hover over the charts to display crosshairs with specific numeric data. @@ -54,7 +51,6 @@ The **Pings over time** chart is a graphical representation of the check statuse :screenshot: ::: - ## Check history [uptime-history-panel] The **History** table lists the total count of this monitor’s checks for the selected date range. To help find recent problems on a per-check basis, you can filter by `status` and `location`. diff --git a/solutions/observability/apps/analyze.md b/solutions/observability/apps/analyze.md index ce96d8f958..63c70c5962 100644 --- a/solutions/observability/apps/analyze.md +++ b/solutions/observability/apps/analyze.md @@ -23,6 +23,3 @@ Learn how to view and interpret data in the {{uptime-app}}: * [Analyze monitors](analyze-monitors.md) * [Inspect uptime duration anomalies](inspect-uptime-duration-anomalies.md) - - - diff --git a/solutions/observability/apps/anonymous-authentication.md b/solutions/observability/apps/anonymous-authentication.md index ac30a12e13..811c04be37 100644 --- a/solutions/observability/apps/anonymous-authentication.md +++ b/solutions/observability/apps/anonymous-authentication.md @@ -16,7 +16,6 @@ Elastic APM agents can send unauthenticated (anonymous) events to the APM Server In some cases, however, it makes sense to allow both authenticated and anonymous requests. For example, it isn’t possible to authenticate requests from front-end services as the secret token or API key can’t be protected. This is the case with the Real User Monitoring (RUM) agent running in a browser, or the Android or iOS/Swift agent running in a user application. However, you still likely want to authenticate requests from back-end services. To solve this problem, you can enable anonymous authentication in the APM Server to allow the ingestion of unauthenticated client-side APM data while still requiring authentication for server-side services. - ## Configuring anonymous auth for client-side services [apm-anonymous-auth-config] ::::{note} @@ -24,7 +23,6 @@ You can only enable and configure anonymous authentication if an [API key](api-k :::: - :::::::{tab-set} ::::::{tab-item} Fleet-managed @@ -57,7 +55,6 @@ The remote IP address of an incoming request might be different from the end-use If none of these headers are present, the remote address for the incoming request is used. - ### Using a reverse proxy or load balancer [apm-derive-client-ip-concerns] HTTP headers are easily modified; it’s possible for anyone to spoof the derived `client.ip` value by changing or setting, for example, the value of the `X-Forwarded-For` header. For this reason, if any of your clients are not trusted, we recommend setting up a reverse proxy or load balancer in front of the APM Server. diff --git a/solutions/observability/apps/api-keys.md b/solutions/observability/apps/api-keys.md index 4da797006a..e5ccead4be 100644 --- a/solutions/observability/apps/api-keys.md +++ b/solutions/observability/apps/api-keys.md @@ -11,7 +11,6 @@ applies_to: API keys are sent as plain-text, so they only provide security when used in combination with [TLS](apm-agent-tls-communication.md). :::: - When enabled, API keys are used to authorize requests to the APM Server. API keys are not applicable for APM agents running on clients, like the RUM agent, as there is no way to prevent them from being publicly exposed. You can assign one or more unique privileges to each API key: @@ -26,7 +25,6 @@ To secure the communication between APM Agents and the APM Server with API keys, 3. [Create an API key in {{kib}}](#apm-create-an-api-key) 4. [Set the API key in your APM agents](#apm-agent-api-key) - ## Enable API keys [apm-enable-api-key] :::::::{tab-set} @@ -84,10 +82,8 @@ POST /_security/role/apm_agent_key_role 1. This example assigns privileges for the default space. - Assign the newly created `apm_agent_key_role` role to any user that wishes to create {{apm-agent}} API keys. - ## Create an API key in the Applications UI [apm-create-an-api-key] The Applications UI has a built-in workflow that you can use to easily create and view {{apm-agent}} API keys. Only API keys created in the Applications UI will show up here. @@ -103,7 +99,6 @@ Click **Create APM Agent key** and copy the Base64 encoded API key. You will nee :screenshot: ::: - ## Set the API key in your APM agents [apm-agent-api-key] You can now apply your newly created API keys in the configuration of each of your APM agents. See the relevant agent documentation for additional information: @@ -118,7 +113,6 @@ You can now apply your newly created API keys in the configuration of each of yo * **Python agent**: [`api_key`](apm-agent-python://reference/configuration.md#config-api-key) * **Ruby agent**: [`api_key`](apm-agent-ruby://reference/configuration.md#config-api-key) - ## Alternate API key creation methods [apm-configure-api-key-alternative] API keys can also be created and validated outside of {{kib}}: @@ -126,7 +120,6 @@ API keys can also be created and validated outside of {{kib}}: * [APM Server API key workflow](#apm-create-api-key-workflow-apm-server) * [{{es}} API key workflow](#apm-create-api-key-workflow-es) - ### APM Server API key workflow [apm-create-api-key-workflow-apm-server] This API creation method only works with the APM Server binary. @@ -137,10 +130,8 @@ This API creation method only works with the APM Server binary. Users should create API Keys through {{kib}} or the {{es}} REST API :::: - APM Server provides a command line interface for creating, retrieving, invalidating, and verifying API keys. Keys created using this method can only be used for communication with APM Server. - #### `apikey` subcommands [apm-create-api-key-subcommands] **`create`** @@ -163,10 +154,8 @@ APM Server provides a command line interface for creating, retrieving, invalidat * To **ingest agent data**, assign `event:write`. * To **upload source maps**, assign `sourcemap:write`. - :::: - **`info`** : Query API Key(s). `--id` or `--name` required. @@ -176,7 +165,6 @@ APM Server provides a command line interface for creating, retrieving, invalidat **`verify`** : Check if a credentials string has the given privilege(s). `--credentials` required. - #### Privileges [apm-create-api-key-privileges] If privileges are not specified at creation time, the created key will have all privileges. @@ -185,7 +173,6 @@ If privileges are not specified at creation time, the created key will have all * `--ingest` grants the `event:write` privilege * `--sourcemap` grants the `sourcemap:write` privilege - #### Create an API key [apm-create-api-key-workflow] Create an API key with the `create` subcommand. @@ -238,7 +225,6 @@ Error count ........ 0 A full list of `apikey` subcommands and flags is available in the [API key command reference](apm-server-command-reference.md#apm-apikey-command). - ### {{es}} API key workflow [apm-create-api-key-workflow-es] It is also possible to create API keys using the {{es}} [create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key). @@ -268,7 +254,6 @@ POST /_security/api_key 2. The expiration time of the API key 3. Any assigned privileges - The response will look similar to this: ```console-result diff --git a/solutions/observability/apps/apm-agent-authorization.md b/solutions/observability/apps/apm-agent-authorization.md index e38fe12fa1..e0ef819c33 100644 --- a/solutions/observability/apps/apm-agent-authorization.md +++ b/solutions/observability/apps/apm-agent-authorization.md @@ -14,7 +14,6 @@ Most options in this section are supported by all APM Server deployment methods. :::: - Agent authorization APM Server configuration options. :::::::{tab-set} @@ -55,8 +54,6 @@ These settings apply to API key communication between the APM Server and APM Age These settings are different from the API key settings used for {{es}} output and monitoring. :::: - - ### API key for agent authentication [_api_key_for_agent_authentication] Enable API key authorization by setting `enabled` to `true`. By default, `enabled` is set to `false`, and API key support is disabled. (bool) @@ -70,8 +67,6 @@ Enable API key authorization by setting `enabled` to `true`. By default, `enable Not using Elastic APM agents? When enabled, third-party APM agents must include a valid API key in the following format: `Authorization: ApiKey `. The key must be the base64 encoded representation of the API key’s `id:name`. :::: - - ### API key limit [_api_key_limit] Each unique API key triggers one request to {{es}}. This setting restricts the number of unique API keys are allowed per minute. The minimum value for this setting should be the number of API keys configured in your monitored services. The default `limit` is `100`. (int) @@ -81,7 +76,6 @@ Each unique API key triggers one request to {{es}}. This setting restricts the n | APM Server binary | `auth.api_key.limit` | | Fleet-managed | `Number of keys` | - ### Secret token [_secret_token] Authorization token for sending APM data. The same token must also be set in each {{apm-agent}}. This token is not used for RUM endpoints. (text) @@ -91,7 +85,6 @@ Authorization token for sending APM data. The same token must also be set in eac | APM Server binary | `auth.api_key.token` | | Fleet-managed | `Secret token` | - ## `auth.api_key.elasticsearch.*` configuration options [_auth_api_key_elasticsearch_configuration_options] ::::{note} @@ -103,83 +96,66 @@ All of the `auth.api_key.elasticsearch.*` configurations are optional. If none a :::: - - ### `elasticsearch.hosts` [_elasticsearch_hosts] API keys are fetched from {{es}}. This configuration needs to point to a secured {{es}} cluster that is able to serve API key requests. - ### `elasticsearch.protocol` [_elasticsearch_protocol] The name of the protocol {{es}} is reachable on. The options are: `http` or `https`. The default is `http`. If nothing is configured, configuration settings from the `output` section will be reused. - ### `elasticsearch.path` [_elasticsearch_path] An optional HTTP path prefix that is prepended to the HTTP API calls. If nothing is configured, configuration settings from the `output` section will be reused. - ### `elasticsearch.proxy_url` [_elasticsearch_proxy_url] The URL of the proxy to use when connecting to the {{es}} servers. The value may be either a complete URL or a "host[:port]", in which case the "http"scheme is assumed. If nothing is configured, configuration settings from the `output` section will be reused. - ### `elasticsearch.timeout` [_elasticsearch_timeout] The HTTP request timeout in seconds for the {{es}} request. If nothing is configured, configuration settings from the `output` section will be reused. - ## `auth.api_key.elasticsearch.ssl.*` configuration options [_auth_api_key_elasticsearch_ssl_configuration_options] SSL is off by default. Set `elasticsearch.protocol` to `https` if you want to enable `https`. - ### `elasticsearch.ssl.enabled` [_elasticsearch_ssl_enabled] Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication. - ### `elasticsearch.ssl.verification_mode` [_elasticsearch_ssl_verification_mode] Configure SSL verification mode. If `none` is configured, all server hosts and certificates will be accepted. In this mode, SSL based connections are susceptible to man-in-the-middle attacks. **Use only for testing**. Default is `full`. - ### `elasticsearch.ssl.supported_protocols` [_elasticsearch_ssl_supported_protocols] List of supported/valid TLS versions. By default, all TLS versions from 1.0 to 1.2 are enabled. - ### `elasticsearch.ssl.certificate_authorities` [_elasticsearch_ssl_certificate_authorities] List of root certificates for HTTPS server verifications. - ### `elasticsearch.ssl.certificate` [_elasticsearch_ssl_certificate] The path to the certificate for SSL client authentication. - ### `elasticsearch.ssl.key` [_elasticsearch_ssl_key] The client certificate key used for client authentication. This option is required if certificate is specified. - ### `elasticsearch.ssl.key_passphrase` [_elasticsearch_ssl_key_passphrase] An optional passphrase used to decrypt an encrypted key stored in the configured key file. - ### `elasticsearch.ssl.cipher_suites` [_elasticsearch_ssl_cipher_suites] The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s default suites are used (recommended). - ### `elasticsearch.ssl.curve_types` [_elasticsearch_ssl_curve_types] The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange). - ### `elasticsearch.ssl.renegotiation` [_elasticsearch_ssl_renegotiation] Configure what types of renegotiation are supported. Valid options are `never`, `once`, and `freely`. Default is `never`. diff --git a/solutions/observability/apps/apm-agent-central-configuration.md b/solutions/observability/apps/apm-agent-central-configuration.md index 160882853c..3584143c96 100644 --- a/solutions/observability/apps/apm-agent-central-configuration.md +++ b/solutions/observability/apps/apm-agent-central-configuration.md @@ -6,8 +6,6 @@ applies_to: stack: --- - - # APM agent central configuration [apm-agent-configuration] APM Agent configuration allows you to fine-tune your APM agent configuration from within the Applications UI. Changes are automatically propagated to your APM agents, so there’s no need to redeploy. @@ -19,12 +17,10 @@ To get started, choose the services and environments you wish to configure. The :screenshot: ::: - ## Precedence [_precedence] Configurations set from the Applications UI take precedence over configurations set locally in each APM agent. However, if APM Server is slow to respond, is offline, reports an error, etc., APM agents will use local defaults until they’re able to update the configuration. For this reason, it is still essential to set custom default configurations locally in each of your APM agents. - ## Supported configurations [_supported_configurations] Each APM agent has a list of supported configurations. After selecting a Service name and environment in the Applications UI, a list of all supported configuration options, including descriptions and default values, will be displayed. @@ -61,7 +57,6 @@ Ruby agent Real User Monitoring (RUM) agent : [Configuration reference](apm-agent-rum-js://reference/configuration.md) - ## APM Server configuration [_apm_server_configuration] For most users, APM agent configuration should work out-of-the-box. If you run into trouble, it may be because you’re not using the {{es}} output, or because your {{es}} credentials don’t have sufficient privileges. diff --git a/solutions/observability/apps/apm-agent-compatibility.md b/solutions/observability/apps/apm-agent-compatibility.md index 486c0b600d..d490c0174e 100644 --- a/solutions/observability/apps/apm-agent-compatibility.md +++ b/solutions/observability/apps/apm-agent-compatibility.md @@ -9,21 +9,18 @@ applies_to: The chart below outlines the compatibility between different versions of Elastic APM agents and extensions with the APM integration. - ## APM AWS Lambda extension [_apm_aws_lambda_extension] | {{apm-agent}} version | APM integration version | | --- | --- | | `1.x` | ≥ `8.2` | - ## Android agent [_android_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `0.x` | ≥ `8.12` | - ## Go agent [_go_agent] | {{apm-agent}} version | APM integration version | @@ -31,14 +28,12 @@ The chart below outlines the compatibility between different versions of Elastic | `1.x` | ≥ `6.5` | | `2.x` | ≥ `6.5` | - ## iOS agent [_ios_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `1.x` | ≥ `8.12` | - ## Java agent [_java_agent] | {{apm-agent}} version | APM integration version | @@ -50,36 +45,30 @@ Java agent < 1.43.0 not fully compatible with APM Server >= 8.11.0. For more inf :::: - - ## .NET agent [_net_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `1.x` | ≥ `6.5` | - ## Node.js agent [_node_js_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `3.x` | ≥ `6.6` | - ## PHP agent [_php_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `1.x` | ≥ `7.0` | - ## Python agent [_python_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `6.x` | ≥ `6.6` | - ## Ruby agent [_ruby_agent] | {{apm-agent}} version | APM integration version | @@ -87,7 +76,6 @@ Java agent < 1.43.0 not fully compatible with APM Server >= 8.11.0. For more inf | `3.x` | ≥ `6.5` | | `4.x` | ≥ `6.5` | - ## JavaScript RUM agent [_javascript_rum_agent] | {{apm-agent}} version | APM integration version | diff --git a/solutions/observability/apps/apm-agent-explorer.md b/solutions/observability/apps/apm-agent-explorer.md index 066c832f3a..e1381a9175 100644 --- a/solutions/observability/apps/apm-agent-explorer.md +++ b/solutions/observability/apps/apm-agent-explorer.md @@ -7,12 +7,10 @@ applies_to: # APM Agent explorer [apm-agent-explorer] - ::::{important} This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: - APM agent explorer provides a centralized panel to identify APM agent deployment details, like service name, environment, instances, and agent name, version, and documentation. :::{image} /solutions/images/observability-apm-agent-explorer.png diff --git a/solutions/observability/apps/apm-agent-tls-communication.md b/solutions/observability/apps/apm-agent-tls-communication.md index b139517c4d..9919023d20 100644 --- a/solutions/observability/apps/apm-agent-tls-communication.md +++ b/solutions/observability/apps/apm-agent-tls-communication.md @@ -11,10 +11,8 @@ TLS is disabled by default. When TLS is enabled for APM Server inbound communica When TLS is enabled, a certificate and corresponding private key are required. The certificate and private key can either be issued by a trusted certificate authority (CA) or be [self-signed](#apm-agent-self-sign). - ## Use a self-signed certificate [apm-agent-self-sign] - ### Step 1: Create a self-signed certificate [apm-agent-self-sign-1] The {{es}} distribution offers the `certutil` tool for the creation of self-signed certificates: @@ -24,7 +22,6 @@ The {{es}} distribution offers the `certutil` tool for the creation of self-sign 3. Create the self-signed certificate: `./bin/elasticsearch-certutil cert --ca-cert /ca.crt --ca-key /ca.key --pem --name localhost` 4. Extract the certificate and key from the resulted zip archive. - ### Step 2: Configure the APM Server [apm-agent-self-sign-2] Enable TLS and configure the APM Server to point to the extracted certificate and key: @@ -74,7 +71,6 @@ We do not recommend disabling {{apm-agent}} verification of the server’s certi * **Ruby agent**: [`verify_server_cert`](apm-agent-ruby://reference/configuration.md#config-verify-server-cert) * **Node.js agent**: [`verifyServerCert`](apm-agent-nodejs://reference/configuration.md#validate-server-cert) - ## Client certificate authentication [apm-agent-client-cert] APM Server does not require agents to provide a certificate for authentication, and there is no dedicated support for SSL/TLS client certificate authentication in Elastic’s backend agents. diff --git a/solutions/observability/apps/apm-apis.md b/solutions/observability/apps/apm-apis.md index 98c1416944..eed38510db 100644 --- a/solutions/observability/apps/apm-apis.md +++ b/solutions/observability/apps/apm-apis.md @@ -14,5 +14,3 @@ There are two kinds of APIs related to Elastic APM: | [APM UI API](apm-ui-api.md) | {{kib}} APIs specific to working with the Applications UI including updating configuration options, uploading real user monitoring (RUM) source maps, adding annotations, and more. | | [APM Server API](apm-server-api.md) | APIs for working with APM Server. These are mainly intake APIs that accept data from APM agents and are used primarily by APM agent developers. | - - diff --git a/solutions/observability/apps/apm-k8s-attacher.md b/solutions/observability/apps/apm-k8s-attacher.md index d3d8a3fa7e..1748ee5813 100644 --- a/solutions/observability/apps/apm-k8s-attacher.md +++ b/solutions/observability/apps/apm-k8s-attacher.md @@ -6,11 +6,8 @@ applies_to: stack: --- - - # APM K8S Attacher [apm-k8s-attacher] - The [APM attacher for Kubernetes](apm-k8s-attacher://reference/index.md) simplifies the instrumentation and configuration of your application pods. The attacher includes a webhook receiver that modifies pods so they are automatically instrumented by an Elastic APM agent. Ready to get started? See [Instrument and configure pods](apm-k8s-attacher://reference/apm-get-started-webhook.md) to get started. diff --git a/solutions/observability/apps/apm-reader-user.md b/solutions/observability/apps/apm-reader-user.md index 1fb3fd6c15..d8c93ccee0 100644 --- a/solutions/observability/apps/apm-reader-user.md +++ b/solutions/observability/apps/apm-reader-user.md @@ -6,14 +6,10 @@ applies_to: stack: --- - - # APM reader user [apm-app-reader] - APM reader users typically need to view the Applications UI and dashboards and visualizations that use APM data. These users might also need to create and edit dashboards, visualizations, and machine learning jobs. - ## APM reader [apm-app-reader-full] To create an APM reader user: @@ -56,8 +52,6 @@ To create an APM reader user: | `kibana_admin` | Grants access to all features in Kibana. | | `machine_learning_admin` | Grants the privileges required to create, update, and view machine learning jobs | - - ## Partial APM reader [apm-app-reader-partial] In some instances, you may wish to restrict certain Kibana apps that a user has access to. @@ -93,7 +87,6 @@ In some instances, you may wish to restrict certain Kibana apps that a user has :::: - 2. Assign feature privileges to any Kibana feature that the user needs access to. Here are two examples: | Type | Privilege | Purpose | @@ -107,4 +100,3 @@ In some instances, you may wish to restrict certain Kibana apps that a user has | --- | --- | | `machine_learning_admin` | Grants the privileges required to create, update, and view machine learning jobs | - diff --git a/solutions/observability/apps/apm-server-advanced-setup.md b/solutions/observability/apps/apm-server-advanced-setup.md index c6b77196e6..1c59ff90a4 100644 --- a/solutions/observability/apps/apm-server-advanced-setup.md +++ b/solutions/observability/apps/apm-server-advanced-setup.md @@ -6,11 +6,8 @@ applies_to: stack: --- - - # APM Server advanced setup [apm-setting-up-and-running] - Before reading this section, see the [getting started documentation](fleet-managed-apm-server.md) for basic installation and running instructions. This section includes additional information on how to set up and run APM Server, including: @@ -22,9 +19,3 @@ This section includes additional information on how to set up and run APM Server * [High Availability](high-availability.md) * [Run APM Server on Docker](apm-server-binary.md#apm-running-on-docker) - - - - - - diff --git a/solutions/observability/apps/apm-server-api.md b/solutions/observability/apps/apm-server-api.md index d569838929..174b0536c9 100644 --- a/solutions/observability/apps/apm-server-api.md +++ b/solutions/observability/apps/apm-server-api.md @@ -15,8 +15,3 @@ The APM Server exposes endpoints for: * [OpenTelemetry intake API](opentelemetry-intake-api.md) * [Jaeger event intake](jaeger-event-intake.md) - - - - - diff --git a/solutions/observability/apps/apm-server-binary.md b/solutions/observability/apps/apm-server-binary.md index 0bf2e38e29..6fe5fc6bed 100644 --- a/solutions/observability/apps/apm-server-binary.md +++ b/solutions/observability/apps/apm-server-binary.md @@ -9,7 +9,6 @@ applies_to: This guide will explain how to set up and configure the APM Server binary. - ## Prerequisites [_prerequisites_7] First, see the [Elastic Support Matrix](https://www.elastic.co/support/matrix) for information about supported operating systems and product compatibility. @@ -25,14 +24,12 @@ We recommend you use the same version of {{es}}, {{kib}}, and APM Server. See [I :alt: Install Elastic APM yourself ::: - ## Step 1: Install [apm-installing] ::::{note} **Before you begin**: If you haven’t installed the {{stack}}, do that now. See [Learn how to install the {{stack}} on your own hardware](../../../get-started/the-stack.md). :::: - To download and install APM Server, use the commands below that work with your system. If you use `apt` or `yum`, you can [install APM Server from our repositories](#apm-setup-repositories) to update to the newest version more easily. $$$apm-deb$$$ @@ -87,7 +84,6 @@ $$$apm-docker$$$ See [Running on Docker](#apm-running-on-docker) for deploying Docker containers. - ## Step 2: Set up and configure [apm-server-configuration] Configure APM by editing the `apm-server.yml` configuration file. The location of this file varies by platform—​see the [Installation layout](installation-layout.md) for help locating it. @@ -107,10 +103,8 @@ output.elasticsearch: 2. The {{es}} `host:port` to connect to. 3. This example uses basic authentication. The user provided here needs the privileges required to publish events to {{es}}. To create a dedicated user for this role, see [Create a *writer* role](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-to-publish-events). - All available configuration options are outlined in [configuring APM Server](configure-apm-server.md). - ## Step 3: Start [apm-server-starting] In a production environment, you would put APM Server on its own machines, similar to how you run {{es}}. You *can* run it on the same machines as {{es}}, but this is not recommended, as the processes will be competing for resources. @@ -125,14 +119,12 @@ To start APM Server, run: The `-e` [global flag](apm-server-command-reference.md#apm-global-flags) enables logging to stderr and disables syslog/file output. Remove this flag if you’ve enabled logging in the configuration file. For Linux systems, see [APM Server status and logs](apm-server-systemd.md). :::: - You should see APM Server start up. It will try to connect to {{es}} on localhost port `9200` and expose an API to agents on port `8200`. You can change the defaults in `apm-server.yml` or by supplying a different address on the command line: ```bash ./apm-server -e -E output.elasticsearch.hosts=ElasticsearchAddress:9200 -E apm-server.host=localhost:8200 ``` - ### Debian Package / RPM [apm-running-deb-rpm] For Debian package and RPM installations, we recommend the `apm-server` process runs as a non-root user. Therefore, these installation methods create an `apm-server` user which you can use to start the process. In addition, APM Server will only start if the configuration file is [owned by the user running the process](apm-server-systemd.md#apm-config-file-ownership). @@ -145,7 +137,6 @@ sudo -u apm-server apm-server [] By default, APM Server loads its configuration file from `/etc/apm-server/apm-server.yml`. See the [deb & rpm default paths](installation-layout.md) for a full directory layout. - ## Step 4: Install APM agents [apm-next-steps] :::::::{tab-set} @@ -165,7 +156,6 @@ plugins { 1. The Elastic plugin declaration must be added below the Android app plugin declaration (`com.android.application`) and below the Kotlin plugin declaration (if used). - **2. Configure the agent** After adding the agent plugin, configure it. A minimal configuration sets the Elastic APM integration endpoint as shown below: @@ -195,12 +185,10 @@ elasticApm { 4. Defaults to null. More info on API Keys [here](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key). 5. Defaults to null. - ::::{note} When both `secretToken` and `apiKey` are provided, apiKey has priority and secretToken is ignored. :::: - **3. Initialize the agent** After syncing your project with the Gradle changes above, the Elastic APM agent needs to be initialized within your [Application class](https://developer.android.com/reference/android/app/Application). This example shows the simplest way to configure the agent: @@ -220,7 +208,6 @@ class MyApp extends android.app.Application { 1. Initialize the Elastic APM agent once. - All that’s left is to compile and run your application. That’s it! **Learn more in the agent reference** @@ -328,7 +315,6 @@ struct MyApp: App { 1. The APM integration host and port 2. Secret token for APM integration connection - If you’re not using `SwiftUI`, you can alternatively add the same thing to your `AppDelegate.swift` file: ```swift @@ -350,7 +336,6 @@ var config = AgentConfigBuilder() 1. The APM integration host and port 2. Secret token for APM integration connection - **Learn more in the agent reference** Read more in the [APM iOS Agent Reference](apm-agent-ios://reference/index.md). @@ -370,7 +355,6 @@ The first step in getting started with the Elastic APM Java agent is to retrieve curl -o 'elastic-apm-agent.jar' -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST' ``` - **2. Add `-javaagent` flag** When starting your application, add the JVM flag `-javaagent:/path/to/elastic-apm-agent-.jar` @@ -479,7 +463,6 @@ If you can’t find your distribution, you can install the agent by building it The agent is currently only available for Linux operating system. :::: - 1. Download the agent source from [https://github.com/elastic/apm-agent-php/](https://github.com/elastic/apm-agent-php/). 2. Execute the following commands to build the agent and install it: @@ -560,7 +543,6 @@ MIDDLEWARE = ( ) ``` - Flask : **1. Install the {{apm-agent}}** @@ -597,7 +579,6 @@ app.config['ELASTIC_APM'] = { apm = ElasticAPM(app) ``` - **Learn more in the agent reference** * [Supported technologies](apm-agent-python://reference/supported-technologies.md) @@ -632,7 +613,6 @@ secret_token: '' server_url: 'http://localhost:8200' ``` - Rack : For Rack or a compatible framework, like Sinatra, include the middleware in your app and start the agent. @@ -674,7 +654,6 @@ secret_token: '' server_url: 'http://localhost:8200' ``` - **Learn more in the agent reference** * [Supported technologies](apm-agent-ruby://reference/supported-technologies.md) @@ -770,7 +749,6 @@ Once you have at least one {{apm-agent}} sending data to APM Server, you can sta :screenshot: ::: - ## Repositories for APT and YUM [apm-setup-repositories] We have repositories available for APT and YUM-based distributions. Note that we provide binary packages, but no source packages. @@ -782,17 +760,14 @@ We use the PGP key [D88E42B4](https://pgp.mit.edu/pks/lookup?op=vindex&search=0x ``` to sign all our packages. It is available from [https://pgp.mit.edu](https://pgp.mit.edu). - ### APT [_apt] Version 9.0.0-beta1 of apm-server has not yet been released. - ### YUM [_yum] Version 9.0.0-beta1 of apm-server has not yet been released. - ## Run APM Server on Docker [apm-running-on-docker] Docker images for APM Server are available from the Elastic Docker registry. The base image is [ubuntu:22.04](https://hub.docker.com/_/ubuntu). @@ -801,19 +776,16 @@ A list of all published Docker images and tags is available at [www.docker.elast These images are free to use under the Elastic license. They contain open source and free commercial features and access to paid commercial features. [Start a 30-day trial](../../../deploy-manage/license/manage-your-license-in-self-managed-cluster.md) to try out all of the paid commercial features. See the [Subscriptions](https://www.elastic.co/subscriptions) page for information about Elastic license levels. - ### Pull the image [_pull_the_image] Obtaining APM Server for Docker is as simple as issuing a `docker pull` command against the Elastic Docker registry and then, optionally, verifying the image. However, version 9.0.0-beta1 of APM Server has not yet been released, so no Docker image is currently available for this version. - ### Configure APM Server on Docker [_configure_apm_server_on_docker] The Docker image provides several methods for configuring APM Server. The conventional approach is to provide a configuration file via a volume mount, but it’s also possible to create a custom image with your configuration included. - #### Example configuration file [_example_configuration_file] Download this example configuration file as a starting point: @@ -822,7 +794,6 @@ Download this example configuration file as a starting point: curl -L -O https://raw.githubusercontent.com/elastic/apm-server/master/apm-server.docker.yml ``` - #### Volume-mounted configuration [_volume_mounted_configuration] One way to configure APM Server on Docker is to provide `apm-server.docker.yml` via a volume mount. With `docker run`, the volume mount can be specified like this. @@ -841,13 +812,10 @@ docker run -d \ 1. Substitute your {{es}} hosts and ports. 2. If you are using {{ech}}, replace the `-E output.elasticsearch.hosts` line with the Cloud ID and elastic password using the syntax shown earlier. - - #### Customize your configuration [_customize_your_configuration] The `apm-server.docker.yml` downloaded earlier should be customized for your environment. See [Configure APM Server](configure-apm-server.md) for more details. Edit the configuration file and customize it to match your environment then re-deploy your APM Server container. - #### Custom image configuration [_custom_image_configuration] It’s possible to embed your APM Server configuration in a custom image. Here is an example Dockerfile to achieve this: diff --git a/solutions/observability/apps/apm-server-command-reference.md b/solutions/observability/apps/apm-server-command-reference.md index 5cd51ba28d..4a738472bb 100644 --- a/solutions/observability/apps/apm-server-command-reference.md +++ b/solutions/observability/apps/apm-server-command-reference.md @@ -6,16 +6,12 @@ applies_to: stack: --- - - # APM Server command reference [apm-command-line-options] - ::::{important} These commands only apply to the APM Server binary installation method. :::: - APM Server provides a command-line interface for starting APM Server and performing common tasks, like testing configuration files. The command-line also supports [global flags](#apm-global-flags) for controlling global behaviors. @@ -28,7 +24,6 @@ Use `sudo` to run the following commands if: :::: - Some of the features described here require an Elastic license. For more information, see [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions) and [License Management](../../../deploy-manage/license/manage-your-license-in-self-managed-cluster.md). | Commands | | @@ -43,21 +38,18 @@ Some of the features described here require an Elastic license. For more informa Also see [Global flags](#apm-global-flags). - ## `apikey` command [apm-apikey-command] ::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: - ::::{admonition} Deprecated in 8.6.0. :class: warning Users should create API Keys through {{kib}} or the {{es}} REST API. See [API keys](api-keys.md). :::: - Communication between APM agents and APM Server now supports sending an [API Key in the Authorization header](api-keys.md). APM Server provides an `apikey` command that can create, verify, invalidate, and show information about API Keys for agent/server communication. Most operations require the `manage_own_api_key` cluster privilege, and you must ensure that `apm-server.api_key` or `output.elasticsearch` are configured appropriately. **SYNOPSIS** @@ -88,10 +80,8 @@ apm-server apikey SUBCOMMAND [FLAGS] * To **ingest agent data**, assign `event:write`. * To **upload source maps**, assign `sourcemap:write`. - :::: - **`info`** : Query API Key(s). `--id` or `--name` required. @@ -142,7 +132,6 @@ apm-server apikey invalidate --name example-001 For more information, see [API keys](api-keys.md). - ## `export` command [apm-export-command] Exports the configuration, index template, or {{ilm-init}} policy to stdout. You can use this command to quickly view your configuration or see the contents of the index template or the {{ilm-init}} policy. @@ -189,7 +178,6 @@ apm-server export config apm-server export template --es.version 9.0.0-beta1 --index myindexname ``` - ## `help` command [apm-help-command] Shows help for any command. If no command is specified, shows help for the `run` command. @@ -216,7 +204,6 @@ Also see [Global flags](#apm-global-flags). apm-server help export ``` - ## `keystore` command [apm-keystore-command] Manages the [secrets keystore](secrets-keystore-for-secure-settings.md). @@ -265,7 +252,6 @@ apm-server keystore list See [Secrets keystore](secrets-keystore-for-secure-settings.md) for more examples. - ## `run` command [apm-run-command] Runs APM Server. This command is used by default if you start APM Server without specifying a command. @@ -316,7 +302,6 @@ Or: apm-server -e ``` - ## `test` command [apm-test-command] Tests the configuration. @@ -348,7 +333,6 @@ Also see [Global flags](#apm-global-flags). apm-server test config ``` - ## `version` command [apm-version-command] Shows information about the current version. @@ -372,7 +356,6 @@ Also see [Global flags](#apm-global-flags). apm-server version ``` - ## Global flags [apm-global-flags] These global flags are available whenever you run APM Server. @@ -386,7 +369,6 @@ These global flags are available whenever you run APM Server. This setting is applied to the currently running APM Server process. The APM Server configuration file is not changed. - **`-c, --c FILE`** : Specifies the configuration file to use for APM Server. The file you specify here is relative to `path.config`. If the `-c` flag is not specified, the default config file, `apm-server.yml`, is used. diff --git a/solutions/observability/apps/apm-server-information-api.md b/solutions/observability/apps/apm-server-information-api.md index 63d8d50843..c9485fcb86 100644 --- a/solutions/observability/apps/apm-server-information-api.md +++ b/solutions/observability/apps/apm-server-information-api.md @@ -9,7 +9,6 @@ applies_to: The APM Server exposes an API endpoint to query general server information. This lightweight endpoint is useful as a server up/down health check. - ## Server Information endpoint [apm-api-info-endpoint] This is the server information endpoint: @@ -24,7 +23,6 @@ To configure authenticated access to the APM server, the instructions at [APM AP If an [API keys](api-keys.md) or a [Secret token](secret-token.md) is passed along with the `HTTP GET` request, in addition to an HTTP 200, the response payload will include some information about the APM server. - ### Example: GET, without credentials [apm-api-info-example-get-without-credentials] Example APM Server status request with GET, without credentials: @@ -49,7 +47,6 @@ curl --verbose -X GET http://127.0.0.1:8200 * Connection #0 to host 127.0.0.1 left intact ``` - ### Example: GET, with secret token [apm-api-info-example-get-with-secret-token] Example APM Server information request with GET, with a [Secret token](secret-token.md): diff --git a/solutions/observability/apps/apm-server-systemd.md b/solutions/observability/apps/apm-server-systemd.md index ac8eb8db88..2e1b0bbf74 100644 --- a/solutions/observability/apps/apm-server-systemd.md +++ b/solutions/observability/apps/apm-server-systemd.md @@ -11,12 +11,10 @@ applies_to: These commands only apply to the APM Server binary installation method. Fleet-managed users should see [Start and stop {{agent}}s on edge hosts](/reference/fleet/start-stop-elastic-agent.md). :::: - The DEB and RPM packages include a service unit for Linux systems with systemd. On these systems, you can manage APM Server by using the usual systemd commands. We recommend that the apm-server process is run as a non-root user. Therefore, that is the default setup for APM Server’s DEB package and RPM installation. - ## Start and stop APM Server [_start_and_stop_apm_server] Use `systemctl` to start or stop APM Server: @@ -39,7 +37,6 @@ sudo systemctl enable apm-server sudo systemctl disable apm-server ``` - ## APM Server status and logs [_apm_server_status_and_logs] To get the service status, use `systemctl`: @@ -54,7 +51,6 @@ Logs are stored by default in journald. To view the Logs, use `journalctl`: journalctl -u apm-server.service ``` - ## Customize systemd unit for APM Server [_customize_systemd_unit_for_apm_server] The systemd service unit file includes environment variables that you can override to change the default options. @@ -69,7 +65,6 @@ The systemd service unit file includes environment variables that you can overri You can use `BEAT_LOG_OPTS` to set debug selectors for logging. However, to configure logging behavior, set the logging options described in [Configure logging](configure-logging.md). :::: - To override these variables, create a drop-in unit file in the `/etc/systemd/system/apm-server.service.d` directory. For example a file with the following content placed in `/etc/systemd/system/apm-server.service.d/debug.conf` would override `BEAT_LOG_OPTS` to enable debug for {{es}} output. @@ -90,8 +85,6 @@ systemctl restart apm-server It is recommended that you use a configuration management tool to include drop-in unit files. If you need to add a drop-in manually, use `systemctl edit apm-server.service`. :::: - - #### Configuration file ownership [apm-config-file-ownership] On systems with POSIX file permissions, the APM Server configuration file is subject to ownership and file permission checks. These checks prevent unauthorized users from providing or modifying configurations that are run by APM Server. @@ -119,7 +112,6 @@ can only be writable by the owner but the permissions are "-rw-rw-r--" To correct this problem, use `chmod go-w /etc/apm-server/apm-server.yml` to remove write privileges from anyone other than the owner. - ##### Disabling strict permission checks [_disabling_strict_permission_checks] You can disable strict permission checks from the command line by using `--strict.perms=false`, but we strongly encourage you to leave the checks enabled. diff --git a/solutions/observability/apps/apm-ui-api.md b/solutions/observability/apps/apm-ui-api.md index 9d768aaaff..f7aae26d7e 100644 --- a/solutions/observability/apps/apm-ui-api.md +++ b/solutions/observability/apps/apm-ui-api.md @@ -14,7 +14,6 @@ Some Applications UI features are provided via a REST API: * RUM source map API ([{{stack}}](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-sourcemaps) or [{{serverless-short}}](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-apm-sourcemaps)) * APM agent key API ([{{stack}}](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-agent-keys) or [{{serverless-short}}](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-apm-agent-keys)) - ## Using the APIs [apm-api-example] Interact with APM APIs using cURL or another API tool. All APM APIs are Kibana APIs, not Elasticsearch APIs; because of this, the Kibana dev tools console cannot be used to interact with APM APIs. @@ -28,7 +27,6 @@ For all APM APIs, you must use a request header. Supported headers are `Authoriz Whether using `Authorization: ApiKey {{credentials}}`, or `-u $USER:$PASSWORD`, users interacting with APM APIs must have [sufficient privileges](applications-ui-api-user.md). - `kbn-xsrf: true` : By default, you must use `kbn-xsrf` for all API calls, except in the following scenarios: @@ -36,7 +34,6 @@ For all APM APIs, you must use a request header. Supported headers are `Authoriz * The path is allowed using the `server.xsrf.allowlist` setting * XSRF protections are disabled using the `server.xsrf.disableProtection` setting - `Content-Type: application/json` : Applicable only when you send a payload in the API request. {{kib}} API requests and responses use JSON. Typically, if you include the `kbn-xsrf` header, you must also include the `Content-Type` header. @@ -60,12 +57,7 @@ curl -X POST \ }' ``` - ## Kibana API [kibana-api] In addition to the APM specific API endpoints, Kibana provides its own [REST API](/solutions/observability/apps/apm-server-api.md) which you can use to automate certain aspects of configuring and deploying Kibana. - - - - diff --git a/solutions/observability/apps/application-data-security.md b/solutions/observability/apps/application-data-security.md index 4acc8522a7..6fa390d819 100644 --- a/solutions/observability/apps/application-data-security.md +++ b/solutions/observability/apps/application-data-security.md @@ -6,11 +6,8 @@ applies_to: stack: --- - - # Application data security [apm-data-security] - When setting up Elastic APM, it’s essential to review all captured data carefully to ensure it doesn’t contain sensitive information like passwords, credit card numbers, or health data. In addition, you may wish to filter out other identifiable information, like IP addresses, user agent information, or form field data. Depending on the type of data, we offer several different ways to filter, manipulate, or obfuscate sensitive information during or before ingestion: @@ -20,7 +17,6 @@ Depending on the type of data, we offer several different ways to filter, manipu In addition to utilizing filters, you should regularly review the [sensitive fields](#apm-sensitive-fields) table to ensure sensitive data is not being ingested. If it is, it’s possible to remove or redact it. See [Delete sensitive data](delete-sensitive-data.md) for more information. - ## Built-in data filters [apm-built-in-data-filters] Built-in data filters allow you to filter or turn off ingestion of the following types of data: @@ -33,7 +29,6 @@ Built-in data filters allow you to filter or turn off ingestion of the following | [Real user monitoring data](built-in-data-filters.md#apm-filters-real-user-data) | URLs visited, click events, user browser errors, resources used, etc. | | [Database statements](built-in-data-filters.md#apm-filters-database-statements) | Sensitive user or business information | - ## Custom filters [apm-custom-data-filters] Custom filters allow you to filter or redact other types of APM data on ingestion: @@ -43,7 +38,6 @@ Custom filters allow you to filter or redact other types of APM data on ingestio | [Ingest pipelines](custom-filters.md#apm-filters-ingest-pipeline) | Applied at ingestion time.All agents and fields are supported. Data leaves the instrumented service.There are no performance overhead implications on the instrumented service. | | [{{apm-agent}} filters](custom-filters.md#apm-filters-in-agent) | Not supported by all agents.Data is sanitized before leaving the instrumented service.Potential overhead implications on the instrumented service | - ## Sensitive fields [apm-sensitive-fields] You should review the following fields regularly to ensure sensitive data is not being captured: diff --git a/solutions/observability/apps/application-performance-monitoring-apm.md b/solutions/observability/apps/application-performance-monitoring-apm.md index 9ee3d3c727..2dd54adbc7 100644 --- a/solutions/observability/apps/application-performance-monitoring-apm.md +++ b/solutions/observability/apps/application-performance-monitoring-apm.md @@ -20,7 +20,6 @@ Elastic APM also automatically collects unhandled errors and exceptions. Errors Metrics are another vital source of information when debugging production systems. Elastic APM agents automatically pick up basic host-level metrics and agent-specific metrics, like JVM metrics in the Java Agent, and Go runtime metrics in the Go Agent. - ## Give Elastic APM a try [give_elastic_apm_a_try] Use [Get started with application traces and APM](../../../solutions/observability/apps/fleet-managed-apm-server.md) to quickly spin up an APM deployment. Want to host everything yourself instead? See [Get started](../../../solutions/observability/apps/get-started-with-apm.md). \ No newline at end of file diff --git a/solutions/observability/apps/applications-ui-annotation-user.md b/solutions/observability/apps/applications-ui-annotation-user.md index ffa30c71b2..77bbdb754d 100644 --- a/solutions/observability/apps/applications-ui-annotation-user.md +++ b/solutions/observability/apps/applications-ui-annotation-user.md @@ -8,13 +8,10 @@ applies_to: # Applications UI annotation user [apm-app-annotation-user-create] - ::::{note} By default, the `viewer` and `editor` built-in roles provide read access to Observability annotations. You only need to create an annotation user to write to the annotations index ([`xpack.observability.annotations.index`](kibana://reference/configuration-reference/apm-settings.md)). :::: - - ## Annotation user [apm-app-annotation-user] View deployment annotations in the Applications UI. @@ -30,7 +27,6 @@ View deployment annotations in the Applications UI. 2. Assign the `annotation_user` created previously, and the roles and privileges necessary to create a [full](apm-reader-user.md#apm-app-reader-full) or [partial](apm-reader-user.md#apm-app-reader-partial) APM reader to any users that need to view annotations in the Applications UI - ## Annotation API [apm-app-annotation-api] See [Create an API user](applications-ui-api-user.md). diff --git a/solutions/observability/apps/applications-ui-api-user.md b/solutions/observability/apps/applications-ui-api-user.md index f82265b2df..37e20585e3 100644 --- a/solutions/observability/apps/applications-ui-api-user.md +++ b/solutions/observability/apps/applications-ui-api-user.md @@ -8,8 +8,6 @@ applies_to: # Applications UI API user [apm-app-api-user] - - ## Central configuration API [apm-app-api-config-manager] Users can list, search, create, update, and delete central configurations via the Applications UI API. @@ -20,8 +18,6 @@ Users can list, search, create, update, and delete central configurations via th | --- | --- | --- | | Kibana | `all` on the APM and User Experience feature | Allow all access to the Applications and User Experience UIs | - - ## Central configuration API reader [apm-app-api-config-reader] Sometimes a user only needs to list and search central configurations via the Applications UI API. @@ -32,8 +28,6 @@ Sometimes a user only needs to list and search central configurations via the Ap | --- | --- | --- | | Kibana | `read` on the APM and User Experience feature | Allow read access to the Applications and User Experience UIs | - - ## Annotation API [apm-app-api-annotation-manager] Users can use the annotation API to create annotations on their APM data. @@ -53,4 +47,3 @@ Users can use the annotation API to create annotations on their APM data. | --- | --- | --- | | Kibana | `all` on the APM and User Experience feature | Allow all access to the Applications and User Experience UIs | - diff --git a/solutions/observability/apps/applications-ui-central-config-user.md b/solutions/observability/apps/applications-ui-central-config-user.md index 3b6c8884bd..5307089163 100644 --- a/solutions/observability/apps/applications-ui-central-config-user.md +++ b/solutions/observability/apps/applications-ui-central-config-user.md @@ -8,8 +8,6 @@ applies_to: # Applications UI central config user [apm-app-central-config-user] - - ## Central configuration manager [apm-app-central-config-manager] Central configuration users need to be able to view, create, update, and delete APM agent configurations. @@ -57,8 +55,6 @@ Central configuration users need to be able to view, create, update, and delete | --- | --- | --- | | Kibana | `All` on the APM and User Experience feature | Allow full use of the Applications and User Experience UIs | - - ## Central configuration reader [apm-app-central-config-reader] In some instances, you may wish to create a user that can only read central configurations, but not create, update, or delete them. @@ -106,8 +102,6 @@ In some instances, you may wish to create a user that can only read central conf | --- | --- | --- | | Kibana | `read` on the APM and User Experience feature | Allow read access to the Applications and User Experience UIs | - - ## Central configuration API [apm-app-central-config-api] See [Create an API user](applications-ui-api-user.md). diff --git a/solutions/observability/apps/applications-ui-settings.md b/solutions/observability/apps/applications-ui-settings.md index c0e73853b9..4c29dc36df 100644 --- a/solutions/observability/apps/applications-ui-settings.md +++ b/solutions/observability/apps/applications-ui-settings.md @@ -10,7 +10,6 @@ applies_to: # Applications UI settings [observability-apm-kibana-settings] - ::::{note} The **Editor** role or higher is required to modify settings. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). @@ -19,7 +18,6 @@ The **Editor** role or higher is required to modify settings. To learn more, ref You can adjust Application settings to fine-tune your experience in the Applications UI. - ## General settings [observability-apm-kibana-settings-general-settings] ```{applies_to} stack: ga 9.1 @@ -66,7 +64,6 @@ The Applications UI uses data views to query APM indices. In non-serverless vers APM indices are {{kib}} Spaces-aware; Changes to APM index settings will only apply to the currently enabled space. See [Control access to APM data](../../../solutions/observability/apps/control-access-to-apm-data.md) for more information. - ## APM Labs [observability-apm-kibana-settings-apm-labs] **APM Labs** allows you to easily try out new features that are technical preview. diff --git a/solutions/observability/apps/applications-ui-storage-explorer-user.md b/solutions/observability/apps/applications-ui-storage-explorer-user.md index 86c9fece4f..7cb3cffc99 100644 --- a/solutions/observability/apps/applications-ui-storage-explorer-user.md +++ b/solutions/observability/apps/applications-ui-storage-explorer-user.md @@ -8,8 +8,6 @@ applies_to: # Applications UI storage explorer user [apm-app-storage-explorer-user-create] - - ## Storage Explorer user [apm-app-storage-explorer-user] View the **Storage Explorer** in the Applications UI. diff --git a/solutions/observability/apps/built-in-data-filters.md b/solutions/observability/apps/built-in-data-filters.md index 1e5d74dd7c..ea5c2624c1 100644 --- a/solutions/observability/apps/built-in-data-filters.md +++ b/solutions/observability/apps/built-in-data-filters.md @@ -17,7 +17,6 @@ Built-in data filters allow you to filter or turn off ingestion of the following | [Real user monitoring data](#apm-filters-real-user-data) | URLs visited, click events, user browser errors, resources used, etc. | | [Database statements](#apm-filters-database-statements) | Sensitive user or business information | - ## HTTP headers [apm-filters-http-header] By default, APM agents capture HTTP request and response headers (including cookies). Most Elastic APM agents provide the ability to sanitize HTTP header fields, including cookies and `application/x-www-form-urlencoded` data (POST form fields). Query string and captured request bodies, like `application/json` data, are not sanitized. @@ -42,7 +41,6 @@ Alternatively, you can completely disable the capturing of HTTP headers. This se * Python: [`capture_headers`](apm-agent-python://reference/configuration.md#config-capture-headers) * Ruby: [`capture_headers`](apm-agent-ruby://reference/configuration.md#config-capture-headers) - ## HTTP bodies [apm-filters-http-body] By default, the body of HTTP requests is not recorded. Request bodies often contain sensitive data like passwords or credit card numbers, so use care when enabling this feature. @@ -56,7 +54,6 @@ This setting supports [Central configuration](apm-agent-central-configuration.md * Python: [`capture_body`](apm-agent-python://reference/configuration.md#config-capture-body) * Ruby: [`capture_body`](apm-agent-ruby://reference/configuration.md#config-capture-body) - ## Personal data [apm-filters-personal-data] By default, the APM Server captures some personal data associated with trace events: @@ -70,7 +67,6 @@ The capturing of this data can be turned off by setting **Capture personal data* This setting only prevents APM Server from capturing already ingested personal data. It does not prevent such data from appearing in ingestion logs where applicable. See [{{apm-agent}} filters](custom-filters.md#apm-filters-in-agent) for redacting data on ingestion. ::: - ## Real user monitoring data [apm-filters-real-user-data] Protecting user data is important. For that reason, individual RUM instrumentations can be disabled in the RUM agent with the [`disableInstrumentations`](apm-agent-rum-js://reference/configuration.md#disable-instrumentations) configuration variable. Disabled instrumentations produce no spans or transactions. @@ -83,14 +79,12 @@ Protecting user data is important. For that reason, individual RUM instrumentati | User click events including URLs visited, mouse clicks, and navigation events | `eventtarget` | | Single page application route changes | `history` | - ## Database statements [apm-filters-database-statements] For SQL databases, APM agents do not capture the parameters of prepared statements. Note that Elastic APM currently does not make an effort to strip parameters of regular statements. Not using prepared statements makes your code vulnerable to SQL injection attacks, so be sure to use prepared statements. For non-SQL data stores, such as {{es}} or MongoDB, Elastic APM captures the full statement for queries. For inserts or updates, the full document is not stored. To filter or obfuscate data in non-SQL database statements, or to remove the statement entirely, you can set up an ingest node pipeline. - ## Agent-specific options [apm-filters-agent-specific] Certain agents offer additional filtering and obfuscating options: diff --git a/solutions/observability/apps/collect-application-data.md b/solutions/observability/apps/collect-application-data.md index f5f211fc53..9b0f532de0 100644 --- a/solutions/observability/apps/collect-application-data.md +++ b/solutions/observability/apps/collect-application-data.md @@ -7,7 +7,6 @@ applies_to: serverless: --- - # Collect application data [apm-collect-application-data] % is required role serverless only? @@ -33,10 +32,8 @@ Use Elastic APM agents or an OpenTelemetry language SDK to instrument a service * This option includes Elastic Distributions of OpenTelemetry, which are customized versions of [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/) that are optimized to work with an Elastic backend. - **Not sure which method is right for you?** Compare the available options below. - ### Capabilities [_capabilities] | | Elastic APM agent | Elastic Distribution of OpenTelemetry | @@ -61,7 +58,6 @@ Use Elastic APM agents or an OpenTelemetry language SDK to instrument a service | **Python** | Python agent | [preview] EDOT Python | | **Ruby** | Ruby agent | ![Not available](/solutions/images/observability-cross.svg "") | - ## Service-specific options [_service_specific_options] Elastic also offers several tools to help you collect data from specific services: diff --git a/solutions/observability/apps/collect-metrics.md b/solutions/observability/apps/collect-metrics.md index 679114b505..dc104da404 100644 --- a/solutions/observability/apps/collect-metrics.md +++ b/solutions/observability/apps/collect-metrics.md @@ -13,7 +13,6 @@ applies_to: When collecting metrics, please note that the [`DoubleValueRecorder`](https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/metrics/DoubleValueRecorder.md) and [`LongValueRecorder`](https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/metrics/LongValueObserver.md) metrics are not yet supported. :::: - Here’s an example of how to capture business metrics from a Java application. ```java @@ -32,7 +31,6 @@ public void createOrder(HttpServletRequest request) { See the [Open Telemetry Metrics API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md) for more information. - ## Verify OpenTelemetry metrics data [apm-open-telemetry-verify-metrics] Use **Discover** to validate that metrics are successfully reported to {{kib}}. @@ -43,7 +41,6 @@ Use **Discover** to validate that metrics are successfully reported to {{kib}}. 4. Filter the data to only show documents with metrics: `processor.name :"metric"` 5. Narrow your search with a known OpenTelemetry field. For example, if you have an `order_value` field, add `order_value: *` to your search to return only OpenTelemetry metrics documents. - ## Visualize your metrics[apm-open-telemetry-visualize] Use **Lens** to create visualizations for OpenTelemetry metrics. Lens enables you to build visualizations by dragging and dropping data fields. It makes smart visualization suggestions for your data, allowing you to switch between visualization types. diff --git a/solutions/observability/apps/configure-anonymous-authentication.md b/solutions/observability/apps/configure-anonymous-authentication.md index 6a21ccbcaa..8ce55bb5e4 100644 --- a/solutions/observability/apps/configure-anonymous-authentication.md +++ b/solutions/observability/apps/configure-anonymous-authentication.md @@ -8,7 +8,6 @@ applies_to: # Configure anonymous authentication [apm-configuration-anonymous] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-yes.svg "") @@ -16,7 +15,6 @@ Most options on this page are supported by all APM Server deployment methods. :::: - Elastic APM agents can send unauthenticated (anonymous) events to the APM Server. An event is considered to be anonymous if no authentication token can be extracted from the incoming request. This is useful for agents that run on clients, like the Real User Monitoring (RUM) agent running in a browser, or the Android or iOS/Swift agent running in a user application. Enable anonymous authentication in the APM Server to allow the ingestion of unauthenticated client-side APM data while still requiring authentication for server-side services. @@ -49,15 +47,12 @@ Configure and customize Fleet-managed APM settings directly in {{kib}}: All anonymous access configuration is ignored if [authenticated communication](secure-communication-with-apm-agents.md) is disabled. :::: - - ## Real User Monitoring (RUM) [apm-config-auth-anon-rum] If an [API key](api-keys.md) or [secret token](secret-token.md) is configured, then anonymous authentication must be enabled to collect RUM data. For this reason, anonymous auth will be enabled automatically if [Enable RUM](configure-real-user-monitoring-rum.md#apm-rum-enable) is set to `true`, and [Anonymous Agent access](#apm-config-auth-anon-enabled) is not explicitly defined. See [Real User Monitoring (RUM)](configure-real-user-monitoring-rum.md) for additional RUM configuration options. - ### Mitigating malicious requests [apm-config-auth-anon-mitigating] There are a few configuration variables that can mitigate the impact of malicious requests to an unauthenticated APM Server endpoint. @@ -66,7 +61,6 @@ Use the [Allowed anonymous agents](#apm-config-auth-anon-allow-agent) and [Allow Additionally, the APM Server can rate-limit unauthenticated requests based on the client IP address (`client.ip`) of the request with [Event limit](#apm-config-auth-anon-event-limit). This allows you to specify the maximum number of requests allowed per unique IP address, per second. - ### Deriving an incoming request’s `client.ip` address [apm-config-auth-anon-client-ip] The remote IP address of an incoming request might be different from the end-user’s actual IP address, for example, because of a proxy. For this reason, the APM Server attempts to derive the IP address of an incoming request from HTTP headers. The supported headers are parsed in the following order: @@ -77,17 +71,14 @@ The remote IP address of an incoming request might be different from the end-use If none of these headers are present, the remote address for the incoming request is used. - ### Using a reverse proxy or load balancer [apm-config-auth-anon-client-ip-concerns] HTTP headers are easily modified; it’s possible for anyone to spoof the derived `client.ip` value by changing or setting, for example, the value of the `X-Forwarded-For` header. For this reason, if any of your clients are not trusted, we recommend setting up a reverse proxy or load balancer in front of the APM Server. Using a proxy allows you to clear any existing IP-forwarding HTTP headers, and replace them with one set by the proxy. This prevents malicious users from cycling spoofed IP addresses to bypass the APM Server’s rate limiting feature. - ## Configuration reference [apm-config-auth-anon] - ### Anonymous Agent access [apm-config-auth-anon-enabled] Enable or disable anonymous authentication. Default: `false` (disabled). (bool) @@ -97,7 +88,6 @@ Enable or disable anonymous authentication. Default: `false` (disabled). (bool) | APM Server binary | `apm-server.auth.anonymous.enabled` | | Fleet-managed | `Anonymous Agent access` | - ### Allowed anonymous agents [apm-config-auth-anon-allow-agent] A list of permitted {{apm-agent}} names for anonymous authentication. Names in this list must match the agent’s `agent.name`. Default: `[rum-js, js-base]` (only RUM agent events are accepted). (array) @@ -107,7 +97,6 @@ A list of permitted {{apm-agent}} names for anonymous authentication. Names in t | APM Server binary | `apm-server.auth.anonymous.allow_agent` | | Fleet-managed | `Allowed Anonymous agents` | - ## Allowed services [apm-config-auth-anon-allow-service] A list of permitted service names for anonymous authentication. Names in this list must match the agent’s `service.name`. This can be used to limit the number of service-specific indices or data streams created. Default: Not set (any service name is accepted). (array) @@ -117,7 +106,6 @@ A list of permitted service names for anonymous authentication. Names in this li | APM Server binary | `apm-server.auth.anonymous.allow_service` | | Fleet-managed | `Allowed Anonymous services` | - ### IP limit [apm-config-auth-anon-ip-limit] The number of unique IP addresses to track in an LRU cache. IP addresses in the cache will be rate limited according to the [Event limit](#apm-config-auth-anon-event-limit) setting. Consider increasing this default if your application has many concurrent clients. Default: `1000`. (int) @@ -127,7 +115,6 @@ The number of unique IP addresses to track in an LRU cache. IP addresses in the | APM Server binary | `apm-server.auth.anonymous.rate_limit.ip_limit` | | Fleet-managed | `Anonymous Rate limit (IP limit)` | - ### Event limit [apm-config-auth-anon-event-limit] The maximum number of events allowed per second, per agent IP address. Default: `300`. (int) diff --git a/solutions/observability/apps/configure-apm-agent-central-configuration.md b/solutions/observability/apps/configure-apm-agent-central-configuration.md index 415e3e6b55..4134faede7 100644 --- a/solutions/observability/apps/configure-apm-agent-central-configuration.md +++ b/solutions/observability/apps/configure-apm-agent-central-configuration.md @@ -8,7 +8,6 @@ applies_to: # Configure APM agent central configuration [apm-configure-agent-config] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-yes.svg "") @@ -16,12 +15,10 @@ APM agent central configuration is supported by all APM Server deployment method :::: - APM agent central configuration allows you to fine-tune your APM agents from within the Applications UI. Changes are automatically propagated to your APM agents, so there’s no need to redeploy your applications. To learn more about this feature, see [APM agent central configuration](apm-agent-central-configuration.md). - ## APM agent configuration options [_apm_agent_configuration_options] **The following options are only supported for APM Server binary users**. @@ -35,23 +32,18 @@ apm-server.agent.config.elasticsearch.api_key: TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSo 1. You *must* set the API key to be configured to **Beats**. Base64 encoded API keys are not currently supported in this configuration. For details on how to create and configure a compatible API key, refer to [Create an API key for writing events](grant-access-using-api-keys.md#apm-beats-api-key-publish). - - ### `apm-server.agent.config.cache.expiration` [apm-agent-config-cache] When using APM agent central configuration, information fetched from {{es}} will be cached in memory for some time. Specify the cache expiration time via this setting. Defaults to `30s` (30 seconds). - ### `apm-server.agent.config.elasticsearch` [apm-agent-config-elasticsearch] Takes the same options as [output.elasticsearch](configure-elasticsearch-output.md). For APM Server binary users and Elastic Agent standalone-managed APM Server, APM agent central configuration is automatically fetched from {{es}} using the `output.elasticsearch` configuration. If `output.elasticsearch` isn’t set or doesn’t have sufficient privileges, use these {{es}} options to provide {{es}} access. - ### Common problems [_common_problems] - #### HTTP 403 errors [_http_403_errors] You may see either of the following HTTP 403 errors from APM Server when it attempts to fetch the APM agent central configuration: @@ -72,7 +64,6 @@ This occurs because the user or API key set in either `apm-server.agent.config.e To fix this error, ensure that APM Server has all the required privileges. For more details, refer to [Create a *central configuration management* role](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-agent-central-config-server). - #### HTTP 401 errors [_http_401_errors] If you get an HTTP 401 error from APM Server, make sure that you’re using an API key that is configured to **Beats**. For details on how to create and configure a compatible API key, refer to [Create an API key for writing events](grant-access-using-api-keys.md#apm-beats-api-key-publish). diff --git a/solutions/observability/apps/configure-apm-instrumentation.md b/solutions/observability/apps/configure-apm-instrumentation.md index ff5b886fa2..3fd4e36b82 100644 --- a/solutions/observability/apps/configure-apm-instrumentation.md +++ b/solutions/observability/apps/configure-apm-instrumentation.md @@ -8,7 +8,6 @@ applies_to: # Configure APM instrumentation [apm-configuration-instrumentation] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ Instrumentation of APM Server is not yet supported for Fleet-managed APM. :::: - APM Server uses the Elastic APM Go Agent to instrument its publishing pipeline. To gain insight into the performance of APM Server, you can enable this instrumentation and send trace data to APM Server. Currently, only the {{es}} output is instrumented. Example configuration with instrumentation enabled: @@ -30,32 +28,26 @@ instrumentation: api_key: L5ER6FEvjkmlfalBealQ3f3fLqf03fazfOV ``` - ## Configuration options [_configuration_options] You can specify the following options in the `instrumentation` section of the `apm-server.yml` config file: - ### `enabled` [_enabled] Set to `true` to enable instrumentation of APM Server. Defaults to `false`. - ### `environment` [_environment] Set the environment in which APM Server is running, for example, `staging`, `production`, `dev`, etc. Environments can be filtered in the [{{kib}} Applications UI](overviews.md). - ### `hosts` [_hosts] The [APM Server](get-started-with-apm.md) hosts to report instrumentation data to. Defaults to `http://localhost:8200`. - ### `api_key` [_api_key] [API key](api-keys.md) used to secure communication with the APM Server(s). If `api_key` is set then `secret_token` will be ignored. - ### `secret_token` [_secret_token_2] [Secret token](secret-token.md) used to secure communication with the APM Server(s). diff --git a/solutions/observability/apps/configure-apm-server.md b/solutions/observability/apps/configure-apm-server.md index de6207680a..814f0fcf11 100644 --- a/solutions/observability/apps/configure-apm-server.md +++ b/solutions/observability/apps/configure-apm-server.md @@ -39,13 +39,11 @@ Change how Elastic APM runs by providing your own user settings. Starting in {{s Check [APM configuration reference](/solutions/observability/apps/configure-apm-server.md) for information on how to configure Elastic APM in this mode. - Standalone APM Server (legacy) : Deployments created prior to {{stack}} version 8.0 are in legacy mode. Upgrading to or past {{stack}} 8.0 will not remove you from legacy mode. Check [Edit standalone APM settings (legacy)](../../../solutions/observability/apps/configure-apm-server.md#ec-edit-apm-standalone-settings) and [Supported standalone APM settings (legacy)](../../../solutions/observability/apps/configure-apm-server.md#ec-apm-settings) for information on how to configure Elastic APM in this mode. - To learn more about the differences between these modes, or to switch from Standalone APM Server (legacy) mode to {{fleet}}-managed, check [Switch to the Elastic APM integration](/solutions/observability/apps/switch-to-elastic-apm-integration.md). ## Edit standalone APM settings (legacy) [ec-edit-apm-standalone-settings] @@ -68,8 +66,6 @@ To add user settings: If a setting is not supported on {{ecloud}}, you will get an error message when you try to save. :::: - - ## Supported standalone APM settings (legacy) [ec-apm-settings] {{ech}} supports the following setting when running APM in standalone mode (legacy). @@ -78,7 +74,6 @@ If a setting is not supported on {{ecloud}}, you will get an error message when Some settings that could break your cluster if set incorrectly are blocklisted. The following settings are generally safe in cloud environments. For detailed information about APM settings, check the [APM documentation](/solutions/observability/apps/configure-apm-server.md). :::: - ### Version 8.0+ [ec_version_8_0_3] This stack version removes support for some previously supported settings. These are all of the supported settings for this version: @@ -169,7 +164,6 @@ The following `apm-server.auth.anonymous.*` settings can be configured to restri `output.elasticsearch.flush_bytes` : The bulk request size threshold, in bytes, before flushing to Elasticsearch. The value must have a suffix. The default is 5MB. - ### Version 7.17+ [ec_version_7_17] This stack version includes all of the settings from 7.16 and the following: @@ -194,7 +188,6 @@ Allow anonymous access only for specified agents and/or services. This is primar `apm-server.auth.api_key.limit` : Restrict how many unique API keys are allowed per minute. Should be set to at least the amount of different API keys configured in your monitored services. Every unique API key triggers one request to Elasticsearch. This will be replacing the config option `apm-server.api_key.limit` from `8.0` on. - ### Supported versions before 8.x [ec_supported_versions_before_8_x_3] `apm-server.aggregation.transactions.*` @@ -340,7 +333,6 @@ Allow anonymous access only for specified agents and/or services. This is primar | 4GB | 762 | | 8GB | 1067 | - `output.elasticsearch.indices` : Array of index selector rules supporting conditionals and formatted string. @@ -358,7 +350,6 @@ Allow anonymous access only for specified agents and/or services. This is primar | 4GB | 14 | | 8GB | 20 | - `queue.mem.events` : Maximum number of events to concurrently store in the internal queue. By default, this number changes based on the size of the instance: @@ -370,14 +361,12 @@ Allow anonymous access only for specified agents and/or services. This is primar | 4GB | 16000 | | 8GB | 32000 | - `queue.mem.flush.min_events` : Minimum number of events to have before pushing them to Elasticsearch. By default, this number changes based on the size of the instance. `queue.mem.flush.timeout` : Maximum duration before sending the events to the output if the `min_events` is not crossed. - ### Logging settings [ec_logging_settings] `logging.level` diff --git a/solutions/observability/apps/configure-console-output.md b/solutions/observability/apps/configure-console-output.md index 01d31d0a2c..03560ee239 100644 --- a/solutions/observability/apps/configure-console-output.md +++ b/solutions/observability/apps/configure-console-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the Console output [apm-console-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,14 +15,12 @@ The Console output is not yet supported by {{fleet}}-managed APM Server. :::: - The Console output writes events in JSON format to stdout. ::::{warning} The Console output should be used only for debugging issues as it can produce a large amount of logging data. :::: - To use this output, edit the APM Server configuration file to disable the {{es}} output by commenting it out, and enable the console output by adding `output.console`. Example configuration: @@ -33,31 +30,26 @@ output.console: pretty: true ``` - ## Configuration options [_configuration_options_7] You can specify the following `output.console` options in the `apm-server.yml` config file: - ### `enabled` [_enabled_6] The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled. The default value is `true`. - ### `pretty` [_pretty] If `pretty` is set to true, events written to stdout will be nicely formatted. The default is false. - ### `codec` [_codec_3] Output codec configuration. If the `codec` section is missing, events will be JSON encoded using the `pretty` option. See [Change the output codec](#apm-configuration-output-codec) for more information. - ### `bulk_max_size` [_bulk_max_size_4] The maximum number of events to buffer internally during publishing. The default is 2048. @@ -66,7 +58,6 @@ Specifying a larger batch size may add some latency and buffering during publish Setting `bulk_max_size` to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch. - ## Change the output codec [apm-configuration-output-codec] For outputs that do not require a specific encoding, you can change the encoding by using the codec configuration. You can specify either the `json` or `format` codec. By default the `json` codec is used. diff --git a/solutions/observability/apps/configure-elasticsearch-output.md b/solutions/observability/apps/configure-elasticsearch-output.md index bedc17e6ea..b146268df7 100644 --- a/solutions/observability/apps/configure-elasticsearch-output.md +++ b/solutions/observability/apps/configure-elasticsearch-output.md @@ -6,11 +6,8 @@ applies_to: stack: --- - - # Configure the Elasticsearch output [apm-elasticsearch-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -18,7 +15,6 @@ This documentation only applies to APM Server binary users. Fleet-managed users :::: - The {{es}} output sends events directly to {{es}} using the {{es}} HTTP API. Example configuration: @@ -30,7 +26,6 @@ output.elasticsearch: 1. To enable SSL, add `https` to all URLs defined under *hosts*. - When sending data to a secured cluster through the `elasticsearch` output, APM Server can use any of the following authentication methods: * Basic authentication credentials (username and password). @@ -56,7 +51,6 @@ output.elasticsearch: 1. You *must* set the API key to be configured to **Beats**. Base64 encoded API keys are not currently supported in this configuration. For details on how to create and configure a compatible API key, refer to [Create an API key for writing events](grant-access-using-api-keys.md#apm-beats-api-key-publish). - **PKI certificate authentication:** ```yaml @@ -68,24 +62,20 @@ output.elasticsearch: See [Secure communication with {{es}}](#apm-securing-communication-elasticsearch) for details on each authentication method. - ## Compatibility [_compatibility] This output works with all compatible versions of {{es}}. See the [Elastic Support Matrix](https://www.elastic.co/support/matrix#matrix_compatibility). - ## Configuration options [_configuration_options_3] You can specify the following options in the `elasticsearch` section of the `apm-server.yml` config file: - ### `enabled` [_enabled_2] The enabled config is a boolean setting to enable or disable the output. If set to `false`, the output is disabled. The default value is `true`. - ### `hosts` [apm-hosts-option] The list of {{es}} nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each {{es}} node can be defined as a `URL` or `IP:PORT`. For example: `http://192.15.3.2`, `https://es.found.io:9230` or `192.24.3.2:9300`. If no port is specified, `9200` is used. @@ -94,7 +84,6 @@ The list of {{es}} nodes to connect to. The events are distributed to these node When a node is defined as an `IP:PORT`, the *scheme* and *path* are taken from the [`protocol`](#apm-protocol-option) and [`path`](#apm-path-option) config options. :::: - ```yaml output.elasticsearch: hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] @@ -104,7 +93,6 @@ output.elasticsearch: In the previous example, the {{es}} nodes are available at `https://10.45.3.2:9220/elasticsearch` and `https://10.45.3.1:9230/elasticsearch`. - ### `compression_level` [_compression_level] The gzip compression level. Setting this value to `0` disables compression. The compression level must be in the range of `1` (best speed) to `9` (best compression). @@ -113,14 +101,12 @@ Increasing the compression level will reduce the network usage but will increase The default value is `0`. - ### `escape_html` [_escape_html] Configure escaping of HTML in strings. Set to `true` to enable escaping. The default value is `false`. - ### `api_key` [_api_key_2] Instead of using a username and password, you can use API keys to secure communication with {{es}}. The value must be the ID of the API key and the API key joined by a colon: `id:api_key`. @@ -131,34 +117,28 @@ You *must* set the API key to be configured to **Beats**. Base64 encoded API key :alt: API key dropdown highlighting the Beats option ::: - ### `username` [_username] The basic authentication username for connecting to {{es}}. This user needs the privileges required to publish events to {{es}}. To create a user like this, see [Create a *writer* role](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-to-publish-events). - ### `password` [_password] The basic authentication password for connecting to {{es}}. - ### `parameters` [_parameters] Dictionary of HTTP parameters to pass within the URL with index operations. - ### `protocol` [apm-protocol-option] The name of the protocol {{es}} is reachable on. The options are: `http` or `https`. The default is `http`. However, if you specify a URL for [`hosts`](#apm-hosts-option), the value of `protocol` is overridden by whatever scheme you specify in the URL. - ### `path` [apm-path-option] An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where {{es}} listens behind an HTTP reverse proxy that exports the API under a custom prefix. - ### `headers` [_headers] Custom HTTP headers to add to each request created by the {{es}} output. Example: @@ -170,12 +150,10 @@ output.elasticsearch.headers: It is possible to specify multiple header values for the same header name by separating them with a comma. - ### `proxy_url` [_proxy_url] The URL of the proxy to use when connecting to the {{es}} servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the [Go documentation](https://golang.org/pkg/net/http/#ProxyFromEnvironment) for more information about the environment variables. - ### `max_retries` [_max_retries] The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. @@ -184,39 +162,32 @@ Set `max_retries` to a value less than 0 to retry until all events are published The default is 3. - ### `flush_bytes` [_flush_bytes] The bulk request size threshold, in bytes, before flushing to {{es}}. The value must have a suffix, e.g. `"2MB"`. The default is `1MB`. - ### `flush_interval` [_flush_interval] The maximum duration to accumulate events for a bulk request before being flushed to {{es}}. The value must have a duration suffix, e.g. `"5s"`. The default is `1s`. - ### `backoff.init` [_backoff_init] The number of seconds to wait before trying to reconnect to {{es}} after a network error. After waiting `backoff.init` seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is `1s`. - ### `backoff.max` [_backoff_max] The maximum number of seconds to wait before attempting to connect to {{es}} after a network error. The default is `60s`. - ### `timeout` [_timeout] The HTTP request timeout in seconds for the {{es}} request. The default is 90. - ### `ssl` [_ssl] Configuration options for SSL parameters like the certificate authority to use for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to {{es}}. See the [secure communication with {{es}}](#apm-securing-communication-elasticsearch) guide or [SSL configuration reference](ssltls-output-settings.md) for more information. - ## Secure communication with {{es}} [apm-securing-communication-elasticsearch] When sending data to a secured cluster through the `elasticsearch` output, APM Server can use any of the following authentication methods: @@ -248,7 +219,6 @@ Authentication is specified in the APM Server configuration file: 1. This API key must have the privileges required to publish events to {{es}}. You *must* set the API key to be configured to **Beats**. Base64 encoded API keys are not currently supported in this configuration. For details on how to create and configure a compatible API key, refer to [Create an API key for writing events](grant-access-using-api-keys.md#apm-beats-api-key-publish). - * To use **Public Key Infrastructure (PKI) certificates** to authenticate users, specify the `certificate` and `key` settings under `output.elasticsearch`. For example: ```yaml @@ -261,7 +231,6 @@ Authentication is specified in the APM Server configuration file: 1. The path to the certificate for SSL client authentication 2. The client certificate key - These settings assume that the distinguished name (DN) in the certificate is mapped to the appropriate roles in the `role_mapping.yml` file on each node in the {{es}} cluster. For more information, see [Using role mapping files](../../../deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md#mapping-roles-file). By default, APM Server uses the list of trusted certificate authorities (CA) from the operating system where APM Server is running. If the certificate authority that signed your node certificates is not in the host system’s trusted certificate authorities list, you need to add the path to the `.pem` file that contains your CA’s certificate to the APM Server configuration. This will configure APM Server to use a specific list of CA certificates instead of the default list from the OS. @@ -282,14 +251,10 @@ Authentication is specified in the APM Server configuration file: 2. The path to the certificate for SSL client authentication 3. The client certificate key - ::::{note} For any given connection, the SSL/TLS certificates must have a subject that matches the value specified for `hosts`, or the SSL handshake fails. For example, if you specify `hosts: ["foobar:9200"]`, the certificate MUST include `foobar` in the subject (`CN=foobar`) or as a subject alternative name (SAN). Make sure the hostname resolves to the correct IP address. If no DNS is available, then you can associate the IP address with your hostname in `/etc/hosts` (on Unix) or `C:\Windows\System32\drivers\etc\hosts` (on Windows). :::: - - - ### Learn more about secure communication [apm-securing-communication-learn-more] More information on sending data to a secured cluster is available in the configuration reference: diff --git a/solutions/observability/apps/configure-individual-browser-monitors.md b/solutions/observability/apps/configure-individual-browser-monitors.md index 5de38c17b2..ddc54a1440 100644 --- a/solutions/observability/apps/configure-individual-browser-monitors.md +++ b/solutions/observability/apps/configure-individual-browser-monitors.md @@ -10,13 +10,11 @@ applies_to: # Configure individual browser monitors [synthetics-monitor-use] - ::::{note} This is only relevant for monitors that are created and managed using a [Synthetics project](../../../solutions/observability/apps/get-started.md#observability-synthetics-get-started-synthetics-project). For more information on configuring browser monitors added in the UI, refer to [Create monitors in the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md). :::: - After [writing synthetic journeys](../../../solutions/observability/apps/write-synthetic-test.md), you can use `monitor.use` to configure the browser monitors that will run your tests. You’ll need to set a few configuration options: diff --git a/solutions/observability/apps/configure-kafka-output.md b/solutions/observability/apps/configure-kafka-output.md index 74113039d2..9ad546a72c 100644 --- a/solutions/observability/apps/configure-kafka-output.md +++ b/solutions/observability/apps/configure-kafka-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the Kafka output [apm-kafka-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ The Kafka output is not yet supported by {{fleet}}-managed APM Server. :::: - The Kafka output sends events to Apache Kafka. To use this output, edit the APM Server configuration file to disable the {{es}} output by commenting it out, and enable the Kafka output by uncommenting the Kafka section. @@ -42,30 +40,24 @@ output.kafka: Events bigger than [`max_message_bytes`](#apm-kafka-max_message_bytes) will be dropped. To avoid this problem, make sure APM Server does not generate events bigger than [`max_message_bytes`](#apm-kafka-max_message_bytes). :::: - - ## Compatibility [apm-kafka-compatibility] This output works with all Kafka versions in between 0.11 and 2.2.2. Older versions might work as well, but are not supported. - ## Configuration options [_configuration_options_5] You can specify the following options in the `kafka` section of the `apm-server.yml` config file: - ### `enabled` [_enabled_4] The `enabled` config is a boolean setting to enable or disable the output. If set to false, the output is disabled. The default value is `false`. - ### `hosts` [_hosts_2] The list of Kafka broker addresses from where to fetch the cluster metadata. The cluster metadata contain the actual Kafka brokers events are published to. - ### `version` [_version] Kafka version apm-server is assumed to run against. Defaults to 1.0.0. @@ -76,24 +68,20 @@ Valid values are all Kafka releases in between `0.8.2.0` and `2.0.0`. See [Compatibility](#apm-kafka-compatibility) for information on supported versions. - ### `username` [_username_2] The username for connecting to Kafka. If username is configured, the password must be configured as well. - ### `password` [_password_2] The password for connecting to Kafka. - ### `sasl.mechanism` [_sasl_mechanism] ::::{warning} This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: - The SASL mechanism to use when connecting to Kafka. It can be one of: * `PLAIN` for SASL/PLAIN. @@ -102,7 +90,6 @@ The SASL mechanism to use when connecting to Kafka. It can be one of: If `sasl.mechanism` is not set, `PLAIN` is used if `username` and `password` are provided. Otherwise, SASL authentication is disabled. - ### `topic` [apm-topic-option-kafka] The Kafka topic used for produced events. @@ -115,7 +102,6 @@ topic: '%{[fields.log_topic]}' See the [`topics`](#apm-topics-option-kafka) setting for other ways to set the topic dynamically. - ### `topics` [apm-topics-option-kafka] An array of topic selector rules. Each rule specifies the `topic` to use for events that match the rule. During publishing, APM Server sets the `topic` for each event based on the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the `topics` setting is missing or no rule matches, the [`topic`](#apm-topic-option-kafka) field is used. @@ -151,14 +137,12 @@ output.kafka: This configuration results in topics named `critical-9.0.0-beta1`, `error-9.0.0-beta1`, and `logs-9.0.0-beta1`. - ### `key` [_key] Optional formatted string specifying the Kafka event key. If configured, the event key can be extracted from the event using a format string. See the Kafka documentation for the implications of a particular choice of key; by default, the key is chosen by the Kafka cluster. - ### `partition` [_partition] Kafka output broker event partitioning strategy. Must be one of `random`, `round_robin`, or `hash`. By default the `hash` partitioner is used. @@ -177,25 +161,20 @@ All partitioners will try to publish events to all partitions by default. If a p Publishing to a subset of available partitions potentially increases resource usage because events may become unevenly distributed. :::: - - ### `client_id` [_client_id] The configurable client ID used for logging, debugging, and auditing purposes. The default is "beats". - ### `worker` [_worker_2] The number of concurrent load-balanced Kafka output workers. - ### `codec` [_codec] Output codec configuration. If the `codec` section is missing, events will be JSON encoded. See [Change the output codec](configure-console-output.md#apm-configuration-output-codec) for more information. - ### `metadata` [_metadata] Kafka metadata update settings. The metadata do contain information about brokers, topics, partition, and active leaders to use for publishing. @@ -212,7 +191,6 @@ Kafka metadata update settings. The metadata do contain information about broker **`retry.backoff`** : Waiting time between retries during leader elections. Default is `250ms`. - ### `max_retries` [_max_retries_3] The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. @@ -221,47 +199,38 @@ Set `max_retries` to a value less than 0 to retry until all events are published The default is 3. - ### `backoff.init` [_backoff_init_3] The number of seconds to wait before trying to republish to Kafka after a network error. After waiting `backoff.init` seconds, APM Server tries to republish. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful publish, the backoff timer is reset. The default is `1s`. - ### `backoff.max` [_backoff_max_3] The maximum number of seconds to wait before attempting to republish to Kafka after a network error. The default is `60s`. - ### `bulk_max_size` [_bulk_max_size_2] The maximum number of events to bulk in a single Kafka request. The default is 2048. - ### `bulk_flush_frequency` [_bulk_flush_frequency] Duration to wait before sending bulk Kafka request. 0 is no delay. The default is 0. - ### `timeout` [_timeout_3] The number of seconds to wait for responses from the Kafka brokers before timing out. The default is 30 (seconds). - ### `broker_timeout` [_broker_timeout] The maximum duration a broker will wait for number of required ACKs. The default is `10s`. - ### `channel_buffer_size` [_channel_buffer_size] Per Kafka broker number of messages buffered in output pipeline. The default is 256. - ### `keep_alive` [_keep_alive] The keep-alive period for an active network connection. If `0s`, keep-alives are disabled. The default is `0s`. - ### `compression` [_compression] Sets the output compression codec. Must be one of `none`, `snappy`, `lz4` and `gzip`. The default is `gzip`. @@ -273,8 +242,6 @@ When targeting Azure Event Hub for Kafka, set `compression` to `none` as the pro :::: - - ### `compression_level` [_compression_level_3] Sets the compression level used by gzip. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression). @@ -283,29 +250,24 @@ Increasing the compression level will reduce the network usage but will increase The default value is 4. - ### `max_message_bytes` [apm-kafka-max_message_bytes] The maximum permitted size of JSON-encoded messages. Bigger messages will be dropped. The default value is 1000000 (bytes). This value should be equal to or less than the broker’s `message.max.bytes`. - ### `required_acks` [_required_acks] The ACK reliability level required from broker. 0=no response, 1=wait for local commit, -1=wait for all replicas to commit. The default is 1. Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error. - ### `enable_krb5_fast` [_enable_krb5_fast] ::::{warning} This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: - Enable Kerberos FAST authentication. This may conflict with some Active Directory installations. It is separate from the standard Kerberos settings because this flag only applies to the Kafka output. The default is `false`. - ### `ssl` [_ssl_3] Configuration options for SSL parameters like the root CA for Kafka connections. The Kafka host keystore should be created with the `-keyalg RSA` argument to ensure it uses a cipher supported by [{{filebeat}}'s Kafka library](https://github.com/Shopify/sarama/wiki/Frequently-Asked-Questions#why-cant-sarama-connect-to-my-kafka-cluster-using-ssl). See [SSL/TLS output settings](ssltls-output-settings.md) for more information. diff --git a/solutions/observability/apps/configure-kibana-endpoint.md b/solutions/observability/apps/configure-kibana-endpoint.md index c977dae335..ca61ad46ab 100644 --- a/solutions/observability/apps/configure-kibana-endpoint.md +++ b/solutions/observability/apps/configure-kibana-endpoint.md @@ -8,7 +8,6 @@ applies_to: # Configure the Kibana endpoint [apm-setup-kibana-endpoint] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -18,7 +17,6 @@ For all other use-cases, starting in version 8.7.0, APM agent configurations is :::: - Here’s a sample configuration: ```yaml @@ -26,17 +24,14 @@ apm-server.kibana.enabled: true apm-server.kibana.host: "http://localhost:5601" ``` - ## {{kib}} endpoint configuration options [_kib_endpoint_configuration_options] You can specify the following options in the `apm-server.kibana` section of the `apm-server.yml` config file. These options are not required for a Fleet-managed APM Server. - ### `apm-server.kibana.enabled` [apm-kibana-enabled] Defaults to `false`. Must be `true` to use APM Agent configuration. - ### `apm-server.kibana.host` [apm-kibana-host] The {{kib}} host that APM Server will communicate with. The default is `127.0.0.1:5601`. The value of `host` can be a `URL` or `IP:PORT`. For example: `http://192.15.3.2`, `192:15.3.2:5601` or `http://192.15.3.2:6701/path`. If no port is specified, `5601` is used. @@ -45,10 +40,8 @@ The {{kib}} host that APM Server will communicate with. The default is `127.0.0. When a node is defined as an `IP:PORT`, the *scheme* and *path* are taken from the [apm-server.kibana.protocol](#apm-kibana-protocol-option) and [apm-server.kibana.path](#apm-kibana-path-option) config options. :::: - IPv6 addresses must be defined using the following format: `https://[2001:db8::1]:5601`. - ### `apm-server.kibana.protocol` [apm-kibana-protocol-option] The name of the protocol {{kib}} is reachable on. The options are: `http` or `https`. The default is `http`. However, if you specify a URL for host, the value of `protocol` is overridden by whatever scheme you specify in the URL. @@ -61,27 +54,22 @@ apm-server.kibana.protocol: "http" apm-server.kibana.path: /kibana ``` - ### `apm-server.kibana.username` [_apm_server_kibana_username] The basic authentication username for connecting to {{kib}}. - ### `apm-server.kibana.password` [_apm_server_kibana_password] The basic authentication password for connecting to {{kib}}. - ### `apm-server.kibana.api_key` [_apm_server_kibana_api_key] Authentication with an API key. Formatted as `id:api_key` - ### `apm-server.kibana.path` [apm-kibana-path-option] An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where {{kib}} listens behind an HTTP reverse proxy that exports the API under a custom prefix. - ### `apm-server.kibana.ssl.enabled` [_apm_server_kibana_ssl_enabled] Enables APM Server to use SSL settings when connecting to {{kib}} via HTTPS. If you configure APM Server to connect over HTTPS, this setting defaults to `true` and APM Server uses the default SSL settings. diff --git a/solutions/observability/apps/configure-lightweight-monitors.md b/solutions/observability/apps/configure-lightweight-monitors.md index 3e20508063..8325f3bc29 100644 --- a/solutions/observability/apps/configure-lightweight-monitors.md +++ b/solutions/observability/apps/configure-lightweight-monitors.md @@ -17,7 +17,6 @@ Monitor the status of network endpoints using the following lightweight checks: Lightweight monitors can be configured using either the [Synthetics UI](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-ui) or [{{project-monitors-cap}}](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-projects). - ## Synthetics UI [synthetics-lightweight-ui] To use the UI, go to the Synthetics UI in {{kib}} or in your Observability Serverless project to create and configure monitors. For step-by-step instructions, refer to [Use the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md). @@ -27,7 +26,6 @@ To use the UI, go to the Synthetics UI in {{kib}} or in your Observability Serve :screenshot: ::: - ## {{project-monitors-cap}} [synthetics-lightweight-projects] To use YAML files to create lightweight monitors in a Synthetics project, [set up the Synthetics project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md) and configure monitors in YAML files in the `lightweight` directory. @@ -68,7 +66,6 @@ Each monitor type also has additional configuration options that are specific to The `tcp` and `http` monitor types both support SSL/TLS and some proxy settings. - ### Common options [synthetics-lightweight-common-options] You can specify the following options when defining a synthetic monitor in any location. These options are the same for all monitors. Each monitor type has additional configuration options that are specific to that monitor type. @@ -84,7 +81,6 @@ $$$monitor-type$$$ * `icmp`: Uses an ICMP (v4 and v6) Echo Request to ping the configured hosts. Requires special permissions or root access. * `tcp`: Connects via TCP and optionally verifies the endpoint by sending and/or receiving a custom payload. - $$$monitor-id$$$ **`id`** @@ -109,7 +105,6 @@ $$$monitor-id$$$ :::: - $$$monitor-name$$$ **`name`** @@ -127,7 +122,6 @@ $$$monitor-name$$$ name: Example website ``` - $$$monitor-service_name$$$ **`service.name`** @@ -135,7 +129,6 @@ $$$monitor-service_name$$$ APM service name for this monitor. Corresponds to the `service.name` ECS field. Set this when monitoring an app that is also using APM to enable integrations between Synthetics and APM data in Kibana or your Observability Serverless project. - $$$monitor-enabled$$$ **`enabled`** @@ -151,7 +144,6 @@ $$$monitor-enabled$$$ enabled: false ``` - $$$monitor-schedule$$$ **`schedule`** @@ -163,14 +155,12 @@ $$$monitor-schedule$$$ Schedules with less than 1 minute resolution will be saved to the nearest minute. For example, `@every 5s` will be changed to `@every 60s` when the monitor is pushed using the CLI. :::: - **Example**: Run the task every 5 minutes from the time the monitor was started. ```yaml schedule: @every 5m ``` - $$$monitor-timeout$$$ **`timeout`** @@ -186,7 +176,6 @@ $$$monitor-timeout$$$ timeout: 2m ``` - $$$monitor-tags$$$ **`tags`** @@ -206,7 +195,6 @@ $$$monitor-tags$$$ tags: ["tag one", "tag two"] ``` - $$$monitor-mode$$$ **`mode`** @@ -221,7 +209,6 @@ $$$monitor-mode$$$ **Example**: If you’re using a DNS-load balancer and want to ping every IP address for the specified hostname, you should use `all`. - $$$monitor-ipv4$$$ **`ipv4`** @@ -237,7 +224,6 @@ $$$monitor-ipv4$$$ ipv4: false ``` - $$$monitor-ipv6$$$ **`ipv6`** @@ -253,7 +239,6 @@ $$$monitor-ipv6$$$ ipv6: false ``` - $$$monitor-alert$$$ **`alert`** @@ -270,7 +255,6 @@ $$$monitor-alert$$$ alert.status.enabled: true ``` - **`tls.enabled`** ([boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool)) : Enable TLS certificate alerts on this monitor. @@ -282,7 +266,6 @@ $$$monitor-alert$$$ alert.tls.enabled: true ``` - $$$monitor-retest_on_failure$$$ **`retest_on_failure`** @@ -298,7 +281,6 @@ $$$monitor-retest_on_failure$$$ retest_on_failure: false ``` - $$$monitor-locations$$$ **`locations`** @@ -330,7 +312,6 @@ $$$monitor-locations$$$ :::: - $$$monitor-private_locations$$$ **`private_locations`** @@ -362,7 +343,6 @@ $$$monitor-private_locations$$$ :::: - $$$monitor-fields$$$ **`fields`** @@ -381,8 +361,6 @@ $$$monitor-fields$$$ fields.team: synthetics ``` - - ### HTTP options [synthetics-lightweight-http] The options described here configure Synthetics to connect via HTTP and optionally verify that the host returns the expected response. @@ -396,7 +374,6 @@ $$$monitor-http-urls$$$ **Required**. The URL to ping. - $$$monitor-http-max_redirects$$$ **`max_redirects`** @@ -410,7 +387,6 @@ $$$monitor-http-max_redirects$$$ **Default**: `0` - $$$monitor-http-proxy_headers$$$ **`proxy_headers`** @@ -427,7 +403,6 @@ $$$monitor-http-proxy_url$$$ http://proxy.mydomain.com:3128 ``` - $$$monitor-http-username$$$ **`username`** @@ -437,7 +412,6 @@ $$$monitor-http-username$$$ You need to specify credentials when your `check.response` settings require it. For example, you can check for a 403 response (`check.response.status: [403]`) without setting credentials. - $$$monitor-http-password$$$ **`password`** @@ -445,7 +419,6 @@ $$$monitor-http-password$$$ The password for authenticating with the server. This setting is optional. - $$$monitor-http-ssl$$$ **`ssl`** @@ -466,7 +439,6 @@ $$$monitor-http-ssl$$$ supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"] ``` - $$$monitor-http-headers$$$ **`headers`** @@ -476,7 +448,6 @@ $$$monitor-http-headers$$$ **Default**: `true` - $$$monitor-http-response$$$ **`response`** @@ -489,13 +460,11 @@ $$$monitor-http-response$$$ * `never`: Never include the body. * `always`: Always include the body with checks. - **`include_body_max_bytes`** ([number](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-numbers)) : Set `response.include_body_max_bytes` to control the maximum size of the stored body contents. **Default**: `1024` - $$$monitor-http-check$$$ **`check`** @@ -507,19 +476,16 @@ $$$monitor-http-check$$$ The HTTP method to use. - **`headers`** : Type: [HTTP headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) A dictionary of additional HTTP headers to send. By default Synthetics will set the *User-Agent* header to identify itself. - **`body`** : Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) Optional request body content. - **Example**: This monitor POSTs an `x-www-form-urlencoded` string to the endpoint `/demo/add`. ```yaml @@ -531,7 +497,6 @@ check.request: body: "name=first&email=someemail%40someemailprovider.com" ``` - **`response`** : The expected `response`. @@ -549,13 +514,11 @@ check.request: status: [200, 201] ``` - **`headers`** : Type: [HTTP headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) The required response headers. - **`body.positive`** : Type: list of [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s @@ -574,7 +537,6 @@ check.request: - Foo ``` - **`body.negative`** (list of [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s) : A list of regular expressions to match the body output negatively. Return match failed if single expression matches. HTTP response bodies of up to 100MiB are supported. @@ -607,7 +569,6 @@ check.request: - Bar ``` - **`json`** : A list of expressions executed against the body when parsed as JSON. Body sizes must be less than or equal to 100 MiB. @@ -627,8 +588,6 @@ check.request: expression: 'foo.bar == "myValue"' ``` - - ### ICMP options [synthetics-lightweight-icmp] The options described here configure Synthetics to use ICMP (v4 and v6) Echo Requests to check the configured hosts. On most platforms you must execute Synthetics with elevated permissions to perform ICMP pings. @@ -652,7 +611,6 @@ $$$monitor-icmp-hosts$$$ hosts: "myhost" ``` - $$$monitor-icmp-wait$$$ **`wait`** @@ -668,8 +626,6 @@ $$$monitor-icmp-wait$$$ wait: 1m ``` - - ### TCP options [synthetics-lightweight-tcp] The options described here configure Synthetics to connect via TCP and optionally verify the endpoint by sending and/or receiving a custom payload. @@ -690,7 +646,6 @@ $$$monitor-tcp-hosts$$$ * `host` is the hostname. * `port` is the port number. - **Examples**: ```yaml @@ -701,7 +656,6 @@ $$$monitor-tcp-hosts$$$ hosts: "tcp://localhost:8000" ``` - $$$monitor-tcp-check$$$ **`check`** @@ -720,7 +674,6 @@ $$$monitor-tcp-check$$$ receive: 'Hello World' ``` - $$$monitor-tcp-proxy_url$$$ **`proxy_url`** @@ -738,7 +691,6 @@ $$$monitor-tcp-proxy_url$$$ proxy_url: socks5://user:password@socks5-proxy:2233 ``` - $$$monitor-tcp-proxy_use_local_resolver$$$ **`proxy_use_local_resolver`** @@ -748,7 +700,6 @@ $$$monitor-tcp-proxy_use_local_resolver$$$ **Default**: `false` - $$$monitor-tcp-ssl$$$ **`ssl`** @@ -766,13 +717,10 @@ $$$monitor-tcp-ssl$$$ Also see [Configure SSL](beats://reference/heartbeat/configuration-ssl.md) for a full description of the `ssl` options. - - ### Data types reference [synthetics-lightweight-data-types] Values of configuration settings are interpreted as required by Synthetics. If a value can’t be correctly interpreted as the required type - for example a string is given when a number is required - Synthetics will fail to start up. - #### Boolean [synthetics-lightweight-data-bool] Boolean values can be either `true` or `false`. Alternative names for `true` are `yes` and `on`. Instead of `false` the values `no` and `off` can be used. @@ -782,7 +730,6 @@ enabled: true disabled: false ``` - #### Number [synthetics-lightweight-data-numbers] Number values require you to enter the number *without* single or double quotes. @@ -797,8 +744,6 @@ float: 5.4 Some settings only support a restricted number range. :::: - - #### String [synthetics-lightweight-data-string] In [YAML](http://www.yaml.org), multiple styles of string definitions are supported: double-quoted, single-quoted, unquoted. @@ -813,8 +758,6 @@ Unquoted style requires no quotes, but does not support any escaping and can’t Single-quoted style is recommended when defining regular expressions, event format strings, windows file paths, or non-alphabetical symbolic characters. :::: - - #### Duration [synthetics-lightweight-data-duration] Durations require a numeric value with optional fraction and required unit. Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Sometimes features based on durations can be disabled by using zero or negative durations. @@ -825,7 +768,6 @@ duration2: 6h duration_disabled: -1s ``` - #### Regular expression [synthetics-lightweight-data-regex] Regular expressions are special strings that are compiled into regular expressions at load time. diff --git a/solutions/observability/apps/configure-logging.md b/solutions/observability/apps/configure-logging.md index 58d8940518..cddbf9b419 100644 --- a/solutions/observability/apps/configure-logging.md +++ b/solutions/observability/apps/configure-logging.md @@ -8,7 +8,6 @@ applies_to: # Configure logging [apm-configuration-logging] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ These configuration options are only relevant to APM Server binary users. Fleet- :::: - The `logging` section of the `apm-server.yml` config file contains options for configuring the logging output. The logging system can write logs to the syslog or rotate log files. If logging is not explicitly configured the file output is used. @@ -35,23 +33,18 @@ logging.files: In addition to setting logging options in the config file, you can modify the logging output configuration from the command line. See [Command reference](apm-server-command-reference.md). :::: - ::::{warning} When APM Server is running on a Linux system with systemd, it uses by default the `-e` command line option, that makes it write all the logging output to stderr so it can be captured by journald. Other outputs are disabled. See [APM Server and systemd](apm-server-systemd.md) to know more and learn how to change this. :::: - - ## Configuration options [_configuration_options_2] You can specify the following options in the `logging` section of the `apm-server.yml` config file: - ### `logging.to_stderr` [_logging_to_stderr] When true, writes all logging output to standard error output. This is equivalent to using the `-e` command line option. - ### `logging.to_syslog` [_logging_to_syslog] When true, writes all logging output to the syslog. @@ -60,13 +53,10 @@ When true, writes all logging output to the syslog. This option is not supported on Windows. :::: - - ### `logging.to_eventlog` [_logging_to_eventlog] When true, writes all logging output to the Windows Event Log. - ### `logging.to_files` [_logging_to_files] When true, writes all logging output to files. The log files are automatically rotated when the log file size limit is reached. @@ -75,8 +65,6 @@ When true, writes all logging output to files. The log files are automatically r APM Server only creates a log file if there is logging output. For example, if you set the log [`level`](#apm-level) to `error` and there are no errors, there will be no log file in the directory specified for logs. :::: - - ### `logging.level` [apm-level] Minimum log level. One of `debug`, `info`, `warning`, or `error`. The default log level is `info`. @@ -93,7 +81,6 @@ Minimum log level. One of `debug`, `info`, `warning`, or `error`. The default lo `error` : Logs errors and critical errors. - ### `logging.selectors` [apm-selectors] The list of debugging-only selector tags used by different APM Server components. Use `*` to enable debug output for all components. Use `publisher` to display debug messages related to event publishing. @@ -105,7 +92,6 @@ To see which selectors are available, run APM Server in debug mode (set `logging :::: - To configure multiple selectors, use the following [YAML list syntax](beats://reference/libbeat/config-file-format.md): ```yaml @@ -114,7 +100,6 @@ logging.selectors: [ harvester, input ] To override selectors at the command line, use the `-d` global flag (`-d` also sets the debug log level). For more information, see [Command reference](apm-server-command-reference.md). - ### `logging.metrics.enabled` [_logging_metrics_enabled] By default, APM Server periodically logs its internal metrics that have changed in the last period. For each metric that changed, the delta from the value at the beginning of the period is logged. Also, the total values for all non-zero internal metrics are logged on shutdown. Set this to false to disable this behavior. The default is true. @@ -127,32 +112,26 @@ Here is an example log line: Note that we currently offer no backwards compatible guarantees for the internal metrics and for this reason they are also not documented. - ### `logging.metrics.period` [_logging_metrics_period] The period after which to log the internal metrics. The default is `30s`. - ### `logging.files.path` [_logging_files_path] The directory that log files are written to. The default is the logs path. See the [Installation layout](installation-layout.md) section for details. - ### `logging.files.name` [_logging_files_name] The name of the file that logs are written to. The default is *apm-server*. - ### `logging.files.rotateeverybytes` [_logging_files_rotateeverybytes] The maximum size of a log file. If the limit is reached, a new log file is generated. The default size limit is 10485760 (10 MB). - ### `logging.files.keepfiles` [_logging_files_keepfiles] The number of most recent rotated log files to keep on disk. Older files are deleted during log rotation. The default value is 7. The `keepfiles` options has to be in the range of 2 to 1024 files. - ### `logging.files.permissions` [_logging_files_permissions] The permissions mask to apply when rotating log files. The default value is 0600. The `permissions` option must be a valid Unix-style file permissions mask expressed in octal notation. In Go, numbers in octal notation must start with *0*. @@ -164,22 +143,18 @@ Examples: * 0640: give read and write access to the file owner, and read access to members of the group associated with the file. * 0600: give read and write access to the file owner, and no access to all others. - ### `logging.files.interval` [_logging_files_interval] Enable log file rotation on time intervals in addition to size-based rotation. Intervals must be at least `1s`. Values of `1m`, `1h`, `24h`, `7*24h`, `30*24h`, and `365*24h` are boundary-aligned with minutes, hours, days, weeks, months, and years as reported by the local system clock. All other intervals are calculated from the Unix epoch. Defaults to disabled. - ### `logging.files.rotateonstartup` [_logging_files_rotateonstartup] If the log file already exists on startup, immediately rotate it and start writing to a new file instead of appending to the existing one. Defaults to true. - ### `logging.files.redirect_stderr` [preview] [_logging_files_redirect_stderr] When true, diagnostic messages printed to APM Server’s standard error output will also be logged to the log file. This can be helpful in situations were APM Server terminates unexpectedly because an error has been detected by Go’s runtime but diagnostic information is not present in the log file. This feature is only available when logging to files (`logging.to_files` is true). Disabled by default. - ## Logging format [_logging_format] The logging format is generally the same for each logging output. The one exception is with the syslog output where the timestamp is not included in the message because syslog adds its own timestamp. diff --git a/solutions/observability/apps/configure-logstash-output.md b/solutions/observability/apps/configure-logstash-output.md index 016ad82ff1..693bbe053f 100644 --- a/solutions/observability/apps/configure-logstash-output.md +++ b/solutions/observability/apps/configure-logstash-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the Logstash output [apm-logstash-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,10 +15,8 @@ The {{ls}} output is not yet supported by {{fleet}}-managed APM Server. :::: - {{ls}} allows for additional processing and routing of APM events. The {{ls}} output sends events directly to {{ls}} using the lumberjack protocol, which runs over TCP. - ## Send events to {{ls}} [_send_events_to_ls] To send events to {{ls}}, you must: @@ -27,7 +24,6 @@ To send events to {{ls}}, you must: 1. [Enable the {{ls}} output in APM Server](#apm-ls-output-config) 2. [Create a {{ls}} configuration pipeline in {{ls}}](#apm-ls-config-pipeline) - ### {{ls}} output configuration [apm-ls-output-config] To enable the {{ls}} output in APM Server, edit the `apm-server.yml` file to: @@ -43,8 +39,6 @@ To enable the {{ls}} output in APM Server, edit the `apm-server.yml` file to: 1. The `hosts` option specifies the {{ls}} server and the port (`5044`) where {{ls}} is configured to listen for incoming APM Server connections. - - ### {{ls}} configuration pipeline [apm-ls-config-pipeline] Finally, you must create a {{ls}} configuration pipeline that listens for incoming APM Server connections and indexes received events into {{es}}. @@ -74,7 +68,6 @@ Finally, you must create a {{ls}} configuration pipeline that listens for incomi 1. Enables indexing into {{es}} data streams. 2. This example assumes you’re sending data to {{ecloud}}. If you’re using a self-hosted version of {{es}}, use `hosts` instead. See [{{es}} output plugin](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) for more information. - Here’s what your basic {{ls}} configuration file will look like when we put everything together: ```json @@ -93,7 +86,6 @@ output { } ``` - ## Accessing the @metadata field [_accessing_the_metadata_field] Every event sent to {{ls}} contains a special field called [`@metadata`](logstash://reference/event-dependent-configuration.md#metadata) that you can use in {{ls}} for conditionals, filtering, indexing and more. APM Server sends the following `@metadata` to {{ls}}: @@ -111,7 +103,6 @@ Every event sent to {{ls}} contains a special field called [`@metadata`](logstas 1. To change the default `apm-server` value, set the [`index`](#apm-logstash-index) option in the APM Server config file. 2. The current version of APM Server. - In addition to `@metadata`, APM Server provides other potentially useful fields, like the `data_stream` field, which can be used to conditionally operate on [event types](learn-about-application-data-types.md), namespaces, or datasets. As an example, you might want to use {{ls}} to route all `metrics` events to the same custom metrics data stream, rather than to service-specific data streams. @@ -144,32 +135,26 @@ output { 3. Add a Logstash mutate filter to update the value of `data_stream.dataset`. 4. In this example, `cloud_id` and `cloud_auth` are stored as [environment variables](logstash://reference/environment-variables.md). - - ### Compatibility [_compatibility_2] This output works with all compatible versions of {{ls}}. See the [Elastic Support Matrix](https://www.elastic.co/support/matrix#matrix_compatibility). - ### Configuration options [_configuration_options_4] You can specify the following options in the `logstash` section of the `apm-server.yml` config file: - #### `enabled` [_enabled_3] The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled. The default value is `false`. - #### `hosts` [apm-hosts] The list of known {{ls}} servers to connect to. If load balancing is disabled, but multiple hosts are configured, one host is selected randomly (there is no precedence). If one host becomes unreachable, another one is selected randomly. All entries in this list can contain a port number. The default port number 5044 will be used if no number is given. - #### `compression_level` [_compression_level_2] The gzip compression level. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression). @@ -178,19 +163,16 @@ Increasing the compression level will reduce the network usage but will increase The default value is 3. - #### `escape_html` [_escape_html_2] Configure escaping of HTML in strings. Set to `true` to enable escaping. The default value is `false`. - #### `worker` [_worker] The number of workers per configured host publishing events to {{ls}}. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host). - #### `loadbalance` [apm-loadbalance] If set to true and multiple {{ls}} hosts are configured, the output plugin load balances published events onto all {{ls}} hosts. If set to false, the output plugin sends all events to only one host (determined at random) and will switch to another host if the selected one becomes unresponsive. The default value is false. @@ -202,7 +184,6 @@ output.logstash: index: apm-server ``` - #### `ttl` [_ttl] Time to live for a connection to {{ls}} after which the connection will be re-established. Useful when {{ls}} hosts represent load balancers. Since the connections to {{ls}} hosts are sticky, operating behind load balancers can lead to uneven load distribution between the instances. Specifying a TTL on the connection allows to achieve equal connection distribution between the instances. Specifying a TTL of 0 will disable this feature. @@ -213,13 +194,10 @@ The default value is 0. The "ttl" option is not yet supported on an asynchronous {{ls}} client (one with the "pipelining" option set). :::: - - #### `pipelining` [_pipelining] Configures the number of batches to be sent asynchronously to {{ls}} while waiting for ACK from {{ls}}. Output only becomes blocking once number of `pipelining` batches have been written. Pipelining is disabled if a value of 0 is configured. The default value is 2. - #### `proxy_url` [_proxy_url_2] The URL of the SOCKS5 proxy to use when connecting to the {{ls}} servers. The value must be a URL with a scheme of `socks5://`. The protocol used to communicate to {{ls}} is not based on HTTP so a web-proxy cannot be used. @@ -234,12 +212,10 @@ output.logstash: proxy_url: socks5://user:password@socks5-proxy:2233 ``` - #### `proxy_use_local_resolver` [apm-logstash-proxy-use-local-resolver] The `proxy_use_local_resolver` option determines if {{ls}} hostnames are resolved locally when using a proxy. The default value is false, which means that when a proxy is used the name resolution occurs on the proxy server. - #### `index` [apm-logstash-index] The index root name to write events to. The default is `apm-server`. For example `"apm"` generates `"[apm-]9.0.0-beta1-YYYY.MM.DD"` indices (for example, `"apm-9.0.0-beta1-2017.04.26"`). @@ -248,18 +224,14 @@ The index root name to write events to. The default is `apm-server`. For example This parameter’s value will be assigned to the `metadata.beat` field. It can then be accessed in {{ls}}'s output section as `%{[@metadata][beat]}`. :::: - - #### `ssl` [_ssl_2] Configuration options for SSL parameters like the root CA for {{ls}} connections. See [SSL/TLS output settings](ssltls-output-settings.md) for more information. To use SSL, you must also configure the [{{beats}} input plugin for {{ls}}](logstash-docs-md://lsr/plugins-inputs-beats.md) to use SSL/TLS. - #### `timeout` [_timeout_2] The number of seconds to wait for responses from the {{ls}} server before timing out. The default is 30 (seconds). - #### `max_retries` [_max_retries_2] The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. @@ -268,7 +240,6 @@ Set `max_retries` to a value less than 0 to retry until all events are published The default is 3. - #### `bulk_max_size` [_bulk_max_size] The maximum number of events to bulk in a single {{ls}} request. The default is 2048. @@ -279,24 +250,20 @@ Specifying a larger batch size can improve performance by lowering the overhead Setting `bulk_max_size` to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch. - #### `slow_start` [_slow_start] If enabled, only a subset of events in a batch of events is transferred per transaction. The number of events to be sent increases up to `bulk_max_size` if no error is encountered. On error, the number of events per transaction is reduced again. The default is `false`. - #### `backoff.init` [_backoff_init_2] The number of seconds to wait before trying to reconnect to {{ls}} after a network error. After waiting `backoff.init` seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is `1s`. - #### `backoff.max` [_backoff_max_2] The maximum number of seconds to wait before attempting to connect to {{ls}} after a network error. The default is `60s`. - ## Secure communication with {{ls}} [apm-configuring-ssl-logstash] You can use SSL mutual authentication to secure connections between APM Server and {{ls}}. This ensures that APM Server sends encrypted data to trusted {{ls}} servers only, and that the {{ls}} server receives data from trusted APM Server clients only. @@ -350,8 +317,6 @@ To use SSL mutual authentication: For more information about these options, see the [documentation for the {{beats}} input plugin](logstash-docs-md://lsr/plugins-inputs-beats.md). - - ### Validate the {{ls}} server’s certificate [apm-testing-ssl-logstash] Before running APM Server, you should validate the {{ls}} server’s certificate. You can use `curl` to validate the certificate even though the protocol used to communicate with {{ls}} is not based on HTTP. For example: diff --git a/solutions/observability/apps/configure-output-for-elasticsearch-service-on-elastic-cloud.md b/solutions/observability/apps/configure-output-for-elasticsearch-service-on-elastic-cloud.md index 66aae9d732..b9814fe31c 100644 --- a/solutions/observability/apps/configure-output-for-elasticsearch-service-on-elastic-cloud.md +++ b/solutions/observability/apps/configure-output-for-elasticsearch-service-on-elastic-cloud.md @@ -8,7 +8,6 @@ applies_to: # Configure the output for {{ech}} [apm-configure-cloud-id] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,12 +15,10 @@ This documentation only applies to APM Server binary users. :::: - ::::{note} This page refers to using a separate instance of APM Server with an existing [{{ech}} deployment](https://www.elastic.co/cloud/elasticsearch-service?page=docs&placement=docs-body). If you want to use APM on {{ech}}, see: [Create your deployment](../../../deploy-manage/deploy/elastic-cloud/create-an-elastic-cloud-hosted-deployment.md) and [Add APM user settings](configure-apm-server.md). :::: - APM Server comes with two settings that simplify the output configuration when used together with [{{ech}}](https://www.elastic.co/cloud/elasticsearch-service?page=docs&placement=docs-body). When defined, these setting overwrite settings from other parts in the configuration. Example: @@ -37,12 +34,10 @@ These settings can be also specified at the command line, like this: apm-server -e -E cloud.id="" -E cloud.auth="" ``` - ## `cloud.id` [_cloud_id] The Cloud ID, which can be found in the {{ecloud}} Console, is used by APM Server to resolve the {{es}} and {{kib}} URLs. This setting overwrites the `output.elasticsearch.hosts` and `setup.kibana.host` settings. - ## `cloud.auth` [_cloud_auth] When specified, the `cloud.auth` overwrites the `output.elasticsearch.username` and `output.elasticsearch.password` settings. Because the {{kib}} settings inherit the username and password from the {{es}} output, this can also be used to set the `setup.kibana.username` and `setup.kibana.password` options. diff --git a/solutions/observability/apps/configure-output.md b/solutions/observability/apps/configure-output.md index c7ceb8003a..70db626f2d 100644 --- a/solutions/observability/apps/configure-output.md +++ b/solutions/observability/apps/configure-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the output [apm-configuring-output] - Output configuration options. * [{{ech}}](configure-output-for-elasticsearch-service-on-elastic-cloud.md) @@ -18,14 +17,7 @@ Output configuration options. * [Redis](configure-redis-output.md) * [Console](configure-console-output.md) - ## Source maps [apm-sourcemap-output] Source maps can be uploaded through all outputs but must eventually be stored in {{es}}. When using outputs other than {{es}}, `source_mapping.elasticsearch` must be set for source maps to be applied. Be sure to update `source_mapping.index_pattern` if source maps are stored in the non-default location. See [`source_mapping.elasticsearch`](configure-real-user-monitoring-rum.md#apm-config-sourcemapping-elasticsearch) for more details. - - - - - - diff --git a/solutions/observability/apps/configure-project-paths.md b/solutions/observability/apps/configure-project-paths.md index 0814617a5c..f6cfff9dc0 100644 --- a/solutions/observability/apps/configure-project-paths.md +++ b/solutions/observability/apps/configure-project-paths.md @@ -8,7 +8,6 @@ applies_to: # Configure project paths [apm-configuration-path] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ This documentation is only relevant for APM Server binary users. Fleet-managed p :::: - The `path` section of the `apm-server.yml` config file contains configuration options that define where APM Server looks for its files. For example, APM Server looks for the {{es}} template file in the configuration path and writes log files in the logs path. Please see the [Installation layout](installation-layout.md) section for more details. @@ -32,12 +30,10 @@ path.logs: /var/log/ Note that it is possible to override these options by using command line flags. - ## Configuration options [_configuration_options_8] You can specify the following options in the `path` section of the `apm-server.yml` config file: - ### `home` [_home] The home path for the APM Server installation. This is the default base path for all other path settings and for miscellaneous files that come with the distribution (for example, the sample dashboards). If not set by a CLI flag or in the configuration file, the default for the home path is the location of the APM Server binary. @@ -48,7 +44,6 @@ Example: path.home: /usr/share/beats ``` - ### `config` [_config] The configuration path for the APM Server installation. This is the default base path for configuration files, including the main YAML configuration file and the {{es}} template file. If not set by a CLI flag or in the configuration file, the default for the configuration path is the home path. @@ -59,7 +54,6 @@ Example: path.config: /usr/share/beats/config ``` - ### `data` [_data] The data path for the APM Server installation. This is the default base path for all the files in which APM Server needs to store its data. If not set by a CLI flag or in the configuration file, the default for the data path is a `data` subdirectory inside the home path. @@ -74,8 +68,6 @@ path.data: /var/lib/beats When running multiple APM Server instances on the same host, make sure they each have a distinct `path.data` value. :::: - - ### `logs` [_logs] The logs path for a APM Server installation. This is the default location for APM Server’s log files. If not set by a CLI flag or in the configuration file, the default for the logs path is a `logs` subdirectory inside the home path. @@ -86,7 +78,6 @@ Example: path.logs: /var/log/beats ``` - ### `system.hostfs` [_system_hostfs] Specifies the mount point of the host’s file system for use in monitoring a host. This can either be set in the config, or with the `--system.hostfs` CLI flag. This is used for cgroup self-monitoring. diff --git a/solutions/observability/apps/configure-real-user-monitoring-rum.md b/solutions/observability/apps/configure-real-user-monitoring-rum.md index ebbdf99559..91e5930fce 100644 --- a/solutions/observability/apps/configure-real-user-monitoring-rum.md +++ b/solutions/observability/apps/configure-real-user-monitoring-rum.md @@ -8,7 +8,6 @@ applies_to: # Configure Real User Monitoring (RUM) [apm-configuration-rum] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-yes.svg "") @@ -16,7 +15,6 @@ Most options in this section are supported by all APM Server deployment methods. :::: - The [Real User Monitoring (RUM) agent](apm-agent-rum-js://reference/index.md) captures user interactions with clients such as web browsers. These interactions are sent as events to the APM Server. Because the RUM agent runs on the client side, the connection between agent and server is unauthenticated. As a security precaution, RUM is therefore disabled by default. :::::::{tab-set} @@ -55,10 +53,8 @@ Configure and customize Fleet-managed APM settings directly in {{kib}}: ::::::: In addition, if APM Server is deployed in an origin different than the page’s origin, you will need to configure [Cross-Origin Resource Sharing (CORS)](apm-agent-rum-js://reference/configuring-cors.md) in the Agent. - ## Configuration reference [apm-enable-rum-support] - ### Enable RUM [apm-rum-enable] To enable RUM support, set to `true`. By default this is disabled. (bool) @@ -73,8 +69,6 @@ If an [API key](api-keys.md) or [secret token](secret-token.md) is configured, e :::: - - ### Allowed Origins [apm-rum-allow-origins] A list of permitted origins for RUM support. User-agents send an Origin header that will be validated against this list. This is done automatically by modern browsers as part of the [CORS specification](https://www.w3.org/TR/cors/). An origin is made of a protocol scheme, host and port, without the URL path. @@ -86,7 +80,6 @@ Default: `['*']` (allows everything). (text) | APM Server binary | `apm-server.rum.allow_origins` | | Fleet-managed | `Allowed Origins` | - ### Access-Control-Allow-Headers [apm-rum-allow-headers] HTTP requests made from the RUM agent to the APM Server are limited in the HTTP headers they are allowed to have. If any other headers are added, the request will be rejected by the browser due to Cross-Origin Resource Sharing (CORS) restrictions. Use this setting to allow additional headers. The default list of allowed headers includes "Content-Type", "Content-Encoding", and "Accept"; custom values configured here are appended to the default list and used as the value for the `Access-Control-Allow-Headers` header. @@ -98,7 +91,6 @@ Default: `[]`. (text) | APM Server binary | `apm-server.rum.allow_headers` | | Fleet-managed | `Access-Control-Allow-Headers` | - ### Custom HTTP response headers [apm-rum-response-headers] Custom HTTP headers to add to RUM responses. This can be useful for security policy compliance. @@ -112,7 +104,6 @@ Default: none. (text) | APM Server binary | `apm-server.rum.response_headers` | | Fleet-managed | `Custom HTTP response headers` | - ### Library Frame Pattern [apm-rum-library-pattern] RegExp to be matched against a stack trace frame’s `file_name` and `abs_path` attributes. If the RegExp matches, the stack trace frame is considered to be a library frame. When source mapping is applied, the `error.culprit` is set to reflect the *function* and the *filename* of the first non library frame. This aims to provide an entry point for identifying issues. @@ -124,7 +115,6 @@ Default: `"node_modules|bower_components|~"`. (text) | APM Server binary | `apm-server.rum.library_pattern` | | Fleet-managed | `Library Frame Pattern` | - ### Exclude from grouping [_exclude_from_grouping] RegExp to be matched against a stack trace frame’s `file_name`. If the RegExp matches, the stack trace frame is excluded from being used for calculating error groups. @@ -136,7 +126,6 @@ Default: `"^/webpack"` (excludes stack trace frames that have a filename startin | APM Server binary | `apm-server.rum.exclude_from_grouping` | | Fleet-managed | `Exclude from grouping` | - ## Source map configuration options [apm-rum-source-map] ::::{admonition} @@ -146,34 +135,28 @@ Source maps are supported by all APM Server deployment methods, however, the opt :::: - - ### `source_mapping.enabled` [apm-config-sourcemapping-enabled] Used to enable/disable [source mapping](create-upload-source-maps-rum.md) for RUM events. When enabled, the APM Server needs additional privileges to read source maps. See [Use feature roles](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-rum-source-mapping) for more details. Default: `true` - ### `source_mapping.elasticsearch` [apm-config-sourcemapping-elasticsearch] Configure the {{es}} source map retrieval location, taking the same options as [output.elasticsearch](configure-elasticsearch-output.md). This must be set when using an output other than {{es}}, and that output is writing to {{es}}. Otherwise leave this section empty. - ### `source_mapping.cache.expiration` [apm-rum-sourcemap-cache] If a source map has been uploaded to the APM Server, [source mapping](create-upload-source-maps-rum.md) is automatically applied to documents sent to the RUM endpoint. Source maps are fetched from {{es}} and then kept in an in-memory cache for the configured time. Values configured without a time unit are treated as seconds. Default: `5m` (5 minutes) - ### `source_mapping.index_pattern` [_source_mapping_index_pattern] Previous versions of APM Server stored source maps in `apm-%{[observer.version]}-sourcemap` indices. Search source maps stored in an older version with this setting. Default: `"apm-*-sourcemap*"` - ## Ingest pipelines [_ingest_pipelines] The default APM Server pipeline includes processors that enrich RUM data prior to indexing in {{es}}. See [Parse data using ingest pipelines](parse-data-using-ingest-pipelines.md) for details on how to locate, edit, or disable this preprocessing. diff --git a/solutions/observability/apps/configure-redis-output.md b/solutions/observability/apps/configure-redis-output.md index 6df3d2a89a..a0c09abf65 100644 --- a/solutions/observability/apps/configure-redis-output.md +++ b/solutions/observability/apps/configure-redis-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the Redis output [apm-redis-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ The Redis output is not yet supported by {{fleet}}-managed APM Server. :::: - The Redis output inserts the events into a Redis list or a Redis channel. This output plugin is compatible with the [Redis input plugin](logstash-docs-md://lsr/plugins-inputs-redis.md) for {{ls}}. To use this output, edit the APM Server configuration file to disable the {{es}} output by commenting it out, and enable the Redis output by adding `output.redis`. @@ -32,34 +30,28 @@ output.redis: timeout: 5 ``` - ## Compatibility [_compatibility_3] This output is expected to work with all Redis versions between 3.2.4 and 5.0.8. Other versions might work as well, but are not supported. - ## Configuration options [_configuration_options_6] You can specify the following `output.redis` options in the `apm-server.yml` config file: - ### `enabled` [_enabled_5] The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled. The default value is `true`. - ### `hosts` [_hosts_3] The list of Redis servers to connect to. If load balancing is enabled, the events are distributed to the servers in the list. If one server becomes unreachable, the events are distributed to the reachable servers only. You can define each Redis server by specifying `HOST` or `HOST:PORT`. For example: `"192.15.3.2"` or `"test.redis.io:12345"`. If you don’t specify a port number, the value configured by `port` is used. Configure each Redis server with an `IP:PORT` pair or with a `URL`. For example: `redis://localhost:6379` or `rediss://localhost:6379`. URLs can include a server-specific password. For example: `redis://:password@localhost:6379`. The `redis` scheme will disable the `ssl` settings for the host, while `rediss` will enforce TLS. If `rediss` is specified and no `ssl` settings are configured, the output uses the system certificate store. - ### `index` [_index] The index name added to the events metadata for use by {{ls}}. The default is "apm-server". - ### `key` [apm-key-option-redis] The name of the Redis list or channel the events are published to. If not configured, the value of the `index` setting is used. @@ -74,7 +66,6 @@ output.redis: See the [`keys`](#apm-keys-option-redis) setting for other ways to set the key dynamically. - ### `keys` [apm-keys-option-redis] An array of key selector rules. Each rule specifies the `key` to use for events that match the rule. During publishing, APM Server uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the `keys` setting is missing or no rule matches, the [`key`](#apm-key-option-redis) setting is used. @@ -113,54 +104,44 @@ output.redis: mysql: "backend_list" ``` - ### `password` [_password_3] The password to authenticate with. The default is no authentication. - ### `db` [_db] The Redis database number where the events are published. The default is 0. - ### `datatype` [_datatype] The Redis data type to use for publishing events.If the data type is `list`, the Redis RPUSH command is used and all events are added to the list with the key defined under `key`. If the data type `channel` is used, the Redis `PUBLISH` command is used and means that all events are pushed to the pub/sub mechanism of Redis. The name of the channel is the one defined under `key`. The default value is `list`. - ### `codec` [_codec_2] Output codec configuration. If the `codec` section is missing, events will be JSON encoded. See [Change the output codec](configure-console-output.md#apm-configuration-output-codec) for more information. - ### `worker` [_worker_3] The number of workers to use for each host configured to publish events to Redis. Use this setting along with the `loadbalance` option. For example, if you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host). - ### `loadbalance` [_loadbalance] If set to true and multiple hosts or workers are configured, the output plugin load balances published events onto all Redis hosts. If set to false, the output plugin sends all events to only one host (determined at random) and will switch to another host if the currently selected one becomes unreachable. The default value is true. - ### `timeout` [_timeout_4] The Redis connection timeout in seconds. The default is 5 seconds. - ### `backoff.init` [_backoff_init_4] The number of seconds to wait before trying to reconnect to Redis after a network error. After waiting `backoff.init` seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is `1s`. - ### `backoff.max` [_backoff_max_4] The maximum number of seconds to wait before attempting to connect to Redis after a network error. The default is `60s`. - ### `max_retries` [_max_retries_4] The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. @@ -169,7 +150,6 @@ Set `max_retries` to a value less than 0 to retry until all events are published The default is 3. - ### `bulk_max_size` [_bulk_max_size_3] The maximum number of events to bulk in a single Redis request or pipeline. The default is 2048. @@ -180,12 +160,10 @@ Specifying a larger batch size can improve performance by lowering the overhead Setting `bulk_max_size` to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch. - ### `ssl` [_ssl_4] Configuration options for SSL parameters like the root CA for Redis connections guarded by SSL proxies (for example [stunnel](https://www.stunnel.org)). See [SSL/TLS output settings](ssltls-output-settings.md) for more information. - ### `proxy_url` [_proxy_url_3] The URL of the SOCKS5 proxy to use when connecting to the Redis servers. The value must be a URL with a scheme of `socks5://`. You cannot use a web proxy because the protocol used to communicate with Redis is not based on HTTP. @@ -194,7 +172,6 @@ If the SOCKS5 proxy server requires client authentication, you can embed a usern When using a proxy, hostnames are resolved on the proxy server instead of on the client. You can change this behavior by setting the [`proxy_use_local_resolver`](#apm-redis-proxy-use-local-resolver) option. - ### `proxy_use_local_resolver` [apm-redis-proxy-use-local-resolver] This option determines whether Redis hostnames are resolved locally when using a proxy. The default value is false, which means that name resolution occurs on the proxy server. diff --git a/solutions/observability/apps/configure-settings.md b/solutions/observability/apps/configure-settings.md index dd2e930f2e..05abcd98de 100644 --- a/solutions/observability/apps/configure-settings.md +++ b/solutions/observability/apps/configure-settings.md @@ -26,8 +26,6 @@ Uptime settings apply to the current space only. To segment different uptime use :::: - - ## Configure indices [configure-uptime-indices] Specify a comma-separated list of index patterns to match indices in {{es}} that contain {{heartbeat}} data. @@ -37,13 +35,11 @@ The pattern set here only restricts what the {{uptime-app}} displays. You can st :::: - :::{image} /solutions/images/observability-heartbeat-indices.png :alt: {{heartbeat}} indices :screenshot: ::: - ## Configure connectors [configure-uptime-alert-connectors] **Alerts** work by running checks on a schedule to detect conditions defined by a rule. When a condition is met, the rule tracks it as an **alert** and responds by triggering one or more **actions**. Actions typically involve interaction with {{kib}} services or third party integrations. **Connectors** allow actions to talk to these services and integrations. @@ -57,7 +53,6 @@ For more information about each connector, see [action types and connectors](../ :screenshot: ::: - ## Configure certificate thresholds [configure-cert-thresholds] You can modify certificate thresholds to control how Uptime displays your TLS values in the [TLS Certificates](uptime-monitoring-deprecated.md#view-certificate-status) page. These settings also determine which certificates are selected by any TLS rule you create. diff --git a/solutions/observability/apps/configure-synthetics-projects.md b/solutions/observability/apps/configure-synthetics-projects.md index 0eeee3772f..7f5fe20313 100644 --- a/solutions/observability/apps/configure-synthetics-projects.md +++ b/solutions/observability/apps/configure-synthetics-projects.md @@ -103,15 +103,12 @@ export default env => { :::: - The configuration file can either export an object, or a function that when called should return the generated configuration. To know more about configuring the tests based on environments, look at the [dynamic configuration](../../../solutions/observability/apps/work-with-params-secrets.md#synthetics-dynamic-configs) documentation. - ## `params` [synthetics-configuration-params] A JSON object that defines any variables your tests require. Read more in [Work with params and secrets](../../../solutions/observability/apps/work-with-params-secrets.md). - ## `playwrightOptions` [synthetics-configuration-playwright-options] For all available options, refer to the [Playwright documentation](https://playwright.dev/docs/test-configuration). @@ -121,10 +118,8 @@ Do not attempt to run in headful mode (using `headless:false`) when running thro :::: - Below are details on a few Playwright options that are particularly relevant to Elastic Synthetics including TLS client authentication, timeouts, timezones, and device emulation. - ### TLS client authentication [synthetics-configuration-playwright-options-client-certificates] To enable TLS client authentication, set the [`clientCertificates`](https://playwright.dev/docs/api/class-testoptions#test-options-client-certificates) option in the configuration file: @@ -134,7 +129,6 @@ Path-based options `{certPath, keyPath, pfxPath}` are only supported on Private :::: - ```js playwrightOptions: { clientCertificates: [ @@ -168,15 +162,12 @@ clientCertificates: [ :::: - - ### Timeouts [synthetics-configuration-playwright-options-timeouts] Playwright has two types of timeouts that are used in Elastic Synthetics: [action and navigation timeouts](https://playwright.dev/docs/test-timeouts#action-and-navigation-timeouts). Elastic Synthetics uses a default action and navigation timeout of 50 seconds. You can override this default using [`actionTimeout`](https://playwright.dev/docs/api/class-testoptions#test-options-action-timeout) and [`navigationTimeout`](https://playwright.dev/docs/api/class-testoptions#test-options-navigation-timeout) in `playwrightOptions`. - ### Timezones and locales [synthetics-configuration-playwright-options-timezones] The Elastic global managed testing infrastructure does not currently set the timezone. For {{private-location}}s, the monitors will use the timezone of the host machine running the {{agent}}. This is not always desirable if you want to test how a web application behaves across different timezones. To specify what timezone to use when the monitor runs, you can use `playwrightOptions` on a per monitor or global basis. @@ -192,7 +183,6 @@ playwrightOptions: { To use a timezone and/or locale for a *specific* monitor, add these options to a journey using [`monitor.use`](../../../solutions/observability/apps/configure-individual-browser-monitors.md). - ### Device emulation [synthetics-config-device-emulation] Users can emulate a mobile device using the configuration file. The example configuration below runs tests in "Pixel 5" emulation mode. @@ -210,7 +200,6 @@ const config: SyntheticsConfig = { export default config; ``` - ## `project` [synthetics-configuration-project] Information about the Synthetics project. @@ -220,14 +209,12 @@ Information about the Synthetics project. If you used [`init` to create a Synthetics project](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-init-command), this is the `` you specified. - `url` (`string`) : The URL for the Serverless Observability project or the {{kib}} URL for the deployment to which you want to upload the monitors. space (`string`) : For {{kib}} deployments 9.0 or higher, the identifier of the target [{{kib}} space](../../../deploy-manage/manage-spaces.md) for the pushed monitors. Spaces help you organize pushed monitors. Pushes to the "default" space if not specified. - ## `monitor` [synthetics-configuration-monitor] Default values to be applied to *all* monitors when using the [`@elastic/synthetics` `push` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). @@ -255,7 +242,6 @@ $$$synthetics-configuration-monitor-tags$$$ `tags` (`Array`) * Run the [`elastic-synthetics locations` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). * Find `Synthetics` in the [global search field](../../../get-started/the-stack.md#kibana-navigation-search) and click **Create monitor**. Locations will be listed in *Locations*. - `privateLocations` (`Array`) : The [{{private-location}}s](../../../solutions/observability/apps/monitor-resources-on-private-networks.md) to which the monitors will be deployed. These {{private-location}}s refer to locations hosted and managed by you, whereas `locations` are hosted by Elastic. You can specify a {{private-location}} using the location’s name. @@ -264,7 +250,6 @@ $$$synthetics-configuration-monitor-tags$$$ `tags` (`Array`) * Run the [`elastic-synthetics locations` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) with the URL for the Observability project or the {{kib}} URL for the deployment from which to fetch available locations. * Find `Synthetics` in the [global search field](../../../get-started/the-stack.md#kibana-navigation-search) and click **Create monitor**. {{private-location}}s will be listed in *Locations*. - `throttling` (`boolean` | [`ThrottlingOptions`](https://github.com/elastic/synthetics/blob/v1.3.0/src/common_types.ts#L194-L198)) : Control the monitor’s download speeds, upload speeds, and latency to simulate your application’s behavior on slower or laggier networks. Set to `false` to disable throttling altogether. @@ -281,7 +266,6 @@ $$$synthetics-configuration-monitor-tags$$$ `tags` (`Array`) Using `retestOnFailure` can reduce noise related to transient problems. - `fields` (`object`) : A list of key-value pairs that will be sent with each monitor event. The `fields` are appended to {{es}} documents as `labels`, and those labels are displayed in {{kib}} in the *Monitor details* panel in the [individual monitor’s *Overview* tab](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-individual-monitors-overview). @@ -294,7 +278,6 @@ $$$synthetics-configuration-monitor-tags$$$ `tags` (`Array`) } ``` - For information on configuring monitors individually, refer to: * [Configure individual browser monitors](../../../solutions/observability/apps/configure-individual-browser-monitors.md) for browser monitors diff --git a/solutions/observability/apps/configure-synthetics-settings.md b/solutions/observability/apps/configure-synthetics-settings.md index 50f0633fbf..0f8caaf025 100644 --- a/solutions/observability/apps/configure-synthetics-settings.md +++ b/solutions/observability/apps/configure-synthetics-settings.md @@ -11,7 +11,6 @@ applies_to: There are several Synthetics settings you can adjust in Observability. - ## Alerting [synthetics-settings-alerting] Alerting enables you to detect complex conditions using **rules** across Observability and send a notification using **connectors**. @@ -24,7 +23,6 @@ When you create a new synthetic monitor, new default synthetics rules will be ap * **Monitor status rule** for receiving notifications for errors and outages. * **TLS certificate rule** for receiving notifications when one or more of your HTTP or TCP lightweight monitors has a TLS certificate expiring within a specified threshold or when it exceeds an age limit. - However, the automatically created Synthetics internal alert is intentionally preconfigured, and some configuration options can’t be changed. For example, you can’t change how often it checks the rule. If you need specific alerting behavior, set up a different rule. To view all existing rules or create a new rule: @@ -48,7 +46,6 @@ You can enable and disable default alerts for individual monitors in a few ways: :::: - In the **Alerting** tab on the Synthetics Settings page, you can add and configure connectors. If you are running in Elastic Cloud, then an SMTP connector will automatically be configured, allowing you to easily set up email alerts. Read more about all available connectors in [Action types](../../../solutions/observability/incident-management/create-an-apm-anomaly-rule.md). :::{image} /solutions/images/observability-synthetics-settings-alerting.png @@ -56,7 +53,6 @@ In the **Alerting** tab on the Synthetics Settings page, you can add and configu :screenshot: ::: - ## {{private-location}}s [synthetics-settings-private-locations] {{private-location}}s allow you to run monitors from your own premises. @@ -68,7 +64,6 @@ In the **{{private-location}}s** tab, you can add and manage {{private-location} :screenshot: ::: - ## Global parameters [synthetics-settings-global-parameters] Global parameters can be defined once and used across the configuration of lightweight and browser-based monitors. @@ -80,7 +75,6 @@ In the **Global parameters** tab, you can define variables and parameters. This :screenshot: ::: - ## Data retention [synthetics-settings-data-retention] When you set up a synthetic monitor, data from the monitor is saved in [Elasticsearch data streams](../../../manage-data/data-store/data-streams.md), an append-only structure in Elasticsearch. You can customize how long synthetics data is stored by creating your own index lifecycle policy and attaching it to the relevant custom Component Template in Stack Management. @@ -92,7 +86,6 @@ In the **Data retention** tab, use the links to jump to the relevant policy for :screenshot: ::: - ## Project API keys [synthetics-settings-api-keys] Project API keys are used to push {{project-monitors}} remotely from a CLI or CD pipeline. @@ -106,7 +99,6 @@ In a serverless project, to create a Project API key you must be logged in as a :::: - :::{image} /solutions/images/observability-synthetics-settings-api-keys.png :alt: Project API keys tab on the Synthetics Settings page in {kib} :screenshot: diff --git a/solutions/observability/apps/control-access-to-apm-data.md b/solutions/observability/apps/control-access-to-apm-data.md index 550fa9bfde..c0c6811a3a 100644 --- a/solutions/observability/apps/control-access-to-apm-data.md +++ b/solutions/observability/apps/control-access-to-apm-data.md @@ -13,7 +13,6 @@ To take advantage of this feature, your APM data needs to be written to differen Multiple APM integration instances is not required though. The simplest way to take advantage of this feature is by creating filtered aliases. See the guide below for more information. - ## Guide: Separate staging and production data [apm-spaces-example] This guide will explain how to separate your staging and production data. This can be helpful to either remove noise when troubleshooting a production issue, or to create more granular access control for certain data. @@ -23,7 +22,6 @@ This guide assumes that you: * Are sending both staging and production APM data to an {{es}} cluster. * Have configured the `environment` variable in your APM agent configurations. This variable sets the `service.environment` field in APM documents. You should have documents where `service.environment: production` and `service.environment: staging`. If this field is empty, see [service environment filter](filter-application-data.md#apm-filter-your-data-service-environment-filter) to learn how to set this value. - ### Step 1: Create filtered aliases [_step_1_create_filtered_aliases] The Applications UI uses index patterns to query your APM data. An index pattern can match data streams, indices, and/or aliases. The default values are: @@ -38,7 +36,6 @@ The Applications UI uses index patterns to query your APM data. An index pattern The default index settings also query the `apm-*` data view. This data view matches APM data shipped in earlier versions of APM (prior to v8.0). :::: - Instead of querying the default APM data views, we can create filtered aliases for the Applications UI to query. A filtered alias is a secondary name for a group of data streams that has a user-defined filter to limit the documents that the alias can access. To separate `staging` and `production` APM data, we’d need to create six filtered aliases—​three aliases for each service environment: @@ -79,10 +76,8 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`traces-apm*,apm-*`) 3. Only match documents where `service.environment: production` - :::: - ::::{dropdown} `logs-apm*` production alias example ```console POST /_aliases?pretty @@ -109,10 +104,8 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`logs-apm*,apm-*`) 3. Only match documents where `service.environment: production` - :::: - ::::{dropdown} `metrics-apm*` production alias example ```console POST /_aliases?pretty @@ -139,10 +132,8 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`metrics-apm*,apm-*`) 3. Only match documents where `service.environment: production` - :::: - ::::{dropdown} `traces-apm*` staging alias example ```console POST /_aliases?pretty @@ -169,10 +160,8 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`traces-apm*,apm-*`) 3. Only match documents where `service.environment: staging` - :::: - ::::{dropdown} `logs-apm*` staging alias example ```console POST /_aliases?pretty @@ -199,10 +188,8 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`logs-apm*,apm-*`) 3. Only match documents where `service.environment: staging` - :::: - ::::{dropdown} `metrics-apm*` staging alias example ```console POST /_aliases?pretty @@ -229,18 +216,14 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`metrics-apm*,apm-*`) 3. Only match documents where `service.environment: staging` - :::: - - ### Step 2: Create {{kib}} spaces [_step_2_create_kib_spaces] Next, you’ll need to create a {{kib}} space for each service environment. To open **Spaces**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). To create a new space, click **Create a space**. For this guide, we’ve created two Kibana spaces, one named `production` and one named `staging`. See [Kibana spaces](../../../deploy-manage/manage-spaces.md) for more information on creating a space. - ### Step 3: Update APM index settings in each space [_step_3_update_apm_index_settings_in_each_space] Now we can change the default data views that the Applications UI queries in each space. @@ -254,7 +237,6 @@ Open the Applications UI and navigate to **Settings** → **Indices**. Use the t | Transaction indices | `production-traces-apm` | `staging-traces-apm` | | Metrics indices | `production-metrics-apm` | `staging-metrics-apm` | - ### Step 4: Create {{kib}} access roles [_step_4_create_kib_access_roles] To open **Roles**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Click **Create role**. @@ -306,10 +288,8 @@ POST /_security/role/production_apm_viewer 2. Assigns `read` privileges for the Applications and User Experience UIs. 3. Provides access to the space named `production`. - :::: - ::::{dropdown} Create a `staging_apm_viewer` role This request creates a `staging_apm_viewer` role: @@ -341,11 +321,8 @@ POST /_security/role/staging_apm_viewer 2. Assigns `read` privileges for the Applications and User Experience UIs. 3. Provides access to the space named `staging`. - :::: - - ### Step 5: Assign users to roles [_step_5_assign_users_to_roles] The last thing to do is assign users to the newly created roles above. Users will only have access to the data within the spaces that they are granted. @@ -368,7 +345,6 @@ POST /_security/user/production-apm-user 1. Assigns the previously created `production_apm_viewer` role. - This example creates a new user and assigns them the `staging_apm_viewer` role created in the previous step. This user will only have access to the staging space and data with a `service.environment` of `staging`. Remember to change the `password`, `full_name`, and `email` fields. ```console @@ -383,8 +359,6 @@ POST /_security/user/staging-apm-user 1. Assigns the previously created `staging_apm_viewer` role. - - ### Step 6: Marvel [_step_6_marvel] That’s it! Head back to the Applications UI and marvel at your space-specific data. diff --git a/solutions/observability/apps/create-apm-rules-alerts.md b/solutions/observability/apps/create-apm-rules-alerts.md index 3756d7cd8d..6f03509080 100644 --- a/solutions/observability/apps/create-apm-rules-alerts.md +++ b/solutions/observability/apps/create-apm-rules-alerts.md @@ -10,10 +10,8 @@ applies_to: # Create APM rules and alerts [apm-alerts] - The Applications UI allows you to define **rules** to detect complex conditions within your APM data and trigger built-in **actions** when those conditions are met. - ## APM rules [apm_rules] The following APM rules are supported: @@ -37,12 +35,10 @@ stack: all View and manage rules and alerts in the Applications UI. - ### View active alerts [apm-alert-view-active] Active alerts are displayed and grouped in multiple ways in the Applications UI. - #### View alerts by service group [apm-alert-view-group] If you’re using the [service groups](../../../solutions/observability/apps/services.md#service-groups) feature, you can view alerts by service group. From the service group overview page, click the red alert indicator to open the **Alerts** tab with a predefined filter that matches the filter used when creating the service group. @@ -52,7 +48,6 @@ If you’re using the [service groups](../../../solutions/observability/apps/ser :screenshot: ::: - #### View alerts by service [apm-alert-view-service] Alerts can be viewed within the context of any service. After selecting a service, go to the **Alerts** tab to view any alerts that are active for the selected service. @@ -62,12 +57,10 @@ Alerts can be viewed within the context of any service. After selecting a servic :screenshot: ::: - ### Manage alerts and rules [apm-alert-manage] From the Applications UI, select **Alerts and rules** → **Manage rules** to be taken to the {{kib}} **{{rules-ui}}** page. From this page, you can disable, mute, and delete APM alerts. - ### More information [apm-alert-more-info] See [Alerting](../../../explore-analyze/alerts-cases.md) for more information. diff --git a/solutions/observability/apps/create-assign-feature-roles-to-apm-server-users.md b/solutions/observability/apps/create-assign-feature-roles-to-apm-server-users.md index f13a4efadb..33bfb8ef68 100644 --- a/solutions/observability/apps/create-assign-feature-roles-to-apm-server-users.md +++ b/solutions/observability/apps/create-assign-feature-roles-to-apm-server-users.md @@ -6,17 +6,13 @@ applies_to: stack: --- - - # Create and assign feature roles to APM Server users [apm-feature-roles] - ::::{note} Kibana custom roles are *not* compatible with [{{serverless-full}}](https://docs.elastic.co/serverless). :::: - Manage access on a feature-by-feature basis by creating several custom feature-related *roles* and assigning one or more of these roles to each *user or group* based on which features they need to access. ::::{tip} @@ -30,7 +26,6 @@ In general, there are three types of privileges you’ll work with when creating :::: - The following are common roles that APM Server users might need: * [**Writer role**](#apm-privileges-to-publish-events): Allows a user to publish events collected by APM Server, which is **required** to write to {{es}}. @@ -47,8 +42,6 @@ If you want to create an APM Server user who can use the Elastic APM Real User M :::: - - ## Create a *writer* role [apm-privileges-to-publish-events] APM users that publish events to {{es}} *must* have privileges to write to APM data streams. @@ -58,7 +51,6 @@ This is not needed when APM Server doesn’t write to {{es}} directly. For examp :::: - To grant an APM Server user the required privileges for writing events to {{es}}: 1. Create a **general writer role**, called something like `apm_writer`, that has the following privileges: @@ -69,12 +61,10 @@ To grant an APM Server user the required privileges for writing events to {{es}} | Index | `create_doc` on `traces-apm*`, `logs-apm*`, and `metrics-apm*` indices | Write events into {{es}} | | Cluster | `monitor` | Allows cluster UUID checks, which are performed as part of APM server startup preconditionsif [Elasticsearch security](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md) is enabled (it is enabled by default), and allows a license check, which is required if [tail-based sampling](transaction-sampling.md#apm-tail-based-sampling) is enabled. | - ::::{note} If you have explicitly disabled Elastic security *and* you are *not* using tail-based sampling, the `monitor` privilege may not be necessary. :::: - 1. Assign the **general writer role** to APM Server users who need to publish APM data. ::::{note} @@ -82,8 +72,6 @@ Assign additional APM feature roles to users as needed including the *Central co :::: - - ## Create a *central configuration management* role [apm-privileges-agent-central-config] ::::{important} @@ -93,7 +81,6 @@ The privileges included in this role are **required** for all users when [centra :::: - $$$apm-privileges-agent-central-config-server$$$ APM Server acts as a proxy between your APM agents and the Applications UI. The Applications UI communicates any changed settings to APM Server so that your agents only need to poll the Server to determine which central configuration settings have changed. @@ -124,13 +111,10 @@ Assign additional APM feature roles to users as needed including the *Writer rol :::: - ::::{tip} Looking for privileges and roles needed to use central configuration from the Applications UI or APM UI API? See [Applications UI central configuration user](applications-ui-central-config-user.md). :::: - - ## Create a *monitoring* role [apm-privileges-to-publish-monitoring] {{es-security-features}} provides built-in users and roles for publishing and viewing monitoring data. The privileges and roles needed to publish monitoring data depend on the method used to collect that data. @@ -142,7 +126,6 @@ Looking for privileges and roles needed to use central configuration from the Ap * [View monitoring data](#apm-privileges-to-publish-monitoring-view) - ### Publish monitoring data [apm-privileges-to-publish-monitoring-write] ::::{important} @@ -150,8 +133,6 @@ Looking for privileges and roles needed to use central configuration from the Ap :::: - - #### Internal collection [apm-privileges-to-publish-monitoring-internal] If you’re using [internal collection](use-internal-collection-to-send-monitoring-data.md) to collect metrics about APM Server, either: @@ -183,15 +164,12 @@ Assign additional APM feature roles to users as needed including the [*Writer ro :::: - - #### {{metricbeat}} collection [apm-privileges-to-publish-monitoring-metricbeat] ::::{note} When using {{metricbeat}} to collect metrics, no roles or users need to be created with APM Server. See [Use {{metricbeat}} collection](use-metricbeat-to-send-monitoring-data.md) for complete details on setting up {{metricbeat}} collection. :::: - If you’re [using {{metricbeat}}](use-metricbeat-to-send-monitoring-data.md) to collect metrics about APM Server, you can either: * Use the built-in `remote_monitoring_user` user or role @@ -214,14 +192,11 @@ If you don’t use the `remote_monitoring_user` user, you can create a custom us | `remote_monitoring_collector` | Collect monitoring metrics from APM Server | | `remote_monitoring_agent` | Send monitoring data to the monitoring cluster | - ::::{note} Assign additional APM feature roles to users as needed including the [*Writer role*](#apm-privileges-to-publish-events) and [*Central configuration management role*](#apm-central-config-role-note), both of which are required in most cases. :::: - - ### View monitoring data [apm-privileges-to-publish-monitoring-view] To grant users the required privileges for viewing monitoring data: @@ -239,14 +214,11 @@ To grant users the required privileges for viewing monitoring data: | --- | --- | | `monitoring_user` | Grants access to monitoring indices for APM Server | - ::::{note} Assign additional APM feature roles to users as needed including the [*Writer role*](#apm-privileges-to-publish-events) and [*Central configuration management role*](#apm-central-config-role-note), both of which are required in most cases. :::: - - ## Create a *source map* role [apm-privileges-rum-source-map] $$$apm-privileges-rum-source-mapping$$$ @@ -263,5 +235,4 @@ Assign additional APM feature roles to users as needed including the [*Writer ro :::: - The previous privileges should be sufficient for RUM source mapping to work properly as long as APM Server communicates with {{es}} successfully. If it fails, it may fallback to read source maps through {{kib}} if configured, which requires additional {{kib}} privileges. For more details, refer to the [{{stack}}](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-sourcemaps) or [{{serverless-short}}](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-apm-sourcemaps) API documentation. \ No newline at end of file diff --git a/solutions/observability/apps/create-custom-links.md b/solutions/observability/apps/create-custom-links.md index eddf88c559..f31421860b 100644 --- a/solutions/observability/apps/create-custom-links.md +++ b/solutions/observability/apps/create-custom-links.md @@ -16,19 +16,16 @@ applies_to: :::: - Elastic’s custom link feature allows you to easily create up to 500 dynamic links based on your specific APM data. Custom links can be filtered to only appear in the Applications UI for relevant services, environments, transaction types, or transaction names. Ready to dive in? Jump straight to the [examples](#custom-links-examples). - ## Create a link [custom-links-create] Each custom link consists of a label, URL, and optional filter. The easiest way to create a custom link is from within the actions dropdown in the transaction detail page. This method will automatically apply filters, scoping the link to that specific service, environment, transaction type, and transaction name. Alternatively, you can create a custom link in the Applications UI by navigating to **Settings** > **Customize UI**, and selecting **Create custom link**. - ### Label [custom-links-label] The name of your custom link. The actions context menu displays this text, so keep it as short as possible. @@ -37,8 +34,6 @@ The name of your custom link. The actions context menu displays this text, so ke Custom links are displayed alphabetically in the actions menu. :::: - - ### URL [custom-links-url] The URL your link points to. URLs support dynamic field name variables, encapsulated in double curly brackets: `{{field.name}}`. These variables will be replaced with transaction metadata when the link is clicked. @@ -50,7 +45,6 @@ Because everyone’s data is different, you’ll need to examine your traces to :screenshot: ::: - ### Filters [custom-links-filters] Filter each link to only appear for specific services or transactions. You can filter on the following fields: @@ -62,12 +56,10 @@ Filter each link to only appear for specific services or transactions. You can f Multiple values are allowed when comma-separated. - ## Custom link examples [custom-links-examples] Not sure where to start with custom links? Take a look at the examples below and customize them to your liking! - ### Email [custom-links-examples-email] Email the owner of a service. @@ -88,7 +80,6 @@ This link opens an email addressed to the team or owner of `python-backend`. It | Link | `mailto:python_team@elastic.co` | | Filters | `service.name:python-backend` | - ### GitHub issue [custom-links-examples-gh] Open a GitHub issue with pre-populated metadata from the selected trace sample. @@ -116,7 +107,6 @@ This link opens a new GitHub issue in the apm-agent-rum repository. It populates See the [GitHub automation documentation](https://help.github.com/en/github/managing-your-work-on-github/about-automation-for-issues-and-pull-requests-with-query-parameters) for a full list of supported query parameters. - ### Jira task [custom-links-examples-jira] Create a Jira task with pre-populated metadata from the selected trace sample. @@ -142,7 +132,6 @@ This link creates a new task on the Engineering board in Jira. It populates the See the [Jira application administration knowledge base](https://confluence.atlassian.com/jirakb/how-to-create-issues-using-direct-html-links-in-jira-server-159474.html) for a full list of supported query parameters. - ### Dashboards [custom-links-examples-kib] Link to a custom dashboard. @@ -161,7 +150,6 @@ This link opens the current `transaction.id` in a custom dashboard. There are no | Label | `Open transaction in Python drilldown viz` | | URL | `https://kibana-instance/app/kibana#/dashboard?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),panels:!((embeddableConfig:(),gridData:(h:15,i:cb79c1c0-1af8-472c-aaf7-d158a76946fb,w:24,x:0,y:0),id:c8c74b20-6a30-11ea-92ab-b5d3feff11df,panelIndex:cb79c1c0-1af8-472c-aaf7-d158a76946fb,type:visualization,version:'7.7')),query:(language:kuery,query:'transaction.id:{{transaction.id}}'),timeRestore:!f,title:'',viewMode:edit)` | - ### Slack channel [custom-links-examples-slack] Open a specified slack channel. @@ -182,7 +170,6 @@ This link opens a company slack channel, #apm-user-support. It only appears when | Link | `https://COMPANY_SLACK.slack.com/archives/efk52kt23k` | | Filters | `transaction.name:GET user/login` | - ### Website [custom-links-examples-web] Open an internal or external website. diff --git a/solutions/observability/apps/create-monitors-in-synthetics-app.md b/solutions/observability/apps/create-monitors-in-synthetics-app.md index 20032ad0cc..6e80d4c723 100644 --- a/solutions/observability/apps/create-monitors-in-synthetics-app.md +++ b/solutions/observability/apps/create-monitors-in-synthetics-app.md @@ -18,7 +18,6 @@ You can create synthetic monitors directly in the UI by opening an Observability This is one of [two approaches](../../../solutions/observability/apps/get-started.md) you can use to set up a synthetic monitor. - ## Prerequisites [synthetics-get-started-ui-prerequisites] For **serverless Observability projects**, you must be signed in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access. @@ -32,7 +31,6 @@ You should decide where you want to run the monitors before getting started. You Executing synthetic tests on Elastic’s global managed testing infrastructure incurs an additional charge. Tests are charged under one of two new billing dimensions depending on the monitor type. For *browser monitor* usage, there is a fee per test run. For *lightweight monitor* usage, there is a fee per region in which you run any monitors regardless of the number of test runs. For more details, refer to the [{{obs-serverless}} pricing page](https://www.elastic.co/pricing/serverless-observability). - ## Add a lightweight monitor [synthetics-get-started-ui-add-a-lightweight-monitor] To use the UI to add a lightweight monitor: @@ -47,7 +45,6 @@ To use the UI to add a lightweight monitor: :::: - :::::{note} If you’ve [added a {{private-location}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md), you’ll see your the {{private-location}} in the list of *Locations*. @@ -68,8 +65,6 @@ To use the UI to add a lightweight monitor: :screenshot: ::: - - ## Add a browser monitor [synthetics-get-started-ui-add-a-browser-monitor] You can also create a browser monitor in the UI using an **Inline script**. @@ -110,7 +105,6 @@ To use the UI to add a browser monitor: 7. (Optional) Click **Run test** to verify that the test is valid. 8. Click **Create monitor**. - ## View in your Observability project [synthetics-get-started-ui-view-in-your-observability-project] Navigate to **Synthetics**, where you can see screenshots of each run, set up alerts in case of test failures, and more. @@ -122,7 +116,6 @@ When a monitor is created or updated, the first run might not occur immediately, :::: - ## Next steps [synthetics-get-started-ui-next-steps] Learn more about: diff --git a/solutions/observability/apps/create-monitors-with-project-monitors.md b/solutions/observability/apps/create-monitors-with-project-monitors.md index 6e8530b3a7..4ec0f4001c 100644 --- a/solutions/observability/apps/create-monitors-with-project-monitors.md +++ b/solutions/observability/apps/create-monitors-with-project-monitors.md @@ -10,7 +10,6 @@ applies_to: # Create monitors with a Synthetics project [observability-synthetics-get-started-project] - A Synthetics project is the most powerful and sophisticated way to configure synthetic monitors. A Synthetics project lets you define your infrastructure as code, more commonly known as IaaC or Git-ops. With monitors created and managed in Synthetics projects, you organize your YAML configuration and JavaScript- or TypeScript-defined monitors on the filesystem, use Git for version control, and deploy via a CLI tool, usually executed on a CI/CD platform. :::{image} /solutions/images/observability-synthetics-get-started-projects.png @@ -19,7 +18,6 @@ A Synthetics project is the most powerful and sophisticated way to configure syn This is one of [two approaches](../../../solutions/observability/apps/get-started.md) you can use to set up a synthetic monitor. - ## Prerequisites [synthetics-get-started-project-prerequisites] For **serverless Observability projects**, you must be signed in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access. @@ -41,7 +39,6 @@ Working with a Synthetics project requires working with the Elastic Synthetics C npx @elastic/synthetics -h ``` - You should also decide where you want to run the monitors before getting started. You can run monitors in Synthetics projects on one or both of the following: * **Elastic’s global managed testing infrastructure**: With Elastic’s global managed testing infrastructure, you can create and run monitors in multiple locations without having to manage your own infrastructure. Elastic takes care of software updates and capacity planning for you. @@ -54,7 +51,6 @@ If you are setting up Synthetics for a deployment configured with [traffic filte :::: - ## Create a Synthetics project [synthetics-get-started-project-create-a-synthetics-project] Start by creating your first Synthetics project. Run the command below to create a new Synthetics project named `synthetic-project-test` in the current directory. @@ -83,7 +79,6 @@ Then, follow the prompts on screen to set up the correct default variables for y :::: - :::{image} /solutions/images/serverless-synthetics-monitor-management-api-key.png :alt: Project API Keys tab in Synthetics settings :screenshot: @@ -126,7 +121,6 @@ Then, take a look at key files and directories inside your Synthetics project: :::: - :::{image} /solutions/images/serverless-synthetics-monitor-management-api-key.png :alt: Project API Keys tab in Synthetics settings :screenshot: @@ -197,7 +191,6 @@ journey('My Example Journey', ({ page, params }) => { For more details on writing journeys and configuring browser monitors, refer to [Scripting browser monitors](../../../solutions/observability/apps/scripting-browser-monitors.md). - ## Test and connect to your Observability project or Elastic Stack deployment[synthetics-get-started-project-test-and-connect-to-your-observability-project] ::::{tab-set} @@ -220,7 +213,6 @@ While inside the project directory you can do two things with the `npx @elastic/ npx @elastic/synthetics push --auth $SYNTHETICS_API_KEY --url ``` - One monitor will appear in the {{synthetics-app}} for each journey or lightweight monitor, and you’ll manage all monitors from your local environment. For more details on using the `push` command, refer to [`@elastic/synthetics push`](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). ::::{note} @@ -249,7 +241,6 @@ While inside the Synthetics project directory you can do two things with the `np npx @elastic/synthetics push --auth $SYNTHETICS_API_KEY --url ``` - One monitor will appear in the Synthetics UI for each journey or lightweight monitor, and you’ll manage all monitors from your local environment. For more details on using the `push` command, refer to [`@elastic/synthetics push`](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). ::::{note} @@ -263,7 +254,6 @@ To list available {{private-location}}s, run the [`elastic-synthetics locations` :::: - ## View in the Synthetics UI [synthetics-get-started-project-view-in-your-observability-project] Then, go to **Synthetics** in your serverless Observability project or in {{kib}}. You should see your newly pushed monitors running. You can also go to the **Management** tab to see the monitors' configuration settings. @@ -273,8 +263,6 @@ When a monitor is created or updated, the first run might not occur immediately, :::: - - ## Next steps [observability-synthetics-get-started-project-next-steps] Learn more about: diff --git a/solutions/observability/apps/create-upload-source-maps-rum.md b/solutions/observability/apps/create-upload-source-maps-rum.md index 69525748cd..0959dfbc0b 100644 --- a/solutions/observability/apps/create-upload-source-maps-rum.md +++ b/solutions/observability/apps/create-upload-source-maps-rum.md @@ -31,7 +31,6 @@ Follow the steps below to enable source mapping your error stack traces in the A * [Generate a source map](#apm-source-map-rum-generate) * [Upload the source map](#apm-source-map-rum-upload) - ## Initialize the RUM Agent [apm-source-map-rum-initialize] Set the service name and version of your application when initializing the RUM Agent. To make uploading subsequent source maps easier, the `serviceVersion` you choose might be the `version` from your `package.json`. For example: @@ -55,7 +54,6 @@ const serviceVersion = git.short() It can also be any other unique string that indicates a specific version of your application. The APM integration uses the service name and version to match the correct source map file to each stack trace. - ## Generate a source map [apm-source-map-rum-generate] To be compatible with Elastic APM, source maps must follow the [source map revision 3 proposal spec](https://sourcemaps.info/spec.html). @@ -84,15 +82,12 @@ module.exports = { 1. If you’re using a different method of defining `serviceVersion`, you can set it here. - - ## Upload the source map [apm-source-map-rum-upload] ::::{tip} When uploading a source map, ensure that RUM support is enabled in the APM integration. :::: - {{kib}} exposes a source map endpoint for uploading source maps. Source maps can be uploaded as a string, or as a file upload. Let’s look at two different ways to upload a source map: curl and a custom application. Each example includes the four fields necessary for APM Server to later map minified code to its source: @@ -104,7 +99,6 @@ Let’s look at two different ways to upload a source map: curl and a custom app If you have multiple source maps, you’ll need to upload each individually. - ### Upload via curl [apm-source-map-curl] Here’s an example curl request that uploads the source map file created in the previous step. This request uses an API key for authentication. @@ -152,7 +146,6 @@ request.post({url: 'http://localhost:5601/api/apm/sourcemaps',formData: formData }) ``` - ## What happens next [apm-source-map-next] Source maps are stored in {{es}}. When you upload a source map, a new {{es}} document is created containing the contents of the source map. When a RUM request comes in, APM Server will make use of these source map documents to apply the source map logic to the event’s stack traces. diff --git a/solutions/observability/apps/cross-cluster-search-with-application-data.md b/solutions/observability/apps/cross-cluster-search-with-application-data.md index 9b44279032..bce33142ae 100644 --- a/solutions/observability/apps/cross-cluster-search-with-application-data.md +++ b/solutions/observability/apps/cross-cluster-search-with-application-data.md @@ -8,10 +8,8 @@ applies_to: # Cross-cluster search with application data [apm-cross-cluster-search] - Elastic APM utilizes {{es}}'s cross-cluster search functionality. Cross-cluster search lets you run a single search request against one or more [remote clusters](/deploy-manage/remote-clusters/remote-clusters-self-managed.md) — making it easy to search APM data across multiple sources. This means you can also have deployments per data type, making sizing and scaling more predictable, and allowing for better performance while managing multiple observability use cases. - ## Set up cross-cluster search [apm-set-up-cross-cluster-search] **Step 1. Set up remote clusters.** diff --git a/solutions/observability/apps/custom-filters.md b/solutions/observability/apps/custom-filters.md index d1037c0f61..aa861a1c7c 100644 --- a/solutions/observability/apps/custom-filters.md +++ b/solutions/observability/apps/custom-filters.md @@ -9,7 +9,6 @@ applies_to: Custom filters, including [ingest pipeline filters](#apm-filters-ingest-pipeline) and [APM agent filters](#apm-filters-in-agent), allow you to filter or redact APM data on ingestion. - ## Ingest pipeline filters [apm-filters-ingest-pipeline] Ingest pipelines specify a series of processors that transform data in a specific way. Transformation happens prior to indexing—​inflicting no performance overhead on the monitored application. Pipelines are a flexible and easy way to filter or obfuscate Elastic APM data. @@ -23,7 +22,6 @@ Features of this approach: For a step-by-step example, refer to [Tutorial: Use an ingest pipeline to redact sensitive information](#apm-filters-ingest-pipeline-tutorial). - ## APM agent filters [apm-filters-in-agent] Some APM agents offer a way to manipulate or drop APM events *before* they are sent to APM Server. @@ -41,7 +39,6 @@ Refer to the relevant agent’s documentation for more information and examples: * Python: [custom processors](apm-agent-python://reference/sanitizing-data.md). * Ruby: [`add_filter()`](apm-agent-ruby://reference/api-reference.md#api-agent-add-filter). - ## Tutorial: Use an ingest pipeline to redact sensitive information [apm-filters-ingest-pipeline-tutorial] Say you decide to [capture HTTP request bodies](built-in-data-filters.md#apm-filters-http-body) but quickly notice that sensitive information is being collected in the `http.request.body.original` field: @@ -55,14 +52,12 @@ Say you decide to [capture HTTP request bodies](built-in-data-filters.md#apm-fil To obfuscate the passwords stored in the request body, you can use a series of [ingest processors](elasticsearch://reference/enrich-processor/index.md). - ### Create a pipeline [_create_a_pipeline] ::::{tip} This tutorial uses the [Ingest APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ingest), but it’s also possible to create a pipeline using the UI. In Kibana, go to **Stack Management** → **Ingest Pipelines** → **Create pipeline** → **New pipeline** or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). :::: - To start, create a pipeline with a simple description and an empty array of processors: ```json @@ -76,8 +71,6 @@ To start, create a pipeline with a simple description and an empty array of proc 1. The processors defined below will go in this array - - #### Add a JSON processor [_add_a_json_processor] Add your first processor to the processors array. Because the agent captures the request body as a string, use the [JSON processor](elasticsearch://reference/enrich-processor/json-processor.md) to convert the original field value into a structured JSON object. Save this JSON object in a new field: @@ -92,7 +85,6 @@ Add your first processor to the processors array. Because the agent captures the } ``` - #### Add a set processor [_add_a_set_processor] If `body.original_json` is not `null`, i.e., it exists, we’ll redact the `password` with the [set processor](elasticsearch://reference/enrich-processor/set-processor.md), by setting the value of `body.original_json.password` to `"redacted"`: @@ -107,7 +99,6 @@ If `body.original_json` is not `null`, i.e., it exists, we’ll redact the `pass } ``` - #### Add a convert processor [_add_a_convert_processor] Use the [convert processor](elasticsearch://reference/enrich-processor/convert-processor.md) to convert the JSON value of `body.original_json` to a string and set it as the `body.original` value: @@ -124,7 +115,6 @@ Use the [convert processor](elasticsearch://reference/enrich-processor/convert-p } ``` - #### Add a remove processor [_add_a_remove_processor] Finally, use the [remove processor](elasticsearch://reference/enrich-processor/remove-processor.md) to remove the `body.original_json` field: @@ -139,7 +129,6 @@ Finally, use the [remove processor](elasticsearch://reference/enrich-processor/r } ``` - #### Register the pipeline [_register_the_pipeline] Then put it all together, and use the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) to register the new pipeline in {{es}}. Name the pipeline `apm_redacted_body_password`: @@ -183,7 +172,6 @@ PUT _ingest/pipeline/apm_redacted_body_password } ``` - ### Test the pipeline [_test_the_pipeline] Prior to enabling this new pipeline, you can test it with the [simulate pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-simulate). This API allows you to run multiple documents through a pipeline to ensure it is working correctly. @@ -229,7 +217,6 @@ POST _ingest/pipeline/apm_redacted_body_password/_simulate 2. This document only contains an unrelated field 3. This document contains invalid JSON - The API response should be similar to this: ```json @@ -277,7 +264,6 @@ The API response should be similar to this: As expected, only the first simulated document has a redacted password field. All other documents are unaffected. - ### Create a `@custom` pipeline [_create_a_custom_pipeline] The final step in this process is to call the newly created `apm_redacted_body_password` pipeline from the `@custom` pipeline of the data stream you wish to edit. @@ -314,10 +300,8 @@ PUT _ingest/pipeline/traces-apm@custom 1. The name of the pipeline we previously created - That’s it! Passwords will now be redacted from your APM HTTP body data. - ### Next steps [_next_steps] To learn more about ingest pipelines, see [View the {{es}} index template](view-elasticsearch-index-template.md). diff --git a/solutions/observability/apps/data-streams.md b/solutions/observability/apps/data-streams.md index e51db657bb..89cf3d4940 100644 --- a/solutions/observability/apps/data-streams.md +++ b/solutions/observability/apps/data-streams.md @@ -19,13 +19,10 @@ See the [{{fleet}} and {{agent}} Guide](/reference/fleet/data-streams.md) to lea :::: - - ## Data stream naming scheme [apm-data-streams-naming-scheme] APM data follows the `--` naming scheme. The `type` and `dataset` are predefined by the {{es}} apm-data plugin, but the `namespace` is your opportunity to customize how different types of data are stored in {{es}}. There is no recommendation for what to use as your namespace—​it is intentionally flexible. For example, you might create namespaces for each of your environments, like `dev`, `prod`, `production`, etc. Or, you might create namespaces that correspond to strategic business units within your organization. - ## APM data streams [apm-data-streams-list] By type, the APM data streams are: @@ -36,7 +33,6 @@ Traces * Application traces: `traces-apm-` * RUM and iOS agent application traces: `traces-apm.rum-` - Metrics : Metrics include application-based metrics, aggregation metrics, and basic system metrics. Metrics are stored in the following data streams: @@ -57,24 +53,19 @@ Metrics '\\', '/', '*', '?', '"', '<', '>', '|', ' ', ',', '#', ':', '-' ``` - :::: - ::::{important} Additional storage efficiencies provided by [Synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md) are available to users with an [appropriate license](https://www.elastic.co/subscriptions). :::: - Logs : Logs include application error events and application logs. Logs are stored in the following data streams: * APM error/exception logging: `logs-apm.error-` * Applications UI logging: `logs-apm.app.-` - - ## APM data stream rerouting [apm-data-stream-rerouting] APM supports rerouting APM data to user-defined APM data stream names other than the defaults. This can be achieved by using a [`reroute` processor](elasticsearch://reference/enrich-processor/reroute-processor.md) in ingest pipelines to set the data stream dataset or namespace. The benefit of separating APM data streams is that custom retention and security policies can be used. @@ -98,7 +89,6 @@ To find other ingest pipelines from the {{es}} apm-data plugin that are called b For more custom APM ingest pipeline guides, see [parse data using ingest pipelines](parse-data-using-ingest-pipelines.md). - ## What’s next? [apm-data-streams-next] * Data streams define not only how data is stored in {{es}}, but also how data is retained over time. See [{{ilm-cap}}](index-lifecycle-management.md) to learn how to create your own data retention policies. diff --git a/solutions/observability/apps/delete-sensitive-data.md b/solutions/observability/apps/delete-sensitive-data.md index 078d263a74..aaf0f25a95 100644 --- a/solutions/observability/apps/delete-sensitive-data.md +++ b/solutions/observability/apps/delete-sensitive-data.md @@ -15,8 +15,6 @@ If you accidentally ingest sensitive data, follow these steps to remove or redac * [Redact specific fields](#apm-redact-field-data) * [Delete {{es}} documents](#apm-delete-doc-data) - - ## Redact specific fields [apm-redact-field-data] To redact sensitive data in a specific field, use the [update by query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query). @@ -63,14 +61,12 @@ POST /logs-apm.error-default/_update_by_query See [update by query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query) for more information and examples. - ## Delete {{es}} documents [apm-delete-doc-data] ::::{warning} This will permanently delete your data. You should test your queries with the [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) prior to deleting data. :::: - To delete an {{es}} document, you can use the [delete by query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-delete-by-query). For example, to delete all documents in the `apm-traces-*` data stream with a `user.email` value, run the following query: diff --git a/solutions/observability/apps/dependencies.md b/solutions/observability/apps/dependencies.md index 9ae372f7bc..195e577ebd 100644 --- a/solutions/observability/apps/dependencies.md +++ b/solutions/observability/apps/dependencies.md @@ -29,7 +29,6 @@ When viewing a dependency, consider your pattern of usage with that dependency. If your usage pattern *has* changed, the dependency view can quickly show you whether that pattern change exists in all upstream services, or just a subset of your services. You might then start digging into traces coming from impacted services to determine why that pattern change has occurred. - ## Operations [dependencies-operations] ::::{Warning} diff --git a/solutions/observability/apps/drill-down-into-data.md b/solutions/observability/apps/drill-down-into-data.md index a41d1af453..95a6f84e38 100644 --- a/solutions/observability/apps/drill-down-into-data.md +++ b/solutions/observability/apps/drill-down-into-data.md @@ -10,7 +10,6 @@ applies_to: # Drill down into application data [apm-ui-drill-down] - Notice something awry? Select a service or trace and dive deeper with: * [Transactions](../../../solutions/observability/apps/transactions-2.md) diff --git a/solutions/observability/apps/elastic-apm-agent-configuration-api.md b/solutions/observability/apps/elastic-apm-agent-configuration-api.md index 0afdd353fe..9986f994c4 100644 --- a/solutions/observability/apps/elastic-apm-agent-configuration-api.md +++ b/solutions/observability/apps/elastic-apm-agent-configuration-api.md @@ -9,7 +9,6 @@ applies_to: APM Server exposes API endpoints that allow Elastic APM agents to query the APM Server for configuration changes. More information on this feature is available in [{{apm-agent}} configuration in {{kib}}](apm-agent-central-configuration.md). - ## Agent configuration endpoints [apm-api-config-endpoint] | Name | Endpoint | @@ -19,7 +18,6 @@ APM Server exposes API endpoints that allow Elastic APM agents to query the APM The Agent configuration endpoints accepts both `HTTP GET` and `HTTP POST` requests. If an [API keys](api-keys.md) or [Secret token](secret-token.md) is configured, requests to this endpoint must be authenticated. - ### HTTP GET [apm-api-config-api-get] `service.name` is a required query string parameter. @@ -28,7 +26,6 @@ The Agent configuration endpoints accepts both `HTTP GET` and `HTTP POST` reques http(s)://{hostname}:{port}/config/v1/agents?service.name=SERVICE_NAME ``` - ### HTTP POST [apm-api-config-api-post] Encode parameters as a JSON object in the body. `service.name` is a required parameter. @@ -44,14 +41,12 @@ http(s)://{hostname}:{port}/config/v1/agents } ``` - ### Responses [apm-api-config-api-response] * Successful - `200` * APM Server is configured to fetch agent configuration from {{es}} but the configuration is invalid - `403` * APM Server is starting up or {{es}} is unreachable - `503` - ### Example request [apm-api-config-api-example] Example Agent configuration `GET` request including the service name "test-service": @@ -69,7 +64,6 @@ curl -X POST http://127.0.0.1:8200/config/v1/agents \ -d '{"service": {"name": "test-service"}}' ``` - ### Example response [apm-api-config-api-ex-response] ```sh diff --git a/solutions/observability/apps/elastic-apm-agents.md b/solutions/observability/apps/elastic-apm-agents.md index fcee458be4..d5135d0814 100644 --- a/solutions/observability/apps/elastic-apm-agents.md +++ b/solutions/observability/apps/elastic-apm-agents.md @@ -10,14 +10,12 @@ applies_to: # Elastic APM agents [observability-apm-agents-elastic-apm-agents] - ::::{note} **For Observability Serverless projects**, the **Admin** role or higher is required to use APM agents. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). :::: - Elastic APM agents automatically measure application performance and track errors. They offer built-in support for popular frameworks and technologies, and provide easy-to-use APIs that allow you to instrument any application. Elastic APM agents are built and maintained by Elastic. While they are similar, different programming languages have different nuances and requirements. Select your preferred language below to learn more about how each agent works. diff --git a/solutions/observability/apps/elastic-apm-events-intake-api.md b/solutions/observability/apps/elastic-apm-events-intake-api.md index 0684abb017..6fe5ff519d 100644 --- a/solutions/observability/apps/elastic-apm-events-intake-api.md +++ b/solutions/observability/apps/elastic-apm-events-intake-api.md @@ -11,7 +11,6 @@ applies_to: Most users do not need to interact directly with the events intake API. :::: - The events intake API is what we call the internal protocol that APM agents use to talk to the APM Server. Agents communicate with the Server by sending events — captured pieces of information — in an HTTP request. Events can be: * Transactions @@ -25,7 +24,6 @@ With NDJSON, agents can open an HTTP POST request and use chunked encoding to st See the [APM data model](learn-about-application-data-types.md) to learn more about the different types of events. - ### Endpoints [apm-api-events-endpoint] APM Server exposes the following endpoints for Elastic APM agent data intake: @@ -36,7 +34,6 @@ APM Server exposes the following endpoints for Elastic APM agent data intake: | RUM event intake (v2) | `/intake/v2/rum/events` | | RUM event intake (v3) | `/intake/v3/rum/events` | - ### Request [apm-api-events-example] Send an `HTTP POST` request to the APM Server `intake/v2/events` endpoint: @@ -55,21 +52,18 @@ http(s)://{hostname}:{port}/intake/v2/events?async=true Since asynchronous processing defers some of the event processing to the background and takes place after the client has closed the request, some errors can’t be communicated back to the client and are logged by the APM Server. Furthermore, asynchronous processing requests will only be scheduled if the APM Server can service the incoming request, requests that cannot be serviced will receive an internal error `503` "queue is full" error. :::: - For [RUM](real-user-monitoring-rum.md) send an `HTTP POST` request to the APM Server `intake/v3/rum/events` endpoint instead: ```bash http(s)://{hostname}:{port}/intake/v3/rum/events ``` - ### Response [apm-api-events-response] On success, the server will respond with a 202 Accepted status code and no body. Keep in mind that events can succeed and fail independently of each other. Only if all events succeed does the server respond with a 202. - ### Errors [apm-api-events-errors] There are two types of errors that the APM Server may return to an agent: @@ -108,10 +102,8 @@ An example error response might look something like this: 3. An immediately returning non-event related error 4. The number of accepted events - If you’re developing an agent, these errors can be useful for debugging. - ### Event API Schemas [apm-api-events-schema-definition] The APM Server uses a collection of JSON Schemas for validating requests to the intake API: @@ -133,11 +125,9 @@ Rather than send this metadata information from the agent multiple times, the AP Metadata is stored under `context` when viewing documents in {{es}}. :::: - * [Kubernetes data](#apm-api-kubernetes-data) * [Metadata Schema](#apm-api-metadata-schema) - #### Kubernetes data [apm-api-kubernetes-data] APM agents automatically read Kubernetes data and send it to the APM Server. In most instances, agents are able to read this data from inside the container. If this is not the case, or if you wish to override this data, you can set environment variables for the agents to read. These environment variable are set via the Kubernetes [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables). Here’s how you would add the environment variables to your Kubernetes pod spec: @@ -170,7 +160,6 @@ The table below maps these environment variables to the APM metadata event field | `KUBERNETES_NAMESPACE` | system.kubernetes.namespace | | `KUBERNETES_POD_UID` | system.kubernetes.pod.uid | - #### Metadata Schema [apm-api-metadata-schema] APM Server uses JSON Schema to validate requests. The specification for metadata is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/metadata.json) and included below: @@ -753,12 +742,10 @@ APM Server uses JSON Schema to validate requests. The specification for metadata } ``` - ## Transactions [apm-api-transaction] Transactions are events corresponding to an incoming request or similar task occurring in a monitored service. - #### Transaction Schema [apm-api-transaction-schema] APM Server uses JSON Schema to validate requests. The specification for transactions is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/transaction.json) and included below: @@ -1897,12 +1884,10 @@ APM Server uses JSON Schema to validate requests. The specification for transact } ``` - ## Spans [apm-api-span] Spans are events captured by an agent occurring in a monitored service. - #### Span Schema [apm-api-span-schema] APM Server uses JSON Schema to validate requests. The specification for spans is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/span.json) and included below: @@ -2816,12 +2801,10 @@ APM Server uses JSON Schema to validate requests. The specification for spans is } ``` - ## Errors [apm-api-error] An error or a logged error message captured by an agent occurring in a monitored service. - #### Error Schema [apm-api-error-schema] APM Server uses JSON Schema to validate requests. The specification for errors is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/error.json) and included below: @@ -4122,12 +4105,10 @@ APM Server uses JSON Schema to validate requests. The specification for errors i } ``` - ## Metrics [apm-api-metricset] Metrics contain application metric data captured by an {{apm-agent}}. - #### Metric Schema [apm-api-metricset-schema] APM Server uses JSON Schema to validate requests. The specification for metrics is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/metricset.json) and included below: @@ -4436,7 +4417,6 @@ APM Server uses JSON Schema to validate requests. The specification for metrics } ``` - ## Example request body [apm-api-event-example] A request body example containing one event for all currently supported event types. @@ -4449,4 +4429,3 @@ A request body example containing one event for all currently supported event ty {"metricset":{"samples":{"transaction.breakdown.count":{"value":12},"transaction.duration.sum.us":{"value":12},"transaction.duration.count":{"value":2},"transaction.self_time.sum.us":{"value":10},"transaction.self_time.count":{"value":2},"span.self_time.count":{"value":1},"span.self_time.sum.us":{"value":633.288},"byte_counter":{"value":1},"short_counter":{"value":227},"integer_gauge":{"value":42767},"long_gauge":{"value":3147483648},"float_gauge":{"value":9.16},"double_gauge":{"value":3.141592653589793},"dotted.float.gauge":{"value":6.12},"negative.d.o.t.t.e.d":{"value":-1022}},"tags":{"code":200,"success":true},"transaction":{"type":"request","name":"GET/"},"span":{"type":"db","subtype":"mysql"},"timestamp":1571657444929001}} ``` - diff --git a/solutions/observability/apps/errors.md b/solutions/observability/apps/errors.md index 35e7882a27..b34698e612 100644 --- a/solutions/observability/apps/errors.md +++ b/solutions/observability/apps/errors.md @@ -23,17 +23,14 @@ An Error contains: * URL - full, domain, port, query, etc. * [User](metadata.md#apm-data-model-user) - (if supplied) email, ID, username, etc. - In addition, agents provide options for users to capture custom [metadata](metadata.md). Metadata can be indexed - [`labels`](metadata.md#apm-data-model-labels), or not-indexed - [`custom`](metadata.md#apm-data-model-custom). ::::{tip} Most agents limit keyword fields (e.g. `error.id`) to 1024 characters, non-keyword fields (e.g. `error.exception.message`) to 10,000 characters. :::: - Errors are stored in error indices. - ## Data streams [_data_streams_3] Errors are stored in the following data streams: @@ -43,7 +40,6 @@ Errors are stored in the following data streams: See [Data streams](data-streams.md) to learn more. - ## Example error document [_example_error_document] This example shows what error documents can look like when indexed in {{es}}. @@ -643,4 +639,3 @@ This example shows what error documents can look like when indexed in {{es}}. :::: - diff --git a/solutions/observability/apps/explore-data-in-elasticsearch.md b/solutions/observability/apps/explore-data-in-elasticsearch.md index cc52979f29..c297684b12 100644 --- a/solutions/observability/apps/explore-data-in-elasticsearch.md +++ b/solutions/observability/apps/explore-data-in-elasticsearch.md @@ -9,7 +9,6 @@ applies_to: * [{{es}} query examples](#apm-elasticsearch-query-examples) - ## {{es}} query examples [apm-elasticsearch-query-examples] Elastic APM data is stored in [Data streams](data-streams.md). diff --git a/solutions/observability/apps/explore-mobile-sessions-with-discover.md b/solutions/observability/apps/explore-mobile-sessions-with-discover.md index b4d5635368..353fb01e67 100644 --- a/solutions/observability/apps/explore-mobile-sessions-with-discover.md +++ b/solutions/observability/apps/explore-mobile-sessions-with-discover.md @@ -9,7 +9,6 @@ applies_to: Elastic Mobile APM provides session tracking by attaching a `session.id`, a guid, to every span and event. This allows for the recall of the activities of a specific user during a specific period of time. The best way recall these data points is using [Discover](../../../explore-analyze/discover/document-explorer.md). This guide will explain how to do that. - ## Viewing sessions with Discover [viewing-sessions-with-discover] The first step is to find the relevant `session.id`. In this example, we’ll walk through investigating a crash. Since all events and spans have `session.id` attributes, a crash is no different. diff --git a/solutions/observability/apps/filter-application-data.md b/solutions/observability/apps/filter-application-data.md index 651afe5137..e952bce7d2 100644 --- a/solutions/observability/apps/filter-application-data.md +++ b/solutions/observability/apps/filter-application-data.md @@ -10,7 +10,6 @@ applies_to: # Filter application data [apm-filter-your-data] - Global filters are ways you can filter your APM data based on a specific time range or environment. When viewing a specific service, the filter persists as you move between tabs. :::{image} /solutions/images/observability-global-filters.png @@ -23,13 +22,10 @@ If you prefer to use advanced queries on your data to filter on specific pieces :::: - - ## Global time range [apm-filter-your-data-global-time-range] The global time range filter restricts APM data to a specific time period. - ## Service environment filter [apm-filter-your-data-service-environment-filter] The environment selector is a global filter for `service.environment`. It allows you to view only relevant data and is especially useful for separating development from production environments. By default, all environments are displayed. If there are no environment options, you’ll see "not defined". diff --git a/solutions/observability/apps/filter-search-application-data.md b/solutions/observability/apps/filter-search-application-data.md index 3c7313b044..ea4c0dba41 100644 --- a/solutions/observability/apps/filter-search-application-data.md +++ b/solutions/observability/apps/filter-search-application-data.md @@ -10,7 +10,6 @@ applies_to: # Filter and search application data [apm-filter-and-search-data] - Because Elastic APM is built on top of the {{stack}}, you have the full power of Elastic’s powerful search capabilities to filter and search through your application data. Mastering how to filter and search your data can help you find bottlenecks in your code faster: * Use global filters to [filter data](../../../solutions/observability/apps/filter-application-data.md) across the Applications UI based on a specific time range or environment. diff --git a/solutions/observability/apps/find-transaction-latency-failure-correlations.md b/solutions/observability/apps/find-transaction-latency-failure-correlations.md index 60d93bc4a3..a98ac9760a 100644 --- a/solutions/observability/apps/find-transaction-latency-failure-correlations.md +++ b/solutions/observability/apps/find-transaction-latency-failure-correlations.md @@ -21,7 +21,6 @@ To find correlations: Select a service on the **Services** page in the Applications UI then select a transaction group from the **Transactions** tab. - ::: :::{tab-item} Serverless @@ -40,7 +39,6 @@ Select a service on the **Services** page in the Applications UI then select a t Active queries *are* applied to correlations. :::: - ## Find high transaction latency correlations [observability-apm-find-transaction-latency-and-failure-correlations-find-high-transaction-latency-correlations] The correlations on the **Latency correlations** tab help you discover which attributes are contributing to increased transaction latency. @@ -63,7 +61,6 @@ You can also click the icon beside the field name to view and filter its most po In this example screenshot, there are transactions that are skewed to the right with slower response times than the overall latency distribution. If you select the `+` filter in the appropriate row of the table, it creates a new query in the Applications UI for transactions with this attribute. With the "noise" now filtered out, you can begin viewing sample traces to continue your investigation. - ## Find failed transaction correlations [correlations-error-rate] The correlations on the **Failed transaction correlations** tab help you discover which attributes are most influential in distinguishing between transaction failures and successes. In this context, the success or failure of a transaction is determined by its [event.outcome](ecs://reference/ecs-event.md#field-event-outcome) value. For example, APM agents set the `event.outcome` to `failure` when an HTTP transaction returns a `5xx` status code. diff --git a/solutions/observability/apps/fleet-managed-apm-server.md b/solutions/observability/apps/fleet-managed-apm-server.md index 30bc00b353..39548ed47b 100644 --- a/solutions/observability/apps/fleet-managed-apm-server.md +++ b/solutions/observability/apps/fleet-managed-apm-server.md @@ -9,7 +9,6 @@ applies_to: This guide will explain how to set up and configure a Fleet-managed APM Server. - ## Prerequisites [_prerequisites_6] You need {{es}} for storing and searching your data, and {{kib}} for visualizing and managing it. When setting these components up, you need: @@ -41,8 +40,6 @@ xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters" 1. The password should be stored in the {{kib}} keystore as described in the [{{es}} security documentation](../../../deploy-manage/security/set-up-minimal-security.md). - - ## Step 1: Set up Fleet [_step_1_set_up_fleet] Use {{fleet}} in {{kib}} to get APM data into the {{stack}}. @@ -52,7 +49,6 @@ If you already have a {{fleet-server}} set up, you can choose to skip this step. :::: - The first time you use {{fleet}}, you’ll need to set it up and add a {{fleet-server}} using the steps outlined below. To deploy a self-managed {{fleet-server}}, you install an {{agent}} and enroll it in an agent policy containing the {{fleet-server}} integration. @@ -61,7 +57,6 @@ To deploy a self-managed {{fleet-server}}, you install an {{agent}} and enroll i You can install only a single {{agent}} per host, which means you cannot run {{fleet-server}} and another {{agent}} on the same host unless you deploy a containerized {{fleet-server}}. :::: - 1. In {{fleet}}, open the **Settings** tab. For more information about these settings, see [{{fleet}} settings](/reference/fleet/fleet-settings.md). 2. Under **Fleet Server hosts**, click **Edit hosts** and specify one or more host URLs your {{agent}}s will use to connect to {{fleet-server}}. For example, `https://192.0.2.1:8220`, where `192.0.2.1` is the host IP where you will install {{fleet-server}}. Save and apply your settings. @@ -78,7 +73,6 @@ You can install only a single {{agent}} per host, which means you cannot run {{f :screenshot: ::: - **Notes:** * Choose **Quick Start** if you want {{fleet}} to generate a {{fleet-server}} policy and enrollment token for you. The {{fleet-server}} policy will include a {{fleet-server}} integration plus a system integration for monitoring {{agent}}. This option generates self-signed certificates and is not recommended for production use cases. @@ -101,10 +95,8 @@ If installation is successful, you’ll see confirmation that {{fleet-server}} c If you’re unable to add a {{fleet}}-managed agent, click the **Agents** tab and confirm that the agent running {{fleet-server}} is healthy. :::: - For more information, refer to [{{fleet-server}}](/reference/fleet/fleet-server.md). - ## Step 2: Add and configure the APM integration [add-apm-integration] ::::{note} @@ -112,7 +104,6 @@ If you don’t have a {{fleet}} setup already in place, the easiest way to get s :::: - 1. In {{kib}}, find **Integrations** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). 2. Select **Elastic APM**. @@ -145,7 +136,6 @@ If you don’t have a {{fleet}} setup already in place, the easiest way to get s Any {{agent}}s assigned to this policy will collect APM data from your instrumented services. - ::::{tip} An internet connection is required to install the APM integration via the Fleet UI in Kibana. @@ -163,7 +153,6 @@ xpack.fleet.packages: See [Configure Kibana](kibana://reference/configuration-reference/general-settings.md) to learn more about how to edit the Kibana configuration file. - Option 2: Use the {{fleet}} API : Use the {{fleet}} API to install the APM integration. To be successful, this needs to be run against the {{kib}} API, not the {{es}} API. @@ -174,14 +163,10 @@ POST kbn:/api/fleet/epm/packages/apm/9.0.0 See [Kibana API](https://www.elastic.co/guide/en/kibana/current/api.html) to learn more about how to use the Kibana APIs. - :::: - :::: - - ## Step 3: Install APM agents [_step_3_install_apm_agents] APM agents are written in the same language as your service. To monitor a new service, you must install the agent and configure it with a service name, APM Server host, and Secret token. @@ -194,7 +179,6 @@ APM agents are written in the same language as your service. To monitor a new se You can edit your APM integration settings if you need to change the APM Server URL or secret token to match your APM agents. :::: - :::::::{tab-set} ::::::{tab-item} Android @@ -212,7 +196,6 @@ plugins { 1. The Elastic plugin declaration must be added below the Android app plugin declaration (`com.android.application`) and below the Kotlin plugin declaration (if used). - **2. Configure the agent** After adding the agent plugin, configure it. A minimal configuration sets the Elastic APM integration endpoint as shown below: @@ -242,12 +225,10 @@ elasticApm { 4. Defaults to null. More info on API Keys [here](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key). 5. Defaults to null. - ::::{note} When both `secretToken` and `apiKey` are provided, apiKey has priority and secretToken is ignored. :::: - **3. Initialize the agent** After syncing your project with the Gradle changes above, the Elastic APM agent needs to be initialized within your [Application class](https://developer.android.com/reference/android/app/Application). This example shows the simplest way to configure the agent: @@ -267,7 +248,6 @@ class MyApp extends android.app.Application { 1. Initialize the Elastic APM agent once. - All that’s left is to compile and run your application. That’s it! **Learn more in the agent reference** @@ -375,7 +355,6 @@ struct MyApp: App { 1. The APM integration host and port 2. Secret token for APM integration connection - If you’re not using `SwiftUI`, you can alternatively add the same thing to your `AppDelegate.swift` file: ```swift @@ -397,7 +376,6 @@ var config = AgentConfigBuilder() 1. The APM integration host and port 2. Secret token for APM integration connection - **Learn more in the agent reference** Read more in the [APM iOS Agent Reference](apm-agent-ios://reference/index.md). @@ -417,7 +395,6 @@ The first step in getting started with the Elastic APM Java agent is to retrieve curl -o 'elastic-apm-agent.jar' -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST' ``` - **2. Add `-javaagent` flag** When starting your application, add the JVM flag `-javaagent:/path/to/elastic-apm-agent-.jar` @@ -526,7 +503,6 @@ If you can’t find your distribution, you can install the agent by building it The agent is currently only available for Linux operating system. :::: - 1. Download the agent source from [https://github.com/elastic/apm-agent-php/](https://github.com/elastic/apm-agent-php/). 2. Execute the following commands to build the agent and install it: @@ -607,7 +583,6 @@ MIDDLEWARE = ( ) ``` - Flask : **1. Install the {{apm-agent}}** @@ -644,7 +619,6 @@ app.config['ELASTIC_APM'] = { apm = ElasticAPM(app) ``` - **Learn more in the agent reference** * [Supported technologies](apm-agent-python://reference/supported-technologies.md) @@ -679,7 +653,6 @@ secret_token: '' server_url: 'http://localhost:8200' ``` - Rack : For Rack or a compatible framework, like Sinatra, include the middleware in your app and start the agent. @@ -721,7 +694,6 @@ secret_token: '' server_url: 'http://localhost:8200' ``` - **Learn more in the agent reference** * [Supported technologies](apm-agent-ruby://reference/supported-technologies.md) diff --git a/solutions/observability/apps/general-configuration-options.md b/solutions/observability/apps/general-configuration-options.md index a7989d933f..d4f1b405a0 100644 --- a/solutions/observability/apps/general-configuration-options.md +++ b/solutions/observability/apps/general-configuration-options.md @@ -14,7 +14,6 @@ Most options on this page are supported by all APM Server deployment methods. :::: - General APM Server configuration options. :::::::{tab-set} @@ -45,7 +44,6 @@ Configure and customize Fleet-managed APM settings directly in {{kib}}: ## Configuration options [apm-configuration-apm-server] - ### Host [apm-host] Defines the host and port the server is listening on. Use `"unix:/path/to.sock"` to listen on a Unix domain socket. Defaults to *localhost:8200*. (text) @@ -55,7 +53,6 @@ Defines the host and port the server is listening on. Use `"unix:/path/to.sock"` | APM Server binary | `apm-server.host` | | Fleet-managed | `Host` | - ### URL [_url] The publicly reachable server URL. For deployments on Elastic Cloud or ECK, the default is unchangeable. @@ -65,7 +62,6 @@ The publicly reachable server URL. For deployments on Elastic Cloud or ECK, the | APM Server binary | N/A | | Fleet-managed | `URL` | - ### Max header size [apm-max_header_size] Maximum permitted size of a request’s header accepted by the server to be processed (in Bytes). Defaults to 1048576 Bytes (1 MB). (int) @@ -75,7 +71,6 @@ Maximum permitted size of a request’s header accepted by the server to be proc | APM Server binary | `apm-server.max_header_size` | | Fleet-managed | `Maximum size of a request's header` | - ### Idle timeout [apm-idle_timeout] Maximum amount of time to wait for the next incoming request before underlying connection is closed. Defaults to `45s` (45 seconds). (text) @@ -85,7 +80,6 @@ Maximum amount of time to wait for the next incoming request before underlying c | APM Server binary | `apm-server.idle_timeout` | | Fleet-managed | `Idle time before underlying connection is closed` | - ### Read timeout [apm-read_timeout] Maximum permitted duration for reading an entire request. Defaults to `3600s` (3600 seconds). (text) @@ -95,7 +89,6 @@ Maximum permitted duration for reading an entire request. Defaults to `3600s` (3 | APM Server binary | `apm-server.read_timeout` | | Fleet-managed | `Maximum duration for reading an entire request` | - ### Write timeout [apm-write_timeout] Maximum permitted duration for writing a response. Defaults to `30s` (30 seconds). (text) @@ -105,7 +98,6 @@ Maximum permitted duration for writing a response. Defaults to `30s` (30 seconds | APM Server binary | `apm-server.write_timeout` | | Fleet-managed | `Maximum duration for writing a response` | - #### Shutdown timeout [apm-shutdown_timeout] Maximum duration in seconds before releasing resources when shutting down the server. Defaults to `30s` (30 seconds). (text) @@ -115,7 +107,6 @@ Maximum duration in seconds before releasing resources when shutting down the se | APM Server binary | `apm-server.shutdown_timeout` | | Fleet-managed | `Maximum duration before releasing resources when shutting down` | - ### Max event size [apm-max_event_size] Maximum permitted size of an event accepted by the server to be processed (in Bytes). Defaults to `307200` Bytes. (int) @@ -125,7 +116,6 @@ Maximum permitted size of an event accepted by the server to be processed (in By | APM Server binary | `apm-server.max_event_size` | | Fleet-managed | `Maximum size per event` | - ### Max connections [apm-max_connections] Maximum number of TCP connections to accept simultaneously. Default value is 0, which means *unlimited*. (int) @@ -135,7 +125,6 @@ Maximum number of TCP connections to accept simultaneously. Default value is 0, | APM Server binary | `apm-server.max_connections` | | Fleet-managed | `Simultaneously accepted connections` | - ### Custom HTTP response headers [apm-custom_http_headers] Custom HTTP headers to add to HTTP responses. Useful for security policy compliance. (text) @@ -145,7 +134,6 @@ Custom HTTP headers to add to HTTP responses. Useful for security policy complia | APM Server binary | `apm-server.response_headers` | | Fleet-managed | `Custom HTTP response headers` | - ### Capture personal data [apm-capture_personal_data] If true, APM Server captures the IP of the instrumented service and its User Agent if any. Enabled by default. (bool) @@ -155,7 +143,6 @@ If true, APM Server captures the IP of the instrumented service and its User Age | APM Server binary | `apm-server.capture_personal_data` | | Fleet-managed | `Capture personal data` | - ### Default service environment [apm-default_service_environment] Sets the default service environment to associate with data and requests received from agents which have no service environment defined. Default: none. (text) @@ -165,7 +152,6 @@ Sets the default service environment to associate with data and requests receive | APM Server binary | `apm-server.default_service_environment` | | Fleet-managed | `Default Service Environment` | - ### expvar support [apm-expvar.enabled] When set to true APM Server exposes [golang expvar](https://golang.org/pkg/expvar/) under `/debug/vars`. Disabled by default. @@ -175,7 +161,6 @@ When set to true APM Server exposes [golang expvar](https://golang.org/pkg/expva | APM Server binary | `apm-server.expvar.enabled` | | Fleet-managed | `Enable APM Server Golang expvar support` | - ### expvar URL [apm-expvar.url] Configure the URL to expose expvar. Defaults to `debug/vars`. @@ -185,7 +170,6 @@ Configure the URL to expose expvar. Defaults to `debug/vars`. | APM Server binary | `apm-server.expvar.url` | | Fleet-managed | N/A | - ### Data stream namespace [apm-data_streams.namespace] Change the default namespace. This setting changes the name of the data stream. diff --git a/solutions/observability/apps/get-started-with-uptime.md b/solutions/observability/apps/get-started-with-uptime.md index da0134e998..676763d32f 100644 --- a/solutions/observability/apps/get-started-with-uptime.md +++ b/solutions/observability/apps/get-started-with-uptime.md @@ -34,15 +34,12 @@ If you’ve used the Elastic Synthetics integration to create monitors in the pa :::: - - ## Pull the Docker image [uptime-set-up-docker] Elastic provides Docker images that you can use to run monitors. Start by pulling the {{heartbeat}} Docker image. Version 9.0.0-beta1 has not yet been released. - ## Configure [uptime-set-up-config] Next, create a `heartbeat.yml` configuration file. @@ -62,7 +59,6 @@ heartbeat.monitors: 1. Each `monitor` gets its own ID in the {{uptime-app}} and its own schedule entry. This allows tests to be run in parallel and analyzed separately. - Read more about configuration options in [Configure {{heartbeat}} monitors](beats://reference/heartbeat/configuration-heartbeat-options.md). ::::{warning} @@ -72,22 +68,18 @@ If you previously used {{heartbeat}} to set up **`browser`** monitor, you can fi :::: - - ## Connect to the {{stack}} [uptime-set-up-connect] After configuring the monitor, run it in Docker and connect the monitor to the {{stack}}. Version 9.0.0-beta1 has not yet been released. - ## View in {{kib}} [uptime-set-up-kibana] {{heartbeat}} is now sending synthetic monitoring data to the {{stack}}. Navigate to the {{uptime-app}} in {{kib}}, where you can see screenshots of each run, set up alerts in case of test failures, and more. If a test does fail (shown as `down` in the {{uptime-app}}), you’ll be able to view the step script that failed, any errors, and a stack trace. For more information, refer to [Analyze](analyze.md). - ## Manage monitors [uptime-manage] After you’ve created a monitor, you’ll need to manage that monitor over time. This might include updating or permanently deleting an existing monitor. diff --git a/solutions/observability/apps/get-started.md b/solutions/observability/apps/get-started.md index 021543f223..a4bcbcd97a 100644 --- a/solutions/observability/apps/get-started.md +++ b/solutions/observability/apps/get-started.md @@ -18,7 +18,6 @@ There are two ways to set up a synthetic monitor: Read more about each option below, and choose the approach that works best for you. - ## Synthetics project [observability-synthetics-get-started-synthetics-project] With a Synthetics project, you write tests in an external version-controlled Node.js project using YAML for lightweight monitors and JavaScript or TypeScript for browser monitors. Then, you use the `@elastic/synthetics` NPM library’s `push` command to create monitors. @@ -31,7 +30,6 @@ Get started in [Create monitors in a Synthetics project](../../../solutions/obse :alt: Diagram showing which pieces of software are used to configure monitors ::: - ## Synthetics UI [observability-synthetics-get-started-synthetics-ui] You can create monitors directly in the user interface. This approach works well if you want to create and manage your monitors in the browser. diff --git a/solutions/observability/apps/grant-access-using-api-keys.md b/solutions/observability/apps/grant-access-using-api-keys.md index 165a3474ce..b50e67cb42 100644 --- a/solutions/observability/apps/grant-access-using-api-keys.md +++ b/solutions/observability/apps/grant-access-using-api-keys.md @@ -15,8 +15,6 @@ APM Server instances typically send both collected data and monitoring informati For security reasons, we recommend using a unique API key per APM Server instance. You can create as many API keys per user as necessary. :::: - - ## Create an API key for writing events [apm-beats-api-key-publish] To create an API key: @@ -74,7 +72,6 @@ To create an API key: :alt: API key dropdown highlighting the Beats option ::: - You can now use this API key in your `apm-server.yml` configuration file: ```yaml @@ -84,8 +81,6 @@ output.elasticsearch: 1. Format is `id:api_key` (as shown in the {{beats}} dropdown) - - ## Create an API key for monitoring [apm-beats-api-key-monitor] To open **API keys**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Click **Create API key**. @@ -114,7 +109,6 @@ Enter a name for your API key and select **Restrict privileges**. In the role de This example only provides privileges for **publishing monitoring data**. See [Use feature roles](create-assign-feature-roles-to-apm-server-users.md) for additional privileges and information. :::: - To set an expiration date for the API key, select **Expire after time** and input the lifetime of the API key in days. Click **Create API key**. In the dropdown, switch to **{{beats}}** and copy the API key. @@ -128,8 +122,6 @@ monitoring.elasticsearch: 1. Format is `id:api_key` (as shown in the {{beats}} dropdown) - - ## Create an API key with {{es}} APIs [apm-beats-api-key-es] You can also use {{es}}'s [Create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) to create a new API key. For example: @@ -172,10 +164,8 @@ POST /_security/api_key 1. Name of the API key 2. Granted privileges, see [Use feature roles](create-assign-feature-roles-to-apm-server-users.md) - See the [Create API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) reference for more information. - ## Learn more about API keys [apm-learn-more-api-keys] See the {{es}} API key documentation for more information: diff --git a/solutions/observability/apps/grant-users-access-to-secured-resources.md b/solutions/observability/apps/grant-users-access-to-secured-resources.md index b3e88387ea..ceafc25bd8 100644 --- a/solutions/observability/apps/grant-users-access-to-secured-resources.md +++ b/solutions/observability/apps/grant-users-access-to-secured-resources.md @@ -51,7 +51,6 @@ In general, these are types of privileges you’ll work with: * Create, modify, and delete monitors. * View and create visualizations that access Synthetics data. - Read more about user roles in [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md). ::: diff --git a/solutions/observability/apps/high-availability.md b/solutions/observability/apps/high-availability.md index 605e2045b6..f46fbbd1e4 100644 --- a/solutions/observability/apps/high-availability.md +++ b/solutions/observability/apps/high-availability.md @@ -19,4 +19,3 @@ If {{es}} goes down, the APM Server will eventually deny incoming requests. Both Fleet-managed APM Server users might also be interested in [Fleet/Agent proxy support](/reference/fleet/fleet-agent-proxy-support.md). :::: - diff --git a/solutions/observability/apps/index-lifecycle-management.md b/solutions/observability/apps/index-lifecycle-management.md index 87242a7535..ea5663c992 100644 --- a/solutions/observability/apps/index-lifecycle-management.md +++ b/solutions/observability/apps/index-lifecycle-management.md @@ -16,8 +16,6 @@ Indices created in 8.15.x and 8.16.x might be managed by [data stream lifecycle :::: - - ## Default policies [index-lifecycle-policies-default] Each APM data stream has its own default lifecycle policy including a delete definition and a rollover definition. @@ -51,15 +49,12 @@ Rollover (writing to a new index) prevents a single index from growing too large Default lifecycle policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features. :::: - - ## Configure a custom index lifecycle policy [apm-data-streams-custom-policy] Mappings and settings for data streams can be customized through the creation of `*@custom` component templates, which are referenced by the index templates created by the {{es}} apm-data plugin. The easiest way to configure a custom index lifecycle policy per data stream is to edit this template. This tutorial explains how to apply a custom index lifecycle policy to the `traces-apm` data stream. - ## Step 1: View data streams [apm-data-streams-custom-one] The **Data Streams** view in {{kib}} shows you data streams, index templates, and lifecycle policies: @@ -74,8 +69,6 @@ The **Data Streams** view in {{kib}} shows you data streams, index templates, an :screenshot: ::: - - ## Step 2: Create an index lifecycle policy [apm-data-streams-custom-two] 1. To open **Lifecycle Policies**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). @@ -83,7 +76,6 @@ The **Data Streams** view in {{kib}} shows you data streams, index templates, an Name your new policy; For this tutorial, I’ve chosen `custom-traces-apm-policy`. Customize the policy to your liking, and when you’re done, click **Save policy**. - ## Step 3: Apply the index lifecycle policy [apm-data-streams-custom-three] To apply your new index lifecycle policy to the `traces-apm-*` data stream, edit the `@custom` component template. @@ -108,8 +100,6 @@ To apply your new index lifecycle policy to the `traces-apm-*` data stream, edit :screenshot: ::: - - ## Step 4: Roll over the data stream (optional) [apm-data-streams-custom-four] To confirm that the data stream is now using the new index template and {{ilm-init}} policy, you can either repeat [step one](#apm-data-streams-custom-one), or navigate to **{{dev-tools-app}}** and run the following: @@ -120,7 +110,6 @@ GET /_data_stream/traces-apm-default <1> 1. The name of the data stream we’ve been hacking on appended with your - The result should include the following: ```json @@ -139,14 +128,12 @@ The result should include the following: 1. The name of the custom index template created in step three 2. The name of the {{ilm-init}} policy applied to the new component template in step two - New {{ilm-init}} policies only take effect when new indices are created, so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the [{{es}} rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover): ```bash POST /traces-apm-default/_rollover/ ``` - ## Namespace-level index lifecycle policies [apm-data-streams-custom-policy-namespace] It is also possible to create more granular index lifecycle policies that apply to individual namespaces. This process is similar to the above tutorial, but includes cloning and modify the existing index template to use a new `*@custom` component template. diff --git a/solutions/observability/apps/infrastructure.md b/solutions/observability/apps/infrastructure.md index dbf7915890..86a6f630b9 100644 --- a/solutions/observability/apps/infrastructure.md +++ b/solutions/observability/apps/infrastructure.md @@ -15,14 +15,12 @@ The Applications UI Infrastructure functionality is in beta and is subject to ch :::: - The **Infrastructure** tab provides information about the containers, pods, and hosts that the selected service is linked to. * **Pods**: Uses the `kubernetes.pod.name` from the [APM metrics data streams](../../../solutions/observability/apps/metrics.md). * **Containers**: Uses the `container.id` from the [APM metrics data streams](../../../solutions/observability/apps/metrics.md). * **Hosts**: If the application is containerized—​if the APM metrics documents include `container.id`-- the `host.name` is used from the infrastructure data streams (filtered by `container.id`). If not, `host.hostname` is used from the APM metrics data streams. - :::{image} /solutions/images/serverless-infra.png :alt: Example view of the Infrastructure tab in the Applications UI :screenshot: diff --git a/solutions/observability/apps/inspect-uptime-duration-anomalies.md b/solutions/observability/apps/inspect-uptime-duration-anomalies.md index d602ed5ea7..4230994b4f 100644 --- a/solutions/observability/apps/inspect-uptime-duration-anomalies.md +++ b/solutions/observability/apps/inspect-uptime-duration-anomalies.md @@ -16,7 +16,6 @@ Use [Synthetic monitoring](/solutions/observability/apps/synthetic-monitoring.md Each monitor location is modeled, and when a monitor runs for an unusual amount of time, at a particular time, an anomaly is recorded and highlighted on the **Monitor duration** chart. - ## Enable uptime duration {{anomaly-detect}} [uptime-anomaly-detection] Create a {{ml}} job to detect anomalous monitor duration rates automatically. diff --git a/solutions/observability/apps/installation-layout.md b/solutions/observability/apps/installation-layout.md index a8524f63fa..87d7387b6c 100644 --- a/solutions/observability/apps/installation-layout.md +++ b/solutions/observability/apps/installation-layout.md @@ -9,7 +9,6 @@ applies_to: View the installation layout and default paths for both Fleet-managed APM Server and the APM Server binary. - ## Fleet-managed [_fleet_managed] {{agent}} files are installed in the following locations. You cannot override these installation paths because they are required for upgrades. diff --git a/solutions/observability/apps/integrate-with-jaeger-deprecated.md b/solutions/observability/apps/integrate-with-jaeger-deprecated.md index 35a60c42c9..80e6a079cc 100644 --- a/solutions/observability/apps/integrate-with-jaeger-deprecated.md +++ b/solutions/observability/apps/integrate-with-jaeger-deprecated.md @@ -6,19 +6,14 @@ applies_to: stack: deprecated --- - - # Integrate with Jaeger (deprecated) [apm-jaeger-integration] - ::::{warning} Support for Jaeger is deprecated and will be removed in a future version of Elastic APM. [Jaeger clients are deprecated](https://www.jaegertracing.io/docs/1.35/client-libraries/) in favor of OpenTelemetry SDKs, and OpenTelemetry has removed all Jaeger exporters from their [specification](https://github.com/open-telemetry/opentelemetry-specification/pull/2858). :::: - Elastic APM integrates with [Jaeger](https://www.jaegertracing.io/), an open-source, distributed tracing system. This integration allows users with an existing Jaeger setup to switch from the default Jaeger backend, to the {{stack}}. Best of all, no instrumentation changes are needed in your application code. - ## Supported architecture [apm-jaeger-architecture] Jaeger architecture supports different data formats and transport protocols that define how data can be sent to a collector. Elastic APM, as a Jaeger collector, supports communication with **Jaeger agents** via gRPC. @@ -29,7 +24,6 @@ Jaeger architecture supports different data formats and transport protocols that See the [Jaeger docs](https://www.jaegertracing.io/docs/1.27/architecture) for more information on Jaeger architecture. - ## Get started [apm-get-started-jaeger] Connect your preexisting Jaeger setup to Elastic APM in three steps: @@ -42,8 +36,6 @@ Connect your preexisting Jaeger setup to Elastic APM in three steps: There are [caveats](#apm-caveats-jaeger) to this integration. :::: - - ### Configure Jaeger agents [apm-configure-agent-client-jaeger] The APM integration serves Jaeger gRPC over the same host and port as the Elastic {{apm-agent}} protocol. @@ -64,7 +56,6 @@ The APM integration serves Jaeger gRPC over the same host and port as the Elasti --agent.tags="elastic-apm-auth=Bearer " ``` - ::::{tip} For the equivalent environment variables, change all letters to upper-case and replace punctuation with underscores (`_`). See the [Jaeger CLI flags documentation](https://www.jaegertracing.io/docs/1.27/cli/) for more information. :::: @@ -95,7 +86,6 @@ For the equivalent environment variables, change all letters to upper-case and r --agent.tags="elastic-apm-auth=Bearer " ``` - ::::{tip} For the equivalent environment variables, change all letters to upper-case and replace punctuation with underscores (`_`). See the [Jaeger CLI flags documentation](https://www.jaegertracing.io/docs/1.27/cli/) for more information. :::: @@ -112,22 +102,18 @@ There are two different ways to configure the sampling rate of your Jaeger agent * [{{apm-agent}} central configuration (default)](#apm-configure-sampling-central-jaeger) * [Local sampling in each Jaeger client](#apm-configure-sampling-local-jaeger) - #### {{apm-agent}} central configuration (default) [apm-configure-sampling-central-jaeger] Central sampling, with {{apm-agent}} central configuration, allows Jaeger clients to poll APM Server for the sampling rate. This means sample rates can be configured on the fly, on a per-service and per-environment basis. See [Central configuration](apm-agent-central-configuration.md) to learn more. - #### Local sampling in each Jaeger client [apm-configure-sampling-local-jaeger] If you don’t have access to the Applications UI, you’ll need to change the Jaeger client’s `sampler.type` and `sampler.param`. This enables you to set the sampling configuration locally in each Jaeger client. See the official [Jaeger sampling documentation](https://www.jaegertracing.io/docs/1.27/sampling/) for more information. - ### Start sending data [apm-configure-start-jaeger] That’s it! Data sent from Jaeger clients to the APM Server can now be viewed in the Applications UI. - ## Caveats [apm-caveats-jaeger] There are some limitations and differences between Elastic APM and Jaeger that you should be aware of. diff --git a/solutions/observability/apps/integrate-with-machine-learning.md b/solutions/observability/apps/integrate-with-machine-learning.md index 66b687cf46..1f7e64b334 100644 --- a/solutions/observability/apps/integrate-with-machine-learning.md +++ b/solutions/observability/apps/integrate-with-machine-learning.md @@ -29,7 +29,6 @@ Results from machine learning jobs are shown in multiple places throughout the A :screenshot: ::: - ## Enable anomaly detection [observability-apm-integrate-with-machine-learning-enable-anomaly-detection] To enable machine learning anomaly detection: @@ -42,12 +41,10 @@ To enable machine learning anomaly detection: That’s it! After a few minutes, the job will begin calculating results; it might take additional time for results to appear on your service maps. To manage existing jobs, click **Manage jobs** (or go to **Machine learning** → **Jobs**). - ## Anomaly detection warning [observability-apm-integrate-with-machine-learning-anomaly-detection-warning] To make machine learning as easy as possible to set up, Elastic will warn you when filtered to an environment without a machine learning job. - ## Unknown service health [observability-apm-integrate-with-machine-learning-unknown-service-health] After enabling anomaly detection, service health may display as "Unknown". Here are some reasons why this can occur: diff --git a/solutions/observability/apps/interpret-application-data.md b/solutions/observability/apps/interpret-application-data.md index 2fc8ac1017..4448c5034f 100644 --- a/solutions/observability/apps/interpret-application-data.md +++ b/solutions/observability/apps/interpret-application-data.md @@ -10,7 +10,6 @@ applies_to: # Interpret application data [observability-apm-interpret-data] - Learn how to get the most out of your data using the Applications UI. % Stateful only for exploring mobile sessions with Discover? diff --git a/solutions/observability/apps/inventory.md b/solutions/observability/apps/inventory.md index f38489e186..3e18f1a179 100644 --- a/solutions/observability/apps/inventory.md +++ b/solutions/observability/apps/inventory.md @@ -8,18 +8,14 @@ applies_to: serverless: --- - - # Inventory [inventory] - Inventory provides a single place to observe the status of your entire ecosystem of hosts, containers, and services at a glance, even just from logs. From there, you can monitor and understand the health of your entities, check what needs attention, and start your investigations. ::::{note} The new Inventory requires the Elastic Entity Model (EEM). To learn more, refer to [Elastic Entity Model](/reference/observability/elastic-entity-model.md). :::: - :::{image} /solutions/images/observability-inventory-catalog.png :alt: Inventory catalog :screenshot: @@ -48,7 +44,6 @@ Inventory allows you to: * Debug resource bottlenecks with your service caused by their containers and the hosts they run on. * Easily discover all entities related to the host, container or service you are viewing by leveraging your tags and labels - ## Explore your entities [explore-your-entities] 1. To view all your entities, find **Inventory** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). @@ -81,12 +76,10 @@ For each entity, you can click the entity name and get a detailed view. For exam If you open an entity of type `host` or `container` that does not have infrastructure data, some of the visualizations will be blank and some features on the page will not be fully populated. - ## Add entities to the Inventory [add-entities-to-inventory] You can add entities to the Inventory through one of the following approaches: **Add data** or **Associate existing service logs**. - ## Add data [add-data-entities] To add entities, select **Add data** and choose one of the following onboarding journeys: @@ -96,7 +89,6 @@ To add entities, select **Add data** and choose one of the following onboarding * **Application** Detects services * **Cloud** Ingests telemetry data from the Cloud - ## Associate existing service logs [associate-existing-service-logs] To learn how, refer to [Add a service name to logs](../logs/add-service-name-to-logs.md). diff --git a/solutions/observability/apps/jaeger-event-intake.md b/solutions/observability/apps/jaeger-event-intake.md index 77412be6f3..3a353c87e3 100644 --- a/solutions/observability/apps/jaeger-event-intake.md +++ b/solutions/observability/apps/jaeger-event-intake.md @@ -12,7 +12,6 @@ applies_to: :::: - Elastic APM natively supports Jaeger, an open-source, distributed tracing system. [Learn more](integrate-with-jaeger-deprecated.md). **Jaeger/gRPC paths** diff --git a/solutions/observability/apps/limitations.md b/solutions/observability/apps/limitations.md index 8f912dc69e..2541e721df 100644 --- a/solutions/observability/apps/limitations.md +++ b/solutions/observability/apps/limitations.md @@ -9,25 +9,21 @@ applies_to: # Limitations [apm-open-telemetry-known-limitations] - ## OpenTelemetry traces [apm-open-telemetry-traces-limitations] * Traces of applications using `messaging` semantics might be wrongly displayed as `transactions` in the Applications UI, while they should be considered `spans` (see issue [#7001](https://github.com/elastic/apm-server/issues/7001)). * Inability to see Stack traces in spans. * Inability in APM views to view the "Time Spent by Span Type" (see issue [#5747](https://github.com/elastic/apm-server/issues/5747)). - ## OpenTelemetry logs [apm-open-telemetry-logs-intake] * [preview] The OpenTelemetry logs intake via Elastic is in technical preview. * The application logs data stream (`app_logs`) has dynamic mapping disabled. This means the automatic detection and mapping of new fields is disabled (see issue [#9093](https://github.com/elastic/apm-server/issues/9093)). - ## OpenTelemetry Line Protocol (OTLP) [apm-open-telemetry-otlp-limitations] Elastic supports both the [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc) and [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp) protocol with ProtoBuf payload. Elastic does not yet support JSON Encoding for OTLP/HTTP. - ## OpenTelemetry Collector exporter for Elastic [apm-open-telemetry-collector-exporter] The [OpenTelemetry Collector exporter for Elastic](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.57.2/exporter/elasticexporter) has been deprecated and replaced by the native support of the OpenTelemetry Line Protocol in Elastic Observability (OTLP). diff --git a/solutions/observability/apps/manage-data-retention.md b/solutions/observability/apps/manage-data-retention.md index dafa0f0064..6234aa61a7 100644 --- a/solutions/observability/apps/manage-data-retention.md +++ b/solutions/observability/apps/manage-data-retention.md @@ -13,7 +13,6 @@ When you set up a synthetic monitor, data from the monitor is saved in [{{es}} d There are six data streams recorded by synthetic monitors: `http`, `tcp`, `icmp`, `browser`, `browser.network`, `browser.screenshot`. Elastic will retain data from each data stream for some time period, and the default time period varies by data stream. If you want to reduce the amount of storage required or store data for longer, you can customize how long to retain data for each data stream. - ## Synthetics data streams [synthetics-manage-retention-data-streams] There are six data streams recorded by synthetic monitors: @@ -29,7 +28,6 @@ There are six data streams recorded by synthetic monitors: All types of checks record core metadata. Browser-based checks store two additional types of data: network and screenshot documents. These browser-specific indices are usually many times larger than the core metadata. The relative sizes of each vary depending on the sites being checked with network data usually being the larger of the two by a significant factor. - ## Customize data stream lifecycles [synthetics-manage-retention-customize] If Synthetics browser data streams are storing data longer than necessary, you can opt to retain data for a shorter period. diff --git a/solutions/observability/apps/manage-monitors.md b/solutions/observability/apps/manage-monitors.md index 745a4d24d1..9f52b915f3 100644 --- a/solutions/observability/apps/manage-monitors.md +++ b/solutions/observability/apps/manage-monitors.md @@ -13,7 +13,6 @@ After you’ve [created a synthetic monitor](../../../solutions/observability/ap If you’re using {{project-monitors}}, you should also set up a workflow that uses [best practices for managing monitors effectively](../../../solutions/observability/apps/manage-monitors.md#synthetics-projects-best-practices) in a production environment. - ## Update a monitor [manage-monitors-config] You can update a monitor’s configuration, for example, changing the interval at which the monitor runs a test. @@ -82,22 +81,18 @@ If you [set up the monitor using the Synthetics UI](../../../solutions/observabi ::::::: Alternatively, you can temporarily disable a monitor by updating the monitor’s configuration in your journey’s code or in the Synthetics UI using the *Enabled* toggle. - ## Implement best practices for Synthetics projects [synthetics-projects-best-practices] ::::{important} This is only relevant to monitors created using projects. :::: - After you’ve [set up a project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md), there are some best practices you can implement to manage the Synthetics project effectively. - ### Use version control [synthetics-version-control] First, it’s recommended that you version control all files in Git. If your Synthetics project is not already in a version controlled directory add it and push it to your Git host. - ### Set up recommended workflow [synthetics-workflow] While it can be convenient to run the `push` command directly from your workstation, especially when setting up a new Synthetics project, it is not recommended for production environments. diff --git a/solutions/observability/apps/manage-storage.md b/solutions/observability/apps/manage-storage.md index c598f991bd..7bcbc3a9c6 100644 --- a/solutions/observability/apps/manage-storage.md +++ b/solutions/observability/apps/manage-storage.md @@ -13,11 +13,3 @@ The [storage and sizing guide](storage-sizing-guide.md) attempts to define a "ty In addition, the Applications UI makes it easy to visualize your APM data usage with [storage explorer](storage-explorer.md). Storage explorer allows you to analyze the storage footprint of each of your services to see which are producing large amounts of data—​so you can better reduce the data you’re collecting or forecast and prepare for future storage needs. - - - - - - - - diff --git a/solutions/observability/apps/managed-intake-service-event-api.md b/solutions/observability/apps/managed-intake-service-event-api.md index 8dc154b6da..5277804265 100644 --- a/solutions/observability/apps/managed-intake-service-event-api.md +++ b/solutions/observability/apps/managed-intake-service-event-api.md @@ -12,19 +12,16 @@ This API is exclusively for APM agent developers. The vast majority of users sho :::: - The managed intake service exposes endpoints for: * [The managed intake service information API](#observability-apm-server-api-server-information-api) * [Elastic APM events intake API](#observability-apm-server-api-events-intake-api) * [OpenTelemetry intake API](#observability-apm-server-api-opentelemetry-api) - ## Server information API [observability-apm-server-api-server-information-api] The managed intake service exposes an API endpoint to query general server information. This lightweight endpoint is useful as a server up/down health check. - ### Server Information endpoint [api-info-endpoint] Send an `HTTP GET` request to the server information endpoint: @@ -37,7 +34,6 @@ This endpoint always returns an HTTP 200. Requests to this endpoint must be authenticated. - #### Example [api-info-examples] Example managed intake service information request: @@ -54,7 +50,6 @@ curl -X POST http://127.0.0.1:8200/ \ } ``` - ## Events intake API [observability-apm-server-api-events-intake-api] ::::{note} @@ -62,7 +57,6 @@ Most users do not need to interact directly with the events intake API. :::: - The events intake API is what we call the internal protocol that APM agents use to talk to the managed intake service. Agents communicate with the Server by sending events — captured pieces of information — in an HTTP request. Events can be: * Transactions @@ -76,7 +70,6 @@ With NDJSON, agents can open an HTTP POST request and use chunked encoding to st Refer to [Learn about data types](learn-about-application-data-types.md) to learn more about the different types of events. - ### Endpoints [api-events-endpoint] The managed intake service exposes the following endpoints for Elastic APM agent data intake: @@ -85,7 +78,6 @@ The managed intake service exposes the following endpoints for Elastic APM agent | --- | --- | | APM agent event intake | `/intake/v2/events` | - ### Request [api-events-example] Send an `HTTP POST` request to the managed intake service `intake/v2/events` endpoint: @@ -105,15 +97,12 @@ Since asynchronous processing defers some of the event processing to the backgro :::: - - ### Response [api-events-response] On success, the server will respond with a 202 Accepted status code and no body. Keep in mind that events can succeed and fail independently of each other. Only if all events succeed does the server respond with a 202. - ### API Errors [api-events-errors] There are two types of errors that the managed intake service may return to an agent: @@ -152,15 +141,12 @@ An example error response might look something like this: 3. An immediately returning non-event related error 4. The number of accepted events - If you’re developing an agent, these errors can be useful for debugging. - ### Event API Schemas [api-events-schema-definition] The managed intake service uses a collection of JSON Schemas for validating requests to the intake API. - ### Metadata [observability-apm-server-api-metadata] Every new connection to the managed intake service starts with a `metadata` stanza. This provides general metadata concerning the other objects in the stream. @@ -172,8 +158,6 @@ Metadata is stored under `context` when viewing documents in {{es}}. :::: - - #### Metadata Schema [metadata-schema] The managed intake service uses JSON Schema to validate requests. The specification for metadata is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/metadata.json) and included below. @@ -751,8 +735,6 @@ The managed intake service uses JSON Schema to validate requests. The specificat :::: - - #### Kubernetes data [kubernetes-data] APM agents automatically read Kubernetes data and send it to the managed intake service. In most instances, agents are able to read this data from inside the container. If this is not the case, or if you wish to override this data, you can set environment variables for the agents to read. These environment variable are set via the Kubernetes [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables). Here’s how you would add the environment variables to your Kubernetes pod spec: @@ -785,12 +767,10 @@ The table below maps these environment variables to the APM metadata event field | `KUBERNETES_NAMESPACE` | system.kubernetes.namespace | | `KUBERNETES_POD_UID` | system.kubernetes.pod.uid | - ### Transactions [observability-apm-server-api-transactions] Transactions are events corresponding to an incoming request or similar task occurring in a monitored service. - #### Transaction Schema [api-transaction-schema] The managed intake service uses JSON Schema to validate requests. The specification for transactions is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/transaction.json) and included below. @@ -1932,13 +1912,10 @@ The managed intake service uses JSON Schema to validate requests. The specificat :::: - - ### Spans [observability-apm-server-api-spans] Spans are events captured by an agent occurring in a monitored service. - #### Span Schema [api-span-schema] The managed intake service uses JSON Schema to validate requests. The specification for spans is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/span.json) and included below. @@ -2852,13 +2829,10 @@ The managed intake service uses JSON Schema to validate requests. The specificat :::: - - ### Errors [observability-apm-server-api-errors] An error or a logged error message captured by an agent occurring in a monitored service. - #### Error Schema [api-error-schema] The managed intake service uses a JSON Schema to validate requests. The specification for errors is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/error.json) and included below. @@ -4162,13 +4136,10 @@ The managed intake service uses a JSON Schema to validate requests. The specific :::: - - ### Metrics [observability-apm-server-api-metrics] Metrics contain application metric data captured by an {{apm-agent}}. - #### Metric Schema [api-metricset-schema] The managed intake service uses JSON Schema to validate requests. The specification for metrics is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/metricset.json) and included below. @@ -4480,8 +4451,6 @@ The managed intake service uses JSON Schema to validate requests. The specificat :::: - - ## OpenTelemetry API [observability-apm-server-api-opentelemetry-api] Elastic supports receiving traces, metrics, and logs over the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/). OTLP is the default transfer protocol for OpenTelemetry and is supported natively by the managed intake service. @@ -4491,7 +4460,6 @@ The managed intake service supports two OTLP communication protocols on the same * OTLP/HTTP (protobuf) * OTLP/gRPC - ### OTLP/gRPC paths [otlpgrpc-paths] | Name | Endpoint | @@ -4500,7 +4468,6 @@ The managed intake service supports two OTLP communication protocols on the same | OTLP trace intake | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | | OTLP logs intake | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | - ### OTLP/HTTP paths [otlphttp-paths] | Name | Endpoint | @@ -4514,4 +4481,3 @@ See our [OpenTelemetry docs](upstream-opentelemetry-collectors-language-sdks.md) :::: - diff --git a/solutions/observability/apps/metadata.md b/solutions/observability/apps/metadata.md index 9cd46c5394..bc6b593a6e 100644 --- a/solutions/observability/apps/metadata.md +++ b/solutions/observability/apps/metadata.md @@ -10,7 +10,6 @@ applies_to: Metadata can enrich your events and make application performance monitoring even more useful. Let’s explore the different types of metadata that Elastic APM offers. - ## Labels [apm-data-model-labels] Labels add **indexed** information to transactions, spans, and errors. Indexed means the data is searchable and aggregatable in {{es}}. Add additional key-value pairs to define multiple labels. @@ -26,8 +25,6 @@ Label values can be a string, boolean, or number, although some agents only supp Avoid defining too many user-specified labels. Defining too many unique fields in an index is a condition that can lead to a [mapping explosion](../../../manage-data/data-store/mapping.md#mapping-limit-settings). :::: - - ### Agent API reference [_agent_api_reference] * Go: [`SetLabel`](apm-agent-go://reference/api-documentation.md#context-set-label) @@ -39,7 +36,6 @@ Avoid defining too many user-specified labels. Defining too many unique fields i * Ruby: [`set_label`](apm-agent-ruby://reference/api-reference.md#api-agent-set-label) * Rum: [`addLabels`](apm-agent-rum-js://reference/agent-api.md#apm-add-labels) - ## Custom context [apm-data-model-custom] Custom context adds **non-indexed**, custom contextual information to transactions and errors. Non-indexed means the data is not searchable or aggregatable in {{es}}, and you cannot build dashboards on top of the data. This also means you don’t have to worry about [mapping explosions](../../../manage-data/data-store/mapping.md#mapping-limit-settings), as these fields are not added to the mapping. @@ -55,8 +51,6 @@ Non-indexed information is useful for providing contextual information to help y Setting a circular object, a large object, or a non JSON serializable object can lead to errors. :::: - - ### Agent API reference [_agent_api_reference_2] * Go: [`SetCustom`](apm-agent-go://reference/api-documentation.md#context-set-custom) @@ -69,7 +63,6 @@ Setting a circular object, a large object, or a non JSON serializable object can * Ruby: [`set_custom_context`](apm-agent-ruby://reference/api-reference.md#api-agent-set-custom-context) * Rum: [`setCustomContext`](apm-agent-rum-js://reference/agent-api.md#apm-set-custom-context) - ## User context [apm-data-model-user] User context adds **indexed** user information to transactions and errors. Indexed means the data is searchable and aggregatable in {{es}}. @@ -79,7 +72,6 @@ User context adds **indexed** user information to transactions and errors. Index * {{es}} fields: `user.email` | `user.name` | `user.id` * Applies to: [Transactions](transactions.md) | [Errors](errors.md) - ### Agent API reference [_agent_api_reference_3] * Go: [`SetUsername`](apm-agent-go://reference/api-documentation.md#context-set-user-email) diff --git a/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md b/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md index 131cbf2341..6f89782ca9 100644 --- a/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md +++ b/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md @@ -14,7 +14,6 @@ You should *not* use the Elastic Synthetics integration to set up new monitors a * With Elastic **{{project-monitors-cap}}**, you write tests in an external version-controlled project and use a CLI tool to push monitors to the {{stack}}. * The **{{synthetics-app}}** is an application in {{kib}} that you can use to configure and create monitors using a user interface. - ## Compare approaches [synthetics-migrate-integration-compare] Below is a comparison of how you used the {{agent}} integration to create monitors and how you’ll use the {{synthetics-app}} or projects to create monitors: @@ -32,7 +31,6 @@ Below is a comparison of how you used the {{agent}} integration to create monito * Your infrastructure using [{{private-location}}s](monitor-resources-on-private-networks.md) * Elastic’s global managed infrastructure - **Where you configure monitors**: * **{{agent}} integration**: You could configure monitors using: @@ -45,7 +43,6 @@ Below is a comparison of how you used the {{agent}} integration to create monito * A user interface in {{kib}} (all lightweight monitors, browser monitors via inline script only) * Code in an external, version-controlled project (all monitor types) - **How to use projects ([read more](#synthetics-migrate-integration-projects))**: * **{{agent}} integration**: @@ -62,7 +59,6 @@ Below is a comparison of how you used the {{agent}} integration to create monito 3. Write journeys in JavaScript or TypeScript files and configure individual monitors in your journey code using `monitor.use` or configure all monitors using the `synthetics.config.ts` file. 4. Use the `elastic/synthetics push` command to create monitors. - Find more details in [Use {{project-monitors-cap}}](create-monitors-with-project-monitors.md). **How to use the UI ([read more](#synthetics-migrate-integration-ui))**: @@ -82,10 +78,8 @@ Find more details in [Use {{project-monitors-cap}}](create-monitors-with-project 4. Configure the monitor. 5. Create the monitor. - Find more details in [Use the {{synthetics-app}}](create-monitors-in-synthetics-app.md). - ## Where monitors run [synthetics-migrate-integration-location] If you want to continue hosting on your infrastructure, you will need to create a {{private-location}} before creating monitors. If you have already have an {{agent}} running using `elastic-agent-complete`, you can [add it as a new {{private-location}}](monitor-resources-on-private-networks.md#synthetics-private-location-add) in the {{synthetics-app}}. To create a new {{private-location}} from scratch, follow all instructions in [Monitor resources on private networks](monitor-resources-on-private-networks.md). @@ -94,7 +88,6 @@ Alternatively, you can start hosting on Elastic’s global managed infrastructur Executing synthetic tests on Elastic’s global managed testing infrastructure incurs an additional charge. Tests are charged under one of two new billing dimensions depending on the monitor type. For *browser monitor* usage, there is a fee per test run. For *lightweight monitor* usage, there is a fee per region in which you run any monitors regardless of the number of test runs. For more details, refer to [full details and current pricing](https://www.elastic.co/pricing). - ## How to use projects [synthetics-migrate-integration-projects] If you already have an external project you were adding via a ZIP URL you can use the same project, but you will have to make some changes. @@ -128,7 +121,6 @@ First, upgrade the existing project to use the latest version of `@elastic/synth 4. `.github/` will contain sample workflow files to use with GitHub Actions. - Then, you can further configure monitors as needed. In the upgraded project, you’ll use code (instead of the Integrations UI) to define settings like the name of the monitor and the frequency at which it will run. There are two ways you can configure monitors using code: * For individual monitors, use `monitor.use` directly in the journey code. Read more in [Configure individual monitors](configure-individual-browser-monitors.md). @@ -140,7 +132,6 @@ Optionally, you can also add lightweight monitors to the project in YAML files. For more information on getting started with projects, refer to [Use {{project-monitors-cap}}](create-monitors-with-project-monitors.md). - ## How to use the UI [synthetics-migrate-integration-ui] If you created monitors solely via the Elastic Synthetics integration UI, you can recreate monitors in the {{synthetics-app}}. diff --git a/solutions/observability/apps/mobile-service-overview.md b/solutions/observability/apps/mobile-service-overview.md index 9160fe7808..a0f0337cb3 100644 --- a/solutions/observability/apps/mobile-service-overview.md +++ b/solutions/observability/apps/mobile-service-overview.md @@ -22,31 +22,26 @@ For example, see: All of these metrics & insights can help SREs and developers better understand the health of their mobile application environment and the impact of backend errors and bottlenecks on end-user experience. - ## Quick stats [mobile-service-stats] Understand the impact of slow application load times and variations in application crash rate on user traffic (coming soon). Visualize session and HTTP trends, and see where your users are located—​enabling you to optimize your infrastructure deployment and routing topology. Note: due to the way crash rate is calculated (crashes per session) it is possible to have greater than 100% rate, due to fact that a session may contain multiple crashes. - :::{image} /solutions/images/observability-mobile-location.png :alt: mobile service overview centered on location map :screenshot: ::: - ## Most used [mobile-service-most-used] Optimize your end-user experience and your application QA strategy based on your most used device models and operating system versions. - :::{image} /solutions/images/observability-mobile-most-used.png :alt: mobile service overview showing most used devices :screenshot: ::: - ## Throughput and transactions [mobile-throughput-transactions] The **Throughput** chart visualizes the average number of transactions per minute for the selected service. @@ -55,13 +50,11 @@ The **Transactions** table displays a list of *transaction groups* for the selec By default, transaction groups are sorted by *Impact* to show the most used and slowest endpoints in your service. If there is a particular endpoint you are interested in, click **View transactions** to view a list of similar transactions on the [transactions overview](transactions-2.md) page. - :::{image} /solutions/images/observability-traffic-transactions.png :alt: Traffic and transactions :screenshot: ::: - ## Failed transaction rate and dependencies [mobile-error-and-dependencies] The failed transaction rate represents the percentage of failed transactions from the perspective of the selected service. It’s useful for visualizing unexpected increases, decreases, or irregular patterns in a service’s transactions. @@ -75,21 +68,17 @@ If there is no HTTP status, both transactions and spans are considered successfu :::: - The **Dependencies** table displays a list of downstream services or external connections relevant to the service at the selected time range. The table displays latency, throughput, failed transaction rate, and the impact of each dependency. By default, dependencies are sorted by *Impact* to show the most used and the slowest dependency. If there is a particular dependency you are interested in, click **[View dependencies](dependencies.md)** to learn more about it. - ::::{note} Displaying dependencies for services instrumented with the Real User Monitoring (RUM) agent requires an agent version ≥ v5.6.3. :::: - :::{image} /solutions/images/observability-spans-dependencies.png :alt: Span type duration and dependencies :screenshot: ::: - :::{image} /solutions/images/observability-mobile-tp.png :alt: mobile service overview showing latency :screenshot: diff --git a/solutions/observability/apps/monitor-apm-server-binary.md b/solutions/observability/apps/monitor-apm-server-binary.md index 20e28434d6..32fde3bd29 100644 --- a/solutions/observability/apps/monitor-apm-server-binary.md +++ b/solutions/observability/apps/monitor-apm-server-binary.md @@ -8,7 +8,6 @@ applies_to: # Monitor the APM Server binary [apm-monitoring] - There are two methods to monitor the APM Server binary. Make sure monitoring is enabled on your {{es}} cluster, then configure one of these methods to collect APM Server metrics: * [Internal collection](use-internal-collection-to-send-monitoring-data.md) - Internal collectors send monitoring data directly to your monitoring cluster. diff --git a/solutions/observability/apps/monitor-apm-server.md b/solutions/observability/apps/monitor-apm-server.md index fc8f14fe29..28b95ba90e 100644 --- a/solutions/observability/apps/monitor-apm-server.md +++ b/solutions/observability/apps/monitor-apm-server.md @@ -15,7 +15,6 @@ Select your deployment method to get started: * [Fleet-managed](monitor-fleet-managed-apm-server.md) * [APM Server binary](monitor-apm-server-binary.md) - ## {{ecloud}} [apm-monitor-apm-cloud] {{ecloud}} manages the installation and configuration of a monitoring agent for you — so all you have to do is flip a switch and watch the data pour in. diff --git a/solutions/observability/apps/monitor-fleet-managed-apm-server.md b/solutions/observability/apps/monitor-fleet-managed-apm-server.md index f022329e55..315e302b86 100644 --- a/solutions/observability/apps/monitor-fleet-managed-apm-server.md +++ b/solutions/observability/apps/monitor-fleet-managed-apm-server.md @@ -6,22 +6,17 @@ applies_to: stack: all --- - - # Monitor a Fleet-managed APM Server [apm-monitor-apm-self-install] - ::::{note} This guide assumes you are already ingesting APM data into the {{stack}}. :::: - In 8.0 and later, you can use {{metricbeat}} to collect data about APM Server and ship it to a monitoring cluster. To collect and ship monitoring data: 1. [Configure {{agent}} to send monitoring data](#apm-configure-ea-monitoring-data) 2. [Install and configure {{metricbeat}} to collect monitoring data](#apm-install-config-metricbeat) - ## Configure {{agent}} to send monitoring data [apm-configure-ea-monitoring-data] ::::{admonition} @@ -29,7 +24,6 @@ Before you can monitor APM, you must have monitoring data for the {{es}} product :::: - 1. Enable monitoring of {{agent}} by adding the following settings to your `elastic-agent.yml` configuration file: ```yaml @@ -72,10 +66,8 @@ Before you can monitor APM, you must have monitoring data for the {{es}} product [global-flags] ``` - See the [{{agent}} command reference](/reference/fleet/agent-command-reference.md) for more information on the enroll command. - ## Install and configure {{metricbeat}} to collect monitoring data [apm-install-config-metricbeat] 1. Install {{metricbeat}} on the same server as {{agent}}. To learn how, see [Get started with {{metricbeat}}](beats://reference/metricbeat/metricbeat-installation-configuration.md). If you already have {{metricbeat}} installed, skip this step. @@ -134,7 +126,6 @@ See the [{{agent}} command reference](/reference/fleet/agent-command-reference.m In production environments, you should send your deployment logs and metrics to a dedicated monitoring deployment (referred to as the *monitoring cluster*). Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. :::: - For example, specify the {{es}} output information in the {{metricbeat}} configuration file (`metricbeat.yml`): ```yaml @@ -152,14 +143,12 @@ See the [{{agent}} command reference](/reference/fleet/agent-command-reference.m 1. In this example, the data is stored on a monitoring cluster with nodes `es-mon-1` and `es-mon-2`. 2. Specify one of `api_key` or `username`/`password`. - If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200`. ::::{important} The {{es}} {{monitor-features}} use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node. :::: - If the {{es}} {{security-features}} are enabled on the monitoring cluster, you must provide a valid user ID and password so that {{metricbeat}} can send metrics successfully: 1. Create a user on the monitoring cluster that has the `remote_monitoring_agent` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). diff --git a/solutions/observability/apps/monitor-resources-on-private-networks.md b/solutions/observability/apps/monitor-resources-on-private-networks.md index d57011f80b..c3f97e2430 100644 --- a/solutions/observability/apps/monitor-resources-on-private-networks.md +++ b/solutions/observability/apps/monitor-resources-on-private-networks.md @@ -16,14 +16,12 @@ To monitor resources on private networks you can either: {{private-location}}s via Elastic Agent require only outbound connections from your network, while allowing Elastic’s global managed infrastructure to access a private endpoint requires inbound access, thus posing an additional risk that users must assess. - ## Allow access to your private network [monitor-via-access-control] To give Elastic’s global managed infrastructure access to a private endpoint, use IP address filtering, HTTP authentication, or both. To grant access via IP, use [this list of egress IPs](https://manifest.synthetics.elastic-cloud.com/v1/ip-ranges.json). The addresses and locations on this list may change, so automating updates to filtering rules is recommended. IP filtering alone will allow all users of Elastic’s global managed infrastructure access to your endpoints, if this is a concern consider adding additional protection via user/password authentication via a proxy like nginx. - ## Monitor via a private agent [monitor-via-private-agent] {{private-location}}s allow you to run monitors from your own premises. Before running a monitor on a {{private-location}}, you’ll need to: @@ -37,7 +35,6 @@ To grant access via IP, use [this list of egress IPs](https://manifest.synthetic :::: - ## Set up {{fleet-server}} and {{agent}} [synthetics-private-location-fleet-agent] Start by setting up {{fleet-server}} and {{agent}}: @@ -52,7 +49,6 @@ By default {{private-location}}s are configured to allow two simultaneous browse :::: - ## Connect to the {{stack}} or your Observability Serverless project [synthetics-private-location-connect] After setting up {{fleet}}, you’ll connect {{fleet}} to the {{stack}} or your Observability Serverless project and enroll an {{agent}} in {{fleet}}. @@ -64,7 +60,6 @@ The `elastic-agent-complete` Docker image is the only way to have all available :::: - To pull the Docker image run: ```sh @@ -90,13 +85,11 @@ The `elastic-agent-complete` Docker image requires additional capabilities to op :::: - ::::{note} You may need to set other environment variables. Learn how in [{{agent}} environment variables guide](/reference/fleet/agent-environment-variables.md). :::: - ## Add a {{private-location}} [synthetics-private-location-add] When the {{agent}} is running you can add a new {{private-location}} in the UI: @@ -112,7 +105,6 @@ When the {{agent}} is running you can add a new {{private-location}} in the UI: It is not currently possible to use custom CAs for synthetics browser tests in private locations without following a workaround. To learn more about the workaround, refer to the following GitHub issue: [elastic/synthetics#717](https://github.com/elastic/synthetics/issues/717). :::: - ## Scaling {{private-location}}s [synthetics-private-location-scaling] By default {{private-location}}s are configured to allow two simultaneous browser tests, and an unlimited number of lightweight checks. These limits can be set via the environment variables `SYNTHETICS_LIMIT_{{TYPE}}`, where `{{TYPE}}` is one of `BROWSER`, `HTTP`, `TCP`, and `ICMP` for the container running the {{agent}} docker image. @@ -121,7 +113,6 @@ It is critical to allocate enough memory and CPU capacity to handle configured l These limits are for simultaneous tests, not total tests. For example, if 60 browser tests were scheduled to run once per hour and each took 1 minute to run, that would fully occupy one execution slot. However, it is a good practice to set up execution slots with extra capacity. A good starting point would be to over-allocate by a factor of 5. In the previous example that would mean allocating 5 slots. - ## Next steps [synthetics-private-location-next] Now you can add monitors to your {{private-location}} in [the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md) or using the [Elastic Synthetics library’s `push` method](../../../solutions/observability/apps/create-monitors-with-project-monitors.md). \ No newline at end of file diff --git a/solutions/observability/apps/monitoring-aws-lambda-functions.md b/solutions/observability/apps/monitoring-aws-lambda-functions.md index e9398ce822..90c664524b 100644 --- a/solutions/observability/apps/monitoring-aws-lambda-functions.md +++ b/solutions/observability/apps/monitoring-aws-lambda-functions.md @@ -10,7 +10,6 @@ applies_to: # Monitoring AWS Lambda Functions [apm-monitoring-aws-lambda] - Elastic APM lets you monitor your AWS Lambda functions. The natural integration of [distributed tracing](../../../solutions/observability/apps/traces.md#apm-distributed-tracing) into your AWS Lambda functions provides insights into the function’s execution and runtime behavior as well as its relationships and dependencies to other services. ## AWS Lambda architecture [aws-lambda-arch] diff --git a/solutions/observability/apps/multi-factor-authentication-mfa-for-browser-monitors.md b/solutions/observability/apps/multi-factor-authentication-mfa-for-browser-monitors.md index 0dd1224cc1..ac80ba8ae8 100644 --- a/solutions/observability/apps/multi-factor-authentication-mfa-for-browser-monitors.md +++ b/solutions/observability/apps/multi-factor-authentication-mfa-for-browser-monitors.md @@ -10,12 +10,10 @@ applies_to: # Multi-factor Authentication (MFA) for browser monitors [synthetics-mfa] - Multi-factor Authentication (MFA) adds an essential layer of security to applications login processes, protecting against unauthorized access. A very common use case in Synthetics is testing user journeys involving websites protected by MFA. Synthetics supports testing websites secured by Time-based One-Time Password (TOTP), a common MFA method that provides short-lived one-time tokens to enhance security. - ## Configuring TOTP for MFA [configuring_totp_for_mfa] To test a browser journey that uses TOTP for MFA, first configure the Synthetics authenticator token in the target application. To do this, generate a One-Time Password (OTP) using the Synthetics CLI; refer to [`@elastic/synthetics totp `](../../../solutions/observability/apps/use-synthetics-cli.md). @@ -27,7 +25,6 @@ npx @elastic/synthetics totp OTP Token: 123456 ``` - ## Applying the TOTP Token in Browser Journeys [applying_the_totp_token_in_browser_journeys] Once the Synthetics TOTP Authentication is configured in your application, you can now use the OTP token in the synthetics browser journeys using the `mfa` object imported from `@elastic/synthetics`. diff --git a/solutions/observability/apps/observe-lambda-functions.md b/solutions/observability/apps/observe-lambda-functions.md index ce583543bd..67119a29f3 100644 --- a/solutions/observability/apps/observe-lambda-functions.md +++ b/solutions/observability/apps/observe-lambda-functions.md @@ -18,26 +18,22 @@ To set up Lambda monitoring, refer to [AWS Lambda functions](/solutions/observab :screenshot: ::: - ## Cold starts [apm-lambda-cold-start-info] A cold start occurs when a Lambda function has not been used for a certain period of time. A lambda worker receives a request to run the function and prepares an execution environment. Cold starts are an unavoidable byproduct of the serverless world, but visibility into how they impact your services can help you make better decisions about factors like how much memory to allocate to a function, whether to enable provisioned concurrency, or if it’s time to consider removing a large dependency. - ### Cold start rate [apm-lambda-cold-start-rate] The cold start rate (i.e. proportion of requests that experience a cold start) is displayed per service and per transaction. Cold start is also displayed in the trace waterfall, where you can drill-down into individual traces and see trace metadata like AWS request ID, trigger type, and trigger request ID. - ### Latency distribution correlation [apm-lambda-cold-start-latency] The [latency correlations](../../../solutions/observability/apps/find-transaction-latency-failure-correlations.md) feature can be used to visualize the impact of Lambda cold starts on latency—​just select the `faas.coldstart` field. - ## AWS Lambda function grouping [apm-lambda-service-config] The default APM agent configuration results in one APM service per AWS Lambda function, where the Lambda function name is the service name. diff --git a/solutions/observability/apps/opentelemetry-intake-api.md b/solutions/observability/apps/opentelemetry-intake-api.md index f051269423..815a0c065f 100644 --- a/solutions/observability/apps/opentelemetry-intake-api.md +++ b/solutions/observability/apps/opentelemetry-intake-api.md @@ -14,7 +14,6 @@ APM Server supports two OTLP communication protocols on the same port: * OTLP/HTTP (protobuf) * OTLP/gRPC - ## OTLP/gRPC paths [_otlpgrpc_paths] | Name | Endpoint | @@ -23,7 +22,6 @@ APM Server supports two OTLP communication protocols on the same port: | OTLP trace intake | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | | OTLP logs intake | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | - ## OTLP/HTTP paths [_otlphttp_paths] | Name | Endpoint | @@ -36,4 +34,3 @@ APM Server supports two OTLP communication protocols on the same port: See our OpenTelemetry documentation to learn how to send data to the APM Server from an [OpenTelemetry agent](upstream-opentelemetry-collectors-language-sdks.md#apm-instrument-apps-otel) or [OpenTelemetry collector](upstream-opentelemetry-collectors-language-sdks.md#apm-connect-open-telemetry-collector). :::: - diff --git a/solutions/observability/apps/overviews.md b/solutions/observability/apps/overviews.md index 80e991c331..53d4251f19 100644 --- a/solutions/observability/apps/overviews.md +++ b/solutions/observability/apps/overviews.md @@ -10,7 +10,6 @@ applies_to: # High-level overviews of application data [apm-ui] - For a quick, high-level overview of the health and performance of your application, start with: * [Services](../../../solutions/observability/apps/services.md) @@ -18,7 +17,6 @@ For a quick, high-level overview of the health and performance of your applicati * [Dependencies](../../../solutions/observability/apps/dependencies.md) * [Service Map](../../../solutions/observability/apps/service-map.md) - ## View an individual service [apm-ui-individual-service] ```{applies_to} stack: all diff --git a/solutions/observability/apps/parse-data-using-ingest-pipelines.md b/solutions/observability/apps/parse-data-using-ingest-pipelines.md index 6055144a4b..354b385237 100644 --- a/solutions/observability/apps/parse-data-using-ingest-pipelines.md +++ b/solutions/observability/apps/parse-data-using-ingest-pipelines.md @@ -11,7 +11,6 @@ Ingest pipelines preprocess and enrich APM documents before indexing them. For e The default APM pipelines are defined in {{es}} apm-data plugin index templates. {{es}} then uses the index pattern in these index templates to match pipelines to APM data streams. - ## Custom ingest pipelines [custom-ingest-pipelines] Elastic APM supports custom ingest pipelines. A custom pipeline allows you to transform data to better match your specific use case. This can be useful, for example, to ensure data security by removing or obfuscating sensitive information. @@ -20,7 +19,6 @@ Each data stream ships with a default pipeline. This default pipeline calls an i In addition, ingest pipelines can also be used to direct application metrics (`metrics-apm.app.*`) to a data stream with a different dataset, e.g. to combine metrics for two applications. Sending other APM data to alternate data streams, like traces (`traces-apm.*`), logs (`logs-apm.*`), and internal metrics (`metrics-apm.internal*`) is not currently supported. - ## `@custom` ingest pipeline naming convention [custom-ingest-pipeline-naming] `@custom` pipelines are specific to each data stream and follow a similar naming convention: `-@custom`. As a reminder, the default APM data streams are: @@ -40,7 +38,6 @@ To match a custom ingest pipeline with a data stream, follow the `-[7](https://web.dev/cls/)] @@ -113,8 +102,6 @@ Cumulative layout shift (CLS) [Beginning in May 2021](https://webmasters.googleblog.com/2020/11/timing-for-page-experience.md), Google will start using Core Web Vitals as part of their ranking algorithm and will open up the opportunity for websites to rank in the "top stories" position without needing to leverage [AMP](https://amp.dev/). [8](https://webmasters.googleblog.com/2020/05/evaluating-page-experience.md)] :::: - - ### Load/view distribution [user-experience-distribution] Operating system, browser family, and geographic location can all have a massive impact on how visitors experience your website. This data can help you understand when and where your users are visiting from, and can help you prioritize optimizations — for example, prioritizing improvements for the most popular browsers visiting your site. @@ -126,7 +113,6 @@ Don’t forget, this data also influences search engine page rankings and placem :screenshot: ::: - ### Error breakdown [user-experience-errors] JavaScript errors can be detrimental to a users experience on your website. But variation in users' software and hardware makes it nearly impossible to test for every combination. And, as JavaScript continues to get more and more complex, the need for user experience monitoring and error reporting only increases. Error monitoring makes this visible by surfacing JavaScript errors that are occurring on your website in production. @@ -138,11 +124,9 @@ JavaScript errors can be detrimental to a users experience on your website. But Open error messages in APM for additional analysis tools, like occurrence rates, transaction ids, user data, and more. - ### Feedback and troubleshooting [user-experience-feedback] Have a question? Want to leave feedback? Visit the [{{user-experience}} discussion forum](https://discuss.elastic.co/c/observability/user-experience/87). - #### References [user-experience-references] diff --git a/solutions/observability/apps/reduce-storage.md b/solutions/observability/apps/reduce-storage.md index df19ecccbe..f86787aaf8 100644 --- a/solutions/observability/apps/reduce-storage.md +++ b/solutions/observability/apps/reduce-storage.md @@ -11,21 +11,18 @@ applies_to: The richness and volume of APM data provides unique insights into your applications, but it can also mean higher costs and more noise when analyzing data. There are a couple strategies you can use to reduce your data usage while continuing to get the full value of APM data. - ## Reduce the sample rate [apm-reduce-sample-rate] Distributed tracing can generate a substantial amount of data. More data can mean higher costs and more noise. Sampling aims to lower the amount of data ingested and the effort required to analyze that data. See [Transaction sampling](../../../solutions/observability/apps/transaction-sampling.md) to learn more. - ## Enable span compression [enable_span_compression] In some cases, APM agents may collect large amounts of very similar or identical spans in a transaction. These repeated, similar spans often don’t provide added benefit, especially if they are of very short duration. Span compression takes these similar spans and compresses them into a single span-- retaining important information but reducing processing and storage overhead. See [Span compression](/solutions/observability/apps/spans.md#apm-spans-span-compression) to learn more. - ## Reduce collected stack trace information [observability-apm-reduce-stacktrace] Elastic APM agents collect `stacktrace` information under certain circumstances. This can be very helpful in identifying issues in your code, but it also comes with an overhead at collection time and increases your storage usage. @@ -49,12 +46,10 @@ Depending on your use case, you can delete data: If you want to delete data for security or privacy reasons, see [Secure data](../../../solutions/observability/apps/application-data-security.md). - ### Delete data with {{ilm}} ({{ilm-init}}) [apm-delete-data-with-ilm] Index lifecycle management enables you to automate how you want to manage your indices over time. You can base actions on factors such as shard size and performance requirements. See [{{ilm-cap}}](../../../solutions/observability/apps/index-lifecycle-management.md) to learn more. - ### Delete data matching a query [apm-delete-data-query] You can delete all APM documents matching a specific query with the [Delete By Query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-delete-by-query). For example, to delete all documents with a given `service.name`, use the following request: @@ -72,14 +67,12 @@ POST /.ds-*-apm*/_delete_by_query } ``` - ### Delete data with {{kib}} Index Management [apm-delete-data-in-kibana] {{kib}}'s [Index Management](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) allows you to manage your cluster’s indices, data streams, index templates, and much more. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Select **Data Streams**. Select the data streams you want to delete, and click **Delete data streams**. - ## Update existing data [apm-update-data] ```{applies_to} stack: all diff --git a/solutions/observability/apps/scale-architect-synthetics-deployment.md b/solutions/observability/apps/scale-architect-synthetics-deployment.md index d7453d027a..73a93434cf 100644 --- a/solutions/observability/apps/scale-architect-synthetics-deployment.md +++ b/solutions/observability/apps/scale-architect-synthetics-deployment.md @@ -25,12 +25,10 @@ You may, however, use Dashboards and the Discover feature with CCS to view `synt The Synthetics UI does *not* work with CCS/CCR because it would limit the rich user experience that the Synthetics UI provides. Unlike the majority of {{kib}} apps, the Synthetics UI relies heavily on state stored in {{kib}} shared objects in order to provide a rich user experience. Because {{kib}} saved objects cannot be shared via CCS/CCR, the Synthetics UI will not show any user data even if CCS/CCR is configured. - ## Manage large numbers of Synthetic monitors with tags [synthetics-tagging] When managing larger numbers of synthetic monitors, use tags to keep them organized. Many of the views in the Synthetics UI are tag-aware and can group data by tag. - ## Create custom dashboards [synthetics-custom-dashboards] If we don't provide a UI for your exact needs, you can use [dashboards](../../../explore-analyze/dashboards.md) to build custom visualizations. For a complete accounting of fields used by the Synthetics UI, refer to [{{heartbeat}}'s exported fields](beats://reference/heartbeat/exported-fields.md). \ No newline at end of file diff --git a/solutions/observability/apps/secret-token.md b/solutions/observability/apps/secret-token.md index 28f4c91e9a..32c20644e8 100644 --- a/solutions/observability/apps/secret-token.md +++ b/solutions/observability/apps/secret-token.md @@ -11,7 +11,6 @@ applies_to: Secret tokens are sent as plain-text, so they only provide security when used in combination with [TLS](apm-agent-tls-communication.md). :::: - When defined, secret tokens are used to authorize requests to the APM Server. Both the {{apm-agent}} and APM Server must be configured with the same secret token for the request to be accepted. To secure the communication between APM agents and the APM Server with a secret token: @@ -24,8 +23,6 @@ To secure the communication between APM agents and the APM Server with a secret Secret tokens are not applicable for the RUM Agent, as there is no way to prevent them from being publicly exposed. :::: - - ## Create a secret token [apm-create-secret-token] % This should be checked and updated, it doesn't looks like it's there by default vv @@ -34,7 +31,6 @@ Secret tokens are not applicable for the RUM Agent, as there is no way to preven {{ech}} and {{ece}} deployments provision a secret token when the deployment is created. The secret token can be found and reset in the {{ecloud}} Console under the deployment's **APM & Fleet** page. :::: - :::::::{tab-set} ::::::{tab-item} Fleet-managed diff --git a/solutions/observability/apps/secrets-keystore-for-secure-settings.md b/solutions/observability/apps/secrets-keystore-for-secure-settings.md index 559d79fbab..d9445b30db 100644 --- a/solutions/observability/apps/secrets-keystore-for-secure-settings.md +++ b/solutions/observability/apps/secrets-keystore-for-secure-settings.md @@ -6,16 +6,12 @@ applies_to: stack: all --- - - # Secrets keystore for secure settings [apm-keystore] - ::::{important} The APM Server keystore only applies to the APM Server binary installation method. :::: - When you configure APM Server, you might need to specify sensitive settings, such as passwords. Rather than relying on file system permissions to protect these values, you can use the APM Server keystore to securely store secret values for use in configuration settings. After adding a key and its secret value to the keystore, you can use the key in place of the secret value when you configure sensitive settings. @@ -41,8 +37,6 @@ To create and manage keys, use the `keystore` command. See the [command referenc The `keystore` command must be run by the same user who will run APM Server. :::: - - ## Create a keystore [apm-creating-keystore] To create a secrets keystore, use: @@ -53,7 +47,6 @@ apm-server keystore create APM Server creates the keystore in the directory defined by the `path.data` configuration setting. - ## Add keys [apm-add-keys-to-keystore] To store sensitive values, such as authentication credentials for {{es}}, use the `keystore add` command: @@ -76,7 +69,6 @@ To pass the value through stdin, use the `--stdin` flag. You can also use `--for cat /file/containing/setting/value | apm-server keystore add ES_PWD --stdin --force ``` - ## List keys [apm-list-settings] To list the keys defined in the keystore, use: @@ -85,7 +77,6 @@ To list the keys defined in the keystore, use: apm-server keystore list ``` - ## Remove keys [apm-remove-settings] To remove a key from the keystore, use: diff --git a/solutions/observability/apps/secure-access-to-applications-ui.md b/solutions/observability/apps/secure-access-to-applications-ui.md index a600cdcced..1ad46f76a3 100644 --- a/solutions/observability/apps/secure-access-to-applications-ui.md +++ b/solutions/observability/apps/secure-access-to-applications-ui.md @@ -8,7 +8,6 @@ applies_to: # Secure access to the Applications UI [apm-app-users] - Use role-based access control to grant users access to secured resources. The roles that you set up depend on your organization’s security requirements and the minimum privileges required to use specific features. {{es-security-features}} provides [built-in roles](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) that grant a subset of the privileges needed by APM users. When possible, assign users the built-in roles to minimize the affect of future changes on your security strategy. If no built-in role is available, you can assign users the privileges needed to accomplish a specific task. In general, there are three types of privileges you’ll work with: diff --git a/solutions/observability/apps/secure-communication-with-apm-agents.md b/solutions/observability/apps/secure-communication-with-apm-agents.md index 3aed4796d3..b875edeb53 100644 --- a/solutions/observability/apps/secure-communication-with-apm-agents.md +++ b/solutions/observability/apps/secure-communication-with-apm-agents.md @@ -17,7 +17,3 @@ As soon as an authenticated communication is enabled, requests without a valid t In some use-cases, like when an {{apm-agent}} is running on the client side, authentication is not possible. See [Anonymous authentication](anonymous-authentication.md) for more information. - - - - diff --git a/solutions/observability/apps/secure-communication-with-elastic-stack.md b/solutions/observability/apps/secure-communication-with-elastic-stack.md index 6e51dec046..587bf885a8 100644 --- a/solutions/observability/apps/secure-communication-with-elastic-stack.md +++ b/solutions/observability/apps/secure-communication-with-elastic-stack.md @@ -11,24 +11,20 @@ applies_to: This documentation only applies to the APM Server binary. :::: - Use role-based access control or API keys to grant APM Server users access to secured resources. - ## Role-based access [apm-secure-comms-stack-role-based] Manage access on a feature-by-feature basis by creating several custom feature-related roles and assigning one or more of these roles to each APM Server user based on which features they need to access. [**Read more in Use feature roles →**](create-assign-feature-roles-to-apm-server-users.md) - ## API keys [apm-secure-comms-stack-api-keys] Instead of using usernames and passwords, you can use API keys to grant access to Elasticsearch resources. You can set API keys to expire at a certain time, and you can explicitly invalidate them. [**Read more in Grant access using API keys →**](grant-access-using-api-keys.md) - ## More resources [_more_resources] After privileged users have been created, use authentication to connect to a secured Elastic cluster. @@ -47,6 +43,3 @@ APM Server exposes an HTTP endpoint, and as with anything that opens ports on yo :::: - - - diff --git a/solutions/observability/apps/service-map.md b/solutions/observability/apps/service-map.md index 3b7db18c9e..90036a5ea7 100644 --- a/solutions/observability/apps/service-map.md +++ b/solutions/observability/apps/service-map.md @@ -16,12 +16,10 @@ We currently surface two types of service maps: * **Global**: All services instrumented with APM agents and the connections between them are shown. * **Service-specific**: Highlight connections for a selected service. - ## How do service maps work? [service-maps-how] Service Maps rely on distributed traces to draw connections between services. As [distributed tracing](/solutions/observability/apps/traces.md) is enabled out-of-the-box for supported technologies, so are service maps. However, if a service isn’t instrumented, or a `traceparent` header isn’t being propagated to it, distributed tracing will not work, and the connection will not be drawn on the map. - ## Visualize your architecture [visualize-your-architecture] Select the **Service Map** tab to get started. By default, all instrumented services and connections are shown. Whether you’re onboarding a new engineer, or just trying to grasp the big picture, drag things around, zoom in and out, and begin to visualize how your services are connected. @@ -30,13 +28,11 @@ Customize what the service map displays using either the query bar or the enviro If there’s a specific service that interests you, select that service to highlight its connections. Click **Focus map** to refocus the map on the selected service and lock the connection highlighting. Click the **Transactions** tab to jump to the Transaction overview for the selected service. You can also use the tabs at the top of the page to easily jump to the **Errors** or **Metrics** overview. - :::{image} /solutions/images/observability-service-maps-java.png :alt: Example view of service maps in the Applications UI in Kibana :screenshot: ::: - ## Anomaly detection with machine learning [service-map-anomaly-detection] You can create machine learning jobs to calculate anomaly scores on APM transaction durations within the selected service. When these jobs are active, service maps will display a color-coded anomaly indicator based on the detected anomaly score: @@ -56,7 +52,6 @@ If an anomaly has been detected, click **View anomalies** to view the anomaly de To learn how to create a machine learning job, refer to [Integrate with machine learning](../../../solutions/observability/apps/integrate-with-machine-learning.md). - ## Legend [service-maps-legend] Nodes appear on the map in one of two shapes: @@ -64,7 +59,6 @@ Nodes appear on the map in one of two shapes: * **Circle**: Instrumented services. Interior icons are based on the language of the APM agent used. * **Diamond**: Databases, external, and messaging. Interior icons represent the generic type, with specific icons for known entities, like Elasticsearch. Type and subtype are based on `span.type`, and `span.subtype`. - ## Supported APM agents [service-maps-supported] Service Maps are supported for the following APM agent versions: diff --git a/solutions/observability/apps/service-overview.md b/solutions/observability/apps/service-overview.md index f8eba3d080..4f9f985e64 100644 --- a/solutions/observability/apps/service-overview.md +++ b/solutions/observability/apps/service-overview.md @@ -37,7 +37,6 @@ Select the **Comparison** box to apply a time-based or expected bounds compariso The expected bounds comparison is powered by [machine learning](../../../solutions/observability/apps/integrate-with-machine-learning.md) and requires anomaly detection to be enabled. - ## Latency [service-latency] Response times for the service. You can filter the **Latency** chart to display the average, 95th, or 99th percentile latency times for the service. @@ -47,7 +46,6 @@ Response times for the service. You can filter the **Latency** chart to display :screenshot: ::: - ## Throughput and transactions [service-throughput-transactions] The **Throughput** chart visualizes the average number of transactions per minute for the selected service. @@ -56,7 +54,6 @@ The **Transactions** table displays a list of *transaction groups* for the selec By default, transaction groups are sorted by *Impact* to show the most used and slowest endpoints in your service. If there is a particular endpoint you are interested in, click **View transactions** to view a list of similar transactions on the [transactions overview](../../../solutions/observability/apps/transactions-2.md) page. - ## Failed transaction rate and errors [service-error-rates] The failed transaction rate represents the percentage of failed transactions from the perspective of the selected service. It’s useful for visualizing unexpected increases, decreases, or irregular patterns in a service’s transactions. @@ -70,7 +67,6 @@ If there is no HTTP status, both transactions and spans are considered successfu :::: - The **Errors** table provides a high-level view of each error message when it first and last occurred, along with the total number of occurrences. This makes it very easy to quickly see which errors affect your services and take actions to rectify them. To do so, click **View errors**. :::{image} /solutions/images/observability-error-rate.png @@ -78,7 +74,6 @@ The **Errors** table provides a high-level view of each error message when it fi :screenshot: ::: - ## Span types average duration and dependencies [service-span-duration] The **Time spent by span type** chart visualizes each span type’s average duration and helps you determine which spans could be slowing down transactions. The "app" label displayed under the chart indicates that something was happening within the application. This could signal that the APM agent does not have auto-instrumentation for whatever was happening during that time or that the time was spent in the application code and not in database or external requests. @@ -91,30 +86,25 @@ The **Dependencies** table displays a list of downstream services or external co **In Elastic Stack deployments**, displaying dependencies for services instrumented with the Real User Monitoring (RUM) agent requires an agent version ≥ v5.6.3. :::: - ## Cold start rate [service-cold-start] The cold start rate chart is specific to serverless services, and displays the percentage of requests that trigger a cold start of a serverless function. A cold start occurs when a serverless function has not been used for a certain period of time. Analyzing the cold start rate can be useful for deciding how much memory to allocate to a function, or when to remove a large dependency. The cold start rate chart is currently supported for [AWS Lambda](../../../solutions/observability/apps/observe-lambda-functions.md#apm-lambda-cold-start-info) functions and Azure functions. - ## Instances [service-instances] The **Instances** table displays a list of all the available service instances within the selected time range. Depending on how the service runs, the instance could be a host or a container. The table displays latency, throughput, failed transaction, CPU usage, and memory usage for each instance. By default, instances are sorted by *Throughput*. - :::{image} /solutions/images/observability-all-instances.png :alt: All instances :screenshot: ::: - ## Service metadata [service-metadata] To view metadata relating to the service agent, and if relevant, the container and cloud provider, click on each icon located at the top of the page beside the service name. - :::{image} /solutions/images/observability-metadata-icons.png :alt: Service metadata :screenshot: diff --git a/solutions/observability/apps/services.md b/solutions/observability/apps/services.md index 70b0eb4be5..832c9ab89b 100644 --- a/solutions/observability/apps/services.md +++ b/solutions/observability/apps/services.md @@ -26,7 +26,6 @@ In addition to health status, active alerts for each service are prominently dis Want to monitor service logs without instrumenting all your services? Learn about our [Inventory](../../../solutions/observability/apps/inventory.md). :::: - ## Service groups [service-groups] ::::{note} @@ -35,14 +34,12 @@ For Observability Serverless projects, the **Editor** role or higher is required :::: - ::::{important} The Service grouping functionality is in beta and is subject to change. The design and code is less mature than official generally available features and is being provided as-is with no warranties. :::: - Group services together to build meaningful views that remove noise, simplify investigations across services, and combine related alerts. :::{image} /solutions/images/observability-apm-service-group.png diff --git a/solutions/observability/apps/setup-role.md b/solutions/observability/apps/setup-role.md index 8bd84e4764..b8e8e30c91 100644 --- a/solutions/observability/apps/setup-role.md +++ b/solutions/observability/apps/setup-role.md @@ -38,9 +38,7 @@ Create a **setup role**, called something like `synthetics_setup`: | [Index](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `auto_configure` | Permits auto-creation of indices and data streams. | | [Index](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `monitor` | Gives access to all actions that are required for monitoring (recovery, segments info, index stats, and status). | - ::::{note} If users with the setup role also need to create, modify, and delete monitors, add the privileges defined in the [writer role](writer-role.md). :::: - diff --git a/solutions/observability/apps/ssltls-input-settings.md b/solutions/observability/apps/ssltls-input-settings.md index 1bf160abdd..ea734d4efd 100644 --- a/solutions/observability/apps/ssltls-input-settings.md +++ b/solutions/observability/apps/ssltls-input-settings.md @@ -14,7 +14,6 @@ Most options on this page are supported by all APM Server deployment methods. :::: - These settings apply to SSL/TLS communication between the APM Server and APM Agents. See [{{apm-agent}} TLS communication](apm-agent-tls-communication.md) to learn more. :::::::{tab-set} @@ -48,7 +47,6 @@ Enable or disable TLS. Disabled by default. | APM Server binary | `apm-server.ssl.enabled` | | Fleet-managed | `Enable TLS` | - ## File path to server certificate [_file_path_to_server_certificate] The path to the file containing the certificate for Server authentication. Required if TLS is enabled. @@ -58,7 +56,6 @@ The path to the file containing the certificate for Server authentication. Requi | APM Server binary | `apm-server.ssl.certificate` | | Fleet-managed | `File path to server certificate` | - ## File path to server certificate key [_file_path_to_server_certificate_key] The path to the file containing the Server certificate key. Required if TLS is enabled. @@ -68,7 +65,6 @@ The path to the file containing the Server certificate key. Required if TLS is e | APM Server binary | `apm-server.ssl.key` | | Fleet-managed | `File path to server certificate key` | - ## Key passphrase [_key_passphrase] The passphrase used to decrypt an encrypted key stored in the configured `apm-server.ssl.key` file. @@ -78,7 +74,6 @@ The passphrase used to decrypt an encrypted key stored in the configured `apm-se | APM Server binary | `apm-server.ssl.key_passphrase` | | Fleet-managed | N/A | - ## Supported protocol versions [_supported_protocol_versions] This setting is a list of allowed protocol versions: `SSLv3`, `TLSv1.0`, `TLSv1.1`, `TLSv1.2` and `TLSv1.3`. We do not recommend using `SSLv3` or `TLSv1.0`. The default value is `[TLSv1.1, TLSv1.2, TLSv1.3]`. @@ -88,7 +83,6 @@ This setting is a list of allowed protocol versions: `SSLv3`, `TLSv1.0`, `TLSv1. | APM Server binary | `apm-server.ssl.supported_protocols` | | Fleet-managed | `Supported protocol versions` | - ## Cipher suites for TLS connections [_cipher_suites_for_tls_connections] The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s [default suites](https://golang.org/pkg/crypto/tls/) are used (recommended). Note that TLS 1.3 cipher suites are not individually configurable in Go, so they are not included in this list. @@ -137,7 +131,6 @@ Here is a list of acronyms used in defining the cipher suites: * RSA: Cipher suites using RSA. * SHA, SHA256, SHA384: Cipher suites using SHA-1, SHA-256 or SHA-384. - ## Curve types for ECDHE based cipher suites [_curve_types_for_ecdhe_based_cipher_suites] The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange). @@ -147,7 +140,6 @@ The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key e | APM Server binary | `apm-server.ssl.curve_types` | | Fleet-managed | `Curve types for ECDHE based cipher suites` | - ## List of root certificates for verifying client certificates [_list_of_root_certificates_for_verifying_client_certificates] The list of root certificates for verifying client certificates. If `certificate_authorities` is empty or not set, the trusted certificate authorities of the host system are used. If `certificate_authorities` is set, `client_authentication` will be automatically set to `required`. Sending client certificates is currently only supported by the RUM agent through the browser, the Java agent (see [Agent certificate authentication](apm-agent-java://reference/ssl-configuration.md)), and the Jaeger agent. @@ -157,7 +149,6 @@ The list of root certificates for verifying client certificates. If `certificate | APM Server binary | `apm-server.ssl.certificate_authorities` | | Fleet-managed | N/A | - ## Client authentication [_client_authentication] This configures what types of client authentication are supported. The valid options are `none`, `optional`, and `required`. The default is `none`. If `certificate_authorities` has been specified, this setting will automatically change to `required`. This option only needs to be configured when the agent is expected to provide a client certificate. Sending client certificates is currently only supported by the RUM agent through the browser, the Java agent (see [Agent certificate authentication](apm-agent-java://reference/ssl-configuration.md)), and the Jaeger agent. diff --git a/solutions/observability/apps/ssltls-output-settings.md b/solutions/observability/apps/ssltls-output-settings.md index c17a098e53..eb08880893 100644 --- a/solutions/observability/apps/ssltls-output-settings.md +++ b/solutions/observability/apps/ssltls-output-settings.md @@ -14,7 +14,6 @@ These configuration options are only relevant to APM Server binary users. Fleet- :::: - You can specify SSL/TLS options with any output that supports SSL, like {{es}}, {{ls}}, or Kafka. Example output config with SSL/TLS enabled: ```yaml @@ -30,12 +29,10 @@ There are a number of SSL/TLS configuration options available to you: * [Client configuration options](#apm-ssl-client-config) * [Server configuration options](#apm-ssl-server-config) - ## Common configuration options [apm-ssl-common-config] Common SSL configuration options can be used in both client and server configurations. You can specify the following options in the `ssl` section of each subsystem that supports SSL. - ### `enabled` [apm-enabled] To disable SSL configuration, set the value to `false`. The default value is `true`. @@ -45,15 +42,12 @@ SSL settings are disabled if either `enabled` is set to `false` or the `ssl` sec :::: - - ### `supported_protocols` [apm-supported-protocols] List of allowed SSL/TLS versions. If SSL/TLS server decides for protocol versions not configured, the connection will be dropped during or after the handshake. The setting is a list of allowed protocol versions: `SSLv3`, `TLSv1` for TLS version 1.0, `TLSv1.0`, `TLSv1.1`, `TLSv1.2`, and `TLSv1.3`. The default value is `[TLSv1.1, TLSv1.2, TLSv1.3]`. - ### `cipher_suites` [apm-cipher-suites] The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s [default suites](https://golang.org/pkg/crypto/tls/) are used (recommended). Note that TLS 1.3 cipher suites are not individually configurable in Go, so they are not included in this list. @@ -97,7 +91,6 @@ Here is a list of acronyms used in defining the cipher suites: * RSA: Cipher suites using RSA. * SHA, SHA256, SHA384: Cipher suites using SHA-1, SHA-256 or SHA-384. - ### `curve_types` [apm-curve-types] The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange). @@ -109,7 +102,6 @@ The following elliptic curve types are available: * P-521 * X25519 - ### `ca_sha256` [apm-ca-sha256] This configures a certificate pin that you can use to ensure that a specific certificate is part of the verified chain. @@ -120,13 +112,10 @@ The pin is a base64 encoded string of the SHA-256 of the certificate. This check is not a replacement for the normal SSL validation, but it adds additional validation. If this option is used with `verification_mode` set to `none`, the check will always fail because it will not receive any verified chains. :::: - - ## Client configuration options [apm-ssl-client-config] You can specify the following options in the `ssl` section of each subsystem that supports SSL. - ### `certificate_authorities` [apm-client-certificate-authorities] The list of root certificates for verifications is required. If `certificate_authorities` is self-signed, the host system needs to trust that CA cert as well. @@ -157,7 +146,6 @@ certificate_authorities: -----END CERTIFICATE----- ``` - ### `certificate: "/etc/pki/client/cert.pem"` [apm-client-certificate] The path to the certificate for SSL client authentication is only required if `client_authentication` is specified. If the certificate is not specified, client authentication is not available. The connection might fail if the server requests client authentication. If the SSL server does not require client authentication, the certificate will be loaded, but not requested or used by the server. @@ -187,7 +175,6 @@ certificate: | -----END CERTIFICATE----- ``` - ### `key: "/etc/pki/client/cert.key"` [apm-client-key] The client certificate key used for client authentication and is only required if `client_authentication` is configured. The key option support embedding of the private key: @@ -224,12 +211,10 @@ key: | -----END PRIVATE KEY----- ``` - ### `key_passphrase` [apm-client-key-passphrase] The passphrase used to decrypt an encrypted key stored in the configured `key` file. - ### `verification_mode` [apm-client-verification-mode] Controls the verification of server certificates. Valid values are: @@ -248,13 +233,10 @@ Controls the verification of server certificates. Valid values are: The default value is `full`. - - ## Server configuration options [apm-ssl-server-config] You can specify the following options in the `ssl` section of each subsystem that supports SSL. - ### `certificate_authorities` [apm-server-certificate-authorities] The list of root certificates for client verifications is only required if `client_authentication` is configured. If `certificate_authorities` is empty or not set, and `client_authentication` is configured, the system keystore is used. @@ -285,7 +267,6 @@ certificate_authorities: -----END CERTIFICATE----- ``` - ### `certificate: "/etc/pki/server/cert.pem"` [apm-server-certificate] For server authentication, the path to the SSL authentication certificate must be specified for TLS. If the certificate is not specified, startup will fail. @@ -315,7 +296,6 @@ certificate: | -----END CERTIFICATE----- ``` - ### `key: "/etc/pki/server/cert.key"` [apm-server-key] The server certificate key used for authentication is required. The key option support embedding of the private key: @@ -352,12 +332,10 @@ key: | -----END PRIVATE KEY----- ``` - ### `key_passphrase` [apm-server-key-passphrase] The passphrase is used to decrypt an encrypted key stored in the configured `key` file. - ### `verification_mode` [apm-server-verification-mode] Controls the verification of client certificates. Valid values are: @@ -376,8 +354,6 @@ Controls the verification of client certificates. Valid values are: The default value is `full`. - - ### `renegotiation` [apm-server-renegotiation] This configures what types of TLS renegotiation are supported. The valid options are: @@ -393,4 +369,3 @@ This configures what types of TLS renegotiation are supported. The valid options The default value is `never`. - diff --git a/solutions/observability/apps/storage-explorer.md b/solutions/observability/apps/storage-explorer.md index e699b43f2c..f50127f53d 100644 --- a/solutions/observability/apps/storage-explorer.md +++ b/solutions/observability/apps/storage-explorer.md @@ -14,7 +14,6 @@ Analyze your APM data and manage costs with **storage explorer**. For example, a :screenshot: ::: - ## Index lifecycle phases [_index_lifecycle_phases] A default [index lifecycle policy](/solutions/observability/apps/index-lifecycle-management.md) is applied to each APM data stream, but can be customized depending on your business needs. Use the **Index lifecycle phase** dropdown to visualize and analyze your storage by phase. @@ -27,12 +26,10 @@ Customizing the default APM index lifecycle policies can save money by specifyin See [Index lifecycle management](/solutions/observability/apps/index-lifecycle-management.md) to learn more about customizing the default APM index lifecycle policies. - ## Service size chart [_service_size_chart] The service size chart displays the estimated size of each service over time. Expand the time filter to visualize data trends and estimate daily data generation. - ## Service statistics table [_service_statistics_table] The service statistics table provides detailed information on each service: @@ -49,32 +46,26 @@ The service statistics table provides detailed information on each service: As you explore your service statistics, you might want to take action to reduce the number of documents and therefore storage size of a particular service. - ### Reduce the number of transactions [_reduce_the_number_of_transactions] To reduce the number of transactions a service generates, configure a more aggressive [transaction sampling policy](/solutions/observability/apps/transaction-sampling.md). Transaction sampling lowers the amount of data ingested without negatively impacting the usefulness of your data. - ### Reduce the number of spans [_reduce_the_number_of_spans] To reduce the number of spans a service generates, enable [span compression](/solutions/observability/apps/spans.md#apm-spans-span-compression). Span compression saves on data and transfer costs by compressing multiple, similar spans into a single span. - ### Reduce the number of metrics [_reduce_the_number_of_metrics] To reduce the number of system, runtime, and application metrics, tune the APM agent or agents that are collecting the data. You can disable the collection of specific metrics with the **disable metrics** configuration. Or, you can set the **metrics interval** to zero seconds to deactivate metrics entirely. Most APM agents support both options. See the relevant [APM agent configuration options](https://www.elastic.co/guide/en/apm/agent) for more details. - ### Reduce the number of errors [_reduce_the_number_of_errors] To reduce the number of errors a service generate, work with your developers to change how exceptions are handled in your code. - ## Privileges [_privileges] Storage Explorer requires expanded privileges to view. See [Create a storage explorer user](applications-ui-storage-explorer-user.md) for more information. - ## Limitations [_limitations_4] Multi-cluster deployments are not supported. diff --git a/solutions/observability/apps/storage-sizing-guide.md b/solutions/observability/apps/storage-sizing-guide.md index 548ec85df3..19875d1c50 100644 --- a/solutions/observability/apps/storage-sizing-guide.md +++ b/solutions/observability/apps/storage-sizing-guide.md @@ -13,7 +13,6 @@ APM processing and storage costs are largely dominated by transactions, spans, a * [**Spans**](spans.md) belong to transactions. They measure from the start to end of an activity, and contain information about a specific code path that has been executed. * **Stack frames** belong to spans. Stack frames represent a function call on the call stack, and include attributes like function name, file name and path, line number, etc. Stack frames can heavily influence the size of a span. - ## Typical transactions [_typical_transactions] Due to the high variability of APM data, it’s difficult to classify a transaction as typical. Regardless, this guide will attempt to classify Transactions as *Small*, *Medium*, or *Large*, and make recommendations based on those classifications. @@ -34,7 +33,6 @@ Here’s a speculative reference: There will always be transaction outliers with hundreds of spans or stack frames, but those are very rare. Small transactions are the most common. - ## Typical storage [_typical_storage] Consider the following typical storage reference. These numbers do not account for {{es}} compression. diff --git a/solutions/observability/apps/switch-an-elastic-cloud-cluster-to-apm-integration.md b/solutions/observability/apps/switch-an-elastic-cloud-cluster-to-apm-integration.md index b96beba06a..b58c84d53d 100644 --- a/solutions/observability/apps/switch-an-elastic-cloud-cluster-to-apm-integration.md +++ b/solutions/observability/apps/switch-an-elastic-cloud-cluster-to-apm-integration.md @@ -6,22 +6,17 @@ applies_to: stack: --- - - # Switch an Elastic Cloud cluster to the APM integration [apm-integration-upgrade-steps-ess] - 1. [Upgrade the {{stack}}](#apm-integration-upgrade-ess-1) 2. [Switch to {{agent}}](#apm-integration-upgrade-ess-2) 3. [Configure the APM integration](#apm-integration-upgrade-ess-3) 4. [Scale APM and {{fleet}}](#apm-integration-upgrade-ess-4) - ## Upgrade the {{stack}} [apm-integration-upgrade-ess-1] Use the {{ecloud}} Console to upgrade the {{stack}} to version 9.0.0-beta1. See the [Upgrade guide](../../../deploy-manage/upgrade/deployment-or-cluster.md) for details. - ## Switch to {{agent}} [apm-integration-upgrade-ess-2] APM data collection will be interrupted while the migration is in progress. The process of migrating should only take a few minutes. @@ -40,17 +35,14 @@ With a Superuser account, complete the following steps: :alt: {{agent}} settings migration ::: - {{ecloud}} will now create a {{fleet}} Server instance to contain the new APM integration, and then will shut down the old APM server instance. Within minutes your data should begin appearing in the Applications UI again. - ## Configure the APM integration [apm-integration-upgrade-ess-3] You can now update settings that were removed during the upgrade. See [Configure APM Server](configure-apm-server.md) for a reference of all available settings. In {{kib}}, navigate to **Management** > **Fleet**. Select the **Elastic Cloud Agent Policy**. Next to the **Elastic APM** integration, select **Actions** > **Edit integration**. - ## Scale APM and {{fleet}} [apm-integration-upgrade-ess-4] Certain {{es}} output configuration options are not available with the APM integration. To ensure data is not lost, you can scale APM and {{fleet}} up and out. APM’s capacity to process events increases with the instance memory size. diff --git a/solutions/observability/apps/switch-self-installation-to-apm-integration.md b/solutions/observability/apps/switch-self-installation-to-apm-integration.md index 45402caa25..3a56a62deb 100644 --- a/solutions/observability/apps/switch-self-installation-to-apm-integration.md +++ b/solutions/observability/apps/switch-self-installation-to-apm-integration.md @@ -6,25 +6,20 @@ applies_to: stack: --- - - # Switch a self-installation to the APM integration [apm-integration-upgrade-steps] - 1. [Upgrade the {{stack}}](#apm-integration-upgrade-1) 2. [Add a {{fleet}} Server](#apm-integration-upgrade-2) 3. [Install a {{fleet}}-managed {{agent}}](#apm-integration-upgrade-3) 4. [Add the APM integration](#apm-integration-upgrade-4) 5. [Stop the APM Server](#apm-integration-upgrade-5) - ## Upgrade the {{stack}} [apm-integration-upgrade-1] The {{stack}} ({{es}} and {{kib}}) must be upgraded to version 7.14 or higher. See the [{{stack}} Installation and Upgrade Guide](../../../deploy-manage/upgrade/deployment-or-cluster.md) for guidance. Review the [Elastic APM release notes](apm-server://release-notes/index.md) and [Elastic {{observability}} release notes](/release-notes/elastic-observability/index.md) for important changes between your current APM version and this one. - ## Add a {{fleet}} Server [apm-integration-upgrade-2] {{fleet}} Server is a component of the {{stack}} used to centrally manage {{agent}}s. The APM integration requires a {{fleet}} Server to be running and accessible to your hosts. Add a {{fleet}} Server by following [this guide](/reference/fleet/deployment-models.md). @@ -33,22 +28,18 @@ Review the [Elastic APM release notes](apm-server://release-notes/index.md) and If you’re upgrading a self-managed deployment of the {{stack}}, you’ll need to enable [{{es}} security](../../../deploy-manage/deploy/self-managed/installing-elasticsearch.md) and the [API key service](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md). :::: - After adding your {{fleet}} Server host and generating a service token, the in-product help in {{kib}} will provide a command to run to start an {{agent}} as a {{fleet}} Server. Commands may require administrator privileges. Verify {{fleet}} Server is running by navigating to **{{fleet}}** > **Agents** in {{kib}}. - ## Install a {{fleet}}-managed {{agent}} [apm-integration-upgrade-3] ::::{note} It’s possible to install the Elastic APM integration on the same {{agent}} that is running the {{fleet}} Server integration. For this use case, skip this step. :::: - The {{fleet}}-managed {{agent}} will run the Elastic APM integration on your edge nodes, next to your applications. To install a {{fleet}}-managed {{agent}}, follow [this guide](/reference/fleet/install-fleet-managed-elastic-agent.md). - ## Add the APM integration [apm-integration-upgrade-4] The APM integration receives performance data from your APM agents, validates and processes it, and then transforms the data into {{es}} documents. @@ -59,8 +50,6 @@ To add the APM integration, see [Step 2: Add and configure the APM integration]( You’ll configure the APM integration in this step. See [Configure APM Server](configure-apm-server.md) for a reference of all available settings. As long as the APM integration is configured with the same secret token or you have API keys enabled on the same host, no reconfiguration is required in your APM agents. :::: - - ## Stop the APM Server [apm-integration-upgrade-5] Once data from upgraded APM agents is visible in the Applications UI, it’s safe to stop the APM Server process. diff --git a/solutions/observability/apps/switch-to-elastic-apm-integration.md b/solutions/observability/apps/switch-to-elastic-apm-integration.md index b261d3fb9d..f732d51735 100644 --- a/solutions/observability/apps/switch-to-elastic-apm-integration.md +++ b/solutions/observability/apps/switch-to-elastic-apm-integration.md @@ -30,7 +30,6 @@ The APM integration offers a number of benefits over the standalone method of ru * Easier and less error-prone upgrade path * Zero-downtime configuration changes - ## APM integration architecture [apm-arch-upgrade] Elastic APM consists of four components: **APM agents**, the **Elastic APM integration**, **{{es}}**, and **{{kib}}**. Generally, there are two ways that these four components can work together: @@ -47,7 +46,6 @@ Or, APM agents and the APM integration live on edge machines and enroll via a ce In order to collect data from RUM and mobile agents, which run in browser and mobile applications, you must run {{agent}} centrally. For other applications, such as backend services, {{agent}} may be co-located on the edge machine. :::: - ## Limitations [apm-integration-upgrade-limitations] There are some limitations to be aware of: @@ -57,7 +55,6 @@ There are some limitations to be aware of: * APM runs under {{agent}} which, depending on the installation method, might require root privileges * An {{agent}} with the APM integration enabled must be managed by {{fleet}}. - ## Make the switch [_make_the_switch] Select a guide below to get started. diff --git a/solutions/observability/apps/synthetic-monitoring.md b/solutions/observability/apps/synthetic-monitoring.md index b15673da80..d4c9f4618b 100644 --- a/solutions/observability/apps/synthetic-monitoring.md +++ b/solutions/observability/apps/synthetic-monitoring.md @@ -14,7 +14,6 @@ The Synthetics UI is for viewing result data from monitors created and managed d :::: - Synthetics periodically checks the status of your services and applications. Monitor the availability of network endpoints and services using the following types of monitors: * [Lightweight HTTP/S, TCP, and ICMP monitors](../../../solutions/observability/apps/synthetic-monitoring.md#monitoring-uptime) @@ -25,7 +24,6 @@ Synthetics periodically checks the status of your services and applications. Mon :screenshot: ::: - ## Lightweight HTTP/S, TCP, and ICMP monitors [monitoring-uptime] You can monitor the status of network endpoints using the following lightweight checks: @@ -38,7 +36,6 @@ You can monitor the status of network endpoints using the following lightweight To set up your first monitor, refer to [Get started](../../../solutions/observability/apps/get-started.md). - ## Browser monitors [monitoring-synthetics] Real browser synthetic monitoring enables you to test critical actions and requests that an end-user would make on your site at predefined intervals and in a controlled environment. Synthetic monitoring extends traditional end-to-end testing techniques because it allows your tests to run continuously on the cloud. The result is rich, consistent, and repeatable data that you can trend and alert on. diff --git a/solutions/observability/apps/synthetics-encryption-security.md b/solutions/observability/apps/synthetics-encryption-security.md index fb55d279a5..2c5c886422 100644 --- a/solutions/observability/apps/synthetics-encryption-security.md +++ b/solutions/observability/apps/synthetics-encryption-security.md @@ -11,17 +11,14 @@ applies_to: Elastic Synthetics was designed with security in mind encrypting both persisted and transmitted data. This page catalogs the points within Elastic Synthetics where data is either stored or transmitted in an encrypted fashion. - ## Synthetics UI (Kibana App) [_synthetics_ui_kibana_app] Data is stored in [Kibana Secure Saved Objects](../../../deploy-manage/security/secure-saved-objects.md), with sensitive fields encrypted. These fields include your script source, params, and global params. - ## Synthetics Service [_synthetics_service] The Global Elastic Synthetics Service performs all communication of sensitive data (both internally, and with Kibana) over encrypted connections and encrypts all data persisted to disk as well. - ## Synthetics Private Locations [_synthetics_private_locations] In Kibana configuration for private locations is stored in two places, Synthetics saved objects which always encrypt sensitive fields using [Kibana Secure Saved Objects](../../../deploy-manage/security/secure-saved-objects.md) and also in Fleet, which uses unencrypted saved objects restricted by user permissions. For Elastic Cloud customers all data is secured on disk regardless of whether additional saved object encryption is present. See our [Cloud Security Statement](https://www.elastic.co/cloud/security) for more information. We recommend that self-managed customers encrypt disks for their Elasticsearch instances if this is a concern. diff --git a/solutions/observability/apps/synthetics-support-matrix.md b/solutions/observability/apps/synthetics-support-matrix.md index b62ba2ab53..53bed3ad7e 100644 --- a/solutions/observability/apps/synthetics-support-matrix.md +++ b/solutions/observability/apps/synthetics-support-matrix.md @@ -9,7 +9,6 @@ applies_to: There are various components that make up the Synthetics solution, which are supported in the following configurations: - ## {{synthetics-app}} [_synthetics_app] * **GA support**: 8.8.0 and higher @@ -18,14 +17,11 @@ There are various components that make up the Synthetics solution, which are sup * For creating and managing lightweight and browser monitors configured through the [{{synthetics-app}}](create-monitors-in-synthetics-app.md) * For reporting for lightweight and browser monitors configured through the [{{synthetics-app}}](create-monitors-in-synthetics-app.md) and/or [{{project-monitors-cap}}](create-monitors-with-project-monitors.md) - - ## {{project-monitors-cap}} [_project_monitors_cap] * **GA support**: 8.8.0 and higher * **Notes**: For creating and managing lightweight and browser monitors configured as [{{project-monitors-cap}}](create-monitors-with-project-monitors.md) - ## Elastic’s global managed testing infrastructure [_elastics_global_managed_testing_infrastructure_2] * **GA support**: 8.8.0 and higher @@ -33,8 +29,6 @@ There are various components that make up the Synthetics solution, which are sup Executing synthetic tests on Elastic’s global managed testing infrastructure incurs an additional charge. Tests are charged under one of two new billing dimensions depending on the monitor type. For *browser monitor* usage, there is a fee per test run. For *lightweight monitor* usage, there is a fee per region in which you run any monitors regardless of the number of test runs. For more details, refer to [full details and current pricing](https://www.elastic.co/pricing). - - ## {{private-location}}s [_private_locations_2] * **GA support**: 8.8.0 and higher @@ -50,8 +44,6 @@ There are various components that make up the Synthetics solution, which are sup * Do not configure any ingest pipelines or Logstash output - - ## Heartbeat with Uptime [_heartbeat_with_uptime] ```{applies_to} @@ -65,8 +57,6 @@ serverless: unavailable * For running lightweight monitors via YML configuration running on self-managed infrastructure * Browser-based monitors are not supported in this configuration - - ## Standalone {{agent}} [_standalone_agent] * **GA support**: As defined in the standard [Support matrix](https://www.elastic.co/support/matrix) @@ -76,8 +66,6 @@ serverless: unavailable * Browser-based monitors are not supported in this configuration * Results for monitors configured using the standalone {{agent}} are available in the {{uptime-app}} (*not* the {{synthetics-app}}) - - ## Synthetics Recorder [_synthetics_recorder] System requirements: @@ -90,8 +78,6 @@ System requirements: * Fedora - 24 and newer * Debian - 8 and newer - - ## Output to Elasticsearch [_output_to_elasticsearch] Synthetics must have a direct connection to {{es}}, whether running monitors from Elastic’s global managed testing infrastructure or from {{private-location}}s. diff --git a/solutions/observability/apps/tail-based-sampling.md b/solutions/observability/apps/tail-based-sampling.md index 922e30e8ef..e0a7e4258c 100644 --- a/solutions/observability/apps/tail-based-sampling.md +++ b/solutions/observability/apps/tail-based-sampling.md @@ -14,7 +14,6 @@ Most options on this page are supported by all APM Server deployment methods whe :::: - Tail-based sampling configuration options. :::::::{tab-set} @@ -51,7 +50,6 @@ Configure and customize Fleet-managed APM settings directly in {{kib}}: See [Tail-based sampling](transaction-sampling.md#apm-tail-based-sampling) to learn more. - ### Enable tail-based sampling [sampling-tail-enabled-ref] Set to `true` to enable tail based sampling. Disabled by default. (bool) @@ -61,7 +59,6 @@ Set to `true` to enable tail based sampling. Disabled by default. (bool) | APM Server binary | `sampling.tail.enabled` | | Fleet-managed | `Enable tail-based sampling` | - ### Interval [sampling-tail-interval-ref] Synchronization interval for multiple APM Servers. Should be in the order of tens of seconds or low minutes. Default: `1m` (1 minute). (duration) @@ -71,7 +68,6 @@ Synchronization interval for multiple APM Servers. Should be in the order of ten | APM Server binary | `sampling.tail.interval` | | Fleet-managed | `Interval` | - ### Policies [sampling-tail-policies-ref] Criteria used to match a root transaction to a sample rate. @@ -83,7 +79,6 @@ Policies map trace events to a sample rate. Each policy must specify a sample ra | APM Server binary | `sampling.tail.policies` | | Fleet-managed | `Policies` | - ### Storage limit [sampling-tail-storage_limit-ref] The amount of storage space allocated for trace events matching tail sampling policies. Caution: Setting this limit higher than the allowed space may cause APM Server to become unhealthy. @@ -101,39 +96,32 @@ Default: `0GB`. (text) | APM Server binary | `sampling.tail.storage_limit` | | Fleet-managed | `Storage limit` | - ## Policy-level tail-based sampling settings [apm-configuration-tbs-policy] See [Tail-based sampling](transaction-sampling.md#apm-tail-based-sampling) to learn more. - ### **`sample_rate`** [sampling-tail-sample-rate-ref] The sample rate to apply to trace events matching this policy. Required in each policy. The sample rate must be greater than or equal to `0` and less than or equal to `1`. For example, a `sample_rate` of `0.01` means that 1% of trace events matching the policy will be sampled. A `sample_rate` of `1` means that 100% of trace events matching the policy will be sampled. (int) - ### **`trace.name`** [sampling-tail-trace-name-ref] The trace name for events to match a policy. A match occurs when the configured `trace.name` matches the `transaction.name` of the root transaction of a trace. A root transaction is any transaction without a `parent.id`. (string) - ### **`trace.outcome`** [sampling-tail-trace-outcome-ref] The trace outcome for events to match a policy. A match occurs when the configured `trace.outcome` matches a trace’s `event.outcome` field. Trace outcome can be `success`, `failure`, or `unknown`. (string) - ### **`service.name`** [sampling-tail-service-name-ref] The service name for events to match a policy. (string) - ### **`service.environment`** [sampling-tail-service-environment-ref] The service environment for events to match a policy. (string) - ## Monitoring tail-based sampling [sampling-tail-monitoring-ref] APM Server produces metrics to monitor the performance and estimate the workload being processed by tail-based sampling. In order to use these metrics, you need to [enable monitoring for the APM Server](/solutions/observability/apps/monitor-apm-server.md). The following metrics are produced by the tail-based sampler (note that the metrics might have a different prefix, for example `beat.stats` for ECH deployments, based on how the APM Server is running): diff --git a/solutions/observability/apps/trace-sample-timeline.md b/solutions/observability/apps/trace-sample-timeline.md index d1393f9db9..710acac06f 100644 --- a/solutions/observability/apps/trace-sample-timeline.md +++ b/solutions/observability/apps/trace-sample-timeline.md @@ -22,13 +22,11 @@ View a span in detail by clicking on it in the timeline waterfall. For example, A [span](/solutions/observability/apps/spans.md) is the duration of a single event. Spans are automatically captured by APM agents, and you can also define custom spans. Each span has a type and is defined by a different color in the timeline/waterfall visualization. :::: - :::{image} /solutions/images/observability-apm-span-detail.png :alt: Example view of a span detail in the Applications UI :screenshot: ::: - ## Investigate [trace-sample-investigate] The trace sample timeline features an **Investigate** button which provides a quick way to jump to other areas of the Elastic Observability UI while maintaining the context of the currently selected trace sample. For example, quickly view: @@ -41,7 +39,6 @@ The trace sample timeline features an **Investigate** button which provides a qu * the selected transaction in **Discover** * your [custom links](../../../solutions/observability/apps/create-custom-links.md) - ## Distributed tracing [distributed-tracing] When a trace travels through multiple services it is known as a *distributed trace*. In APM, the colors in a distributed trace represent different services and are listed in the order they occur. diff --git a/solutions/observability/apps/traces-2.md b/solutions/observability/apps/traces-2.md index 391fc39185..775d1e7be9 100644 --- a/solutions/observability/apps/traces-2.md +++ b/solutions/observability/apps/traces-2.md @@ -13,7 +13,6 @@ applies_to: Traces link together related transactions to show an end-to-end performance of how a request was served and which services were part of it. In addition to the Traces overview, you can view your application traces in the [trace sample timeline waterfall](../../../solutions/observability/apps/trace-sample-timeline.md). :::: - **Traces** displays your application’s entry (root) transactions. Transactions with the same name are grouped together and only shown once in this table. If you’re using [distributed tracing](../../../solutions/observability/apps/trace-sample-timeline.md#distributed-tracing), this view is key to finding the critical paths within your application. By default, transactions are sorted by *Impact*. Impact helps show the most used and slowest endpoints in your service — in other words, it’s the collective amount of pain a specific endpoint is causing your users. If there’s a particular endpoint you’re worried about, select it to view its [transaction details](../../../solutions/observability/apps/transactions-2.md#transaction-details). @@ -25,14 +24,12 @@ You can also use queries to filter and search the transactions shown on this pag :screenshot: ::: - ## Trace explorer [trace-explorer] ::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: - **Trace explorer** is an experimental top-level search tool that allows you to query your traces using [{{kib}} Query Language (KQL)](../../../explore-analyze/query-filter/languages/kql.md) or [Event Query Language (EQL)](../../../explore-analyze/query-filter/languages/eql.md). Curate your own custom queries, or use the [**Service Map**](../../../solutions/observability/apps/service-map.md) to find and select edges to automatically generate queries based on your selection: diff --git a/solutions/observability/apps/traces.md b/solutions/observability/apps/traces.md index a131ffef85..40ed67765c 100644 --- a/solutions/observability/apps/traces.md +++ b/solutions/observability/apps/traces.md @@ -11,7 +11,6 @@ applies_to: A trace is a group of [transactions](../../../solutions/observability/apps/transactions.md) and [spans](../../../solutions/observability/apps/spans.md) with a common root. Each trace tracks the entirety of a single request. It describes the individual operations and their causality that ensue from a single logical operation. - ## Distributed tracing [apm-distributed-tracing] When a trace travels through multiple services, as is common in a microservice architecture, it is known as a **distributed trace**. A distributed trace comprises operations across multiple distributed components, crossing process, network, and security boundaries. @@ -24,14 +23,12 @@ Tracking requests as they propagate through your services provides an end-to-end For supported technologies, distributed tracing works out-of-the-box, with no additional configuration required. - ### How distributed tracing works [apm-how-distributed-tracing] Distributed tracing works by injecting a custom `traceparent` HTTP header into outgoing requests. This header includes information, like `trace-id`, which is used to identify the current trace, and `parent-id`, which is used to identify the parent of the current span on incoming requests or the current span on an outgoing request. When a service is working on a request, it checks for the existence of this HTTP header. If it’s missing, the service starts a new trace. If it exists, the service ensures the current action is added as a child of the existing trace, and continues to propagate the trace. - ### Trace propagation examples [apm-trace-propagation] In this example, Elastic’s Ruby agent communicates with Elastic’s Java agent. Both support the `traceparent` header, and trace data is successfully propagated. @@ -52,7 +49,6 @@ In this example, the trace meets a piece of middleware that doesn’t propagate :alt: How traceparent propagation works ::: - ### W3C Trace Context specification [apm-w3c-tracecontext-spec] All Elastic agents now support the official W3C Trace Context specification and `traceparent` header. See the table below for the minimum required agent version: @@ -74,8 +70,6 @@ All Elastic agents now support the official W3C Trace Context specification and Older Elastic agents use a unique `elastic-apm-traceparent` header. For backward-compatibility purposes, new versions of Elastic agents still support this header. :::: - - ### Visualize distributed tracing [apm-visualize-distributed-tracing] APM's timeline visualization provides a visual deep-dive into each of your application’s traces: @@ -85,12 +79,10 @@ APM's timeline visualization provides a visual deep-dive into each of your appli :screenshot: ::: - ### Manual distributed tracing [apm-manual-distributed-tracing] Elastic agents automatically propagate distributed tracing context for supported technologies. If your service communicates over a different, unsupported protocol, you can manually propagate distributed tracing context from a sending service to a receiving service with each agent’s API. - #### Add the `traceparent` header to outgoing requests [apm-distributed-tracing-outgoing] Sending services must add the `traceparent` header to outgoing requests. @@ -380,7 +372,6 @@ $receiverTransaction = ElasticApm::beginCurrentTransaction( <1> 1. Start a new transaction 2. Pass in the received distributed tracing data (serialized as string) - Once this new transaction has been created in the receiving service, you can create child spans, or use any other agent API methods as you typically would. :::::: @@ -398,7 +389,6 @@ client.begin_transaction('processors', trace_parent=parent) <2> 1. Create a `TraceParent` object from HTTP headers formed as a dictionary 2. Begin a new transaction as a child of the received `TraceParent` - ::::{tip} See the [`TraceParent` API](apm-agent-python://reference/api-reference.md#traceparent-api) for additional examples. :::: diff --git a/solutions/observability/apps/transactions-2.md b/solutions/observability/apps/transactions-2.md index 163973f301..784491fd23 100644 --- a/solutions/observability/apps/transactions-2.md +++ b/solutions/observability/apps/transactions-2.md @@ -11,7 +11,6 @@ applies_to: A *transaction* describes an event captured by an Elastic APM agent instrumenting a service. APM agents automatically collect performance metrics on HTTP requests, database queries, and much more. - :::{image} /solutions/images/observability-apm-transactions-overview.png :alt: Example view of transactions table in the Applications UI :screenshot: @@ -37,7 +36,6 @@ The **Latency**, **Throughput**, **Failed transaction rate**, **Time spent by sp :::: - **Time spent by span type** Visualize where your application is spending most of its time. For example, is your app spending time in external calls, database processing, or application code execution? @@ -45,11 +43,9 @@ The **Latency**, **Throughput**, **Failed transaction rate**, **Time spent by sp It’s important to note that if you have asynchronous spans, the sum of all span times may exceed the duration of the transaction. - **Cold start rate** Only applicable to serverless transactions, this chart displays the percentage of requests that trigger a cold start of a serverless function. See [Cold starts](../../../solutions/observability/apps/observe-lambda-functions.md#apm-lambda-cold-start-info) for more information. - ## Transactions table [transactions-table] The **Transactions** table displays a list of *transaction groups* for the selected service. In other words, this view groups all transactions of the same name together, and only displays one entry for each group. @@ -81,7 +77,6 @@ The transaction overview page is customized for the JavaScript RUM agent. Specif Additional RUM goodies, like core vitals, and visitor breakdown by browser, location, and device, are available in the Observability User Experience tab. - ## Transaction details [transaction-details] Selecting a transaction group will bring you to the **transaction** details. This page is visually similar to the transaction overview, but it shows data from all transactions within the selected transaction group. @@ -91,7 +86,6 @@ Selecting a transaction group will bring you to the **transaction** details. Thi :screenshot: ::: - ### Latency distribution [transaction-duration-distribution] The latency distribution shows a plot of all transaction durations for the given time period. The following screenshot shows a typical distribution and indicates most of our requests were served quickly — awesome! The requests on the right are taking longer than average; we probably need to focus on them. @@ -103,7 +97,6 @@ The latency distribution shows a plot of all transaction durations for the given Click and drag to select a latency duration *bucket* to display up to 500 trace samples. - ### Trace samples [transaction-trace-sample] Trace samples are based on the *bucket* selection in the **Latency distribution** chart; update the samples by selecting a new *bucket*. The number of requests per bucket is displayed when hovering over the graph, and the selected bucket is highlighted to stand out. @@ -124,7 +117,6 @@ More information on timeline waterfalls is available in [spans](../../../solutio :::: - **Trace sample metadata** Learn more about a trace sample in the **Metadata** tab: @@ -145,7 +137,6 @@ All of this data is stored in documents in Elasticsearch. This means you can sel :::: - **Trace sample logs** The **Logs** tab displays logs related to the sampled trace. @@ -161,7 +152,6 @@ To learn how to correlate your logs with your instrumented services, see [Stream :screenshot: ::: - ### Correlations [transaction-latency-correlations] Correlations surface attributes of your data that are potentially correlated with high-latency or erroneous transactions. To learn more, see [Find transaction latency and failure correlations](../../../solutions/observability/apps/find-transaction-latency-failure-correlations.md). diff --git a/solutions/observability/apps/transactions.md b/solutions/observability/apps/transactions.md index 63dc3d9603..00818dc3d4 100644 --- a/solutions/observability/apps/transactions.md +++ b/solutions/observability/apps/transactions.md @@ -42,8 +42,6 @@ Transactions are grouped by their `type` and `name` in the Applications UI’s [ Most agents limit keyword fields (e.g. `labels`) to 1024 characters, non-keyword fields (e.g. `span.db.statement`) to 10,000 characters. :::: - - ## Data streams [_data_streams_2] Transactions are stored with spans in the following data streams: @@ -53,7 +51,6 @@ Transactions are stored with spans in the following data streams: See [Data streams](data-streams.md) to learn more. - ## Example transaction document [_example_transaction_document] This example shows what transaction documents can look like when indexed in {{es}}. @@ -467,5 +464,3 @@ This example shows what transaction documents can look like when indexed in {{es :::: - - diff --git a/solutions/observability/apps/tune-data-ingestion.md b/solutions/observability/apps/tune-data-ingestion.md index c0e257923f..8645d1274c 100644 --- a/solutions/observability/apps/tune-data-ingestion.md +++ b/solutions/observability/apps/tune-data-ingestion.md @@ -9,21 +9,18 @@ applies_to: This section explains how to adapt data ingestion according to your needs. - ## Tune APM Server [apm-tune-apm-server] * [Add APM Server or {{agent}} instances](#apm-add-apm-server-instances) * [Reduce the payload size](#apm-reduce-payload-size) * [Adjust anonymous auth rate limit](#apm-adjust-event-rate) - ### Add APM Server or {{agent}} instances [apm-add-apm-server-instances] If the APM Server cannot process data quickly enough, you will see request timeouts. One way to solve this problem is to increase processing power. Increase processing power by either migrating to a more powerful machine or adding more APM Server/Elastic Agent instances. Having several instances will also increase [availability](high-availability.md). - ### Reduce the payload size [apm-reduce-payload-size] Large payloads may result in request timeouts. You can reduce the payload size by decreasing the flush interval in the agents. This will cause agents to send smaller and more frequent requests. @@ -32,7 +29,6 @@ Optionally you can also [reduce the sample rate](reduce-storage.md#apm-reduce-sa Read more in the [agents documentation](https://www.elastic.co/guide/en/apm/agent/index.html). - ### Adjust anonymous auth rate limit [apm-adjust-event-rate] Agents make use of long running requests and flush as many events over a single request as possible. Thus, the rate limiter for anonymous authentication is bound to the number of *events* sent per second, per IP. @@ -46,7 +42,6 @@ Increasing the default value for the following configuration variable will help | APM Server binary | [`rate_limit.event_limit`](configure-anonymous-authentication.md#apm-config-auth-anon-event-limit) | | Fleet-managed | `Anonymous Event rate limit (event limit)` | - ## Tune {{es}} [apm-tune-elasticsearch] The {{es}} Reference provides insight on tuning {{es}}. diff --git a/solutions/observability/apps/tutorial-monitor-java-application.md b/solutions/observability/apps/tutorial-monitor-java-application.md index 888ec0f693..de8a907246 100644 --- a/solutions/observability/apps/tutorial-monitor-java-application.md +++ b/solutions/observability/apps/tutorial-monitor-java-application.md @@ -9,7 +9,6 @@ applies_to: In this guide, you’ll learn how to monitor a Java application using Elastic {{observability}}: Logs, Infrastructure metrics, APM, and Uptime. - ## What you’ll learn [_what_youll_learn] You’ll learn how to: @@ -20,12 +19,10 @@ You’ll learn how to: * Instrument your application using the [Elastic APM Java agent](apm-agent-java://reference/index.md). * Monitor your services using {{heartbeat}} and view your uptime data in {{kib}}. - ## Before you begin [_before_you_begin] Create an [{{ech}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body) deployment. The deployment includes an {{es}} cluster for storing and searching your data, {{kib}} for visualizing and managing your data, and an APM server. If you do not want to follow all those steps listed here and take a look at the final java code, check out the [observability-contrib GitHub repository](https://github.com/elastic/observability-contrib/tree/main/monitor-java-app) for the sample application. - ## Step 1: Create a Java application [_step_1_create_a_java_application] To create the Java application, you require OpenJDK 14 (or higher) and the [Javalin](https://javalin.io/) web framework. The application will include the main endpoint, an artificially long-running endpoint, and an endpoint that needs to poll another data source. There will also be a background job running. @@ -201,13 +198,10 @@ To create the Java application, you require OpenJDK 14 (or higher) and the [Java ./gradlew clean check shadowJar ``` - - ## Step 2: Ingest logs [_step_2_ingest_logs] Logs can be events such as checkout, an exception, or an HTTP request. For this tutorial, let’s use log4j2 as our logging implementation. - ### Add logging implementation [_add_logging_implementation] 1. Add the dependency to the `build.gradle` file. @@ -228,7 +222,6 @@ Logs can be events such as checkout, an exception, or an HTTP request. For this :::: - ```java package de.spinscale.javalin; @@ -281,8 +274,6 @@ Logs can be events such as checkout, an exception, or an HTTP request. For this 17:17:40.019 [INFO ] de.spinscale.javalin.App - This is an informative logging message, user agent [curl/7.64.1] ``` - - ### Log requests [_log_requests] Depending on the application traffic and whether it happens outside of the application, it makes sense to log each request on the application level. @@ -321,8 +312,6 @@ Depending on the application traffic and whether it happens outside of the appli 10:43:50.066 [INFO ] de.spinscale.javalin.App - GET / 200 0:0:0:0:0:0:0:1 "curl/7.64.1" 7 ``` - - ### Create an ISO8601 timestamp [_create_an_iso8601_timestamp] Before ingesting logs into {{ech}}, create an ISO8601 timestamp by editing the `log4j2.xml` file. @@ -332,7 +321,6 @@ Creating an ISO8601 timestamp removes the need to do any calculation for timesta :::: - ```text ``` @@ -343,7 +331,6 @@ The log entries are ingested containing timestamps like the following. 2020-07-03T14:25:40,378+02:00 [INFO ] de.spinscale.javalin.App GET / 200 0:0:0:0:0:0:0:1 "curl/7.64.1" 0 ``` - ### Log to a file and stdout [_log_to_a_file_and_stdout] 1. To read the logging output, let’s write data into a file and to stdout. This is a new `log4j2.xml` file. @@ -371,7 +358,6 @@ The log entries are ingested containing timestamps like the following. 2. Restart the application and send a request. The logs will be sent to `/tmp/javalin/app.log`. - ### Install and configure {{filebeat}} [_install_and_configure_filebeat] To read the log file and send it to {{es}}, {{filebeat}} is required. To download and install {{filebeat}}, use the commands that work with your system: @@ -433,7 +419,6 @@ NOTE: If script execution is disabled on your system, you need to set the execut :::: - ```bash ./filebeat keystore create echo -n "" | ./filebeat keystore add CLOUD_ID --stdin @@ -472,7 +457,6 @@ NOTE: If script execution is disabled on your system, you need to set the execut :::: - To see if both settings have been stored, run `./filebeat keystore list`. 3. To load the {{filebeat}} dashboards, use the `elastic` super user. @@ -501,8 +485,6 @@ NOTE: If script execution is disabled on your system, you need to set the execut api_key: ${ES_API_KEY} ``` - - ### Send data to {{es}} [_send_data_to_es] To send data to {{es}}, start {{filebeat}}. @@ -518,7 +500,6 @@ sudo service filebeat start If you use an `init.d` script to start Filebeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/filebeat/master/command-line-options.html)). To specify flags, start Filebeat in the foreground. :::: - Also see [Filebeat and systemd](https://www.elastic.co/guide/en/beats/filebeat/master/running-with-systemd.html). :::::: @@ -531,7 +512,6 @@ sudo service filebeat start If you use an `init.d` script to start Filebeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/filebeat/master/command-line-options.html)). To specify flags, start Filebeat in the foreground. :::: - Also see [Filebeat and systemd](https://www.elastic.co/guide/en/beats/filebeat/master/running-with-systemd.html). :::::: @@ -570,7 +550,6 @@ wrk -t1 -c 100 -d10s http://localhost:7000 This command results in roughly 8,000 requests per second, and the equivalent number of log lines are also written. - ## Step 3: View logs in {{kib}} [_step_3_view_logs_in_kib] 1. Log into {{kib}} and select the **Discover** app. @@ -632,11 +611,8 @@ This command results in roughly 8,000 requests per second, and the equivalent nu * When you compare the `@timestamp` field with the timestamp of the log message, you will notice that it differs. This means that you don’t get the results you expect when filtering based on the `@timestamp` field. The current `@timestamp` field reflects the timestamp when the event was created within {{filebeat}}, not the timestamp of when the log event occurred in the application. * One cannot filter on specific fields like the HTTP verb, the HTTP status code, the log level or the class that generated the log message - - ## Step 4: Work with your logs [_step_4_work_with_your_logs] - ### Structure logs [_structure_logs] To extract more data from a single log line into several fields requires additional structuring of the logs. @@ -683,7 +659,6 @@ The removal of parts of the original message is debatable. Keeping the original :::: - There is also a slight difference in the parsing of a timestamp as the go time parser only accepts dots as a separator between seconds and milliseconds. Still, our default output of the log4j2 is using a comma. Either one can fix the timestamp in the logging output to look like one expected from {{filebeat}}. This results in the following pattern layout. @@ -779,7 +754,6 @@ When the log level is `INFO`, it is logged with additional space at the end. You ``` - ### Parse exceptions [_parse_exceptions] Exceptions are a special treat in the case of logging. They span multiple lines, so the old rule of one message per line does not exist in exceptions. @@ -830,10 +804,8 @@ This introduces state into your logging. You cannot split a log file among sever :::: - After restarting {{filebeat}} and your Javalin app, trigger an exception, and you will see a long stack trace in the `message` field of your logs. - ### Configure log rotation [_configure_log_rotation] To ensure that logs don’t grow infinitely, let’s add some log rotation to your logging configuration. @@ -870,7 +842,6 @@ The sample added a `JavalinAppLogRolling` appender to our configuration that use If a new log file is created, older log files are gzipped as well to take less space on disk. The size of 50 megabytes refers to the unpacked file size, so the potentially twenty files on disk will be much smaller each. - ### Ingest node [_ingest_node] The built-in modules are almost entirely using the [Ingest node](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md) feature of {{es}} instead of the {{beats}} processors. @@ -939,7 +910,6 @@ One of the most helpful parts of the ingest pipeline is the ability to debug by 4. Restart {{filebeat}} and see if logs are flowing in as expected. - ### Write logs as JSON [_write_logs_as_json] You have now learned about parsing logs in either {{beats}} or {{es}}. What if we didn’t need to think about parsing our logs and extract data manually? @@ -1007,8 +977,6 @@ While log4j2 has a [JSONLayout](https://logging.apache.org/log4j/2.x/manual/layo As you can see, just by writing out logs as JSON, our whole logging setup got a ton easier, so whenever possible, try to directly write your logs as JSON. - - ## Step 5: Ingest metrics [_step_5_ingest_metrics] A metric is considered a point in time value that can change anytime. The number of current requests can change any millisecond. You could have a spike of 1000 requests, and then everything goes back to one request. This also means that these kinds of metrics may not be accurate, and you also want to pull min/max values to get some more indication. Furthermore, this implies that you need to think about the duration of those metrics as well. Do you need those once per minute or every 10 seconds? @@ -1017,7 +985,6 @@ To get a different angled view of your application, let’s ingest some metrics. The underlying library used in our app is [micrometer.io](http://micrometer.io/), a vendor-neutral application metrics facade in combination with its [Prometheus support](http://micrometer.io/docs/registry/prometheus) to implement a pull-based model. You could use the [elastic support](http://micrometer.io/docs/registry/elastic) to achieve a push-based model. This would require users to store credential data of the {{es}} cluster in our app. This example keeps this data in the surrounding tools. - ### Add metrics to the application [_add_metrics_to_the_application] 1. Add the dependencies to our `build.gradle` file. @@ -1117,8 +1084,6 @@ The underlying library used in our app is [micrometer.io](http://micrometer.io/) This returns a line based response with one metric per line. This is the standard Prometheus format. - - ### Install and configure {{metricbeat}} [_install_and_configure_metricbeat] To send metrics to {{es}}, {{metricbeat}} is required. To download and install {{metricbeat}}, use the commands that work with your system: @@ -1260,7 +1225,6 @@ sudo service metricbeat start If you use an `init.d` script to start Metricbeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/metricbeat/master/command-line-options.html)). To specify flags, start Metricbeat in the foreground. :::: - Also see [Metricbeat and systemd](https://www.elastic.co/guide/en/beats/metricbeat/master/running-with-systemd.html). :::::: @@ -1273,7 +1237,6 @@ sudo service metricbeat start If you use an `init.d` script to start Metricbeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/metricbeat/master/command-line-options.html)). To specify flags, start Metricbeat in the foreground. :::: - Also see [Metricbeat and systemd](https://www.elastic.co/guide/en/beats/metricbeat/master/running-with-systemd.html). :::::: @@ -1321,7 +1284,6 @@ GET metricbeat-*/_search?filter_path=**.prometheus,hits.total } ``` - ## Step 6: View metrics in {{kib}} [_step_6_view_metrics_in_kib] As this is custom data from our Javalin app, there is no predefined dashboard for displaying this data. @@ -1352,8 +1314,6 @@ Visualize the number of log messages over time, split by the log level. Since th ![Date Histogram of the log rate per log level](/solutions/images/observability-monitor-java-app-metrics-kibana-create-visualization-log-rate.png "") - - ### Visualize open files over time [_visualize_open_files_over_time] The second visualization is to be a check for the number of open files in our application. @@ -1436,8 +1396,6 @@ Let’s look at the `process_files_open_files` metric. This should be a rather s This is an area chart of the total events counter the Javalin app emits. It’s rising because there is a component polling an endpoint that, in turn, produces another log message. The steeper peek was due to sending more requests. But where is the sudden drop-off coming from? A JVM restart. As those metrics are not persisted, they reset on a JVM restart. With that in mind, it’s often better to log the `rate` instead of the `counter` field. - - ## Step 7: Instrument the application [_step_7_instrument_the_application] The third piece of {{observability}} is Application Performance Management (APM). An APM setup consists of an APM server which accepts the data (and is already running within our {{ecloud}} setup) and an agent delivering the data to the server. @@ -1448,7 +1406,6 @@ One of the APM’s core ideas is the ability to follow the flow of a user sessio To fully capture user activity, you need to start in the browser of the user using Real User Monitoring (RUM) down to your application, which sends a SQL query to your database. - ### Data Model [_data_model] Despite a heavily fragmented APM landscape, the terminology often is similar. The two most important terms are **Spans** and **Transactions**. @@ -1463,7 +1420,6 @@ The Java {{apm-agent}} can instrument specific frameworks automatically. Spring There are currently agents for Go, .NET, Node, Python, Ruby, and the browser (RUM). Agents keep getting added so you may want to check the [APM agent documentation](https://www.elastic.co/guide/en/apm/agent/index.html). - ### Add the {{apm-agent}} to your code [_add_the_apm_agent_to_your_code] You have two options to add Java agent instrumentation to your application. @@ -1491,7 +1447,6 @@ java -javaagent:/path/to/elastic-apm-agent-1.17.0.jar\ You could now go ahead and open up the Applications UI and you should see the data flowing in. - ### Automatic attachment [_automatic_attachment] If you do not want to change the startup options of your application, the standalone agent allows you to attach to running JVMs on a host. @@ -1525,7 +1480,6 @@ So now the agent was attached to a running application with a special configurat While both of the first two possibilities work, you can also use the third one: using the {{apm-agent}} as a direct dependency. This allows you to write custom spans and transactions within our application. - ### Programmatic setup [_programmatic_setup] A programmatic setup allows you to attach the agent via a line of java in your source code. @@ -1589,8 +1543,6 @@ A programmatic setup allows you to attach the agent via a line of java in your s The only transaction comes from a single servlet, which is not too helpful. Let’s try to fix this by introducing custom programmatic transactions. - - ### Custom transactions [_custom_transactions] 1. Add another dependency. @@ -1660,8 +1612,6 @@ A programmatic setup allows you to attach the agent via a line of java in your s + " " + ctx.endpointHandlerPath())); ``` - - ### Method tracing via agent configuration [_method_tracing_via_agent_configuration] Instead of writing code to trace methods, you can also configure the agent to do this. Let’s try to figure out if logging is a bottleneck for our application and trace the request logger statements we added earlier. @@ -1713,8 +1663,6 @@ ELASTIC_APM_TRACE_METHODS="de.spinscale.javalin.Log4j2RequestLogger#handle" If you are on the quest for a faster service, you may want to rethink logging. However, this logging happens after the result is written to the client, so while the total processing time increases with logging, responding to the client does not (closing the connection however might be). Also note that these tests were conducted without a proper warm-up. I assume that after appropriate JVM warm-up, you will have much faster processing of requests. - - ### Automatic profiling of inferred spans [_automatic_profiling_of_inferred_spans] Once you have a bigger application with more code paths than our sample app, you can try to enable the [automatic profiling of inferred spans](apm-agent-java://reference/config-profiling.md#config-profiling-inferred-spans-enabled) by setting the following. @@ -1725,7 +1673,6 @@ ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED=true This mechanism uses the [async profiler](https://github.com/jvm-profiling-tools/async-profiler) to create spans without you having to instrument anything allowing you to find bottlenecks faster. - ### Log correlation [_log_correlation] Transaction ids are automatically added to logs. You can check the generated log files that are sent to {{es}} via {{filebeat}}. An entry looks like this. @@ -1751,8 +1698,6 @@ We have not covered the [Elastic APM OpenTracing bridge](apm-agent-java://refere :::: - - ## Step 8: Ingest Uptime data [_step_8_ingest_uptime_data] ```{applies_to} @@ -1765,7 +1710,6 @@ How about checking if our users have the same experience that our APM data is su [Uptime](https://www.elastic.co/uptime-monitoring) not only enables you to monitor the availability of a service, but also graph latencies over time, and get notified about expiring TLS certificates. - ### Setup [_setup] To send uptime data to {{es}}, {{heartbeat}} (the polling component) is required. To download and install {{heartbeat}}, use the commands that work with your system: @@ -1902,7 +1846,6 @@ sudo service heartbeat start If you use an `init.d` script to start Heartbeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/heartbeat/master/command-line-options.html)). To specify flags, start Heartbeat in the foreground. :::: - Also see [Heartbeat and systemd](https://www.elastic.co/guide/en/beats/heartbeat/master/running-with-systemd.html). :::::: @@ -1915,7 +1858,6 @@ sudo service heartbeat start If you use an `init.d` script to start Heartbeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/heartbeat/master/command-line-options.html)). To specify flags, start Heartbeat in the foreground. :::: - Also see [Heartbeat and systemd](https://www.elastic.co/guide/en/beats/heartbeat/master/running-with-systemd.html). :::::: @@ -1960,7 +1902,6 @@ The duration of the monitor is in the low milliseconds, as it is really fast. Ch Do not underestimate the importance of this kind of monitoring. Also, consider this just the beginning as the next step is to have synthetics that monitor the correct behavior of your application, for example, to ensure that your checkout process works all the time. - ## What’s next? [_whats_next] For more information about using Elastic {{observability}}, see the [{{observability}} documentation](../get-started/what-is-elastic-observability.md). diff --git a/solutions/observability/apps/upgrade-elastic-cloud-apm-server-standalone-to-90.md b/solutions/observability/apps/upgrade-elastic-cloud-apm-server-standalone-to-90.md index f26ced3e9b..ad7abf0768 100644 --- a/solutions/observability/apps/upgrade-elastic-cloud-apm-server-standalone-to-90.md +++ b/solutions/observability/apps/upgrade-elastic-cloud-apm-server-standalone-to-90.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade-8.0-cloud-standalone.html --- - - # Upgrade Elastic Cloud APM Server standalone to 8. [apm-upgrade-8.0-cloud-standalone] - This upgrade guide is for the standalone method of running APM Server. Only use this guide if both of the following are true: * You’re using {{ecloud}}. diff --git a/solutions/observability/apps/upgrade-elastic-cloud-with-apm-integration-to-90.md b/solutions/observability/apps/upgrade-elastic-cloud-with-apm-integration-to-90.md index 6d691df7f9..ba1db80cff 100644 --- a/solutions/observability/apps/upgrade-elastic-cloud-with-apm-integration-to-90.md +++ b/solutions/observability/apps/upgrade-elastic-cloud-with-apm-integration-to-90.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade-8.0-cloud-integration.html --- - - # Upgrade Elastic Cloud with the APM integration to 8.0 [apm-upgrade-8.0-cloud-integration] - This upgrade guide is for the Elastic APM integration. Only use this guide if both of the following are true: * You’re using {{ecloud}}. @@ -24,4 +21,3 @@ Follow these steps to upgrade: {{ece}} users require additional TLS setup. Refer to [Add APM user settings](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-manage-apm-settings.html) for more information. :::: - diff --git a/solutions/observability/apps/upgrade-self-installation-of-apm-integration-to-90.md b/solutions/observability/apps/upgrade-self-installation-of-apm-integration-to-90.md index 27bcf9202d..2c1adb4fe3 100644 --- a/solutions/observability/apps/upgrade-self-installation-of-apm-integration-to-90.md +++ b/solutions/observability/apps/upgrade-self-installation-of-apm-integration-to-90.md @@ -4,24 +4,19 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade-8.0-self-integration.html --- - - # Upgrade a self-installation of the APM integration to 9.0 [apm-upgrade-9.0-self-integration] - This upgrade guide is for the Elastic APM integration. Only use this guide if both of the following are true: * You have a self-installation of the {{stack}}, i.e. you’re not using {{ecloud}}. * You have already switched to and are running {{fleet}} and the Elastic APM integration. - ## Prerequisites [_prerequisites_9] 1. Before you upgrade to version 9.0, you must upgrade {{es}} and {{kib}} to the latest patch version of 8.18. For more details, refer to the [{{stack}} Installation and Upgrade Guide](https://www.elastic.co/guide/en/elastic-stack/8.18/upgrading-elastic-stack.html). 2. Review the [Elastic APM release notes](apm-server://release-notes/index.md) and [Elastic {{observability}} release notes](/release-notes/elastic-observability/index.md). 3. Review the [Elastic APM breaking changes](apm-server://release-notes/breaking-changes.md). - ## Upgrade steps [_upgrade_steps_2] 1. Upgrade the {{stack}} to version 9.0. diff --git a/solutions/observability/apps/upgrade-self-installation-of-apm-server-standalone-to-90.md b/solutions/observability/apps/upgrade-self-installation-of-apm-server-standalone-to-90.md index 0d4bdd8484..2810855e26 100644 --- a/solutions/observability/apps/upgrade-self-installation-of-apm-server-standalone-to-90.md +++ b/solutions/observability/apps/upgrade-self-installation-of-apm-server-standalone-to-90.md @@ -4,17 +4,13 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade-8.0-self-standalone.html --- - - # Upgrade a self-installation of APM Server standalone to 9.0 [apm-upgrade-9.0-self-standalone] - This upgrade guide is for the standalone method of running APM Server. Only use this guide if both of the following are true: * You have a self-installation of the {{stack}}, i.e. you’re not using {{ecloud}}. * You’re running the APM Server binary, i.e. you haven’t switched to the Elastic APM integration. - ## Prerequisites [_prerequisites_8] 1. Prior to upgrading to version 9.0, {{es}}, {{kib}}, and APM Server must be upgraded to version 8.18. @@ -25,8 +21,6 @@ This upgrade guide is for the standalone method of running APM Server. Only use 2. Review the [Elastic APM release notes](apm-server://release-notes/index.md) and [Elastic {{observability}} release notes](/release-notes/elastic-observability/index.md). 3. Review the [Elastic APM breaking changes](apm-server://release-notes/breaking-changes.md). - - ## Upgrade steps [_upgrade_steps] 1. **Upgrade the {{stack}} to version 9.0** diff --git a/solutions/observability/apps/upgrade-to-version-90.md b/solutions/observability/apps/upgrade-to-version-90.md index 97c05e30e9..dfcb10b15b 100644 --- a/solutions/observability/apps/upgrade-to-version-90.md +++ b/solutions/observability/apps/upgrade-to-version-90.md @@ -11,7 +11,6 @@ This guide explains the upgrade process for version 9.0. For a detailed look at * [What’s new in {{kib}}](https://www.elastic.co/guide/en/kibana/current/whats-new.html) * [{{es}} release highlights](https://www.elastic.co/guide/en/elasticsearch/reference/current/release-highlights.html) - ## Notable APM changes [_notable_apm_changes] * All index management has been removed from APM Server; the built-in {{es}} apm-data plugin is entirely responsible for setting up index templates, index lifecycle polices, and index pipelines. @@ -20,7 +19,6 @@ This guide explains the upgrade process for version 9.0. For a detailed look at As a result of the above changes, a number of index management and index tuning configuration variables have been removed. See the APM [release notes](apm-server://release-notes/index.md), [breaking changes](https://www.elastic.co/guide/en/observability/current/apm-breaking.html) for full details. - ## Find your upgrade guide [_find_your_upgrade_guide] Starting in version 7.14, there are two ways to run Elastic APM. Determine which method you’re using, then use the links below to find the correct upgrading guide. diff --git a/solutions/observability/apps/uptime-monitoring-deprecated.md b/solutions/observability/apps/uptime-monitoring-deprecated.md index 7997d7819c..7b5ccc8001 100644 --- a/solutions/observability/apps/uptime-monitoring-deprecated.md +++ b/solutions/observability/apps/uptime-monitoring-deprecated.md @@ -14,7 +14,6 @@ applies_to: Use [Synthetic monitoring](/solutions/observability/apps/synthetic-monitoring.md) instead of the {{uptime-app}}. :::: - ::::{important} The {{uptime-app}} is for viewing result data from lightweight monitors running through {{heartbeat}} and [configured with a traditional `heartbeat.yml` file](get-started-with-uptime.md). This is for TCP, HTTP or ICMP monitors that you have configured and run from your own infrastructure with {{heartbeat}} natively. @@ -24,10 +23,8 @@ Note that the {{uptime-app}} is hidden from the interface when there is no recen :::: - The {{uptime-app}} uses {{agent}} to periodically check the status of your services and applications. Monitor the availability of network endpoints and services using [Lightweight HTTP/S, TCP, and ICMP monitors](synthetic-monitoring.md#monitoring-uptime). - ## Lightweight HTTP/S, TCP, and ICMP monitors [uptime-lightweight] In the {{uptime-app}}, you can monitor the status of network endpoints using the following lightweight checks: @@ -45,7 +42,6 @@ In the {{uptime-app}}, you can monitor the status of network endpoints using the To set up your first monitor, refer to [Get started with Uptime](get-started-with-uptime.md). - ## TLS Certificates [view-certificate-status] The TLS Certificates page in the {{uptime-app}} lists the TLS certificates that are being monitored and shows the TLS certificate data in your indices. diff --git a/solutions/observability/apps/use-advanced-queries-on-application-data.md b/solutions/observability/apps/use-advanced-queries-on-application-data.md index c4820bb1bc..83b6d8ef03 100644 --- a/solutions/observability/apps/use-advanced-queries-on-application-data.md +++ b/solutions/observability/apps/use-advanced-queries-on-application-data.md @@ -10,7 +10,6 @@ applies_to: # Use advanced queries on your application data [apm-advanced-queries] - Querying your APM data is an essential tool that can make finding bottlenecks in your code even more straightforward. Using the query bar, a powerful data query feature, you can pass advanced queries on your data to filter on specific pieces of information you’re interested in. APM queries entered into the query bar are added as parameters to the URL, so it’s easy to share a specific query or view with others. @@ -29,7 +28,6 @@ To learn more about the {{kib}} query language capabilities, see the [Kibana Que :::: - ### APM queries [apm-app-queries] APM queries can be handy for removing noise from your data in the [Services](../../../solutions/observability/apps/services.md), [Transactions](../../../solutions/observability/apps/transactions-2.md), [Errors](../../../solutions/observability/apps/errors-2.md), [Metrics](../../../solutions/observability/apps/metrics-2.md), and [Traces](../../../solutions/observability/apps/traces-2.md) views. @@ -40,12 +38,10 @@ On the **Traces** view, you might want to view failed transaction results from a On the **Transactions** view, you may want to list only the slower transactions than a specified time threshold: `transaction.duration.us > 2000000`. Or filter the list by including the service version and the Kubernetes pod it’s running on: `transaction.duration.us > 2000000 and service.version : "7.12.0" and kubernetes.pod.name : "pod-5468b47f57-pqk2m"`. - ## Querying in Discover [discover-advanced-queries] Alternatively, you can query your APM documents in [**Discover**](../../../explore-analyze/discover.md). Querying documents in **Discover** works the same way as queries in the Applications UI, and **Discover** supports all of the example APM queries shown on this page. - ### Discover queries [discover-queries] One example where you may want to make use of **Discover** is to view *all* transactions for an endpoint instead of just a sample. diff --git a/solutions/observability/apps/use-environment-variables-in-configuration.md b/solutions/observability/apps/use-environment-variables-in-configuration.md index 8e5ba8920f..a48b70b619 100644 --- a/solutions/observability/apps/use-environment-variables-in-configuration.md +++ b/solutions/observability/apps/use-environment-variables-in-configuration.md @@ -14,7 +14,6 @@ This documentation is only relevant for APM Server binary users. :::: - You can use environment variable references in the config file to set values that need to be configurable during deployment. To do this, use: `${VAR}` @@ -46,8 +45,6 @@ You can also specify environment variables when you override a config setting fr :::: - - ## Examples [_examples] Here are some examples of configurations that use environment variables and what each configuration looks like after replacement: @@ -61,7 +58,6 @@ Here are some examples of configurations that use environment variables and what | `name: ${NAME:?You need to set the NAME environment variable}` | no setting | None. Returns an error message that’s prepended with the custom text. | | `name: ${NAME:?You need to set the NAME environment variable}` | `export NAME=elastic` | `name: elastic` | - ## Specify complex objects in environment variables [_specify_complex_objects_in_environment_variables] You can specify complex objects, such as lists or dictionaries, in environment variables by using a JSON-like syntax. @@ -83,7 +79,6 @@ output.elasticsearch: When APM Server loads the config file, it resolves the environment variable and replaces it with the specified list before reading the `hosts` setting. - ### Opt-out of complex object parsing [_opt_out_of_complex_object_parsing] To opt-out of this complex object parsing behavior such that the environment variable is always parsed as a string, wrap the environment variable in double-quotes (`"`), and escape all double quotes within them. @@ -105,4 +100,3 @@ ES_PASSWORD='"foo,bar\"baz"' Do not use double-quotes (`"`) to wrap regular expressions. Otherwise, the backslash (`\`) will be interpreted as an escape character. :::: - diff --git a/solutions/observability/apps/use-internal-collection-to-send-monitoring-data.md b/solutions/observability/apps/use-internal-collection-to-send-monitoring-data.md index c95fda3d5c..ba2516a67a 100644 --- a/solutions/observability/apps/use-internal-collection-to-send-monitoring-data.md +++ b/solutions/observability/apps/use-internal-collection-to-send-monitoring-data.md @@ -8,7 +8,6 @@ applies_to: # Use internal collection to send monitoring data [apm-monitoring-internal-collection] - Use internal collectors to send {{beats}} monitoring data directly to your monitoring cluster. Or as an alternative to internal collection, use [Use {{metricbeat}} collection](use-metricbeat-to-send-monitoring-data.md). The benefit of using internal collection instead of {{metricbeat}} is that you have fewer pieces of software to install and maintain. 1. Create an API key or user that has appropriate authority to send system-level monitoring data to {{es}}. For example, you can use the built-in `apm_system` user or assign the built-in `apm_system` role to another user. For more information on the required privileges, see [Create a *monitoring* role](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-to-publish-monitoring). For more information on how to use API keys, see [Grant access using API keys](grant-access-using-api-keys.md). @@ -25,7 +24,6 @@ Use internal collectors to send {{beats}} monitoring data directly to your monit 1. Specify one of `api_key` or `username`/`password`. - If you want to send monitoring events to an [{{ecloud}}](https://cloud.elastic.co/) monitoring cluster, you can use two simpler settings. When defined, these settings overwrite settings from other parts in the configuration. For example: ```yaml @@ -52,7 +50,6 @@ Use internal collectors to send {{beats}} monitoring data directly to your monit 2. This setting identifies the hosts and port numbers of {{es}} nodes that are part of the monitoring cluster. 3. Specify one of `api_key` or `username`/`password`. - If you want to use PKI authentication to send monitoring events to {{es}}, you must specify a different set of configuration options. For example: ```yaml @@ -72,106 +69,86 @@ Use internal collectors to send {{beats}} monitoring data directly to your monit 3. Start APM Server. 4. [View the monitoring data in {{kib}}](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). - ## Settings for internal collection [apm-configuration-monitor] Use the following settings to configure internal collection when you are not using {{metricbeat}} to collect monitoring data. You specify these settings in the X-Pack monitoring section of the `apm-server.yml` config file: - ### `monitoring.enabled` [_monitoring_enabled] The `monitoring.enabled` config is a boolean setting to enable or disable {{monitoring}}. If set to `true`, monitoring is enabled. The default value is `false`. - ### `monitoring.elasticsearch` [_monitoring_elasticsearch] The {{es}} instances that you want to ship your APM Server metrics to. This configuration option contains the following fields: - #### `api_key` [_api_key_3] The detail of the API key to be used to send monitoring information to {{es}}. See [Grant access using API keys](grant-access-using-api-keys.md) for more information. - #### `bulk_max_size` [_bulk_max_size_5] The maximum number of metrics to bulk in a single {{es}} bulk API index request. The default is `50`. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `backoff.init` [_backoff_init_5] The number of seconds to wait before trying to reconnect to {{es}} after a network error. After waiting `backoff.init` seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is `1s`. - #### `backoff.max` [_backoff_max_5] The maximum number of seconds to wait before attempting to connect to {{es}} after a network error. The default is `60s`. - #### `compression_level` [_compression_level_4] The gzip compression level. Setting this value to `0` disables compression. The compression level must be in the range of `1` (best speed) to `9` (best compression). The default value is `0`. Increasing the compression level reduces the network usage but increases the CPU usage. - #### `headers` [_headers_2] Custom HTTP headers to add to each request. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `hosts` [_hosts_4] The list of {{es}} nodes to connect to. Monitoring metrics are distributed to these nodes in round robin order. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `max_retries` [_max_retries_5] The number of times to retry sending the monitoring metrics after a failure. After the specified number of retries, the metrics are typically dropped. The default value is `3`. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `parameters` [_parameters_2] Dictionary of HTTP parameters to pass within the URL with index operations. - #### `password` [_password_4] The password that APM Server uses to authenticate with the {{es}} instances for shipping monitoring data. - #### `metrics.period` [_metrics_period] The time interval (in seconds) when metrics are sent to the {{es}} cluster. A new snapshot of APM Server metrics is generated and scheduled for publishing each period. The default value is 10 * time.Second. - #### `state.period` [_state_period] The time interval (in seconds) when state information are sent to the {{es}} cluster. A new snapshot of APM Server state is generated and scheduled for publishing each period. The default value is 60 * time.Second. - #### `protocol` [_protocol] The name of the protocol to use when connecting to the {{es}} cluster. The options are: `http` or `https`. The default is `http`. If you specify a URL for `hosts`, however, the value of protocol is overridden by the scheme you specify in the URL. - #### `proxy_url` [_proxy_url_4] The URL of the proxy to use when connecting to the {{es}} cluster. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `timeout` [_timeout_5] The HTTP request timeout in seconds for the {{es}} request. The default is `90`. - #### `ssl` [_ssl_5] Configuration options for Transport Layer Security (TLS) or Secure Sockets Layer (SSL) parameters like the certificate authority (CA) to use for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to {{es}}. For more information, see [SSL/TLS output settings](ssltls-output-settings.md). - #### `username` [_username_3] The user ID that APM Server uses to authenticate with the {{es}} instances for shipping monitoring data. diff --git a/solutions/observability/apps/use-metricbeat-to-send-monitoring-data.md b/solutions/observability/apps/use-metricbeat-to-send-monitoring-data.md index eceb54f4d4..ae24fcf574 100644 --- a/solutions/observability/apps/use-metricbeat-to-send-monitoring-data.md +++ b/solutions/observability/apps/use-metricbeat-to-send-monitoring-data.md @@ -8,7 +8,6 @@ applies_to: # Use Metricbeat to send monitoring data [apm-monitoring-metricbeat-collection] - In 7.3 and later, you can use {{metricbeat}} to collect data about APM Server and ship it to the monitoring cluster. The benefit of using {{metricbeat}} instead of internal collection is that the monitoring agent remains active even if the APM Server instance dies. To collect and ship monitoring data: @@ -16,7 +15,6 @@ To collect and ship monitoring data: 1. [Configure the shipper you want to monitor](#apm-configure-shipper) 2. [Install and configure {{metricbeat}} to collect monitoring data](#apm-configure-metricbeat) - ## Configure the shipper you want to monitor [apm-configure-shipper] 1. Enable the HTTP endpoint to allow external collection of monitoring data: @@ -61,7 +59,6 @@ To collect and ship monitoring data: 5. Start APM Server. - ## Install and configure {{metricbeat}} to collect monitoring data [apm-configure-metricbeat] 1. Install {{metricbeat}} on the same server as APM Server. To learn how, see [Get started with {{metricbeat}}](beats://reference/metricbeat/metricbeat-installation-configuration.md). If you already have {{metricbeat}} installed on the server, skip this step. @@ -122,7 +119,6 @@ To collect and ship monitoring data: In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. :::: - For example, specify the {{es}} output information in the {{metricbeat}} configuration file (`metricbeat.yml`): ```yaml @@ -140,14 +136,12 @@ To collect and ship monitoring data: 1. In this example, the data is stored on a monitoring cluster with nodes `es-mon-1` and `es-mon-2`. 2. Specify one of `api_key` or `username`/`password`. - If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200`. ::::{important} The {{es}} {{monitor-features}} use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node. :::: - If the {{es}} {{security-features}} are enabled on the monitoring cluster, you must provide a valid user ID and password so that {{metricbeat}} can send metrics successfully: 1. Create a user on the monitoring cluster that has the `remote_monitoring_agent` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). diff --git a/solutions/observability/apps/use-select-metrics-emitted-directly-to-monitoring-cluster.md b/solutions/observability/apps/use-select-metrics-emitted-directly-to-monitoring-cluster.md index 0cb53f746b..e5992b80d6 100644 --- a/solutions/observability/apps/use-select-metrics-emitted-directly-to-monitoring-cluster.md +++ b/solutions/observability/apps/use-select-metrics-emitted-directly-to-monitoring-cluster.md @@ -8,7 +8,6 @@ applies_to: # Use the select metrics emitted directly to your monitoring cluster [apm-monitoring-local-collection] - In 8.11 and later, we emit a selected set of metrics directly to the monitoring cluster. The benefit of using local collection instead of internal collection is that the metrics are sent directly to your main monitoring index, making it easier to view shared data. ## The select metrics [apm-select-metrics] diff --git a/solutions/observability/apps/use-synthetics-cli.md b/solutions/observability/apps/use-synthetics-cli.md index 577a1c19b4..61a8db18d4 100644 --- a/solutions/observability/apps/use-synthetics-cli.md +++ b/solutions/observability/apps/use-synthetics-cli.md @@ -10,8 +10,6 @@ applies_to: # Use the Synthetics CLI [synthetics-command-reference] - - ## `@elastic/synthetics` [elastic-synthetics-command] Elastic uses the [@elastic/synthetics](https://www.npmjs.com/package/@elastic/synthetics) Node.js library to run synthetic browser tests and report the test results. The library also provides a CLI to help you scaffold, develop/run tests locally, and push tests to {{kib}}. @@ -36,19 +34,16 @@ You will not need to use most command line flags. However, there are some you ma Params passed will be merged with params defined in your [`synthetics.config.js` file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-params). Params defined via the CLI take precedence. - `--playwright-options ` : JSON object to pass in custom Playwright options for the agent. For more details on relevant Playwright options, refer to the [the configuration docs](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-playwright-options). Options passed will be merged with Playwright options defined in your [`synthetics.config.js` file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-playwright-options). Options defined via the CLI take precedence. - `--screenshots ` : Control whether or not to capture screenshots at the end of each step. Options include `'on'`, `'off'`, or `'only-on-failure'`. This can also be set in the configuration file using [`monitor.screenshot`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - `-c, --config ` : Path to the configuration file. By default, test runner looks for a `synthetics.config.(js|ts)` file in the current directory. Synthetics configuration provides options to configure how your tests are run and pushed to Elastic. Allowed options are described in the [configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md). @@ -63,25 +58,21 @@ You will not need to use most command line flags. However, there are some you ma Throttling can also be disabled in the configuration file using [`monitor.throttling`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - ::::{note} Network throttling for browser based monitors is disabled. See this [documention](https://github.com/elastic/synthetics/blob/main/docs/throttling.md) for more details. :::: - `--no-headless` : Runs with the browser in headful mode. This is the same as setting [Playwright’s `headless` option](https://playwright.dev/docs/api/class-testoptions#test-options-headless) to `false` by running `--playwright-options '{"headless": false}'`. - ::::{note} Headful mode should only be used locally to see the browser and interact with DOM elements directly for testing purposes. Do not attempt to run in headful mode when running through Elastic’s global managed testing infrastructure or {{private-location}}s as this is not supported. :::: - `-h, --help` : Shows help for the `npx @elastic/synthetics` command. @@ -90,14 +81,11 @@ The `--pattern`, `--tags`, and `--match` flags for filtering are only supported :::: - ::::{note} For debugging synthetic tests locally, you can set an environment variable, `DEBUG=synthetics npx @elastic/synthetics`, to capture Synthetics agent logs. :::: - - ## `@elastic/synthetics init` [elastic-synthetics-init-command] Scaffold a new Synthetics project using Elastic Synthetics. @@ -110,7 +98,6 @@ npx @elastic/synthetics init Read more about what’s included in a template Synthetics project in [Create a Synthetics project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md). - ## `@elastic/synthetics push` [elastic-synthetics-push-command] Create monitors by using your local journeys. By default, running `push` command will use the `project` settings field from the `synthetics.config.ts` file, which is set up using the `init` command. However, you can override these settings using the CLI flags. @@ -127,7 +114,6 @@ The `push` command includes interactive prompts to prevent you from accidentally :::: - ::::{note} If the journey contains external NPM packages other than the `@elastic/synthetics`, those packages will be bundled along with the journey code when the `push` command is invoked. However there are some limitations when using external packages: @@ -137,7 +123,6 @@ If the journey contains external NPM packages other than the `@elastic/synthetic :::: - `--auth ` : API key used for [authentication](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). You can also set the API key via the `SYNTHETICS_API_KEY` environment variable. @@ -147,7 +132,6 @@ If the journey contains external NPM packages other than the `@elastic/synthetic On {{obs-serverless}}, you must be logged in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access to create an API key. - `--id ` : A unique id associated with your project. It will be used for logically grouping monitors. @@ -155,7 +139,6 @@ If the journey contains external NPM packages other than the `@elastic/synthetic This can also be set in the configuration file using [`project.id`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-project). The value defined via the CLI will take precedence. - `--url ` : The URL for the deployment or Observability Serverless project to which you want to upload the monitors. @@ -168,13 +151,11 @@ If the journey contains external NPM packages other than the `@elastic/synthetic This can also be set in the configuration file using [`project.space`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-project). The value defined via the CLI will take precedence. - `--schedule ` : The interval (in minutes) at which the monitor should run. This can also be set in the configuration file using [`monitor.schedule`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - [`--locations Array`](https://github.com/elastic/synthetics/blob/v1.3.0/src/locations/public-locations.ts#L28-L37) : Where to deploy the monitor. Monitors can be deployed in multiple locations so that you can detect differences in availability and response times across those locations. @@ -182,7 +163,6 @@ If the journey contains external NPM packages other than the `@elastic/synthetic This can also be set in the configuration file using [`monitor.locations` in the configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - `--private-locations Array` : The [{{private-location}}s](../../../solutions/observability/apps/monitor-resources-on-private-networks.md) to which the monitors will be deployed. These {{private-location}}s refer to locations hosted and managed by you, whereas `locations` are hosted by Elastic. You can specify a {{private-location}} using the location’s name. @@ -190,7 +170,6 @@ If the journey contains external NPM packages other than the `@elastic/synthetic This can also be set in the configuration file using [`monitor.privateLocations` in the configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - `--fields ` : A list of key-value pairs that will be sent with each monitor event. The `fields` are appended to {{es}} documents as `labels`, and those labels are displayed in {{kib}} in the *Monitor details* panel in the [individual monitor’s *Overview* tab](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-individual-monitors-overview). @@ -198,15 +177,12 @@ If the journey contains external NPM packages other than the `@elastic/synthetic This can also be set in the configuration file using [the `monitor.fields` option](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - `--yes` : The `push` command includes interactive prompts to prevent you from accidentally deleting or duplicating monitors. If running the CLI non-interactively, you can override these prompts using the `--yes` option. When the `--yes` option is passed to `push`: * If you `push` a Synthetics project that used to contain one or more monitors but no longer contains any monitors, all monitors associated with the Synthetics project ID being pushed will be deleted. * If you `push` a Synthetics project that’s already been pushed using one Synthetics project ID and then try to `push` it using a *different* ID, it will create duplicates of all monitors in the Synthetics project. - - ## Tag monitors [tagging-and-filtering] Synthetics journeys can be tagged with one or more tags. Use tags to filter journeys when running tests locally or pushing them to Elastic. @@ -234,7 +210,6 @@ tags: To apply tags to all browser and lightweight monitors, configure using [the `monitor.tags`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor) field in the `synthetics.config.ts` file. - ### Filter monitors [_filtering_monitors] When running the `npx @elastic/synthetics push` command, you can filter the monitors that are pushed to {{kib}} using the following flags: @@ -255,7 +230,6 @@ npx @elastic/synthetics push --config synthetics.qa.config.ts --tags env:qa npx @elastic/synthetics push --config synthetics.prod.config.ts --tags env:prod ``` - ## `@elastic/synthetics locations` [elastic-synthetics-locations-command] List all available locations for running synthetics monitors. @@ -278,7 +252,6 @@ To list both locations on Elastic’s global managed infrastructure and {{privat If an administrator has disabled Elastic managed locations for the role you are assigned and you do *not* include `--url` and `--auth`, all global locations managed by Elastic will be listed. However, you will not be able to push to these locations with your API key and will see an error: *You don’t have permission to use Elastic managed global locations*. For more details, refer to the [troubleshooting docs](../../../troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-public-locations-disabled). :::: - ## `@elastic/synthetics totp ` [elastic-synthetics-totp-command] Generate a Time-based One-Time Password (TOTP) for multifactor authentication (MFA) in Synthetics. @@ -307,7 +280,6 @@ npx @elastic/synthetics totp --issuer --label