diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc new file mode 100644 index 000000000..6dcf0f939 --- /dev/null +++ b/CHANGELOG.asciidoc @@ -0,0 +1 @@ +This is a placeholder file. Release notes are published in CHANGELOG.md \ No newline at end of file diff --git a/docs/advanced-topics.asciidoc b/docs/advanced-topics.asciidoc index 39e67ff29..8c6ca1010 100644 --- a/docs/advanced-topics.asciidoc +++ b/docs/advanced-topics.asciidoc @@ -4,10 +4,9 @@ * <> * <> * <> -* <> -* <> include::./custom-instrumentation.asciidoc[Custom Instrumentation] include::./sanitizing-data.asciidoc[Sanitizing Data] +include::./how-the-agent-works.asciidoc[How the Agent works] include::./run-tests-locally.asciidoc[Run Tests Locally] -include::./logging.asciidoc[Logging Integrations] + diff --git a/docs/api.asciidoc b/docs/api.asciidoc index 961b36436..81ff36135 100644 --- a/docs/api.asciidoc +++ b/docs/api.asciidoc @@ -1,5 +1,5 @@ [[api]] -== Public API +== API reference The Elastic APM Python agent has several public APIs. Most of the public API functionality is not needed when using one of our <>, diff --git a/docs/configuration.asciidoc b/docs/configuration.asciidoc index e535358ff..f4e5233ba 100644 --- a/docs/configuration.asciidoc +++ b/docs/configuration.asciidoc @@ -135,11 +135,11 @@ Note however that this can have adverse effects on performance. The name of the environment this service is deployed in, e.g. "production" or "staging". -Environments allow you to easily filter data on a global level in the APM UI. +Environments allow you to easily filter data on a global level in the APM app. It's important to be consistent when naming environments across agents. -See {kibana-ref}/filters.html#environment-selector[environment selector] in the Kibana UI for more information. +See {apm-app-ref}/filters.html#environment-selector[environment selector] in the APM app for more information. -NOTE: This feature is fully supported in the APM UI in Kibana versions >= 7.2. +NOTE: This feature is fully supported in the APM app in Kibana versions >= 7.2. You must use the query bar to filter for a specific environment in versions prior to 7.2. [float] @@ -662,7 +662,7 @@ NOTE: This feature requires APM Server and Kibana >= 7.3. When enabled, the agent will make periodic requests to the APM Server to fetch updated configuration. -See {kibana-ref}/agent-configuration.html#agent-configuration[APM Agent Configuration] for more documentation on central agent configuration. +See {apm-app-ref}/agent-configuration.html#agent-configuration[APM Agent Configuration] for more documentation on central agent configuration. NOTE: This feature requires APM Server and Kibana >= 7.3. diff --git a/docs/django.asciidoc b/docs/django.asciidoc index d544765ca..2efcb568a 100644 --- a/docs/django.asciidoc +++ b/docs/django.asciidoc @@ -1,11 +1,11 @@ [[django-support]] -== Django Support +=== Django support Getting Elastic APM set up for your Django project is easy, and there are various ways you can tweak it to fit to your needs. [float] [[django-installation]] -=== Installation +==== Installation Install the Elastic APM agent using pip: @@ -25,7 +25,7 @@ threads]. [float] [[django-setup]] -=== Setup +==== Setup Set up the Elastic APM agent in Django with these two steps: @@ -77,7 +77,7 @@ ELASTIC_APM = { [float] [[django-performance-metrics]] -=== Performance Metrics +==== Performance metrics To send performance metrics to Elastic APM, simply add our tracing middleware to your `MIDDLEWARE` settings: @@ -95,14 +95,14 @@ Make sure that it is the first middleware in the list. [float] [[django-instrumenting-custom-python-code]] -==== Instrumenting custom Python code +===== Instrumenting custom Python code To gain further insights into the performance of your code, please see <>. [float] [[django-ignoring-specific-views]] -==== Ignoring specific views +===== Ignoring specific views You can use the `TRANSACTIONS_IGNORE_PATTERNS` configuration option to ignore specific views. The list given should be a list of regular expressions which are matched against the transaction name as seen in the Elastic APM user interface: @@ -116,7 +116,7 @@ This example ignores any requests using the `OPTIONS` method and any requests co [float] [[django-transaction-name-route]] -==== Using the route as transaction name +===== Using the route as transaction name By default, we use the function or class name of the view as the transaction name. Starting with Django 2.2, Django makes the route (e.g. `users//`) available on the `request.resolver_match` object. @@ -131,7 +131,7 @@ NOTE: in versions previous to Django 2.2, changing this setting will have no eff [float] [[django-integrating-with-the-rum-agent]] -==== Integrating with the RUM agent +===== Integrating with the RUM Agent To correlate performance measurement in the browser with measurements in your Django app, you can help the RUM (Real User Monitoring) agent by configuring it with the Trace ID and Span ID of the backend request. @@ -173,7 +173,7 @@ See the {apm-rum-ref}[JavaScript RUM agent documentation] for more information. [float] [[django-enabling-and-disabling-the-agent]] -=== Enabling and disabling the agent +==== Enabling and disabling the agent The easiest way to disable the agent is to set Django’s `DEBUG` option to `True` in your development configuration. No errors or metrics will be logged to Elastic APM. @@ -191,7 +191,7 @@ ELASTIC_APM = { [float] [[django-logging]] -=== Integrating with Python logging +==== Integrating with Python logging To easily send Python `logging` messages as "error" objects to Elasticsearch, we provide a `LoggingHandler` which you can use in your logging setup. @@ -270,7 +270,7 @@ Without it, only the message is sent. [float] [[django-extra-data]] -==== Extra data +===== Extra data If you want to send more data than what you get with the agent by default, logging can be done like so: @@ -293,7 +293,7 @@ except MyModel.DoesNotExist: [float] [[django-celery-integration]] -=== Celery Integration +==== Celery integration For a general guide on how to set up Django with Celery, head over to Celery's http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html#django-first-steps[Django @@ -303,7 +303,7 @@ Elastic APM will automatically log errors from your celery tasks, and record per [float] [[django-logging-http-404-not-found-errors]] -=== Logging "HTTP 404 Not Found" Errors +==== Logging "HTTP 404 Not Found" errors By default, Elastic APM does not log HTTP 404 errors. If you wish to log these errors, add @@ -325,7 +325,7 @@ setting. [float] [[django-disable-agent-during-tests]] -=== Disable the agent during tests +==== Disable the agent during tests To prevent the agent from sending any data to the APM Server during tests, set the `ELASTIC_APM_DISABLE_SEND` environment variable to `true`, e.g.: @@ -336,7 +336,7 @@ ELASTIC_APM_DISABLE_SEND=true python manage.py test [float] [[django-troubleshooting]] -=== Troubleshooting +==== Troubleshooting Elastic APM comes with a Django command that helps troubleshooting your setup. To check your configuration, run @@ -369,6 +369,6 @@ Success! We tracked the error successfully! You should be able to see it in a fe [float] [[supported-django-and-python-versions]] -=== Supported Django and Python versions +==== Supported Django and Python versions A list of supported <> and <> versions can be found on our <> page. diff --git a/docs/flask.asciidoc b/docs/flask.asciidoc index 1f6b338ee..93e6f138b 100644 --- a/docs/flask.asciidoc +++ b/docs/flask.asciidoc @@ -1,12 +1,12 @@ [[flask-support]] -== Flask support +=== Flask support Getting Elastic APM set up for your Flask project is easy, and there are various ways you can tweak it to fit to your needs. [float] [[flask-installation]] -=== Installation +==== Installation Install the Elastic APM agent using pip: @@ -25,7 +25,7 @@ threads]. [float] [[flask-setup]] -=== Setup +==== Setup To set up the agent, you need to initialize it with appropriate settings. @@ -69,7 +69,7 @@ apm = ElasticAPM(app, service_name='', secret_token='') [float] [[flask-debug-mode]] -==== Debug Mode +===== Debug mode Please note that errors and transactions will only be sent to the APM Server if your app is *not* in http://flask.pocoo.org/docs/0.12/quickstart/#debug-mode[debug mode]. @@ -88,7 +88,7 @@ app.config['ELASTIC_APM'] = { [float] [[flask-building-applications-on-the-fly]] -==== Building applications on the fly? +===== Building applications on the fly? You can use the agent's `init_app` hook for adding the application on the fly: @@ -105,7 +105,7 @@ def create_app(): [float] [[flask-usage]] -=== Usage +==== Usage Once you have configured the agent, it will automatically track transactions and capture uncaught exceptions within Flask. @@ -132,7 +132,7 @@ apm.capture_message('hello, world!') [float] [[flask-logging]] -=== Logging +==== Logging Passing `logging=LEVEL` to the ElasticAPM constructor will make the agent automatically log all log messages from Python's built-in `logging` module, with the given level or higher. @@ -198,7 +198,7 @@ Without it, only the message is sent. [float] [[flask-extra-data]] -==== Extra data +===== Extra data In addition to what the agents log by default, you can send extra information: @@ -220,20 +220,20 @@ def bar(): [float] [[flask-celery-tasks]] -==== Celery tasks +===== Celery tasks The Elastic APM agent will automatically send errors and performance data from your Celery tasks to the APM Server. [float] [[flask-performance-metrics]] -=== Performance Metrics +==== Performance metrics If you've followed the instructions above, the agent has already hooked into the right signals and should be reporting performance metrics. [float] [[flask-ignoring-specific-views]] -==== Ignoring specific routes +===== Ignoring specific routes You can use the `TRANSACTIONS_IGNORE_PATTERNS` configuration option to ignore specific routes. The list given should be a list of regular expressions which are matched against the transaction name: @@ -253,7 +253,7 @@ and any requests containing `/api/`. [float] [[flask-integrating-with-the-rum-agent]] -==== Integrating with the RUM agent +===== Integrating with the RUM Agent To correlate performance measurement in the browser with measurements in your Flask app, you can help the RUM (Real User Monitoring) agent by configuring it with the Trace ID and Span ID of the backend request. @@ -277,6 +277,6 @@ See the {apm-rum-ref}[JavaScript RUM agent documentation] for more information. [float] [[supported-flask-and-python-versions]] -=== Supported Flask and Python versions +==== Supported Flask and Python versions A list of supported <> and <> versions can be found on our <> page. diff --git a/docs/getting-started.asciidoc b/docs/getting-started.asciidoc index 12cf28c6b..e63653ecf 100644 --- a/docs/getting-started.asciidoc +++ b/docs/getting-started.asciidoc @@ -1,30 +1,31 @@ [[getting-started]] +== Introduction -ifdef::env-github[] -NOTE: For the best reading experience, -please view this documentation at https://www.elastic.co/guide/en/apm/agent/python/current/getting-started.html[elastic.co] -endif::[] +The Elastic APM Python agent sends performance metrics and error logs to the APM Server. +It has built-in support for Django and Flask performance metrics and error logging, as well as generic support of other WSGI frameworks for error logging. + +[float] +[[how-it-works]] +=== How does the Agent work? -== Getting started +The Python Agent instruments your application to collect APM events in a few different ways: -Welcome to the APM Python agent docs. +To collect data about incoming requests and background tasks, the Agent integrates with <> to make use of hooks and signals provided by the framework. +These framework integrations require limited code changes in your application. -The Elastic APM Python agent sends performance metrics and error logs to the APM Server. -It has built-in support for Django and Flask performance metrics and error logging, as well as generic support of other WSGI frameworks for error logging. +To collect data from database drivers, HTTP libraries etc., +we instrument certain functions and methods in these libraries. +Instrumentations are set up automatically and do not require any code changes. + +In addition to APM and error data, +the Python agent also collects system and application metrics in regular intervals. +This collection happens in a background thread that is started by the agent. + +More detailed information on how the Agent works can be found in the <>. [float] [[additional-components]] -=== Additional Components +=== Additional components APM Agents work in conjunction with the {apm-server-ref-v}/index.html[APM Server], {ref}/index.html[Elasticsearch], and {kibana-ref}/index.html[Kibana]. Please view the {apm-overview-ref-v}/index.html[APM Overview] for details on how these components work together. - -[[framework-support]] -The Elastic APM Python Agent comes with support for the following frameworks: - - * <> 1.8 - 2.1 - * <> 0.10+ - -For other frameworks and custom Python code, the agent exposes a set of <> for integration. - -NOTE: The Elastic APM Python agent does currently not support asynchronous frameworks like Twisted or Tornado. \ No newline at end of file diff --git a/docs/how-the-agent-works.asciidoc b/docs/how-the-agent-works.asciidoc new file mode 100644 index 000000000..b81061543 --- /dev/null +++ b/docs/how-the-agent-works.asciidoc @@ -0,0 +1,64 @@ +[[how-the-agent-works]] +=== How the Agent works + +To gather APM events (called Transactions and Spans), errors and metrics, +the Python agent instruments your application in a few different ways. +These events, are then sent to the APM Server. +The APM Server converts them to a format suitable for Elasticsearch, and sends them to an Elasticsearch cluster. +You can then use the APM app in Kibana to gain insight into latency issues and error culprits within your application. + +Broadly, we differentiate between three different approaches to collect the necessary data: +framework integration, instrumentation, and background collection. + +[float] +[[how-it-works-framework-integration]] +==== Framework integration + +To collect data about incoming requests and background tasks, +we integrate with frameworks like <>, <> and Celery. +Whenever possible, framework integrations make use of hooks and signals provided by the framework. +Examples of this are + + * `request_started`, `request_finished`, and `got_request_exception` signals from `django.core.signals` + * `request_started`, `request_finished`, and `got_request_exception` signals from `flask.signals` + * `task_prerun`, `task_postrun`, and `task_failure` signals from `celery.signals` + +Framework integrations require some limited code changes in your app. +E.g. for Django, you need to add `elasticapm.contrib.django` to `INSTALLED_APPS`. + +[float] +[[how-it-works-instrumentation]] +==== Instrumentation + +To collect data from database drivers, HTTP libraries etc., +we instrument certain functions and methods in these libraries. +Our instrumentation wraps these callables and collects additional data, like + + * time spent in the call + * the executed query for database drivers + * the fetched URL for HTTP libraries + +We use a 3rd party library, https://github.com/GrahamDumpleton/wrapt[`wrapt`], to wrap the callables. +You can read more on how `wrapt` works in Graham Dumpleton's +excellent series of http://blog.dscpl.com.au/search/label/wrapt[blog posts]. + +Instrumentations are set up automatically and do not require any code changes. +See <> to learn more about which libraries we support. + +[float] +[[how-it-works-background-collection]] +==== Background collection + +In addition to APM and error data, +the Python agent also collects system and application metrics in regular intervals. +This collection happens in a background thread that is started by the agent. + +In addition to the metrics collection background thread, +the agent starts two additional threads per process: + + * a thread to regularly fetch remote configuration from the APM Server + * a thread to process the collected data and send it to the APM Server via HTTP. + +Note that every process that instantiates the agent will have these three threads. +This means that when you e.g. use gunicorn or uwsgi workers, +each worker will have three threads started by the Python agent. \ No newline at end of file diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 32eed862d..2aeaed91e 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -1,7 +1,4 @@ -:branch: current -:server-branch: 6.5 -:ecs_version: current - +include::{asciidoc-dir}/../../shared/versions/stack/current.asciidoc[] include::{asciidoc-dir}/../../shared/attributes.asciidoc[] ifdef::env-github[] @@ -11,26 +8,26 @@ endif::[] = APM Python Agent Reference -include::./getting-started.asciidoc[Getting Started] +include::./getting-started.asciidoc[] -include::./configuration.asciidoc[Configuration] +include::./set-up.asciidoc[] -include::./supported-technologies.asciidoc[Supported Technologies] +include::./supported-technologies.asciidoc[] -include::./django.asciidoc[Django support] +include::./configuration.asciidoc[] -include::./flask.asciidoc[Flask support] +include::./advanced-topics.asciidoc[] -include::./metrics.asciidoc[Metrics] +include::./api.asciidoc[] -include::./opentracing.asciidoc[OpenTracing] +include::./metrics.asciidoc[] -include::./advanced-topics.asciidoc[Advanced Topics] +include::./opentracing.asciidoc[] -include::./api.asciidoc[API documentation] +include::./logging.asciidoc[] -include::./upgrading.asciidoc[Upgrading from previous versions] +include::./tuning.asciidoc[] -include::./tuning.asciidoc[Tuning and Overhead considerations] +include::./upgrading.asciidoc[] -include::./release-notes.asciidoc[Release notes] +include::./release-notes.asciidoc[] diff --git a/docs/logging.asciidoc b/docs/logging.asciidoc index fed74518d..a39cf4d96 100644 --- a/docs/logging.asciidoc +++ b/docs/logging.asciidoc @@ -1,9 +1,19 @@ -[[logging-integrations]] -=== Logging integrations +[[log-correlation]] +== Log correlation + +Log correlation allows you to navigate to all logs belonging to a particular trace, and vice-versa -- for a specific log, see in which context it has been logged, and which parameters the user provided. -The Elastic APM agent provides integrations with both the default Python logging library, +The Agent provides integrations with both the default Python logging library, as well as http://www.structlog.org/en/stable/[`structlog`]. +* <> +* <> + +[float] +[[logging-integrations]] +=== Logging integrations + +[float] [[logging]] ==== `logging` @@ -20,16 +30,12 @@ This factory also adds these fields to a dictionary attribute, You can disable this automatic behavior by using the <> setting -in your configuration +in your configuration. For Python versions <3.2, we also provide a https://docs.python.org/3/library/logging.html#filter-objects[filter] which will add the same new attributes to any filtered `LogRecord`: -Note: Because https://docs.python.org/3/library/logging.html#filter-objects[filters -are not propagated to descendent loggers], you should add the filter to each of -your log handlers, as handlers are propagated, along with their attached filters. - [source,python] ---- import logging @@ -41,7 +47,11 @@ console.addFilter(LoggingFilter()) logging.getLogger("").addHandler(console) ---- +NOTE: Because https://docs.python.org/3/library/logging.html#filter-objects[filters +are not propagated to descendent loggers], you should add the filter to each of +your log handlers, as handlers are propagated, along with their attached filters. +[float] [[structlog]] ==== `structlog` @@ -64,6 +74,7 @@ log = logger.new() log.msg("some_event") ---- +[float] ===== Use structlog for agent-internal logging The Elastic APM Python agent uses logging to log internal events and issues. @@ -71,7 +82,8 @@ By default, it will use a `logging` logger. If your project uses structlog, you can tell the agent to use a structlog logger by setting the environment variable `ELASTIC_APM_USE_STRUCTLOG` to `true`. -[[log-correlation]] +[float] +[[log-correlation-in-es]] === Log correlation in Elasticsearch In order to correlate logs from your app with transactions captured by the @@ -134,7 +146,6 @@ formatstring = formatstring + " | elasticapm " \ Then, you could use a grok pattern like this (for the {ref}/grok-processor.html[Elasticsearch Grok Processor]): - [source, json] ---- { diff --git a/docs/metrics.asciidoc b/docs/metrics.asciidoc index bc2eb0fdc..b47023e8a 100644 --- a/docs/metrics.asciidoc +++ b/docs/metrics.asciidoc @@ -4,11 +4,17 @@ With Elastic APM, you can capture system and process metrics. These metrics will be sent regularly to the APM Server and from there to Elasticsearch +[float] [[metric-sets]] -=== Metric Sets +=== Metric sets +* <> +* <> +* <> + +[float] [[cpu-memory-metricset]] -==== CPU/Memory Metric Set +==== CPU/Memory metric set `elasticapm.metrics.sets.cpu.CPUMetricSet` @@ -21,7 +27,7 @@ type: scaled_float format: percent -The percentage of CPU time in states other than Idle and IOWait, normalised by the number of cores. +The percentage of CPU time in states other than Idle and IOWait, normalized by the number of cores. -- @@ -81,8 +87,9 @@ he Resident Set Size. The amount of memory the process occupied in main memory ( NOTE: if you do *not* use Linux, you need to install https://pypi.org/project/psutil/[`psutil`] for this metric set. +[float] [[transactions-metricset]] -==== Transactions Metric Set +==== Transactions metric set This metric set collects metrics in relation with transactions. @@ -104,10 +111,11 @@ You can filter and group by these dimensions: * `transaction.type`: The type of the transaction, for example `request` -- +[float] [[breakdown-metricset]] -==== Breakdown Metric Set +==== Breakdown metric set -NOTE: tracking and collection of this metric set can be disabled using the `<>` setting. +NOTE: Tracking and collection of this metric set can be disabled using the <> setting. *`transaction.breakdown.count`*:: + diff --git a/docs/opentracing.asciidoc b/docs/opentracing.asciidoc index cf4f56982..eb58bb1de 100644 --- a/docs/opentracing.asciidoc +++ b/docs/opentracing.asciidoc @@ -1,15 +1,10 @@ -ifdef::env-github[] -NOTE: For the best reading experience, -please view this documentation at https://www.elastic.co/guide/en/apm/agent/java[elastic.co] -endif::[] - [[opentracing-bridge]] -== Elastic APM OpenTracing bridge +== OpenTracing API -The Elastic APM OpenTracing bridge allows to create Elastic APM `Transactions` and `Spans`, +The Elastic APM OpenTracing bridge allows you to create Elastic APM `Transactions` and `Spans`, using the OpenTracing API. In other words, -it translates the calls to the OpenTracing API to Elastic APM and thus allows for reusing existing instrumentation. +it translates calls to the OpenTracing API to Elastic APM events, which allows for the reuse of existing instrumentation. The first span of a service will be converted to an Elastic APM {apm-overview-ref-v}/transactions.html[`Transaction`], @@ -81,7 +76,7 @@ Not all features of the OpenTracing API are supported. [[opentracing-scope-managers]] ==== Scope Managers Currently, only the `ThreadLocalScopeManager` is supported. -Using other scope managers will lead to unpredictable and possibly app-breaking behaviour. +Using other scope managers will lead to unpredictable and possibly app-breaking behavior. [float] [[opentracing-instrumentation]] @@ -99,7 +94,7 @@ This bridge only supports the formats `Format.Builtin.TEXT_MAP` and `Format.Buil [float] [[opentracing-references]] -==== Span References +==== Span references Currently, this bridge only supports `child_of` references. Other references, like `follows_from` are not supported yet. diff --git a/docs/set-up.asciidoc b/docs/set-up.asciidoc new file mode 100644 index 000000000..c3fc8aa70 --- /dev/null +++ b/docs/set-up.asciidoc @@ -0,0 +1,13 @@ +[[set-up]] +== Set up the Agent + +To get you off the ground, we’ve prepared guides for setting up the Agent with different frameworks: + + * <> + * <> + +For custom instrumentation, see the <>. + +include::./django.asciidoc[] + +include::./flask.asciidoc[] diff --git a/docs/supported-technologies.asciidoc b/docs/supported-technologies.asciidoc index 2ffbd2f20..589db33ab 100644 --- a/docs/supported-technologies.asciidoc +++ b/docs/supported-technologies.asciidoc @@ -1,6 +1,16 @@ [[supported-technologies]] == Supported Technologies +[[framework-support]] +The Elastic APM Python Agent comes with support for the following frameworks: + + * <> + * <> + +For other frameworks and custom Python code, the agent exposes a set of <> for integration. + +NOTE: The Elastic APM Python agent does currently not support asynchronous frameworks like Twisted or Tornado. + [float] [[supported-python]] === Python diff --git a/docs/tuning.asciidoc b/docs/tuning.asciidoc index a26ea48d8..1cb0b3422 100644 --- a/docs/tuning.asciidoc +++ b/docs/tuning.asciidoc @@ -1,5 +1,5 @@ [[tuning-and-overhead]] -== Tuning and Overhead considerations +== Performance tuning Using an APM solution comes with certain trade-offs, and the Python agent for Elastic APM is no different. Instrumenting your code, measuring timings, recording context data, etc., all need resources: @@ -84,7 +84,7 @@ Other contextual information, like the SQL query, will still be available. [[tuning-frame-context]] === Collecting Frame Context -When a stack trace is captured, the agent will also capture several lines of source code around each frame location in the stack trace. This allows the APM UI to give greater insight into where exactly the error or span happens. +When a stack trace is captured, the agent will also capture several lines of source code around each frame location in the stack trace. This allows the APM app to give greater insight into where exactly the error or span happens. There are four settings you can modify to control this behavior: