Skip to content

Commit

Permalink
thehesiod aiohttp branch work (#1368)
Browse files Browse the repository at this point in the history
* chore: update six to 1.14.0 (#1313)

* chore(ci): test only with grpcio wheels (#1310)

* chore(ci): test only with grpcio wheels

* fix

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

* fix(profiling/pprof): provide timestamp to export() (#1312)

Currently, the Exporters have to guess what the time range is for the provided
events.
This is not optimal in cases where we don't have any events — even if it
_should_ not happen in practice.

By forcing the caller of the exporter to provide the collection time range, we
can generate data that are actually more precise.

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

* ci: import our Dockerfile in the repository (#1227)

That should make it easier to manage our CI image.

Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>
Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

* fmt: black celery tests (#1314)

* celery: black fmt tests

* celery: use self.assert_is_measured

Co-authored-by: Julien Danjou <julien@danjou.info>

* Add uds_path to ddtrace.opentracer.Tracer (#1275)

Fixes #1274

* logging: only use global setting for env, service and version (#1317)

* logging: update documentation (#1318)

* chore(docs): consolidate environment variable configuration (#1320)

* docs: consolidate environment variable configuration

* Update docs/configuration.rst

Co-Authored-By: Kyle Verhoog <kyle@verhoog.ca>

* remove env vars from usage

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

* logging: fix typo (#1323)

* fix(tests/profiling): use a string with setenv, not a int (#1321)

* fix(profiling/http): converts nanoseconds timestamp to seconds (#1325)

* fix(profiling): ignore failure on shutdown (#1327)

On interpreter shutdown, we can see this kind of errors:

Traceback (most recent call last):
  File "ddtrace/profiling/_periodic.py", line 104, in run
    del threading._active[self._tident]
TypeError: 'NoneType' object does not support item deletion

This changes the code to use the same try/except mechanism in the above block
to avoid logging those errors on shutdown.

* aiobotocore: pin to <1.0 (#1330)

* aiobotocore: pin to <1.0

* aiobotocore: don't test 0.12 with Python 3.5

* Update error event handler name within SQLAlchemy engine (#1324)

* Updating event handler name

Update method name for handling db errors in sqlalchemy

* Listen for dbapi events

* Accommodating difference in positional args among sqlalchemy versions

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

* fix: tag runtime metrics for only internal services (#1311)

* fix: tag runtime metrics only for internal services

* only add internal services

* handle case where span type not set as internal

* remove deprecated functions

* check lang

* revert to deprecated regexp asserts

* docs(opentracing): add celery example (#1329)

* Document how to propagate the tracer across celery tasks

* docs(opentracing): add celery distributed tracing example

Co-authored-by: Lie Ryan <lie.1296@gmail.com>
Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com>

* fix(profiling): allow to override options after load (#1332)

Using default=os.getenv in attr attributes will make the default unchangeable
after the library is loaded.

Most options already leverage ddtrace.profiling._attr.from_env to avoid
this, but not the HTTP exporter.

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

* ci: allow to use latest pytest version (#1326)

* feat(profiling): add the pip install when recompilation is needed (#1334)

If an incompatible ABI is detected, users are usually lost and don't know what
to do.
Indicate the command to use to install and compile this module.

* tracer: add support for cross process tracing (#1331)

* feat(profiling): add support for DD_VERSION (#1340)

* feat(profiling): add support for DD_SERVICE (#1339)

* fix: avoid context deadlock with logs injection enabled (#1338)

* don't try to get current span on patched internal logger

* run black

* add span directly to log record

* add test

* run black

* contextually group test a bit better

* don't add issue number

Co-Authored-By: Tahir H. Butt <tahir@tahirbutt.com>

* rename logger_arg to logger_override 1/2

Co-Authored-By: Tahir H. Butt <tahir@tahirbutt.com>

* rename logger_arg to logger_override 2/2

Co-Authored-By: Tahir H. Butt <tahir@tahirbutt.com>

* fix arg name

Co-authored-by: Tahir H. Butt <tahir@tahirbutt.com>

* fix(profiling): use formats.asbool to convert bool from env (#1342)

Using the `bool` primitive will only work with 0 or 1, whereas we want to
support true/false strings.

* feat(profiling): add support for DD_ENV environment variable (#1343)

* span: handle non-string tag keys (#1345)

* span: handle non-string keys

* span: use six.string_types for type checking

* fix(profiling): pthread_t is defined as unsigned long, not int (#1347)

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

* Allow Profiler to finish upload data in the background when stopped (#1322)

* feat(profiling/periodic): allow to call a shutdown function

* feat(profiling): do not reset event list of no exporter configured

* feat(profiling): allow to stop/join collectors in 2 steps

This should make the Profiler slightly faster to stop.

* refactor: introduce Service class

This introduces a new class Service that describes a Service that can be
started, stopped and joined.

The Service tracks its own status in order to avoid starting twice the same
service.

The collectors are now inheriting from this common class as they all provide a
startable/stoppable service.

* refactor(scheduler): leverage PeriodicService class

This changes the Scheduler class to leverage the PeriodicService class.

It also makes sure the Scheduler always flush on exit.

The Profiler is now responsible for joining or not the Scheduler if it wants to
block on the last upload before returning from its `stop` method.

* feat(profiling): add support for DD_API_KEY (#1350)

* feat(profiling): add support for DD_SITE (#1351)

* tracer: stop previous writer if a new one is created (#1356)

* Updates configuration docs (#1360)

This updates the configuration docs, calling out that the DD_SERVICE, DD_ENV, and DD_VERSION env vars were added in v0.36.0

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

* feat(profiling): allow to pass service_name to HTTP exporter (#1358)

* Add config to omit `django.user.name` tag from request root span (#1361)

* Add `user_name_enabled` config to Django contrib

* Elaborate on user_name_enabled comment

* Change test to only check for user name tag instead of all meta tags

* Check that "django.user.name" key does not exist in meta

* Change `user_name_enabled` to `include_user_name`

* feat(profiling): replace http_client by urllib for uploading (#1359)

This is slightly higher level than http_client and supports features such as
the `HTTP_PROXY` environment variable.

* test(profiling): check for thread presence rather than number of threads (#1357)

This test would fail occasionally if a thread from another test is still
running, depending on the order the tests are run.

This replaces check with a more accurate one.

* Fix task context management for asyncio in Python <3.7 (#1352)

* fix for context propagation when first span is closed

* fix tests and merge

* fix for context propagation when first span is closed

* fix tests and merge

* test without context changes

* asyncio: use context.clone()

Co-authored-by: Alexander Mohr <thehesiod@hotmail.com>
Co-authored-by: Alexander Mohr <amohr@farmersbusinessnetwork.com>

* fix: make C extensions mandatory (#1333)

If no C compiler is available, the installation do not fail and only a warning
is being print. This is wrong as the profiler won't work — at all.

Since we need a C compiler to build the profiler extensions that are part of
the wheel anyway,  let's not make wrapt or psutil optional.

Fixes #1328

* writer: allow configuration of queue maxsize via env var (#1364)

* writer: allow configuration of queue maxsize via env var

* rename to DD_TRACE_MAX_TPS

* cleanup

Co-authored-by: Julien Danjou <julien@danjou.info>
Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com>
Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>
Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>
Co-authored-by: Sarah Harvey <worldwise001@users.noreply.github.com>
Co-authored-by: Robert Townley <Me@RobertTownley.com>
Co-authored-by: Lie Ryan <lie.1296@gmail.com>
Co-authored-by: Zach Hammer <zhammer@seatgeek.com>
Co-authored-by: Tahir H. Butt <tahir@tahirbutt.com>
Co-authored-by: Scott Burns <scott.s.burns@gmail.com>
Co-authored-by: Sebastian Coetzee <mail@sebastiancoetzee.com>
  • Loading branch information
12 people committed Apr 17, 2020
1 parent 5da14d7 commit 31df275
Show file tree
Hide file tree
Showing 69 changed files with 1,523 additions and 916 deletions.
22 changes: 18 additions & 4 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ s3_bucket: &s3_bucket pypi.datadoghq.com

resource_class: &resource_class medium
busybox_image: &busybox_image busybox:latest
cimg_base_image: &cimg_base_image cimg/base:2020.01
python38_image: &python38_image circleci/python:3.8
python37_image: &python37_image circleci/python:3.7
python36_image: &python36_image circleci/python:3.6
Expand Down Expand Up @@ -152,19 +153,21 @@ commands:
# Ensure we didn't cache from previous runs
- run: rm -rf build/ dist/
# Manually build any extensions to ensure they succeed
# DEV: `DDTRACE_BUILD_RAISE=TRUE` will ensure we don't swallow any build errors
- run: DDTRACE_BUILD_RAISE=TRUE python setup.py build_ext --force
- run: python setup.py build_ext --force
# Ensure source package will build
- run: python setup.py sdist
# Ensure wheel will build
# DEV: `DDTRACE_BUILD_RAISE=TRUE` will ensure we don't swallow any build errors
- run: DDTRACE_BUILD_RAISE=TRUE python setup.py bdist_wheel
- run: python setup.py bdist_wheel
# Ensure package long description is valid and will render
# https://github.com/pypa/twine/tree/6c4d5ecf2596c72b89b969ccc37b82c160645df8#twine-check
- run: twine check dist/*


executors:
cimg_base:
docker:
- image: *cimg_base_image
resource_class: *resource_class
python38:
docker:
- image: *python38_image
Expand Down Expand Up @@ -246,6 +249,15 @@ jobs:
- run: tox -e 'flake8' --result-json /tmp/flake8.results
- save_results

build-docker-ci-image:
executor: cimg_base
steps:
- checkout
- setup_remote_docker:
docker_layer_caching: true
- run: |
docker build .
test_build_py38:
executor: python38
steps:
Expand Down Expand Up @@ -871,6 +883,7 @@ workflows:
- tracer: *requires_pre_test
- unit_tests: *requires_pre_test
- vertica: *requires_pre_test
- build-docker-ci-image: *requires_pre_test
- verify_all_tests_ran:
requires:
# Individual tests do not need this to start running
Expand Down Expand Up @@ -936,6 +949,7 @@ workflows:
- tornado
- unit_tests
- vertica
- build-docker-ci-image
- deploy_master:
requires:
- verify_all_tests_ran
Expand Down
70 changes: 70 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# DEV: Use `debian:slim` instead of an `alpine` image to support installing wheels from PyPI
# this drastically improves test execution time since python dependencies don't all
# have to be built from source all the time (grpcio takes forever to install)
FROM debian:stretch-slim

# http://bugs.python.org/issue19846
# > At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.
ENV LANG C.UTF-8

RUN \
# Install system dependencies
apt-get update \
&& apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
curl \
git \
jq \
libbz2-dev \
libffi-dev \
liblzma-dev \
libmariadb-dev \
libmariadb-dev-compat \
libmemcached-dev \
libmemcached-dev \
libncurses5-dev \
libncursesw5-dev \
libpq-dev \
libreadline-dev \
libsasl2-dev \
libsqlite3-dev \
# Needed to support Python 3.4
libssl1.0-dev \
# Can be re-enabled once we drop 3.4
# libssh-dev \
# libssl-dev \
patch \
python-openssl\
wget \
zlib1g-dev \
# Cleaning up apt cache space
&& rm -rf /var/lib/apt/lists/*


# Configure PATH environment for pyenv
ENV PYENV_ROOT /root/.pyenv
ENV PATH ${PYENV_ROOT}/shims:${PYENV_ROOT}/bin:$PATH

# Install pyenv
RUN git clone git://github.com/yyuu/pyenv.git "${PYENV_ROOT}"


# Install all required python versions
RUN \
pyenv install 2.7.17 \
&& pyenv install 3.4.9 \
&& pyenv install 3.5.9 \
&& pyenv install 3.6.9 \
&& pyenv install 3.7.6 \
&& pyenv install 3.8.1 \
&& pyenv install 3.9-dev \
&& pyenv global 2.7.17 3.4.9 3.5.9 3.6.9 3.7.6 3.8.1 3.9-dev \
&& pip install --upgrade pip

# Install Python dependencies
# DEV: `tox==3.7` introduced parallel execution mode
# https://tox.readthedocs.io/en/3.7.0/example/basic.html#parallel-mode
RUN pip install "tox>=3.7,<4.0"

CMD ["/bin/bash"]
14 changes: 2 additions & 12 deletions ddtrace/commands/ddtrace_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,10 @@

USAGE = """
Execute the given Python program after configuring it to emit Datadog traces.
Append command line arguments to your program as usual.
Usage: [ENV_VARS] ddtrace-run <my_program>
Available environment variables:
Append command line arguments to your program as usual.
DATADOG_ENV : override an application's environment (no default)
DATADOG_TRACE_ENABLED=true|false : override the value of tracer.enabled (default: true)
DATADOG_TRACE_DEBUG=true|false : enabled debug logging (default: false)
DATADOG_PATCH_MODULES=module:patch,module:patch... e.g. boto:true,redis:false : override the modules patched for this execution of the program (default: none)
DATADOG_TRACE_AGENT_HOSTNAME=localhost: override the address of the trace agent host that the default tracer will attempt to submit to (default: localhost)
DATADOG_TRACE_AGENT_PORT=8126: override the port that the default tracer will submit to (default: 8126)
DD_SERVICE: the service name to be used for this program (default only for select web frameworks)
DATADOG_PRIORITY_SAMPLING=true|false: (default: false): enables Priority Sampling.
Usage: ddtrace-run <my_program>
""" # noqa: E501


Expand Down
2 changes: 2 additions & 0 deletions ddtrace/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,5 @@

MANUAL_DROP_KEY = 'manual.drop'
MANUAL_KEEP_KEY = 'manual.keep'

LOG_SPAN_KEY = '__datadog_log_span'
17 changes: 5 additions & 12 deletions ddtrace/context.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import logging
import threading

from .constants import HOSTNAME_KEY, SAMPLING_PRIORITY_KEY, ORIGIN_KEY
from .constants import HOSTNAME_KEY, SAMPLING_PRIORITY_KEY, ORIGIN_KEY, LOG_SPAN_KEY
from .internal.logger import get_logger
from .internal import hostname
from .settings import config
Expand Down Expand Up @@ -76,11 +76,9 @@ def clone(self):
"""
with self._lock:
new_ctx = Context(
trace_id=self._parent_trace_id,
span_id=self._parent_span_id,
sampling_priority=self._sampling_priority,
)
new_ctx._current_span = self._current_span
new_ctx._set_current_span(self._current_span)
return new_ctx

def get_current_root_span(self):
Expand Down Expand Up @@ -139,12 +137,13 @@ def close_span(self, span):
# some children. On the other hand, asynchronous web frameworks still expect
# to close the root span after all the children.
if span.tracer and span.tracer.log.isEnabledFor(logging.DEBUG) and span._parent is None:
extra = {LOG_SPAN_KEY: span}
unfinished_spans = [x for x in self._trace if not x.finished]
if unfinished_spans:
log.debug('Root span "%s" closed, but the trace has %d unfinished spans:',
span.name, len(unfinished_spans))
span.name, len(unfinished_spans), extra=extra)
for wrong_span in unfinished_spans:
log.debug('\n%s', wrong_span.pprint())
log.debug('\n%s', wrong_span.pprint(), extra=extra)

def _is_sampled(self):
return any(span.sampled for span in self._trace)
Expand Down Expand Up @@ -182,12 +181,6 @@ def get(self):
self._trace = []
self._finished_spans = 0

# Don't clear out the parent trace IDs as this may be a cloned
# context in a new thread/task without a outer span
if not self._parent_trace_id or (trace and trace[0].trace_id == self._parent_trace_id):
self._parent_trace_id = None
self._parent_span_id = None
self._sampling_priority = None
return trace, sampled

elif self._partial_flush_enabled:
Expand Down
6 changes: 4 additions & 2 deletions ddtrace/contrib/aiohttp/middlewares.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import asyncio
import functools
import pkg_resources

from ..asyncio import context_provider
from ...compat import stringify
Expand All @@ -8,12 +9,13 @@
from ...propagation.http import HTTPPropagator
from ...settings import config

import aiohttp

try:
if pkg_resources.parse_version(aiohttp.__version__) < pkg_resources.parse_version('3'):
from aiohttp.web import middleware

AIOHTTP_2x = True
except ImportError:
else:
AIOHTTP_2x = False

def middleware(f):
Expand Down

0 comments on commit 31df275

Please sign in to comment.