Merge main (#874)

* Exclude command line functionality from test coverage (#855)

* FIX: resilient environment settings (#825)

if the application uses generalimport to manage optional depedencies,
it's possible that generalimport.MissingOptionalDependency is raised.
In this case, we should not report the module as it is not actually loaded and
is not a runtime dependency of the application.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>

* Replace drop_transaction logic by using transaction context manager (#832)

* Replace drop_transaction call

* [Mega-Linter] Apply linters fixes

* Empty commit to start tests

* Change logic in BG Wrappers

---------

Co-authored-by: lrafeei <lrafeei@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Upgrade to Pypy38 for TypedDict (#861)

* Fix base branch

* Revert tox dependencies

* Replace all pypy37 with pypy38

* Remove action.yml file

* Push Empty Commit

* Fix skip_missing_interpreters behavior

* Fix skip_missing_interpreters behavior

* Pin dev CI image sha

* Remove unsupported Tornado tests

* Add latest tests to Tornado

* Remove pypy38 (for now)

---------

Co-authored-by: Tim Pansino <timpansino@gmail.com>

* Add profile_trace testing (#858)

* Include isort stdlibs for determining stdlib modules

* Use isort & sys to eliminate std & builtin modules

Previously, the logic would fail to identify third party modules installed within the
local user socpe. This fixes that issue by skipping builtin and stdlib modules by name,
instead of attempting to identify third party modules based on file paths.

* Handle importlib_metadata.version being a callable

* Add isort into third party notices

* [Mega-Linter] Apply linters fixes

* Remove Python 2.7 and pypy2 testing (#835)

* Change setup-python to @v2 for py2.7

* Remove py27 and pypy testing

* Fix syntax errors

* Fix comma related syntax errors

* Fix more issues in tox

* Remove gearman test

* Containerized CI Pipeline (#836)

* Revert "Remove Python 2.7 and pypy2 testing (#835)"

This reverts commit abb6405.

* Containerize CI process

* Publish new docker container for CI images

* Rename github actions job

* Copyright tag scripts

* Drop debug line

* Swap to new CI image

* Move pip install to just main python

* Remove libcurl special case from tox

* Install special case packages into main image

* Remove unused packages

* Remove all other triggers besides manual

* Add make run command

* Cleanup small bugs

* Fix CI Image Tagging (#838)

* Correct templated CI image name

* Pin pypy2.7 in image

* Fix up scripting

* Temporarily Restore Old CI Pipeline (#841)

* Restore old pipelines

* Remove python 2 from setup-python

* Rework CI Pipeline (#839)

Change pypy to pypy27 in tox.

Fix checkout logic

Pin tox requires

* Fix Tests on New CI (#843)

* Remove non-root user

* Test new CI image

* Change pypy to pypy27 in tox.

* Fix checkout logic

* Fetch git tags properly

* Pin tox requires

* Adjust default db settings for github actions

* Rename elasticsearch services

* Reset to new pipelines

* [Mega-Linter] Apply linters fixes

* Fix timezone

* Fix docker networking

* Pin dev image to new sha

* Standardize gearman DB settings

* Fix elasticsearch settings bug

* Fix gearman bug

* Add missing odbc headers

* Add more debug messages

* Swap out dev ci image

* Fix required virtualenv version

* Swap out dev ci image

* Swap out dev ci image

* Remove aioredis v1 for EOL

* Add coverage paths for docker container

* Unpin ci container

---------

Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>

* Trigger tests

* Add testing for profile trace.

* [Mega-Linter] Apply linters fixes

* Ignore __call__ from coverage on profile_trace.

* [Mega-Linter] Apply linters fixes

---------

Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: hmstepanek <hmstepanek@users.noreply.github.com>
Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com>

* Add Transaction API Tests (#857)

* Test for suppress_apdex_metric

* Add custom_metrics tests

* Add distributed_trace_headers testing in existing tests

* [Mega-Linter] Apply linters fixes

* Remove redundant if-statement

* Ignore deprecated transaction function from coverage

* [Mega-Linter] Apply linters fixes

* Push empty commit

* Update newrelic/api/transaction.py

---------

Co-authored-by: lrafeei <lrafeei@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>

* Add tests for jinja2. (#842)

* Add tests for jinja2.

* [Mega-Linter] Apply linters fixes

* Update tox.ini

Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>

---------

Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>

* Add tests for newrelic/config.py (#860)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Fix starlette testing matrix for updated behavior. (#869)

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>
Co-authored-by: Uma Annamalai <umaannamalai@users.noreply.github.com>

* Correct Serverless Distributed Tracing Logic (#870)

* Fix serverless logic for distributed tracing

* Test stubs

* Collapse testing changes

* Add negative testing to regular DT test suite

* Apply linter fixes

* [Mega-Linter] Apply linters fixes

---------

Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>

* Fix Kafka CI (#863)

* Reenable kafka testing

* Add kafka dev lib

* Sync install python with devcontainer

* Fix kafka local host setting

* Drop set -u flag

* Pin CI image dev sha

* Add parallel flag to kafka

* Fix proper exit status

* Build librdkafka from source

* Updated dev image sha

* Remove coverage exclusions

* Add new options to better emulate GHA

* Reconfigure kafka networking

Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>

* Fix kafka ports on GHA

* Run kafka tests serially

* Separate kafka consumer groups

* Put CI container makefile back

* Remove confluent kafka Py27 for latest

* Roll back ubuntu version update

* Update dev ci sha

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>

* Change image tag to latest (#871)

* Change image tag to latest

* Use built sha

* Fixup

* Replace w/ latest

* Add full version for pypy3.8 to tox (#872)

* Add full version for pypy3.8

* Remove solrpy from tests

* Fix merge conflict

* Fix tests for scikit-learn >= 1.3.0

In 1.3.0 sklearn renamed fit to _fit in BaseDecisionTree.

* Add gfortran to container

* Use ci image sha

* Add pkg-config

* New CI build

---------

Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
Co-authored-by: Ahmed Helil <aaeabdo@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: lrafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Tim Pansino <timpansino@gmail.com>
Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
Co-authored-by: hmstepanek <hmstepanek@users.noreply.github.com>
Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com>

.github/containers/Dockerfile

@@ -26,6 +26,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
         freetds-common \
         freetds-dev \
         gcc \
+        gfortran \
         git \
         libbz2-dev \
         libcurl4-openssl-dev \
@@ -43,6 +44,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
         make \
         odbc-postgresql \
         openssl \
+        pkg-config \
         python2-dev \
         python3-dev \
         python3-pip \
@@ -55,10 +57,22 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
         zlib1g-dev && \
     rm -rf /var/lib/apt/lists/*
 
+# Build librdkafka from source
+ARG LIBRDKAFKA_VERSION=2.1.1
+RUN cd /tmp && \
+    wget https://github.com/confluentinc/librdkafka/archive/refs/tags/v${LIBRDKAFKA_VERSION}.zip -O ./librdkafka.zip && \
+    unzip ./librdkafka.zip && \
+    rm ./librdkafka.zip && \
+    cd ./librdkafka-${LIBRDKAFKA_VERSION} && \
+    ./configure && \
+    make all install && \
+    cd /tmp && \
+    rm -rf ./librdkafka-${LIBRDKAFKA_VERSION}
+
 # Setup ODBC config
-RUN sed -i 's/Driver=psqlodbca.so/Driver=\/usr\/lib\/x86_64-linux-gnu\/odbc\/psqlodbca.so/g' /etc/odbcinst.ini && \
-    sed -i 's/Driver=psqlodbcw.so/Driver=\/usr\/lib\/x86_64-linux-gnu\/odbc\/psqlodbcw.so/g' /etc/odbcinst.ini && \
-    sed -i 's/Setup=libodbcpsqlS.so/Setup=\/usr\/lib\/x86_64-linux-gnu\/odbc\/libodbcpsqlS.so/g' /etc/odbcinst.ini
+RUN sed -i 's|Driver=psqlodbca.so|Driver=/usr/lib/x86_64-linux-gnu/odbc/psqlodbca.so|g' /etc/odbcinst.ini && \
+    sed -i 's|Driver=psqlodbcw.so|Driver=/usr/lib/x86_64-linux-gnu/odbc/psqlodbcw.so|g' /etc/odbcinst.ini && \
+    sed -i 's|Setup=libodbcpsqlS.so|Setup=/usr/lib/x86_64-linux-gnu/odbc/libodbcpsqlS.so|g' /etc/odbcinst.ini
 
 # Set the locale
 RUN locale-gen --no-purge en_US.UTF-8
@@ -79,7 +93,7 @@ RUN echo 'eval "$(pyenv init -)"' >>$HOME/.bashrc && \
     pyenv update
 
 # Install Python
-ARG PYTHON_VERSIONS="3.10 3.9 3.8 3.7 3.11 2.7 pypy2.7-7.3.11 pypy3.7"
+ARG PYTHON_VERSIONS="3.10 3.9 3.8 3.7 3.11 2.7 pypy2.7-7.3.12 pypy3.8-7.3.11"
 COPY --chown=1000:1000 --chmod=+x ./install-python.sh /tmp/install-python.sh
 COPY ./requirements.txt /requirements.txt
 RUN /tmp/install-python.sh && \

.github/containers/Makefile

@@ -22,7 +22,9 @@ default: test
 .PHONY: build
 build:
 	@# Perform a shortened build for testing
-	@docker build --build-arg='PYTHON_VERSIONS=3.10 2.7' $(MAKEFILE_DIR) -t ghcr.io/newrelic/newrelic-python-agent-ci:local
+	@docker build $(MAKEFILE_DIR) \
+		-t ghcr.io/newrelic/newrelic-python-agent-ci:local \
+		--build-arg='PYTHON_VERSIONS=3.10 2.7'
 
 .PHONY: test
 test: build
@@ -38,7 +40,9 @@ run: build
 	@docker run --rm -it \
 		--mount type=bind,source="$(REPO_ROOT)",target=/home/github/python-agent \
 		--workdir=/home/github/python-agent \
+		--add-host=host.docker.internal:host-gateway \
 		-e NEW_RELIC_HOST="${NEW_RELIC_HOST}" \
 		-e NEW_RELIC_LICENSE_KEY="${NEW_RELIC_LICENSE_KEY}" \
 		-e NEW_RELIC_DEVELOPER_MODE="${NEW_RELIC_DEVELOPER_MODE}" \
+		-e GITHUB_ACTIONS="true" \
 		ghcr.io/newrelic/newrelic-python-agent-ci:local /bin/bash

.github/containers/install-python.sh

@@ -13,10 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-set -e
-
-SCRIPT_DIR=$(dirname "$0")
-PIP_REQUIREMENTS=$(cat /requirements.txt)
+set -eo pipefail
 
 main() {
     # Coerce space separated string to array
@@ -50,7 +47,7 @@ main() {
     pyenv global ${PYENV_VERSIONS[@]}
 
     # Install dependencies for main python installation
-    pyenv exec pip install --upgrade $PIP_REQUIREMENTS
+    pyenv exec pip install --upgrade -r /requirements.txt
 }
 
 main