Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(grpc): fix segfault with grpc.aio streaming responses #9233

Merged
merged 21 commits into from
May 15, 2024

Conversation

wconti27
Copy link
Contributor

@wconti27 wconti27 commented May 10, 2024

This PR fixes a few issues with the grpc aio integration. Most notably, the integration was causing segfaults when wrapping async stream responses, most likely since these spans were never being finished. This issue was uncovered when customers upgraded their google-api-core dependencies to 2.17.0; with this upgrade, the package changed many grpc calls to use async streaming. In addition to fixing the segfault, this PR also fixes the Pin object to be correctly placed on the grpcio module.

Fixes #9139

Checklist

  • Change(s) are motivated and described in the PR description
  • Testing strategy is described if automated tests are not included in the PR
  • Risks are described (performance impact, potential for breakage, maintainability)
  • Change is maintainable (easy to change, telemetry, documentation)
  • Library release note guidelines are followed or label changelog/no-changelog is set
  • Documentation is included (in-code, generated user docs, public corp docs)
  • Backport labels are set (if applicable)
  • If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Description motivates each change
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Change is maintainable (easy to change, telemetry, documentation)
  • Release note makes sense to a user of the library
  • Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

@wconti27 wconti27 requested review from a team as code owners May 10, 2024 18:53
@datadog-dd-trace-py-rkomorn
Copy link

datadog-dd-trace-py-rkomorn bot commented May 10, 2024

Datadog Report

Branch report: conti/fix-grpc-pin
Commit report: 556d80a
Test service: dd-trace-py

✅ 0 Failed, 447 Passed, 439 Skipped, 4m 11.68s Total duration (1m 11.41s time saved)

@wconti27 wconti27 changed the title fix grpc segfault fix(grpc): fix segfault May 10, 2024
@wconti27 wconti27 requested a review from majorgreys May 10, 2024 19:18
@wconti27 wconti27 marked this pull request as draft May 10, 2024 19:23
@pr-commenter
Copy link

pr-commenter bot commented May 10, 2024

Benchmarks

Benchmark execution time: 2024-05-15 18:03:47

Comparing candidate commit 556d80a in PR branch conti/fix-grpc-pin with baseline commit 1cb3487 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 209 metrics, 9 unstable metrics.

@wconti27 wconti27 marked this pull request as ready for review May 14, 2024 14:23
@wconti27 wconti27 requested a review from majorgreys May 14, 2024 21:51
ddtrace/contrib/grpc/aio_client_interceptor.py Outdated Show resolved Hide resolved
ddtrace/contrib/grpc/aio_client_interceptor.py Outdated Show resolved Hide resolved
ddtrace/contrib/grpc/aio_client_interceptor.py Outdated Show resolved Hide resolved
ddtrace/contrib/grpc/aio_client_interceptor.py Outdated Show resolved Hide resolved
ddtrace/contrib/grpc/aio_client_interceptor.py Outdated Show resolved Hide resolved
@wconti27 wconti27 requested a review from a team as a code owner May 15, 2024 14:44
@wconti27 wconti27 requested a review from majorgreys May 15, 2024 14:58
@majorgreys majorgreys changed the title fix(grpc): fix segfault fix(grpc): fix segfault with grpc.aio streaming responses May 15, 2024
@majorgreys
Copy link
Contributor

We should expand on the problem in the PR description so we have context later on about the solution.

@wconti27 wconti27 merged commit 5897cab into main May 15, 2024
72 checks passed
@wconti27 wconti27 deleted the conti/fix-grpc-pin branch May 15, 2024 18:17
Copy link
Contributor

The backport to 1.20 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.20 1.20
# Navigate to the new working tree
cd .worktrees/backport-1.20
# Create a new branch
git switch --create backport-9233-to-1.20
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5897cabe7651f30b0f865ed211c84e1316b49628
# Push it to GitHub
git push --set-upstream origin backport-9233-to-1.20
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.20

Then, create a pull request where the base branch is 1.20 and the compare/head branch is backport-9233-to-1.20.

github-actions bot pushed a commit that referenced this pull request May 15, 2024
This PR fixes a few issues with the `grpc aio` integration. Most
notably, the integration was causing segfaults when wrapping async
stream responses, most likely since these spans were never being
finished. This issue was uncovered when customers upgraded their
`google-api-core` dependencies to `2.17.0`; with this upgrade, the
package changed many grpc calls to use async streaming. In addition to
fixing the segfault, this PR also fixes the Pin object to be correctly
placed on the grpcio module.

Fixes #9139

## Checklist

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

(cherry picked from commit 5897cab)
github-actions bot pushed a commit that referenced this pull request May 15, 2024
This PR fixes a few issues with the `grpc aio` integration. Most
notably, the integration was causing segfaults when wrapping async
stream responses, most likely since these spans were never being
finished. This issue was uncovered when customers upgraded their
`google-api-core` dependencies to `2.17.0`; with this upgrade, the
package changed many grpc calls to use async streaming. In addition to
fixing the segfault, this PR also fixes the Pin object to be correctly
placed on the grpcio module.

Fixes #9139

## Checklist

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

(cherry picked from commit 5897cab)
wconti27 added a commit that referenced this pull request May 15, 2024
This PR fixes a few issues with the `grpc aio` integration. Most
notably, the integration was causing segfaults when wrapping async
stream responses, most likely since these spans were never being
finished. This issue was uncovered when customers upgraded their
`google-api-core` dependencies to `2.17.0`; with this upgrade, the
package changed many grpc calls to use async streaming. In addition to
fixing the segfault, this PR also fixes the Pin object to be correctly
placed on the grpcio module.

Fixes #9139

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

(cherry picked from commit 5897cab)
wconti27 added a commit that referenced this pull request May 15, 2024
….8] (#9274)

Backport 5897cab from #9233 to 2.8.

This PR fixes a few issues with the `grpc aio` integration. Most
notably, the integration was causing segfaults when wrapping async
stream responses, most likely since these spans were never being
finished. This issue was uncovered when customers upgraded their
`google-api-core` dependencies to `2.17.0`; with this upgrade, the
package changed many grpc calls to use async streaming. In addition to
fixing the segfault, this PR also fixes the Pin object to be correctly
placed on the grpcio module.

Fixes #9139

## Checklist

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

---------

Co-authored-by: William Conti <58711692+wconti27@users.noreply.github.com>
Co-authored-by: William Conti <william.conti@datadoghq.com>
wconti27 added a commit that referenced this pull request May 15, 2024
….9] (#9275)

Backport 5897cab from #9233 to 2.9.

This PR fixes a few issues with the `grpc aio` integration. Most
notably, the integration was causing segfaults when wrapping async
stream responses, most likely since these spans were never being
finished. This issue was uncovered when customers upgraded their
`google-api-core` dependencies to `2.17.0`; with this upgrade, the
package changed many grpc calls to use async streaming. In addition to
fixing the segfault, this PR also fixes the Pin object to be correctly
placed on the grpcio module.

Fixes #9139

## Checklist

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

---------

Co-authored-by: William Conti <58711692+wconti27@users.noreply.github.com>
Co-authored-by: William Conti <william.conti@datadoghq.com>
emmettbutler added a commit that referenced this pull request May 17, 2024
….20] (#9276)

Backport
5897cab
from #9233 to 1.20.

This PR fixes a few issues with the grpc aio integration. Most notably,
the integration was causing segfaults when wrapping async stream
responses, most likely since these spans were never being finished. This
issue was uncovered when customers upgraded their google-api-core
dependencies to 2.17.0; with this upgrade, the package changed many grpc
calls to use async streaming. In addition to fixing the segfault, this
PR also fixes the Pin object to be correctly placed on the grpcio
module.

Fixes #9139

## Checklist

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

---------

Co-authored-by: Emmett Butler <723615+emmettbutler@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

grpc integration causes exception with grpc.aio streaming
2 participants