Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix opentracing.propagation.InvalidCarrierException: carrier not a collection #1720

Merged
merged 5 commits into from Sep 17, 2021

Conversation

wei840222
Copy link

@wei840222 wei840222 commented Jun 29, 2021

issue: #1591
When open the jaeger tracing like guide: https://docs.bentoml.org/en/stable/guides/tracing.html.
It will raise an exception on runtime and all api request become s to 500.
log:

opentracing.propagation.InvalidCarrierException: carrier not a collection
[2021-06-29 11:48:20,120] ERROR - Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/bentoml/marshal/marshal.py", line 109, in request_dispatcher
    resp = await func(request)
  File "/opt/conda/lib/python3.6/site-packages/bentoml/marshal/marshal.py", line 329, in request_dispatcher
    resp = await self.relay_handler(request)
  File "/opt/conda/lib/python3.6/site-packages/bentoml/marshal/marshal.py", line 342, in relay_handler
    request_headers=request.headers,
  File "/opt/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/opt/conda/lib/python3.6/site-packages/bentoml/tracing/jaeger.py", line 112, in async_span
    with self.span(*args, async_transport=True, **kwargs) as ctx:
  File "/opt/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/opt/conda/lib/python3.6/site-packages/bentoml/tracing/jaeger.py", line 104, in span
    scope.span.context, Format.HTTP_HEADERS, request_headers,
  File "/opt/conda/lib/python3.6/site-packages/jaeger_client/tracer.py", line 266, in inject
    codec.inject(span_context=span_context, carrier=carrier)
  File "/opt/conda/lib/python3.6/site-packages/jaeger_client/codecs.py", line 59, in inject
    raise InvalidCarrierException('carrier not a collection')
opentracing.propagation.InvalidCarrierException: carrier not a collection

and found inject function in https://github.com/jaegertracing/jaeger-client-python/blob/master/jaeger_client/codecs.py#L57 only accept dict type, and seems request header not need to pass to the span.
so, I follow the opentracing example on github https://github.com/opentracing/opentracing-python#outbound-request only pass a empty dict to it.

@wei840222 wei840222 changed the title fix opentracing.propagation.InvalidCarrierException: carrier not a collection Fix opentracing.propagation.InvalidCarrierException: carrier not a collection Jun 29, 2021
@codecov
Copy link

codecov bot commented Jun 29, 2021

Codecov Report

Merging #1720 (49494da) into master (1648a29) will decrease coverage by 0.15%.
The diff coverage is 0.00%.

❗ Current head 49494da differs from pull request most recent head 828ea2e. Consider uploading reports for the commit 828ea2e to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1720      +/-   ##
==========================================
- Coverage   60.98%   60.82%   -0.16%     
==========================================
  Files         167      167              
  Lines       12104    11958     -146     
==========================================
- Hits         7382     7274     -108     
+ Misses       4722     4684      -38     
Impacted Files Coverage Δ
bentoml/tracing/jaeger.py 0.00% <0.00%> (ø)
bentoml/server/instruments.py 79.54% <0.00%> (-17.33%) ⬇️
bentoml/tracing/noop.py 80.00% <0.00%> (-5.72%) ⬇️
bentoml/frameworks/tensorflow.py 81.13% <0.00%> (-0.95%) ⬇️
bentoml/service/inference_api.py 84.27% <0.00%> (-0.30%) ⬇️
.../yatai/grpc_interceptor/prom_server_interceptor.py 82.55% <0.00%> (-0.21%) ⬇️
bentoml/yatai/yatai_service.py 23.96% <0.00%> (-0.20%) ⬇️
bentoml/yatai/deployment/docker_utils.py 94.73% <0.00%> (-0.14%) ⬇️
bentoml/saved_bundle/loader.py 60.60% <0.00%> (-0.14%) ⬇️
bentoml/yatai/yatai_service_impl.py 45.37% <0.00%> (-0.03%) ⬇️
... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1648a29...828ea2e. Read the comment docs.

@wei840222
Copy link
Author

I think maybe dict(request_headers) is beter?

@parano parano requested a review from bojiang June 29, 2021 17:03
@wei840222
Copy link
Author

The failure on CI does not seem to be related to this modification?

@parano
Copy link
Member

parano commented Jun 30, 2021

The failure on CI does not seem to be related to this modification?

That's right, I think that particular test is flaky on github CI environment, let me trigger a re-run

bentoml/tracing/jaeger.py Outdated Show resolved Hide resolved
bentoml/tracing/jaeger.py Outdated Show resolved Hide resolved
@wei840222 wei840222 requested review from parano and bojiang July 12, 2021 03:31
@pep8speaks
Copy link

pep8speaks commented Jul 13, 2021

Hello @wei840222, Thanks for updating this PR.

There are currently no PEP 8 issues detected in this PR. Cheers! 🍻

Comment last updated at 2021-07-13 12:10:58 UTC

@parano parano merged commit 5d74d93 into bentoml:master Sep 17, 2021
parano pushed a commit that referenced this pull request Dec 2, 2021
…llection (#1720)

* fix opentracing.propagation.InvalidCarrierException: carrier not a collection

* Update jaeger.py

* update injected headers back to origin header

* Update jaeger.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants