AttributeError: '_Span' object has no attribute 'add_link'

[mukund-ananthu]

Describe your environment

OS: (e.g, Ubuntu)
Python version: Python 3.7
SDK version: (e.g., 1.25.0)
API version: (e.g., 1.25.0)

What happened?

Python version: Python 3.7

LInk to the failed run: https://github.com/googleapis/python-pubsub/actions/runs/9637364916/job/26577285125?pr=1194
This error happens only for Python 3.7 and succeeds for other versions.

1. tracer.start_as_current_span() is supposed to be returning a Span() object, on which add_link() method should work :

   opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

   Line 1086 in 754fc36

   ) -> Iterator[trace_api.Span]:

Unexpectedly, the function returns a _Span() object when we use:

with tracer.start_as_current_span(name="foo", end_on_exit=False) as create_span1:

where create_span1 is of type _Span instead of Span().

Why is this happening?

self = <google.cloud.pubsub_v1.publisher._batch.thread.Batch object at 0x7f0e97085c90>

    def _commit(self) -> None:
        """Actually publish all of the messages on the active batch.
    
        This moves the batch out from being the active batch to an in progress
        batch on the publisher, and then the batch is discarded upon
        completion.
    
        .. note::
    
            This method blocks. The :meth:`commit` method is the non-blocking
            version, which calls this one.
        """
        with self._state_lock:
            if self._status in _CAN_COMMIT:
                self._status = base.BatchStatus.IN_PROGRESS
            else:
                # If, in the intervening period between when this method was
                # called and now, the batch started to be committed, or
                # completed a commit, then no-op at this point.
                _LOGGER.debug(
                    "Batch is already in progress or has been cancelled, "
                    "exiting commit"
                )
                return
    
        # Once in the IN_PROGRESS state, no other thread can publish additional
        # messages or initiate a commit (those operations become a no-op), thus
        # it is safe to release the state lock here. Releasing the lock avoids
        # blocking other threads in case api.publish() below takes a long time
        # to complete.
        # https://github.com/googleapis/google-cloud-python/issues/[80](https://github.com/googleapis/python-pubsub/actions/runs/9637364916/job/26577285125?pr=1194#step:5:81)36
    
        # Sanity check: If there are no messages, no-op.
        if not self._message_wrappers:
            _LOGGER.debug("No messages to publish, exiting commit")
            self._status = base.BatchStatus.SUCCESS
            return
    
        # Begin the request to publish these messages.
        # Log how long the underlying request takes.
        start = time.time()
    
        batch_transport_succeeded = True
        try:
            if self._client._open_telemetry_enabled:
                tracer = trace.get_tracer("com.google.cloud.pubsub.v1")
                links = []
                for wrapper in self._message_wrappers:
                    span = wrapper.create_span
                    if span.get_span_context().trace_flags.sampled:
                        links.append(trace.Link(span.get_span_context()))
                with tracer.start_as_current_span(
                    name=f"{self._topic} publish",
                    attributes={
                        "messaging.system": "com.google.cloud.pubsub.v1",
                        "messaging.destination.name": self._topic,
                        "gcp.project_id": self._topic.split("/")[1],
                        "messaging.batch.message_count": len(self._message_wrappers),
                        "messaging.operation": "publish",
                        "code.function": "_commit",
                    },
                    links=links if len(links) > 0 else None,
                    kind=trace.SpanKind.CLIENT,
                    end_on_exit=False,
                ) as publish_rpc_span:
                    ctx = publish_rpc_span.get_span_context()
                    for wrapper in self._message_wrappers:
                        if wrapper.create_span.get_span_context().trace_flags.sampled:
>                           wrapper.create_span.add_link(ctx)
E                           AttributeError: '_Span' object has no attribute 'add_link'

google/cloud/pubsub_v1/publisher/_batch/thread.py:301: AttributeError
------------------------------ Captured log setup ------------------------------
WARNING  opentelemetry.trace:__init__.py:521 Overriding of current TracerProvider is not allowed
- generated xml file: /home/runner/work/python-pubsub/python-pubsub/unit_3.7_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/pubsub_v1/publisher/batch/test_thread.py::test_commit_otel_publish_rpc_span - AttributeError: '_Span' object has no attribute 'add_link'
1 failed, 1551 passed, 2 skipped in 24.65s
nox > Command py.test --quiet --junitxml=unit_3.7_sponge_log.xml --cov=google/cloud --cov=tests/unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=0 tests/unit failed with exit code 1
nox > Session unit-3.7 failed.

Steps to Reproduce

Already provided in the description above

Expected Result

Already provided in the description above

Actual Result

Already provided in the description above

Additional context

No response

Would you like to implement a fix?

None

[mukund-ananthu]

I noticed that the Python Open Telemetry docs indicate that it supports only Python 3.8 and above. Is that the reason for this issue: https://opentelemetry.io/docs/languages/python/#version-support

[xrmx]

@mukund-ananthu what version of the sdk are you using?

[mukund-ananthu]

xrmx opentelemetry-sdk=1.25.0

[mukund-ananthu]

@xrmx Could you PTAL the issue.

[emdneto]

@mukund-ananthu add_link is only available from sdk > 1.23.0, are you sure you're using sdk version 1.25.0 in the CI?

[mukund-ananthu]

If you check the setup.py file in my PR: https://github.com/googleapis/python-pubsub/pull/1194/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7 , you will see that the I just specify:

    "opentelemetry-api",
    "opentelemetry-sdk",

in the setup.py file. I assume this takes in the latest version of the library. When I check the pip freeze of my local development environment, I see that I am using 1.25.0

[emdneto]

The thing is, if you run pip install using python3.7 in CI, it will resolve to the last version that supports python3.7, which is 1.22.0. In 1.22.0, we didn't have add_link yet.

[mukund-ananthu]

Okay. What solution would you recommend in this case. To have code level checks for the python version and only enable / use OpenTelemetry if the python version is 3.8 or above

[xrmx]

@mukund-ananthu I think that depends on you. If you want to use features of a recent enough sdk you somewhat need to add a minimum version required and if that's not available on 3.7 I'll just make it a python 3.8+ feature.

[emdneto]

@mukund-ananthu Did you have a chance to take a look at the comment here? can we close this as done?

[mukund-ananthu]

Yes, I'm closing the bug. I see that the Open Telemetry Python docs also specify the min version of Python supported as 3.8. We will disable open telemetry for Python versions 3.7 or below. Thanks.