Opentelemetry stops sending traces with 9.0.0

[moserke]

Summary

When going from 8.6.2 to 9.0.0 the opentelemetry callback stops sending traces to the endpoint. Same exact configuration and traces get forwarded in 8.6.2 but go nowhere in 9.0.0. I suspect it's due to how the exporter is getting picked but can't seem to figure out how to make it work.

otel_exporter = None
        if store_spans_in_file:
            otel_exporter = InMemorySpanExporter()
            processor = SimpleSpanProcessor(otel_exporter)
        else:
            if otel_exporter_otlp_traces_protocol == 'grpc':
                otel_exporter = GRPCOTLPSpanExporter()
            else:
                otel_exporter = HTTPOTLPSpanExporter()
            processor = BatchSpanProcessor(otel_exporter)

Issue Type

Bug Report

Component Name

opentelemetry callback

Ansible Version

$ ansible --version
ansible [core 2.17.1]
  config file = /ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.12/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.12.3 (main, Apr 17 2024, 00:00:00) [GCC 14.0.1 20240411 (Red Hat 14.0.1-0)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general
Collection        Version
----------------- -------
community.general 9.1.0  

Configuration

$ ansible-config dump --only-changed

OS / Environment

No response

Steps to Reproduce

ansibile config:
[defaults]
callbacks_enabled = community.general.opentelemetry

Run playbook
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 ansible-playbook playbook.yml

Expected Results

Expect traces to be sent to endpoint

Actual Results

Traces are never forwarded

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/callback/opentelemetry.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @v1v
click here for bot help

[v1v]

#8321 is the PR that introduced the support for the http exporter.

As far as I see, the change uses the same exporter by default.

Can you try to run the plugin with the explicit configuration entries?

ansible.cfg:

    [defaults]
    callbacks_enabled = community.general.opentelemetry
    [callback_opentelemetry]
    otel_exporter_otlp_traces_protocol = grpc
    store_spans_in_file = None

IIUC, you tried locally running OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317/ ansible-playbook playbook.yml against your OTEL collector, right?

[moserke]

Tried setting all of the possible config options to their defaults in the ansible.cfg and still the same issue, it just simply isn't trying to send the traces. If I do a store_spans_in_file=/dev/stdout instead just to see, it prints them to the screen, so I know it's tracing, it's just for some reason not sending to the otlp endpoint...

[rojon8]

Seeing the same issue here. Works nicely in 8.6, but silently stops sending traces in >=9.0.0.

[v1v]

I can see a few changes were added to v9.0:

• https://github.com/ansible-collections/community.general/blob/stable-9/CHANGELOG.md#v9-0-0

IIUC, from the description, the issue might be related to supporting HTTP exporters and the existing GRPC support.

@wilfriedroset @russoz, since you worked and helped on #8321, would you mind if I asked you to double-check if things work nicely on your end if you use >=9.0.0? 🙇

[cervajs]

tested with 9.2.0. problem persist

8.6.3 works ok

ansible [core 2.14.14]
config file = /home/cervenka/.ansible.cfg
configured module search path = ['/home/cervenka/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.9/site-packages/ansible
ansible collection location = /home/cervenka/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.9.18 (main, Jan 4 2024, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True

[russoz]

I can see a few changes were added to v9.0:

• stable-9/CHANGELOG.md#v9-0-0

IIUC, from the description, the issue might be related to supporting HTTP exporters and the existing GRPC support.

@wilfriedroset @russoz, since you worked and helped on #8321, would you mind if I asked you to double-check if things work nicely on your end if you use >=9.0.0? 🙇

Hi @v1v I pretty much helped review it from a Python/Ansible perspective, I am not familiar enough with OpenTelemetry to make a call on the plugin logic.

@wilfriedroset Would it be possible for you to double check the code change? TIA

[russoz]

I have just reviewed the changes in that PR, and to the best of my ability I could not find anything that would be a problem. There are 4 other PRs after #8321 that might have introduced a problem (I have no x-ref' d them with the version tag, so probably not all of them apply).