Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

open telemetry callback error: Transient error StatusCode.UNAVAILABLE #7888

Closed
1 task done
mihai-satmarean opened this issue Jan 24, 2024 · 4 comments · Fixed by #8321
Closed
1 task done

open telemetry callback error: Transient error StatusCode.UNAVAILABLE #7888

mihai-satmarean opened this issue Jan 24, 2024 · 4 comments · Fixed by #8321
Labels
bug This issue/PR relates to a bug callback callback plugin plugins plugin (any type)

Comments

@mihai-satmarean
Copy link

Summary

We have configured Ansible withe the Opentelemetry plugin
and the Grafanacloud confugurations suggested on the https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry_callback.html page, including the token.

Issue Type

Bug Report

Component Name

community.general.opentelemetry

Ansible Version

$ ansible --version
ansible [core 2.13.1]
  config file = None
  configured module search path = ['/REDACTED/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /REDACTED/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.17 (main, Jun 15 2023, 07:46:17) [Clang 14.0.3 (clang-1403.0.22.14.1)]
  jinja version = 3.1.2
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general


# /REDACTED/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 8.2.0

Configuration

$ ansible-config dump --only-changed
CALLBACKS_ENABLED(/REDACTED/ansible.cfg) = ['community.general.opentelemetry']
CONFIG_FILE() = /REDACTED/ansible.cfg

OS / Environment

macosx, docker container with Ubuntu

Steps to Reproduce

Steps to reproduce the problem:

we configured Ansible like here: https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry_callback.html
mainly adding this in the ansible.cfg:
[defaults]
callbacksenabled = community.general.opentelemetry [callbackopentelemetry]
enablefromenvironment = ANSIBLEOPENTELEMETRYENABLED

created and configured the environment with the token and ID
export OTELEXPORTEROTLPPROTOCOL="http/protobuf" export OTELEXPORTEROTLPENDPOINT="https://otlp-gateway-prod-eu-west-2.grafana.net/otlp"
export OTELEXPORTEROTLP_HEADERS="Authorization=Basic REDACTED"
3.
Run playbooks and got this:

Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 2s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 4s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 8s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 16s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 32s.

We also have an open question: https://grafana.slack.com/archives/C038S86TZGF/p1705057683915289
and here: https://forum.ansible.com/t/anyone-managed-to-use-opentelemetry-from-awx-ansible/3005

Expected Results

Running Ansible playbooks should send traces to the grafana cloud

Actual Results

PLAY RECAP *********************************************************************
localhost                  : ok=9    changed=0    unreachable=0    failed=0    skipped=14   rescued=0    ignored=0

Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 2s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 4s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 8s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 16s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 32s.

Code of Conduct

  • I agree to follow the Ansible Code of Conduct
@ansibullbot
Copy link
Collaborator

Files identified in the description:

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

@ansibullbot
Copy link
Collaborator

cc @v1v
click here for bot help

@ansibullbot ansibullbot added bug This issue/PR relates to a bug callback callback plugin plugins plugin (any type) labels Jan 24, 2024
@v1v
Copy link
Contributor

v1v commented Mar 14, 2024

This particular implementation is vendor-agnostic.

Would you mind if I ask you to try the same settings but using an OpenTelemetry Collector1 with some debug traces2? Pretty much update OTEL_EXPORTER_OTLP_ENDPOINT to point to your collector. You can try that locally.

Footnotes

  1. https://opentelemetry.io/docs/collector/ ↩

  2. https://opentelemetry.io/docs/collector/configuration/ ↩

@wilfriedroset
Copy link
Contributor

Would you mind if I ask you to try the same settings but using an OpenTelemetry Collector1 with some debug traces2? Pretty much update OTEL_EXPORTER_OTLP_ENDPOINT to point to your collector

I'm in the same situation with OTEL_EXPORTER_OTLP_ENDPOINT correctly define. I've other applications in the same network who manage to successfully send their traces.

wilfriedroset added a commit to wilfriedroset/community.general that referenced this issue May 6, 2024
@wilfriedroset
[opentelemetry][callback] Add support for http exporter
330971d 
The previous version of the callback was supporting only the grpc
exporter. This was counter intuitive as the documentation was
mentioning `<your endpoint (OTLP/HTTP)>`. Users were left with a error
similar to
`Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in 1s.`

The following commit fix this situation by support both HTTP and GRPC
via the standard environment variables and ansible.cfg

See as well ansible-collections#7888

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>
wilfriedroset added a commit to wilfriedroset/community.general that referenced this issue May 6, 2024
@wilfriedroset
[opentelemetry][callback] Add support for http exporter
b53a1da 
The previous version of the callback was supporting only the grpc
exporter. This was counter intuitive as the documentation was
mentioning `<your endpoint (OTLP/HTTP)>`. Users were left with a error
similar to
`Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in 1s.`

The following commit fix this situation by support both HTTP and GRPC
via the standard environment variables and ansible.cfg

See as well ansible-collections#7888

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>
@wilfriedroset wilfriedroset mentioned this issue May 6, 2024
Merged
wilfriedroset added a commit to wilfriedroset/community.general that referenced this issue May 6, 2024
@wilfriedroset
[opentelemetry][callback] Add support for http exporter
71e8670 
The previous version of the callback was supporting only the grpc
exporter. This was counter intuitive as the documentation was
mentioning `<your endpoint (OTLP/HTTP)>`. Users were left with a error
similar to
`Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in 1s.`

The following commit fix this situation by support both HTTP and GRPC
via the standard environment variables and ansible.cfg

See as well ansible-collections#7888

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>
wilfriedroset added a commit to wilfriedroset/community.general that referenced this issue May 6, 2024
@wilfriedroset
[opentelemetry][callback] Add support for http exporter
9f18813 
The previous version of the callback was supporting only the grpc
exporter. This was counter intuitive as the documentation was
mentioning `<your endpoint (OTLP/HTTP)>`. Users were left with a error
similar to
`Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in 1s.`

The following commit fix this situation by support both HTTP and GRPC
via the standard environment variables and ansible.cfg

See as well ansible-collections#7888

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>
felixfontein pushed a commit that referenced this issue May 15, 2024
@wilfriedroset
[opentelemetry][callback] Add support for http exporter (#8321)
6889e04 
* [opentelemetry][callback] Add support for http exporter

The previous version of the callback was supporting only the grpc
exporter. This was counter intuitive as the documentation was
mentioning `<your endpoint (OTLP/HTTP)>`. Users were left with a error
similar to
`Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in 1s.`

The following commit fix this situation by support both HTTP and GRPC
via the standard environment variables and ansible.cfg

See as well #7888

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>

* [opentelemetry][callback] Take into account review

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>

---------

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>
austinlucaslake pushed a commit to austinlucaslake/community.general that referenced this issue May 25, 2024
@wilfriedroset @austinlucaslake
[opentelemetry][callback] Add support for http exporter (ansible-coll…
d65b930 
…ections#8321)

* [opentelemetry][callback] Add support for http exporter

The previous version of the callback was supporting only the grpc
exporter. This was counter intuitive as the documentation was
mentioning `<your endpoint (OTLP/HTTP)>`. Users were left with a error
similar to
`Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in 1s.`

The following commit fix this situation by support both HTTP and GRPC
via the standard environment variables and ansible.cfg

See as well ansible-collections#7888

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>

* [opentelemetry][callback] Take into account review

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>

---------

Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue/PR relates to a bug callback callback plugin plugins plugin (any type)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[opentelemetry][callback] Add support for http exporter wilfriedroset/community.general
4 participants
@v1v @mihai-satmarean @wilfriedroset @ansibullbot