Last span is not dispalyed by tracing-telemetry #58
Comments
|
I agree this seems to be an issue. For example for every root span in the Jaeger UI it displays the warning "invalid parent span IDs=c877373268c0dfe2; skipping clock skew adjustment". |
|
I think this may be related to open-telemetry/opentelemetry-rust#888 |
|
Can this be reproduced using the |
Maybe by modifying the reproducer repo there: https://github.com/shivjm/rust-opentelemetry-jaeger-test |
|
I was not able to reproduce it without tracing-opentelemetry yet. |
|
I think opentelemetry#888 has something to do with tracing integration within HTTP clients. If we use an HTTP client without tracing integration the spans seem to be fine. I plan to take another look over the weekend. |
|
@TommyCpp did you get any more insights on that? |
|
Hi there @izderadicka would you mind checking with this code? use opentelemetry::sdk::trace::TracerProvider;
use opentelemetry::trace::TracerProvider as _;
use tracing::{info, instrument, Level, span, Subscriber};
use tracing_subscriber::{layer::Layer, layer::SubscriberExt, Registry};
struct MyLayer;
impl <S:Subscriber> Layer<S> for MyLayer {
fn on_enter(&self, id: &span::Id, _ctx: tracing_subscriber::layer::Context<'_, S>) {
println!(">>> Entering span {:?}", id)
}
fn on_exit(&self, id: &span::Id, _ctx: tracing_subscriber::layer::Context<'_, S>) {
println!("<<< Leaving span {:?}", id)
}
}
mod old {
use log::info;
pub fn legacy(s: &str) {
info!("I'm legacy logging {msg}", msg = s);
}
}
#[instrument(name = "fn.answer")]
fn answer(n: i32) {
info!("universal answer is {}", n);
}
struct OtelGuard;
impl Drop for OtelGuard {
fn drop(&mut self) {
opentelemetry::global::shutdown_tracer_provider();
}
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let _guard = OtelGuard;
tracing_log::LogTracer::init()?;
let provider = TracerProvider::builder()
.with_simple_exporter(opentelemetry_stdout::SpanExporter::default())
.build();
let tracer = provider.tracer("readme_example");
let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);
let log = tracing_subscriber::fmt::layer();
let subscriber = Registry::default().with(telemetry)
.with(log).with(MyLayer);
tracing::subscriber::set_global_default(subscriber)?;
let span = span!(Level::INFO, "root", "root span");
span.in_scope(|| {
let span = span!(Level::INFO, "inner", "inner span");
span.in_scope(|| {
println!("Hello, world!");
info!(fun = "main", "First event");
answer(42);
old::legacy("abcd");
});
});
Ok(())
}Cargo.toml [package]
name = "delete"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
log = "0.4.20"
opentelemetry = "0.20.0"
opentelemetry-stdout = { version = "0.1.0", features = ["trace"] }
tracing = { version = "0.1.37", features = ["log"] }
tracing-log = "0.1.3"
tracing-opentelemetry = "0.21.0"
tracing-subscriber = "0.3.17"the main difference is here: because potentially the global shutdown could run before the root span is dropped, which i assume it would be "stuck" somehow, the drop guard ensures that it will be executed after the other variables inside main's body get dropped |
|
@jaysonsantos Just using new versions of crate solved the issue , neither guard, nor final call to So works almost as expected, however there is still small issue - an out put from my custom layer {"resourceSpans":[{"resource":{"<<< Leaving span Id(2) If I remember correctly |
|
@jtescher For me this helped - an encoder that first encodes whole structure to buffer: fn bulk_encoder(out: &mut dyn Write, data: SpanData) -> TraceResult<()> {
let mut out_buffer: Vec<u8> = Vec::new();
serde_json::to_writer(&mut out_buffer, &data)
.map_err(|err| TraceError::Other(Box::new(err)))?;
out.write_all(&out_buffer)
.map_err(|err| TraceError::Other(Box::new(err)))
}and use it like this: let provider = TracerProvider::builder()
.with_simple_exporter(
opentelemetry_stdout::SpanExporterBuilder::default()
.with_encoder(bulk_encoder)
.build(),
)
.build();From my part issue can be closed - or I can provide PR based on the above code. |
|
Ok let's close for now, if you want to open an issue about it on http://github.com/open-telemetry/opentelemetry-rust/ we can look at the current exporter behavior. |
Bug Report
Version
rust: rustc 1.64.0 (a55dd71d5 2022-09-19)
Platform
Crates
tracing-opentelemetry ?
Description
Top - outer most span is not exported/printed by tracing-opentelemetry
As you can there are only 2 SpanData top most "root" is missing.
In cases when I had only one span, nothing was usually printed, but sometimes it did work, I though it may be because main thread finished before last span was output, so I tried to add code to end of main to keep main thread live for a while, but it does not seem the case.
The text was updated successfully, but these errors were encountered: