Empty traces

[Filipcsupka]

Describe the bug

I set up autoinstrumentation-go but every trace in Tempo i see is empty Get Post or /opentelemetry.proto.collector.trace.v1.TraceServi

Environment

collectorImage:
repository: "otel/opentelemetry-collector-k8s"
tag: 0.122.1
autoInstrumentationImage:
go:
repository: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go
tag: v0.21.0
google.golang.org/grpc on version 1.64 or 1.69

To Reproduce

Steps to reproduce the behavior:

1. i followed these steps https://opentelemetry.io/docs/platforms/kubernetes/operator/automatic/
2. deployed operator in helm chart with version opentelemetry-operator:0.122.0
3. deployed collector otel/opentelemetry-collector-k8s:0.122.1
4. instrumentation as per steps provided on official web i shared.
5. apps are instrumented with annotations
6. sidecar added by operator with envs
7. tried with multiple version of google.golang.org/grpc on version 1.64 or 1.69
8. collector pointing to tempo to see traces
9. as i said every trace is empty or internal /opentelemetry.proto.collector.trace.v1.TraceServi

Expected behavior

Proper traces + database traces as on exmaple https://github.com/dash0hq/go-instrumentation-ebpf-demo?tab=readme-ov-file

Additional context

I also tried with deploying directly the sidecar within app deployment and with needed envs, without operator and collector pointing it directly to tempo. Still same result with traces.

Cannot find anything on this topic, is it bug ? versions are mismatched ? Kind of blackhole

[RonFed]

Can you please provide a minimal reproducer? i.e a snippet of the gROC setup code and message sending.

[Filipcsupka]

Can you please provide a minimal reproducer? i.e a snippet of the gROC setup code and message sending.

I am not sure what you want/need.
We use
go 1.24.2

require (
git.prosoftke.sk/nghis/modules/go-libraries/logging v1.0.0
git.prosoftke.sk/nghis/modules/go-libraries/opentelemetry v1.0.2
git.prosoftke.sk/nghis/openapi/clients/go/nghisclinicalclient/v2 v2.51.2
git.prosoftke.sk/nghis/openapi/clients/go/nghisorganizationclient/v3 v3.3.6
git.prosoftke.sk/nghis/openapi/clients/go/nghisorganizationgoclient v0.8.0
git.prosoftke.sk/nghis/openapi/clients/go/nghispersonserviceclient v0.45.1
github.com/go-chi/chi/v5 v5.0.12
github.com/go-chi/cors v1.2.1
github.com/go-playground/validator/v10 v10.20.0
github.com/google/uuid v1.6.0
github.com/ilyakaznacheev/cleanenv v1.5.0
github.com/jackc/pgx/v5 v5.5.5
github.com/stretchr/testify v1.9.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.48.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.23.1
go.uber.org/dig v1.17.1
gopkg.in/h2non/gock.v1 v1.1.2
)

require (
github.com/BurntSushi/toml v1.2.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/joho/godotenv v1.5.1 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect
github.com/mdobak/go-xerrors v0.3.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.18.0 // indirect
github.com/prometheus/client_model v0.5.0 // indirect
github.com/prometheus/common v0.45.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
github.com/rogpeppe/go-internal v1.11.0 // indirect
go.opentelemetry.io/otel v1.31.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.23.1 // indirect
go.opentelemetry.io/otel/exporters/prometheus v0.45.2 // indirect
go.opentelemetry.io/otel/metric v1.31.0 // indirect
go.opentelemetry.io/otel/sdk v1.31.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.31.0 // indirect
go.opentelemetry.io/otel/trace v1.31.0 // indirect
go.opentelemetry.io/proto/otlp v1.1.0 // indirect
golang.org/x/crypto v0.28.0 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/text v0.19.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20241015192408-796eee8c2d53 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53 // indirect
google.golang.org/grpc v1.69.0 // indirect
google.golang.org/protobuf v1.35.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 // indirect
)

k8s kernel "5.15.0-1052-azure"
Otel operator/collector/auto-go images are mentioned above (i even build new image from latest commits, still empty traces...)

Logs show:
2025-04-25T09:59:33.135375015Z DEBUG .../probe.go:206
package constraint not meet, skipping uprobe
probe={"SpanKind":2,"InstrumentedPkg":"google.golang.org/grpc"}
symbol="google.golang.org/grpc/internal/transport.(*http2Server).WriteStatus"
package="google.golang.org/grpc"
constraint=">1.40.0 <1.69.0"
version="1.69.0"

2025-04-25T09:59:33.084439375Z DEBUG .../probe.go:206
package constraint not meet, skipping uprobe
probe={"SpanKind":2,"InstrumentedPkg":"google.golang.org/grpc"}
symbol="google.golang.org/grpc.(*Server).handleStream"
package="google.golang.org/grpc"
constraint="<1.69.0"
version="1.69.0"

But grpc version is missleading since before we had 1.64.0 and the error was pointing me to updating the version. Since version went to 1.69 it shows it should be diffrent...

[MrAlias]

What are the full debug logs of the auto instrumentation container?

Do you send the spans to a collector? Are the spans getting there?

[Filipcsupka]

What are the full debug logs of the auto instrumentation container?

Do you send the spans to a collector? Are the spans getting there?

---

Lot of logs but i dont see any problem with them, only the WARN

{"time":"2025-05-05T14:00:15.754743742Z","level":"WARN","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/internal/global"},"symbol":"go.opentelemetry.io/otel/internal/global.(*tracer).Start","package":"std","constraint":"<1.24.0+0","version":"1.24.2"}

Which shouldnt be a problem at all.

{"time":"2025-05-05T14:00:15.755712281Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).ConfigLoop","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":244},"msg":"Configuration provider closed, configuration updates will no longer be received"}
{"time":"2025-05-05T14:00:15.755590804Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":397},"msg":"loaded probes to memory","total_probes":5}
{"time":"2025-05-05T14:00:15.754814975Z","level":"WARN","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/internal/global"},"symbol":"go.opentelemetry.io/otel/internal/global.(*nonRecordingSpan).SetName","package":"std","constraint":"<1.24.0+0","version":"1.24.2"}
{"time":"2025-05-05T14:00:15.75479628Z","level":"WARN","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/internal/global"},"symbol":"go.opentelemetry.io/otel/internal/global.(*nonRecordingSpan).SetStatus","package":"std","constraint":"<1.24.0+0","version":"1.24.2"}
{"time":"2025-05-05T14:00:15.754775511Z","level":"WARN","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/internal/global"},"symbol":"go.opentelemetry.io/otel/internal/global.(*nonRecordingSpan).SetAttributes","package":"std","constraint":"<1.24.0+0","version":"1.24.2"}
{"time":"2025-05-05T14:00:15.754761405Z","level":"WARN","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/internal/global"},"symbol":"go.opentelemetry.io/otel/internal/global.(*nonRecordingSpan).End","package":"std","constraint":"<1.24.0+0","version":"1.24.2"}
{"time":"2025-05-05T14:00:15.754743742Z","level":"WARN","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/internal/global"},"symbol":"go.opentelemetry.io/otel/internal/global.(*tracer).Start","package":"std","constraint":"<1.24.0+0","version":"1.24.2"}
{"time":"2025-05-05T14:00:15.754694149Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/internal/global"},"symbol":"go.opentelemetry.io/otel/internal/global.(*tracer).newSpan","package":"go.opentelemetry.io/otel","constraint":">=1.33.0-0","version":"1.31.0"}
{"time":"2025-05-05T14:00:15.280510979Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"tracer_provider_tracers_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/internal/global","Struct":"tracerProvider","Field":"tracers"},"offset":24}
{"time":"2025-05-05T14:00:15.280490581Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"tracer_provider_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/internal/global","Struct":"tracer","Field":"provider"},"offset":56}
{"time":"2025-05-05T14:00:15.280472638Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"tracer_name_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/internal/global","Struct":"tracer","Field":"name"},"offset":16}
{"time":"2025-05-05T14:00:15.280436881Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"tracer_delegate_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/internal/global","Struct":"tracer","Field":"delegate"},"offset":64}
{"time":"2025-05-05T14:00:15.278424695Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":382},"msg":"loading probe","name":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/internal/global"}}
{"time":"2025-05-05T14:00:14.922474054Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"url_host_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Host"},"offset":40}
{"time":"2025-05-05T14:00:14.922464567Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"username_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"Userinfo","Field":"username"},"offset":0}
{"time":"2025-05-05T14:00:14.922459097Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"raw_fragment_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"RawFragment"},"offset":128}
{"time":"2025-05-05T14:00:14.922450781Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"fragment_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Fragment"},"offset":112}
{"time":"2025-05-05T14:00:14.922445511Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"raw_query_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"RawQuery"},"offset":96}
{"time":"2025-05-05T14:00:14.922438598Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"force_query_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"ForceQuery"},"offset":89}
{"time":"2025-05-05T14:00:14.922424882Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"omit_host_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"OmitHost"},"offset":88}
{"time":"2025-05-05T14:00:14.9224131Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"raw_path_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"RawPath"},"offset":72}
{"time":"2025-05-05T14:00:14.922397221Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"user_ptr_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"User"},"offset":32}
{"time":"2025-05-05T14:00:14.92238623Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"opaque_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Opaque"},"offset":16}
{"time":"2025-05-05T14:00:14.922380459Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"scheme_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Scheme"},"offset":0}
{"time":"2025-05-05T14:00:14.922371342Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"io_writer_n_pos","id":{"ModPath":"std","PkgPath":"bufio","Struct":"Writer","Field":"n"},"offset":40}
{"time":"2025-05-05T14:00:14.922365281Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"io_writer_buf_ptr_pos","id":{"ModPath":"std","PkgPath":"bufio","Struct":"Writer","Field":"buf"},"offset":16}
{"time":"2025-05-05T14:00:14.922356284Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"request_proto_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"Proto"},"offset":24}
{"time":"2025-05-05T14:00:14.9223483Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"request_host_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"Host"},"offset":128}
{"time":"2025-05-05T14:00:14.9223326Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"status_code_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Response","Field":"StatusCode"},"offset":16}
{"time":"2025-05-05T14:00:14.922320167Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"ctx_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"ctx"},"offset":248}
{"time":"2025-05-05T14:00:14.922314316Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"headers_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"Header"},"offset":56}
{"time":"2025-05-05T14:00:14.922308064Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"path_ptr_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Path"},"offset":56}
{"time":"2025-05-05T14:00:14.922299378Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"url_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"URL"},"offset":16}
{"time":"2025-05-05T14:00:14.922274451Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"method_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"Method"},"offset":0}
{"time":"2025-05-05T14:00:14.921487872Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":382},"msg":"loading probe","name":{"SpanKind":3,"InstrumentedPkg":"net/http"}}
{"time":"2025-05-05T14:00:14.647227118Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"headerFrame_streamid_pos","id":{"ModPath":"google.golang.org/grpc","PkgPath":"google.golang.org/grpc/internal/transport","Struct":"headerFrame","Field":"streamID"},"offset":0}
{"time":"2025-05-05T14:00:14.647214304Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"headerFrame_hf_pos","id":{"ModPath":"google.golang.org/grpc","PkgPath":"google.golang.org/grpc/internal/transport","Struct":"headerFrame","Field":"hf"},"offset":8}
{"time":"2025-05-05T14:00:14.647198424Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"httpclient_nextid_pos","id":{"ModPath":"google.golang.org/grpc","PkgPath":"google.golang.org/grpc/internal/transport","Struct":"http2Client","Field":"nextID"},"offset":440}
{"time":"2025-05-05T14:00:14.647166555Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"clientconn_target_ptr_pos","id":{"ModPath":"google.golang.org/grpc","PkgPath":"google.golang.org/grpc","Struct":"ClientConn","Field":"target"},"offset":24}
{"time":"2025-05-05T14:00:14.646273226Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":382},"msg":"loading probe","name":{"SpanKind":3,"InstrumentedPkg":"google.golang.org/grpc"}}
{"time":"2025-05-05T14:00:14.387121346Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"proto_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"Proto"},"offset":24}
{"time":"2025-05-05T14:00:14.387112359Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"host_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"Host"},"offset":128}
{"time":"2025-05-05T14:00:14.387101699Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"remote_addr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"RemoteAddr"},"offset":176}
{"time":"2025-05-05T14:00:14.387079437Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"status_code_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"response","Field":"status"},"offset":120}
{"time":"2025-05-05T14:00:14.387073155Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"req_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"response","Field":"req"},"offset":8}
{"time":"2025-05-05T14:00:14.387066062Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"headers_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"Header"},"offset":56}
{"time":"2025-05-05T14:00:14.387051264Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"path_ptr_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Path"},"offset":56}
{"time":"2025-05-05T14:00:14.387040785Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"ctx_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"ctx"},"offset":248}
{"time":"2025-05-05T14:00:14.387032459Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"url_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"URL"},"offset":16}
{"time":"2025-05-05T14:00:14.387009006Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"method_ptr_pos","id":{"ModPath":"std","PkgPath":"net/http","Struct":"Request","Field":"Method"},"offset":0}
{"time":"2025-05-05T14:00:14.385945249Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":382},"msg":"loading probe","name":{"SpanKind":2,"InstrumentedPkg":"net/http"}}
{"time":"2025-05-05T14:00:14.385165021Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":2,"InstrumentedPkg":"google.golang.org/grpc"},"symbol":"google.golang.org/grpc/internal/transport.(*http2Server).WriteStatus","package":"google.golang.org/grpc","constraint":">1.40.0 <1.69.0","version":"1.69.0"}
{"time":"2025-05-05T14:00:14.358007808Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).loadUprobes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":206},"msg":"package constraint not meet, skipping uprobe","probe":{"SpanKind":2,"InstrumentedPkg":"google.golang.org/grpc"},"symbol":"google.golang.org/grpc.(*Server).handleStream","package":"google.golang.org/grpc","constraint":"<1.69.0","version":"1.69.0"}
{"time":"2025-05-05T14:00:14.120252888Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"TCPAddr_Port_offset","id":{"ModPath":"std","PkgPath":"net","Struct":"TCPAddr","Field":"Port"},"offset":24}
{"time":"2025-05-05T14:00:14.120243791Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"TCPAddr_IP_offset","id":{"ModPath":"std","PkgPath":"net","Struct":"TCPAddr","Field":"IP"},"offset":0}
{"time":"2025-05-05T14:00:14.120222391Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"frame_stream_id_pod","id":{"ModPath":"golang.org/x/net","PkgPath":"golang.org/x/net/http2","Struct":"FrameHeader","Field":"StreamID"},"offset":8}
{"time":"2025-05-05T14:00:14.120202384Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"frame_fields_pos","id":{"ModPath":"golang.org/x/net","PkgPath":"golang.org/x/net/http2","Struct":"MetaHeadersFrame","Field":"Fields"},"offset":8}
{"time":"2025-05-05T14:00:14.120181425Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"stream_ctx_pos","id":{"ModPath":"google.golang.org/grpc","PkgPath":"google.golang.org/grpc/internal/transport","Struct":"Stream","Field":"ctx"},"offset":8}
{"time":"2025-05-05T14:00:14.120169723Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"stream_id_pos","id":{"ModPath":"google.golang.org/grpc","PkgPath":"google.golang.org/grpc/internal/transport","Struct":"Stream","Field":"id"},"offset":0}
{"time":"2025-05-05T14:00:14.120137142Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/probe/probe.go","line":537},"msg":"Offset found","key":"stream_method_ptr_pos","id":{"ModPath":"google.golang.org/grpc","PkgPath":"google.golang.org/grpc/internal/transport","Struct":"Stream","Field":"method"},"offset":24}
{"time":"2025-05-05T14:00:14.118160421Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":382},"msg":"loading probe","name":{"SpanKind":2,"InstrumentedPkg":"google.golang.org/grpc"}}
{"time":"2025-05-05T14:00:14.116610827Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":374},"msg":"Mounting bpffs"}
{"time":"2025-05-05T14:00:14.116526028Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).filterUnusedProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":155},"msg":"no functions found for probe, removing","name":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/auto"}}
{"time":"2025-05-05T14:00:14.116515228Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).filterUnusedProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":155},"msg":"no functions found for probe, removing","name":{"SpanKind":5,"InstrumentedPkg":"github.com/segmentio/kafka-go"}}
{"time":"2025-05-05T14:00:14.116500471Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).filterUnusedProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":155},"msg":"no functions found for probe, removing","name":{"SpanKind":4,"InstrumentedPkg":"github.com/segmentio/kafka-go"}}
{"time":"2025-05-05T14:00:14.116466117Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).filterUnusedProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":155},"msg":"no functions found for probe, removing","name":{"SpanKind":3,"InstrumentedPkg":"go.opentelemetry.io/otel/trace"}}
{"time":"2025-05-05T14:00:14.116430961Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).filterUnusedProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":155},"msg":"no functions found for probe, removing","name":{"SpanKind":3,"InstrumentedPkg":"database/sql"}}

[Filipcsupka]

What are the full debug logs of the auto instrumentation container?

Do you send the spans to a collector? Are the spans getting there?

And yes i send spans to collector and they arrive as i said, they are just empty. But sending them to collector is not even required, i also tried with direct sidecar to app with collector backend or tempo backend and exactly same behavior. I see just GET in tempo and not a single information about the span. Seems like something is missing. Normal tracing via go libs works like a charm but there are no database querys

[Filipcsupka]

Any response ?

[MrAlias]

@Filipcsupka can you clarify what you are looking for? What is an empty trace?

Does this just mean it has not children and you are expecting there to be database spans as children to the HTTP span you show in the description? If so this following log line indicates there is no database/sql use in the target binary and therefore no database probe is loaded:

{"time":"2025-05-05T14:00:14.116430961Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).filterUnusedProbes","file":"/usr/src/go.opentelemetry.io/auto/internal/pkg/instrumentation/manager.go","line":155},"msg":"no functions found for probe, removing","name":{"SpanKind":3,"InstrumentedPkg":"database/sql"}}

Does empty mean that the spans you are seeing do not have the expected attributes or some other data?

[Filipcsupka]

@MrAlias empty traces means that traces are empty.. I see them in Tempo but they have nothing inside, no info, no spans,traces,informations just nothing. When i open trace the trace is GET / (without any information what so ever what where how long) or the traces are errored as some otel stuff. But i sent the screenshot of how traces looks like in the description... And about database traces its kind of right that our code should use database/sql to trace properly. BUt a year ago colleague got it working with same config as now without changing the code.
In comparison:

• Grafana Beyla epbf tracing solution, without any configuraiton provides full traces from start and no errors (but we dont want it since its so resoruce heavy we would need separate node just for tracing)
• otel go libs directly integrated in app and collected by collector and propagated via Tempo works GREAT but we are trying to go away from that since we have like 100 microservices and maintaining this is useless when there is this solution which should work as a no code.