Support uploading traces to UI in OpenTelemetry format (OTLP JSON) #5155

NavinShrinivas · 2024-01-30T05:24:18Z

Which problem is this PR solving?

Part of Ability to upload OTLP data into UI #4949
In combination with Support uploading traces in OpenTelemetry format (OTLP JSON) jaeger-ui#2145

Description of the changes

Incorporates changes from previous draft PR review
Adds middleware in frontend as well

How was this change tested?

Unit tests and supplying valid and invalid OTLP traces using insomnia to test the API

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: yarn lint and yarn test

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com> partial work for issue jaegertracing#4949

cmd/query/app/apiv3/otlp_translator.go

yurishkuro

there should be no UI submodule changes in this PR.

NavinShrinivas · 2024-01-30T05:39:58Z

there should be no UI submodule changes in this PR.

How do I make pr's for jaeger-ui changes?

yurishkuro · 2024-01-30T05:42:30Z

in jaeger-ui repository

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com>

NavinShrinivas · 2024-01-30T17:06:04Z

hey @yurishkuro, I have implemented the group by traceID and I have excluded jaeger-ui changes and added some tests. I'll make a pr to jaeger-ui only after the backend is confirmed

yurishkuro

and I have excluded jaeger-ui changes

can you submit another PR for UI changes? The functionality is not useful without both of them.

cmd/query/app/apiv3/package_test.go

model/trace.go

cmd/query/app/apiv3/otlp_translator.go

cmd/query/app/http_handler.go

NavinShrinivas · 2024-01-31T02:11:11Z

and I have excluded jaeger-ui changes

can you submit another PR for UI changes? The functionality is not useful without both of them.

I wont be able to link the issue in the jaeger-ui repo, is that alright? Or do you want me to push a seperate pr changing the commit version tracked in this repo for jaeger-ui?

I've made a PR anyways : jaegertracing/jaeger-ui#2145

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com>

cmd/query/app/http_handler.go

cmd/query/app/otlp_translator.go

cmd/query/app/http_handler.go

cmd/query/app/otlp_translator.go

yurishkuro · 2024-01-31T02:48:11Z

I wont be able to link the issue in the jaeger-ui repo, is that alright?

Yes you can, see how I just edited UI PR description - you just include URL to the ticket and GitHub links them

cmd/query/app/fixture/OTLP-transform.json

cmd/query/app/handler_otlp_translator_test.go

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com>

NavinShrinivas · 2024-02-07T06:31:49Z

Im unable to add the required label for ci/cd

codecov · 2024-02-08T02:25:17Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (8ae2e05) 94.38% compared to head (463556c) 94.41%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5155      +/-   ##
==========================================
+ Coverage   94.38%   94.41%   +0.02%     
==========================================
  Files         331      332       +1     
  Lines       19156    19189      +33     
==========================================
+ Hits        18081    18117      +36     
+ Misses        886      884       -2     
+ Partials      189      188       -1

Flag	Coverage Δ
cassandra-3.x	`25.58% <ø> (ø)`
cassandra-4.x	`25.58% <ø> (ø)`
elasticsearch-5.x	`19.86% <ø> (+0.01%)`	⬆️
elasticsearch-6.x	`19.85% <ø> (-0.02%)`	⬇️
elasticsearch-7.x	`19.98% <ø> (ø)`
elasticsearch-8.x	`20.08% <ø> (+0.01%)`	⬆️
grpc-badger	`19.48% <ø> (ø)`
kafka	`14.09% <ø> (ø)`
opensearch-1.x	`20.00% <ø> (ø)`
opensearch-2.x	`19.98% <ø> (-0.02%)`	⬇️
unittests	`92.18% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

NavinShrinivas · 2024-02-08T17:54:35Z

I just saw the missing tests, from what I can see, golang testify lets you mock functions that are methods of a struct, here the model2otlp functions are one-off separate functions. Would it be okay to include them as part of aH (APIHandler), allowing me to mock them like tracesByID function? Or is there some alternative method I'm missing out on?

cmd/query/app/http_handler.go

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com>

cmd/query/app/http_handler_test.go

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com>

yurishkuro

Found one issue on the UI side: jaegertracing/jaeger-ui#2145 (comment)

cmd/query/app/http_handler_test.go

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com>

Signed-off-by: Yuri Shkuro <github@ysh.us>

cmd/query/app/http_handler.go

yurishkuro · 2024-02-14T02:06:34Z

cmd/query/app/fixture/otlp2jaeger-out.json

@@ -0,0 +1,110 @@
+{


reformatted per cat file | jq .

yurishkuro · 2024-02-14T02:07:48Z

cmd/query/app/http_handler_test.go

+		out, err := json.MarshalIndent(obj, "", "  ")
+		require.NoError(t, err)
+		return out
+	}


you were right the first time about employing marshaling to ensure consistent formatting. I changed the expected fixture to be readable - also added a 2nd trace to test a branch in the translator

Signed-off-by: Yuri Shkuro <github@ysh.us>

…31709) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [github.com/jaegertracing/jaeger](https://togithub.com/jaegertracing/jaeger) | `v1.54.0` -> `v1.55.0` | [![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fjaegertracing%2fjaeger/v1.55.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fjaegertracing%2fjaeger/v1.55.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fjaegertracing%2fjaeger/v1.54.0/v1.55.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fjaegertracing%2fjaeger/v1.54.0/v1.55.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>jaegertracing/jaeger (github.com/jaegertracing/jaeger)</summary> ### [`v1.55.0`](https://togithub.com/jaegertracing/jaeger/releases/tag/v1.55.0): Release 1.55.0 [Compare Source](https://togithub.com/jaegertracing/jaeger/compare/v1.54.0...v1.55.0) ##### Backend Changes ##### ✨ New Features: - Support uploading traces to UI in OpenTelemetry format (OTLP/JSON) ([@NavinShrinivas](https://togithub.com/NavinShrinivas) in [#5155](https://togithub.com/jaegertracing/jaeger/pull/5155)) - Add Elasticsearch storage support for adaptive sampling ([@Pushkarm029](https://togithub.com/Pushkarm029) in [#5158](https://togithub.com/jaegertracing/jaeger/pull/5158)) ##### 🐞 Bug fixes, Minor Improvements: - Add the `print-config` subcommand ([@gmafrac](https://togithub.com/gmafrac) in [#5200](https://togithub.com/jaegertracing/jaeger/pull/5200)) - Return more detailed errors from ES storage ([@yurishkuro](https://togithub.com/yurishkuro) in [#5209](https://togithub.com/jaegertracing/jaeger/pull/5209)) - Bump go version ([@yurishkuro](https://togithub.com/yurishkuro) in [#5180](https://togithub.com/jaegertracing/jaeger/pull/5180)) ##### 🚧 Experimental Features: - \[jaeger-v2] Add support for gRPC storarge ([@james-ryans](https://togithub.com/james-ryans) in [#5228](https://togithub.com/jaegertracing/jaeger/pull/5228)) - \[jaeger-v2] Add support for Elasticsearch ([@akagami-harsh](https://togithub.com/akagami-harsh) in [#5152](https://togithub.com/jaegertracing/jaeger/pull/5152)) ##### 📊 UI Changes - UI pinned to version [1.39.0](https://togithub.com/jaegertracing/jaeger-ui/blob/main/CHANGELOG.md#v1390-2024-03-04). ##### 👏 New Contributors - [@h4shk4t](https://togithub.com/h4shk4t) made their first contribution in [jaegertracing/jaeger#5162 - [@NavinShrinivas](https://togithub.com/NavinShrinivas) made their first contribution in [jaegertracing/jaeger#5155 - [@prakrit55](https://togithub.com/prakrit55) made their first contribution in [jaegertracing/jaeger#5197 - [@gmafrac](https://togithub.com/gmafrac) made their first contribution in [jaegertracing/jaeger#5200 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).  --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>

…pen-telemetry#31709) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [github.com/jaegertracing/jaeger](https://togithub.com/jaegertracing/jaeger) | `v1.54.0` -> `v1.55.0` | [![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fjaegertracing%2fjaeger/v1.55.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fjaegertracing%2fjaeger/v1.55.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fjaegertracing%2fjaeger/v1.54.0/v1.55.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fjaegertracing%2fjaeger/v1.54.0/v1.55.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>jaegertracing/jaeger (github.com/jaegertracing/jaeger)</summary> ### [`v1.55.0`](https://togithub.com/jaegertracing/jaeger/releases/tag/v1.55.0): Release 1.55.0 [Compare Source](https://togithub.com/jaegertracing/jaeger/compare/v1.54.0...v1.55.0) ##### Backend Changes ##### ✨ New Features: - Support uploading traces to UI in OpenTelemetry format (OTLP/JSON) ([@&open-telemetry#8203;NavinShrinivas](https://togithub.com/NavinShrinivas) in [#&open-telemetry#8203;5155](https://togithub.com/jaegertracing/jaeger/pull/5155)) - Add Elasticsearch storage support for adaptive sampling ([@&open-telemetry#8203;Pushkarm029](https://togithub.com/Pushkarm029) in [#&open-telemetry#8203;5158](https://togithub.com/jaegertracing/jaeger/pull/5158)) ##### 🐞 Bug fixes, Minor Improvements: - Add the `print-config` subcommand ([@&open-telemetry#8203;gmafrac](https://togithub.com/gmafrac) in [#&open-telemetry#8203;5200](https://togithub.com/jaegertracing/jaeger/pull/5200)) - Return more detailed errors from ES storage ([@&open-telemetry#8203;yurishkuro](https://togithub.com/yurishkuro) in [#&open-telemetry#8203;5209](https://togithub.com/jaegertracing/jaeger/pull/5209)) - Bump go version ([@&open-telemetry#8203;yurishkuro](https://togithub.com/yurishkuro) in [#&open-telemetry#8203;5180](https://togithub.com/jaegertracing/jaeger/pull/5180)) ##### 🚧 Experimental Features: - \[jaeger-v2] Add support for gRPC storarge ([@&open-telemetry#8203;james-ryans](https://togithub.com/james-ryans) in [#&open-telemetry#8203;5228](https://togithub.com/jaegertracing/jaeger/pull/5228)) - \[jaeger-v2] Add support for Elasticsearch ([@&open-telemetry#8203;akagami-harsh](https://togithub.com/akagami-harsh) in [#&open-telemetry#8203;5152](https://togithub.com/jaegertracing/jaeger/pull/5152)) ##### 📊 UI Changes - UI pinned to version [1.39.0](https://togithub.com/jaegertracing/jaeger-ui/blob/main/CHANGELOG.md#v1390-2024-03-04). ##### 👏 New Contributors - [@&open-telemetry#8203;h4shk4t](https://togithub.com/h4shk4t) made their first contribution in [jaegertracing/jaeger#5162 - [@&open-telemetry#8203;NavinShrinivas](https://togithub.com/NavinShrinivas) made their first contribution in [jaegertracing/jaeger#5155 - [@&open-telemetry#8203;prakrit55](https://togithub.com/prakrit55) made their first contribution in [jaegertracing/jaeger#5197 - [@&open-telemetry#8203;gmafrac](https://togithub.com/gmafrac) made their first contribution in [jaegertracing/jaeger#5200 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).  --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>

Add endpoints to allow uploading OTLP traces

817db96

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com> partial work for issue jaegertracing#4949

NavinShrinivas changed the title ~~Uploading jaeger traces~~ Uploading OTLP traces Jan 30, 2024

yurishkuro reviewed Jan 30, 2024

View reviewed changes

cmd/query/app/apiv3/otlp_translator.go Outdated Show resolved Hide resolved

yurishkuro reviewed Jan 30, 2024

View reviewed changes

NavinShrinivas force-pushed the uploading_jaeger_traces branch from 02c039f to 0dc3464 Compare January 30, 2024 16:50

Refactoring and fixes for endpoint in backend

18cd089

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com>

NavinShrinivas force-pushed the uploading_jaeger_traces branch from 0dc3464 to 18cd089 Compare January 30, 2024 16:56

yurishkuro reviewed Jan 30, 2024

View reviewed changes

Placing functions in correct files

d1f2e8c

Signed-off-by: Navin Shrinivas <karupal2002@gmail.com>

yurishkuro reviewed Jan 31, 2024

View reviewed changes

cmd/query/app/http_handler.go Show resolved Hide resolved

yurishkuro reviewed Jan 31, 2024

View reviewed changes

cmd/query/app/otlp_translator.go Outdated Show resolved Hide resolved

yurishkuro reviewed Jan 31, 2024

View reviewed changes

cmd/query/app/http_handler.go Outdated Show resolved Hide resolved

yurishkuro reviewed Jan 31, 2024

View reviewed changes

cmd/query/app/otlp_translator.go Outdated Show resolved Hide resolved