feature: adding support for AWS X-RAY #1987

xoscar · 2023-02-13T21:13:49Z

This PR enables Amazon X-Ray as a data store for Tracetest.

Changes

Adds a Node.js example to use Tracetest with X-Ray
Includes BE and FE support for the new Data Store

Fixes

Support Amazon x-ray #179

Checklist

tested locally
added new dependencies
updated the docs
added a test

Using AWS XRAY SDK to instrument plus the ADOT COllector

https://www.loom.com/share/5441578aef2442c78684916834335428

Using AWX XRAY SDK to instrument plus AWS XRAY SDK to pull Traces

https://www.loom.com/share/4ca15c89ba48455b813d6c605c828063

Using OpenTelemetry (Pokeshop) to instrument plus XRAY ADOT collector exporter plus AWS XRAY SDK to pull Traces

https://www.loom.com/share/3a4f3ad4955a4390ae475c18cad30f99

examples/tracetest-amazon-x-ray/tracetest-config.yaml

server/tracedb/tracedb.go

* added quick-start golnag app in example * Added quick-start-python in examples

* chore(examples): adding documentation and example of datadog integration * adding datadog recipe * fixing details in docs * Apply suggestions from code review Co-authored-by: Julianne Fermi <julianne@kubeshop.io> --------- Co-authored-by: Julianne Fermi <julianne@kubeshop.io>

* docs(1919): flatten tools and integration * docs(1919): flatten examples and tutorials * docs(recipes): edit naming * docs(config): fix 1982 * examples(tempo): edit config * docs(structure): add redirect * docs(config): update dd

* feature(examples): adding net core x tracetest example * adding readme * addressing feedback comments * adding more information

* refactor connectivity test to reuse components * only run next step if previous step succeeded * simplify tester TestConnection Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * add protocol in IsReachable function * ignore unused ctx * remove boolean return from IsReachable * use multierrors to wrap all endpoint errors * fix build * use const for IsReachable timeout * rename IsReachable to CheckReachability --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com>

xoscar · 2023-02-15T22:26:05Z

server/id/generator.go

-	tid := trace.TraceID{}
-	g.rand.Read(tid[:])
-	return tid
+	var r [16]byte


@mathnogueira @schoren there is quite an annoying rule by x-ray that requires traces to include a prefix to calculate the creation time, if the trace is older than 30 days the API returns an error, if the prefix is missing x-ray returns an error, I have this code here knowing that is wrong to put it in this place, should we also support a different way of creating traces based on data store? I'm open to ideas

Brainstorming: Maybe we should have specializations of this generator (or about the traceid logic) and do something similar with the connectivity PR?

xoscar · 2023-02-15T22:26:46Z

server/tracedb/awsxray.go

@@ -0,0 +1,801 @@
+package tracedb
+
+import (


Big chunky file that contains everything related to the x-ray connectivity and parsing of the data

Makes sense to create a package for XRay inside of tracedb and break this file by responsibilities/concerns?

…1999)

examples/tracetest-amazon-x-ray-adot/.env

danielbdias · 2023-02-16T13:59:18Z

api/dataStores.yaml

+      properties:
+        region:
+          type: string
+        accessKeyId:


To think of tech debt in the future: this is sensitive data that we should store in our database hashed.

Will be addressed in a separate PR

danielbdias · 2023-02-16T14:04:32Z

server/id/generator.go

-	tid := trace.TraceID{}
-	g.rand.Read(tid[:])
-	return tid
+	var r [16]byte


Brainstorming: Maybe we should have specializations of this generator (or about the traceid logic) and do something similar with the connectivity PR?

danielbdias · 2023-02-16T14:08:45Z

server/tracedb/awsxray.go

+
+		spans = append(spans, nestedSpans...)
+
+		// if seg.Cause != nil &&


Can we remove this code?

danielbdias · 2023-02-16T14:12:38Z

server/tracedb/awsxray.go

+
+func convertToAmazonTraceID(traceID trace.TraceID) (string, error) {
+	const (
+		// maxAge of 28 days.  AWS has a 30 day limit, let's be conservative rather than


Suggested change

// maxAge of 28 days. AWS has a 30 day limit, let's be conservative rather than

// maxAge of 28 days in seconds. AWS has a 30-day limit, let's be conservative rather than

danielbdias · 2023-02-16T14:15:31Z

server/tracedb/awsxray.go

@@ -0,0 +1,801 @@
+package tracedb
+
+import (


Makes sense to create a package for XRay inside of tracedb and break this file by responsibilities/concerns?

…1999)

disable app exporter in demo

* feature: adding support for AWS X-RAY * updating README * cleanup * feat(server): implement viper (#1977) * added quick-start golang app in example (#1940) * added quick-start golnag app in example * Added quick-start-python in examples * chore(docs): add recipe for datadog (#1995) * chore(examples): adding documentation and example of datadog integration * adding datadog recipe * fixing details in docs * Apply suggestions from code review Co-authored-by: Julianne Fermi <julianne@kubeshop.io> --------- Co-authored-by: Julianne Fermi <julianne@kubeshop.io> * Update docs structure and config guides (#1984) * docs(1919): flatten tools and integration * docs(1919): flatten examples and tutorials * docs(recipes): edit naming * docs(config): fix 1982 * examples(tempo): edit config * docs(structure): add redirect * docs(config): update dd * feature(examples): adding net core x tracetest example (#1978) * feature(examples): adding net core x tracetest example * adding readme * addressing feedback comments * adding more information * adding support for the native xray client * refactor connectivity test to reuse components (#1994) * refactor connectivity test to reuse components * only run next step if previous step succeeded * simplify tester TestConnection Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * add protocol in IsReachable function * ignore unused ctx * remove boolean return from IsReachable * use multierrors to wrap all endpoint errors * fix build * use const for IsReachable timeout * rename IsReachable to CheckReachability --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * cleanup * using the new test connection logic * config(server): allow provisioning from file or environent variable (#1999) * cleanup * adding support for session token * feat(server): implement viper (#1977) * config(server): allow provisioning from file or environent variable (#1999) * moving trace id generation to data store * fixing unit tests * chore: disable application exporter in demo (#2012) disable app exporter in demo * removing examples * cleanup --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> Co-authored-by: Mahesh Kasbe <60398112+maheshkasabe@users.noreply.github.com> Co-authored-by: Daniel Baptista Dias <danielbdias@users.noreply.github.com> Co-authored-by: Julianne Fermi <julianne@kubeshop.io> Co-authored-by: Adnan Rahić <adnan@kubeshop.io> Co-authored-by: Matheus Nogueira <matheus.nogueira2008@gmail.com>

* feat(server): implement viper (#1977) * config(server): allow provisioning from file or environent variable (#1999) * chore: disable application exporter in demo (#2012) disable app exporter in demo * feature(examples): adding aws terraform example * updating the tracetest image * feat(server): implement viper (#1977) * config(server): allow provisioning from file or environent variable (#1999) * feat: config deprecation (#2015) * update options to use named fields * add postgresConnString as deprecated * remove nil validations * fix deprecated config fields detection * add logger to config New function * replace zap with go's log package * invert logic of printing the deprecation note Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * unexport logger interface * use camelCase names in tests * update test with Sebastian notes and update logic to make it pass --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * fix: postgres config retrocompatibility (#2043) feat: ensure retrocompatibility if users is using postgresConnString * feature: adding support for AWS X-RAY (#1987) * feature: adding support for AWS X-RAY * updating README * cleanup * feat(server): implement viper (#1977) * added quick-start golang app in example (#1940) * added quick-start golnag app in example * Added quick-start-python in examples * chore(docs): add recipe for datadog (#1995) * chore(examples): adding documentation and example of datadog integration * adding datadog recipe * fixing details in docs * Apply suggestions from code review Co-authored-by: Julianne Fermi <julianne@kubeshop.io> --------- Co-authored-by: Julianne Fermi <julianne@kubeshop.io> * Update docs structure and config guides (#1984) * docs(1919): flatten tools and integration * docs(1919): flatten examples and tutorials * docs(recipes): edit naming * docs(config): fix 1982 * examples(tempo): edit config * docs(structure): add redirect * docs(config): update dd * feature(examples): adding net core x tracetest example (#1978) * feature(examples): adding net core x tracetest example * adding readme * addressing feedback comments * adding more information * adding support for the native xray client * refactor connectivity test to reuse components (#1994) * refactor connectivity test to reuse components * only run next step if previous step succeeded * simplify tester TestConnection Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * add protocol in IsReachable function * ignore unused ctx * remove boolean return from IsReachable * use multierrors to wrap all endpoint errors * fix build * use const for IsReachable timeout * rename IsReachable to CheckReachability --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * cleanup * using the new test connection logic * config(server): allow provisioning from file or environent variable (#1999) * cleanup * adding support for session token * feat(server): implement viper (#1977) * config(server): allow provisioning from file or environent variable (#1999) * moving trace id generation to data store * fixing unit tests * chore: disable application exporter in demo (#2012) disable app exporter in demo * removing examples * cleanup --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> Co-authored-by: Mahesh Kasbe <60398112+maheshkasabe@users.noreply.github.com> Co-authored-by: Daniel Baptista Dias <danielbdias@users.noreply.github.com> Co-authored-by: Julianne Fermi <julianne@kubeshop.io> Co-authored-by: Adnan Rahić <adnan@kubeshop.io> Co-authored-by: Matheus Nogueira <matheus.nogueira2008@gmail.com> * feature(examples) adding tracetest x aws-xray examples (#2030) * feature: adding support for AWS X-RAY * updating README * cleanup * feat(server): implement viper (#1977) * added quick-start golang app in example (#1940) * added quick-start golnag app in example * Added quick-start-python in examples * chore(docs): add recipe for datadog (#1995) * chore(examples): adding documentation and example of datadog integration * adding datadog recipe * fixing details in docs * Apply suggestions from code review Co-authored-by: Julianne Fermi <julianne@kubeshop.io> --------- Co-authored-by: Julianne Fermi <julianne@kubeshop.io> * Update docs structure and config guides (#1984) * docs(1919): flatten tools and integration * docs(1919): flatten examples and tutorials * docs(recipes): edit naming * docs(config): fix 1982 * examples(tempo): edit config * docs(structure): add redirect * docs(config): update dd * feature(examples): adding net core x tracetest example (#1978) * feature(examples): adding net core x tracetest example * adding readme * addressing feedback comments * adding more information * adding support for the native xray client * refactor connectivity test to reuse components (#1994) * refactor connectivity test to reuse components * only run next step if previous step succeeded * simplify tester TestConnection Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * add protocol in IsReachable function * ignore unused ctx * remove boolean return from IsReachable * use multierrors to wrap all endpoint errors * fix build * use const for IsReachable timeout * rename IsReachable to CheckReachability --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * cleanup * using the new test connection logic * config(server): allow provisioning from file or environent variable (#1999) * cleanup * adding support for session token * feat(server): implement viper (#1977) * config(server): allow provisioning from file or environent variable (#1999) * moving trace id generation to data store * fixing unit tests * chore: disable application exporter in demo (#2012) disable app exporter in demo * removing examples * feature(examples) adding tracetest x aws-xray examples * cleanup * cleanup * removing mysql dependency --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> Co-authored-by: Mahesh Kasbe <60398112+maheshkasabe@users.noreply.github.com> Co-authored-by: Daniel Baptista Dias <danielbdias@users.noreply.github.com> Co-authored-by: Julianne Fermi <julianne@kubeshop.io> Co-authored-by: Adnan Rahić <adnan@kubeshop.io> Co-authored-by: Matheus Nogueira <matheus.nogueira2008@gmail.com> * cleanup --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> Co-authored-by: Matheus Nogueira <matheus.nogueira2008@gmail.com> Co-authored-by: Mahesh Kasbe <60398112+maheshkasabe@users.noreply.github.com> Co-authored-by: Daniel Baptista Dias <danielbdias@users.noreply.github.com> Co-authored-by: Julianne Fermi <julianne@kubeshop.io> Co-authored-by: Adnan Rahić <adnan@kubeshop.io>

* feature: adding support for AWS X-RAY * updating README * cleanup * feat(server): implement viper (#1977) * added quick-start golang app in example (#1940) * added quick-start golnag app in example * Added quick-start-python in examples * chore(docs): add recipe for datadog (#1995) * chore(examples): adding documentation and example of datadog integration * adding datadog recipe * fixing details in docs * Apply suggestions from code review Co-authored-by: Julianne Fermi <julianne@kubeshop.io> --------- Co-authored-by: Julianne Fermi <julianne@kubeshop.io> * Update docs structure and config guides (#1984) * docs(1919): flatten tools and integration * docs(1919): flatten examples and tutorials * docs(recipes): edit naming * docs(config): fix 1982 * examples(tempo): edit config * docs(structure): add redirect * docs(config): update dd * feature(examples): adding net core x tracetest example (#1978) * feature(examples): adding net core x tracetest example * adding readme * addressing feedback comments * adding more information * adding support for the native xray client * refactor connectivity test to reuse components (#1994) * refactor connectivity test to reuse components * only run next step if previous step succeeded * simplify tester TestConnection Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * add protocol in IsReachable function * ignore unused ctx * remove boolean return from IsReachable * use multierrors to wrap all endpoint errors * fix build * use const for IsReachable timeout * rename IsReachable to CheckReachability --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> * cleanup * using the new test connection logic * config(server): allow provisioning from file or environent variable (#1999) * cleanup * adding support for session token * feat(server): implement viper (#1977) * config(server): allow provisioning from file or environent variable (#1999) * moving trace id generation to data store * fixing unit tests * chore: disable application exporter in demo (#2012) disable app exporter in demo * removing examples * cleanup --------- Co-authored-by: Sebastian Choren <schoren@users.noreply.github.com> Co-authored-by: Mahesh Kasbe <60398112+maheshkasabe@users.noreply.github.com> Co-authored-by: Daniel Baptista Dias <danielbdias@users.noreply.github.com> Co-authored-by: Julianne Fermi <julianne@kubeshop.io> Co-authored-by: Adnan Rahić <adnan@kubeshop.io> Co-authored-by: Matheus Nogueira <matheus.nogueira2008@gmail.com>

xoscar added 2 commits February 13, 2023 15:05

feature: adding support for AWS X-RAY

9c68b35

updating README

d3376d2

xoscar added frontend backend tracing backend Improve or implement support for a new tracing backend labels Feb 13, 2023

xoscar self-assigned this Feb 13, 2023

xoscar linked an issue Feb 13, 2023 that may be closed by this pull request

Support Amazon x-ray #179

Closed

xoscar marked this pull request as ready for review February 13, 2023 21:13

xoscar added the do not merge label Feb 13, 2023

xoscar requested review from danielbdias, schoren, jorgeepc and mathnogueira February 13, 2023 21:14

jorgeepc reviewed Feb 13, 2023

View reviewed changes

examples/tracetest-amazon-x-ray/tracetest-config.yaml Outdated Show resolved Hide resolved

cleanup

8ce8300

mathnogueira reviewed Feb 13, 2023

View reviewed changes

server/tracedb/tracedb.go Outdated Show resolved Hide resolved

schoren and others added 10 commits February 14, 2023 14:07

feat(server): implement viper (#1977)

2c8d644

added quick-start golang app in example (#1940)

8482bfa

* added quick-start golnag app in example * Added quick-start-python in examples

Update docs structure and config guides (#1984)

820604d

* docs(1919): flatten tools and integration * docs(1919): flatten examples and tutorials * docs(recipes): edit naming * docs(config): fix 1982 * examples(tempo): edit config * docs(structure): add redirect * docs(config): update dd

feature(examples): adding net core x tracetest example (#1978)

df1adb7

* feature(examples): adding net core x tracetest example * adding readme * addressing feedback comments * adding more information

adding support for the native xray client

71ce8c4

Merge branch 'main' into 179-support-amazon-x-ray

0ba0c42

cleanup

ef5dcce

Merge branch 'main' into 179-support-amazon-x-ray

f3d09f0

xoscar commented Feb 15, 2023

View reviewed changes

xoscar and others added 2 commits February 15, 2023 16:32

using the new test connection logic

059d5d2

config(server): allow provisioning from file or environent variable (#…

e445ff9

…1999)

Merge branch 'main' into 179-support-amazon-x-ray

300b13e

danielbdias reviewed Feb 16, 2023

View reviewed changes

cleanup

ffe260c

xoscar removed the do not merge label Feb 16, 2023

xoscar and others added 3 commits February 16, 2023 15:39

adding support for session token

dd0a468

feat(server): implement viper (#1977)

49f5412

config(server): allow provisioning from file or environent variable (#…

13e7a89

…1999)

schoren force-pushed the main branch from e13504c to 16b2765 Compare February 16, 2023 21:44

xoscar and others added 5 commits February 17, 2023 09:33

Merge branch 'main' into 179-support-amazon-x-ray

4647b98

moving trace id generation to data store

8107c44

fixing unit tests

330ac6b

chore: disable application exporter in demo (#2012)

f05caf6

disable app exporter in demo

Merge branch 'main' into 179-support-amazon-x-ray

e40ea02

xoscar changed the base branch from main to new-config February 22, 2023 21:16

xoscar changed the base branch from new-config to main February 22, 2023 21:16

Merge branch 'new-config' into 179-support-amazon-x-ray

18ae9e4

xoscar changed the base branch from main to new-config February 22, 2023 21:54

removing examples

42c9bda

mathnogueira force-pushed the new-config branch from b7f9f9a to 64d99a0 Compare February 23, 2023 19:35

xoscar added 2 commits February 27, 2023 11:52

Merge branch 'new-config' into 179-support-amazon-x-ray

dd7c36a

cleanup

502c7ac

xoscar merged commit 427db01 into new-config Feb 27, 2023

xoscar deleted the 179-support-amazon-x-ray branch February 27, 2023 17:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: adding support for AWS X-RAY #1987

feature: adding support for AWS X-RAY #1987

xoscar commented Feb 13, 2023 •

edited

xoscar Feb 15, 2023

danielbdias Feb 16, 2023

xoscar Feb 15, 2023

danielbdias Feb 16, 2023

danielbdias Feb 16, 2023

xoscar Feb 16, 2023

danielbdias Feb 16, 2023

danielbdias Feb 16, 2023

danielbdias Feb 16, 2023

danielbdias Feb 16, 2023


		spans = append(spans, nestedSpans...)

		// if seg.Cause != nil &&

	// maxAge of 28 days. AWS has a 30 day limit, let's be conservative rather than
	// maxAge of 28 days in seconds. AWS has a 30-day limit, let's be conservative rather than

feature: adding support for AWS X-RAY #1987

feature: adding support for AWS X-RAY #1987

Conversation

xoscar commented Feb 13, 2023 • edited

Changes

Fixes

Checklist

Using AWS XRAY SDK to instrument plus the ADOT COllector

Using AWX XRAY SDK to instrument plus AWS XRAY SDK to pull Traces

Using OpenTelemetry (Pokeshop) to instrument plus XRAY ADOT collector exporter plus AWS XRAY SDK to pull Traces

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xoscar commented Feb 13, 2023 •

edited