Create test trace data and ingest it into a test instance of Open Search in aws

[sarahzinger]

Worth checking to see if we already have something like this set up, but we need a way to create a view traces in open search.

Assuming we do not already have a test instance of open search service in aws with valid traces, we will likely need to:

• make some kind of test app that is instrumented with open telemetry
• then we need someway to collect and format that data and send it to an instance of open search (ideally one that is deployed in aws, so we can share between team members) to be ingested.

In this pre-req section there's some talk on how to do this: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/trace-analytics.html although it's possible there might be better sample apps already. They mention X-Ray I wonder if it's possible to use the same test apps we have from x-ray and send that data to open search somehow? I think theoretically it should also be easy to sort of fake a trace if we know the proper format, as opensearch can ingest data with a curl script, but is probably best long term to have a test app with real traces we can play with.

found another test data app here: https://opensearch.org/docs/2.6/observing-your-data/trace/getting-started/

more helpful info here: https://docs.google.com/document/d/1l8HYKvCddF9jUdkAV0TkHU6fsWFs8T1s-gLWbaGLtog/edit

[fridgepoet]

Some notes:

• Not yet working in AWS: Found this guide and tried it out on us-east-1 on our external dev account https://aws.amazon.com/blogs/big-data/part-1-microservice-observability-with-amazon-opensearch-service-trace-and-log-correlation/ however I didn't see traces in the dashboard after clicking around on the "Logs and traces generation" step.

• OK locally: the steps here https://docs.google.com/document/d/1l8HYKvCddF9jUdkAV0TkHU6fsWFs8T1s-gLWbaGLtog/edit#heading=h.afsljqmneoby be sure to replace the image version as specified "change the image:jaegertracing/example-hotrod ‘s :latest to :1.41.0" -- otherwise there is this issue

I think that development with local traces might be possible for now.

[iwysiu]

Further notes:

• Attempted both the guide that Shirley found and the workshop that guide was created from and was not able to get them working.
• Development is possible with just local traces
• After talking to AWS about it, they have been made aware the guides don't work and suggest running the local steps (cloning the example repo, running docker) in an aws instance instead.

[fridgepoet]

Let us know if you want us to look into this!