[SYNPY-1305] Collect trace data from integration tests

[BryanFauble]

Problem:

• We want to automatically collect trace information for integration tests that are ran - allowing us to inspect this data without us manually running the tests on local machine.

Solution:

• Created the aws config file and encrypted the values - it contains the PAT needed to get the AWS STS token to start a port forwarding session
• Start a port-forward session in the CICD pipeline
• Export trace data to the jaeger server running in the Sage EC2 service catalog
• I re-worked how we determine if we are running integration tests to be earlier in the job
• I added a run_opentelemetry boolean during the aws-check step

Testing:

• I could not get this working on windows so I disabled it connecting to EC2
• I verified that I could export trace data for both linux and macos:
  
  

[BryanFauble]

This all shows as changed because I removed the if check here - since I am doing it earlier in the steps.

The only notable change from this block is L175:

          if [ ${{ steps.aws-check.outputs.run_opentelemetry }} == "true" ]; then
            export SYNAPSE_OTEL_INTEGRATION_TEST_EXPORTER="otlp"
          fi

[zaro0508]

This is neat however the GH workflow is pretty complex, would it make sense to create a custom GH action for it?

[BryanFauble]

This is neat however the GH workflow is pretty complex, would it make sense to create a custom GH action for it?

@zaro0508 Maybe - I had originally intended to use https://github.com/marketplace/actions/aws-ssm-port-forwarding-session - But I tried it in this pipeline and I couldn't get trace data to export properly, so I fell back to doing it myself

[thomasyu888]

🔥 nice - I'm going to pre-approve, as I just had a couple comments.

[thomasyu888]

out of scope of this review, but did we hear back from Xa about the stfp server?

[BryanFauble]

I haven't heard anything additional