Investigate Cortex exporter test panics

[MrAlias]

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-go-contrib/814/workflows/0312cefe-6305-4bf6-aac5-410ba6a5e5cc/jobs/1444

Looks like a timeout is causing the testing to panic after 30 seconds. Can this behavior be designed away otherwise it should be tuned to not panic.

[evantorrie]

This always seems to be for the 386 build only. Profiling it shows 90+% of the time in math/big.addMulVVW which is part of the crypto random number generation for the CA cert, server cert and client key/cert.

[MrAlias]

Might be related to golang/go#22643

[MrAlias]

From the SIG meeting today we talked about making the certs static and storing in a testdata directory that is checked in.

[evantorrie]

I think this has been addressed already, but an alternative would be to drop the number of bits in the RSA private key generation. Dropping from 4096 to 1024 bits improves runtime by around two orders of magnitude on my machine.

[ercl]

Thanks for that tip! I didn't think lowering it to 1024 would improve performance so much. While I was testing that out, I also tried using ECDSA P256 and I ended up changing the key generation functions to use that since it seems to have even better performance than RSA 1024. I added some of my results from pprof on my PR comment.

[ercl]

I think this issue this can be closed since it was resolved by #315.