workload/schemachange: export OTLP traces to disk

[chrisseto]

aa05213 workload/schemachange: export OTLP traces to disk

Previous commits instrumented the schemachange workload with OTeL but
did not provide a way to access the output spans.

This commit implements a OTLP client that exports to a gzipped JSON+nl
file. As noted in the client's documentation, tooling for this format is
sparse but JSON wrangling tools can provide extremely rich insights. Due
to versioning issues, loading traces into a tool like Jaeger will
require a small amount of post-processing.

Epic: none
Release note (cli change): workload schemachange now writes a
.otlp.ndjson.gz archive containing OTLP trace bundles for debugging
purposes.

[cockroach-teamcity]

This change is 

[rafiss]

lgtm! just had minor questions

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @chrisseto, @DarrylWong, and @renatolabs)

---

pkg/workload/schemachange/schemachange.go line 119 at r2 (raw file):

			`If provided, transactions will be written to this file in JSON form`)
		s.flags.StringVar(&s.traceFilePath, `trace-file`, "",
			`The file to write OTeL traces to. Defaults to schemachange-workload.{timestamp}.otlp.jsonnl.gz`)

nit: should we juse use *.json.gz or *.ndjson.gz? (same for other usages below)

---

pkg/workload/schemachange/tracing.go line 119 at r2 (raw file):

	var err error
	c.file, err = os.OpenFile(c.Path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0660)

does this mean we'll append to the file if it exists already? is it more usable to recreate the file?

[chrisseto]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DarrylWong, @rafiss, and @renatolabs)

---

pkg/workload/schemachange/schemachange.go line 119 at r2 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

nit: should we juse use *.json.gz or *.ndjson.gz? (same for other usages below)

Fixed!

---

pkg/workload/schemachange/tracing.go line 119 at r2 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

does this mean we'll append to the file if it exists already? is it more usable to recreate the file?

Yes it does and now that you ask not in the slightest. If the file was just ndjson, we could append just fine but the addition of gzip means we'll just corrupt the archive 😬

Fixed to just use os.Create. Good catch!

[rafiss]

lgtm! just asked about a pending todo

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @chrisseto, @DarrylWong, and @renatolabs)

---

pkg/workload/schemachange/schemachange.go line 822 at r3 (raw file):

	path := s.traceFilePath
	if path == "" {
		// TODO(chrisseto): Before merge ensure this default output is to a

were you able to check this?

[chrisseto]

TFTR!

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DarrylWong, @rafiss, and @renatolabs)

---

pkg/workload/schemachange/schemachange.go line 822 at r3 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

were you able to check this?

Yep! It works as expected though it's not super ergonomic to access as I noted in slack. I might spend some later trying to make the logged directory a clickable URL in TC.

[chrisseto]

bors r=rafiss

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)