Add spans data schema

[dbanda]

Add spans data schema. This can help us validate messages in the spans data bag.

Also:

• Fixed typo in the readme
• Add title to the top level transaction message

[github-actions]

versions in use:

The following repositories use one of the schemas you are editing. It is recommended to roll out schema changes in small PRs, meaning that if those used versions lag behind the latest, it is probably best to update those services before rolling out your change.

• getsentry/sentry: pip:sentry-kafka-schemas==0.1.12
• getsentry/snuba: pip:sentry-kafka-schemas==0.1.12

latest version: 0.1.12

benign changes

schemas/transactions.v1.schema.json

• {"path": ".2.data.spans.?.data", "change": {"TypeAdd": {"added": "string"}}}
  

• {"path": ".2.data.spans.?.data", "change": {"TypeAdd": {"added": "number"}}}
  

• {"path": ".2.data.spans.?.data", "change": {"TypeAdd": {"added": "integer"}}}
  

• {"path": ".2.data.spans.?.data", "change": {"TypeAdd": {"added": "array"}}}
  

• {"path": ".2.data.spans.?.data", "change": {"TypeAdd": {"added": "boolean"}}}
  

✅ This PR should be safe to roll out to consumers first. Make sure to bump
the library in the following repos first:

getsentry/snuba

...then in the other repos:

getsentry/sentry

Take a look at the README for how to release a new version of sentry-kafka-schemas.

[lynnagara]

Is there a reason why all the spans/starfish experiments has to be on the transactions topic still, and we are not creating a separate one for the new pipeline? With all the schema changes that are going on, I think it's much safer to keep it separate.

These additional validations are not required on the transactions consumer.

[dbanda]

Is there a reason why all the spans/starfish experiments has to be on the transactions topic still

@lynnagara eventually we will have spans on a new topic and ultimately spans will replace transactions. But right now, we decided to leave it there as we iterate. My understanding is starfish wants to test things and move fast. The issue is that we are stuffing a lot of things in the spans data bag and don't have any checking on it. I could create a separate schema for SpansData, but that would have to mean adding some customized code within the spans consumer to check the data field. Either way if fine with me, but I would like to have a schema defined somewhere so that we have a unified source of truth and catch typing bugs.

[untitaker]

We have had a similar problem already. I think long-term we might want to have specialized schemas per consumer group.

With all the schema changes that are going on, I think it's much safer to keep it separate.

The schemas are not changing. We are just parsing more/less data out of the same payloads.

What is more concerning to me is that we are trying to make sense of this data while there is absolutely zero validation for spans data in Relay. The next buggy SDK could take down a spans consumer that tries to assume anything about the payloads.

[lynnagara]

My concern was coming from a place mostly unrelated to this schema change. I'm just not sure it's a good idea to build and iterate fast on things inside the existing pipeline just because it's "faster". We are adding risk into our existing transactions pipeline. Though if we definitely want a separate topic and schema eventually anyway, should we just go ahead and create it now?

[untitaker]

We are adding risk into our existing transactions pipeline

yes, multiple schemas per topic would solve this. there's not really a risk of one consumer group impacting another if they do not share a schema

[lynnagara]

@untitaker In this case, I don't think they should even be sharing a topic at all though. My understanding is that's a shortcut we are taking right now.

[untitaker]

Currently it's only sentry-kafka-schemas that makes this dangerous. I don't think there's any other reason to split out topics right now

[lynnagara]

It isn't sentry-kafka-schemas that makes it dangerous though, as we are currently not enforcing the schema. I'm actually more worried about the changes we are doing in consumers and producers and generally changing the shape of messages.

Anyway, most of these concerns are unrelated with this change. This particular change LGTM so will approve.