Creating a sample event fails to be stored

[armenzg]

This issue may be revealing really worse underlying issues.

This project requested sample events three times. The group was created but the events did not make it into Snuba.

You can see the group in the issue stream (stored in PG) [1]. Loading the issue will show "sorry, the events for this issue could not be found" [2]. You can see that we count the events but they're not stored [3].

Action items:

• Troubleshoot what happened
• Create metric and alert for this

[1]

[2]

[3]

[barkbarkimashark]

Was able to repro this and whats curious is when I click the create sample event button and the issue shows up on the issue stream in the for review tab. The issue has an event count of 0/0.

It seems like somewhere down the event creation pipeline, the event never makes it to snuba which powers the issue search functionality for tabs that aren't For Review. So if we figure out why the event never gets saved, I suspect this issue will also be fixed.

I don't think this issue is specifically related to issue states and whether the feature flag is on/off.

[armenzg]

Thanks @barkbarkimashark !
I've removed the issue states milestone relation.
This is likely related to #inc-417 which we're still investigating.

[brianthi]

Synced with Armen, and he doesn't think it's related to #inc-417. For some reason, the event is just not making it into snuba. The thing that'll need to be done is reproduce, and work with SnS or Ops to trace the event and understand why it's not making it's way in.

[snigdhas]

The sample event creation creates 3-5 events and groups them into a new issue. Looking at the network tab, I see that the issue gets created, but the call to get the latest event (events/latest) 404s. It keeps retrying for a while before failing silently.

If I get the issue ID from the request and visit the URL, I see the Sorry, the events for this issue could not be found. error above.

I'll post this in discuss-ops/sns to see if anyone has insight/can help further.

[snigdhas]

We've confirmed that sample events are not making it into Snuba. The path taken to save a sample event is different than that for regular event ingestion, so this shouldn't indicate that we're dropping events elsewhere. The next step is to add logging to understand exactly where in the event save logic this is failing.

[brianthi]

#54748 fixed this issue