bug-1886018: support pubsub crashqueue

[relud]

No description provided.

[relud]

the make setup step pauses for like 6 minutes and then ends up with a "504 Deadline exceeded". What's going on is that make runservices doesn't start pubsub.

pubsub definitely needs to be added to runservices, but also i feel like either make setup should use a different container for the shell that defines depends_on in docker-compose.yml, or app should define depends_on. given that we don't have any extends in docker-compose.yml.

i'm in favor of changing app, but i don't have the context to be sure that's the right choice.'

edit: i'm going to commit the change to app, but i'm happy to remove/change that solution

[relud]

sometimes the crash report will get processed twice. I saw this happen twice in the minimal testing I did. Also, it looks like two threads each get the crash id, so it's getting processed twice at the same time.

I don't remember having this problem years ago when I did pubsub the first time around. Are we missing a setting somewhere in either the topic or subscription creation? Maybe this is a problem with the emulator we're using?

Also, I took a break to get a cup of coffee and after I came back, I can't reproduce it anymore.

I'm pretty sure the issue isn't the emulator, it's the pubsub client sdk. the data platform's ingestion-edge service has a flaky test where sometimes messages are published twice, and that's using a custom emulator.

last time I looked the data platform had a duplicate rate of ~6% between between the edge and bigquery, which goes edge->pubsub->dataflow->pubsub->java app->bigquery, and the dataflow job uses beam's built-in deduplication based on document_id over a 10-minute window.

is this something that will break socorro, or is it just wasted compute? if it doesn't break anything i'd be inclined to let it slide.

[willkg]

Getting processed multiple times and even having two things processing at the same time where one stomps on the other's results--Socorro should be fine with both of those situations. It's just wasted compute and if we're looking at numbers of crash reports collected vs. processed, there will be some discrepancy.

[willkg]

I pulled down the changes in this PR. I have a mostly default local dev env--very little is set in my my.env file. So it should be running in "AWS mode". When I run ./bin/process_crash.sh, it fails with an "unbound variable" error. That should get fixed. I provided a diff.

There were a couple of other comments of minor things.

I went through and tested the CLOUD_PROVIDER=GCP mode and processing, reprocessing with webapp, and reprocessing with api all work fine.

Almost there!

[relud]

ready for another pass

[willkg]

Looks good--thank you!

I went through this test plan:

1. with CLOUD_PROVIDER unset
   1. does it build? does it rebuild?
   2. can i verify it's using sqs and not pubsub?
   3. does processing work?
   4. does reprocessing api work?
   5. does reprocessing via webapp work?
2. with CLOUD_PROVIDER=GCP
   1. does it build? does it rebuild?
   2. can i verify it's using pubsub and not sqs?
   3. does processing work?
   4. does reprocessing api work?
   5. does reprocessing via webapp work?

[willkg]

Wow! That's bonkers! I had no idea that existed.