Not all EC2 instances tag themselves

[ansoncfit]

After #468, workers are supposed to tag themselves. On a static site analysis, I used the EC2 console to "launch more like this" in two rounds. In the first round, I requested 99 instances and all tagged themselves. In the second round, a few minutes later, I requested 200 instances and only ~2/3 of them did.

[abyrd]

That's strange. Maybe EC2 API rate limiting?

[ansoncfit]

Good idea, I'll look into it and consider adding a small random delay. The batches of 150 just started are exhibiting similar inconsistent behavior.

[ansoncfit]

Starting many batches of 100 yesterday, it appeared that all instances successfully tagged themselves. I didn't see limits on tagging following the instructions at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html, but rate limiting above ~100 tagging requests at a time does appear to be a likely suspect.

[abyrd]

Sounds promising. I think adding some random jitter of say 0-20 seconds (as we do with the worker polling delay) would do the job.

[ansoncfit]

Starting a batch of 200 with a random delay of up to 20 seconds did not seem to slow things down enough. If we move the tagging operation after the R5 startup script so we don't delay users, we could try a longer delay (maybe ~4 minutes)?

[ansoncfit]

Fixed in #488, then moved to analysis-backend in conveyal/analysis-backend#203