Send notification emails

[samreid]

See #4

[samreid]

We are in the habit of checking Bayes regularly, do we need to add an email component too? I'll create voting by thumbs ups: You can unassign yourself after you vote.

[samreid]

Yes, email integration would still be valuable and we should take the time to implement it.

[samreid]

No, email integration is not worth the effort because we are in the habit of checking Bayes regularly.

[Denz1994]

phetsims/chipper#410 (comment)
Agreed if new bugs are emailed.

[jonathanolson]

I see a lot of "yays" for this. I'm interested in what the "new bugs" would really mean then.

For instance, if I cause a runtime error that hits every simulation on startup, that will cause failures in ~87 runnables, each currently with 8 tests (~700 individual tests that were passing will start to fail).

Furthermore, the same test can fail with many different stack traces (depending on the fuzz).

We also have flaky tests (ones that fail randomly mostly based on the fuzz events - once a day/week/month).

If we have a system that will give us useful information on failures (but won't spam us with 1000s of emails - or even more than 1 email per type of failure), then that sounds great (and non-trivial).

No, email integration is not worth the effort because we are in the habit of checking Bayes regularly.

Presumably we don't need to check bayes regularly because (a) we are almost always linting/testing before pushing, (b) usually for commits that would have the potential to break things, we'll push, wait and check, and (c) QA is checking bayes regularly, to catch things that weren't anticipated above. I'm not even checking bayes weekly, but usually only if I need to do some testing that requires knowing about currently failures (or branching a sim from master), or if I notice something broken locally and want to quickly confirm if it's an issue for others.

So I'm fine if someone wants to work at email notification, but it doesn't currently seem like it's worth the effort to me.

[jbphet]

I voted no, and thought I should explain myself. I'm in the habit of checking Bayes every morning, and if I see anything that needs attending to, I put it on my to-do list for the day and prioritize it relative to everything else. If I change something in common code, I set a timer and check Bayes some time later. Email would be unlikely to change this behavior for me, but I wouldn't object to it being added if others would find it valuable.

[pixelzoom]

I'm changing my "yay" to a "nay". I think it would be difficult to identify "new issues", and I don't want to hear about chronic issues repeatedly. And it sounds like people are already successfully identifying problems manually. So I don't think this is worth the work to automate this. Perhaps better to emphasize that, if you make a common code change, then you'd better have a look at bayes after you commit and push.

[samreid]

@jonathanolson made a compelling argument, I'm going to move my thumb's up to the "not now" column. Once things are very stable and errors are more rare, we can re-evaluate.

[samreid]

Based on preceding remarks, I'm inclined to close the issue.