Improvements for Jenkins CI setup.

[simitt]

This is a Meta Issue to collect ideas and improvement suggestions for further Improvements regarding Jenkins setup.

• Slack Notifications on every build
• Email Notifications on failure
• Change how often the dependency tests are run or prioritize them lower somehow

[watson]

Change how often the dependency tests are run or prioritize them lower somehow

I'd love input on how to do this better. The reason why we just do it every time right now is because a change in the agent can break support for an old version of one of the modules we instrument. And since we don't control which versions of say mysql that users are running, we can't just only test against the latest version. Instead we have to also test that we still work as expected with old versions. But there might be a smarter way to run the tests, so I'm open to suggestions 😃

[ruflin]

@watson How often did you break these in the past and how likely are they to break? Reason I'm asking because my suggestion would be to run them only after a merge into master if breaking it is pretty rare but allow a command like jenkins, test dependencies when you want to check it also on a PR directly. We do this in beats for the packaging for example. Like this the PR's get green much faster as breaking packaging is unlikely but still can happen. In case we change code that we know is critical for pacakging, we comment jenkins, package it to see if the packaging still works before merged into master. If we don't do it and a pacakging is broken after merging into master, we get a notification later on that we did.

[watson]

@ruflin Hard to say. It has happened before, but it's not that often. Your solution could work. I guess I just have to get comfortable with potentially broken code sneaking into master, but in many cases I'm able to know exactly when instrumented modules need to be tested. Maybe we could even narrow it down to say only test the pg module or something.

[ruflin]

Sounds like we could try this and see where it breaks? :-)

[watson]

@ruflin let's do that

[watson]

@simitt would you be able to take this on?

[simitt]

I first have to figure out the current issues with the webhooks. As soon as the general setup is fixed I can take this over.

[watson]

Just thought I'd do a quick status update on our progress with the Jenkins integration:

These tasks are all implemented:

• Run basic test suite on all new PR's and when merging to master
• Email Notifications on failure
• Only run dependency tests once a day and not as part of PR tests

These tasks are still not implemented:

• Add GitHub comment command to trigger dependency tests in a PR (just as with jenkins test this please)
• Slack notifications on every build

[simitt]

thanks @watson.
There are discussions going on about how to provide the possibility to trigger jenkins.
The slack notification is simply not implemented yet.

[watson]

I'd say the slack notifications are pretty low priority anyway, so I'm fine with that 👍

[watson]

Since we got the ability to trigger jenkins builds for tav tests, I'll go ahead and close this issue