Added support of unstable and aborted build statuses

[gmykhailiuta]

With unstable and aborted statuses team can save their nerves when some test failed or somebody stopped the build.

[gmykhailiuta]

Fixed some bugs and added elapsed time in #11. Not sure if I should close this one.

[jan-molak]

Hi gmykhailiuta and thanks for the time you spent contributing to the project!
I'm on holidays at the moment, but I'll have a look at your pull request over the next two weeks and give you some feedback.

[ghost]

I really need this functionality, how long before the code makes it into a new release?

[gmykhailiuta]

Jan didn't implemented this functionality for some reason. Anyway, you can find it done already in my clone: https://github.com/gmykhailiuta/jenkins-build-monitor-plugin. Please, mail me if you don't want to build it, I'll provide you the built module.

[ghost]

It would be a great help if you could build it for me thanks.

---

From: Gennadiy Mykhailiuta [mailto:notifications@github.com]
Sent: Wednesday, 30 October 2013 0:21
To: jan-molak/jenkins-build-monitor-plugin
Cc: SUMNER Andrew
Subject: Re: [jenkins-build-monitor-plugin] Added support of unstable and aborted build statuses (#9)

Jan didn't implemented this functionality for some reason. Anyway, you can find it done already in my clone: https://github.com/gmykhailiuta/jenkins-build-monitor-plugin. Please, mail me if you don't want to build it, I'll provide you the built module.

Reply to this email directly or view it on GitHubhttps://github.com//pull/9#issuecomment-27294928.

The information contained in this email message is intended only for the addressee and is not necessarily the official view or communication of the New Zealand Customs Service.

This email may contain information that is confidential or legally privileged. If you have received it by mistake, please:
(a) reply promptly to that effect, and remove this email and the reply from your system; and
(b) do not act on this email in any other way.

[jan-molak]

Hi Gennadiy and thanks again for your contribution.

The reason why I didn't want to distinguish between an "unstable" and "broken" build is because I consider the concept of an "unstable" build flawed and harmful. Before you accuse me of heresy please allow me to quickly explain why that is the case :)

According to Jenkins terminology "a build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable." What this means is that your build will be considered "unstable" should any of your tests fail...

Now, if we'd like to distinguish between a "broken" and an "unstable" state this implies that we're also happy to accept the fact that some of our tests might fail from time to time. And here's where I see the problem - if you have tests that you know are failing, is there anything you can do about them rather than accepting that your project is "unstable"?

On several occasions I've worked with teams that started with accepting say 5% of their tests failing as "normal and to be expected" (justifying this state of things by saying that "those 5% are caused by a flaky browser-based test, and hey, 95% percent of our tests is still passing, right?").

Over time, however, 5% became 10%, then 15% then 50%, eventually ignoring test failures got them to a state where they started questioning automated testing all together! (consider the Broken Windows Theory).

My suggestion is to avoid using the very concept of "unstable builds" and react to build failures by either:

• fixing the offending test
• stabilising a test or moving it to a different test suite (i.e. do you need to use a UI-based test where a lower-level test would suffice? See Martin Fowler's Test Pyramid)
• if the failure is caused by one of Jenkins' publishers, could you use a more stable one to achieve the same result?

I believe that the primary task of Build Monitor is to alert the team should anything be wrong with their project. Giving them a false sense of security is not compatible with this goal I'm afraid.

Having said all this, the team should also be made aware of anyone trying to fix a broken build (that's especially useful for distributed teams), that's why I'm currently working on adding support for the Jenkins Claim Plugin (See #19). Hopefully, making a build amber when someone is actually actively working on fixing it should help your team save their nerves :)

What do you think?

[gmykhailiuta]

Wow! It could be a separate article. I think it make sense and it's an interesting point that I can present to my team too. Maybe it's not widespread approach, but we have jobs which deploy application and run smoke test on it. So, with "unstable" status it is easier to distinguish failed deploy (failed status) from "deployment succeeded, but some tests failed" (unstable status). The first is most likely devops' responsibility, but the second is developers' one. Also, you can configure job not to use unstable status and mark it as failed if it's unstable. Anyway, the best solution might be giving the users an option to choose, enable unstable status or not in the plugin's configuration.

[ghost]

I agree that failing tests should fail the build - that's exactly what ours do. My need for the distinction is around code analysis. Eventually I would like to get to zero tolerance but at the moment I would like to mark builds as success when zero check style faults, unstable if under a threshold, and failed if over threshold. That way developers can also distinguish between a build that failed to compile and needs immediate attention or one that just needs a bit of tidying up.

[ghost]

I'm trying to build the plugin and get the following error, any ideas?

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default) on project build-monitor-plugin: Command execution failed. Cannot run program "npm" (in directory "C:\Work_Java\eclipse_workspace\jenkins-build-monitor-plugin-master"): CreateProcess error=2, The system cannot find the file specified -> [Help 1]

[gmykhailiuta]

You need node.js and npm (node.js package manager) to build plugin. See http://nodejs.org/download/

[fonsecas72]

Jenkins builds not always have tests in them. Sometimes they may refer to static analysis results so I may want to distinguish from a bad build that exceeded threshold by 20 from another very bad build that exceeded threshold by 50.
For my company, this functionality is the only thing missing for us to use it. It would be awesome if this was added even if optionally, as this plugin seams to be much attractive then the one we use know. Please please add this functionality.

[wilriker]

I would also love to see at least an option to have different colors for all the different Result possibilities (ABORTED, FAILURE, NOT_BUILT, SUCCESS and UNSTABLE). I see the point in your argumentation, Jan, but we also do distinguish between compile/build errors (FAILURE) and tests/post-build-trigger failing (UNSTABLE). Also we add some new CI jobs once in a while and I am always asked why this is failed even though it is just NOT_BUILT instead (and believe me it has always been recognized before Jenkins had a free executor ;-) ).

So please at least add it as an opt-in option.

[ghost]

An opt in option for unstable builds does seem a good compromise :-)

[jan-molak]

Hi guys and thanks for all your feedback - it will definitely help to improve the Build Monitor even further :)

My main focus is to ensure that the information radiated by the monitor is clear and simple to understand.
In order to achieve this goal, the monitor has to make it extremely easy to find out at a first glance whether a given Jenkins job is:

• working as expected - green
• broken, but under control (failing, unstable or aborted, but claimed) - amber
• broken, and no one is looking at it (failing, unstable or aborted, but not claimed) - red

If you considered all the states a Jenkins job can be in, you'd end up with the following list:

• SUCCESS
• UNSTABLE
• FAILURE
• NOT_BUILT
• ABORTED

If we wanted to have a separate colour for any of those states, we'd end up with a bit of a rainbow and it seems that this defeats the main objective. (Hey, did purple mean unstable? No, I think that was the deep blue that meant that... or The build is red! No it's not, it's PINK, it means it's ABORTED not FAILING ... ;-) )

I see your point of view, @hugofonseca, and I understand that Jenkins jobs don't necessarily need to have tests in them, and as you mentioned can be used for static code analysis for example.
Even in this case you might easily end up with a rotting build scenario I described earlier if you don't address problems as soon as they arise... That's precisely why I'm reluctant to encourage dev teams using this humble plugin to start ignoring builds that are failing tests they wrote and exceeding thresholds they set.

What's your opinion? Would you treat a build that exceeded the threshold by 20 any different from the one that exceeded it by 50?

I'd apply similar logic to post-build tests and triggers. Would you ever want to leave a job in an UNSTABLE state?
Also, what does it mean that a build is unstable? Sometimes it works, sometimes it doesn't? It mostly works? Can an unstable application be seen by your customers? Is it OK to be unstable? What's your view?

@wilriker - To help you answer the question you're being asked: why this is failed even though it is just NOT_BUILT - in test-driven development (TDD) you start with writing a test before writing the class you're testing. What happens when you run your test before implementing the class? :)
But more seriously: How long do you usually have your newly created jobs in a NOT_BUILT state? Do you think it would be beneficial to have NOT_BUILT state distinguished on the same level as SUCCESSFUL, FAILING and CLAIMED?

Looking forward to hearing back from you and thanks again for helping me make this plugin better!

[hakanai]

The task has a haltonfailure property which lets you configure the build such that any failing test results in a failing build.

If you really want all unstable builds to be marked as failed, why don't you just set that flag in your build, rather than forcing your beliefs on everyone else? Software development isn't supposed to be a religion.

[ghost]

I think you have some very valid points @jan-molak, and I agree that clarity is the goal here. I could see that I might eventually want SUCCESS, FAILING, CLAIMED, UNSTABLE, ABORTED, and DISABLED. I don't necessarily agree that we'd end up with with an unclear monitor as the goal is to keep things green.

If you choose not to implement them I'm not likely to use another monitor tool anyway as this is the best I've come across so far. As much as I'd like the flexibility to use these colours if I need them, I'd prefer sound alerts :-)

[wilriker]

I agree with you @jan-molak about having a preselected individual color for every possible state could be confusing. As an example our settings would probably look like the following:
SUCCESS: green
FAILING: red
everything else: amber (maybe explicitly NOT_BUILT: grey, not sure about this though)

To explain this shortly: you also asked whether we would treat UNSTABLE and FAILING builds differently. Yes, we do - but only in priority. A FAILING build needs immediate attention (because this might block other team members from continuing with their work), whereas an UNSTABLE build usually means that retriggering the build will solve it because local Artifactory was overloaded or some of our not so stable tests was failing. Of course there needs to be a solution for the cause and not the symptoms and we are working an that but it takes time. Also we don't use the Claim Plugin because we have another way of communicating such things.

That will obviously not fit everyone's needs and workflows. Therefore my proposal here would be to have the possibility to select the colors for every state. This might either be from a predefined list (which would probably make implementation for you a lot more easy) or even a color-chooser. This way at least everyone could adapt the Build Monitor to their individual workflow.

And about the NOT_BUILT state: for example we do our production deployments using a Jenkins instance as the UI for a command-line tool. If a "build" fails there this usually means at least one production server is down and this requires immediate attention. But on the other hand it could be that we already add a job for a new cluster which is currently being set up on Wednesday and the next regular deployment would be the following Tuesday for example. That means we have one deployment job for nearly one week which is red and everyone gets shocked when they enter the room although there is no problem at all because is has just not yet being started at all.

As you can see this plugin is being used for a wide variety of use cases and not all of them fit into what you intended. No offence, as I already said you have some valid points but you should at least give users the opportunity to configure differently.

[fonsecas72]

I agree completely with you @wilriker! At where I work we also look at failing builds as you, so every red build WILL take our attention immediately.
We use unstable for example for things that QA team will fix or report to PM. This means that no one else should interrupt their work for yellow builds.
About grey builds: We just use them for disabled builds just to remind us that those builds are under construction, meaning that work is still to be done but not right now.
I also agree with @jan-molak point of view but not everyone of us use his methodology and being so we should be free to choose the best one to use.
Please :)

[dickschoeller]

We run into an area where this is an issue. The HP Application Automation Tools plugin to report on QTP runs. The QTP runs always generate a warning because of the method that is used to find certain elements on the page. There is no way to tell that plugin not to mark those runs unstable. If this were distinguished from actual failures of the tests we would benefit.

One could make the argument that we should change the way the test scripts find page elements. Our UI test automation lead is working on that.

One could also make the argument that the HP Application Automation Tools plugin should have control over the thresholds for stable/unstable and success/failure as do other plugins. I plan to request that too.

[jan-molak]

Thanks for all the feedback guys, it's really useful :) I'll most likely make this a configurable option at some point.

[drekbour]

Ditto. Test fail = Unstable doesn't feel right to me either but the generic concept of Unstable/Failed does have value. I'd vote for a BuildMonitor that reflects unstable in yellow and has a boolean setting "Show Unstable builds as Failed"

FWIW Bamboo has a Quarantine option for failing tests. Users can manually Quarantine a failed test which will no longer fail a build (until passed again). There is much discussion about whether this is sensible or not :)

[gstamac]

This is an excellent idea. With Warnings plugin you can have unstable build if compiler generated warnings. This build is not broken. I do agree that one failing test means the build is broken but you have other settings to force this policy. Please include this pull request. It would make Build Monitor much better.

[Mostyn]

Great work Jan. We are using the Build Monitor on two screens in our office. One has an overview of more than 64 jobs whilst the other is more project focused with only twelve jobs displayed.

Whilst I appreciate your thinking about your own requirements and judgements as to what is green and what is red for your purposes there are other teams that have broader range of severity.

The Jenkins system has a status for each job and a coloured sphere to denote this. As far as I am aware, on our system, there are only four coloured statuses: GREEN (success); RED (failed); AMBER (unstable) and GREY (Disabled, Aborted, Not built...)

We would be very happy if the Build Monitor simply used those four colours that Jenkins already uses. We would also be happy if it was optional to have the binary (red/green) or the standard Jenkins status colours (red, amber, green, grey). Can you possible release a feature that does this soon?

Of course anything other than Green would have cause to be evaluated. The amber, red and grey colours would indicate some kind of severity. For us a Build failing to install is more severe (red) than any unstable job (amber) which could be unit test failing or, often for us, a warning during parsing of the console log.

Looking forward to any further updates.

[weissmag]

Just for the sake of voting, We would also appreciate using the full color range by Jenkins, as for us yellow is a test issue and red is a deployment issue... and each is addressed by separate teams...

You can default the configuration to binary and let us choose to remove it.

in any case Great Job !

[stephane-benoist]

Isn't this a misuse of "unstable".
In my mind, "unstable" has to be used for abnormal metrics (e.g. high amount of duplicate code, plenty of cs violations...)
In this case, I think it's logical to allows to distinguish both.

[qse]

Hi there,
for us it would be very great to have the 4 different colors (green, amber, red and grey) of Jenkins, because in my team (QA team) we use our wallboards (information radiator) to monitor automated tests (ci). And we differentiate the job based on the number of failed tests.
In the standup we use the wallboards to get an impression over the tests and if most of the tests are red (even if there are only 0.001% test steps failed) there is confusion.
Over that, if the team only reacts based on a color of a information radiator ... there are other issues in this team. ;-)
Anyway, I love the idea of having the option (opt-in) to use the full color range. :-)

[ilor]

++, I could really use this or an equivalent feature. More options than just green/red, please 👍

[gusran]

We need this. We do not fail the build on code analysis warnings and we want to distinguish between if builds are produced, builds are produced but the build step has warnings and the build step is broken and no builds are produced. Thanks for an otherwise great plugin.

[kerrhome]

Hopefully, this can be configurable soon and the user/team can decide their own experience regarding color meaning. I think the plugin should not enforce a standard, unless it is a Jenkins standard. I see your points Jan, but I think allowing users to treat unstable builds, whether claimed or not, as just unstable (yellow/amber) is in keeping with how Jenkins operates today.

[or-shachar]

Is it that bad?

@gmykhailiuta - can you upload screenshots of your patch?

[kerrhome]

Looks great!

On Aug 31, 2014, at 8:29 AM, Or Shachar notifications@github.com wrote:

Is it that bad?
https://cloud.githubusercontent.com/assets/2962963/4101893/6375b8f2-310a-11e4-868f-61a0936f7029.png
@gmykhailiuta https://github.com/gmykhailiuta - can you upload screenshots of your patch?

—
Reply to this email directly or view it on GitHub #9 (comment).

[dickschoeller]

I agree! Look great to me too.

On Tue, Sep 2, 2014 at 1:57 PM, Shannon Kerr notifications@github.com
wrote:

Looks great!

On Aug 31, 2014, at 8:29 AM, Or Shachar notifications@github.com
wrote:

Is it that bad?
<
https://cloud.githubusercontent.com/assets/2962963/4101893/6375b8f2-310a-11e4-868f-61a0936f7029.png>

@gmykhailiuta https://github.com/gmykhailiuta - can you upload
screenshots of your patch?

—
Reply to this email directly or view it on GitHub <
https://github.com/jan-molak/jenkins-build-monitor-plugin/pull/9#issuecomment-53986375>.

—
Reply to this email directly or view it on GitHub
#9 (comment)
.

[gmykhailiuta]

Here is mine. Sorry, had to hide job names.

[kerrhome]

I would prefer traditional colors like these. What would you use for
aborted and never run?
On Sep 2, 2014 5:13 PM, "Gennadiy Mykhailiuta" notifications@github.com
wrote:

Here is mine. Sorry, had to hide job names.
[image: bmv]
https://cloud.githubusercontent.com/assets/1215548/4125879/e4aaa220-32e5-11e4-9337-9ec58d3258a7.png

—
Reply to this email directly or view it on GitHub
#9 (comment)
.

[gmykhailiuta]

Just standard named html gray tints:

[kerrhome]

Prefer this latest. Standard Jenkins colors. Convention.

[or-shachar]

@gmykhailiuta - Did you drop support in claimed builds?