New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support of unstable and aborted build statuses #9

Closed
wants to merge 5 commits into
base: master
from

Conversation

Projects
None yet
@gmykhailiuta

gmykhailiuta commented Sep 10, 2013

With unstable and aborted statuses team can save their nerves when some test failed or somebody stopped the build.

@gmykhailiuta

This comment has been minimized.

Show comment
Hide comment
@gmykhailiuta

gmykhailiuta Sep 11, 2013

Fixed some bugs and added elapsed time in #11. Not sure if I should close this one.

gmykhailiuta commented Sep 11, 2013

Fixed some bugs and added elapsed time in #11. Not sure if I should close this one.

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Sep 13, 2013

Owner

Hi gmykhailiuta and thanks for the time you spent contributing to the project!
I'm on holidays at the moment, but I'll have a look at your pull request over the next two weeks and give you some feedback.

Owner

jan-molak commented Sep 13, 2013

Hi gmykhailiuta and thanks for the time you spent contributing to the project!
I'm on holidays at the moment, but I'll have a look at your pull request over the next two weeks and give you some feedback.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 28, 2013

I really need this functionality, how long before the code makes it into a new release?

ghost commented Oct 28, 2013

I really need this functionality, how long before the code makes it into a new release?

@gmykhailiuta

This comment has been minimized.

Show comment
Hide comment
@gmykhailiuta

gmykhailiuta Oct 29, 2013

Jan didn't implemented this functionality for some reason. Anyway, you can find it done already in my clone: https://github.com/gmykhailiuta/jenkins-build-monitor-plugin. Please, mail me if you don't want to build it, I'll provide you the built module.

gmykhailiuta commented Oct 29, 2013

Jan didn't implemented this functionality for some reason. Anyway, you can find it done already in my clone: https://github.com/gmykhailiuta/jenkins-build-monitor-plugin. Please, mail me if you don't want to build it, I'll provide you the built module.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 29, 2013

It would be a great help if you could build it for me thanks.


From: Gennadiy Mykhailiuta [mailto:notifications@github.com]
Sent: Wednesday, 30 October 2013 0:21
To: jan-molak/jenkins-build-monitor-plugin
Cc: SUMNER Andrew
Subject: Re: [jenkins-build-monitor-plugin] Added support of unstable and aborted build statuses (#9)

Jan didn't implemented this functionality for some reason. Anyway, you can find it done already in my clone: https://github.com/gmykhailiuta/jenkins-build-monitor-plugin. Please, mail me if you don't want to build it, I'll provide you the built module.

Reply to this email directly or view it on GitHubhttps://github.com/jan-molak/jenkins-build-monitor-plugin/pull/9#issuecomment-27294928.

The information contained in this email message is intended only for the addressee and is not necessarily the official view or communication of the New Zealand Customs Service.

This email may contain information that is confidential or legally privileged. If you have received it by mistake, please:
(a) reply promptly to that effect, and remove this email and the reply from your system; and
(b) do not act on this email in any other way.

ghost commented Oct 29, 2013

It would be a great help if you could build it for me thanks.


From: Gennadiy Mykhailiuta [mailto:notifications@github.com]
Sent: Wednesday, 30 October 2013 0:21
To: jan-molak/jenkins-build-monitor-plugin
Cc: SUMNER Andrew
Subject: Re: [jenkins-build-monitor-plugin] Added support of unstable and aborted build statuses (#9)

Jan didn't implemented this functionality for some reason. Anyway, you can find it done already in my clone: https://github.com/gmykhailiuta/jenkins-build-monitor-plugin. Please, mail me if you don't want to build it, I'll provide you the built module.

Reply to this email directly or view it on GitHubhttps://github.com/jan-molak/jenkins-build-monitor-plugin/pull/9#issuecomment-27294928.

The information contained in this email message is intended only for the addressee and is not necessarily the official view or communication of the New Zealand Customs Service.

This email may contain information that is confidential or legally privileged. If you have received it by mistake, please:
(a) reply promptly to that effect, and remove this email and the reply from your system; and
(b) do not act on this email in any other way.

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Oct 30, 2013

Owner

Hi Gennadiy and thanks again for your contribution.

The reason why I didn't want to distinguish between an "unstable" and "broken" build is because I consider the concept of an "unstable" build flawed and harmful. Before you accuse me of heresy please allow me to quickly explain why that is the case :)

According to Jenkins terminology "a build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable." What this means is that your build will be considered "unstable" should any of your tests fail...

Now, if we'd like to distinguish between a "broken" and an "unstable" state this implies that we're also happy to accept the fact that some of our tests might fail from time to time. And here's where I see the problem - if you have tests that you know are failing, is there anything you can do about them rather than accepting that your project is "unstable"?

On several occasions I've worked with teams that started with accepting say 5% of their tests failing as "normal and to be expected" (justifying this state of things by saying that "those 5% are caused by a flaky browser-based test, and hey, 95% percent of our tests is still passing, right?").

Over time, however, 5% became 10%, then 15% then 50%, eventually ignoring test failures got them to a state where they started questioning automated testing all together! (consider the Broken Windows Theory).

My suggestion is to avoid using the very concept of "unstable builds" and react to build failures by either:

  • fixing the offending test
  • stabilising a test or moving it to a different test suite (i.e. do you need to use a UI-based test where a lower-level test would suffice? See Martin Fowler's Test Pyramid)
  • if the failure is caused by one of Jenkins' publishers, could you use a more stable one to achieve the same result?

I believe that the primary task of Build Monitor is to alert the team should anything be wrong with their project. Giving them a false sense of security is not compatible with this goal I'm afraid.

Having said all this, the team should also be made aware of anyone trying to fix a broken build (that's especially useful for distributed teams), that's why I'm currently working on adding support for the Jenkins Claim Plugin (See #19). Hopefully, making a build amber when someone is actually actively working on fixing it should help your team save their nerves :)

What do you think?

Owner

jan-molak commented Oct 30, 2013

Hi Gennadiy and thanks again for your contribution.

The reason why I didn't want to distinguish between an "unstable" and "broken" build is because I consider the concept of an "unstable" build flawed and harmful. Before you accuse me of heresy please allow me to quickly explain why that is the case :)

According to Jenkins terminology "a build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable." What this means is that your build will be considered "unstable" should any of your tests fail...

Now, if we'd like to distinguish between a "broken" and an "unstable" state this implies that we're also happy to accept the fact that some of our tests might fail from time to time. And here's where I see the problem - if you have tests that you know are failing, is there anything you can do about them rather than accepting that your project is "unstable"?

On several occasions I've worked with teams that started with accepting say 5% of their tests failing as "normal and to be expected" (justifying this state of things by saying that "those 5% are caused by a flaky browser-based test, and hey, 95% percent of our tests is still passing, right?").

Over time, however, 5% became 10%, then 15% then 50%, eventually ignoring test failures got them to a state where they started questioning automated testing all together! (consider the Broken Windows Theory).

My suggestion is to avoid using the very concept of "unstable builds" and react to build failures by either:

  • fixing the offending test
  • stabilising a test or moving it to a different test suite (i.e. do you need to use a UI-based test where a lower-level test would suffice? See Martin Fowler's Test Pyramid)
  • if the failure is caused by one of Jenkins' publishers, could you use a more stable one to achieve the same result?

I believe that the primary task of Build Monitor is to alert the team should anything be wrong with their project. Giving them a false sense of security is not compatible with this goal I'm afraid.

Having said all this, the team should also be made aware of anyone trying to fix a broken build (that's especially useful for distributed teams), that's why I'm currently working on adding support for the Jenkins Claim Plugin (See #19). Hopefully, making a build amber when someone is actually actively working on fixing it should help your team save their nerves :)

What do you think?

@gmykhailiuta

This comment has been minimized.

Show comment
Hide comment
@gmykhailiuta

gmykhailiuta Oct 30, 2013

Wow! It could be a separate article. I think it make sense and it's an interesting point that I can present to my team too. Maybe it's not widespread approach, but we have jobs which deploy application and run smoke test on it. So, with "unstable" status it is easier to distinguish failed deploy (failed status) from "deployment succeeded, but some tests failed" (unstable status). The first is most likely devops' responsibility, but the second is developers' one. Also, you can configure job not to use unstable status and mark it as failed if it's unstable. Anyway, the best solution might be giving the users an option to choose, enable unstable status or not in the plugin's configuration.

gmykhailiuta commented Oct 30, 2013

Wow! It could be a separate article. I think it make sense and it's an interesting point that I can present to my team too. Maybe it's not widespread approach, but we have jobs which deploy application and run smoke test on it. So, with "unstable" status it is easier to distinguish failed deploy (failed status) from "deployment succeeded, but some tests failed" (unstable status). The first is most likely devops' responsibility, but the second is developers' one. Also, you can configure job not to use unstable status and mark it as failed if it's unstable. Anyway, the best solution might be giving the users an option to choose, enable unstable status or not in the plugin's configuration.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 30, 2013

I agree that failing tests should fail the build - that's exactly what ours do. My need for the distinction is around code analysis. Eventually I would like to get to zero tolerance but at the moment I would like to mark builds as success when zero check style faults, unstable if under a threshold, and failed if over threshold. That way developers can also distinguish between a build that failed to compile and needs immediate attention or one that just needs a bit of tidying up.

ghost commented Oct 30, 2013

I agree that failing tests should fail the build - that's exactly what ours do. My need for the distinction is around code analysis. Eventually I would like to get to zero tolerance but at the moment I would like to mark builds as success when zero check style faults, unstable if under a threshold, and failed if over threshold. That way developers can also distinguish between a build that failed to compile and needs immediate attention or one that just needs a bit of tidying up.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 30, 2013

I'm trying to build the plugin and get the following error, any ideas?

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default) on project build-monitor-plugin: Command execution failed. Cannot run program "npm" (in directory "C:\Work_Java\eclipse_workspace\jenkins-build-monitor-plugin-master"): CreateProcess error=2, The system cannot find the file specified -> [Help 1]

ghost commented Oct 30, 2013

I'm trying to build the plugin and get the following error, any ideas?

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default) on project build-monitor-plugin: Command execution failed. Cannot run program "npm" (in directory "C:\Work_Java\eclipse_workspace\jenkins-build-monitor-plugin-master"): CreateProcess error=2, The system cannot find the file specified -> [Help 1]

@gmykhailiuta

This comment has been minimized.

Show comment
Hide comment
@gmykhailiuta

gmykhailiuta Oct 30, 2013

You need node.js and npm (node.js package manager) to build plugin. See http://nodejs.org/download/

gmykhailiuta commented Oct 30, 2013

You need node.js and npm (node.js package manager) to build plugin. See http://nodejs.org/download/

jan-molak added a commit that referenced this pull request Nov 10, 2013

Feature: Build Monitor supports Jenkins Claim Plugin (https://wiki.je…
…nkins-ci.org/display/JENKINS/Claim+plugin)

This closes #19 and can address #9, provided that "unstable" and "aborted" builds are claimed.
@fonsecas72

This comment has been minimized.

Show comment
Hide comment
@fonsecas72

fonsecas72 Nov 21, 2013

Jenkins builds not always have tests in them. Sometimes they may refer to static analysis results so I may want to distinguish from a bad build that exceeded threshold by 20 from another very bad build that exceeded threshold by 50.
For my company, this functionality is the only thing missing for us to use it. It would be awesome if this was added even if optionally, as this plugin seams to be much attractive then the one we use know. Please please add this functionality.

fonsecas72 commented Nov 21, 2013

Jenkins builds not always have tests in them. Sometimes they may refer to static analysis results so I may want to distinguish from a bad build that exceeded threshold by 20 from another very bad build that exceeded threshold by 50.
For my company, this functionality is the only thing missing for us to use it. It would be awesome if this was added even if optionally, as this plugin seams to be much attractive then the one we use know. Please please add this functionality.

@wilriker

This comment has been minimized.

Show comment
Hide comment
@wilriker

wilriker Nov 22, 2013

I would also love to see at least an option to have different colors for all the different Result possibilities (ABORTED, FAILURE, NOT_BUILT, SUCCESS and UNSTABLE). I see the point in your argumentation, Jan, but we also do distinguish between compile/build errors (FAILURE) and tests/post-build-trigger failing (UNSTABLE). Also we add some new CI jobs once in a while and I am always asked why this is failed even though it is just NOT_BUILT instead (and believe me it has always been recognized before Jenkins had a free executor ;-) ).

So please at least add it as an opt-in option.

wilriker commented Nov 22, 2013

I would also love to see at least an option to have different colors for all the different Result possibilities (ABORTED, FAILURE, NOT_BUILT, SUCCESS and UNSTABLE). I see the point in your argumentation, Jan, but we also do distinguish between compile/build errors (FAILURE) and tests/post-build-trigger failing (UNSTABLE). Also we add some new CI jobs once in a while and I am always asked why this is failed even though it is just NOT_BUILT instead (and believe me it has always been recognized before Jenkins had a free executor ;-) ).

So please at least add it as an opt-in option.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Nov 24, 2013

An opt in option for unstable builds does seem a good compromise :-)

ghost commented Nov 24, 2013

An opt in option for unstable builds does seem a good compromise :-)

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Nov 24, 2013

Owner

Hi guys and thanks for all your feedback - it will definitely help to improve the Build Monitor even further :)

My main focus is to ensure that the information radiated by the monitor is clear and simple to understand.
In order to achieve this goal, the monitor has to make it extremely easy to find out at a first glance whether a given Jenkins job is:

  • working as expected - green
  • broken, but under control (failing, unstable or aborted, but claimed) - amber
  • broken, and no one is looking at it (failing, unstable or aborted, but not claimed) - red

If you considered all the states a Jenkins job can be in, you'd end up with the following list:

  • SUCCESS
  • UNSTABLE
  • FAILURE
  • NOT_BUILT
  • ABORTED

If we wanted to have a separate colour for any of those states, we'd end up with a bit of a rainbow and it seems that this defeats the main objective. (Hey, did purple mean unstable? No, I think that was the deep blue that meant that... or The build is red! No it's not, it's PINK, it means it's ABORTED not FAILING ... ;-) )

I see your point of view, @hugofonseca, and I understand that Jenkins jobs don't necessarily need to have tests in them, and as you mentioned can be used for static code analysis for example.
Even in this case you might easily end up with a rotting build scenario I described earlier if you don't address problems as soon as they arise... That's precisely why I'm reluctant to encourage dev teams using this humble plugin to start ignoring builds that are failing tests they wrote and exceeding thresholds they set.

What's your opinion? Would you treat a build that exceeded the threshold by 20 any different from the one that exceeded it by 50?

I'd apply similar logic to post-build tests and triggers. Would you ever want to leave a job in an UNSTABLE state?
Also, what does it mean that a build is unstable? Sometimes it works, sometimes it doesn't? It mostly works? Can an unstable application be seen by your customers? Is it OK to be unstable? What's your view?

@wilriker - To help you answer the question you're being asked: why this is failed even though it is just NOT_BUILT - in test-driven development (TDD) you start with writing a test before writing the class you're testing. What happens when you run your test before implementing the class? :)
But more seriously: How long do you usually have your newly created jobs in a NOT_BUILT state? Do you think it would be beneficial to have NOT_BUILT state distinguished on the same level as SUCCESSFUL, FAILING and CLAIMED?

Looking forward to hearing back from you and thanks again for helping me make this plugin better!

Owner

jan-molak commented Nov 24, 2013

Hi guys and thanks for all your feedback - it will definitely help to improve the Build Monitor even further :)

My main focus is to ensure that the information radiated by the monitor is clear and simple to understand.
In order to achieve this goal, the monitor has to make it extremely easy to find out at a first glance whether a given Jenkins job is:

  • working as expected - green
  • broken, but under control (failing, unstable or aborted, but claimed) - amber
  • broken, and no one is looking at it (failing, unstable or aborted, but not claimed) - red

If you considered all the states a Jenkins job can be in, you'd end up with the following list:

  • SUCCESS
  • UNSTABLE
  • FAILURE
  • NOT_BUILT
  • ABORTED

If we wanted to have a separate colour for any of those states, we'd end up with a bit of a rainbow and it seems that this defeats the main objective. (Hey, did purple mean unstable? No, I think that was the deep blue that meant that... or The build is red! No it's not, it's PINK, it means it's ABORTED not FAILING ... ;-) )

I see your point of view, @hugofonseca, and I understand that Jenkins jobs don't necessarily need to have tests in them, and as you mentioned can be used for static code analysis for example.
Even in this case you might easily end up with a rotting build scenario I described earlier if you don't address problems as soon as they arise... That's precisely why I'm reluctant to encourage dev teams using this humble plugin to start ignoring builds that are failing tests they wrote and exceeding thresholds they set.

What's your opinion? Would you treat a build that exceeded the threshold by 20 any different from the one that exceeded it by 50?

I'd apply similar logic to post-build tests and triggers. Would you ever want to leave a job in an UNSTABLE state?
Also, what does it mean that a build is unstable? Sometimes it works, sometimes it doesn't? It mostly works? Can an unstable application be seen by your customers? Is it OK to be unstable? What's your view?

@wilriker - To help you answer the question you're being asked: why this is failed even though it is just NOT_BUILT - in test-driven development (TDD) you start with writing a test before writing the class you're testing. What happens when you run your test before implementing the class? :)
But more seriously: How long do you usually have your newly created jobs in a NOT_BUILT state? Do you think it would be beneficial to have NOT_BUILT state distinguished on the same level as SUCCESSFUL, FAILING and CLAIMED?

Looking forward to hearing back from you and thanks again for helping me make this plugin better!

@trejkaz

This comment has been minimized.

Show comment
Hide comment
@trejkaz

trejkaz Nov 25, 2013

The task has a haltonfailure property which lets you configure the build such that any failing test results in a failing build.

If you really want all unstable builds to be marked as failed, why don't you just set that flag in your build, rather than forcing your beliefs on everyone else? Software development isn't supposed to be a religion.

trejkaz commented Nov 25, 2013

The task has a haltonfailure property which lets you configure the build such that any failing test results in a failing build.

If you really want all unstable builds to be marked as failed, why don't you just set that flag in your build, rather than forcing your beliefs on everyone else? Software development isn't supposed to be a religion.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Nov 25, 2013

I think you have some very valid points @jan-molak, and I agree that clarity is the goal here. I could see that I might eventually want SUCCESS, FAILING, CLAIMED, UNSTABLE, ABORTED, and DISABLED. I don't necessarily agree that we'd end up with with an unclear monitor as the goal is to keep things green.

If you choose not to implement them I'm not likely to use another monitor tool anyway as this is the best I've come across so far. As much as I'd like the flexibility to use these colours if I need them, I'd prefer sound alerts :-)

ghost commented Nov 25, 2013

I think you have some very valid points @jan-molak, and I agree that clarity is the goal here. I could see that I might eventually want SUCCESS, FAILING, CLAIMED, UNSTABLE, ABORTED, and DISABLED. I don't necessarily agree that we'd end up with with an unclear monitor as the goal is to keep things green.

If you choose not to implement them I'm not likely to use another monitor tool anyway as this is the best I've come across so far. As much as I'd like the flexibility to use these colours if I need them, I'd prefer sound alerts :-)

@wilriker

This comment has been minimized.

Show comment
Hide comment
@wilriker

wilriker Nov 25, 2013

I agree with you @jan-molak about having a preselected individual color for every possible state could be confusing. As an example our settings would probably look like the following:
SUCCESS: green
FAILING: red
everything else: amber (maybe explicitly NOT_BUILT: grey, not sure about this though)

To explain this shortly: you also asked whether we would treat UNSTABLE and FAILING builds differently. Yes, we do - but only in priority. A FAILING build needs immediate attention (because this might block other team members from continuing with their work), whereas an UNSTABLE build usually means that retriggering the build will solve it because local Artifactory was overloaded or some of our not so stable tests was failing. Of course there needs to be a solution for the cause and not the symptoms and we are working an that but it takes time. Also we don't use the Claim Plugin because we have another way of communicating such things.

That will obviously not fit everyone's needs and workflows. Therefore my proposal here would be to have the possibility to select the colors for every state. This might either be from a predefined list (which would probably make implementation for you a lot more easy) or even a color-chooser. This way at least everyone could adapt the Build Monitor to their individual workflow.

And about the NOT_BUILT state: for example we do our production deployments using a Jenkins instance as the UI for a command-line tool. If a "build" fails there this usually means at least one production server is down and this requires immediate attention. But on the other hand it could be that we already add a job for a new cluster which is currently being set up on Wednesday and the next regular deployment would be the following Tuesday for example. That means we have one deployment job for nearly one week which is red and everyone gets shocked when they enter the room although there is no problem at all because is has just not yet being started at all.

As you can see this plugin is being used for a wide variety of use cases and not all of them fit into what you intended. No offence, as I already said you have some valid points but you should at least give users the opportunity to configure differently.

wilriker commented Nov 25, 2013

I agree with you @jan-molak about having a preselected individual color for every possible state could be confusing. As an example our settings would probably look like the following:
SUCCESS: green
FAILING: red
everything else: amber (maybe explicitly NOT_BUILT: grey, not sure about this though)

To explain this shortly: you also asked whether we would treat UNSTABLE and FAILING builds differently. Yes, we do - but only in priority. A FAILING build needs immediate attention (because this might block other team members from continuing with their work), whereas an UNSTABLE build usually means that retriggering the build will solve it because local Artifactory was overloaded or some of our not so stable tests was failing. Of course there needs to be a solution for the cause and not the symptoms and we are working an that but it takes time. Also we don't use the Claim Plugin because we have another way of communicating such things.

That will obviously not fit everyone's needs and workflows. Therefore my proposal here would be to have the possibility to select the colors for every state. This might either be from a predefined list (which would probably make implementation for you a lot more easy) or even a color-chooser. This way at least everyone could adapt the Build Monitor to their individual workflow.

And about the NOT_BUILT state: for example we do our production deployments using a Jenkins instance as the UI for a command-line tool. If a "build" fails there this usually means at least one production server is down and this requires immediate attention. But on the other hand it could be that we already add a job for a new cluster which is currently being set up on Wednesday and the next regular deployment would be the following Tuesday for example. That means we have one deployment job for nearly one week which is red and everyone gets shocked when they enter the room although there is no problem at all because is has just not yet being started at all.

As you can see this plugin is being used for a wide variety of use cases and not all of them fit into what you intended. No offence, as I already said you have some valid points but you should at least give users the opportunity to configure differently.

@fonsecas72

This comment has been minimized.

Show comment
Hide comment
@fonsecas72

fonsecas72 Nov 25, 2013

I agree completely with you @wilriker! At where I work we also look at failing builds as you, so every red build WILL take our attention immediately.
We use unstable for example for things that QA team will fix or report to PM. This means that no one else should interrupt their work for yellow builds.
About grey builds: We just use them for disabled builds just to remind us that those builds are under construction, meaning that work is still to be done but not right now.
I also agree with @jan-molak point of view but not everyone of us use his methodology and being so we should be free to choose the best one to use.
Please :)

fonsecas72 commented Nov 25, 2013

I agree completely with you @wilriker! At where I work we also look at failing builds as you, so every red build WILL take our attention immediately.
We use unstable for example for things that QA team will fix or report to PM. This means that no one else should interrupt their work for yellow builds.
About grey builds: We just use them for disabled builds just to remind us that those builds are under construction, meaning that work is still to be done but not right now.
I also agree with @jan-molak point of view but not everyone of us use his methodology and being so we should be free to choose the best one to use.
Please :)

@dickschoeller

This comment has been minimized.

Show comment
Hide comment
@dickschoeller

dickschoeller Nov 26, 2013

We run into an area where this is an issue. The HP Application Automation Tools plugin to report on QTP runs. The QTP runs always generate a warning because of the method that is used to find certain elements on the page. There is no way to tell that plugin not to mark those runs unstable. If this were distinguished from actual failures of the tests we would benefit.

One could make the argument that we should change the way the test scripts find page elements. Our UI test automation lead is working on that.

One could also make the argument that the HP Application Automation Tools plugin should have control over the thresholds for stable/unstable and success/failure as do other plugins. I plan to request that too.

dickschoeller commented Nov 26, 2013

We run into an area where this is an issue. The HP Application Automation Tools plugin to report on QTP runs. The QTP runs always generate a warning because of the method that is used to find certain elements on the page. There is no way to tell that plugin not to mark those runs unstable. If this were distinguished from actual failures of the tests we would benefit.

One could make the argument that we should change the way the test scripts find page elements. Our UI test automation lead is working on that.

One could also make the argument that the HP Application Automation Tools plugin should have control over the thresholds for stable/unstable and success/failure as do other plugins. I plan to request that too.

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Jan 5, 2014

Owner

Thanks for all the feedback guys, it's really useful :) I'll most likely make this a configurable option at some point.

Owner

jan-molak commented Jan 5, 2014

Thanks for all the feedback guys, it's really useful :) I'll most likely make this a configurable option at some point.

@drekbour

This comment has been minimized.

Show comment
Hide comment
@drekbour

drekbour Jan 8, 2014

Contributor

Ditto. Test fail = Unstable doesn't feel right to me either but the generic concept of Unstable/Failed does have value. I'd vote for a BuildMonitor that reflects unstable in yellow and has a boolean setting "Show Unstable builds as Failed"

FWIW Bamboo has a Quarantine option for failing tests. Users can manually Quarantine a failed test which will no longer fail a build (until passed again). There is much discussion about whether this is sensible or not :)

Contributor

drekbour commented Jan 8, 2014

Ditto. Test fail = Unstable doesn't feel right to me either but the generic concept of Unstable/Failed does have value. I'd vote for a BuildMonitor that reflects unstable in yellow and has a boolean setting "Show Unstable builds as Failed"

FWIW Bamboo has a Quarantine option for failing tests. Users can manually Quarantine a failed test which will no longer fail a build (until passed again). There is much discussion about whether this is sensible or not :)

@gstamac

This comment has been minimized.

Show comment
Hide comment
@gstamac

gstamac Feb 21, 2014

This is an excellent idea. With Warnings plugin you can have unstable build if compiler generated warnings. This build is not broken. I do agree that one failing test means the build is broken but you have other settings to force this policy. Please include this pull request. It would make Build Monitor much better.

gstamac commented Feb 21, 2014

This is an excellent idea. With Warnings plugin you can have unstable build if compiler generated warnings. This build is not broken. I do agree that one failing test means the build is broken but you have other settings to force this policy. Please include this pull request. It would make Build Monitor much better.

@Mostyn

This comment has been minimized.

Show comment
Hide comment
@Mostyn

Mostyn Feb 28, 2014

Great work Jan. We are using the Build Monitor on two screens in our office. One has an overview of more than 64 jobs whilst the other is more project focused with only twelve jobs displayed.

Whilst I appreciate your thinking about your own requirements and judgements as to what is green and what is red for your purposes there are other teams that have broader range of severity.

The Jenkins system has a status for each job and a coloured sphere to denote this. As far as I am aware, on our system, there are only four coloured statuses: GREEN (success); RED (failed); AMBER (unstable) and GREY (Disabled, Aborted, Not built...)

We would be very happy if the Build Monitor simply used those four colours that Jenkins already uses. We would also be happy if it was optional to have the binary (red/green) or the standard Jenkins status colours (red, amber, green, grey). Can you possible release a feature that does this soon?

Of course anything other than Green would have cause to be evaluated. The amber, red and grey colours would indicate some kind of severity. For us a Build failing to install is more severe (red) than any unstable job (amber) which could be unit test failing or, often for us, a warning during parsing of the console log.

Looking forward to any further updates.

Mostyn commented Feb 28, 2014

Great work Jan. We are using the Build Monitor on two screens in our office. One has an overview of more than 64 jobs whilst the other is more project focused with only twelve jobs displayed.

Whilst I appreciate your thinking about your own requirements and judgements as to what is green and what is red for your purposes there are other teams that have broader range of severity.

The Jenkins system has a status for each job and a coloured sphere to denote this. As far as I am aware, on our system, there are only four coloured statuses: GREEN (success); RED (failed); AMBER (unstable) and GREY (Disabled, Aborted, Not built...)

We would be very happy if the Build Monitor simply used those four colours that Jenkins already uses. We would also be happy if it was optional to have the binary (red/green) or the standard Jenkins status colours (red, amber, green, grey). Can you possible release a feature that does this soon?

Of course anything other than Green would have cause to be evaluated. The amber, red and grey colours would indicate some kind of severity. For us a Build failing to install is more severe (red) than any unstable job (amber) which could be unit test failing or, often for us, a warning during parsing of the console log.

Looking forward to any further updates.

@weissmag

This comment has been minimized.

Show comment
Hide comment
@weissmag

weissmag Mar 4, 2014

Just for the sake of voting, We would also appreciate using the full color range by Jenkins, as for us yellow is a test issue and red is a deployment issue... and each is addressed by separate teams...

You can default the configuration to binary and let us choose to remove it.

in any case Great Job !

weissmag commented Mar 4, 2014

Just for the sake of voting, We would also appreciate using the full color range by Jenkins, as for us yellow is a test issue and red is a deployment issue... and each is addressed by separate teams...

You can default the configuration to binary and let us choose to remove it.

in any case Great Job !

@stephane-benoist

This comment has been minimized.

Show comment
Hide comment
@stephane-benoist

stephane-benoist Mar 26, 2014

Isn't this a misuse of "unstable".
In my mind, "unstable" has to be used for abnormal metrics (e.g. high amount of duplicate code, plenty of cs violations...)
In this case, I think it's logical to allows to distinguish both.

stephane-benoist commented Mar 26, 2014

Isn't this a misuse of "unstable".
In my mind, "unstable" has to be used for abnormal metrics (e.g. high amount of duplicate code, plenty of cs violations...)
In this case, I think it's logical to allows to distinguish both.

@qse

This comment has been minimized.

Show comment
Hide comment
@qse

qse Jun 24, 2014

Hi there,
for us it would be very great to have the 4 different colors (green, amber, red and grey) of Jenkins, because in my team (QA team) we use our wallboards (information radiator) to monitor automated tests (ci). And we differentiate the job based on the number of failed tests.
In the standup we use the wallboards to get an impression over the tests and if most of the tests are red (even if there are only 0.001% test steps failed) there is confusion.
Over that, if the team only reacts based on a color of a information radiator ... there are other issues in this team. ;-)
Anyway, I love the idea of having the option (opt-in) to use the full color range. :-)

qse commented Jun 24, 2014

Hi there,
for us it would be very great to have the 4 different colors (green, amber, red and grey) of Jenkins, because in my team (QA team) we use our wallboards (information radiator) to monitor automated tests (ci). And we differentiate the job based on the number of failed tests.
In the standup we use the wallboards to get an impression over the tests and if most of the tests are red (even if there are only 0.001% test steps failed) there is confusion.
Over that, if the team only reacts based on a color of a information radiator ... there are other issues in this team. ;-)
Anyway, I love the idea of having the option (opt-in) to use the full color range. :-)

@ilor

This comment has been minimized.

Show comment
Hide comment
@ilor

ilor Jul 10, 2014

++, I could really use this or an equivalent feature. More options than just green/red, please 👍

ilor commented Jul 10, 2014

++, I could really use this or an equivalent feature. More options than just green/red, please 👍

@gusran

This comment has been minimized.

Show comment
Hide comment
@gusran

gusran Aug 13, 2014

We need this. We do not fail the build on code analysis warnings and we want to distinguish between if builds are produced, builds are produced but the build step has warnings and the build step is broken and no builds are produced. Thanks for an otherwise great plugin.

gusran commented Aug 13, 2014

We need this. We do not fail the build on code analysis warnings and we want to distinguish between if builds are produced, builds are produced but the build step has warnings and the build step is broken and no builds are produced. Thanks for an otherwise great plugin.

@jan-molak jan-molak added the backlog label Aug 13, 2014

@kerrhome

This comment has been minimized.

Show comment
Hide comment
@kerrhome

kerrhome Aug 15, 2014

Hopefully, this can be configurable soon and the user/team can decide their own experience regarding color meaning. I think the plugin should not enforce a standard, unless it is a Jenkins standard. I see your points Jan, but I think allowing users to treat unstable builds, whether claimed or not, as just unstable (yellow/amber) is in keeping with how Jenkins operates today.

kerrhome commented Aug 15, 2014

Hopefully, this can be configurable soon and the user/team can decide their own experience regarding color meaning. I think the plugin should not enforce a standard, unless it is a Jenkins standard. I see your points Jan, but I think allowing users to treat unstable builds, whether claimed or not, as just unstable (yellow/amber) is in keeping with how Jenkins operates today.

@or-shachar

This comment has been minimized.

Show comment
Hide comment
@or-shachar

or-shachar Aug 31, 2014

Is it that bad?
31-08-2014 14-21-00

@gmykhailiuta - can you upload screenshots of your patch?

or-shachar commented Aug 31, 2014

Is it that bad?
31-08-2014 14-21-00

@gmykhailiuta - can you upload screenshots of your patch?

@kerrhome

This comment has been minimized.

Show comment
Hide comment
@kerrhome

kerrhome Sep 2, 2014

Looks great!

On Aug 31, 2014, at 8:29 AM, Or Shachar notifications@github.com wrote:

Is it that bad?
https://cloud.githubusercontent.com/assets/2962963/4101893/6375b8f2-310a-11e4-868f-61a0936f7029.png
@gmykhailiuta https://github.com/gmykhailiuta - can you upload screenshots of your patch?


Reply to this email directly or view it on GitHub #9 (comment).

kerrhome commented Sep 2, 2014

Looks great!

On Aug 31, 2014, at 8:29 AM, Or Shachar notifications@github.com wrote:

Is it that bad?
https://cloud.githubusercontent.com/assets/2962963/4101893/6375b8f2-310a-11e4-868f-61a0936f7029.png
@gmykhailiuta https://github.com/gmykhailiuta - can you upload screenshots of your patch?


Reply to this email directly or view it on GitHub #9 (comment).

@dickschoeller

This comment has been minimized.

Show comment
Hide comment
@dickschoeller

dickschoeller Sep 2, 2014

I agree! Look great to me too.

On Tue, Sep 2, 2014 at 1:57 PM, Shannon Kerr notifications@github.com
wrote:

Looks great!

On Aug 31, 2014, at 8:29 AM, Or Shachar notifications@github.com
wrote:

Is it that bad?
<
https://cloud.githubusercontent.com/assets/2962963/4101893/6375b8f2-310a-11e4-868f-61a0936f7029.png>

@gmykhailiuta https://github.com/gmykhailiuta - can you upload
screenshots of your patch?


Reply to this email directly or view it on GitHub <
https://github.com/jan-molak/jenkins-build-monitor-plugin/pull/9#issuecomment-53986375>.


Reply to this email directly or view it on GitHub
#9 (comment)
.

dickschoeller commented Sep 2, 2014

I agree! Look great to me too.

On Tue, Sep 2, 2014 at 1:57 PM, Shannon Kerr notifications@github.com
wrote:

Looks great!

On Aug 31, 2014, at 8:29 AM, Or Shachar notifications@github.com
wrote:

Is it that bad?
<
https://cloud.githubusercontent.com/assets/2962963/4101893/6375b8f2-310a-11e4-868f-61a0936f7029.png>

@gmykhailiuta https://github.com/gmykhailiuta - can you upload
screenshots of your patch?


Reply to this email directly or view it on GitHub <
https://github.com/jan-molak/jenkins-build-monitor-plugin/pull/9#issuecomment-53986375>.


Reply to this email directly or view it on GitHub
#9 (comment)
.

@gmykhailiuta

This comment has been minimized.

Show comment
Hide comment
@gmykhailiuta

gmykhailiuta Sep 2, 2014

Here is mine. Sorry, had to hide job names.
bmv

gmykhailiuta commented Sep 2, 2014

Here is mine. Sorry, had to hide job names.
bmv

@kerrhome

This comment has been minimized.

Show comment
Hide comment
@kerrhome

kerrhome Sep 2, 2014

I would prefer traditional colors like these. What would you use for
aborted and never run?
On Sep 2, 2014 5:13 PM, "Gennadiy Mykhailiuta" notifications@github.com
wrote:

Here is mine. Sorry, had to hide job names.
[image: bmv]
https://cloud.githubusercontent.com/assets/1215548/4125879/e4aaa220-32e5-11e4-9337-9ec58d3258a7.png


Reply to this email directly or view it on GitHub
#9 (comment)
.

kerrhome commented Sep 2, 2014

I would prefer traditional colors like these. What would you use for
aborted and never run?
On Sep 2, 2014 5:13 PM, "Gennadiy Mykhailiuta" notifications@github.com
wrote:

Here is mine. Sorry, had to hide job names.
[image: bmv]
https://cloud.githubusercontent.com/assets/1215548/4125879/e4aaa220-32e5-11e4-9337-9ec58d3258a7.png


Reply to this email directly or view it on GitHub
#9 (comment)
.

@gmykhailiuta

This comment has been minimized.

Show comment
Hide comment
@gmykhailiuta

gmykhailiuta Sep 2, 2014

Just standard named html gray tints:
snapshot2

gmykhailiuta commented Sep 2, 2014

Just standard named html gray tints:
snapshot2

@gmykhailiuta gmykhailiuta reopened this Sep 2, 2014

@kerrhome

This comment has been minimized.

Show comment
Hide comment
@kerrhome

kerrhome Sep 2, 2014

Prefer this latest. Standard Jenkins colors. Convention.

kerrhome commented Sep 2, 2014

Prefer this latest. Standard Jenkins colors. Convention.

@or-shachar

This comment has been minimized.

Show comment
Hide comment
@or-shachar

or-shachar Sep 3, 2014

@gmykhailiuta - Did you drop support in claimed builds?

or-shachar commented Sep 3, 2014

@gmykhailiuta - Did you drop support in claimed builds?

@jan-molak jan-molak added in progress and removed backlog labels Sep 14, 2014

@jan-molak jan-molak closed this in 35c6ab9 Sep 14, 2014

@jan-molak jan-molak removed the in progress label Sep 14, 2014

@jan-molak jan-molak referenced this pull request Oct 1, 2014

Closed

Colorblind Palette? #30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment