[WIP] Dont include working specs as errors

[vsoch]

And I still need to do the reclustering. This is a WIP.

[harshithamenon]

We can go ahead with this workflow, but here are some thoughts I have and maybe rethink whether we want to cluster any error message or cluster error messages occurring in failed builds

• Many a times a build can fail due to system related issues like locks/network/filesystem errors and later succeed. How should we handle that? Should we re-cluster every time this happens?
• How do we want to handle the case where a user is interested in identifying fixes for error messages that didn't necessarily cause the build to fail and decides to submit the error message to the monitor. I think technically it is still a valid use case.

[vsoch]

Many a times a build can fail due to system related issues like locks/network/filesystem errors and later succeed. How should we handle that? Should we re-cluster every time this happens?

A spec that is failed (by spack) is going to be spent to spack monitor. The model needs to be streamling online ML so it's updated immediately.

How do we want to handle the case where a user is interested in identifying fixes for error messages that didn't necessarily cause the build to fail and decides to submit the error message to the monitor. I think technically it is still a valid use case.

If the build doesn't fail I don't think the user will fail to look into that?

[harshithamenon]

Many a times a build can fail due to system related issues like locks/network/filesystem errors and later succeed. How should we handle that? Should we re-cluster every time this happens?

A spec that is failed (by spack) is going to be spent to spack monitor. The model needs to be streamling online ML so it's updated immediately.

The situation I am describing is where it succeeds in the next attempt (or sometime in the future). Initially the spec was in errors/, but now that it succeeded do we go find the corresponding the error message from the db, remove the error message and recluster?

[vsoch]

ohh I see - so it's some kind of ephemeral error (and not related to the spec so the hash would be the same?)

I think that could be a case in and of itself - if we truly find that a previously broken spec is now magically working, we'd report back to the user that there is an exact match for a working spec (and suggest to try again, and if still broken open an issue?) You are right it's not relevant for the demo here where things are hard coded as working/error but it could come up in some real world case. I do hope we don't have too many of those because it suggests there are things we aren't accounting for. Either we could look at those cases and try to add the variables to spack, or add some extra metadata to send to spack monitor to figure out.

[vsoch]

And if we are holding a clustering, I don't think it would be feasible to manually remove/add points based on a change of state, we'd simply recluster a some frequency, and the temporal aspect of the model (e.g.,the build working from then into the future, for the most part) would eventually be dominant.