[ADR] Define quality levels #219
Conversation
|
Thank you for this initiative. What is not clear to me is: is this aspirational, or descriptive? What I am advocating for is something that is descriptive of the actual state of James, so people new to James know exactly what to expect (or not), both from the code as well as from the community. |
It is descriptive. Note that some features of supported products are however "experimental" or "unsupported", leading to an unclear overview as a user. An example would be fetchmail usage. I believe that a better extension packaging would allow to take these experimental/unsupported features out of the default supported products, and would better educate our operators about what is supported or not, and the underlying risks.
Note that as a consequence, we should open JIRA in order to maintain a clear description of James current status, and encourage contributions on experimental / unsupported components. This ADR serve as a basis to build this exact diagnostic. Does it makes sense? |
|
Thank you. That clarifies a few points, but muddies a few others. Notably, you use the word "supported" in your description, but in the mail discussion there was a fierce reaction to any notion of "support". From what I understood from the mail thread and if this is indeed descriptive and not aspirational, everything should be labeled as "unsupported", which kind of makes this whole initiative moot. At least I am quite confused. But then again, producing this ADR is a very good idea because hopefully it will clarify all the confusion. :) |
Good point. Then let's change that terms as it might bring unreasonable expectations from operators.
Let's refine our vocabulary to make this cristal clear. Here are some proposals coming to my mind for a better wording:
|
There was a problem hiding this comment.
Good initiative, thanks
By the way, in the mail there wasa link to https://github.com/chibenwa/james-project/blob/v3.0_roadmap/v3.0_roadmap.adoc defining what is supported / experimental / unsupported. But it seems quite outdated.
Do we have a more up to date version of this somewhere or is it something that needs to be done too?
We release 3.0.0, so the related roadmap had been carried out. |
Co-authored-by: Matthieu Baechler <matthieu.baechler@gmail.com>
|
I removed the only "code" mention. |
| understand the underlying quality of the artifact they use, as well as the level of risk associated, | ||
| we need to better define some quality levels. |
There was a problem hiding this comment.
The document defines "quality levels" without defining "quality".
I think that "level" is obvious enough that it doesn't require any further definition, but "quality" may be interpreted differently by different people.
There was a problem hiding this comment.
The document defines "quality levels" without defining "quality".
At one point it becomes hard for me to better defines such a generic term. I believes the levels definition disambiguate clearly this.
There was a problem hiding this comment.
Writing is really hard, I agree. The hardest part is writing something "objective" that everybody understands the same way, especially in a project like this with people from different countries, backgrounds, cultures, education, generations... We'll never have a completely common understanding, so the trick is to decide the right point at which we should draw a boundary given the time and resources we have available. We will never completely agree on where that boundary should be. :)
There was a problem hiding this comment.
Actually this document, and the "mature" quality level dis-ambiguify what we mean by quality level, no?
There was a problem hiding this comment.
Are you saying that "mature" implies "high quality"? In the definitions I proposed, a component could be mature, but unreliable.
I do agree that it's not easy figuring out how all the pieces fit together. Quality really depends on the objective. There is not "absolute". That is why context is so important.
|
|
||
| For a given artifact or feature, by **mature** we mean that: | ||
|
|
||
| - *interfaces* in components need a contract test suite |
There was a problem hiding this comment.
Would it be useful to describe "separation of API/implementation"? It is a term I have used a few times on the list, but I'm not sure I use it in the same way as others. (I have strongly corrupted by OSGi thinking, and it has a very clear meaning over there.)
"Interface" as you use it seems to suggest at the class level, whereas the real "interface" of a component is actualy at the package level, which may be why they call it "API" instead of "interface", to avoid confusion. Usually it is not just a single Java interface that defines behaviour, but rather an entire package. (Should generally not expand beyond a package.)
There was a problem hiding this comment.
Would it be useful to describe "separation of API/implementation"?
👍 and luckily it is the case almost everywhere.
"Interface" as you use it seems to suggest at the class level, whereas the real "interface" of a component is actualy at the package level, which may be why they call it "API" instead of "interface", to avoid confusion. Usually it is not just a single Java interface that defines behaviour, but rather an entire package
If others don't complain, I will replace "interfaces" terms by "APIs".
| - *implementation* of these interfaces need to pass this contract test suite which provides unit tests | ||
| - Decent integration tests coverage is needed | ||
| - Performance tests need to be conducted out | ||
| - Quality Assurance with external clients needs to be conducted out |
There was a problem hiding this comment.
What do you mean by "Quality Assurance"? Is that defined in a different ADR?
There was a problem hiding this comment.
I mean "Manual Testing". Quality Assurance is in my opinion a common practice of the industry. Defining it is in my opinion not rely needed.
There was a problem hiding this comment.
You may be surprised as to how something that ought to be clearly defined in the industry creates a lot of conflict. :)
If there is already a clear de facto modus operandi for the James community, or if nobody challenges it, then I guess it's fine. :)
| ## Context | ||
|
|
||
| We hereby define as an artifact compiled artifact that external people consumes. This includes: | ||
|
|
||
| - libraries | ||
| - Mail servers | ||
| - Extensions for James Mail Servers | ||
| - Command line tools | ||
|
|
||
| We designate as a feature an optional, opt-in behaviour of a James server that can be configured by | ||
| user willing to rely on it. | ||
|
|
||
| James as a project delivers several artifacts, and features. In order for project users to better |
There was a problem hiding this comment.
These seem to me like important and fundamental enough concepts to either expand the scope of this ADR, or to have a separate ADR that clearly defines (as a base concept for other ADRs like this one) the concepts of "artifact" and "feature".
There was a problem hiding this comment.
I think the context part does it well enough. If it is not clear enough, please provide alternative wordings.
There was a problem hiding this comment.
That is not quite what I meant. I was not commenting on the words, I was commenting on the scope. This ADR is about "quality". However, the terms "artifacts" and "features" seem to be so fundamental that they should not be hidden away in a document that describes something different.
So we should either:
- Expand the scope of this document (i.e. it defines "Artifacts", "Features", and "Quality Levels" and not just "Quality Levels", or
- Have a separate ADR that defines "Artifacts" and "Features".
Then this can be referred to from other discussions which will surely use these fundamental concepts.
Just something to consider, maybe.
There was a problem hiding this comment.
I have a different approach here.
I started another document to explain which server we ship as part of the James project, then I hope that with @ieugen we come up with a list of libraries we deliver. Each one of them will get their own ADR, giving a de-facto definition (and exhaustive list of artifacts).
Regarding features, refining extension mechanisms and planning the work for their extraction would do the same.
| Users should have low expectations regarding experimental artifacts or features. They are encouraged to contribute to them | ||
| in order to raise its quality. | ||
|
|
||
| By **unsupported** we mean that an artifact or feature do not match most of the *mature* quality conditions. Active |
There was a problem hiding this comment.
In the referenced email, @chibenwa wrote:
This is a fair point, but 'quality' differs from 'support' concerns.
I agree with this point, which is why I have trouble with the "unsupported" label.
There was a problem hiding this comment.
Propose an alternative please.
I want a term that reflexes the "nothing to expect" concepts. "unsupported" convey this idea quite well to be honest.
There was a problem hiding this comment.
I wrote in a separate comment below how I think the concepts are a bit mixed up. But again, there comes a point at which a boundary has to be drawn, so even though IMO it is not entirely clear, I leave the decision up to you. :)
|
First, I want to mention again that I think this is a really good initiative. Personally, I think it's really important that people can "know" to some extent what they are getting, especially from a system as huge and important as James.
It's difficult nailing down a vocabulary that is commonly shared. Maybe the orthogonal concepts would be something like:
(I just realized that this all implies that the contract be very clearly stated.) I quickly tested these, and they look at first glance to be orthogonal. (High quality but low reliability vs low quality and high reliability, etc.) They are debatable, so feel free to suggest improvements. The list "mature", "experimental", "unsupported" seems to mix these concerns a little. But it is difficult to come up with a practical list. "Quality" needs a lot of definition, and likely depends on the other terms. So based on these, just a thought, but perhaps the first step is to clearly define the "Formality" and "Design Principles" of the community (which to some extent you already do in this ADR), then it would be very easy to map a quality level. |
Co-authored-by: Raphaël Ouazana <rouazana@linagora.com>
|
What would you think about |
|
Kubernetes uses some words to describe the API stability as well https://kubernetes.io/docs/concepts/overview/kubernetes-api/ . |
|
I would like to make a push and merge this. Any comments you would like me to address before ? |
|
Let's use it like this and see where it goes. |
|
Merged |
See https://www.mail-archive.com/server-dev@james.apache.org/msg66909.html