Badging for project lifecycle

[cboettig]

I greatly enjoy using the use_* commands to get the markdown for my README badges. Curious what your team thinks of the http://www.repostatus.org/ badges, and if you would consider a l'lil function to generate the appropriate badge. I know there's some risk of these becoming stale since they are just static badges, but personally I find the categories quite helpful, particularly on the sunset era of a package.

[hadley]

I like the idea, but I don't love repostatus - it just feels a bit too fine grained to me.

Another option is https://github.com/badges/stability-badges but they don't feel quite right for the R community.

[cderv]

Just for pointing other interesting links. Here is an initiative for stability badges for R : https://github.com/joethorley/stability-badges based on shield.io services.

[cboettig]

The stability-badges makes sense to me once I realized I should think of them as package lifecycle terms, but I think stability is the wrong term. I see the motivation: e.g. plyr would be Locked, in the sense that commits would only patch bugs; perhaps closest to what repostatus calls Inactive, and unstable is approximately WIP, e.g. pre-CRAN release.

Yet when I first looked at stability-badges, I was greatly confused because I was not thinking of the in lifecycle terms but just as a generic indicator of the API. I think the unstable & stable are the most confusing: on the face of it, it sounds like stable is a pledge not never to break the API and never to bump the major version. Closer inspection shows that it does not make that claim, but then it becomes a bit unclear how this differs from an unstable package -- is an unstable package just a stable package that fails to bump the major version when it makes such a change? (This also gnaws at my dislike of semantic versioning -- sure there are big changes and small changes, changes that do or don't alter the API; but any change can break things, and I think the promise that "these changes won't hurt" is misleading or dangerous).

Focusing on the term stability, using unstable or stable, in my opinion, puts too much emphasis on classifying how the API changes and makes it easy to miss the lifecycle aspect (e.g. it is easy to miss the implication that stable is a more mature version of unstable, and instead think this is just a message of how much the maintainer gives a crap about the idea of backwards compatibility.).

If the those badges were called lifecycle badges instead of stability badges, maybe replacing unstable with the more life-cycle focused term beta and locked with a term that better indicates "this is an old package that gets limited maintenance priority / only 'security' updates / bug patches" I'd feel much better about it...

[hadley]

Ooh yes, I love the idea of focussing on the lifecycle!

[hadley]

Thoughts on how I see the life cycle of an R package:

• Experimental - very early days, a lot of churn in search of a good API (e.g. monads, strict). Not on CRAN. Might not go anywhere.

• Maturing - will strive to maintain backward compatibility (e.g. rlang, pkgdown), but need wider usage in order to get more feedback.

• Dormant - not currently working on it, (e.g. ggvis) but plan to come back to in the future.

• Stable - we're happy with the API (e.g. ggplot2). There are unlikely to be major changes, and backward incompatible changes will only be made if absolutely critical.

• Questioning - no long convinced this is a good approach.

• Retired - better replacement elsewhere (e.g. reshape, reshape2, lazyeval, pryr). Still on CRAN, and will fix R CMD check failures to keep it there.

• Archived - development complete. Archived from CRAN; github issues turned off.

And in a figure with most common transitions marked:

This lifecycle applies not just to packages but to components within packages. For example, the big pieces of dplyr (mutate, select etc) are stable, but I'm questioning do() in favour of list columns, and we're retired the SE version of functions. So as well as package badges, we might want roxygen2 infrastructure so that you can badge individual functions (e.g. @lifecycle mature would copy badge image into man/figures then generate embed linked image).

Note that it's particularly hard to know when a package is "stable" because it's hard to know when you're "done" with a problem. Sometimes it's straightforward (e.g. with smaller packages like forcats/stringr), but for bigger packages like ggplot2/dplyr they might be maturing for 5+ years, and they only become stable once I move on to the next generation of the tool.

[joethorley]

Some random thoughts

This is a good summary of a developers perspective. As a user stumbling upon a package I want to be able to quickly assess how stable it is and how likely it is to be supported long-term. I understand that stability is a continuum but its still important to get an indication of the frequency of and severity of breaking changes. And I want to know if the developer is committed in the long-term. These two pieces of information allow me to decide whether I am going to use the package (which also depends on how important it is to my work)

The categories above can be classified in this way

• experimental: unstable and may be short-lived (use if inspired)
• maturing: more stable and looking like it may have longer shelf life (use if directly relevant to workflow)
• stable: stable and long-term support (use)
• questioning: stable but may not have long-term support (use if no better alternatives)
• retired: stable but no longer long-term support (don't start using and transition to alternatives)
• archive: so stable changes in other packages are going to break it with no long-term support (stop using)
• dormant: unstable and no long-term-support (use if willing to take on)

I think the schema at https://github.com/joethorley/stability-badges is reasonable but could be improved - its good to see the topic getting attention.

I also think dormant and questioning are unnecessary categories for the end user because a) both are potentially reversible; b) dormant can be inferred from the commit history and c) its not until a package is retired that a user needs to switch.

If these are dropped one is left with a linear sequence of

experimental, maturing, stable, retired and archive

these roughly correspond to the five categories in https://github.com/joethorley/stability-badges of

Experimental, Unstable, Stable, Locked, Deprecated

I struggled with the naming scheme and definitely prefer Maturing over Unstable. However I prefer Locked to Retired since its clearer what the change is. Archived seems superior to Deprecated.

I also don't like the references to CRAN since a whole package can live out its lifecycle on GitHub.

Hopefully these reflections have some value.

[mwaldstein]

I agree about the difference between developer view and user view and want to amplify that idea.

The model outlined here is partially compatible with semantic versioning (http://semver.org/). One challenge R faces that the js world has been rapidly iterating around is the idea that you don't always need to be on the latest version - once you have a dependency that is stable for your use case, don't upgrade unless there is a convincing reason. R tooling doesn't really support this approach - e.g. stay on 1.0 rather than upgrade to 2.0 because it still works. In the context of lifecycle, this means that "stable" just means "the current design is well tested and has some guarantee of working" not "the api won't change".

I feel w/in this discussion, more valuable is a communication around level of support and iteration as @joethorley suggests - e.g. the range between "Wild ideas welcome" to "If it works for you, great, if not fork it b.c. I'm no longer supporting"

[cboettig]

@joethorley Great feedback, thanks for sharing. I agree picking the right adjectives is definitely tricky. A few quick thoughts:

• re: dormant: I think this is here to distinguish from retired. Both would have a quiet commit history. What badge would you suggest using for, say, ggvis in Hadley's example if not dormant? (Obviously leaving it unbadged is also ambiguous).

Re developer vs user perspective: I think the concept of lifecycle is actually more about communicating with users than developers. I believe the concept of API stability is primarily a concern only of relatively sophisticated users (aka other developers who use or depend on the package. Not saying users shouldn't care about API stability, but that is not a significant factor in how I see most R users selecting packages). For instance, many R users will simply try to interpret these badges in a hierarchy, where it is easy to think "Locked > Stable > Experimental", and I don't think most R users would be better served choosing plyr because it is "locked" stability! yay! over dplyr which is merely stable, say. It takes a developer's mindset to realize that the rock-solid stability of "locked" really means "old package replaced by something better".

Re semver, I agree with @joethorley 's perspective that semver is really focused on packages in the stable bucket as way to signal to developers the nature of changes being made within versions of an individual package. It doesn't speak to the issue of whether I should use plyr or dplyr, or let me know that pkgdown API is likely to change.

Re references to CRAN: I agree the syntax should be CRAN agnostic and R agnostic, but "not yet on CRAN", "on CRAN", and "archived from CRAN" are currently the closest thing we have to functional lifecycle signposts in the R community, so making the analogy explicit is helpful.

[joethorley]

@cboettig I would suggest using Experimental for ggvis because they only way it differs from an experimental package is that it is not currently being worked on (as indicated by a quiet commit history). If it was retired it could be badged Retired.

You are right Locked might be misleading for less advanced users and therefore Retired might be better. The question of how to interpret the terms is partly dealt with by clicking on the badges which can lead to a definition of what the term means. I also think that a short page could explain to users what the five or so badges mean and how to relate to them.

[hadley]

Re: questioning - this is very important for me (albeit more at the function level than the package level). I need some way to communicate to people that I think there's likely to be a better approach in the future and not to invest heavily in the current way of doing things.

Re: dormant - I don't think it's reasonable to expect to carefully examine the commit history to determine the repository status. By chance, there may have been a brief flurry of activity on a retired package because a patch was needed. Dormant explicitly says that we're not working on this right now, but we expect to return to it in the future. I don't believe that the life cycle of a package is a linear sequence.

Re: CRAN - I think it's important to start out concrete and specific before generalising. Specifically to R, I would never depend on any package that doesn't have a long term plan to get on CRAN. Not aiming for CRAN suggests that the developer doesn't particularly care about the package (in the same way not using unit tests suggests that they're not very committed)

Re: semver - I think it's naive because there's no way to tell what is "backwards-compatible bug fix" without knowing all user code. I've fixed plenty of bugs that have broken analysis code because people were dependent on the broken behaviour. To me, this seems like a large unacknowledged flaw and makes it very difficult for me to consider semver as a realistic proposal.

[hadley]

And here's what the badges might look like:

[joethorley]

I like the colouring scheme but would suggest dormant should be orange because it exists between experimental and retired which are both orange and because it should not be rated as highly as maturing or questioning.