doi's for plugins

[dsarahstamps]

Do you mind if for each plugin that is “merged” after community review there
is an automatically generated and citable doi when compiling the manual? For
example, within the manual next to the title of the plugin:

If this plugin is used for a publication or the development of another plugin
cite the appropriate ASPECT publications found in the manual and:

• Austermman (2016), Plugin: Heatflux Postprocess, Advanced Solver for
  Problems in Earth’s Convection manual v1.4.pre, doi: XXXX
• Menno F., D.S. Stamps, W. Bangerth (2013), Plugin: Chunk Geometry,
  Advanced Solver for Problems in Earth’s Convection manual v1.4, doi: XXXX
• Naliboff, J. (2016), Plugin: Stress Limiter Rheology, Advanced Solver for
  Problems in Earth’s Convection manual v1.4, doi: XXXX
• Thieliot, C., A. Glerum, etc. (20XX), Plugin: XX, Advanced Solver for
  Problems in Earth’s Convection manual vXX, doi: XXXX

In this way when an author of a publication cites the plugin the plugin
authors get citation credit. Citation numbers in the academic community weigh
heavily in evaluations for jobs and promotions.

[bangerth]

Related to #1014 which contains a mock-up for how some kind of citation system may work.

[bangerth]

We had a long discussion about this in the dark last night on the porch between myself, @tjhei , @ljhwang and @lkellogg (and others). CIG has worked on ways to give appropriate credit for software contributions for a good while now, and is funded by NSF to do this. In all of the discussions we have had on this issue, the overarching goal we all (obviously!) agree upon is that we should give credit where credit is due, and make sure that those in our community who contribute code and other things do well in their careers.

With this out of the way, let me more specifically address @dsarahstamps 's suggestion of DOIs for each plugin. Here's how I think our consensus was last night:

• Assigning DOIs to anything (plugins, manual sections, etc -- collectively called artifacts) is easy, as is putting references to DOIs into papers. But it only helps if indexing services actually index what you assign these DOIs to. Otherwise, people may reference these artifacts but nobody will ever know that that happened and the author will not get any credit. We don't know what Thompson-Reuters and google scholar index, but it is to the best of our knowledge only papers in journals and on select preprint servers, books, and similar things. The likelihood that they will index plugins is pretty small. We also have no influence on what they index. In other words, we think that there is not actually any measurable benefit to the author of a plugin that could be visible to a hiring or promotion committee.
• There are many plugins in ASPECT. I think that right now, there are 177. Every ASPECT run will use quite a lot of them -- I would suspect 15-20 quite easily. It would be a long list of references everyone would have to put into their papers, and journals would almost certainly balk at the idea of having that many ASPECT references alone at the end of a paper.
• If this were the standard for ASPECT, it would only be fair to use the same standard transitively for deal.II, Trilinos and p4est. github lists currently 33 people who have contributed code to ASPECT (https://github.com/geodynamics/aspect/graphs/contributors), and 74 for deal.II (https://github.com/dealii/dealii/graphs/contributors). Every ASPECT run almost certainly runs through code that was written by 30 or more people.
• Fairness would dictate that we retroactively also identify who wrote the existing 177 plugins. That would be a significant amount of work. But it is also complicated because over time, people write and rewrite pieces of code, and the original authors of a plugin may no longer be the ones who actually contributed most to it. In other words, it's difficult to say who actually wrote a particular plugin.

Given all this, it seems to us that giving credit in the form of DOIs at the granularity of individual plugins may not be the appropriate way of achieving the goals laid out above. Rather, let me lay out how we think this should work.

First, citation credit is really only one way to get credit where credit is due. There are many others. For example:

• People who work on a code get jobs offered that are related to this code. I hired Timo, Juliane, and Rene all because of their prior contributions to deal.II and ASPECT. CIG has hired several people as staff because they were working on specific codes CIG wanted to support. I got my job in small part because one of the members of the hiring committee had had a phone conversation on the morning of my job talk with a colleague who told him how excited he was to have found deal.II and what a great tool it was for his work. If I get asked by a colleague about potential candidates for postdoc or faculty positions, the people who I have worked with on deal.II and ASPECT are on my mind, and I mention the ones who I think are good and have contributed a lot. In other words, knowledge about who is good and contributes spreads in the community, and people listen and take it into account.
• Hiring and promotion rely heavily on letters of evaluation. I write them all the time, and must have written a dozen for members of the deal.II and ASPECT communities over the year, as well as several for people from other open source communities I'm familiar with. Of course, the number, quality, and level of contributions these people have made enters the discussion in these letters, and I want to think that the positive things we have to say about many members of the ASPECT community help them find employment and get promoted.
• We also write letters on other occasions. I have written several letters of commitment over the years for NSF proposals by colleagues who, as part of their projects, wanted to contribute to deal.II or ASPECT. I won't write such letters for anyone, but I definitely do for people who have contributed in the past and who I know can work with our community.
• Dynamic Citation Thing #1014 is an attempt at providing a way to at least ensure that papers that describe significant extensions of ASPECT are cited. We do something similar with the papers listed at the top of https://dealii.org/publications.html . This may not be quite at the granularity of plugins, but it does recognize the largest contributions explicitly.
• Ralf, Guido and I got the 2007 Wilkinson Prize for Numerical Software based on our work on deal.II, not for a specific paper. In fact, the deal.II paper everyone cites was only published later that same year. But people knew about what we were doing, and suggested us to the prize committee. Whether anyone does or does not get a prize is a pretty random process, given that there are so few to hand out. But you bet the older ones of us are thinking about nominating others we know and whose work we appreciate.
• In deal.II, we have started to write a short paper for each release with all of the major contributors to that release. This provides a way to get citations for things they worked on.
• In deal.II, we have also created a formal "developer council" that those who get elevated to it can put on their CV. @jdannberg and @gassmoeller were elevated to the rank of "maintainer of ASPECT" just yesterday.

I think the upshot is that all of really want the members of our community to succeed, and that there are many mechanisms to help with this. Providing ways to collect citations is certainly one approach, but it is not the only one and maybe not even the best. Citations alone are not a particularly good measure of a scientific community's appreciation for someone, and ultimately it is the community's appreciation that predicates career success.

[tjhei]

Thanks for writing this summary, @bangerth . I agree with the points you made. I have a few more comments:

• Assigning a DOI to a module/plugin/part of a source file is technically not that trivial. It is easy to get a DOI for a git hash inside a repo (so that could be a merge-commit or some version of ASPECT), but this is not a DOI "for a plugin".
• Most contributions are not a single, self-contained plugin. How do you refer to improvements of the entropy viscosity stabilization that changed a couple lines of code here and there?
• I see one very good reason to figure out a way to cite parts of ASPECT or deal.II. This is not to ask others to give me credit, but to refer to my work myself. For example, I would like to document in my annual report (to NSF or my employer) that I wrote deal.II step-55. Here, I never bothered to create a DOI (and I think that doesn't matter). A reference could have the form "plugin XY in ASPECT manual chapter 2.3, bla bla".
• We should talk about if we want to copy the "release paper" idea for ASPECT or generate citations for ASPECT releases (zenodo or something else).

[bangerth]

In my annual reports, I've taken to explaining in quite some detail (because my colleagues are not used to these sorts of things) what I've done. This includes listing the tutorial programs I (co-)wrote (including HTML reference), but I've also listed the number of commits and the number of pull requests that have been submitted (with an adjustment for the fraction that I probably commented on).

Release paper: Yes.

[egpuckett]

I concur with the points made by both Wolfgang and Timo. Thank you for taking the time to write this!

[ljhwang]

Historically, projects like GMT and PyLIth have issued release notices in EOS as a short article. These often became the citable reference as there was no paper and the implicitly also carried versioning information.

[dsarahstamps]

Such EOS release notices don't give individual credit for the contributors. The ASPECT community review process is rigorous and each plugin is a substantial contribution to the academic community. There must be a better way forward.

On Jul 1, 2016, at 6:40 PM, Lorraine Hwang notifications@github.com wrote:

Historically, projects like GMT and PyLIth have issued release notices in EOS as a short article. These often became the citable reference as there was no paper and the implicitly also carried versioning information.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[bangerth]

@dsarahstamps -- I think we all agree that people deserve to get more credit for what they contribute, but we haven't come up with a good solution that works within the citation system we've been using for a few centuries by now :-( Unless you want to have high-energy-style publications with 500 authors for which the list of affiliations is larger than the actual paper.