-
Notifications
You must be signed in to change notification settings - Fork 11
/
appendix.Rmd
691 lines (622 loc) · 26.9 KB
/
appendix.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
---
always_allow_html: true
---
# Appendices
## Notes on Scope and the Python Statistical Ecosystem {#python}
```{r startup-appendix}
#| echo: false
#| message: false
library(dplyr)
library(ggplot2)
library(rvest)
library(igraph)
theme_set(theme_minimal())
```
Two factors may be usefully noted in this regard:
```{r pypi-projects}
#| echo: false
u <- "https://pypi.org/"
x <- read_html(u) %>%
# html_nodes ("div.statistics-bar") %>%
html_nodes("p.statistics-bar__statistic") %>%
html_text()
pypi_n_projects <- x [grep("projects", x)] %>%
gsub("projects|[[:space:]]", "", .)
cran_n_projects <- format(nrow(available.packages()), big.mark = ",")
pypi_cran_ratio <- floor(as.integer(gsub(",", "", pypi_n_projects)) /
as.integer(gsub(",", "", cran_n_projects)))
```
1. The potential number of python packages for statistical analyses is likely
to be relatively more restricted than relative numbers of **R** packages.
Taking as indicative presentations at the previous three Joint Statistical
Meetings (JSMs; 2018-2020), no python packages were referred to in any
abstract, while 32 **R** packages were presented, along with two
meta-platforms for **R** packages. Presentations at the Symposium of Data
Science and Statistics (SDSS) for 2018-19 similarly including numerous
presentations of **R** packages, along with presentation of
[three](https://altair-viz.github.io)
[python](https://github.com/ajboyd2/salmon)
[packages](https://github.com/dlsun/symbulate). It may accordingly be
expected that potential expansion to include python packages will demand
relatively very little time or effort compared with that devoted to **R**
packages as the primary software scope.
2. In spite of the above, the community of python users is enormously greater,
reflected in the currently `r pypi_n_projects` packages compared with
`r cran_n_projects` packages on CRAN, or over `r pypi_cran_ratio` times as
many python packages. Similarly, 41.7% of all respondents to the [2019
stackoverflow developer
survey](https://insights.stackoverflow.com/survey/2019) nominated python as
their most popular language, compared with only 5.8% who nominated **R**.
The relative importance of python is powerfully reflected in temporal trends
from the [stackoverflow developer
survey](https://insights.stackoverflow.com/survey/2019) from the previous three
years, with results shown in the following graphic.
```{r py_v_r}
#| echo: false
#| message: false
year <- 2017:2019
python_used <- c(31.7, 38.8, 41.7)
r_used <- c(4.4, 6.1, 5.8)
python_loved <- c(62.7, 68.0, 73.1)
r_loved <- c(49.9, 49.4, 51.7)
dat <- rbind(
tibble(
year = year,
proportion = python_used,
language = "python-used",
context = "used"
),
tibble(
year = year,
proportion = r_used,
language = "R-used",
context = "used"
),
tibble(
year = year,
proportion = python_loved,
language = "python-loved",
context = "loved"
),
tibble(
year = year,
proportion = r_loved,
language = "R-loved",
context = "loved"
)
)
ggplot(dat, aes(x = year, y = proportion, color = language)) +
geom_line() +
geom_point() +
scale_x_continuous(breaks = year)
```
Python is not only more used and more loved than **R**, but both statistics for
python have consistently grown at a faster rate over the past three years as
have equivalent statistics for **R**.
Both languages nevertheless have relative well-defined standards for software
packaging, python via the [Python Package Index
(pypi)](https://packaging.python.org/tutorials/packaging-projects), and **R**
via [CRAN](https://cran.r-project.org). In contrast to CRAN, which runs its own
checks on all packages on a daily basis, there are no automatic checks for
[pypi](https://packaging.python.org/tutorials/packaging-projects) packages, and
almost any form of package that minimally conforms to the standards may be
submitted. This much lower effective barrier to entry likely partially
contributes to the far greater numbers of
[pypi](https://packaging.python.org/tutorials/packaging-projects)
(`r pypi_n_projects`) than
[CRAN](https://cran.r-project.org) (`r cran_n_projects`) packages.
## Empirical Derivation of Categories {#appendix-categories}
We attempted to derive a realistic categorisation through using empirical data
from several sources of potential software submissions, including all
apparently "statistical" R packages published in the [Journal of Open Source
Software (JOSS](https://joss.theoj.org)), packages published in the [Journal of
Statistical Software](https://www.jstatsoft.org/index), software presented at
the 2018 and 2019 Joint Statistical Meetings (JSM), and Symposia on Data
Science and Statistics (SDSS), well as CRAN task views. We have also compiled
a list of the descriptions of [all packages rejected by
rOpenSci](https://github.com/mpadge/statistical-software/blob/master/abstracts/ropensci-abstracts.md)
as being out of current scope because of current inability to consider
statistical packages, along with a selection of [recent statistical
R packages](https://github.com/mpadge/statistical-software/blob/master/abstracts/joss-abstracts.md)
accepted by JOSS. (The full list of all R package published by JOSS can be
viewed at <https://joss.theoj.org/papers//in/R>).
We allocated one or more key words (or phrases) to each abstract, and use the
frequencies and inter-connections between these to inform the following
categorisation are represented in the [interactive
graphic](https://ropenscilabs.github.io/statistical-software/abstracts/network-terms/index.html)
(also included in the [Appendix](#appendix-keywords)), itself derived from
analyses of abstracts from all statistical software submitted to both rOpenSci
and JOSS. (Several additional analyses and graphical representations of these
raw data are included an [auxiliary github
repository](https://github.com/ropenscilabs/statistical-software).) The primary
nodes that emerge from these empirical analyses (with associated *relative*
sizes in parentheses) are shown in the following table.
```{r top10-terms}
#| message: false
#| echo: false
x <- readLines("scripts/joss-abstracts/categories.Rmd")
x <- x [grep("[0-9]*\\. \\[", x)]
names <- vapply(x, function(i) {
res <- strsplit(i, "\\[\\`") [[1]]
strsplit(res, "\\`\\]") [[2]] [1]
},
character(1),
USE.NAMES = FALSE
)
terms <- lapply(x, function(i) {
res <- strsplit(i, ":\\s") [[1]] [2]
res <- strsplit(res, ";\\s") [[1]] [1] # rm "; input"
strsplit(res, ",\\s") [[1]]
})
input <- lapply(x, function(i) {
res <- strsplit(i, "input: ") [[1]] [2]
res <- strsplit(res, ";\\s") [[1]] [1]
if (grepl(",\\s", res)) {
res <- strsplit(res, ",\\s") [[1]]
}
return(res)
})
output <- lapply(x, function(i) {
res <- strsplit(i, "output: ") [[1]] [2]
if (grepl(",\\s", res)) {
res <- strsplit(res, ",\\s") [[1]]
}
return(res)
})
names(terms) <- names(input) <- names(output) <- names
n_abstracts <- length(terms)
terms0 <- terms # used below
terms <- unlist(terms) %>%
table() %>%
sort(decreasing = TRUE)
terms <- data.frame(
n = seq_along(terms),
term = names(terms),
proportion = as.integer(terms) / n_abstracts,
stringsAsFactors = FALSE
)
terms$term [terms$term == "scores"] <- "statistical indices and scores"
n <- max(which(terms$proportion > 0.05))
cp <- paste0 ("Most frequent key words from all JOSS abstracts (N = 92) ",
"for statistical software. Proportions are scaled *per ",
"abstract*, with each abstract generally having multiple ",
"key words, and so sum of proportions exceeds one.")
knitr::kable(head(terms, n = n), digits = c(0, NA, 3), caption = cp)
```
```{r collate-some-categories}
#| echo: false
terms$term [terms$term %in% c("Bayesian", "Monte Carlo")] <-
"Bayesian & Monte Carlo"
terms$term [terms$term %in% c("dimensionality reduction",
"feature selection")] <-
"dimensionality reduction & feature selection"
terms$term [terms$term %in% c("regression", "splines", "interpolation")] <-
"regression/splines/interpolation"
terms$term [terms$term == "EDA"] <- "Exploratory Data Analysis (EDA)"
terms <- terms %>%
group_by(term) %>%
summarise(proportion = sum(proportion)) %>%
arrange(by = desc(proportion))
```
The top key words and their inter-relationships within the main [network
diagram](https://ropenscilabs.github.io/statistical-software/abstracts/network-terms/index.html)
were used to distinguish the following primary categories representing all
terms which appear in over 5% of all abstracts, along with the two additional
categories of "spatial" and "education". We have excluded the key word
"Estimates" as being too generic to usefully inform standards, and have also
collected a few strongly-connected terms into single categories.
```{r collate-methods-categories}
#| echo: false
out <- data.frame(terms [which(terms$proportion > 0.05), ])
out <- out [which(!out$term == "estimates"), ]
out <- rbind(
out, terms [which(terms$term == "spatial"), ],
terms [which(terms$term == "education"), ]
)
out$comment <- ""
out$comment [out$term == "dimensionality reduction & feature selection"] <-
"Commonly as a result of ML algorithms"
out$comment [out$term == "regression/splines/interpolation"] <-
"Including function data analysis"
out$comment [out$term == "probability distributions"] <-
paste0 ("Including kernel densities, likelihood estimates and estimators, ",
"and sampling routines")
out$comment [out$term == "categorical variables"] <-
paste0 ("Including latent variables, and those output from ML algorithms. ",
"Note also that method for dimensionality reduction (such as ",
"clustering) often transform data to categorical forms.")
out$comment [out$term == "Exploratory Data Analysis (EDA)"] <-
paste0 ("Including information statistics such as Akaike's criterion, ",
"and techniques such as random forests. ",
"Often related to workflow software.")
out$comment [out$term == "survival"] <-
paste0 ("strongly related to EDA, yet differing in being strictly ",
"descriptive of software *outputs* whereas EDA may include ",
"routines to explore data *inputs* and other pre-output ",
"stages of analysis.")
out$comment [out$term == "summary statistics"] <-
paste0 ("Primarily related in the empirical data to regression and ",
"survival analyses, yet clearly a distinct category of its own.")
out$comment [out$term == "workflow"] <-
"Often related to EDA, and very commonly also to ML."
out$comment [out$term == "model selection"] <-
paste0 ("A important intermediate node between such categories as ML, ",
"Bayesian and Monte Carlo techniques, visualisation, and EDA.")
out$comment [out$term == "statistical indices and scores"] <-
paste0 ("Software generally intended to produce specific ",
"indices or scores as statistical output")
out$comment [out$term == "spatial"] <-
paste0 ("Also an important intermediate node connecting several ",
"other nodes, yet defining its own distinct cluster ",
"reflecting a distinct area of expertise.")
out <- data.frame(
n = seq(nrow(out)),
term = out$term,
proprtion = out$proportion,
comment = out$comment,
stringsAsFactors = FALSE
)
cp <- paste0 ("Proposed categorisation of statistical software, ",
"with corresponding proportions of all JOSS software ",
"matching each category")
knitr::kable(out, digits = c(0, NA, 3, NA), caption = cp)
```
The full network diagram can then be reduced down to these categories only,
with interconnections weighted by all first- and second-order interconnections
between intermediate categories, to give the following, simplified diagram
(in which "scores" denotes "statistical indices and scores"; with the diagram
best inspected by dragging individual nodes to see their connections to
others).
```{r visnetwork-simple}
#| echo: false
#| message: false
nodes <- table(unlist(terms0))
nodes <- data.frame(
id = names(nodes),
label = names(nodes),
value = as.integer(nodes),
stringsAsFactors = FALSE
)
edges <- lapply(terms0, function(i) {
if (length(i) > 1) {
res <- sort(i)
n <- combn(seq_along(res), 2)
cbind(
res [n [1, ]],
res [n [2, ]]
)
}
})
edges <- do.call(rbind, edges)
edges <- data.frame(
from = edges [, 1],
to = edges [, 2],
stringsAsFactors = FALSE
) %>%
group_by(from, to) %>%
summarise(width = length(from)) %>%
ungroup()
# Remove isolated edges:
# annotation, areal weights, binomial distribution, cubature, gene loci,
# generalized least squares, misspecification, classification, interpolation,
# over-dispersion, integration, random effects, probit model
cl <- graph_from_data_frame(edges) %>%
clusters()
out <- names(cl$membership [which(cl$membership != which.max(cl$csize))])
nodes <- nodes [which(!nodes$id %in% out), ]
edges <- edges [which(!(edges$from %in% out | edges$to %in% out)), ]
# manually agglomerate some categories
from <- c(
"Bayesian|Monte Carlo", "dimensionality reduction|feature selection",
"^regression|splines|interpolation"
)
to <- c("Bayes/MC", "dimensionality reduction", "regression")
for (i in seq_along(from)) {
edges$from [grep(from [i], edges$from)] <- to [i]
edges$to [grep(from [i], edges$to)] <- to [i]
nodes$id [grep(from [i], nodes$label)] <- to [i]
nodes$label [grep(from [i], nodes$label)] <- to [i]
}
edges <- edges %>%
group_by(from, to) %>%
summarise(width = sum(width)) %>%
ungroup()
nodes <- nodes %>%
group_by(label) %>%
summarise(value = sum(value)) %>%
transform(id = label)
g <- graph_from_data_frame(edges)
dg <- distances(g)
# add to edge distances all intermediate distances between O(2) connections
edges2 <- edges
dgnames <- rownames(dg)
for (i in seq(nrow(edges2))) {
dgfrom <- match(edges2$from [i], rownames(dg))
dgto <- match(edges$to [i], colnames(dg))
nbs_from <- dgnames [which(dg [dgfrom, ] < 3)]
nbs_to <- dgnames [which(dg [, dgto] < 3)]
nbs <- intersect(nbs_from, nbs_to)
index <- which(edges$from == edges$from [i] & edges$to %in% nbs)
edges2$width [i] <- sum(edges$width [index])
}
ns <- nodes [order(nodes$value, decreasing = TRUE), ]
index <- c(
which(!grepl("estimates", ns$label [1:15])),
grep("spatial|education", ns$label)
)
ns <- ns [index, ]
es <- edges2 [which(edges$from %in% ns$label & edges$to %in% ns$label), ]
# es <- es [which (es$from %in% ns$label & es$to %in% ns$label), ]
es <- es [which(!es$from == es$to), ]
es$width <- es$width * 5 / max(es$width)
```
::: {.content-hidden when-format="pdf"}
```{r visnetwork1}
#| eval: true
#| echo: false
#| message: false
library(visNetwork)
visNetwork(ns, es)
```
:::
Standards considered under any of the ensuing categories must be developed with
reference to inter-relationships between categories, and in particular to
potential ambiguity within and between any categorisation. An example of such
ambiguity, and of potential difficulties associated with categorisation, is the
category of "network" software which appropriate describes the
[`grapherator`](https://github.com/jakobbossek/grapherator) package (with
accompanying [JOSS paper](https://joss.theoj.org/papers/10.21105/joss.00528))
which is effectively a distribution generator for data represented in
a particular format that happens to represent a graph; and three JSM
presentations, one on [network-based clustering of high-dimensional
data](https://ww2.amstat.org/meetings/jsm/2018/onlineprogram/AbstractDetails.cfm?abstractid=327171),
one on [community structure in dynamic
networks](https://ww2.amstat.org/meetings/jsm/2018/onlineprogram/AbstractDetails.cfm?abstractid=328764)
and one on [Gaussian graphical
models](https://ww2.amstat.org/meetings/jsm/2018/onlineprogram/AbstractDetails.cfm?abstractid=328764).
Standards derived for network software must accommodate such diversity of
applications, and must accommodate software for which the "network" category
may pertain only to some relatively minor aspect, while the primary algorithms
or routines may not be related to network software in any direct way.
## Analysis of statistical software keywords {#appendix-keywords}
The
[JOSS](https://joss.theoj.org)
conducts its own peer review process, and publishes textual descriptions of
accepted software. Each piece of software then has its own web page on the
journal's site, on which the text is presented as a compiled `.pdf`-format
document, along with links to the open review, as well as to the software
repository. The published document must be included within the software
repository in a file named `paper.md`, which enables automatic extraction and
analysis of these text descriptions of software. Rather than attempt
a comprehensive, and unavoidably subjective, categorization of software, these
textual descriptions were used to identify key words or phrases (hereafter,
"keywords") which encapsulated the purpose, function, or other general
descriptive elements of each piece of software. Each paper generally yielded
multiple keywords. Extracting these from all papers judged to be potentially in
scope allowed for the construction of a network of topics, in which the nodes
were the key words and phrases, and the connections between any pair of nodes
reflected the number of times those two keywords co-occurred across all papers.
We extracted all papers accepted and published by JOSS (217 at the time of
writing in early 2020), and manually determined which of these were broadly
statistical, reducing the total to 92. We then read through the contents of
each of these, and recorded as many keywords as possible for each paper. The
resultant network is shown in the following interactive graphic, in which nodes
are scaled by numbers of occurrences, and edges by numbers of co-occurrences.
(Or [click
here](https://ropenscilabs.github.io/statistical-software/abstracts/network-terms/index.html)
for full-screen version with link to code.)
```{r visnetwork}
#| message: false
#| echo: false
x <- readLines("stat-software-categories.md")
x <- x [grep("[0-9]*\\. \\[", x)]
names <- vapply(x, function(i) {
res <- strsplit(i, "\\[\\`") [[1]]
strsplit(res, "\\`\\]") [[2]] [1]
},
character(1),
USE.NAMES = FALSE
)
terms <- lapply(x, function(i) {
res <- strsplit(i, ":\\s") [[1]] [2]
res <- strsplit(res, ";\\s") [[1]] [1] # rm "; input"
strsplit(res, ",\\s") [[1]]
})
input <- lapply(x, function(i) {
res <- strsplit(i, "input: ") [[1]] [2]
res <- strsplit(res, ";\\s") [[1]] [1]
if (grepl(",\\s", res)) {
res <- strsplit(res, ",\\s") [[1]]
}
return(res)
})
output <- lapply(x, function(i) {
res <- strsplit(i, "output: ") [[1]] [2]
if (grepl(",\\s", res)) {
res <- strsplit(res, ",\\s") [[1]]
}
return(res)
})
names(terms) <- names(input) <- names(output) <- names
# Then convert to `visNetwork` nodes and edges tables:
nodes <- table(unlist(terms))
nodes <- data.frame(
id = names(nodes),
label = names(nodes),
value = as.integer(nodes),
stringsAsFactors = FALSE
)
edges <- lapply(terms, function(i) {
if (length(i) > 1) {
res <- sort(i)
n <- combn(seq_along(res), 2)
cbind(res [n [1, ]], res [n [2, ]])
}
})
edges <- do.call(rbind, edges)
edges <- data.frame(
from = edges [, 1],
to = edges [, 2],
stringsAsFactors = FALSE
) %>%
group_by(from, to) %>%
summarise(width = length(from)) # Remove isolated edges:
# annotation, areal weights, binomial distribution, cubature, gene loci,
# generalized least squares, misspecification, classification, interpolation,
# over-dispersion, integration, random effects, probit model
cl <- graph_from_data_frame(edges) %>%
clusters()
out <- names(cl$membership [which(cl$membership != which.max(cl$csize))])
nodes <- nodes [which(!nodes$id %in% out), ]
edges <- edges [which(!(edges$from %in% out | edges$to %in% out)), ]
```
::: {.content-hidden when-format="pdf"}
```{r visnetwork2}
#| eval: true
#| message: false
#| echo: false
library(visNetwork)
visNetwork(nodes, edges)
```
:::
Such a network visualization enables immediate identification of more and less
central concepts including, in our case, several that we may not otherwise have
conceived of as having been potentially in scope. We then used this network to
define our set of key "in scope" concepts. This figure also reveals that many
of these keywords are somewhat "lower level" than the kinds of concepts we
might otherwise have used to define scoping categories. For example, keywords
such as "likelihood" or "probability" are not likely to be useful in defining
actual categories of statistical software, yet they turned out to lie at the
centres of relatively well-defined groups of related keywords.
We also examined the forms of both input and output data for each of the 92
pieces of software described in these JOSS papers, and constructed [an
additional
graph](https://ropenscilabs.github.io/statistical-software/abstracts/network-io/index.html)
directionally relating these different data formats.
```{r io-nodes-edges}
#| message: false
#| echo: false
terms <- table(c(unlist(input), unlist(output)))
nodes <- data.frame(
id = seq_along(terms),
label = names(terms),
value = as.integer(terms),
stringsAsFactors = FALSE
)
edges <- list()
for (i in seq_along(input)) {
edges [[i]] <- expand.grid(input [[i]], output [[i]])
}
edges <- do.call(rbind, edges) %>%
data.frame(
from = .$Var1,
to = .$Var2
) %>%
group_by(from, to) %>%
summarise(width = length(from))
edges$from <- nodes$id [match(edges$from, nodes$label)]
edges$to <- nodes$id [match(edges$to, nodes$label)]
# remove isolated edges:
cl <- graph_from_data_frame(edges) %>%
clusters()
out <- names(cl$membership [which(cl$membership != which.max(cl$csize))])
nodes <- nodes [which(!nodes$id %in% out), ]
edges <- edges [which(!(edges$from %in% out | edges$to %in% out)), ]
```
::: {.content-hidden when-format="pdf"}
```{r visnetwork3}
#| eval: true
#| message: false
#| echo: false
library(visNetwork)
visNetwork(nodes, edges)
```
:::
## Other Software Standards {#appendix-other-software-standards}
Among the noteworthy instances of software standards, the following are
particularly relevant:
1. The [Core Infrastructure Initiative's Best Practices
Badge](https://bestpractices.coreinfrastructure.org/en), which is granted to
software meeting an extensive list of
[criteria](https://github.com/coreinfrastructure/best-practices-badge/blob/master/doc/criteria.md).
This list of criteria provides a singularly useful reference for software
standards.
2. The [Software Sustainability Institute](https://www.software.ac.uk/)'s
[*Software Evaluation
Guide*](https://www.software.ac.uk/resources/guides-everything/software-evaluation-guide),
in particular their guide to [*Criteria-based software
evaluation*](http://software.ac.uk/sites/default/files/SSI-SoftwareEvaluationCriteria.pdf),
which considers two primary categories of *Usability* and *Sustainability
and Maintainability*, each of which is divided into numerous sub-categories.
The guide identifies numerous concrete criteria for each sub-category,
explicitly detailed below in order to provide an example of the kind of
standards that might be adapted and developed for application to the present
project.
3. The [*Transparent Statistics
Guidelines*](https://transparentstats.github.io/guidelines/), by the "HCI
(Human Computer Interaction) Working Group". While currently only in its
beginning phases, that document aims to provide concrete guidance on
"transparent statistical communication." If its development continues, it is
likely to provide useful guidelines on best practices for how statistical
software produces and reports results.
4. The more technical considerations of the [Object Management
Group](https://www.omg.org/index.htm)'s [*Automated Source Code CISQ
Maintainability Measure*](https://www.omg.org/spec/ASCMM/) (where CISQ
refers to the [*Consortium for IT Software
Quality*](https://www.it-cisq.org/)). This guide describes a number of
measures which can be automatically extracted and used to quantify the
maintainability of source code. None of these measures are not already
considered in one or both of the preceding two documents, but the
identification of measures particularly amenable to automated assessment
provides a particularly useful reference.
There is also rOpenSci's guide on [package development, maintenance, and peer
review](https://devguide.ropensci.org/), which provides standards of this type
for R packages, primarily within its first chapter. Another notable example is
the [tidyverse design guide](https://principles.tidyverse.org/), and the
section on [Conventions for R Modeling
Packages](https://tidymodels.github.io/model-implementation-principles/) which
provides guidance for model-fitting APIs.
Specific standards for neural network algorithms have also been developed as
part of a [google 2019 Summer Of Code
project](http://www.inmodelia.com/gsoc2019.html), resulting in a dedicated
R package, [`NNbenchmark`](https://akshajverma.com/NNbenchmarkWeb/index.html),
and accompanying results---their so-called
["notebooks"](https://akshajverma.com/NNbenchmarkWeb/notebooks.html)---of
applying their benchmarks to a suite of neural network packages.
## Bibliography
::: {#refs}
:::
# Review request template {#reviewrequesttemplate}
Editors may make use of the e-mail template below in recruiting reviewers.
```{r template-request}
#| echo: false
#| eval: true
#| child: "templates/request.md"
```
# Book release guidance {#bookrelease}
Versions of this book are represented in corresponding version numbers of the
*Statistical Software Review* badges which appear on peer-reviewed
repositories. Maintaining this synchronisation between versions of this book
and peer-review badges requires the following steps to be following for each
release of this book.
1. Increment version number in the repository's "DESCRIPTION" file. Releases
require an increment in at least the minor version number, so for example
from 0.1 to 0.2.
2. Create new badges in [the `svgs` directory of
`ropensci-org/badges`](https://github.com/ropensci-org/badges/tree/main/svgs)
by copying existing versions, and updating the text for the version numbers.
These should only have major and minor versions, "XX.YY". Commit and push
changes to that repository. An example of such an update can be seen in
[this
commit](https://github.com/ropensci-org/badges/commit/2c9ef4bfce7a81a8d633ce2a9b6b2f1766d43830).
3. Ensure that corresponding issue labels exist on [the main `software-review`
repository](https://github.com/ropensci/software-review/labels), or create
issue labels with new version numbers if they don't exist.
Approving a package will then automatically add the badge for the corresponding
stats grade, along with the latest version number. All other processes within
the `ropensci-review-bot` rely on the external endpoint, `stats_badge`,
defined [by the `stats_badge()` function in
`ropensci-review-tools/roreviewapi`](https://github.com/ropensci-review-tools/roreviewapi/blob/main/R/srr.R).
That function reads the version from the badge at the time of approval, or
defaults to the latest version for packages still under review and not yet
approved.