How to stop screening?

[MartijnUX]

This first post is continuously updated based on the discussions in this thread

In the active learning cycle, the model incrementally improves its predictions on the remaining unlabeled records, but hopefully, all relevant records are identified as early in the process as possible. The reviewer decides to stop at some point during the process to conserve resources or when all records have been labeled. In the latter case, no time was saved and therefore the main question is to decide when to stop: i.e. to determine the point at which the cost of labeling more papers by the reviewer is greater than the cost of the errors made by the current model (e.g., Cohen, 2011). Finding 100% of the relevant papers appears to be almost impossible, even for human annotators(Wang, Nayfeh, Tetzlaff, O’Blenis, & Murad, 2020). Therefore, we typically aim to find 95% of the inclusions. However, in the situation of an unlabeled dataset, you don’t know how many relevant papers there are left to be found. So researchers might either stop too early and potentially miss many relevant papers, or stop too late, causing unnecessary further reading(Z. Yu, N. Kraft, & T. Menzies, 2018a).

There are potential stopping rules which have to be implemented, estimating the number of potentially relevant papers or finding an inflection point(Cormack & Grossman, 2015, 2016; Kastner, Straus, McKibbon, & Goldsmith, 2009; Stelfox, Foster, Niven, Kirkpatrick, & Goldsmith, 2013; Ros, Bjarnason, & Runeson, 2017; Wallace et al., 2010, 2012; Webster & Kemp, 2013; Yu & Menzies, 2019).

Another option is to use heuristics (Bloodgood & Vijay-Shanker, 2014; Olsson & Tomanek, 2009; Vlachos, 2008), for example:

Time-based strategy: If you choose a time-based strategy, you decide to stop after an x amount of time. This strategy can be useful when you have a limited amount of time to screen.

Data-driven strategy: When using a data-driven strategy, you e.g. decide to stop after an x amount of consecutive irrelevant papers (this number can be found in the statistics panel). Whether you choose 50, 100, 250, 500, etc. is dependent on the size of the dataset and the goal of the user. You can ask yourself: how important is it to find all the relevant papers?

Mixed strategy: Another option is to stop after an x amount of time unless you exceed the predetermined threshold of consecutive irrelevant papers before that time.

Below we discuss more options in detail. Join the discussion!!

Some useful references:

• Bloodgood, M., & Vijay-Shanker, K. (2014). A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. arXiv preprint arXiv:1409.5165.
• Cohen, A. M. (2011). Performance of Support-Vector-Machine-Based Classification on 15 Systematic Review Topics Evaluated with the WSS@95 Measure [10/cskz4h]. J Am Med Inform Assoc, 18(1), 104-104. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3005879/
• Cormack, G. V., & Grossman, M. R. (2015). Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review. http://arxiv.org/abs/1504.06868
• Cormack, G. V., & Grossman, M. R. (2016). Engineering quality and reliability in technology-assisted review. Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, 75-84.
• Kastner, M., Straus, S. E., McKibbon, K. A., & Goldsmith, C. H. (2009). The capture–mark–recapture technique can be used as a stopping rule when searching in systematic reviews. Journal of clinical epidemiology, 62(2), 149-157.
• Olsson, F., & Tomanek, K. (2009). An intrinsic stopping criterion for committee-based active learning. Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL),
• Ros, R., Bjarnason, E., & Runeson, P. (2017). A machine learning approach for semi-automated search and selection in literature studies. Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering,
• Stelfox, H. T., Foster, G., Niven, D., Kirkpatrick, A. W., & Goldsmith, C. H. (2013). Capture-mark-recapture to estimate the number of missed articles for systematic reviews in surgery. The American Journal of Surgery, 206(3), 439-440.
• Vlachos, A. (2008). A stopping criterion for active learning. Computer Speech & Language, 22(3), 295-312.
• Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. Proceedings of the ACM International Health Informatics Symposium (IHI), 819-824.
• Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010). Semi-Automated Screening of Biomedical Citations for Systematic Reviews [https://doi.org/10.1186/1471-2105-11-55]. BMC Bioinform, 11(1), 55-55. https://doi.org/10.1186/1471-2105-11-55
• Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020). Error rates of human reviewers during abstract screening in systematic reviews. PLoS ONE, 15(1), e0227742.
• Webster, A. J., & Kemp, R. (2013). Estimating omissions from searches. The American Statistician, 67(2), 82-89.
• Yu, Z., Kraft, N., & Menzies, T. (2018). Finding better active learners for faster literature reviews. Empirical Software Engineering. https://doi.org/10.1007/s10664-017-9587-0
• Yu, Z., & Menzies, T. (2019). FAST2: An intelligent assistant for finding relevant papers. Expert Systems with Applications, 120, 57-71.

[LHofstee]

It is recommended to decide on a stopping rule before starting the screening process. Your stopping rule can either be time-based, data-driven or a mix of these two.

Time-based strategy: If you choose a time-based strategy, you decide to stop after an x amount of time. This strategy can be useful when you have a limited amount of time to screen.

Data-driven strategy: When using a data-driven strategy, you e.g. decide to stop after an x amount of consecutive irrelevant papers (this number can be found in the statistics panel). Whether you choose 50, 100, 250, 500, etc. is dependent on the size of the dataset and the goal of the user. You can ask yourself: how important is it to find all the relevant papers?

Mixed strategy: Another option is to stop after an x amount of time, unless you exceed the predetermined threshold of consecutive irrelevant papers before that time.

[JJBDub]

Howard et al. (2020) have proposed a model to estimate recall that could provide a less heuristic stop rule. It applies to the sequential screening of a ranked list of references. They claim that it may or may not be highly accurate in all cases, but is conservative in such a way that the predicted (modeled) stop point based on a set recall threshold would always be predicted to be after the true set recall threshold has exceeded the set threshold.
I would be very curious to know what this community thinks of their claim.

https://doi.org/10.1016/j.envint.2020.105623
See section 2.4: " Here we describe a simple statistical method to estimate recall given a ranked list of references. We assume that a screener begins screening at the head of the list, sequentially providing a label to each item: “Included” or “Excluded.” We are interested in estimating when the screener has achieved a given level of recall (say, 95%) so that s/he can stop screening and realize the benefits of document ranking (while also retaining confidence that most of the relevant documents have been discovered).
The method works by examining the lengths of consecutive spans of excluded documents that occur between each relevant document during screening. The lengths of these spans provide a basis for estimating the local probability of document relevance........"

The estimation method described in the paper makes it possible to generate this type of graphical representation

.

[MaxvanHaastrecht]

I've used ASReview for quite a few reviews now, here is my take.

First of all, there are some really interesting papers discussing this topic. My personal favourite is Cormack & Grossman from 2016: https://doi.org/10.1145/2911451.2911510. But Yu & Menzies also discuss the topic of when to stop (and how to start) quite extensively: https://doi.org/10.1016/j.eswa.2018.11.021.

One thing to note for both is that they are not focused purely on ASReview. This means that although they propose some interesting approaches, they do not have to be the best approaches for ASReview. I discuss my own favourite way to choose a stopping criterion in this video (from about 04:45): https://www.youtube.com/watch?v=HBeAWIsoYiE. The video explains a systematic review methodology incorporating ASReview.

In short, I like screening a predefined set of papers randomly, and using the observed fraction of relevant papers to extrapolate an estimate of relevant papers for the complete set. You can then stop when you have reached (for example) 95% of this estimate. To avoid a case where you grossly overestimate the number of relevant papers and end up screening everything, I usually use a backup stopping criterion (e.g., 20 in a row irrelevant). The neat thing about this approach is that you can use the estimation procedure as your starting procedure for ASReview, since you evaluate papers randomly and will (hopefully) have both relevant and irrelevant papers in your set. Two birds with one stone!

[j0sien]

Nice, Max. I want to apply your strategy too. My supervisor asks for some kind of reference on this methodology (to show that this is a good way of formulating a stoping rule when using asreview). do you have any reference to this strategy? or a published paper? @MaxvanHaastrecht

[j0sien]

@MaxvanHaastrecht how to determine the number of records to screen manually to determine the percentage?

[MaxvanHaastrecht]

Hey @j0sien, thanks for the reply! For a short description of the method using a fraction of relevant papers you can take a look at our paper: https://doi.org/10.3389/frma.2021.685591. I should add the caveat that I like this approach for its simplicity, not because it gives any statistical guarantees. For a more rigorous approach, you could consider one of the other methods discussed in one of the ASReview threads. Here we cover the Cormack & Grossman approach: #1115.

You could also consider formulating a statistical argument for 'the 'simple method'. Evaluating the random papers that ASReview presents yields a Bernoulli process, which means the underlying distribution you are considering is a Binomial (the caveat is that this is not really true, since ASReview will only present 'random' papers that you have not yet evaluated). Since the Binomial approximates the normal under certain conditions, you could use normal distribution confidence intervals to add a statistical argument. The danger is that if you are too conservative, you might end up needing to evaluate a bunch of papers.

I would just discuss the possibilities with your supervisor. In the end, it is 'only' a stopping criterion. Factors like how you set up your database search and which databases you search also influence what you end up having in your final set of inclusions.

[RosannaNagtegaal]

@MartijnUX, where can I find the statistics panel showing the 42 irrelevant records you were referring to?

[J535D165]

In the statistics panel (below the charts), in the review screen. See the 106 for example in this screenshot: https://asreview.readthedocs.io/en/latest/lab/oracle.html#start-reviewing.

[MartijnUX]

@RosannaNagtegaal This is the visual from the current version of the statistics panel.

[vissereva]

For a scoping review I was wondering if a rule like the following would be advisable or not:

Screening saturation (the point we stop screening) is defined in this scoping review when ASReview gives 1% of the total found papers with (a minimum of 25) as consecutive non-relevant papers. When this point is reached we choose to end the screening. (For example if 1500 articles are found, when 25 consecutive articles are not relevant anymore the screening ends; when 3000 articles are found, when 30 consecutive articles are not relevant the screening ends).

[boer0107-zz]

I don't think that a stopping rule for a scoping review will differ from a stopping rule for a systematic review, because the screening process for both types of reviews is similar. Having said that...
I would personally choose a combination of a minimum % of papers to be screened and a certain number/percentage of consecutive irrelevant records like you have described. The simulation data that we have published (https://doi.org/10.1038/s42256-020-00287-7) show a range of screen 8 to 33% of the papers screened before finding 95% of the relevant studies. Depending on the time you have at hand an how thorough you want your scoping review to be you could choose a minimum amount of papers to be screened.

[vissereva]

Thanks a lot for this quick and comprehensive reply! This helps a lot :)

[vissereva]

The stopping rule I now composed (based on your mentioned article and strategies chosen in earlier published articles with ASreview) is this:
The stopping rule for title-abstract screening in this scoping review is defined as: at least 33% of all the papers found in the search are screened AND when ASReview gives 25 consecutive non-relevant papers. When this point is reached we choose to end the screening .

Is this in you opinion an improvement?

(I did read the comments below, for I'm not a data scientist/statistical expert/have knowledge on active learning models, the above stated is my best try at incorporating the knowledge)

[JJBDub]

My intuition is that setting a fixed proportion of the screening set could work quite well, but only if you have worked previously with similar sets, in the same discipline, on similar topics, using the same AS-Review settings, and found from that experience that 33% was sufficient.
OR maybe there is now enough data from diverse projects carried out using AS-Review, such that we have an empirical bound on the proportion of records that must be screened to ensure given recalls. In other words, we may know that in all the projects ever carried out with AS-Review, the least efficient of them all reached, for example, 85% recall after screening 33% of the set. In that case, if your recall requirement is, say, 75%, you would feel quite reassured that you reached it after screening 33%.
BUT does that sort of data on all projects carried out with AS-Review exist yet?

As for stopping after a single string of consecutive misses (e.g. 25 misses), I would propose an improvement.
Track the count of misses between hits, starting with the first record you screen. For example: 3 misses before the first hit, 2 misses before the second hit, 0 misses until the next hit, then 10 misses, etc.... That number will bounce around a lot, but steadily increase. As we all observe, if the ranking model did its job, the count of records between hits grows as you proceed down the ranking.
Rather than waiting for some pre-determined number, monitor the rate of increase in that count, and only then decide what your stopping count of misses will be. Also, because of the bouncing in that count, the danger is that you would stop at the first count of 25 misses, even though the next count of misses may be much smaller. Therefore consider for example a moving average of n counts between hits, rather than a single string. So for example, record the count of misses between hits, take a moving average of three counts, and stop when that moving average gets to a preset threshold.

Again, this is advice based on experience and intuition, not on a rigorous study. The materials at the top of this discussion may provide more rigorous arguments.

[JJBDub]

Same as Jan. I don't see why a scoping review should be different from any review. That said, if we could generate a credible estimate of recall, then we could stop whenever that estimate reaches the recall that is sufficient for the particular project, AND we would be able to generate some measure of confidence in that interval. Work has been published on estimators of recall, and at least one team has implemented theirs for the purpose of providing quantitative stopping information (Howard et al. 2020 https://doi.org/10.1016/j.envint.2020.105623). They think the estimate they generate is valid and conservative, but I would love to have the opinion of the AS-Review developers about that. Note that Howard et al.'s estimator of recall is based on the statistical properties of the gap between successive positive records in an ordered queue, which is also the basis for the heuristic proposed by Jan.

[Rensvandeschoot]

(I removed some of the html code from the initial message)

[Rensvandeschoot]

The method described in Howard et al., works under the condition the records follow a negative binominal distribution, which might make sense for some datasets, but maybe not for others. The problem is you do not know the true distribution before observing the labels. So, for labeled datasets it is fairly easy to apply their method, but not if you start from scratch.

The method also assumes there are no clusters of records, which might be true for some datasets, but not for all. Such clusters can be observed in a recall plot whenever a plateau occurs after which a new batch of relevant records is found.

Also, the method only works for their implementation of the feature extraction (TF-IDF), but not for more advanced feature extraction techniques like Doc2Vec or sBert. I also don't think it will work for neural nets and transformer methods, but I am not sure (I did not test their method).

It would be interesting to test their method on many datasets with other characteristics as they used (would be a nice simulation paper!).

Hopefully to be continued with some simulation results!!

[JJBDub]

In their implementation, we repeatedly notice the "clumping" you mention, where the positive records are not smoothly spread out throughout the queue. We often have "streaks" of positive records followed by "droughts". The recall estimate gets recomputed every time the ranking gets recomputed (every 30 records or by user request). It seems as if the recall estimation just gets adjusted as the work passes through plateaus. I personally notice the similarity between screening data and time-to-event data, such as survival data or reliability data. I have used time-to-event methods to compare the efficiency of different ranking models. Parametric time-to-event methods can generate an estimate of the endpoint of the process (100% events). I wonder if this has been described.

[Rensvandeschoot]

Sounds extremely interesting! Would you like to join one of our weekly meetings in GatherTown? Every Thursday at 1100 (CET) we discuss ongoing discussion and everyone is welcome to join! just send your email address to asreview@uu.nl and I'll add you to the list

[JJBDub]

Thank you for the offer!! I am on the US east coast, so wouldn't that be 5:00 am? That is way too early for me, but the offer is extremely tempting. Add me to the list and I'll see if I can manage to wake up.

[Rensvandeschoot]

will do!
(I deleted your phone number from the post)

[JJBDub]

Those are three great sources. Thank you. Your suggestion of using a preliminary random sample to estimate the population frequency of positives is intuitively very attractive. However, our work deals routinely with populations of 100000-300000 references, among which there may only be 1000-3000 positives. The size of the random sample that would give us a credible estimate of total recall that is also reasonably accurate may be prohibitive. Or it may be acceptable, especially if the margin of error we require is not too small.

[Rensvandeschoot]

I am also not so sure if a random selection makes sense. In most datasets, the percentage of relevant records is <5% of the data and often the total number of relevant records used for the meta-analyses/review is <50 (sorry, just a guess and no reference.... but there probably exists one). Therefore, it would take ages to find enough relevant records to estimate something and you could have stopped much sooner by simply using active learning from the start.

You could combine different query strategies in ASReview: combining max with random or uncertainty:

query.MaxRandomQuery	Mixed (95% Maximum and 5% Random) query strategy.
query.MaxUncertaintyQuery	Mixed (95% Maximum and 5% Uncertainty) query strategy.

[MaxvanHaastrecht]

@JJBDub, very good point, I hadn't really thought about it that way. In principle, software engineering has similar percentages of relevant papers (see e.g., Yu & Menzies 2019). But I'm not sure if Cormack & Grossman had 1% relevant papers in mind when they were proposing their approach. They have a section (4.2) on 'Adjustment for Low Prevalence' but I cannot immediately see how it solves the problem of your setting. Regardless, it may be worth it to read the paper to guide your own ideas.

Of course, if you had a reliable estimate for the percentage of relevant papers that traditionally occur in your field, you could take a more 'Bayesian' approach. You could, for example, update the base percentage based on the relevance percentage you observe in your initial starting procedure. This may help to solve the credibility issue of the estimate, but admittedly doesn't quite address the concern of spending too much time that @Rensvandeschoot also mentioned.

[boer0107-zz]

@Rensvandeschoot we do have more then a guess, our analysis of systematic reviews performed in Utrecht in 2020: https://doi.org/10.5281/zenodo.4725568. We have analyzed 117 systematic reviews and were able to obtain the necessary data on numbers of screened and included titles for 113 of these reviews. 87 of these have less then 5% inclusions. 46 reviews even have less then 1% inclusions.

[j0sien]

You are talking about large datasets.
Does this also apply to smaller datasets (5k-10k)?
Or could you in that case assume that if I take a random sample of 2.5% of the data and screen them manually, I can get a reliable estimate of the percentage of relevant papers in the total dataset?

[boer0107-zz]

You can also check the stopping rules used by other researchers. We have a list of systematic reviews where ASReview was used as a screening tool and the authors have reported their stopping rule.

[JJBDub]

This is very useful! Thank you for the link.

[Rensvandeschoot]

See also the discussion in #1115 about calculating the 'knee' criterion based on the output of the recall plot.

[j0sien]

Hereby I would like to share my stopping rules for two rounds of screening in my review protocol.
What do you think of it? Especially the first one, I think it is quite impossible to determine it up front, so I tried to make a hybrid stopping rule, based on the information you get while reviewing. Is it clear enought? any suggestions?

The stopping rule of the first screening phase is a three-fold rule. (1) screening in this phase will be stopped when at least 25% of the records have been screened. Van de Schoot et al. (2021) showed that 95% of the eligible studies will be found after screening between only 8% to 33% of the total number of records; (2) screening will be stopped only when all key papers have been marked as relevant; (3) Based on the results of the screening of 25% of the records, a ‘knee method’ stopping criterion algorithm (Cormack & Grossman, 2016) will be determined. The knee method is a a geometric stopping procedure, based on the shape of the gain curve (i.e. recall versus effort). The recall plot generated by the software, plotting the number of identified relevant records against the number of viewed records, will be visually inspected after screening batches of 5% of the total number of records, to see whether a plateau (or ‘knee’) is reached. When a plateau is visually identified, we will mathematically evaluate to see if the slope at that earlier evaluation points is more than a predefined slope cutoff ratio (e.g. 6) as high. When this point is reached we choose to end the active learning screening phase.

Screening phase 2: Deep learning
The two reviewers (JB and BS) will work independently and each screen a minimum of 200 papers, with the stopping rule of finding a moving average of >95% irrelevant records in three consecutive batches of 50 papers .

[gspadaro90]

I have little experience with this process, but it looks like a sensible (combination of) approach to me. I would like to do the same.

Can you share a bit more about how would mathematically evaluate to see if the slope at that earlier evaluation point is more than a predefined slope cutoff ratio?

[Rensvandeschoot]

See also the discussion in #1115 about calculating the 'knee' criterion based on the output of the recall plot.

[rohitgarud]

Hello everyone, I hope you're doing well. I have recently conducted a simulation study on utilizing extracted features to develop a stopping criterion for screening. I would be extremely grateful for your expert opinions and any feedback you might have. Thank you in advance for taking the time to review my work.

[MaxvanHaastrecht]

Hey @rohitgarud, sounds like a really interesting approach. I have to say that I'm still wrapping my head around it a bit, but nevertheless wanted to give a first response.

One question that popped into my mind after reading your proposal is if there could potentially be some issues with using a stopping criterion based on text features when the active learning tool you are using to screen (ASReview) also uses text features to sort and choose the papers to present to you. Could this create issues with the robustness of your approach?

I want to stress that this is just a doubt I'm having, not a seriously thought out critique or anything. I think it would be interesting to try out your method on a variety of datasets (as you have partially already done) and compare your method to 2-3 other approaches to choosing a stopping criterion. This would also force you to think about which exact cosine similarity metric(s) you would want to use within your approach.

Having some tables/plots showing how your approach (with e.g., 3 different cosine similarity stopping rules) compares to 2-3 other methods would really help for me to figure out if this is something we should all be thinking about more. Hope this helps!

[rohitgarud]

@MaxvanHaastrecht Thank you very much for your detailed feedback. I think your concern is valid, as Asreview also uses the same features, and the ability of the ASR to suggest the next relevant record will affect the stopping criteria. I am looking into different aspects of this approach.

1. As you mentioned, whether cosine similarly a good metric for comparison
2. Can other dense vector representations like doc2vec or SBERT be used in place of TFIDF (sparse and very high dimensional)
3. Whether calculating the resultant is the right thing to do or should we use other approaches such as cross-product or centroid of the cluster (considering feature as a point in the high dimensional space)

I will present some results soon comparing the approach with existing methods in a more understandable format as highlighted by @rvdinter
Note: The proposed approach is loosely based on the feature-based information retrieval approaches

[rohitgarud]

I have created a Similarity-based classifier to test the proposed approach. The rationale behind it is if all the relevant records have similar features, the similarity classifier should be able to give a high "probability" to the next relevant records. This will also strengthen the assumption made in the proposed feature-based stopping criteria. I am testing this classifier with different feature extraction strategies such as TFIDF, Doc2vec, SBERT, etc. I think comparing the performance of the SimilarityClassifier with the NaiveBayesCLassifier and TFIDF features on benchmark datasets provide a good extrinsic evaluation strategy for the approach. Anybody interested can take a look, install it directly from GitHub from the repository, and give feedback. PRs are also welcome. Thank you.

[rohitgarud]

In order to support the stopping criteria based on the similarity of extracted features, I have performed some simulation studies (presented at #1371) to check the effectiveness of the similarity classifier with doc2vec features as compared to the default Naive Bayes with TFIDF features. Three different dimensions of doc2vec features (40, 120, 200) and three different similarity metrics (cosine similarity, dot product of unnormalised feature vectors, and Euclidean distance of L2 normalised feature vectors) are used in the simulations.
(Note: looking for collaborators to work on this, mentors for guidance and possibly publish on the pre-print server like Arxiv, please ping me if interested)

[Rensvandeschoot]

see also the pre-print "The SAFE Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses": https://psyarxiv.com/c93gq

[Emanuel-1986]

Hi,

What do you think about combining a stopping rule and the human error rate (10.76% [95% CI: 7.43% to 14.09%]) from doi: 10.1371/journal.pone.0227742.

1. First, calculate the error rate of a model by taking a random sample from the titles (calculated by using 95%CI and 5% margin of error). For example, from a RIS file containing 50K titles, one needs to screen 382 random titles.
2. Let’s imagine that after screening 382 random titles, only 1% are found as relevant.
3. Then you would expect (by chance alone) to find approximately 500 expected relevant titles (1% of 50K = 500).
4. If after using ASReview and you stop after X irrelevant titles, you find 400 titles, then you have an estimated error rate of 20%, which is more than the 10% human error rate.
5. Hence, one needs to keep screening to increase the number of included titles from 400 to, for example, 450 relevant titles, which equates 10% error rate. Considering that the lower confidence interval margin is 7%, one must screen further.

All the above numbers are imaginary.

What do you think?

Emanuel

[MaxvanHaastrecht]

Hey @Emanuel-1986, I like your idea of somehow incorporating the imperfection of human reviewing in stopping criteria. However, I do think it's far from trivial how to do this properly. For example, the study you mention (Wang et al., 2020) focuses on systematic reviews in healthcare and state in their limitations section that "the findings may not be generalizable to other systematic review questions or topics." Should we then have an estimate of human error rate for every field to make this work? Are such estimates available for other fields? I honestly would be curious to know, for example for my own field of information systems.

Another thing that plays a role in this strategy is what we define as an error. From my own experience, most disagreements between reviewers are honest differences of opinion, rather than trivial mistakes made by one of the reviewers. If the reviewers eventually reach consensus that a study should be included, did the reviewer that initially indicated an exclusion make an error? I would say this is not the right way to look at this. Both reviewers were 'correct' in their own way, and they simply clarified their prioritisations in how to decide on inclusion/exclusion.

Wang et al. (2020) themselves say: "while we call judgments in study selection that are inconsistent with the final inclusion as errors, we acknowledge that these errors could be due to poor reporting and insufficient data provided in the published abstract. Thus, they may not be avoidable and they are not the fault of human reviewers." Certainly, the quality of abstracts can cause confusion among reviewers leading to misclassification, and this is another point to take into account. I personally think the point I make above regarding honest (initial) disagreement is more fundamental to the discussion whether, and how, we can use human error rates for stopping criteria.

In short, I really like this idea, but do think that more investigation is needed to see how tractable it is in practice. Perhaps a nice idea for a BSc or MSc thesis project!

[JJBernieD]

This is not a comment on the idea Emmanuel proposes, which is quite intriguing.

This is about applying the concept of error to changes in relevance or inclusion label and disagreements regarding the relevance label of some given references. I think that's an unfortunate usage of the concept of error, because it implies that for every reference, there is a unique correct relevance label.

Inter-screener variability is evidence that in many cases there isn't. We may find it uncomfortable, but a unique correct relevance label for every reference is rarely the case. Relevance criteria are only completely void of ambiguity in the most extreme edge case of systematic review: the meta-analysis. There is however a wide range of systematic reviews that would be incomplete if they were to adopt the relevance criteria of meta-analysis, and for those reviews an absolute lack of ambiguity in relevance criteria cannot be achieved. Thus inter-screener variability can even be used as a measure of how much ambiguity remains, and how much judgement is required.

Providing common training for all screeners assigned to a set of references in order to reduce inter-screener variability is a pragmatic solution, but it only gives the set of screeners a common judgement, which may simply be the one of the subject expert providing the training.
A more principled solution is to use decision by a vote of experts who do not consult with each other, thereby aggregating differing judgements. Or else to put the final selection of references for the systematic review up for review by additional subject experts, and add or subtract according to their judgement.

This is just one aspect of screening work where, outside of meta-analyses, the desire to reduce the criteria for inclusion in systematic review to an entirely mechanical system, although appealing, may be ill-conceived.

[Syresearch346]

Stopping decision (approximate numbers to make it easier to follow) :

Hello, for a set of data counting around 5000 records, I am using ASReview in the screening for inclusion and would have questions about the stopping decision:
I am hesitating in following a) the findings in the paper of Van de Schoot et al. 2021 (33% being sufficient to find 95% of the relevant articles) or b) the SAFE procedure.
When following the SAFE procedure :

In phase1: the random screening on 1% led to 25% of relevant records which means around 1250 --> Can I try another random screening as many decisions were taken in the meantime and the inclusion and exclusion criteria were updated 

In phase 2: I understood I should screen 50% (2500)? (PS: I would like to clarify that we used databases that do not distinguish between article types, so although the reviews are not included in the study, the ones relevant to the topic were labeled as RELAVANT in order not to induce the algorithm in error)
                 I would like to ask how to define **key articles** (from the data base or pre-known ones)
                 
In phase 3: which model should I choose? I saw some examples in the table, should I randomly select one of them?

In the actual screening with ASReview, I screened 33% (around 1600) and labeled 15% of relevant records, I haven´t yet reached 50 irrelevant in a row.
Since screening 2500 within the time frame I have is not very realistic, I would like to ask how safe it is to stop with the current stage / or to only reach 50 irrelevant.

Thank you

[Rensvandeschoot]

It depends on how important it is to find all relevant papers, whether finding most of the relevant papers is enough, and how certain you want to be. At the very minimum, I would suggest adhering to your stopping rule of 50 consecutive irrelevant papers.

*Be aware that finding 100% of the relevant papers is a challenging task due to human error and screening fatigue, as well as the fact that relevant papers may not even be part of the dataset due to limitations in the search.

[Syresearch346]

Thank you for your answer.

• Do you think I still can try another random screening of 1% (as I learnt throughout the screening and took decisions that might lead to different results than an initial random screening), or would I be biased?
• And would you recommend a good way to identify key articles?

[Rensvandeschoot]

What is the probability you will find a relevant paper when screening randomly? :-)
A better approach would be switching to a different model that takes context into account, see https://doi.org/10.3389/frma.2023.1178181

[Syresearch346]

From the random screening I meant to check again the percentage of the relevant/total to see whether it changes (not to find more relevant papers). The reason is that I would like to check whether I can still follow the SAFE procedure within the time frame

Thank you for sharing the paper, checking it now

[FelixWdm]

* And would you recommend a good way to identify key articles?

Dear @Syresearch346 , finding or identifying key-articles requires some expertise in the field of the systematic review. There are no strict criteria to determine whether a paper is a key-paper or not. But typically this is a paper that clearly meets all the inclusion criteria and can be seen as an example of what the review (question) is about. You could look at citations, but this favours older papers a lot. I hope this helps.

[jen-pbs]

Hello everyone!
I am conducting a systematic review of preclinical studies, I am going to use active learning during the title and abstract screening (almost 6000 references) and I will follow the SAFE procedure to set my stopping criteria. I am mostly all set with the criteria except for the fact that I am unsure what the number of consecutive irrelevant references before stopping should be. The SAFE procedure proposes 50 references, but in one of the simulations I run with 10% of the references manually screened, I observed that 60 irrelevant references were screened before finding a relevant paper.
Could the ATD (Average Time to Discovery) metric obtained in simulations of the model with the data can be used as guidance? in my simulations the ATD ( different prior knowledge of relevant records) was on average 80.
If not ATD, what can I use to make an informed decision regarding the number of irrelevant records before stopping.
Thanks a lot before hand

[j0sien]

You could use the metric from step S - screen a random subset;
e.g. 1%
you have 60 irrelevant - 1 relevant: 1/61

see also the 'cheat sheet' in step 2:
https://ndownloader.figstatic.com/files/48529133

Custom SAFE procedure Cheat Sheet
Random subset

•
Screen a minimum of
_%
of the
total number of papers;
•
Find at least
one relevant record
.
Complete dataset
Simple model
classifier
:

---

feature extractor
:

---

Labeled records phase 1
•
All
key papers
have been marked
as relevant;
•
At least
twice
the estimate of the
number of relevant records
(in your case 2 x 1/61 * total number)
records
) have been screened;
•
A minimum of __%
of the total
dataset has been screened.
•
No extra relevant records have
been identified in the last
__ record

[tayates]

I am very temped to use ASReview for a systematic review in transplant infectious disease. I am struggling a little with the 'stopping rule'! I have not completed deduplication yet, but there will be approx 15k titles and abstracts. I know of approx 5 relevant papers already.

From the above discussion/references, it seems there are two main approaches to deciding a stopping rule.

First, you can screen a random sample of the total then apply the proportion screening positive to the total sample size to estimate the total number of papers you are expecting to include. You can then stop screening when you have found e.g. 95% of this estimate. However, the uncertainty on the estimate presumably means you could end up targeting more included papers than truly exist?

The alternative approach is to stop when you achieve a run of x irrelevant papers. This makes more sense to me.

Presumably you could stop when the upper limit of the binomial confidence interval on (1/number screened since last positive) x remaining papers to screen is <5% of the number that have screened positive? That would be conservative, as the algorithm places papers most likely to be relevant at the top of the pile.

Are there other conservative stopping rules based on data/calculations that I might apply. I think I am happy with finding 95% of relevant papers, as I can probably mop up the rest by emailing corresponding authors and going through the reference lists of included papers.

Thanks!

[j0sien]

Hi Tayates, see the post above!
We suggest to combine several heuristics to be 'safe'
see also: https://ndownloader.figstatic.com/files/48529133

• At least twice the estimate of the number of relevant records records ) have been screened; (based on preliminary screening)
• A minimum of __% of the total dataset has been screened. (specify youself, e.g. 10%)
• No extra relevant records have been identified in the last __ record (specify youself, depending on the field of study this might be strict or loose, but 50 is a good starting point)

[Syresearch346]

Phase 3 of safe procedure:
To perform phase 3 should I create a new project or continue with the same and change the settings?
I would appreciate it if someone could help me how to exactly proceed step by step?
Thank you

[lastoel]

Hi @Syresearch346, thanks for your question.
In the current version of ASReview, it is indeed required to create a new project for performing phase 3 of the SAFE procedure.

The steps are as follows:

1. Export your dataset from phase 2 of your project: Export --> Dataset (including all labeled and unlabeled records), in whatever file format you prefer.
2. Import that partially labeled dataset into a new project in Oracle mode with new model settings. ASReview will automatically recognise the previously labelled records as priors.
3. Continue screening the remainder of the previously unlabeled records, until you have reached a stopping rule of your choosing.

Hope this helps!

Note: A new version of ASReview is currently being developed, in which switching model within the same project will be an option. This version will be released later this year. If you'd like to read more on model-switching strategies and their efficacy in finding hard-to-find records, check out this paper by @jteijema.

[Annasmits]

Hi, I have a question about the SAFE procedure.
I am a human geographer with little statistics and datascience expertise. I am working on a scoping review and after deduplication I now have 7376 sources to screen. I have already started the title-abstract in ASreview and had some callibration sessions with my supervisors. Then I found the article about the SAFE procedure and I think it is good I follow this best I can seeing I don't feel secure about my knowledge to determine a stopping rule/heuristics myself. Now I have one question about phase 1: is it possible to carry that out using the prior knowledge function when preparing your ASreview project? Or should I do the labeling of the random dataset in a different program and then export it to ASreview? If yes, is there a program you recommend?

PS: I have read the discussions above, but can not follow everything:( hopefully I am not posting a question that has already been answered

[Annasmits]

As I have to be sensible about the time I invest and I am really someone who can loose herself in a search for the best/perfect choice, I am now starting a new project in ASreview and planning to label 74 random sources in the prior knowledge. Then I want to estimate the RR_T by using the FRR_T from these 74 sources and move on to phase 2. I know that there are much better procedures but I also have to start focusing on progressing with the scoping review as my supervisors (and me too;)) are wary of me staying in this design phase for too long.
So even though it's not ideal.. am I doing anything 'incorrect'? (I will be transparent about not using a stratisfied selection of records in this phase in my method section)

[Annasmits]

I made a mistake and closed the prior knowledge function after 70 sources instead of 74, but won't start over for now as I really have progress the screening. 22 relevant sources out of 70!

[j0sien]

Hi Anna,
Thanks for your question:
so if you found 22/70 = 31% of relevant articles, then according to the SAFE procedure you would have to screen at least twice that amount so, 62% of the articles.
Are you sure you took a random subset? Because the FRR_T is quite high in your case.

[Annasmits]

Hi Josie, thanks for your reply.
Yeah I also found the FRR_T quite high and was a bit disappointed about having to manually screen so many articles.
I think it was a random subset because I did the labeling in ASreview in the prior knowledge phase of preparing the project in ASreview. but maybe I misunderstood how the 'search' function in the prior knowledge phase in ASreview works?

[j0sien]

In the new version of ASreview this is possible, but in the current one I do not know whether these are truly random.
To be 'safe' you could do a double check, and draw a random sample using a random number generator.
Just add a unique number to each reference, generate random numbers between 1 and 7376, pick the first 74, and screen those records at random.
see whether FRR_T is similar or very different.
and keep us updated about the results :)

[Annasmits]

Thanks Josien. I did the dubble check... 22 out of 74. So that at least confirms my first approach. I just have to think about my design I am afraid. Some background info: I am deliberately using a broad approach with a very extensive search query and expecting mostly qualitative studies in return. So it does make sense that quite a big chunk is relevant at least for the first round of screening. I am expecting quite a big chunk to be dismissed during full-text screening. So I either have to deal with having to screen 4000+ sources, or have to decide to apply a different stopping rule.

[mcallaghan]

Hi all, and thank you to the developers of ASReview - it's great to have open source software for ML-assisted screening.

We recently published a paper in Systematic Reviews discussing stopping rules.

In this paper, we argue that good stopping rules are absolutely vital to using machine learning responsibly, but that good stopping rules are hardly ever used, and the guidance is very much short of where it needs to be.

We recommend that:

1. We need to agree minimum criteria for stopping rules

• they should make the recall targeted explicit
• confidence estimates on meeting that target should be made explicit, and these should be reliable
• they should not have additional parameters that you need to set that affect the first two points. I.e. With the rule stop after X consecutive records, how do you set X?

In short, they should be based on good statistics, with clear, sensible, and transparent assumptions (just like the rest of the systematic review process!)

2. We evaluate rules robustly across a range of datasets and prioritisation algorithms

We want to demonstrate that rules are reliable, not just in general, but across a broad range of contexts. Reliable in this case, means that a rule targeting 95% recall with 95% confidence should result in recall less than 95% less than 5% of the time. Among reliable rules, we can select the most efficient.

3. The guidelines for systematic review need to be updated.

Cochrane and Campbell guidance on how to conduct a review is short of information on how to use ML-prioritised screening, yet many people are already using it (thanks to great software like this!). The guidance needs to help people (authors, reviews, and the general public) to understand the difference between stopping after 50 consecutive irrelevant records, and using a criterion that is based on solid statistics.

4. Platforms (like ASReview) should provide better guidance on stopping rules, and make them easy to use within their platforms

As noted, many people are already using ML-prioritised screening, but anecdotally, the majority of people are doing things like stopping after X consecutive irrelevant documents, with no justification for X. Platforms need to do a much better job at helping people to use ML reponsibly. ML is not magic, it is just a way of making predictions, and we need statistical methods to understand the risks of relying on those predictions. Platforms should therefore either give better guidance on how to use well-justified stopping criteria, or incorporate them into their software.

5. Papers should not make claims about work savings unless they incorporate a well-justified stopping method

There are a lot of papers that say our system/our algorithm achieves work savings of X%, without addressing the fact that these savings could only have been made if one had a perfect stopping method that predicted exactly when the right moment to stop was, without requiring any extra screening (we would need omniscience for this!). We have enough of these papers now, let's focus on getting stopping criteria right, as this is necessary for actual real-world work-savings

Looking forward to hearing thoughts on this. I would also like to share some resources on a stopping method I developed a few years ago, which I think has a solid statistical foundation, and would be my personal recommendation for anyone looking to solve this problem: https://mcallaghan.github.io/buscar-app/