How to stop screening?

[MartijnUX]

This first post is continuously updated based on the discussions in this thread

In the active learning cycle, the model incrementally improves its predictions on the remaining unlabeled records, but hopefully, all relevant records are identified as early in the process as possible. The reviewer decides to stop at some point during the process to conserve resources or when all records have been labeled. In the latter case, no time was saved and therefore the main question is to decide when to stop: i.e. to determine the point at which the cost of labeling more papers by the reviewer is greater than the cost of the errors made by the current model (e.g., Cohen, 2011). Finding 100% of the relevant papers appears to be almost impossible, even for human annotators(Wang, Nayfeh, Tetzlaff, O’Blenis, & Murad, 2020). Therefore, we typically aim to find 95% of the inclusions. However, in the situation of an unlabeled dataset, you don’t know how many relevant papers there are left to be found. So researchers might either stop too early and potentially miss many relevant papers, or stop too late, causing unnecessary further reading(Z. Yu, N. Kraft, & T. Menzies, 2018a).

There are potential stopping rules which have to be implemented, estimating the number of potentially relevant papers or finding an inflection point(Cormack & Grossman, 2015, 2016; Kastner, Straus, McKibbon, & Goldsmith, 2009; Stelfox, Foster, Niven, Kirkpatrick, & Goldsmith, 2013; Ros, Bjarnason, & Runeson, 2017; Wallace et al., 2010, 2012; Webster & Kemp, 2013; Yu & Menzies, 2019).

Another option is to use heuristics (Bloodgood & Vijay-Shanker, 2014; Olsson & Tomanek, 2009; Vlachos, 2008), for example:

Time-based strategy: If you choose a time-based strategy, you decide to stop after an x amount of time. This strategy can be useful when you have a limited amount of time to screen.

Data-driven strategy: When using a data-driven strategy, you e.g. decide to stop after an x amount of consecutive irrelevant papers (this number can be found in the statistics panel). Whether you choose 50, 100, 250, 500, etc. is dependent on the size of the dataset and the goal of the user. You can ask yourself: how important is it to find all the relevant papers?

Mixed strategy: Another option is to stop after an x amount of time unless you exceed the predetermined threshold of consecutive irrelevant papers before that time.

Below we discuss more options in detail. Join the discussion!!

Some useful references:

• Bloodgood, M., & Vijay-Shanker, K. (2014). A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. arXiv preprint arXiv:1409.5165.
• Cohen, A. M. (2011). Performance of Support-Vector-Machine-Based Classification on 15 Systematic Review Topics Evaluated with the WSS@95 Measure [10/cskz4h]. J Am Med Inform Assoc, 18(1), 104-104. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3005879/
• Cormack, G. V., & Grossman, M. R. (2015). Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review. http://arxiv.org/abs/1504.06868
• Cormack, G. V., & Grossman, M. R. (2016). Engineering quality and reliability in technology-assisted review. Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, 75-84.
• Kastner, M., Straus, S. E., McKibbon, K. A., & Goldsmith, C. H. (2009). The capture–mark–recapture technique can be used as a stopping rule when searching in systematic reviews. Journal of clinical epidemiology, 62(2), 149-157.
• Olsson, F., & Tomanek, K. (2009). An intrinsic stopping criterion for committee-based active learning. Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL),
• Ros, R., Bjarnason, E., & Runeson, P. (2017). A machine learning approach for semi-automated search and selection in literature studies. Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering,
• Stelfox, H. T., Foster, G., Niven, D., Kirkpatrick, A. W., & Goldsmith, C. H. (2013). Capture-mark-recapture to estimate the number of missed articles for systematic reviews in surgery. The American Journal of Surgery, 206(3), 439-440.
• Vlachos, A. (2008). A stopping criterion for active learning. Computer Speech & Language, 22(3), 295-312.
• Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. Proceedings of the ACM International Health Informatics Symposium (IHI), 819-824.
• Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010). Semi-Automated Screening of Biomedical Citations for Systematic Reviews [https://doi.org/10.1186/1471-2105-11-55]. BMC Bioinform, 11(1), 55-55. https://doi.org/10.1186/1471-2105-11-55
• Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020). Error rates of human reviewers during abstract screening in systematic reviews. PLoS ONE, 15(1), e0227742.
• Webster, A. J., & Kemp, R. (2013). Estimating omissions from searches. The American Statistician, 67(2), 82-89.
• Yu, Z., Kraft, N., & Menzies, T. (2018). Finding better active learners for faster literature reviews. Empirical Software Engineering. https://doi.org/10.1007/s10664-017-9587-0
• Yu, Z., & Menzies, T. (2019). FAST2: An intelligent assistant for finding relevant papers. Expert Systems with Applications, 120, 57-71.

[LHofstee]

It is recommended to decide on a stopping rule before starting the screening process. Your stopping rule can either be time-based, data-driven or a mix of these two.

Time-based strategy: If you choose a time-based strategy, you decide to stop after an x amount of time. This strategy can be useful when you have a limited amount of time to screen.

Data-driven strategy: When using a data-driven strategy, you e.g. decide to stop after an x amount of consecutive irrelevant papers (this number can be found in the statistics panel). Whether you choose 50, 100, 250, 500, etc. is dependent on the size of the dataset and the goal of the user. You can ask yourself: how important is it to find all the relevant papers?

Mixed strategy: Another option is to stop after an x amount of time, unless you exceed the predetermined threshold of consecutive irrelevant papers before that time.

[JJBDub]

Howard et al. (2020) have proposed a model to estimate recall that could provide a less heuristic stop rule. It applies to the sequential screening of a ranked list of references. They claim that it may or may not be highly accurate in all cases, but is conservative in such a way that the predicted (modeled) stop point based on a set recall threshold would always be predicted to be after the true set recall threshold has exceeded the set threshold.
I would be very curious to know what this community thinks of their claim.

https://doi.org/10.1016/j.envint.2020.105623
See section 2.4: " Here we describe a simple statistical method to estimate recall given a ranked list of references. We assume that a screener begins screening at the head of the list, sequentially providing a label to each item: “Included” or “Excluded.” We are interested in estimating when the screener has achieved a given level of recall (say, 95%) so that s/he can stop screening and realize the benefits of document ranking (while also retaining confidence that most of the relevant documents have been discovered).
The method works by examining the lengths of consecutive spans of excluded documents that occur between each relevant document during screening. The lengths of these spans provide a basis for estimating the local probability of document relevance........"

The estimation method described in the paper makes it possible to generate this type of graphical representation

.

[MaxvanHaastrecht]

I've used ASReview for quite a few reviews now, here is my take.

First of all, there are some really interesting papers discussing this topic. My personal favourite is Cormack & Grossman from 2016: https://doi.org/10.1145/2911451.2911510. But Yu & Menzies also discuss the topic of when to stop (and how to start) quite extensively: https://doi.org/10.1016/j.eswa.2018.11.021.

One thing to note for both is that they are not focused purely on ASReview. This means that although they propose some interesting approaches, they do not have to be the best approaches for ASReview. I discuss my own favourite way to choose a stopping criterion in this video (from about 04:45): https://www.youtube.com/watch?v=HBeAWIsoYiE. The video explains a systematic review methodology incorporating ASReview.

In short, I like screening a predefined set of papers randomly, and using the observed fraction of relevant papers to extrapolate an estimate of relevant papers for the complete set. You can then stop when you have reached (for example) 95% of this estimate. To avoid a case where you grossly overestimate the number of relevant papers and end up screening everything, I usually use a backup stopping criterion (e.g., 20 in a row irrelevant). The neat thing about this approach is that you can use the estimation procedure as your starting procedure for ASReview, since you evaluate papers randomly and will (hopefully) have both relevant and irrelevant papers in your set. Two birds with one stone!

[j0sien]

Nice, Max. I want to apply your strategy too. My supervisor asks for some kind of reference on this methodology (to show that this is a good way of formulating a stoping rule when using asreview). do you have any reference to this strategy? or a published paper? @MaxvanHaastrecht

[j0sien]

@MaxvanHaastrecht how to determine the number of records to screen manually to determine the percentage?

[MaxvanHaastrecht]

Hey @j0sien, thanks for the reply! For a short description of the method using a fraction of relevant papers you can take a look at our paper: https://doi.org/10.3389/frma.2021.685591. I should add the caveat that I like this approach for its simplicity, not because it gives any statistical guarantees. For a more rigorous approach, you could consider one of the other methods discussed in one of the ASReview threads. Here we cover the Cormack & Grossman approach: #1115.

You could also consider formulating a statistical argument for 'the 'simple method'. Evaluating the random papers that ASReview presents yields a Bernoulli process, which means the underlying distribution you are considering is a Binomial (the caveat is that this is not really true, since ASReview will only present 'random' papers that you have not yet evaluated). Since the Binomial approximates the normal under certain conditions, you could use normal distribution confidence intervals to add a statistical argument. The danger is that if you are too conservative, you might end up needing to evaluate a bunch of papers.

I would just discuss the possibilities with your supervisor. In the end, it is 'only' a stopping criterion. Factors like how you set up your database search and which databases you search also influence what you end up having in your final set of inclusions.

[RosannaNagtegaal]

@MartijnUX, where can I find the statistics panel showing the 42 irrelevant records you were referring to?

[J535D165]

In the statistics panel (below the charts), in the review screen. See the 106 for example in this screenshot: https://asreview.readthedocs.io/en/latest/lab/oracle.html#start-reviewing.

[MartijnUX]

@RosannaNagtegaal This is the visual from the current version of the statistics panel.

[vissereva]

For a scoping review I was wondering if a rule like the following would be advisable or not:

Screening saturation (the point we stop screening) is defined in this scoping review when ASReview gives 1% of the total found papers with (a minimum of 25) as consecutive non-relevant papers. When this point is reached we choose to end the screening. (For example if 1500 articles are found, when 25 consecutive articles are not relevant anymore the screening ends; when 3000 articles are found, when 30 consecutive articles are not relevant the screening ends).

[boer0107]

I don't think that a stopping rule for a scoping review will differ from a stopping rule for a systematic review, because the screening process for both types of reviews is similar. Having said that...
I would personally choose a combination of a minimum % of papers to be screened and a certain number/percentage of consecutive irrelevant records like you have described. The simulation data that we have published (https://doi.org/10.1038/s42256-020-00287-7) show a range of screen 8 to 33% of the papers screened before finding 95% of the relevant studies. Depending on the time you have at hand an how thorough you want your scoping review to be you could choose a minimum amount of papers to be screened.

[vissereva]

Thanks a lot for this quick and comprehensive reply! This helps a lot :)

[vissereva]

The stopping rule I now composed (based on your mentioned article and strategies chosen in earlier published articles with ASreview) is this:
The stopping rule for title-abstract screening in this scoping review is defined as: at least 33% of all the papers found in the search are screened AND when ASReview gives 25 consecutive non-relevant papers. When this point is reached we choose to end the screening .

Is this in you opinion an improvement?

(I did read the comments below, for I'm not a data scientist/statistical expert/have knowledge on active learning models, the above stated is my best try at incorporating the knowledge)

[JJBDub]

My intuition is that setting a fixed proportion of the screening set could work quite well, but only if you have worked previously with similar sets, in the same discipline, on similar topics, using the same AS-Review settings, and found from that experience that 33% was sufficient.
OR maybe there is now enough data from diverse projects carried out using AS-Review, such that we have an empirical bound on the proportion of records that must be screened to ensure given recalls. In other words, we may know that in all the projects ever carried out with AS-Review, the least efficient of them all reached, for example, 85% recall after screening 33% of the set. In that case, if your recall requirement is, say, 75%, you would feel quite reassured that you reached it after screening 33%.
BUT does that sort of data on all projects carried out with AS-Review exist yet?

As for stopping after a single string of consecutive misses (e.g. 25 misses), I would propose an improvement.
Track the count of misses between hits, starting with the first record you screen. For example: 3 misses before the first hit, 2 misses before the second hit, 0 misses until the next hit, then 10 misses, etc.... That number will bounce around a lot, but steadily increase. As we all observe, if the ranking model did its job, the count of records between hits grows as you proceed down the ranking.
Rather than waiting for some pre-determined number, monitor the rate of increase in that count, and only then decide what your stopping count of misses will be. Also, because of the bouncing in that count, the danger is that you would stop at the first count of 25 misses, even though the next count of misses may be much smaller. Therefore consider for example a moving average of n counts between hits, rather than a single string. So for example, record the count of misses between hits, take a moving average of three counts, and stop when that moving average gets to a preset threshold.

Again, this is advice based on experience and intuition, not on a rigorous study. The materials at the top of this discussion may provide more rigorous arguments.

[JJBDub]

Same as Jan. I don't see why a scoping review should be different from any review. That said, if we could generate a credible estimate of recall, then we could stop whenever that estimate reaches the recall that is sufficient for the particular project, AND we would be able to generate some measure of confidence in that interval. Work has been published on estimators of recall, and at least one team has implemented theirs for the purpose of providing quantitative stopping information (Howard et al. 2020 https://doi.org/10.1016/j.envint.2020.105623). They think the estimate they generate is valid and conservative, but I would love to have the opinion of the AS-Review developers about that. Note that Howard et al.'s estimator of recall is based on the statistical properties of the gap between successive positive records in an ordered queue, which is also the basis for the heuristic proposed by Jan.

[Rensvandeschoot]

(I removed some of the html code from the initial message)

[Rensvandeschoot]

The method described in Howard et al., works under the condition the records follow a negative binominal distribution, which might make sense for some datasets, but maybe not for others. The problem is you do not know the true distribution before observing the labels. So, for labeled datasets it is fairly easy to apply their method, but not if you start from scratch.

The method also assumes there are no clusters of records, which might be true for some datasets, but not for all. Such clusters can be observed in a recall plot whenever a plateau occurs after which a new batch of relevant records is found.

Also, the method only works for their implementation of the feature extraction (TF-IDF), but not for more advanced feature extraction techniques like Doc2Vec or sBert. I also don't think it will work for neural nets and transformer methods, but I am not sure (I did not test their method).

It would be interesting to test their method on many datasets with other characteristics as they used (would be a nice simulation paper!).

Hopefully to be continued with some simulation results!!

[JJBDub]

In their implementation, we repeatedly notice the "clumping" you mention, where the positive records are not smoothly spread out throughout the queue. We often have "streaks" of positive records followed by "droughts". The recall estimate gets recomputed every time the ranking gets recomputed (every 30 records or by user request). It seems as if the recall estimation just gets adjusted as the work passes through plateaus. I personally notice the similarity between screening data and time-to-event data, such as survival data or reliability data. I have used time-to-event methods to compare the efficiency of different ranking models. Parametric time-to-event methods can generate an estimate of the endpoint of the process (100% events). I wonder if this has been described.

[Rensvandeschoot]

Sounds extremely interesting! Would you like to join one of our weekly meetings in GatherTown? Every Thursday at 1100 (CET) we discuss ongoing discussion and everyone is welcome to join! just send your email address to asreview@uu.nl and I'll add you to the list

[JJBDub]

Thank you for the offer!! I am on the US east coast, so wouldn't that be 5:00 am? That is way too early for me, but the offer is extremely tempting. Add me to the list and I'll see if I can manage to wake up.

[Rensvandeschoot]

will do!
(I deleted your phone number from the post)

[JJBDub]

Those are three great sources. Thank you. Your suggestion of using a preliminary random sample to estimate the population frequency of positives is intuitively very attractive. However, our work deals routinely with populations of 100000-300000 references, among which there may only be 1000-3000 positives. The size of the random sample that would give us a credible estimate of total recall that is also reasonably accurate may be prohibitive. Or it may be acceptable, especially if the margin of error we require is not too small.

[Rensvandeschoot]

I am also not so sure if a random selection makes sense. In most datasets, the percentage of relevant records is <5% of the data and often the total number of relevant records used for the meta-analyses/review is <50 (sorry, just a guess and no reference.... but there probably exists one). Therefore, it would take ages to find enough relevant records to estimate something and you could have stopped much sooner by simply using active learning from the start.

You could combine different query strategies in ASReview: combining max with random or uncertainty:

query.MaxRandomQuery	Mixed (95% Maximum and 5% Random) query strategy.
query.MaxUncertaintyQuery	Mixed (95% Maximum and 5% Uncertainty) query strategy.

[MaxvanHaastrecht]

@JJBDub, very good point, I hadn't really thought about it that way. In principle, software engineering has similar percentages of relevant papers (see e.g., Yu & Menzies 2019). But I'm not sure if Cormack & Grossman had 1% relevant papers in mind when they were proposing their approach. They have a section (4.2) on 'Adjustment for Low Prevalence' but I cannot immediately see how it solves the problem of your setting. Regardless, it may be worth it to read the paper to guide your own ideas.

Of course, if you had a reliable estimate for the percentage of relevant papers that traditionally occur in your field, you could take a more 'Bayesian' approach. You could, for example, update the base percentage based on the relevance percentage you observe in your initial starting procedure. This may help to solve the credibility issue of the estimate, but admittedly doesn't quite address the concern of spending too much time that @Rensvandeschoot also mentioned.

[boer0107]

@Rensvandeschoot we do have more then a guess, our analysis of systematic reviews performed in Utrecht in 2020: https://doi.org/10.5281/zenodo.4725568. We have analyzed 117 systematic reviews and were able to obtain the necessary data on numbers of screened and included titles for 113 of these reviews. 87 of these have less then 5% inclusions. 46 reviews even have less then 1% inclusions.

[j0sien]

You are talking about large datasets.
Does this also apply to smaller datasets (5k-10k)?
Or could you in that case assume that if I take a random sample of 2.5% of the data and screen them manually, I can get a reliable estimate of the percentage of relevant papers in the total dataset?

[boer0107]

You can also check the stopping rules used by other researchers. We have a list of systematic reviews where ASReview was used as a screening tool and the authors have reported their stopping rule.

[JJBDub]

This is very useful! Thank you for the link.

[Rensvandeschoot]

See also the discussion in #1115 about calculating the 'knee' criterion based on the output of the recall plot.