User manual for Highlight&Go
Highlight&Go is an extension which supports the data extraction phase of a Systematic Literature Review (SLR) or Mapping Study (MS). It records your highlights while reading primary studies, and seamlessly populate the Google Spreadsheet of your SLR/MS.
- To configure the tool and start working is required at least one user (the reviewer). However, following the SLR/MS guidelines [Kitchenham15] it is recommended to involve multiple users (at least one reviewer and one checker to reduce bias) to conduct a data extraction process. The tool does not limit the number of reviewers/checkers, while it doesn't distinguish among users.
- Only one user need to do the configuration use case. Though, the rest of the team must be invited to the SLR spreadsheet.
- Allows classification of primary studies by using color-coding annotation.
- Supports annotation of primary studies on the web, identified by DOI and static URL (for cases without URL redirection), for ACM, IEEE, Springer, ScienceDirect (web version) and Dropbox.
- Supports monovalued and multivalued coding.
- Supports conflict detection for monovalued and multivalued facets.
- Allows detection of facets reviewed by multiple reviewers.
- Supports checking of classification using hyperlinks to the realm where the data extraction activity is conducted.
- Supports validation of classification.
- Supports auditability of data extraction and checking activities through some views in Google Sheet
- (Soon) Will support generation of some charts and visualizations used to report SLR and MS
Changes with screenshots in latest major release here: https://github.com/onekin/WacLine/releases/tag/HighlightAndGo-v0.3.0
A head-start presentation showing most general functionalities of Highlight&Go can be found here: https://docs.google.com/presentation/d/1J0_hYJF7yM38yNtcmiyTY1IRyjBQJjbC88zvqYMirRI/edit?usp=sharing
The rest of this documentation refers to some specificities that maybe you can deal with during use of Highlight&Go or advanced options that you may require to use.
Small starting guide can be found here: https://docs.google.com/presentation/d/1yiJLCSbYuixLXzK7cMTlnrRtdub9aXDx3LLc6TZYloY/edit#slide=id.p This video explains what is Highlight&Go and what for can be used: https://www.youtube.com/watch?v=OIQputVTfvE&feature=emb_logo
Reviewers and checkers must have an account in Google Sheets and Google Chrome synced with your Google account. In the future we want to re-support annotation using Hypothes.is and local storage.
You should access literature using preferably DOIs, but it is possible to access via URLs or local documents. We recommend you to store and share among reviewers a list of DOIs/URLs with papers to be reviewed (e.g.: in a Spreadsheet or Mendeley group) and access documents always from this list.
- DOI: the common identifier for white literature. Most part of the
articles have a DOI, and the tool supports a link with the form
"https://doi.org/". DOIs are supported for:
- DOI links which send you without redirection to the resource to be annotated (a raw PDF or HTML version of the article)
- DOI links which send you to ACM portal of the article (and includes there a link to the PDF version)
- DOI links which send you to ScienceDirect web version of the article.
- URL without redirection: the common identifier for web resources on the internet. Gray literature, old articles or some articles can be identified by a static URL.
For those which have some kind of redirection, the only solution is just to store the PDFs in Dropbox (see later). If your Primary Study selection is done using digital libraries (some of them, ACM or Scopus) have the possibility to export the results to csv with all the metadata (DOI, authors, title, etc.). Importing the CSV to the GSheet can reduce the time required to define PS and its links for Highlight&Go.
Highlight&Go supports PDFs stored in Dropbox. Dropbox, by default, previews PDFs in its own previewer. It is possible to annotate, but its not suitable to work with. For that, Dropbox allow us to view PDFs using the native chrome PDF visor (rendering of the raw version of the PDF) and annotate using the highlighter. To get the raw version of a PDF requires to be publicly shared:
- In dropbox, in the file that we want to share, click on "Share" button and click on "Create a link". We don't have to give editing permissions for the file, with reading is enough.
- The URL should have more or less this format:
https://www.dropbox.com/s/lnek1vbhlmi96kj/Aalst%20-%202014%20-%20Data%20Scientist%20The%20Engineer%20of%20the%20Future.pdf?dl=0 - We must change link's last parameter from dl=0 to raw=1. The
result will be something like this:
https://www.dropbox.com/s/lnek1vbhlmi96kj/Aalst%20-%202014%20-%20Data%20Scientist%20The%20Engineer%20of%20the%20Future.pdf?raw=1 - This is the link we must add to the spreadsheet for further classification.
Facet columns hold hyperlinks to "annotation URLs", i.e. URLs that refer to the quotes of the primary studies that sustain the code decision for the facet at hand. Those are generated while annotating. Reviewers/Checkers can comment (using google sheets comments). Also they are allowed to modify the spreadsheet cells or modify content as they wish, but taking into account that those modifications will be overwritten if you make changes (create or delete) on the annotations for that facet in that primary study.
The mapping is the process where the group of reviewers characterized primary studies along the different facets, where each facet decision can be taken at different times.
The classification process can be done using different procedures [Kitchenham15]:
- Independent extraction by two (or more) reviewers.
- For a lone researcher, taking a test-retest approach and comparing outcomes.
- For a lone researcher, such as PhD student, engaging a member of the supervisor team to extract data for a sample of studies.
Additionally, it is specially recommended for qualitative data, to extract data working together.
As we mentioned before, Highlight&Go allows user to highlight Primary Studies' content. The reviewer, after setting up the tool, will go for each of the PS by clicking on the link of the Spreadsheet. After activating the extension button, the extension will allow to start highlighting content on the article. When user selects some text content, a left sidebar will be prompted. There will appear the different facets (and its defined codes) [See Figure...]. For those facets without any classification codes (thematic analysis ones), a single button will be displayed. Those buttons are used to categorize the selected content.
[Kitchenham15] distinguishes two data types depending on the nature of the Systematic Reviews:
- Quantitative Systematic Reviews: data is most commonly in numerical form although it may include some qualitative data relating, for example, to the context of a primary study.
- Qualitative Systematic Reviews and Mapping Studies: data extraction and aggregation may be performed iteratively with the classification schemes.
Depending on the data type, the result included in the spreadsheet will be slightly different. If there is a facet with defined categories, the data shown in the spreadsheet is the code which classifies each PS. For inductive facets, the text placed on the spreadsheet facet value will be the highlighted text itself. In addition to the value of the cell, as we mentioned before, a hyperlink which redirects to point of the highlighted text is set.
[Al-Zubidy17] states that to reduce bias it is needed a collaboration among multiple researchers. One of the functionalities detected in this collaboration process is the support for conflict resolution. Conflict resolution support is required when multiple reviewers are working on the same Primary Study and they differ in the classification of the primary study's facet. During data extraction, if multiple reviewers are involved, conflicts could arise. A conflict is defined as: when 2 or more reviewers have discrepancies in the classification of a certain facet for a certain primary study. Highlight&Go infers from annotations made by the reviewer team the possible conflicts. Conflicts are detected for monovalued and multivalued facets, when the selected codes for a PS are not the same for all the reviewers. It is displayed in red color in the spreadsheet. The conflict can be resolved deleting the annotations which belongs to the incorrect coding or just validating the correct one. This process should be done after an agreement of the reviewers/checkers.
The data validation is done by a checking process. There are no rules of the number of checks required on a SMS/SLR. A terciary study [Budgen18] mentioned that usually secondary studies don't not clear mentioned the percentage of checks, but it states (as an example) that 5% is very weak. The common procedure is to have more than one independent reviewers which later on resolve any differences better than only one reviewer.
To validate a code for a facet in a PS, the user clicking on the link will go to the paper and the highlighted rationale. There data checkers can validate the mappings so far. The highlighter behaves as a kind of index on top of the set of highlights conducted for this paper, no matter the reviewer person. First, checkers can filter out highlights based on the reviewer (by default all the annotations are shown). In this way, they can know who did what. Second, highlights are indexed by theme: click on a code button for the browser to scroll up to displaying the first paragraph that endorse this code. If this paragraph pertains to a ring, then click on the paragraphs themselves to move along the ring. At any moment, the data checker can either validate the code (right click -> reply) and choosing the agreement/disagreement and tool enables that annotation as a discussion process where users can reply until an agreement is reached (thumb up).