Testing on 20 doctors

[betatim]

This issue is to track what needs to be done until and things that happen during us distributing the app to 20 doctors at USB. We don't have to discuss everything here but would be good to link to it so there is one place to go to for finding all things related to this.

To do (what needs to be done before we can gogogo):

• five test users from the team
• generate 20 unique links
• short email text
• list of issues that need closing: nothing?
• reset spreadsheet and form

(feel free to edit this comment)

[betatim]

@kmader I'd use the text and links from the hackmd to send to our pre-testers. After that send to the 20 doctors.

[kmader]

do we need code names? and if we need code names can we have an easy to parse differentiator between Radiologist vs MD vs Layman (vs Troll?) in the name?

[kmader]

email address for 20 basel doctors

import urllib.request
from urllib.parse import urlencode
from bs4 import BeautifulSoup
import string
from IPython.display import Markdown
rad_url = 'https://www.unispital-basel.ch/das-universitaetsspital/bereiche/medizinische-querschnittsfunktionen/kliniken-institute-abteilungen/departement-radiologie/kliniken-institute/klinik-fuer-radiologie-und-nuklearmedizin/team/assistenzaerzte-und-fellows/'
commit_id = '93b21d289677d14666cb91db1bd874138221bd21'
make_muapp_querystring = lambda user_id, notebook_path='anno_app.ipynb': urlencode({'urlpath': '/muapps/{}/{}'.format(user_id, notebook_path)})
template_url = 'https://mybinder.org/v2/gh/chestrays/jupyanno/{commit_id}?{qs}'
res = urllib.request.urlopen(rad_url)
soup = BeautifulSoup(res.read(), 'html.parser')
all_vcards = soup.findChildren(**{'class':'vcard'})
emails = ['{}@usb.ch'.format(c_card.a.text.split('@')[0]) for c_card in all_vcards]
emails_dict = {string.capwords(email.split('@')[0].replace('.', ' ')): email for email in emails}
out_str = ''
def make_markdown_links(name, email, prefix):
    username = ('{} {}'.format(prefix, name)).replace(' ', '_')
    annotation_url = template_url.format(commit_id=commit_id, qs=make_muapp_querystring(user_id=username))
    results_url = template_url.format(commit_id='master', qs=make_muapp_querystring(user_id=username, notebook_path='dashboard.ipynb'))
    return f'\n# {name}\n - [Annotation Link]({annotation_url})\n - [Results Link]({results_url})\n'

for name, email in emails_dict.items():
    out_str += make_markdown_links(name, email, prefix='Rad')
Markdown(out_str)

5 Beta Testers

other_people = [('Tim Head', 'tim@wildtreetech.com', 'Data')]
other_people += [('Kevin Mader', 'k@4quant.com', 'Data')]
other_people += [('Ivan Nesic', 'ivan.nesic@usb.ch', 'Data')]
other_people += [('Joshy Cyriac', 'joshy.cyriac@usb.ch', 'Data')]
other_people += [('Flavio Trolese', 'flavio.trolese@4quant.com', 'Data')]
other_people += [('Joachim Hagger', 'joachim.hagger@4quant.com', 'Data')]
other_people += [('Bram Stieltjes', 'bram.stieltjes@usb.ch', 'Rad')]
for name, email, prefix in other_people:
    out_str+=make_markdown_links(name, email, prefix=prefix)
Markdown(out_str)

[betatim]

Email sent.

[kmader]

Feedback Trial 1

Reviewer 1

1. On IE, I got minutes long loading sign, worked on Firefox. internet explorer support #36
2. Blue box stays on previous answer, could be a bias inducer Disable answer buttons while loading #26
3. For Pneumothorax, zooming would be good zoom feature of images #18
4. Due to lack of description, I do not understand the result page update dashboard #20

Reviewer 2

• I think its ok to leave out a "cant tell" button and ask for the most likely answer. However we have to keep in mind that in some cases, "cant tell" is clearly the correct answer (e.g. question "cardiomegaly?" and heart boundaries not visible due to another pathology). But with many people annotating, these cases will probably unmask themselves by the inconsistent response behaviour of the readers. introductory text explaining task #35

• Question: "Pneumothorax?"- Due to the fact that its not possible to zoom and window the images, a pneumothorax cant be excluded in any of the cases- maybe we should add this functionality -or- shouldnt insert pneumothoraces -or- should prepare a short introductory text that states that we are only looking for the most likely answer. introductory text explaining task #35 zoom feature of images #18

• Question: "Effusion?"- For a reliable diagnosis, the information if the image was taken in a standing/sitting/lying position is essential. But, again, its not so relevant if we ask for the most likely answer. introductory text explaining task #35

• Question: "No finding?"- We should define that more closely in the introductory text, e.g. "you will be asked if there is one of the following findings: pneumothorax, effusion, cardiomegaly - or none of the three". Because you will have some finding in almost every image (in this stack it was an electrode of a pacemeaker, but it can also be an old fracture or clip material after previous surgery. introductory text explaining task #35