User Evaluation

[cpfaff]

Users will be invited to evaluate an animated set of mock-ups as well as the vocabulary. The following serves as a collection of questions to be answered by the participants.

Interface:

• Is the application logically structured?
• Do we ask for to much or to less information to classify an ecologically related search object.
• Is there aspects missing that would be important to ask for?
• Is the information asked for in an appropriate way (sliders, fields, maps) or is there space for improvement.
• Are the icons meaningful
• Does the graphical user interface facilitate to provide the meta data

Vocabulary:

• Is there any important dimensions missing
• Is the content within the branches of the vocabulary to much/less

@EichenbergBEF Would be good to hear your ideas here as well.

[cpfaff]

GFGio user evaluation for the casual annotation system (CAS) for ecology

Setup

• 10 Persons
• 1 Hour time for evaluation split into half (30 Min) for each part.

Introduction

This evaluation is split into two parts. The first part deals with a user interface (mock-ups) that can be used to create annotations of search objects in the ecological context (e.g. datasets). The second part deals with a controlled vocabulary in form of a thesaurus as basis for the annotation.

User Interface:

The mock-ups of the user interface can be found here. Please browse the mock-ups and answer the questions below. Also feel free to use the comment function of Invision to leaf notes like ideas for improvement, typos in rich detail. The following questions have to be answered on a scale from 1 (fully applies), 2 (partially applies) to 3 (that does not apply).

• Readability
  • The characters on screen are always well readable (+ comment)
  • Colors and icons are understandable and meaningful (+ comment)
• Suitablity
• There is no unnecessary detail (user only gets presents with the details required for each step of the tagging process) (+ comment)
• The software does not require too many steps to reach the goal (+ comment)
• The annotation would take a justifiable amount of time (estimate) (+ comment)
• Self description
  • Meaning of displayed text (help, field labels, choices) is meaningful and does not lead to misinterpretation (+ comment)
• Control
  • The application offers a clear path to finalize an annotation (+ comment)
• Interface
  • The graphical user interface facilitates the provision of meta-data (+ comment)
  • The information is asked for in an appropriate way (sliders, maps, fields) (+ comment)

Vocabulary:

The vocabulary is a thesaurus that can be accessed online here. It has been designed in discussion with domain experts of ecology and adjacent fields. It is designed along 8 dimensions which are essential for describing data in the context of ecology (Time, space, sphere, biome, organism, chemical, process, method). The vocabulary adheres to the following design principles.

• Parsimony

That basically means that we do not get to detailed in terms of information we ask for. This principle is driven by the idea that annotations are typical done in manual fashion and potentially by someone who is not the author of the data (e.g. a data curator).

• Comprehensiveness

Despite the fact that the vocabulary on one hand strifes to be parsimonious we also aim for a good and descriptive annotation. This basically means to extend the vocabulary for more detail if there is a good reason to do so but always keep the first principle in mind.

• Orthogonality

The orthogonality is an important criteria for the axes of the vocabulary. This just means that concepts found on one of the axes do not appear somewhere else and stand in their place for a certain reason. Before a new concept is going to be added first we have to check if the vocabulary could express the same by a combination of existing concepts.

Please browse the vocabulary and answer the questions below. You should use the comment fields attached to each question to provide us with more detail about your classifications. Use these fields to tell us in great detail if something is missing in the interface or needs improvement.

• The vocabulary is well structured (+ comment)
• It does miss essential parts relevant for ecology (+ comment)
• The vocabulary is suitable to tag ecological data (+ comment)
• It contains to many concepts in certain parts (comments: which)
• It contains not enough concepts in certain parts (comments: which)

[cpfaff]

@EichenbergBEF Please check out the plan for the evaluation I made. Comments for a refinement welcome. Would that format also be suitable for the upload sprint you planned? Or is it too long?

[cpfaff]

I refined the description as well as the questions for the user interface part.

Introduction

This evaluation is split into two parts. The first part deals with a user
interface (mock-up) that shows a potential candidate for the implementation of
a tool which can be used to create annotations of search objects in the
ecological context (e.g. datasets). The second part of the evaluation deals
with a controlled vocabulary in form of a thesaurus that provides terms as
basis for the annotation with the tool.

The User Interface Mock-Up

The mock-up of the user interface for the annotation tool is hosted on a web
service called InVision. It allows you to get a first impression of the
application we have in mind for the annotation tool. Please open up the
mock-ups parallel to the questions in that survey (another browser tab or
window). Read the first question switch to the mock-up and check it regarding
the aspects asked for in the question. Then proceed to the next question and so
on switching back and forth.

After answering the question it would be great if you take some time and switch
to InVision. Please use the built-in comment function of InVision to provide
additional comments and hints. This could be hints on typos, ideas for the
arrangement of fields, missing fields, unclear fields and text or general ideas
for the improvement of the interface. The comment function is self explaining.
Just click somewhere to leave a comment. The tool also allows you to create
comments with a snapshot and then you can draw onto the snapshot to better
explain what you mean if necessary.

The mock-up can be found here: https://invis.io/S35HS2P5Q

• Text, colors, icons and other elements
  • The help text and labels are readable (size)
  • Icons are meaningful and colors are used sensible way
  • The text descriptions are meaningful and not misleading or confusing
  • The choices available to pick from in some parts of the annotation meaningful
  • The information in general is captured in an appropriate and intuitive way (sliders, maps, fields, check boxes)
• Suitability for the task
  • The application offers a clear path to finalize an annotation
  • The graphical user interface facilitates the provision of meta-data
  • The 8 categories are suitable to cover the main dimensions in ecology
  • The software asks for too much detail in the categories where some parts should be shortened (+comment what)
  • The software asks for to less detail as it misses some essential information (+comment what)
  • The software will potentially be suitable to create a full annotation for a search object in ecology
  • There is too much detail (user gets confused with too much content for each of the steps)
  • The software would require too much time to create an annotation
  • Would you like to use the tool to create annotations for your data

[cpfaff]

@EichenbergBEF Could you read the post above and comment on it. Some feedback would be good before I finalize this in the survey tool

[EichenbergBEF]

In general I think your questions are quite suphisticated and mine out
the core interests that should be covered by a survey. But I wonder how
you present the evaluation to the PhDs? Do you offer yes/no choices? Do
you intend to do a ranking system (e.g. 1-5)?
In any case I think it would be good to offer some kind of space for
every question that allows to specify either the yes/no or pointing system.
What I mean is:
The application offers a clear path to finalize an annotation
e.g. 1: no, 5, yes
Answer: 3; Opinion: The tool offers good guidance on the dimensions
for annotation. However, I feel that,for my line of work "Biome" is not
important, as I only work in the lab. I am afraid that my datasets will
not be found if I cannot annotate the dimension "Biome".

In this way you can find out what are the main (in this case unfounded)
concerns. You could then (in cases where you think this would be
heplful) e.g. consider these concerns in the text in the green fields of
the mock up or otherwise to tackle these.

As you only have 10-15 people in the evaluation team (this time) I think
this would be a managable amount of work with a potentially high
qualtity outcome.

Am 08.01.2016 um 10:21 schrieb Claas-Thido Pfaff:

@EichenbergBEF https://github.com/EichenbergBEF Could you read the
post above and comment on it. Some feedback would be good before I
finalize this in the survey tool

—
Reply to this email directly or view it on GitHub
#13 (comment).

Dr. rer. nat. David Eichenberg
BEF-China research consortium
Data manager (BEF China Dataportal)
Tel: 0049-341-9738587
Department of Systematic Botany and Functional Biodiversity
University of Leipzig
Room 120
Johannisalles 21
04103 Leipzig GERMANY

[cpfaff]

We use a software to create the survey. Will send you a link if I put the stuff in there. Will present 1-5 choices from disagree to agree and a comment field for each question.

[cpfaff]

Reworked the vocabulary part as well.

Introduction

This evaluation is split into two parts. The first part deals with a user
interface mock-up that shows a potential candidate for the implementation of a
tool which can be used to create annotations of search objects in the
ecological context (e.g. datasets). The second part of the evaluation deals
with a controlled vocabulary in form of a thesaurus that provides terms as
basis for the annotation with the tool.

The vocabulary

The vocabulary is a thesaurus that serves as basis for providing terms for the
annotation with the annotation tool. The vocabulary has been designed in
discussion with domain experts of ecology and adjacent fields. It is designed
along 8 dimensions which are essential for describing data in the context of
ecology (Time, space, sphere, biome, organism, chemical, process, method). The
vocabulary adheres to the following design principles.

Parsimony

That basically means that we do not want to get to detailed. This principle is
driven by the idea that annotation of data is typical done in manual fashion
and potentially by someone who is not the author of the data (e.g. a data
curator).

Comprehensiveness

Despite the fact that the vocabulary on one hand strifes to be parsimonious we
also aim for a useful and sophisticated annotation. This basically means to
extend the vocabulary for more detail if there is a good reason to do so but
always keep the first principle of parsimony in mind.

Orthogonality

Orthogonality is an important design principle for the axes of the vocabulary.
This just means that concepts which are found on one of the axes do not appear
somewhere else and stand in their place for a certain reason. Before a new
concept is going to be added we first have to check if the vocabulary could
express the same meaning by combining the concepts that already exist.

Please browse the vocabulary and answer the questions in the survey. You should
use the comment fields attached to each question to tell us in great detail if
something is missing or needs improvement.

The vocabulary can be accessed online here: http://bit.ly/1TJ5o4O

Questions:

• The vocabulary is logically structured
• The vocabulary does cover all essential areas relevant for ecology
• The vocabulary is suitable to tag ecological data
• The vocabulary contains to many or to detailed concepts in some of its parts
• The vocabulary misses important concepts in one or more of its parts

Each of the questions is scale based (1 disagree - 5 completely agree) and has a comment field attached to allow for the provision of more detailed information.

[cpfaff]

@naouelkaram Could you please read through the post above and let me know what you think. Maybe you have some ideas on how to improve the evaluation for the thesaurus. Maybe some ideas
on the questions.

[cpfaff]

This is done now. Surveys are online. Date of evaluation is 28th of January.