Seen before & retired image flags

[camallen]

I've got an image that i've seen and is retired but it only shows 1 flag (already seen) at a time.

[edpaget]

Do we need to have both flags?

[camallen]

I think so, we could just say retired and already seen.

[chrissnyder]

Is there any circumstance in which a user would be served a retired subject when there is still data that needs classifying on a project? If not, then the retired flag signals something else entirely, in that the project's current dataset is "done", which is a different message than "we just gave you a single subject that is done".

[chrislintott]

3The only example I can think of is if an expert user was working through a dataset that had already been classified by volunteers in order to augment the gold standard dataset

Cheers

Chris

On 17 Jul 2015, at 14:47, Chris Snyder notifications@github.com wrote:

Is there any circumstance in which a user would be served a retired subject when there is still data that needs classifying on a project? If not, then the retired flag signals something else entirely, in that the project's current dataset is "done", which is a different message than "we just gave you a single subject that is done".

—
Reply to this email directly or view it on GitHub #1050 (comment).

[mrniaboc]

I think it would be cool to redirect people at this point, instead of just telling them they've seen the image before, we could give them a message like "Well done! It looks like you've seen all of the images on this project. We could really do with your help over at "

[vrooje]

Agreed, an optional redirect is a good idea and it's a good opportunity to try some experiments too.

However, we need to make sure zooniverse/panoptes#1069 is solved first.

[mschwamb]

Wanted to chime in and say that the out of data message like @mrniaboc suggests has been helpful for me in the past especially before the stats page for Ouroboros projects. When several people saying on Talk they were getting that message, it was the sign that we could push live the next set of subjects that were prepped and waiting.

[mschwamb]

This came up in a talk thread today on Planet Four: Terrains. I noticed two users are very similar in posting and sign with the same name, Well it happens to be a super classifier who is very active on Talk hit all the available data (we have something like 8000 subjects) and kept getting the already seen flag on Planet Four: Terrains and didn't know what that meant apparently so made another account to start classifying again without the message and posting on Talk. This is absolutely an edge case, but I think it points to classifiers don't understand what the 'already seen' means.

[camallen]

@mschwamb i don't consider what that user did as normal...why would they create another account to do the whole set of data again? Did they not think that they would repeat their previous work?

I agree the messaging in the UI can be improved. It'd be nice to run so a/b splits on this to see what works best.

[mschwamb]

@camallen - totally agreed it's an edge case. Never had this happen before as far as I know. The reasoning I got so far is they partly thought it was either glitch since images kept coming and they felt that that with a new account they'd see different images. They didn't quite get from the orange banner that there was no more data. And since the banner went away with a new account, they decided to keep going.

[chrissnyder]

Not sure where to go here, would love more input.

The current messaging is clearly insufficient and unclear if a volunteer didn't make the leap between "you've seen this before" and "you've seen everything there is to see".

Maybe we need a much more explicit banner across the top of the classifier stating something to the effect of "hey, thanks so much, you've seen and classified everything there is to see on this project! go to Talk, etc".

Thoughts?

[mschwamb]

That would be fantastic @chrissnyder . I think it should be a popup like the reminder to sign in.

[aliburchard]

like @chrissnyder's suggestion. I'm partial to having a banner across the top of the interface itself (not on the image itself) that says something along the lines of "all the images for this project have been classified! we'll let you know when we add more data, but for now, please go here"

[mschwamb]

Related to this thread - there's someone on Comet Hunters Talk asking why they're seeing the 'already seen' label all the time now. I think that is another example that people don't really understand the labels going on the images, and a pop up message saying 'thanks you've completed the data' would be a better route, when there is still data live and can be classified but the particular user has seen all of it.

[chrislintott]

I agree with this. @brian-c?

On 21 Feb 2016, at 07:25, Meg Schwamb notifications@github.com wrote:

Related to this thread - there's someone on Comet Hunters Talk https://www.zooniverse.org/projects/mschwamb/comet-hunters/talk/210/40383?page=1&scrollToLastComment=true asking why they're seeing the already seen all the time now. I think that is another example that people don't really understand the labels going on the images, and a pop up message saying thanks you've completed the data would be a better route.

—
Reply to this email directly or view it on GitHub #1050 (comment).

[yshish]

I don't understand your reasons for using a banner instead of making impossible to classify the image again. I'm just afraid that as long as there are images which have been seen already possible to be classified, there always will be volunteers classifying them. (For example kids from my class who almost don't speak English but love easy projects like Snapshot at Sea where you have a simple task with yes/no answer. I'm sure no banner would stop them from clicking as long as they can click on the button.)

[vrooje]

I think you've hit on exactly why! It's important for many projects to be able to stay open so that educators can use them, but of course it's also important to be clear in that case that additional classifications aren't contributing as much to the science. The "Already Seen" banner isn't clear enough, but I don't think we want to completely prevent classifications, either...

[yshish]

Thanks for explaining, Brooke. It makes more sense now, although I still don't see the current banner as a good solution. Maybe adding a pop-up explaining the situation would be more helpful?

[camallen]

Some volunteers are having a similar discussion on talk, https://www.zooniverse.org/talk/17/43234?comment=86295

I agree the current banners aren't messaging the intention well enough. In my opinion we should be signalling the following to the user:

1. The subject is retired but they haven't seen it
   • E.g. when the user has seen all the non-retired data
2. The subject has been seen by the user before
   • User has seen all the data in the workflow (power users)
3. The workflow subjects are all retired
   • Currently fires the we're done now banner.

I think I've covered all our bases above but add more if needed.

All points above should note that new effort may not contribute more information to a project. We currently only provide the seen_before, retired and workflow finished flag to the UI but should probably also provide the user has_finished workflow field as well.

[mschwamb]

@camallen I'm not sure people will understand 'retired but they haven't seen it' messaging. I think it will be the same issues as the current 'retired tag'. Is getting classifications on retired subjects that a person hasn't seen actually useful and productive use of their time or a significant gain for the project? I'm not sure. My suggestion would be to direct the person to Talk or another project until new data is loaded onto the project with a message box if a person has seen all the unretired subjects on the site, and if they want to ignore that message the retired/already seen banner can show on the subject as well

[camallen]

@mschwamb I was talking in general terms and not a specific text based message. As per the link i provided, perhaps a visual UI component / iconography instead of text?

Agree we could do better than just putting a label on an image and instead offer alternatives to classifications.

[mschwamb]

@camallen I think I'd have icons and text, but icons on the project cards on the project list might be useful for people perusing for projects to identify which ones need to help and which ones are complete/out of live data

[alexbfree]

icons on the project cards on the project list ... to identify which ones need to help and which ones are complete/out of live data

I really like that idea!

[mschwamb]

The situation came up again. The volunteer who was helping sort the Already Seen classifications now just finished all the live data on Comet Hunters and didn't know that. This situation again highlights that people don't understand what the Already Seen and Retired tags means and it's not signalling that they've finished. It's jargon that people aren't understanding.

I hope a better solution can be implemented soon. A pop up message saying 'thanks you've completed the data' would be a better route, when there is still data live and can be classified but the particular user has seen all of it. Any thoughts @chrislintott and @brian-c ?

[camallen]

@mschwamb there have been some issues with one of our selection services (one ridges is using). We're working on it and hope to have better news soon. In the meantime you should tell that user that listening to Bowie never hurt anyone ;)

[mschwamb]

@camallen thanks for letting me know. I think these people are close to being done with the live subjects given that they appear to be super users. I think the bigger problem is that this messaging isn't clear. It seems like it wouldn't be too difficult to add something similar to when there is no data on a live site kind of message in a box to the workflow page when a person gets an already seen.

[mschwamb]

Volunteers still don't get these labels. Every time we're close to finishing a dataset, I get questions about these. Many people don't get this means they've seen all the available data on the site but there is still images that other people haven't classified and are not retired. Any chance there can be better messaging?

[vrooje]

Hi all, in the PRN Irma project we are experiencing a phase at the end of each new data release where all volunteers get are the Finished/Already Seen banners and they don't get new subjects (they just keep drawing from the same batch of ~10) if they refresh. But the "out of data" message is not showing up. I'm getting questions as to why this is happening and honestly I'm not sure? The stats page shows that there are about 50 unretired subjects in 1 workflow, but I don't understand why I'm not getting them on the classify page when I refresh.

[vrooje]

Hang on, I think I know why. It's because every subject in the default workflow was finished, but there was another active workflow that wasn't finished, and the classifier wasn't switching to that because it was trying to load the default workflow. When I removed the default flag from that workflow, it switched over to the other one.

Is this intended behavior? If so I don't think it's messaged well to project owners as a potential consequence of selecting a default workflow.

[camallen]

@vrooje yeah - i think this is due to the fact the UI wires up the default workflow on page load so selection will happen on that workflow. You have to switch the default workflow over so that new users get the UI wired up on that.

Existing users that haven't reloaded the page won't get the new workflow, instead the UI will keep requesting data for the default worfklow when the loaded the page. Having the ability to message these people without resorting to finished / seen flags is something we should look to do with a websocket and push from the server.

[srallen]

In addition to activeness, I could have the front end check for completeness too and then fall back to a random selection of the active workflows if the workflow selected is complete.

[camallen]

Sounds ideal!

[simensta]

The Already Seen/Finished flags are causing some confusion in the Decoding the Civil War project, as well. See this Talk discussion for the details.

Is there a way to communicate that a volunteer has seen all of the un-retired subjects for a project?

[mschwamb]

Better messaging would be great. Some volunteers on my projects get confused as well (see previous comments in the thread). I really liked the way the original Planet Hunters did it with a pop message telling you 'thanks you've finished all the available data on the project'. Would that be an option? Leave the current orange label on the top left, but add in a pop up box?

[vrooje]

I agree something like this would be useful on the user level, as it happens to a lot of users well before the global green "this project is out of data" message comes up. It might be as "simple" as more forcefully putting the banner mouseover text in front of the user so they can actively click something like "keep classifying anyway" to keep going, or just come back later. Right now the explanation is pretty well hidden.

[simensta]

Instead of the diagonal banner, we could add a banner just above the image that provides the "Finished" or "Already Seen" labels with the explanation text (without clicking through).

If a volunteer has seen all un-retired images, it would be great to message that information as well.

[srallen]

@beckyrother might have some ideas about this.

[beckyrother]

Thanks for pointing this out; adding to my list of things to work on as we update the classifier. Reading back up, I didn't realize the banners were clickable either! Definitely need more clear language for all of these scenarios.

One thought would be a popup that explains that they've seen all the subjects and gives them the option to classify in that project anyway, or shows them three project cards from the same category (space, physics, nature, etc).

I'm hesitant to request any large changes now, though, since the classification redesign is coming up.

[mschwamb]

@beckyrother yes having suggestions of other projects would be great if a project is out of data. see #3896

[vrooje]

Sometimes refreshing seemed to help people stop seeing the banner, at least in PRN. I'd prefer users only be directed to other projects if the code verifies that user really has seen every subject. Directing them to Talk ("join the conversation and help others who are newer to the project" etc) is another option.

If the project really is completely out of data, that's different.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mschwamb]

@srallen with the interface design will this be addressed, if not could changing the messaging/flags be part of it?

[beckyrother]

@mschwamb yes, we have plans to address these flags in the update.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mschwamb]

wasn't address in any of the recent interface changes, so this is still an issue that should not be marked as stale.

[eatyourgreens]

Already seen and retired subjects are filtered out of the upcoming queue before it's shown to a volunteer:

Panoptes-Front-End/app/pages/project/classify.jsx

Lines 250 to 257 in a29fe0f

	const nonLoadedSubjects = subjects.filter(newSubject => newSubject !== subjectToLoad);
	const filteredSubjects = nonLoadedSubjects.filter((nonLoadedSubject) => {
	const notSeen = !nonLoadedSubject.already_seen &&
	!nonLoadedSubject.retired &&
	!seenThisSession.check(workflow, nonLoadedSubject);
	return notSeen;
	});
	const subjectsToLoad = (filteredSubjects.length > 0) ? filteredSubjects : nonLoadedSubjects;

[mschwamb]

That does not happen when the person has finished all the live data on the site but those subjects are not retired. Then these flag show and people are confused what they mean. I have had more than one volunteer keep classifying thinking they were contributing when they were adding duplicate classifications. The 'good job you completed the data' equivalent message only shows what the entire site is out of data. So if one workflow has data, then these flags do show and are the only indication of the state of the project for the specific user account.

Also in my experience frequently close to reaching the end of the live data for an individual classifier you start seeing the 'already seen' flag pop up.

[srallen]

Yes, the UX part of this still needs addressing when the flags do show and will be with the redesign/rewrite, but keeping open an old generic version of this issue is not helpful for us. When we schedule the next round of UX updates, a specific issue for the UX side of this will be documented.

The latter issue you described is likely an API side issue, not UX.

[srallen]

Project suggestion modal is described in #4556