Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Gold Standard Training" video not showing up for new turkers #13

Open
forresti opened this issue Nov 25, 2014 · 2 comments
Open

"Gold Standard Training" video not showing up for new turkers #13

forresti opened this issue Nov 25, 2014 · 2 comments

Comments

@forresti
Copy link

I set up a "worker training" video:

turkic load train_video /path/train_video.mp4 Car
    --blow-radius 1 
    --for-training 
    --for-training-start 0 
    --for-training-stop 300
    --for-training-overlap 70 
    --for-training-tolerance 70 
    --for-training-mistakes 1

...and I annotated train_video in the UI.

Then, I set up an ordinary video for a worker to annotate... and notice that I do --train-with train_video:

turkic load new_video /path/new_video.mp4  Car 
    --length 300
    --blow-radius 1 
    --title "Label the cars in this video (15sec of video). Per-object bonus." 
    --description "Draw boxes around the cars in this video. When the cars move, you move the boxes. See 'show instructions' in the HIT. Feel free to email us with questions." 
    --duration 600 
    --lifetime 604800 
    --keywords "video, tagging, annotation, cars" 
    --cost 0.05 
    --per-object-bonus 0.02 
    --completion-bonus 0.1
    --train-with training_video

In the worker sandbox, I tried 2 things:

  1. Login as myself and look at the HIT. Instead of seeing train_video, I immediately see new_video.
  2. Perhaps I've annotated my own tasks in the worker sandbox before. So, I created a new worker account. With my new account, I still immediately see new_video instead of train_video.
  3. I tried using mturk.com instead of the worker sandbox. The training video does show up here. But, the training video appears with my ground truth annotations on it! This can't be right...

Questions

  • Am I doing something wrong -- how do I get new workers to see the training video first (in the worker sandbox)?
  • I'd like to grep the server.py logs for the output "Swapping actual segment with training segment". I tried initializing the logger in server.py to dump debug info to a file... but this debug file doesn't appear when a user loads a task.
  • On mturk.com, how do I avoid having the turker see the ground truth annotations for my training video?
@gnetscher
Copy link

I had the same issue and would love to see a response

@pammirato
Copy link

did you accept the hit?

For me, if I just look at the hit it shows the new video, but once I accept the hit it starts with the training video

that being said I do need to find a way to "untrain" myself so I can test the training video multiple times in the sandbox

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants