[Feature request] Prohibit simultaneous annotation of one image by different users

[MrChebur]

Currently, when multiple users annotate images at the same time, different users get the same images.

I believe this is undesirable behavior.

Re-annotation should only occur after all images have been annotated.

Options to solve the problem:

1. Easy to implement - clicking on the "Next Image" button should produce a random image from the list of unannotated images.

2. More complex implementation - when clicking on the "Next image" button, images should be marked "on hold" and not shared with other users for annotation. If an image has been skipped, the "on hold" mark should be removed.

[aliFrancis]

Hi! Thanks for using IRIS.

This sounds like a nice feature when working with a lot of annotators.

To be clear, I think we already have a feature that resolves it in the way of your description of solution 1. By using the "prioritise_unmarked_images": true in your config, the next image should be randomly chosen from the images with the lowest possible number of annotations already saved.

Are you suggesting that in addition to this, there is also a feature for blocking images that are currently being worked on? Or at least de-prioritising them further than the current system which only considers saved masks?

[MrChebur]

Strangely, if images are supposed to come in random order, it's suspiciously common for users to receive the same images.

Perhaps the problem is that our data is similar to each other. We should investigate this problem more thoroughly.

Thanks for the reply.

[aliFrancis]

Thanks for pointing this out, actually looking more closely at our implementation it seems it isn't actually randomized, but just finds the next in the existing order. Not quite a bug but I can see how it causes issues if you have multiple annotators! I will reopen and provide a fix in the coming days.

[markBETA]

Hi! I was checking on that, and I found out that the randomization should occur before pulling the next or previous image, as you can see here. That said, each user should see a constant order of the images, but it should change for each individual user, as the seed used for each one of them is different. You could fix your issue by creating a different user for each annotator.

Implementing a lock system shouldn't be complicated using a framework like Flask-SocketIO. This could also be used to solve the time-counting bug that has already been identified.

[aliFrancis]

Apologies for the huge delay on this, I've had to prioritise other projects.

Thanks for the tip @markBETA, I will look into Flask-SocketIO and what it can offer.