Ways to reset blocked workers for onboarding qualification?

[evelynkyl]

Hi!
I'm trying to reset the existing onboarding qualification data and let the blocked worker (me..) redo the qualification. I've tried removing block_qualification in example.yaml but it didn't work. I've also tried adding the below code to get the current worker id and unblock myself in the init of PerTurnEvalWorld, but that didn't do anything either.

if agent.mephisto_agent.get_worker().is_disqualified(self.block_qualification):
    logging.info("This worker is disqualified!")
    agent.mephisto_agent.get_worker().grant_qualification(
        self.block_qualification, 1
    )
    logging.info("Permission granted.")

Are there any ways to unblock myself? Do I have to modify ModelChatOnboardWorld? Or is there a easy way to wipe existing data so that system can't get the list of blocked workers?

Thanks so much in advance!

[JackUrb]

Hi @evelynkyl, sorry that this flow isn't fully clear. In testing with a LocalArchitect and MockProvider we tend to just change the worker_id url param as described in our tutorial as it's easier than having to pull yourself out of the db and revoke the block qualification.

That being said, you were close to removing the qualification - we key blocking qualifications on the existence of a qualification however, so you'll need to remove it with revoke_qualification:

Mephisto/mephisto/data_model/worker.py

Line 173 in e732de2

def revoke_qualification(self, qualification_name) -> bool:

[evelynkyl]

Thanks so much for the fast response @JackUrb! I wasn't aware of the changing worker_id approach. I used this method and it's working for me now!

Oddly enough, I was actually qualified, but because the validate_onboarding function in world.py in pairwise_per_turn_eval was returning None, it disqualified me.

I removed submitted_data from the below code since data was not in the status_message and it's been fixed:
https://github.com/facebookresearch/ParlAI/blob/5f3fea75d7df55f24e8f0bf14d2b6d15351b005d/parlai/crowdsourcing/tasks/pairwise_per_turn_eval/worlds.py#L509-L524

Thanks again for the help!

[JackUrb]

I removed submitted_data from the below code since data was not in the status_message and it's been fixed:
https://github.com/facebookresearch/ParlAI/blob/5f3fea75d7df55f24e8f0bf14d2b6d15351b005d/parlai/crowdsourcing/tasks/pairwise_per_turn_eval/worlds.py#L509-L524

If this resolved the issue for you, any chance you can submit a PR over there with this fix? I'd be happy to accept

[evelynkyl]

Happy to! Just submitted one, please see above :)