Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Record task start and end times for Acute-Evals #4208

Merged
merged 21 commits into from Jan 6, 2022

Conversation

EricMichaelSmith
Copy link
Contributor

@EricMichaelSmith EricMichaelSmith commented Nov 24, 2021

Patch description
Record HIT start and end times for Acute-Evals so that Mephisto's DataBrowser can access them. This will allow the Acute-Eval analysis script to record the task completion times properly. NOTE: the actual recording of the start and end times is in Mephisto, as part of facebookresearch/Mephisto#636 : this PR bumps the crowdsourcing CI check to use that update.

Testing steps
Existing tests have been updated to handle this change: the tests will discount the 'time_taken' and 'task_start' columns, which are now (correctly) variable from one run to the next

Testing that HIT times are now saved properly:

python parlai/crowdsourcing/tasks/acute_eval/run.py
mkdir -p ${DUMMY_SAVE_FOLDER}
python parlai/crowdsourcing/tasks/acute_eval/analysis.py -ids ${PRINTED_RUN_ID} --root-dir ${DUMMY_SAVE_FOLDER}

@EricMichaelSmith EricMichaelSmith changed the base branch from main to EricMichaelSmith-patch-1 December 22, 2021 21:32
Base automatically changed from EricMichaelSmith-patch-1 to main December 22, 2021 21:41
@EricMichaelSmith EricMichaelSmith changed the title [WIP] Record task start and end times for Acute-Evals Record task start and end times for Acute-Evals Dec 22, 2021
@EricMichaelSmith EricMichaelSmith marked this pull request as ready for review December 22, 2021 22:08
Copy link
Contributor

@JackUrb JackUrb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Glad to have start and end times for these! These get_task_start and get_task_end functions are likely general enough to live in the underlying StaticAgentState though.

@EricMichaelSmith
Copy link
Contributor Author

EricMichaelSmith commented Jan 5, 2022

LGTM! Glad to have start and end times for these! These get_task_start and get_task_end functions are likely general enough to live in the underlying StaticAgentState though.

Good point @JackUrb - just opened up a PR for this at facebookresearch/Mephisto#636

@EricMichaelSmith EricMichaelSmith merged commit d23a623 into main Jan 6, 2022
@EricMichaelSmith EricMichaelSmith deleted the acute-task-times branch January 6, 2022 22:28
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants