Fix Remote Eval Script #68
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ram81
requested changes
Dec 26, 2022
Ram81
requested changes
Dec 26, 2022
| @@ -1,4 +1,3 @@ | |||
| import random | |||
|
|
|||
|
|
|||
| def evaluate(user_submission_file, phase_codename, test_annotation_file=None, **kwargs): | |||
There was a problem hiding this comment.
can you update the order of arguments to be submission_file, test_annotation_file, phase_codename, **kwargs
There was a problem hiding this comment.
@Ram81 I think they will have to change the code in main.py to correcly pass the test annotation file if we make it into a positional argument. I made it None and placed it add the end so they can pass a default, or skip and use phase_codename to find the correct file. Wdyt?
There was a problem hiding this comment.
I see, that makes sense. In that case, can you add a comment in the evaluate method of this form
'''
# Load test annotation file for current phase
test_annotation_file = json.loads(open("{phase_codename}_path", "r"))
'''
There was a problem hiding this comment.
I have made a couple changes accordingly. Please let me know if they look okay.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the remote eval script template with the locally tested and working version of the same.