You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I notice that the default row of dataFrame of launch_gradio_widget is set to 2. Since the gradio does not provide delete_row option in the webui interface, the calculation of some web demos (e.g., exact_match, I don't test all) includes the blank row, which may confuse the user.
For example, if we as the users only hope to calculate the exact match score of one sample, thus we only fill the first row in the webui and left the second unremovable row blank, the returned calculation results is always wrong since it counts the second blank text row as a 1.0 score (exactly match! because they are both 0 length '' str).
The above example is shown in this figure.
Actually, I have posted a discussion in huggingface web demo nearly one month ago and no one responsed.
Here are some potential useful suggestions to fix this issue:
Considering the upstream package gradio is not tending to add delete a row option (related issue discussed in here), a quick fix is set the default row to 1 instead of 2 in evaluate/utils/gradio.py/launch_gradio_widget/LINE 115.
Check the user's input data and discard these null inputs (blank text input), which could be done in evaluate/utils/gradio.py/parse_test_cases function.
And I would like to have a PR for fixing this if the the community welcome.😁
The text was updated successfully, but these errors were encountered:
Hi @Raibows, thanks for pointing this out! Indeed that's an issue that would be great to fix. You suggestion of setting the default to 1 and in addition check for null inputs as a failsafe sounds good to me! Feel free to open a PR and let me know if you need any help! Sorry for not replying on the Hub issue, these sometimes slip :)
I notice that the default row of
dataFrame
oflaunch_gradio_widget
is set to2
. Since the gradio does not providedelete_row
option in the webui interface, the calculation of some web demos (e.g., exact_match, I don't test all) includes the blank row, which may confuse the user.For example, if we as the users only hope to calculate the exact match score of one sample, thus we only fill the first row in the webui and left the second unremovable row blank, the returned calculation results is always wrong since it counts the second blank text row as a
1.0
score (exactly match! because they are both 0 length '' str).The above example is shown in this figure.
Actually, I have posted a discussion in huggingface web demo nearly one month ago and no one responsed.
Here are some potential useful suggestions to fix this issue:
Considering the upstream package gradio is not tending to add
delete a row
option (related issue discussed in here), a quick fix is set the default row to1
instead of2
inevaluate/utils/gradio.py/launch_gradio_widget/LINE 115
.Check the user's input data and discard these null inputs (blank text input), which could be done in
evaluate/utils/gradio.py/parse_test_cases
function.And I would like to have a PR for fixing this if the the community welcome.😁
The text was updated successfully, but these errors were encountered: