-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Return the test details with runtests
?
#139
Comments
thanks for the issue! Can you say more about what the use-case is? Returning Also the whole So what's possible/desirable to do here depends on what exactly you're trying to achieve. Can you share an example? Also, for context: i'm a little wary of committing to any kind of support for interacting with test results without a bit of a plan for what would be in or out of scope, and taking a look at prior art e.g. a natural next step would be to use the returned test results when re-running tests, e.g. to only run the failures/error, or to run those first -- but maybe we can add something that helps you without that, if i can understand the use-case better |
First, thanks for your prompt reply! I understand your concern and your explanation makes sense to me.
Sure! So I'm working on a project similar to openai/human-eval and evalplus, but in JuliaLang. Generally it will ask different LLMs to generate the code based on given prompt and then execute the code to calculate the I think all I need are already in the FYI: you can view the test cases here: https://github.com/oolong-dev/OolongEval.jl/tree/add_human_eval/benchmarks/HumanEval/src/tasks |
i see. yeah, i'm not sure i see an easy way to add this to ReTestItems due to how Test.jl works (at least not one i'd be comfortable committing too 😅) Maybe right now using the But an alternative to parsing the using MetaTesting: EncasedTestSet
using Test
using ReTestItems
# Wrap in a `MetaTesting.EncasedTestSet` so we can capture results rather than throwing if any fail/error
ts = @testset EncasedTestSet "runtests" begin
runtests(...)
end
results = only(ts.results) # <- # Get the `Test.DefaultTestSet` produced by `runtests(...)`
# Do stuff with `results`... e.g. print results to check they're the same as what `runtests(...)` would show:
Test.print_test_results(results) This gives you a |
Great thanks! I wasn't aware of MetaTesting before. I'll try both approaches and see which one works better. |
I used |
I'd like to extract the test results and do some calculations later. My current approach is to set
report=true
and then parse the resultingxml
file. Is there any other better approach?More specifically, can we return
testitems
instead ofnothing
here?ReTestItems.jl/src/ReTestItems.jl
Line 396 in 60d93f1
The text was updated successfully, but these errors were encountered: