Roadmap for v0.5.0

[zimmski]

The v0.5.0 is mainly meant for introducing more variate. There are three main goals

1. Introduce more logical cases, to make sure that "better models" have a bigger difference in score.
2. Introduce more providers so we can test models that have been request and react faster to new releases.

Tasks:

• Management
  • add https://github.com/symflower/eval-dev-quality/milestone/2
  • TODO https://github.com/symflower/eval-dev-quality/pulls?q=is%3Aopen+is%3Apr+milestone%3Av0.5.0
• Documentation
  • Less fuzz and fluff (see Thomas' feedback)
  • Bring in the current blog post, its information, especially the blog post image to showcase the evaluation
  • Readme extension: The nice thing about generating tests is that it is easy to automatically check if the result is correct. Needs to compile and provide 100% coverage. But one can only write such tests if they understand the source, so implicitly we are evaluating the language understanding of the LLM.
• Evaluation
  • Measure processing time of queries Measure processing time of model responses #106 Measure Model response time #105
  • Automate multiple runs for more deterministic results Multiple runs in a single evaluation #109 Multiple Runs #108
  • Empty responses should be marked as error responses, to indicate that they are on the same level Empty model responses should be handled as errors #97 Empty responses should not be tested but should fail  #92
  • when an LLM is evaluated to successfully do "plain" and Go fails bug Java works, then it will be blocked to do ALL repositories. In that case it should only be blocked for Go and not Java. Do not cancel successive runs if previous runs had problems #129
  • Do clean up of generated files Reset repository per task #148 Repository not reset for multiple tasks #147 Use Git to avoid copying the repository on each model run #114 Use empty Git config in temporary repositories #146 The git repository change requires the GPG password #145
  • Java
    • Log Maven commands because they can be faulty (remember that the "surefire" plugin needs a fixed version because GitHub's is tooooo old). With that it is easier to debug symflower v36847
  • even powerful models as GPT4 and Llama3 might return EOF or an error (https://github.com/symflower/eval-dev-quality/tree/105%2B108/evaluation-2024-05-14-09%3A18%3A41), add a retry logic: Give models a retry on error #123 Allow to retry a model when it errors #125
• Support execution of more models
  • Ollama support Integrate Ollama #91 Ollama tool installation #95 Support Ollama provider #96 Fixed Ollama version #117 Ollama version check and update if version is outdated #118 Prepare for Ollama provider #115 Ollama provider #27
  • Allow arbitrary URLs for API provider Generic OpenAI API provider #111 Generic OpenAI API provider #112
• Reporting and Metrics
  • Additional CSVs to sum up overall results, and language individual results Additional CSVs to sum up metrics for all models overall and per language #94 Add additional CSV files that sum up: overall, per-language #83
  • fix, Y axis ticks should be readable Fix svg Y axis ticks #73
  • fix, Deterministic order of rows in CSV exporting Sort map by model before creating the CSV output #99 Non deterministic test output leads to flaky CI Jobs #98
• Multi OS support of eval
  • Support MacOS Support MacOS #102
  • Support Windows Support Windows #103 Unable to run benchmark tasks on windows due to incorrect directory creation syntax #101 fix, Let Windows CI error on failures, and inject API tokens only once per provider #104
• Tools
  • Introduce unique ID for addressing tools Introduce an "ID" method to the tool interface #122 Os-independently
  • Extend symflower test with a deeper execution coverage export
    • Go
      • Extract to file
      • Cover lines of tests that have exceptions (fixed with symflower v36800) Require at least symflower v36800 #144
    • Java
      • Extract to file
      • Cover lines of tests that have exceptions (fixed with symflower v36800) Require at least symflower v36800 #144
• Tasks and cases
  • Introduce more cases with logic in "light" repository
    • Go More "write tests" task cases with more complex logic #124
    • Java More Java task cases for test generation #134 More "write tests" task cases with more complex logic #124
• Release
  • Do a full evaluation with the new version
  • Tag version
  • Blog post
  • Adapt README
  • Announce and eat cake

TODO sort and sort out

• Models
  • For non-selective evaluations exklude certain models e.g. "openrouter/auto" needs to go, because it is not a real model, is just forwarding to a model automatically Exclude openrouter/auto since it is just a random model #126
  • Think about exlcuding the "perplexicty" models because they have a "per request" cost, and they are the only ones that do that.
  • Snowflake against Databricks would be a nice comparision since they align company-wise and are new
  • Include more models (We have the main problem that there are multiple models coming out every day. We should not wait for a "new version" of the eval, we should test these models right away and compare them. big problem: how do we promote findings?)
    • Snowflake
    • DeepSeek for Code https://t.co/jpGY5w0e4Y see https://twitter.com/ak_kim0/status/1786141140376285685
    • CodeQwen 7B
    • Nous-Hermes-2-SOLAR-10.7B; Also "Tree of Thoughts" approach might be interesting as a task
    • Gemma 2
    • https://x.com/osanseviero/status/1793644453007155451
    • https://x.com/osanseviero/status/1793930015047880959
  • Maybe use https://huggingface.co/inference-endpoints/dedicated
• Metrics & Reporting
  • write out results right away so we don't loose anything if the evaluation crashes
  • AST differ Introduce an AST-differ that also gives metrics #80
  • Non-benchmark metrics Include metrics about the models for comparing models #82
  • Add linters where each error is a metric #81
  • Remove absolute paths completely e.g. in stack traces too.
  • Automatically interpret "Extra code" Infer if a model produced "too much" code #44
  • Figure out the "perfect" coverage score so we can display percentage of coverage reached
  • Make coverage metric fair
  • Save the descriptons of the models as well: https://openrouter.ai/api/v1/models The reason is that these can change over time, and we need to know after a while what they where. e.g right now i would like to know if mistral-7b-instruct for the last evaluation was v0.1. or not
  • Bar charts should have have their value on the bar. The axis values do not work that well
  • Pick an example or several examples per category: goal is to find interesting results automatically, because it will get harder and harder to go manually through results.
  • Charts to showcase data
    • Total-scores vs costs scatterplot. Result is upper-left-corner sweat spot: cheap and good results.
    • Piechart of whole evaluations costs: for each LLM show how much it costs. Result is to see which LLMs are costing the most to run the eval.
  • Reporting and documentation on writing deep-dives
    • What are results that align with expectations? what are results against expectations? E.g. are there small LLMs that are better than big ones?
    • Are there big LLMs that totally fail?
    • Are there small LLMs that are surprisingly good?
    • What about LLMs where the commonity doesn't know that much yet: e.g. Snowflake, DBRX, ...
  • Order models by open.weight, allows commercial-use, closed, and price(!) and size: e.g. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 is great because open-weight, and Apache2 so commerical-use allowed. Should be better rated than GPT4
  • Categorize by parameters/experts https://www.reddit.com/r/LocalLLaMA/comments/1cdivc8/comment/l1davhv/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
  • Compare input/output/request/... costs https://twitter.com/oleghuman/status/1786296672785420744
  • Benchmark quantized models, beacuse they need less memory
    • https://twitter.com/HaihaoShen/status/1789178048308543688
  • distinguish between latency (time-to-first-token) and throughput (tokens generated per second)
• Documentation
  • Clean up and extend README
    • Better examples for contributions
    • Overhaul explanation of "why" we need evaluation, i.e. why is it good to evaluate for an empty function that does nothing.
  • Write down a playbook for evaluations, e.g. one thing that should happen is that we let the benchmark play 5 times and then sum up points, but ... the runs should have at least one hour berak in between to not run into cached responses.
  • Write Tutorial for using Ollama
  • YouTube video for using Ollama
• Tooling & Installation
  • Rescore existing models / eval with fixes e.g. when we do a better code repair tool, the LLM answer did not change, so we should rescore right away with new version of tool over a whole result of an eval.
  • Automatic tool installation with fixed version
    • Go
    • Java
  • Ensure that non-critical CLI input validation (such as unavailable models) does not panic
  • Ollama support
    • Install and test Ollama on MacOS
    • Install and test Ollama on Windows
  • Allow to forward CLI commands to be evaluated: Ollama provider #27 (comment)
  • Refactor Model and Provider to be in the same package Preload Ollama models before inference and unload afterwards #121 (comment)
• Outreach
  • Automatically updated leader board for this repository: Do an up-to-date leaderboard/dashboard for current models current evaluation #26
    • Take a look at current leaderboards and evals to know what could be interesting Current popular code leaderboards are [LiveCodeBench](https://huggingface.co/spaces/livecodebench/leaderboard), the [BigCode models leaderboard](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard), [CyberSecEval](https://huggingface.co/spaces/facebook/CyberSecEval) and [CanAICode](https://huggingface.co/spaces/mike-ravkine/can-ai-code-results)
  • Blog post about the different suffixed of models e.g. "chat" and "instruct" and eval them somehow. Idead from https://www.reddit.com/r/LocalLLaMA/comments/1bz5oyx/comment/kyrfap4/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
  • Blog post about HumanEval
  • Blog post about training a small LLM directly on HumanEval
  • Blog post about "non-determinism of LLMs" https://community.openai.com/t/a-question-on-determinism/8185 good starting point, and how we can make them at least more stable.
  • Blogpost idea: misleading comments, weird coding style... how much does it take to confuse the most powerful AI?
• Research
  • Take a look at https://twitter.com/SMT_Solvers/status/1783540994304066006
  • Take a look at all OpenRouter's features of the API e.g. https://openrouter.ai/docs#parameters
    • https://www.reddit.com/r/LocalLLaMA/comments/1cihrdt/comment/l29o97q/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button mentioned that repetition_penalty disabled helps with performance and better results for coding.
    • Requested new models for the eval
      https://www.reddit.com/r/LocalLLaMA/comments/1cihrdt/comment/l2d4im0/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
  • Look at what Paul Gauthier is doing with benchmarking of Aider (and Aider too) https://twitter.com/paulgauthier/status/1787827703158386921?s=46 seems like a perfect match for what we want to do as tasks
  • Look at MLX which maybe helps with execution for us https://twitter.com/loganthorneloe/status/1787845883519775120
  • Take a look at xLSTM https://twitter.com/HochreiterSepp/status/1788072466675335185
  • Take a look at eval https://twitter.com/JiaweiLiu_/status/1783959954321252697
  • Take a look at evaluation framework https://twitter.com/hamelhusain/status/1788936691576975382?s=61
  • Dig through https://arxiv.org/pdf/2405.14782 thanks to https://x.com/clefourrier/status/1793913394871062970
  • Take a look at https://x.com/dottxtai/status/1798443290913853770
• Evaluation
  • Java
    • Let the Java test case for No test files actually identify and error that there are no test files (needs to be implemented in symflower test)
  • LLM
    • Log request and response in their own files, so both can be used 1:1 (character for character) directly for debugging them
    • Improve LLM prompt
      • Add an app-name to the requests so people know we are the eval https://openrouter.ai/docs#quick-start shows that other openapi-packages implement custom headers, but the one Go package we are using does not implement that. So do a PR to contribute.
      • Add markers for system and user e.g. Ollama provider #27 (comment)
      • Think about a standardized way of print outputs e.g. JSON https://twitter.com/WuMinghao_nlp/status/1789094583290507626
  • Prepare language and evaluation logic for multiple files:
    • Use symflower symbols to receive files
  • Sandboxed execution Sandbox execution #17 e.g. with Docker as its first implementation
  • timeout for test execution (we've seen tests that take > 15 minutes to execute in some benchmarks)
  • Do an evaluation with different temperatures
  • Failing tests should receive a score penalty
  • Evaluation tasks
    • Introduce the interface for doing "evaluation tasks" so we can easily add them
    • Add evaluation task for "querying the relative test file path of a relative implementation file path" e.g. "What is the test relative file path for some/implementation/file.go" ... it is "some/implementation/file_test.go" for most cases.
    • Add evaluation task for transpilation Go->Java and Java->Go
    • Scoring, Categorization, Bar Charts split by language.
    • Check determinism of models e.g. execute each plain repository X-times, and then check if they are stable.
  • Code repair
    • Own task category
    • 0-shot, 1-shot, ...
      • With LLM repair
      • With tool repair
  • Do test file paths through
    • symflower symbols
    • Task for models
  • Query REAL costs of all the testing of a model: the reason this is interesting is that some models have HUGE outputs, and since more output means more costs, this should be addressed in the score.
  • Move towards generated cases so models cannot integrate fixed cases to always have 100% score
  • Think about adding more trainings data generation features: This will also help with dynamic cases
    • Heard that Snowflake Arctic is very open with how they gathered trainings data... so we see what LLM creators think and want of trainings data
• Think about a commercial effort of the eval, that we can balance some of the costs that goes into maintaining this eval

[zimmski]

CC @bauersimon

[bauersimon]

Add an app-name to the requests so people know we are the eval https://openrouter.ai/docs#quick-start shows that other openapi-packages implement custom headers, but the one Go package we are using does not implement that. So do a PR to contribute.

• "Features not provided by the OpenAI API will not be accepted."
• Issue: Custom Headers: "customize it with HTTPClient"
• PR: Configurable RequestBuilder for custom headers
• README example custom proxy

seems like a PR does not make sense

[bauersimon]

Blogpost idea: misleading comments... how much does it take to confuse the most powerful AI? (credit to @ahumenberger)

[ahumenberger]

Blogpost idea: misleading comments... how much does it take to confuse the most powerful AI? (credit to @ahumenberger)

Maybe not only comments. What about obfuscated code, e.g. function and variables names are just random strings?

[zimmski]

Take a look at https://x.com/dottxtai/status/1798443290913853770