Home

Overview

Components

1. crawler (Java version here: https://swtrepo.informatik.uni-mannheim.de/lasso/lasso/-/tree/develop/crawler?ref_type=heads)
   • datasources
     • PyPi?
     • GAI Benchmarks (HumanEval, EvalPlus, MultiPL-E etc.) / GAI ad hoc (as part of LSL pipeline ingestion, e.g., https://swtrepo.informatik.uni-mannheim.de/lasso/lasso/-/blob/develop/doc/examples/GenerativeAI.md?ref_type=heads)
     • other freely available python datasets
     • ...
2. analyzer (Java version here: https://swtrepo.informatik.uni-mannheim.de/lasso/lasso/-/tree/develop/analyzer?ref_type=heads)
   • creating a Solr index (based on existing schema) and filling an artifact repository
     • see schema here: https://swtrepo.informatik.uni-mannheim.de/lasso/lasso/-/tree/develop/doc/solr_config?ref_type=heads
   • python parser
   • infer typing challenge
     • pytype?: https://swtrepo.informatik.uni-mannheim.de/lasso/python-playground/-/blob/main/python-support/parser/pytype_parse_type.py?ref_type=heads
3. arena test driver for python
   • adaptation (identifying callable python functions/methods)
     • adapting interface mismatches (interface signatures assumed by sequence sheets vs python functions/methods signatures)
   • execution based on introspection/reflection to obtain outputs
   • serializing python objects to strings
   • storing observations (i.e., actuation sheets/SRMs) using pyignite? (see https://ignite.apache.org/docs/latest/thin-clients/python-thin-client)
   • some rough prototyping here:
     • https://swtrepo.informatik.uni-mannheim.de/lasso/python-playground/-/blob/main/python-support/execute/intercept_recorder.py?ref_type=heads
4. demonstration / evaluation
   • python corpus (large) - index + nexus
   • benchmarking GAI
   • PolyGlot - SRMs/SRHs independent on programming language
     • abstracting from programming language (exploit SRMs in general independent on programming language)

Further playground material is here:

• https://swtrepo.informatik.uni-mannheim.de/lasso/python-playground

(Possible) Data Sources (non-exhaustive)

Python Repositories

• https://pypi.org/
  • packages via pip

Generative AI / Benchmarking

• authors of EvalPlus (HumanEval+ and MBPP+) share their python code completitions here (EvalPlus Leaderboard https://evalplus.github.io/leaderboard.html):
  • https://github.com/evalplus/evalplus?tab=readme-ov-file#-llm-generated-code
• authors of MultiPL-E share their code completitions here (python is included; Java is already in LASSO's corpus)
  • https://huggingface.co/datasets/bigcode/MultiPL-E-completions
  • possible related datasets
    • https://huggingface.co/datasets/bigcode/humanevalpack
    • https://huggingface.co/datasets/nuprl/MultiPL-E-raw-data