This repository will remain anonymous until the notification of a paper submitted to ICSE-2023. If you have any questions for the developers, we will give contact information after the notification.

Table of Contents

• General info
• How to build PyEvolve
• Research
  • How to cite PyEvolve
• API usage guidelines
• Running PyEvolve from the command line

General info

PyEvolve automates the frequently repeated code changes in Python systems. This tool presents a complete pipeline for mining and automating best code evolution practices, ensuring that the your project does not fall behind. The following is a high-level overview of the pipeline.

• Phase 1: We use R-CPATMiner to mine best practices from version history of Python systems. You can find details on executing R-CPATMiner in its project description.
• Phase 2: We infer the initial transformation rules for the patterns detected by R-CPATMiner.
• Phase 3: We identify the potential sites to apply the patterns in the target codes.
• Phase 4: We infer final adapted rules to tranplant the identified pattern.

How to build PyEvolve

To have a fully built PyEvolve, you have to install the following components.

• We use RulePharser to generate an AST for Comby templates that includes both Python and Comby syntaxes. Follow the steps in RulePharser to build it locally and add it to your local maven repository.
• We use ComBy as a backend tool to rewrite code. Please follow the steps in their documentation to install it on your PC.
• We use wala.ml. To install it, follow the instructions on wala.ml.

After completing the above steps, run ./gradlew build from the root directory to build the project. This will build the project and execute the test cases. If you want to build the project without running the tests, use the command ./gradlew build -x test.

API usage guidelines

Using PyEvolve to identify code usages

In this example, we identify two usages of the following pattern in the project PatternTest

Pattern :

# type :[l3] : List[int]" 
# type :[[l1]] : int" 
# type :[[l2]] : int" 
:[[l1]] = 0" 
for :[[l2]] in :[l3]: 
   :[[l1]]=:[[l1]]+:[[l2]]

1. Clone the project PyEvolve
2. Update Configurations.PROJECT_REPOSITORY and Configurations.TYPE_REPOSITORY configurations with valid paths.
3. Clone the project PatternTest to the directory Configurations.PROJECT_REPOSITORY + "/pythonInfer/"
4. Execute the script type_infer.py to infer type information of the project PatternTest
5. Copy the generated json files into the folder Configurations.TYPE_REPOSITORY + "/pythonInfer/PatternTest/"
6. Execute the following code to get all the usages of the Pattern

import com.utils.Utils;
import org.junit.jupiter.api.Test;

import static com.matching.fgpdg.Configurations.PROJECT_REPOSITORY;  // This must be set to root directory of the folder "pythonInfer"


public class DetectPattern {

    @Test
    void testPattern() throws Exception {
        String pattern = "# type number : Any\n" +
                "# type :[l3] : List[int]\n" +
                "# type :[[l1]] : int\n" +
                "# type :[[l2]] : int\n" +
                ":[[l1]] = 0\n" +
                "for :[[l2]] in :[l3]:\n" +
                "   :[[l1]]=:[[l1]]+:[[l2]]";
        String outPath = "./OUTPUT/"; 
        String projectPath =  PROJECT_REPOSITORY +"pythonInfer/PatternTest";
        System.out.println(pattern);
        Utils.processProjectForPattern(projectPath,pattern,outPath);
    }
}

Using PyEvolve for transplanting a rule

To a file

PyEvolve can assist you if you have a rule encoded in ComBy syntax and want to  transplant it to a Python file.

    void transplantPatternToFile() {
         Configurations.TYPE_REPOSITORY = ""; // this should the path to your type repository 
         Configurations.PROJECT_REPOSITORY = "" // this should be the path to your projects repository 
         String projectFile = ""; // this should be the path to your file, this path should be the relative path to Configurations.PROJECT_REPOSITORY
         String LHS = ""; // path to the LHS of the rule file
         String RHS = "";  // path to the RHS of the rule file
         String adaptedFile = MainAdaptor.transplantPatternToFile(projectFile, LHS, RHS);   // adaptedFile is the adapted file
    }

To a function

PyEvolve can assist you if you have a rule encoded in ComBy syntax and want to  transplant it to a perticular Python function.

    void transplantPatternToFunction() {
        Configurations.TYPE_REPOSITORY = ""; // this should the path to your type repository 
        Configurations.PROJECT_REPOSITORY = "" // this should be the path to your projects repository 
        String projectFile = " "; // this should be the path to your file, this path should be the relative path to Configurations.PROJECT_REPOSITOR
        String LHS = ""; // path to the LHS of the rule file
        String RHS = "";  // path to the RHS of the rule file
        Module codeModule = com.utils.Utils.getPythonModule(projectFile);
        stmt stmt = codeModule.getInternalBody().get(i); // i must be the statment number of the function.   
        List<org.python.antlr.base.stmt> imports = codeModule.getInternalBody().stream().filter(x -> x instanceof Import
                || x instanceof ImportFrom).collect(Collectors.toList());
        String adaptedFunction = MainAdaptor.transplantPatternToFunction(projectFile, (FunctionDef) stmt,imports,LHS, RHS);
    }

Using PyEvolve for automation

We will discuss the APIs that can be used for code automation, using the following code example.

The following is a best code evolution practice discovered by R-CPATMiner.

res = 0
for elem in elems:
  res = res + elem

==>

res = np.sum(elems)

Our goal is to transplant the above recommended practice to the target code listed below.

def getSum()
  n_diff = 0
  to_eval = getNumber()
  for dif in to_eval.getDiff():
    total = n_diff + dif
    n_diff = total
return n_diff    

We will now describe the APIs that can be used for above modification.

1. With pattern and target function as input

MainAdaptor adaptor = new MainAdaptor();
Module codeModule = getPythonModule("/pathToFunction.py"); // pathToFunction is the String value of the file path which contains the above target code.
Module lpatternModule = getPythonModuleForTemplate("/pathToLPattern.py");// pathToLPattern is the string value of the file path which has the LHS of the pattern.
Module rpatternModule = getPythonModuleForTemplate("/pathToRPattern.py"); // pathToRFunction is the string value of the file path which has the RHS of the pattern.
List<stmt> imports = lpatternModule.getInternalBody().stream().filter(x -> x instanceof Import
                || x instanceof ImportFrom).collect(Collectors.toList());
Guards guards = new Guards("/pathToLPattern.py",lpatternModule);
String adaptedCode = adaptor.adaptFunction(imports, guards, lpatternModule, rpatternModule, codeModule); // adaptedCode is the final adapted code

Research

We will add citation information as soon as possible.

License

All software provided in this repository is subject to the Apache License Version 2.0.

Executables

To extend PyEvolve - https://github.com/pythonInfer/PyEvolve/tree/master/build/distributions