Couldn't fix QuixBugs lis bugs

[Yuanguo-notebook]

Hello, we saw in your paper that your code could fix bugs in QuixBugs lis with line mode with java. I wonder what parameters we should set to get the results? Currently we ran with epoch 30 and iter 100. We got no patch found. If we ran with 50 epochs, we met with infinite loop and the program just went frozen. Thank you very much!

[agb94]

Hi :), thanks for reaching us.
https://github.com/agb94/pyggi-quixbugs
👆 This repo contains the script & results that I used for the experiment.
Maybe you can try with those scripts. Please let me know if there’s a problem with using it. Thanks!

[Yuanguo-notebook]

Hello, I tried to run your scripts. It just hangs at program.py exec_cmd() shlex.split(cmd) forever. Timeout seems not working.
from terminal:
tmp: ./.pyggi/tmp_variants/java_programs/1638933294 cmd: None

code:
def exec_cmd(self, cmd, timeout=15):
cwd = os.getcwd()
os.chdir(self.tmp_path)
print('tmp: ', self.tmp_path)
print('cmd: ', cmd)
print(shlex.split(cmd))
print('after')
sprocess = subprocess.Popen(
shlex.split(cmd),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# stdout, stderr = process.communicate()
print('after open')
try:
start = time.time()
stdout, stderr = sprocess.communicate(timeout=timeout)
end = time.time()
return (sprocess.returncode, stdout.decode("ascii"), stderr.decode("ascii"), end-start)
except subprocess.TimeoutExpired:
sprocess.kill()
return (None, None, None, None)
finally:
os.chdir(cwd)

[agb94]

Then could you please try it with the recent version of pyggi?
The exec_cmd is slightly updated.

[Yuanguo-notebook]

OK, which branch I should use? I used master branch in https://github.com/coinse/pyggi
Thanks!

[agb94]

I merged a PR into the master branch yesterday, so you can use the current master branch!

[Yuanguo-notebook]

I just pulled the latest version. But it's still hanging on Popen. I think shlex.split(cmd) is still causing this. Also, your latest version misses StatusCode
Thank you!

def exec_cmd(self, cmd, timeout=15):
cwd = os.getcwd()
os.chdir(self.tmp_path)
print('tmp path: ', self.tmp_path)
print('split: ', shlex.split(cmd))
sprocess = subprocess.Popen(
shlex.split(cmd),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
preexec_fn=os.setsid)
print('after popen')
try:
start = time.time()
stdout, stderr = sprocess.communicate(timeout=timeout)
end = time.time()
return (sprocess.returncode, stdout.decode("ascii"), stderr.decode("ascii"), end-start)
except subprocess.TimeoutExpired:
os.killpg(os.getpgid(sprocess.pid), signal.SIGKILL)
_, _ = sprocess.communicate()
return (None, None, None, None)
finally:
os.chdir(cwd)

[agb94]

Oh, right. The StatusCode class is changed into RunResult class (21b1fbc), so the pyggi-quixbug script should be revised. Sorry for the inconvenience.

• Have you separately run the shlex.split(cmd)? (I guess so because your code contains print('split: ', shlex.split(cmd))). Does it produce output?
• I think this problem may be related to the known Popen timeout issues, not the shlex, because the Popen's implementation is different across the system OS and the python version...: https://stackoverflow.com/questions/50338959/subprocess-run-isnt-timing-out-even-though-timeout-is-specified.

[Yuanguo-notebook]

print('split: ', shlex.split(cmd)) doesn't print anything. It just hangs there. So I think the problem might be shlex.split

[agb94]

Are you using Windows?

[Yuanguo-notebook]

No I am using Mac

[agb94]

that's weird... what is the value of cmd? I'll test it on my machine.

[Yuanguo-notebook]

I just printed cmd. It's None. Because self.test_command is None in run().

[agb94]

I just found that the quixbug script on GitHub is completely not compatible with the current version of pyggi.
I may have the latest version of the script, but I guess it is on my previous laptop.
Inquixbugs-java.py, I defined the custom class MyLineProgram and override the run method, which is now evaluate_patch in pyggi/base/program.py.

• MyLineProgram/run in quixbugs-java.py

    def run(self, timeout=10):
        try_compile = self.exec_cmd(COMPILE_COMMAND.format(self.algo.upper()), timeout)
        if try_compile.stderr:
            result = self.__class__.Result(
                            status_code=StatusCode.PARSE_ERROR,
                            elapsed_time=try_compile.elapsed_time,
                            stdout=try_compile.stdout,
                            stderr=try_compile.stderr)
        else:
            failing = 0
            elapsed_time = 0
            for test in self.tests:
                test_in, test_out = test
                algo_input = ['"'+json.dumps(arg)+'"' for arg in copy.deepcopy(test_in)]
                try_test = self.exec_cmd(TEST_COMMAND.format(self.algo, ' '.join(algo_input)), timeout)
                #print(TEST_COMMAND.format(self.algo, ' '.join(algo_input)))
                #print(try_test.stdout.strip(), str(test_out))
                if try_test.stdout is None or try_test.stdout.rstrip() != str(test_out):
                    failing += 1
                elapsed_time += try_test.elapsed_time if try_test.elapsed_time else timeout
            result = self.__class__.Result(
                status_code=StatusCode.NORMAL,
                elapsed_time=elapsed_time,
                stdout=str(failing),
                stderr='')
        return result

• pyggi/base/program.py

 def evaluate_patch(self, patch, timeout=15):
        # apply + run
        self.apply(patch)
        return_code, stdout, stderr, elapsed_time = self.exec_cmd(self.test_command, timeout)
        if return_code is None: # timeout
            return RunResult('TIMEOUT')
        else:
            result = RunResult('SUCCESS', None)
            self.compute_fitness(result, return_code, stdout, stderr, elapsed_time)
            assert not (result.status == 'SUCCESS' and result.fitness is None)
            return result

To use the current pyggi version, Instead of overriding run, you should override evaluate_patch when defining the MyLineProgram,

ex)

class MyLineProgram(LineProgram):
   def evaluate_patch(self, patch, timeout=15):
          # apply the patch
          self.apply(patch)
         # try compile
         return_code, stdout, stderr, elapsed_time  = self.exec_cmd(COMPILE_COMMAND.format(self.algo.upper()), timeout)
         if stderr:
              return RunResult('Compile Error')
        else:
            failing = 0
            elapsed_time = 0
            for test in self.tests: # self.tests is defined in the __main__ function
                test_in, test_out = test
                algo_input = ['"'+json.dumps(arg)+'"' for arg in copy.deepcopy(test_in)]
                return_code, stdout, stderr, elapsed_time = self.exec_cmd(TEST_COMMAND.format(self.algo, ' '.join(algo_input)), timeout)
                if stdout is None or stdout.rstrip() != str(test_out):
                    failing += 1
                elapsed_time += elapsed_time if elapsed_time else timeout
        return RunResult('<something>', fitness=failing)

I haven't run this code, so it may have some minor issues, but I think you'll get the idea.