Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial DSL mutator concept #221

Closed
wants to merge 4 commits into from
Closed

Initial DSL mutator concept #221

wants to merge 4 commits into from

Conversation

velo
Copy link
Contributor

@velo velo commented Jul 27, 2015

Also on users lists:
https://groups.google.com/forum/#!topic/pitusers/o4Vpb_KI9Fc

I'm looking into mutating DSLs, for instance camel DSL:

from("file:src/data?noop=true")
  .to("file:target/messages/uk")
  .to("file:target/messages/others");

So I wanna mutations to remove DSL steps to make sure all my configurations (or at least most) are meaningful and being tested somewhere. So that piece of code would be mutated into something like:

from("file:src/data?noop=true")
  .to("file:target/messages/others");

AND/OR

from("file:src/data?noop=true")
  .to("file:target/messages/uk");

I also wanna test @builders and queries DSL.

So pitest would remove query conditions or builder parameters.

I started creating my own mutator for it (which I will submit as a pull request once properly tested)

Right now, it does remove any method that return the same type as the class that declares it.
https://github.com/hcoles/pitest/pull/221/files#diff-abcacb866451ca350bef8b81ef634c9bR35

It is not merge ready, it is just to illustrate what I'm doing.

But when I do that, dsl.chain(1) become to have the same effect as dsl = null would be
https://github.com/hcoles/pitest/pull/221/files#diff-0f0887a036c22255aa52f202e494fe58R90

And a NPE happen:

java.lang.NullPointerException
    at org.pitest.mutationtest.engine.gregor.mutators.DSLMethodCallMutatorTest$HasDslMethodCall.call(DSLMethodCallMutatorTest.java:90)
    at org.pitest.mutationtest.engine.gregor.mutators.DSLMethodCallMutatorTest$HasDslMethodCall.call(DSLMethodCallMutatorTest.java:1)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.runInClassLoader(MutatorTestBase.java:151)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.mutateAndCall(MutatorTestBase.java:117)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.assertMutantCallableReturns(MutatorTestBase.java:106)
    at org.pitest.mutationtest.engine.gregor.mutators.DSLMethodCallMutatorTest.shouldRemoveDslMethods(DSLMethodCallMutatorTest.java:128)

I think I can find the problem by myself if is there any way to get ASMifier running on a class after mutated, is that possible?
Or, if what I'm doing wrong is obvious, can someone point out?

Is there anything obviously wrong that you guys see on my code?

Keep in mind this is just my initial work, as I master what need to be done I will fix checkstyle and make the design as similar as possible to what is already there.

Also, is this functionality something that worth incorporating to pitest?

Marvin



@Test
@Ignore
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, my bytecode manipulation knowledge is too limited to get this one working, but I appreciate any help!

@velo
Copy link
Contributor Author

velo commented Sep 24, 2015

@hcoles
For your appreciation

@UrsMetz
Copy link
Contributor

UrsMetz commented Sep 25, 2015

@velo I might me mistaken but I think this pull request aims to introduce the same feature as my pull request #218 (naked receiver): Removal of instance method calls that return the same type as the receiver of the call. So I think we should coordinate our pull requests.

@velo
Copy link
Contributor Author

velo commented Sep 26, 2015

@UrsMetz
I tried to run my tests against you change... got this:

java.lang.AssertionError: org.objectweb.asm.tree.analysis.AnalyzerException: Error at instruction 5: Method owner: expected Lorg/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$HasDslMethodCall;, but found I
    at org.objectweb.asm.tree.analysis.Analyzer.analyze(Unknown Source)
    at org.objectweb.asm.util.CheckClassAdapter.verify(Unknown Source)
    at org.objectweb.asm.util.CheckClassAdapter.verify(Unknown Source)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.verifyMutant(MutatorTestBase.java:191)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.getFirstMutant(MutatorTestBase.java:178)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.getFirstMutant(MutatorTestBase.java:184)
    at org.pitest.mutationtest.engine.gregor.mutators.DSLMethodCallMutatorTest.shouldRemoveDslMethods(DSLMethodCallMutatorTest.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: org.objectweb.asm.tree.analysis.AnalyzerException: Method owner: expected Lorg/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$HasDslMethodCall;, but found I
    at org.objectweb.asm.tree.analysis.BasicVerifier.naryOperation(Unknown Source)
    at org.objectweb.asm.tree.analysis.BasicVerifier.naryOperation(Unknown Source)
    at org.objectweb.asm.tree.analysis.Frame.execute(Unknown Source)
    ... 31 more
call()Ljava/lang/String;
00000 DSLMethodCallMutatorTest$HasDslMethodCall .  :  :     ALOAD 0
00001 DSLMethodCallMutatorTest$HasDslMethodCall .  : DSLMethodCallMutatorTest$HasDslMethodCall  :     ASTORE 1
00002 DSLMethodCallMutatorTest$HasDslMethodCall DSLMethodCallMutatorTest$HasDslMethodCall  :  :     ALOAD 1
00003 DSLMethodCallMutatorTest$HasDslMethodCall DSLMethodCallMutatorTest$HasDslMethodCall  : DSLMethodCallMutatorTest$HasDslMethodCall  :     ICONST_1
00004 DSLMethodCallMutatorTest$HasDslMethodCall DSLMethodCallMutatorTest$HasDslMethodCall  : DSLMethodCallMutatorTest$HasDslMethodCall I  :     ICONST_3
00005 DSLMethodCallMutatorTest$HasDslMethodCall DSLMethodCallMutatorTest$HasDslMethodCall  : DSLMethodCallMutatorTest$HasDslMethodCall I I  :     INVOKEVIRTUAL org/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$HasDslMethodCall.nonDsl (I)I
00006 ?      :     POP
00007 ?      :     NEW java/lang/StringBuilder
00008 ?      :     DUP
00009 ?      :     INVOKESPECIAL java/lang/StringBuilder.<init> ()V
00010 ?      :     ALOAD 1
00011 ?      :     INVOKEVIRTUAL java/lang/StringBuilder.append (Ljava/lang/Object;)Ljava/lang/StringBuilder;
00012 ?      :     INVOKEVIRTUAL java/lang/StringBuilder.toString ()Ljava/lang/String;
00013 ?      :     ARETURN


    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.verifyMutant(MutatorTestBase.java:192)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.getFirstMutant(MutatorTestBase.java:178)
    at org.pitest.mutationtest.engine.gregor.MutatorTestBase.getFirstMutant(MutatorTestBase.java:184)
    at org.pitest.mutationtest.engine.gregor.mutators.DSLMethodCallMutatorTest.shouldRemoveDslMethods(DSLMethodCallMutatorTest.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

any ideas?

@UrsMetz
Copy link
Contributor

UrsMetz commented Sep 26, 2015

@velo good catch: the reason why shouldRemoveDslMethods fails is that I forgot to pop the method arguments from the stack in case the method call is removed. I incorporated your test case into the existing test class NakedReceiverMutatorTest. I hope that is all right with you. I'm going to push the fix to my branch for pull request #218 in a minute.

Concerning shouldRemoveInheritedDslMethods: the problem is that to make this test green we have to take inheritance/subtyping relations into consideration. I think there must be some way to do that but I didn't find a solution the last time I looked for one.

@velo
Copy link
Contributor Author

velo commented Sep 26, 2015

Yeah, I hit a dead end for this as well....

mv.visitMethodInsn(INVOKEVIRTUAL, "org/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$ChildHasDslMethodCall", "chain", "(I)Lorg/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$ParentHasDslMethodCall;", false);
mv.visitTypeInsn(CHECKCAST, "org/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$ChildHasDslMethodCall");

I notice the method will be called on parent and after that CAST to the invoking type.

We would need to check for that, them pop the visitMethodInsn(), just don't know how to.

@UrsMetz
Copy link
Contributor

UrsMetz commented Sep 27, 2015

@velo today I tried to tackle the problem with subtyping: I played around with an approach inspired by http://stackoverflow.com/a/14723212. But I think the problem is that removing method calls having a return type that is a sub or super type of the receiver can produce invalid byte code. E.g. mutating (when Bar extends Foo and Foo#bar returns Bar)

Bar bar = new Foo().bar()

to

Bar bar = new Foo()

would lead to an byte code validation error (marked blue in the HTML report).

@velo
Copy link
Contributor Author

velo commented Sep 27, 2015

I found it helps a lot to have the original bytecode and your expected bytecode side by side on beyond compare =D

@velo
Copy link
Contributor Author

velo commented Sep 28, 2015

Just need to pop invokeVirtual out...
image

@UrsMetz
Copy link
Contributor

UrsMetz commented Sep 28, 2015

@velo: here is an example where my experiments led to an error by ASM saying "Incompatible return type" (the mutation consisted in removing the call to baz()):

class Bar {
  public Baz baz() {
    return new Baz();
  }
}

class Baz extends Bar {
}

class Caller implements Callable<Baz> {
  public Baz call() throws Exception {
    return new Bar().baz();
  }
}

So I think without more context of what the type of the "value" (variable or return value of a method) is there is no reliable way produce valid byte code when subtyping should also be taken in consideration. From what I gather @hcoles tends avoid mutation in PIT that lead to invalid byte code (cf. discussion in #139 or also #218). Am I seeing this right, @hcoles?

Update: added extends Bar to Baz to make it a subtype of Bar.

@velo
Copy link
Contributor Author

velo commented Sep 28, 2015

But you can't remove baz() on this example...

you are breaking the method signature, like:

class Caller implements Callable<Baz> {
  public Baz call() throws Exception {
    return new Bar(); //wont compile
  }
}

@velo
Copy link
Contributor Author

velo commented Sep 28, 2015

Also, thinking o a builder for instance... doesn't make sense to remove the build() method. Makes a lot of sense removing a random name("me") in the middle

@UrsMetz
Copy link
Contributor

UrsMetz commented Sep 29, 2015

But you can't remove baz() on this example...

@velo That was actually my point (my comment was a bit unclear, sorry): removing the call to baz() creates invalid Java code (and doing the same on the byte code level creates invalid byte code) but that is what could happen when not only calls that return the same type as the receiver are removed but also calls that return a subtype of the receiver: without more context the mutator cannot know whether invalid byte code would be created. So I think at least for now the case with methods returning subtypes should be left out.

@velo
Copy link
Contributor Author

velo commented Sep 29, 2015

The example you pasted there is no subtypes...

@UrsMetz
Copy link
Contributor

UrsMetz commented Sep 29, 2015

@velo you're right: I forgot the extends Bar on Baz :-(. I will fix my previous comment.

@velo
Copy link
Contributor Author

velo commented Sep 30, 2015

Still, if you remove baz() it does not compile!
image

Unless you make

class Bar extends Baz{

That works like a charm

@UrsMetz
Copy link
Contributor

UrsMetz commented Sep 30, 2015

Still, if you remove baz() it does not compile!

That's actually the point I'm trying to make: we cannot remove calls to methods that return a different type than the type of the receiver (like removing the call to baz()) even when they are in a sub typing relationship. This is because as in my example there are situation where this leads to illegal "code": Both the byte code is not accepted by the JVM and the corresponding Java source code is illegal. This does not always hold true (as your example in the test shouldRemoveInheritedDslMethods shows) but AFAIK there is no way at the moment to distinguish these situations in PIT mutators.

@velo
Copy link
Contributor Author

velo commented Sep 30, 2015

We can remove when subtyping... you removed when "supertyping" =D

@UrsMetz
Copy link
Contributor

UrsMetz commented Oct 5, 2015

@velo depends how you define it :-). For me it is "sub typing" because the method we want to remove returns a subtype (in my example Baz) of the type of the receiver (Bar).

@velo
Copy link
Contributor Author

velo commented Oct 5, 2015

Baz is instance of Bar
Bar it is not Baz

You could do what you suggest if Bar extends Baz

@UrsMetz
Copy link
Contributor

UrsMetz commented Oct 10, 2015

@velo I think I have to clear up my point:

If you take code like:

class Bar {
  public Baz baz() {
    return new Baz();
  }
}

class Baz extends Bar {
}

class Caller implements Callable<Baz> {
  public Baz call() throws Exception {
    return new Bar().baz();
  }
}

When a mutation that removes instance methods that return a type inheriting the type of the receiver is applied the resulting code is not valid (as we both agreed). For that reason I think we can't make the mutation do that (or at least only an optional feature).
There are situations where such a mutation is valid but not generally. So as the default I would opt against mutating in the case there is an inheritance relationship between the type of the receiver and the return type.
Please tell me when I'm think in the wrong direction or missed something.

@velo
Copy link
Contributor Author

velo commented Oct 10, 2015

thinking on DSL I see 2 scenarios:
[x] remove the call when the return type is the same as the calling, that works well with DSLs like camel-dsl, lombok builder and probably many many other
[ ] remove the call, when the return type and the calling type get a cast to be the same. That is what you are going to see on DSLs like queryDSL, that use generics

mv.visitVarInsn(ALOAD, 0);
mv.visitInsn(ICONST_1);
mv.visitMethodInsn(INVOKEVIRTUAL, "org/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$ChildHasDslMethodCall", "chain", "(I)Lorg/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$ParentHasDslMethodCall;", false);
mv.visitTypeInsn(CHECKCAST, "org/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$ChildHasDslMethodCall");
mv.visitInsn(ICONST_3);
mv.visitMethodInsn(INVOKEVIRTUAL, "org/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$ChildHasDslMethodCall", "chain", "(I)Lorg/pitest/mutationtest/engine/gregor/mutators/DSLMethodCallMutatorTest$ParentHasDslMethodCall;", false);
mv.visitInsn(POP);

You can see, the call to one type ChildHasDslMethodCall and the return is a different type ParentHasDslMethodCall! But, on the next line, the return is cast into ChildHasDslMethodCall. So in that case, we can remove it.

I just don't know how to turn that into code.

@velo
Copy link
Contributor Author

velo commented Jul 5, 2016

Closing old pull requests w/o activity

@velo velo closed this Jul 5, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants