New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for parallel test configuration #52

Merged
merged 2 commits into from Jan 9, 2013

Conversation

Projects
None yet
4 participants
@marchof
Member

marchof commented Jan 8, 2013

I've maven-surefire plugin configured with this options :
Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false

This means, that maven runs 4 separate JVM, each JVM has own jacoco-agent and all agents needs to write to the same target/jacoco.exec file.

After all tests are finished, I've malformed target/jacoco.exec file.

Caused by: java.io.UTFDataFormatException: malformed input around byte 0
    at java.io.DataInputStream.readUTF(DataInputStream.java:639)
    at java.io.DataInputStream.readUTF(DataInputStream.java:547)
    at org.jacoco.core.data.ExecutionDataReader.readExecutionData(ExecutionDataReader.java:145)

Is there a synchronisation/write locking for target/jacoco.exec ?

I think that this is continuation of http://sourceforge.net/apps/trac/eclemma/ticket/191

@flozano

This comment has been minimized.

Show comment
Hide comment
@flozano

flozano Jan 8, 2013

Here we reproduced this problem with a different error message, I will add it as a reference for others who find similar problem:

java.io.IOException: Unknown block type b.
        at org.jacoco.core.data.ExecutionDataReader.readBlock(ExecutionDataReader.java:114)
        at org.jacoco.core.data.ExecutionDataReader.read(ExecutionDataReader.java:85)
        at org.sonar.plugins.jacoco.AbstractAnalyzer.readExecutionData(AbstractAnalyzer.java:76)
        at org.sonar.plugins.jacoco.AbstractAnalyzer.analyse(AbstractAnalyzer.java:59)
        ... 38 more

flozano commented Jan 8, 2013

Here we reproduced this problem with a different error message, I will add it as a reference for others who find similar problem:

java.io.IOException: Unknown block type b.
        at org.jacoco.core.data.ExecutionDataReader.readBlock(ExecutionDataReader.java:114)
        at org.jacoco.core.data.ExecutionDataReader.read(ExecutionDataReader.java:85)
        at org.sonar.plugins.jacoco.AbstractAnalyzer.readExecutionData(AbstractAnalyzer.java:76)
        at org.sonar.plugins.jacoco.AbstractAnalyzer.analyse(AbstractAnalyzer.java:59)
        ... 38 more

@Godin Godin referenced this pull request Jan 8, 2013

Merged

Runtime Agent API #61

@Godin

This comment has been minimized.

Show comment
Hide comment
@Godin

Godin Jan 8, 2013

Member

@marchof Indeed we can lock file, when we dump coverage (i.e. for a normal scenario - during JVM shutdown) - http://docs.oracle.com/javase/1.4.2/docs/api/java/nio/channels/FileChannel.html#lock() Do you see a drawbacks or side effects of such solution? For example - I don't know what happens with locks if JVM was killed or crashed. Or we'd like to avoid locks (i.e. concurrent access to dump file) and should find another solution to this problem?

Member

Godin commented Jan 8, 2013

@marchof Indeed we can lock file, when we dump coverage (i.e. for a normal scenario - during JVM shutdown) - http://docs.oracle.com/javase/1.4.2/docs/api/java/nio/channels/FileChannel.html#lock() Do you see a drawbacks or side effects of such solution? For example - I don't know what happens with locks if JVM was killed or crashed. Or we'd like to avoid locks (i.e. concurrent access to dump file) and should find another solution to this problem?

@marchof

This comment has been minimized.

Show comment
Hide comment
@marchof

marchof Jan 8, 2013

Member

Let's give it a try. Writing exec data is quite fast, so waiting for the exclusive lock shoudn't be an issue. Hopefully the OS frees the lock once the process goes away -- but we can test this.

Member

marchof commented Jan 8, 2013

Let's give it a try. Writing exec data is quite fast, so waiting for the exclusive lock shoudn't be an issue. Hopefully the OS frees the lock once the process goes away -- but we can test this.

@ghost ghost assigned Godin Jan 8, 2013

@marchof

This comment has been minimized.

Show comment
Hide comment
@marchof

marchof Jan 8, 2013

Member

@Godin Hi Evgeny, the file lock was an excellent idea and quite simple to implement. Do you have a parallel test example at hand?

Member

marchof commented Jan 8, 2013

@Godin Hi Evgeny, the file lock was an excellent idea and quite simple to implement. Do you have a parallel test example at hand?

@Godin

This comment has been minimized.

Show comment
Hide comment
@Godin

Godin Jan 8, 2013

Member

@marchof Yes, I have. So I will do the tests.

Member

Godin commented Jan 8, 2013

@marchof Yes, I have. So I will do the tests.

return new FileOutputStream(destFile, append);
final FileOutputStream file = new FileOutputStream(destFile, append);
// Avoid concurrent writes from different agents running in parallel:
file.getChannel().lock();

This comment has been minimized.

@Godin

Godin Jan 8, 2013

Member

But even without tests - this method returns FileLock which should be released.

@Godin

Godin Jan 8, 2013

Member

But even without tests - this method returns FileLock which should be released.

This comment has been minimized.

@marchof

marchof Jan 8, 2013

Member

JavaDoc says: "It remains valid until the lock is released by invoking the release method, by closing the channel that was used to acquire it, or by the termination of the Java virtual machine, whichever comes first."

I verified this in OS X: As soon as I close the FileOutputStream another process can acquire the lock.

@marchof

marchof Jan 8, 2013

Member

JavaDoc says: "It remains valid until the lock is released by invoking the release method, by closing the channel that was used to acquire it, or by the termination of the Java virtual machine, whichever comes first."

I verified this in OS X: As soon as I close the FileOutputStream another process can acquire the lock.

This comment has been minimized.

@Godin

Godin Jan 8, 2013

Member

Indeed - FileOutputStream.close closes the channel. But now what about your second statement about concurrent dump operations?

@Godin

Godin Jan 8, 2013

Member

Indeed - FileOutputStream.close closes the channel. But now what about your second statement about concurrent dump operations?

This comment has been minimized.

@marchof

marchof Jan 8, 2013

Member

Sorry, I didn't forget to mention this: Probably we're already safe here: The method

org.jacoco.core.runtime.RuntimeData.collect(IExecutionDataVisitor, ISessionInfoVisitor, boolean)

already is synchronized, so I don't think parallel writes from the same agent are possible.

@marchof

marchof Jan 8, 2013

Member

Sorry, I didn't forget to mention this: Probably we're already safe here: The method

org.jacoco.core.runtime.RuntimeData.collect(IExecutionDataVisitor, ISessionInfoVisitor, boolean)

already is synchronized, so I don't think parallel writes from the same agent are possible.

This comment has been minimized.

@Godin

Godin Jan 8, 2013

Member

Same feelings. But anyway I'll do more tests (including real-life) tomorrow morning and will come back to you about merge.

@Godin

Godin Jan 8, 2013

Member

Same feelings. But anyway I'll do more tests (including real-life) tomorrow morning and will come back to you about merge.

@Godin

This comment has been minimized.

Show comment
Hide comment
@Godin

Godin Jan 9, 2013

Member

@marchof Works fine for me, so lets merge!

Member

Godin commented Jan 9, 2013

@marchof Works fine for me, so lets merge!

marchof added a commit that referenced this pull request Jan 9, 2013

Merge pull request #52 from jacoco/issue-52
Support for parallel test configuration

@marchof marchof merged commit cbb58db into master Jan 9, 2013

1 check passed

default The Travis build passed
Details

@marchof marchof deleted the issue-52 branch Jan 9, 2013

@Raynevun

This comment has been minimized.

Show comment
Hide comment
@Raynevun

Raynevun Sep 9, 2015

Still reproducible for me, when using TeamCity 9.1.1 (build 37059 which should contain JaCoCo 0.7.5) integration and running unit and integration tests in parallel with two agents writing in one exec file.

Raynevun commented Sep 9, 2015

Still reproducible for me, when using TeamCity 9.1.1 (build 37059 which should contain JaCoCo 0.7.5) integration and running unit and integration tests in parallel with two agents writing in one exec file.

@Godin

This comment has been minimized.

Show comment
Hide comment
@Godin

Godin Sep 9, 2015

Member

@Raynevun The best place to get support is our users mailing list - https://groups.google.com/forum/#!forum/jacoco , so could you please switch to it? And describe problem more detailed, at least provide stacktrace of exception, also maybe you know a way to reproduce this issue without need to install TeamCity?

Thanks for your understanding.

Member

Godin commented Sep 9, 2015

@Raynevun The best place to get support is our users mailing list - https://groups.google.com/forum/#!forum/jacoco , so could you please switch to it? And describe problem more detailed, at least provide stacktrace of exception, also maybe you know a way to reproduce this issue without need to install TeamCity?

Thanks for your understanding.

@jacoco jacoco locked and limited conversation to collaborators Jan 11, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.