Dependency conflicts on org.xerial.snappy:snappy-java, leading to inconsistent program behaviors

[HelloCoCooo]

Hi, in KafkaExample-master/demokafka.0.10.1.0, there are mulptiple versions of library org.xerial.snappy:snappy-java. However, according to Maven's dependency management strategy: "first declaration wins", only org.xerial.snappy:snappy-java:1.1.1.6 can be loaded, and org.xerial.snappy:snappy-java:1.1.2.6 will be shadowed.

In total, there are 5 conflicting API pairs between these two library version.

As shown in the following figure, your project expects to invoke method <org.xerial.snappy.SnappyOutputStream: write([BII)V> in library org.xerial.snappy:snappy-java:1.1.2.6 (along the original dependency path). As it has been shadowed, this method defined in org.xerial.snappy:snappy-java:1.1.1.6 is actually forced to be referenced via the following invocation path (along the actual dependency path):

<com.jasongj.kafka.producer.ProducerDemo: main([Ljava/lang/String;)V> /home/wwww/wangSensor/unzip/KafkaExample-master/demokafka.0.10.1.0/target/classes
<org.apache.kafka.clients.producer.KafkaProducer: close()V> /home/wwww/.m2/repository/org/apache/kafka/kafka-clients/0.10.1.0/kafka-clients-0.10.1.0.jar
<org.apache.kafka.clients.producer.KafkaProducer: close(JLjava/util/concurrent/TimeUnit;)V> /home/wwww/.m2/repository/org/apache/kafka/kafka-clients/0.10.1.0/kafka-clients-0.10.1.0.jar
<org.apache.kafka.clients.producer.KafkaProducer: close(JLjava/util/concurrent/TimeUnit;Z)V> /home/wwww/.m2/repository/org/apache/kafka/kafka-clients/0.10.1.0/kafka-clients-0.10.1.0.jar
<org.apache.kafka.clients.ClientUtils: closeQuietly(Ljava/io/Closeable;Ljava/lang/String;Ljava/util/concurrent/atomic/AtomicReference;)V> /home/wwww/.m2/repository/org/apache/kafka/kafka-clients/0.10.1.0/kafka-clients-0.10.1.0.jar
<org.apache.kafka.common.record.KafkaLZ4BlockOutputStream: close()V> /home/wwww/.m2/repository/org/apache/kafka/kafka-clients/0.10.1.0/kafka-clients-0.10.1.0.jar
<org.apache.kafka.common.record.KafkaLZ4BlockOutputStream: flush()V> /home/wwww/.m2/repository/org/apache/kafka/kafka-clients/0.10.1.0/kafka-clients-0.10.1.0.jar
<org.apache.kafka.common.record.KafkaLZ4BlockOutputStream: writeBlock()V> /home/wwww/.m2/repository/org/apache/kafka/kafka-clients/0.10.1.0/kafka-clients-0.10.1.0.jar
<org.xerial.snappy.SnappyOutputStream: write([BII)V>

Although both of these conflicting libraries contain the referenced methods (with the same signature), they have different implementations. This issue will not cause runtime crashes, but it can introduce inconsistent semantic program hehaviors----

Code snippet of <org.xerial.snappy.SnappyOutputStream: write([BII)V> in org.xerial.snappy:snappy-java:1.1.2.6 (shadowed but expected to invoke method):

detailed method body

@Override
    public void write(byte[] b, int byteOffset, int byteLength)
            throws IOException
    {
        if (closed) {
            throw new IOException("Stream is closed");
        }
        int cursor = 0;
        while (cursor < byteLength) {
            int readLen = Math.min(byteLength - cursor, blockSize - inputCursor);
            // copy the input data to uncompressed buffer
            if (readLen > 0) {
                System.arraycopy(b, byteOffset + cursor, inputBuffer, inputCursor, readLen);
                inputCursor += readLen;
            }
            if (inputCursor < blockSize) {
                return;
            }

            compressInput();
            cursor += readLen;
        }
    }

Code snippet of <org.xerial.snappy.SnappyOutputStream: write([BII)V> in org.xerial.snappy:snappy-java:1.1.1.6 (loaded version):

detailed method body

@Override
public void write(byte[] b, int off, int len) throws IOException {
    rawWrite(b, off, len);
}

public void rawWrite(Object array, int byteOffset, int byteLength) throws IOException {

        if(inputCursor + byteLength < MIN_BLOCK_SIZE) {
            // copy the input data to uncompressed buffer
            Snappy.arrayCopy(array, byteOffset, byteLength, inputBuffer, inputCursor);
            inputCursor += byteLength;
            return;
        }

        compressInput();

        for(int readBytes = 0; readBytes < byteLength; ) {
            int inputLen = Math.min(blockSize, byteLength - readBytes);
            if(!hasSufficientOutputBufferFor(inputLen)) {
                dumpOutput();
            }
            int compressedSize = Snappy.rawCompress(array, byteOffset + readBytes, inputLen, outputBuffer, outputCursor + 4);
            writeInt(outputBuffer, outputCursor, compressedSize);
            outputCursor += 4 + compressedSize;
            readBytes += inputLen;
        }
    }

The detailed informantion of the remaining 4 conflicting API pairs can be found in the following attachment.
5 conflicting API pairs in project demokafka.txt

Dependency tree--

[INFO] com.jasongj.kafka:demokafka.0.10.1.0:jar:0.10.1.0
[INFO] +- org.apache.kafka:kafka_2.11:jar:0.10.1.0:compile
[INFO] | +- (org.apache.kafka:kafka-clients:jar:0.10.1.0:compile - omitted for duplicate)
[INFO] | +- net.sf.jopt-simple:jopt-simple:jar:4.9:compile
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] | | - (org.slf4j:slf4j-api:jar:1.7.2:compile - omitted for conflict with 1.7.21)
[INFO] | +- (org.scala-lang:scala-library:jar:2.11.8:compile - omitted for conflict with 2.11.5)
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.21:compile
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.21:compile - omitted for duplicate)
[INFO] | | - (log4j:log4j:jar:1.2.17:compile - omitted for duplicate)
[INFO] | +- (org.apache.zookeeper:zookeeper:jar:3.4.8:compile - omitted for conflict with 3.4.6)
[INFO] | - org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] | - (org.scala-lang:scala-library:jar:2.11.6:compile - omitted for conflict with 2.11.8)
[INFO] +- org.apache.kafka:kafka-clients:jar:0.10.1.0:compile
[INFO] | +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] | +- (org.xerial.snappy:snappy-java:jar:1.1.2.6:compile - omitted for conflict with 1.1.1.6)
[INFO] | - org.slf4j:slf4j-api:jar:1.7.21:compile
[INFO] +- org.apache.kafka:kafka-streams:jar:0.10.1.0:compile
[INFO] | +- (org.apache.kafka:kafka-clients:jar:0.10.1.0:compile - omitted for duplicate)
[INFO] | +- org.apache.kafka:connect-json:jar:0.10.1.0:compile
[INFO] | | +- org.apache.kafka:connect-api:jar:0.10.1.0:compile
[INFO] | | | +- (org.apache.kafka:kafka-clients:jar:0.10.1.0:compile - omitted for duplicate)
[INFO] | | | - (org.slf4j:slf4j-api:jar:1.7.21:compile - omitted for duplicate)
[INFO] | | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.6.3:compile - omitted for duplicate)
[INFO] | | - (org.slf4j:slf4j-api:jar:1.7.21:compile - omitted for duplicate)
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.21:compile - omitted for duplicate)
[INFO] | +- org.rocksdb:rocksdbjni:jar:4.9.0:compile
[INFO] | +- log4j:log4j:jar:1.2.17:compile
[INFO] | - com.fasterxml.jackson.core:jackson-databind:jar:2.6.3:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.0:compile
[INFO] | - com.fasterxml.jackson.core:jackson-core:jar:2.6.3:compile
[INFO] +- com.101tec:zkclient:jar:0.10:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.6.1:compile - omitted for conflict with 1.7.21)
[INFO] | - (org.apache.zookeeper:zookeeper:jar:3.4.8:compile - omitted for duplicate)
[INFO] +- commons-io:commons-io:jar:2.5:compile
[INFO] +- junit:junit:jar:4.11:test (scope not updated to compile)
[INFO] | - org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- com.google.guava:guava:jar:14.0-rc1:compile
[INFO] +- org.scala-lang:scala-library:jar:2.11.5:compile
[INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.6.1:compile - omitted for conflict with 1.7.21)
[INFO] | +- (org.slf4j:slf4j-log4j12:jar:1.6.1:compile - omitted for conflict with 1.7.21)
[INFO] | +- (log4j:log4j:jar:1.2.16:compile - omitted for conflict with 1.2.17)
[INFO] | +- jline:jline:jar:0.9.94:compile
[INFO] | | - (junit:junit:jar:3.8.1:compile - omitted for conflict with 4.11)
[INFO] | - io.netty:netty:jar:3.7.0.Final:compile
[INFO] +- org.xerial.snappy:snappy-java:jar:1.1.1.6:compile
[INFO] - org.apache.commons:commons-lang3:jar:3.5:compile

Suggested solutions:

Solution1: Declare version org.xerial.snappy:snappy-java:1.1.2.6 as a direct dependency, to override the version 1.1.1.6 (based on Maven's nearest wins loading strategy).

Solution2: Remove the conflicting Jars.

Thanks.
Best regards,
Coco

[HelloCoCooo]

@habren Could please help me check this issue?
May I pull a request to fix it?
Thanks again.