Upgraded storm-core to 0.10.0 version

[goldenlink]

To make it work on 0.10.0, I had to migrate guava dependencies to com.google.guava instead of org.apache.storm.guava. This library has been removed from Storm in 0.10.0 version.

[jnioche]

Hi, this should already be covered in branch [https://github.com/DigitalPebble/storm-crawler/tree/storm-0.10]. Can you please test it?

[goldenlink]

Sorry, I did not notice this branch, I'll test with it and let you know about it.

[goldenlink]

I don't really understand while trying to use the 0.10.0 option, I get an error while deploying :
on import org.apache.storm.shade.com.google.common.collect.Iterables;

Exception in thread "main" java.lang.Error: Unresolved compilation problems:
The import org.apache.storm.shade cannot be resolved
Iterables cannot be resolved
Iterables cannot be resolved

at com.digitalpebble.storm.crawler.bolt.FetcherBolt.<init>(FetcherBolt.java:40)

Is this normal ? an issue in my pom.xml maybe ?

[jnioche]

do you get this when running in local mode or only when deploying to a Storm cluster?
Definitely not normal. How does your pom differ from the example one in the 0.10 branch?

[goldenlink]

Well, I'm running in distributed mode. I took storm-0.10 branch HEAD. This happens while integrating the storm-crawler in my own application as a maven dependency :

 <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>1.4</version>
        <configuration>
          <createDependencyReducedPom>true</createDependencyReducedPom>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <filters>
                <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                  </excludes>
                </filter>
              </filters>
              <transformers>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer"
                >
                  <resource>core-site.xml</resource>
                  <file>src/main/resources/core-site.xml</file>
                </transformer>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer"
                >
                  <resource>hdfs-site.xml</resource>
                  <file>src/main/resources/hdfs-site.xml</file>
                </transformer>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer"
                >
                  <resource>my-parsefilters.json</resource>
                  <file>src/main/resources/my-parsefilters.json</file>
                </transformer>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer"
                >
                  <resource>my-urlfilters.json</resource>
                  <file>src/main/resources/my-urlfilters.json</file>
                </transformer>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"
                >
                  <mainClass>com....</mainClass>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>

And here the dependency :

<!-- Storm -->
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-core</artifactId>
      <version>${storm.version}</version>
      <scope>provided</scope>
    </dependency>
<dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-kafka</artifactId>
      <version>${storm.version}</version>
    </dependency>
<dependency>
      <artifactId>guava</artifactId>
      <groupId>com.google.guava</groupId>
      <version>16.0.1</version>
    </dependency>
<dependency>
      <groupId>com.digitalpebble</groupId>
      <artifactId>storm-crawler-core</artifactId>
      <version>${storm-crawler.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>guava</artifactId>
          <groupId>com.google.guava</groupId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.storm</groupId>
          <artifactId>storm-core</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>com.digitalpebble</groupId>
      <artifactId>storm-crawler-elasticsearch</artifactId>
      <version>${storm-crawler.version}</version>
    </dependency>
...

It used to work perfectly with 0.9.5 altogether.

[goldenlink]

Ok, I simply changed the dependency to com.google.common directly to make it deploy.

[jnioche]

great, let's leave this issue open - I'll investigate it later. Thanks @goldenlink

[jnioche]

For some reason we get the shaded dependency when compiling but not at runtime. Will merge your PR, thanks @goldenlink