Package extensions in tarball + run_druid_server goodness #1640

xvrl · 2015-08-19T00:48:48Z

No description provided.

gianm · 2015-08-19T00:55:07Z

👍

xvrl · 2015-08-19T00:55:34Z

👍

drcrallen · 2015-08-19T01:10:48Z

hmmm... when run locally : [ERROR] Failed to execute goal on project extensions-distribution: Could not resolve dependencies for project io.druid:extensions-distribution:pom:0.8.2-SNAPSHOT: The following artifacts could not be resolved: io.druid.extensions:druid-examples:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-namespace-lookup:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-cassandra-storage:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-kafka-eight:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-azure-extensions:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:mysql-metadata-storage:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-hdfs-storage:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:postgresql-metadata-storage:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-kafka-extraction-namespace:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-rabbitmq:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-s3-extensions:jar:0.8.1-rc2-SNAPSHOT, io.druid.extensions:druid-histogram:jar:0.8.1-rc2-SNAPSHOT: Could not find artifact io.druid.extensions:druid-examples:jar:0.8.1-rc2-SNAPSHOT in snapshots (https://metamx.artifactoryonline.com/metamx/libs-snapshots) -> [Help 1]

- move assembly out of druid-services into a 'distribution' module - create separate 'extensions-distribution' module and assembly to package extensions and their dependencies into a local maven repository - include this extensions maven repository in the binaries tarball

- Use local repo in run_example_server. - Add extensions-repo to common.runtime.properties for the tarball.

xvrl · 2015-08-19T01:34:17Z

@drcrallen ah that was just an artifact of us working on the 0.8.1-rc branch. Fixed the pom versions and everything should be happy again.

drcrallen · 2015-08-19T02:56:49Z

Extends build time by 30s on my local box.

Seems to work well, and the packaging mechanism is some crazy maven-foo.

run_druid_server.sh is not documented though. So that should probably be added at some point.

Looks good to me 👍 (waiting for Travis)

himanshug · 2015-08-19T03:20:20Z

LGTM

Package extensions in tarball + run_druid_server goodness

guobingkun · 2015-08-19T15:42:00Z

Is mvn install expected to fail?

05:55:12 [INFO] druid-kafka-extraction-namespace .................. SUCCESS [6.174s]
05:55:12 [INFO] extensions-distribution ........................... FAILURE [5.602s]
05:55:12 [INFO] distribution ...................................... SKIPPED
05:55:12 [INFO] ------------------------------------------------------------------------
05:55:12 [INFO] BUILD FAILURE
05:55:12 [INFO] ------------------------------------------------------------------------
05:55:12 [INFO] Total time: 6:24.537s
05:55:13 [INFO] Finished at: Wed Aug 19 05:55:12 UTC 2015
05:55:13 [INFO] Final Memory: 96M/480M
05:55:13 [INFO] ------------------------------------------------------------------------
05:55:13 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (distro-assembly) on project extensions-distribution: Execution distro-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single failed. NullPointerException -> [Help 1]

drcrallen · 2015-08-19T15:44:35Z

Expected? no

xvrl · 2015-08-19T15:50:11Z

@guobingkun make sure to mvn clean first

guobingkun · 2015-08-19T16:22:40Z

@xvrl Yeah, It happened on a clean state build machine.
I also did mvn -DskipTests clean package locally and got this

[INFO] extensions-distribution ............................ SUCCESS [  7.573 s]
[INFO] distribution ....................................... FAILURE [  1.211 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 48.376 s
[INFO] Finished at: 2015-08-19T11:09:04-05:00
[INFO] Final Memory: 112M/1463M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (distro-assembly) on project distribution: Execution distro-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single failed: group id '285597778' is too big ( > 2097151 ). Use STAR or POSIX extensions to overcome this limit -> [Help 1]

xvrl · 2015-08-19T17:16:21Z

@guobingkun it looks like your tar command does not support file names longer than 100 characters. I have a fix that might help in your case. Can you try this #1644

guobingkun · 2015-08-19T18:16:23Z

@xvrl #1644 fixes the problem I had when I ran mvn -DskipTests clean package locally.
But I still got Execution distro-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single failed. NullPointerException on the build machine after mvn -X -DskipTests clean install

18:03:46 [INFO] druid-kafka-extraction-namespace .................. SUCCESS [0.484s]
18:03:46 [INFO] extensions-distribution ........................... FAILURE [5.073s]
18:03:46 [INFO] distribution ...................................... SKIPPED
18:03:46 [INFO] ------------------------------------------------------------------------
18:03:46 [INFO] BUILD FAILURE
18:03:46 [INFO] ------------------------------------------------------------------------
18:03:46 [INFO] Total time: 1:30.993s
18:03:46 [INFO] Finished at: Wed Aug 19 18:03:46 UTC 2015
18:03:46 [INFO] Final Memory: 98M/454M
18:03:46 [INFO] ------------------------------------------------------------------------
18:03:46 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (distro-assembly) on project extensions-distribution: Execution distro-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single failed. NullPointerException -> [Help 1]
18:03:46 org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (distro-assembly) on project extensions-distribution: Execution distro-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single failed.
18:03:46    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
18:03:46    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
18:03:46    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
18:03:46    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
18:03:46    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
18:03:46    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
18:03:46    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
18:03:46    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
18:03:46    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
18:03:46    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
18:03:46    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
18:03:46    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
18:03:46    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
18:03:46    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
18:03:46    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
18:03:46    at java.lang.reflect.Method.invoke(Method.java:497)
18:03:46    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
18:03:46    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
18:03:46    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
18:03:46    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
18:03:46 Caused by: org.apache.maven.plugin.PluginExecutionException: Execution distro-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single failed.
18:03:46    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
18:03:46    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
18:03:46    ... 19 more
18:03:46 Caused by: java.lang.NullPointerException
18:03:46    at org.apache.maven.artifact.repository.metadata.AbstractRepositoryMetadata.getLocalFilename(AbstractRepositoryMetadata.java:60)
18:03:46    at org.apache.maven.artifact.repository.layout.DefaultRepositoryLayout.pathOfLocalRepositoryMetadata(DefaultRepositoryLayout.java:72)
18:03:46    at org.apache.maven.artifact.repository.DefaultArtifactRepository.pathOfLocalRepositoryMetadata(DefaultArtifactRepository.java:128)
18:03:46    at org.apache.maven.shared.repository.DefaultRepositoryAssembler.assembleRepositoryMetadata(DefaultRepositoryAssembler.java:487)
18:03:46    at org.apache.maven.shared.repository.DefaultRepositoryAssembler.buildRemoteRepository(DefaultRepositoryAssembler.java:232)
18:03:46    at org.apache.maven.plugin.assembly.archive.phase.RepositoryAssemblyPhase.execute(RepositoryAssemblyPhase.java:97)
18:03:46    at org.apache.maven.plugin.assembly.archive.DefaultAssemblyArchiver.createArchive(DefaultAssemblyArchiver.java:178)
18:03:46    at org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(AbstractAssemblyMojo.java:484)
18:03:46    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
18:03:46    ... 20 more

xvrl · 2015-08-19T18:23:51Z

@guobingkun what version of maven are you running?

guobingkun · 2015-08-19T18:29:31Z

@xvrl

Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 13:51:28+0000)
Java version: 1.8.0_40, vendor: Oracle Corporation

xvrl · 2015-08-19T18:34:27Z

@guobingkun can you try a more recent version of maven, e.g. 3.3.3?

guobingkun · 2015-08-19T20:52:52Z

@xvrl I tried Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 2015-03-13T20:10:27+00:00) but still gave the same error.

xvrl · 2015-08-19T23:52:36Z

@guobingkun it looks like you might be running into https://issues.apache.org/jira/browse/MASSEMBLY-677
Do you have a custom maven repository setup with mirrors? Does it work if you remove any repository customizations in your maven settings.xml, to have it default to maven central?

himanshug · 2015-08-20T01:31:40Z

@xvrl that would be a problem. we have corp level maven mirror that contains public maven + internal stuff. All the machines get a [internally certified]wrapped version of maven that uses those mirrors. It is not possible to change things inside settings.xml in most cases and this will continue to fail build for us.

xvrl · 2015-08-20T20:58:21Z

@himanshug @guobingkun I can try disabling embedding the maven metadata into the local maven repo and see if that still works. If not, I suggest we make the tarball assembly optional as part of a separate release profile that we can activate when needed to make the release. Would that work?

himanshug · 2015-08-20T21:06:33Z

@xvrl if it works by disabling the maven metadata, good or else making it optional sounds good to me as most people dont need to build that.

xvrl · 2015-08-20T22:40:28Z

@himanshug @guobingkun can you try with this fix #1650 ?

himanshug · 2015-08-21T03:17:17Z

#1650 fixed the issue, merged.

xvrl force-pushed the maven-repo-tarball-clean branch from a2057d8 to caa7558 Compare August 19, 2015 00:54

xvrl and others added 3 commits August 18, 2015 18:32

Use local repository instead of maven central:

601cd82

- Use local repo in run_example_server. - Add extensions-repo to common.runtime.properties for the tarball.

run_druid_server.sh

5ca440f

xvrl force-pushed the maven-repo-tarball-clean branch from caa7558 to 5ca440f Compare August 19, 2015 01:32

xvrl added the Ease of Use label Aug 19, 2015

xvrl added this to the 0.8.1 milestone Aug 19, 2015

fjy added a commit that referenced this pull request Aug 19, 2015

Merge pull request #1640 from druid-io/maven-repo-tarball-clean

d996e0a

Package extensions in tarball + run_druid_server goodness

fjy merged commit d996e0a into master Aug 19, 2015

fjy deleted the maven-repo-tarball-clean branch August 19, 2015 04:54

xvrl mentioned this pull request Aug 20, 2015

remove maven metadata from embedded repo #1650

Merged

xvrl mentioned this pull request Aug 21, 2015

Merge fixes for 0.8.1 rc2 #1657

Merged

seoeun25 pushed a commit to seoeun25/incubator-druid that referenced this pull request Jan 10, 2020

apache#1640 Add running average window function

24343dc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package extensions in tarball + run_druid_server goodness #1640

Package extensions in tarball + run_druid_server goodness #1640

xvrl commented Aug 19, 2015

gianm commented Aug 19, 2015

xvrl commented Aug 19, 2015

drcrallen commented Aug 19, 2015

xvrl commented Aug 19, 2015

drcrallen commented Aug 19, 2015

himanshug commented Aug 19, 2015

guobingkun commented Aug 19, 2015

drcrallen commented Aug 19, 2015

xvrl commented Aug 19, 2015

guobingkun commented Aug 19, 2015

xvrl commented Aug 19, 2015

guobingkun commented Aug 19, 2015

xvrl commented Aug 19, 2015

guobingkun commented Aug 19, 2015

xvrl commented Aug 19, 2015

guobingkun commented Aug 19, 2015

xvrl commented Aug 19, 2015

himanshug commented Aug 20, 2015

xvrl commented Aug 20, 2015

himanshug commented Aug 20, 2015

xvrl commented Aug 20, 2015

himanshug commented Aug 21, 2015

Package extensions in tarball + run_druid_server goodness #1640

Package extensions in tarball + run_druid_server goodness #1640

Conversation

xvrl commented Aug 19, 2015

gianm commented Aug 19, 2015

xvrl commented Aug 19, 2015

drcrallen commented Aug 19, 2015

xvrl commented Aug 19, 2015

drcrallen commented Aug 19, 2015

himanshug commented Aug 19, 2015

guobingkun commented Aug 19, 2015

drcrallen commented Aug 19, 2015

xvrl commented Aug 19, 2015

guobingkun commented Aug 19, 2015

xvrl commented Aug 19, 2015

guobingkun commented Aug 19, 2015

xvrl commented Aug 19, 2015

guobingkun commented Aug 19, 2015

xvrl commented Aug 19, 2015

guobingkun commented Aug 19, 2015

xvrl commented Aug 19, 2015

himanshug commented Aug 20, 2015

xvrl commented Aug 20, 2015

himanshug commented Aug 20, 2015

xvrl commented Aug 20, 2015

himanshug commented Aug 21, 2015