Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix distribution to not include contrib extensions by default, don't … #6494

Merged
merged 1 commit into from
Oct 19, 2018
Merged

fix distribution to not include contrib extensions by default, don't … #6494

merged 1 commit into from
Oct 19, 2018

Conversation

dclim
Copy link
Contributor

@dclim dclim commented Oct 19, 2018

…pull the entire AWS SDK bundle

When trying to push the 0.13 RC1 binary to the Apache repository, I was getting rejected, likely due to the file being too large (it was 385MB). While investigating the bloat, I found two major factors:

I fixed the first one by splitting off a separate bundle-contrib-exts profile as previous which can be used to add the contrib extensions; otherwise only the core extensions are bundled.

For the second one, I removed the dependency on aws-java-sdk-bundle and only pulled aws-java-sdk-ec2 and aws-java-sdk-s3. I am not sure if the full bundle was pulled in for any particular reason or just as a convenience.

I also excluded a few larger dependencies from extensions that were already included in the core.

Binary tarball reduced in size from 385MB to 210MB.

FYI, I am still waiting on a response to a JIRA ticket to Apache Legal regarding how the MySQL connector should be packaged: https://issues.apache.org/jira/browse/LEGAL-423. If I don't hear back soon, I'll try to push RC1 (for the third time!) and do an RC2 if they express concerns with our current packaging system. My hunch is that we will be required to separate the offending library (GPL-licensed Java MySQL connector library) from the rest of the MySQL extension.. so if others feel good about it, I wouldn't be opposed to going ahead and making the change.

@dclim dclim added this to the 0.13.0 milestone Oct 19, 2018
@b-slim
Copy link
Contributor

b-slim commented Oct 19, 2018

@dclim thanks for doing this! I think removing the Dependency on Mysql is the way to go. Hive does that for Mysql, in fact we do not use it that much in the code.

extensions-core/mysql-metadata-storage/src/main/java/org/apache/druid/metadata/storage/mysql/MySQLConnector.java:import com.mysql.jdbc.exceptions.MySQLTransientException;

Same with LZO libs for hadoop/spark.

Copy link
Contributor

@gianm gianm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I don't see any discussion in #6162 or #5382 indicating that the changes you mentioned in those two PRs were intentional, and I am in favor of reverting them.

The other changes you made look good too.

@gianm gianm merged commit e1a53fd into apache:master Oct 19, 2018
dclim added a commit that referenced this pull request Oct 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants