Add Rackspace Cloud Files Deep Storage Extension #1719

se7entyse7en · 2015-09-10T14:59:35Z

This is the first version to support Rackspace's Cloud Files as druid deep storage. It still lacks of tests. As already discussed here https://groups.google.com/forum/#!topic/druid-development/9O5NMo2-4WU I would also like some suggestions about the pom.xml as I'm having some difficulties in order to build druid along with the cloud-files extension. Here's my current working solution (from the root of the project):

mvn install -DskipTests=true -Dmaven.javadoc.skip=true
cp -f services/target/druid-services-0.8.2-SNAPSHOT-selfcontained.jar /usr/local/druid/lib
java "-Ddruid.extensions.coordinates=[\"io.druid.extensions:druid-kafka-eight:0.8.2-SNAPSHOT\",\"io.druid.extensions:mysql-metadata-storage:0.8.2-SNAPSHOT\",\"io.druid.extensions:druid-cloudfiles-extensions:0.8.2-SNAPSHOT\"]" -Ddruid.extensions.localRepository=/usr/local/druid/repository "-Ddruid.extensions.remoteRepositories=[\"file:///root/.m2/repository/\",\"http://repo1.maven.org/maven2/\",\"https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local\"]" -cp /usr/local/druid/lib/* io.druid.cli.Main tools pull-deps
cd extensions/cloudfiles-extensions/
mvn dependency:copy-dependencies "-DoutputDirectory=/usr/local/druid/jclouds"
cd ../..

and then I run the various nodes:

java -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -cp config/_common:config/coordinator:/usr/local/druid/lib/*:/usr/local/druid/jclouds/* io.druid.cli.Main server coordinator
java -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -cp config/_common:config/historical:/usr/local/druid/lib/*:/usr/local/druid/jclouds/* io.druid.cli.Main server historical
java -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -cp config/_common:config/broker:/usr/local/druid/lib/*:/usr/local/druid/jclouds/* io.druid.cli.Main server broker
java -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=examples/kafka/kafka_realtime.spec -cp config/_common:config/realtime:/usr/local/druid/lib/*:/usr/local/druid/jclouds/* io.druid.cli.Main server realtime

drcrallen · 2015-09-10T16:28:09Z

extensions/cloudfiles-extensions/pom.xml

+	<parent>
+		<groupId>io.druid</groupId>
+		<artifactId>druid</artifactId>
+		<version>0.8.0</version>


This will need to be updated as the PR moves along. Current master is 0.8.2-SNAPSHOT

I missed it while cleaning the branch as I started this feature from 0.8.0, I'm fixing.

drcrallen · 2015-09-10T17:08:46Z

@se7entyse7en change the parent of your extension to match the master pom. Then you should be able to use it like any other extension coordinate

drcrallen · 2015-09-10T17:23:22Z

...-extensions/src/main/java/io/druid/storage/cloudfiles/CloudFilesDataSegmentPusherConfig.java

+
+/**
+ */
+public class CloudFilesDataSegmentPusherConfig {


NotNull annotation but no enforcement

What do you mean? Can you elaborate please? Do you perhaps mean that they're NotNull, but they don't have a default argument such as empty string?

you still need Preconditions.checkNotNull in the getter (or in a constructor for eager failures) Jackson doesn't really enforce NotNull or @JsonProperty(required = true).

Ok, I added Preconditions.checkNotNull in the getters.

se7entyse7en · 2015-09-11T10:07:16Z

Are the tests blocking for accepting the merge?

fjy · 2015-09-11T10:10:18Z

@se7entyse7en we need 2 👍s before we can merge

drcrallen · 2015-09-11T19:09:53Z

...loudfiles-extensions/src/main/java/io/druid/storage/cloudfiles/CloudFilesObjectApiProxy.java

+
+	public CloudFilesObject get(String path) {
+		SwiftObject swiftObject = objectApi.get(path);
+		Payload payload = swiftObject.getPayload();


payload is closeable but never closed

Ok, I added a closeStream method to CloudFilesByteSource and I added the closing in getSegmentFiles in CloudFilesDataSegmentPuller.

Can you make sure there is a unit test that ensures payload is closed?

drcrallen · 2015-09-11T19:31:20Z

Overall the PR is in a pretty good spot. A few things need addressed still

Lots of newlines at end of file are missing
pom.xml for extension needs trimmed to only include stuff that is required rather than taking the example pom.xml directly
How users are supposed to handle the different jclouds providers : https://jclouds.apache.org/reference/providers/ Having docs on which ones are supported and which ones are not and how to enable/use them should be mentioned either in docs or at a minimal in the pom.xml.
Docs on configuration options in docs/content/dependencies/deep-storage.md

Eventually the PR will need squashed but not requried until reviews are done

drcrallen · 2015-09-11T20:41:38Z

extensions/cloudfiles-extensions/pom.xml

+			<version>1.0.13</version>
+		</dependency>
+		<dependency>
+			<groupId>mysql</groupId>


Is this required?

fjy · 2015-11-04T00:33:52Z

👍

se7entyse7en · 2015-11-04T10:30:18Z

I added a comment to the pom file as requested by @xvrl.

se7entyse7en · 2015-11-04T15:33:17Z

Closed and reopened to restart build.

drcrallen · 2015-11-04T16:09:44Z

@se7entyse7en se7entyse7en#1

drcrallen · 2015-11-04T16:10:29Z

Please squash once everything looks good on your end

se7entyse7en · 2015-11-04T16:12:32Z

@drcrallen ok, squashed into a single commit.

drcrallen · 2015-11-04T16:34:32Z

@se7entyse7en please include se7entyse7en#1 in the squash if possible.

drcrallen · 2015-11-04T16:36:02Z

failure in JDBCExtractionNamespaceTest.testFindNew:261->assertUpdated:312 update check expected:<ba[z]> but was:<ba[r]> not a concern

se7entyse7en · 2015-11-04T16:39:59Z

@drcrallen included and squashed

…s as deep storage

drcrallen · 2015-11-04T16:48:48Z

Cool thanks for your patience in the matter!

👍 after travis

drcrallen · 2015-11-04T17:07:15Z

AnnouncerTest.testSessionKilled:168 » NoNode KeeperErrorCode = NoNode for /tes...

drcrallen · 2015-11-04T17:07:25Z

Bouncing for travis

…les-deep-storage Add Rackspace Cloud Files Deep Storage Extension

se7entyse7en · 2015-11-04T20:06:55Z

Thanks to everyone for the help.

drcrallen reviewed Sep 10, 2015
View reviewed changes

drcrallen added Feature Discuss labels Sep 10, 2015

drcrallen changed the title ~~[WIP] Feature Rackspace Cloud Files Deep Storage~~ Add Rackspace Cloud Files Deep Storage Extension Sep 10, 2015

drcrallen reviewed Sep 10, 2015
View reviewed changes

se7entyse7en force-pushed the feature-rackspace-cloud-files-deep-storage branch 2 times, most recently from d27a4d1 to 5e0424b Compare September 11, 2015 13:26

drcrallen reviewed Sep 11, 2015
View reviewed changes

se7entyse7en force-pushed the feature-rackspace-cloud-files-deep-storage branch from e9549f5 to d3fcd36 Compare September 14, 2015 08:58

se7entyse7en force-pushed the feature-rackspace-cloud-files-deep-storage branch 2 times, most recently from bfe0f54 to ac823e4 Compare November 3, 2015 15:50

se7entyse7en force-pushed the feature-rackspace-cloud-files-deep-storage branch from ac823e4 to 4172b58 Compare November 4, 2015 10:29

se7entyse7en force-pushed the feature-rackspace-cloud-files-deep-storage branch from 4172b58 to ea72ec1 Compare November 4, 2015 11:17

se7entyse7en closed this Nov 4, 2015

se7entyse7en reopened this Nov 4, 2015

drcrallen removed the Discuss label Nov 4, 2015

se7entyse7en force-pushed the feature-rackspace-cloud-files-deep-storage branch from ea72ec1 to 24765cf Compare November 4, 2015 16:12

se7entyse7en force-pushed the feature-rackspace-cloud-files-deep-storage branch from dd5fa05 to c924f9f Compare November 4, 2015 16:39

Added cloudfiles-extensions in order to support Rackspace's cloudfile…

c924f9f

…s as deep storage

drcrallen closed this Nov 4, 2015

drcrallen reopened this Nov 4, 2015

drcrallen closed this Nov 4, 2015

drcrallen reopened this Nov 4, 2015

fjy added a commit that referenced this pull request Nov 4, 2015

Merge pull request #1719 from se7entyse7en/feature-rackspace-cloud-fi…

1cbc514

…les-deep-storage Add Rackspace Cloud Files Deep Storage Extension

fjy merged commit 1cbc514 into apache:master Nov 4, 2015

fjy mentioned this pull request Feb 5, 2016

druid-0.9.0 release notes #2404

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Rackspace Cloud Files Deep Storage Extension #1719

Add Rackspace Cloud Files Deep Storage Extension #1719

se7entyse7en commented Sep 10, 2015

drcrallen Sep 10, 2015

se7entyse7en Sep 11, 2015

drcrallen commented Sep 10, 2015

drcrallen Sep 10, 2015

se7entyse7en Sep 11, 2015

drcrallen Sep 11, 2015

se7entyse7en Sep 14, 2015

se7entyse7en commented Sep 11, 2015

fjy commented Sep 11, 2015

drcrallen Sep 11, 2015

se7entyse7en Sep 14, 2015

drcrallen Sep 15, 2015

drcrallen commented Sep 11, 2015

drcrallen Sep 11, 2015

fjy commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

drcrallen commented Nov 4, 2015

drcrallen commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

drcrallen commented Nov 4, 2015

drcrallen commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

drcrallen commented Nov 4, 2015

drcrallen commented Nov 4, 2015

drcrallen commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

Add Rackspace Cloud Files Deep Storage Extension #1719

Add Rackspace Cloud Files Deep Storage Extension #1719

Conversation

se7entyse7en commented Sep 10, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drcrallen commented Sep 10, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

se7entyse7en commented Sep 11, 2015

fjy commented Sep 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drcrallen commented Sep 11, 2015

Choose a reason for hiding this comment

fjy commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

drcrallen commented Nov 4, 2015

drcrallen commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

drcrallen commented Nov 4, 2015

drcrallen commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015

drcrallen commented Nov 4, 2015

drcrallen commented Nov 4, 2015

drcrallen commented Nov 4, 2015

se7entyse7en commented Nov 4, 2015