Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add encryption tests using AWS S3 java SDK #264

Merged
merged 1 commit into from
Apr 18, 2018

Conversation

vadmeste
Copy link
Member

@vadmeste vadmeste commented Mar 7, 2018

Fixes #262

@vadmeste vadmeste force-pushed the add_enc_tests branch 6 times, most recently from 0f35ec9 to f7264f9 Compare March 8, 2018 23:05
@vadmeste vadmeste changed the title [WIP] Add encryption tests using AWS S3 java SDK Add encryption tests using AWS S3 java SDK Mar 8, 2018
@vadmeste vadmeste force-pushed the add_enc_tests branch 2 times, most recently from 102c326 to f18352b Compare March 8, 2018 23:44
@vadmeste
Copy link
Member Author

vadmeste commented Mar 9, 2018

Ready for review & merge

@vadmeste vadmeste force-pushed the add_enc_tests branch 3 times, most recently from 9849a85 to 063c1a1 Compare March 9, 2018 18:58
@vadmeste
Copy link
Member Author

ping @nitisht & @balamurugana

@nitisht
Copy link
Contributor

nitisht commented Mar 16, 2018

@vadmeste I tried building this locally, but the mint build failed. Am I missing something?

Here is the build trace

Step 22/50 : RUN build/aws-sdk-java/install.sh
 ---> Running in 2a3eb89039fa
Buildfile: /mint/build/aws-sdk-java/build.xml

download-ivy:
    [mkdir] Created dir: /root/.ant/lib
      [get] Getting: https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0-rc2/ivy-2.1.0-rc2.jar
      [get] To: /root/.ant/lib/ivy.jar

init-ivy:

BUILD SUCCESSFUL
Total time: 5 seconds
Buildfile: /mint/build/aws-sdk-java/build.xml

resolve:
[ivy:retrieve] :: Ivy 2.1.0-rc2 - 20090704004254 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: url = jar:file:/root/.ant/lib/ivy.jar!/org/apache/ivy/core/settings/ivysettings.xml
[ivy:retrieve] :: resolving dependencies :: org.apache#hello-ivy;working@2a3eb89039fa
[ivy:retrieve] 	confs: [default]
[ivy:retrieve] 	found com.amazonaws#aws-java-sdk-s3;1.11.289 in public
[ivy:retrieve] 	found com.amazonaws#aws-java-sdk-kms;1.11.289 in public
[ivy:retrieve] 	found com.amazonaws#aws-java-sdk-core;1.11.289 in public
[ivy:retrieve] 	found software.amazon.ion#ion-java;1.0.2 in public
[ivy:retrieve] 	found com.fasterxml.jackson.core#jackson-databind;2.6.7.1 in public
[ivy:retrieve] 	found com.fasterxml.jackson.core#jackson-annotations;2.6.0 in public
[ivy:retrieve] 	found com.fasterxml.jackson.core#jackson-core;2.6.7 in public
[ivy:retrieve] 	found com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.6.7 in public
[ivy:retrieve] 	found joda-time#joda-time;2.8.1 in public
[ivy:retrieve] 	found com.amazonaws#jmespath-java;1.11.289 in public
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-s3/1.11.289/aws-java-sdk-s3-1.11.289-sources.jar ...
[ivy:retrieve] .........................................................................................................
[ivy:retrieve] ...............................................................................................................................
[ivy:retrieve] ..................................................................................... (814kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.amazonaws#aws-java-sdk-s3;1.11.289!aws-java-sdk-s3.jar(source) (5265ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-s3/1.11.289/aws-java-sdk-s3-1.11.289-javadoc.jar ...
[ivy:retrieve] ..............................................................................................................................
[ivy:retrieve] ....................................................................................................................................................................
[ivy:retrieve] ......................................................................................................................................................................................................................................................
[ivy:retrieve] ......................................................................................................................................................................................................................................................................................................................................................................................
[ivy:retrieve] ............................................................................................................................................................................................ (3121kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.amazonaws#aws-java-sdk-s3;1.11.289!aws-java-sdk-s3.jar(javadoc) (8029ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-s3/1.11.289/aws-java-sdk-s3-1.11.289.jar ...
[ivy:retrieve] ............................................................................................................................................................................... (817kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.amazonaws#aws-java-sdk-s3;1.11.289!aws-java-sdk-s3.jar (2195ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-kms/1.11.289/aws-java-sdk-kms-1.11.289.jar ...
[ivy:retrieve] ................................................................................................................ (394kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.amazonaws#aws-java-sdk-kms;1.11.289!aws-java-sdk-kms.jar (1667ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.289/aws-java-sdk-core-1.11.289.jar ...
[ivy:retrieve] ................................................................................................................................................................................................................................................................. (840kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.amazonaws#aws-java-sdk-core;1.11.289!aws-java-sdk-core.jar (2002ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/amazonaws/jmespath-java/1.11.289/jmespath-java-1.11.289.jar ...
[ivy:retrieve] ....... (26kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.amazonaws#jmespath-java;1.11.289!jmespath-java.jar (1200ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar ...
[ivy:retrieve] ................... (552kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] software.amazon.ion#ion-java;1.0.2!ion-java.jar(bundle) (2673ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar ...
[ivy:retrieve] .....................................................................................................................................................................................................................
[ivy:retrieve] ................................................................................................................. (1138kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.fasterxml.jackson.core#jackson-databind;2.6.7.1!jackson-databind.jar (3069ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.jar ...
[ivy:retrieve] ...... (47kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.6.7!jackson-dataformat-cbor.jar (1662ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar ...
[ivy:retrieve] .................................................................................................................. (607kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] joda-time#joda-time;2.8.1!joda-time.jar (2005ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar ...
[ivy:retrieve] ................ (45kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.fasterxml.jackson.core#jackson-annotations;2.6.0!jackson-annotations.jar(bundle) (1923ms)
[ivy:retrieve] downloading http://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar ...
[ivy:retrieve] ............................................................. (252kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] 	[SUCCESSFUL ] com.fasterxml.jackson.core#jackson-core;2.6.7!jackson-core.jar(bundle) (1710ms)
[ivy:retrieve] :: resolution report :: resolve 45227ms :: artifacts dl 33448ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   12  |   10  |   10  |   0   ||   12  |   12  |
	---------------------------------------------------------------------
[ivy:retrieve] 
[ivy:retrieve] :: problems summary ::
[ivy:retrieve] :::: WARNINGS
[ivy:retrieve] 	io problem while parsing ivy file: http://repo1.maven.org/maven2/org/apache/commons/commons-parent/28/commons-parent-28.pom: Impossible to load parent for file:/root/.ant/cache/org.apache.commons/commons-parent/ivy-28.xml.original. Parent=org.apache#apache;13
[ivy:retrieve] 	io problem while parsing ivy file: http://repo1.maven.org/maven2/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.pom: Impossible to load parent for file:/root/.ant/cache/commons-logging/commons-logging/ivy-1.1.3.xml.original. Parent=org.apache.commons#commons-parent;28
[ivy:retrieve] 		module not found: commons-logging#commons-logging;1.1.3
[ivy:retrieve] 	==== local: tried
[ivy:retrieve] 	  /root/.ant/local/org.apache/apache/13/ivys/ivy.xml
[ivy:retrieve] 	  -- artifact org.apache#apache;13!apache.jar:
[ivy:retrieve] 	  /root/.ant/local/org.apache/apache/13/jars/apache.jar
[ivy:retrieve] 	==== shared: tried
[ivy:retrieve] 	  /root/.ant/shared/org.apache/apache/13/ivys/ivy.xml
[ivy:retrieve] 	  -- artifact org.apache#apache;13!apache.jar:
[ivy:retrieve] 	  /root/.ant/shared/org.apache/apache/13/jars/apache.jar
[ivy:retrieve] 	==== public: tried
[ivy:retrieve] 	  http://repo1.maven.org/maven2/org/apache/apache/13/apache-13.pom
[ivy:retrieve] 	  -- artifact org.apache#apache;13!apache.jar:
[ivy:retrieve] 	  http://repo1.maven.org/maven2/org/apache/apache/13/apache-13.jar
[ivy:retrieve] 	io problem while parsing ivy file: http://repo1.maven.org/maven2/org/apache/httpcomponents/project/7/project-7.pom: Impossible to load parent for file:/root/.ant/cache/org.apache.httpcomponents/project/ivy-7.xml.original. Parent=org.apache#apache;13
[ivy:retrieve] 	io problem while parsing ivy file: http://repo1.maven.org/maven2/org/apache/httpcomponents/httpcomponents-client/4.5.2/httpcomponents-client-4.5.2.pom: Impossible to load parent for file:/root/.ant/cache/org.apache.httpcomponents/httpcomponents-client/ivy-4.5.2.xml.original. Parent=org.apache.httpcomponents#project;7
[ivy:retrieve] 	io problem while parsing ivy file: http://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.pom: Impossible to load parent for file:/root/.ant/cache/org.apache.httpcomponents/httpclient/ivy-4.5.2.xml.original. Parent=org.apache.httpcomponents#httpcomponents-client;4.5.2
[ivy:retrieve] 		module not found: org.apache.httpcomponents#httpclient;4.5.2
[ivy:retrieve] 	==== local: tried
[ivy:retrieve] 	  /root/.ant/local/org.apache/apache/13/ivys/ivy.xml
[ivy:retrieve] 	  -- artifact org.apache#apache;13!apache.jar:
[ivy:retrieve] 	  /root/.ant/local/org.apache/apache/13/jars/apache.jar
[ivy:retrieve] 	==== shared: tried
[ivy:retrieve] 	  /root/.ant/shared/org.apache/apache/13/ivys/ivy.xml
[ivy:retrieve] 	  -- artifact org.apache#apache;13!apache.jar:
[ivy:retrieve] 	  /root/.ant/shared/org.apache/apache/13/jars/apache.jar
[ivy:retrieve] 	==== public: tried
[ivy:retrieve] 	  http://repo1.maven.org/maven2/org/apache/apache/13/apache-13.pom
[ivy:retrieve] 	  -- artifact org.apache#apache;13!apache.jar:
[ivy:retrieve] 	  http://repo1.maven.org/maven2/org/apache/apache/13/apache-13.jar
[ivy:retrieve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] 		::          UNRESOLVED DEPENDENCIES         ::
[ivy:retrieve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] 		:: commons-logging#commons-logging;1.1.3: not found
[ivy:retrieve] 		:: org.apache.httpcomponents#httpclient;4.5.2: not found
[ivy:retrieve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] 
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

BUILD FAILED
/mint/build/aws-sdk-java/build.xml:25: impossible to resolve dependencies:
	resolve failed - see output for details

Total time: 1 minute 19 seconds
cp: cannot stat 'build/jar/FunctionalTests.jar': No such file or directory
The command '/bin/sh -c build/aws-sdk-java/install.sh' returned a non-zero code: 1

@vadmeste
Copy link
Member Author

@nitisht this is mostly a one time error or network problem.. can you try one time again ?

@nitisht
Copy link
Contributor

nitisht commented Mar 16, 2018

@nitisht this is mostly a one time error or network problem.. can you try one time again ?

You're right, looks like building now

@nitisht
Copy link
Contributor

nitisht commented Mar 16, 2018

tested locally and works fine @vadmeste . Can you update the Dockerfile / Readme.md as well?

@vadmeste
Copy link
Member Author

@nitisht oh yes, Readme needs to be updated but Dockerfile.dev is already updated, do we need to do something with Dockerfile ?

@nitisht
Copy link
Contributor

nitisht commented Mar 17, 2018

do we need to do something with Dockerfile ?

Dockerfile automatically loops through all the directories, so nothing to be done there.

Copy link
Contributor

@nitisht nitisht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM & Tested locally

@vadmeste
Copy link
Member Author

vadmeste commented Mar 19, 2018

ping @balamurugana, please take a look

@@ -0,0 +1,58 @@
<project xmlns:ivy="antlib:org.apache.ivy.ant" name="hello-ivy" default="run">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to use gradle here? gradle is simple and clean.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to use gradle here? gradle is simple and clean.

I liked ivy over gradle because gradle has some internal working that we don't understand. ivy here downloads dependencies in the current folder (gradle is doing it somewhere else) and then it becomes simple to generate a jar file and clean all the stuff and have a minimal docker image.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think gradle does a decent job here. gradle build scripts are simple to read. Similarly we already depends on gradle for minio-java and would reduce one more build dependency in mint.

gradle generally downloads dependencies under ~/.gradle and single jar file can be obtained after successful build from build directory like minio-java-<VERSION>-all.jar.

However the choice is yours.

import java.io.*;

// RandAccessFile wraps RandomAccessFile and overrides
// close() to make it non-operational.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you tell me why close() needs to be non-operational?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you tell me why close() needs to be non-operational?

I added this because AWS S3 java sdk was closing input stream after uploading a file.. but it looks like I can avoid all this and make the code simpler to understand. Will update.



// InputStreamNopCloser wraps the regular input stream but it
// cancels close() call to make it no operational.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you tell me why we need a stream with close() as no-op?

}

// Test download object with Get Range, 64*1024 -> 64*1024
public static void test_downloadGetRange_5() throws Exception {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think downloadGetRange_test5() would be clean.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think downloadGetRange_test5() would be clean.

You are right

@vadmeste vadmeste force-pushed the add_enc_tests branch 4 times, most recently from b1a7e0e to 805f916 Compare April 6, 2018 17:28
@vadmeste
Copy link
Member Author

@balamurugana I removed RandAccessFile & InputStreamNopCloser classes so it is easier to understand. Please review again.

@nitisht nitisht merged commit e55166a into minio:master Apr 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants