[PIO-61] Add S3 Model Data Repository #371
Conversation
| assemblyShadeRules in assembly := Seq( | ||
| ShadeRule.rename("org.apache.http.**" -> "shadeio.data.s3.http.@1").inAll, | ||
| ShadeRule.rename("com.fasterxml.**" -> "shadeio.data.s3.fasterxml.@1").inAll | ||
| ) |
There was a problem hiding this comment.
Hey @marevol can you explain a little bit regarding these shade rules, please?
There was a problem hiding this comment.
aws-java-sdk-s3 depends on them, but they conflict with different version files in spark-assembly jar.
To avoid this conflict, the package replacement is needed in storage/s3.
There was a problem hiding this comment.
I think @mars is seeing some issues with https://github.com/apache/incubator-predictionio/blob/develop/storage/elasticsearch/build.sbt#L42.
There was a problem hiding this comment.
@mars could you provide info?
ES storage also conflicts with httpclient jar file of Spark.
I think an exception occurs if removing it, so it will be needed.
There was a problem hiding this comment.
Hi @marevol,
We're building a predictionio-incubating branch to add authentication to the Elasticsearch REST client with Scala 2.11 & Spark 2.1.
I believe it was during a runtime ES REST client call to performRequest suddenly an underlying method signature could not be located in org.apache.http package. @dszeto helped me to discover that the shadeio.data name was being incorrectly resolved. So, I tried removing the shade rule and it started working. No more errors for that branch.
Maybe the difference is that we're using Spark 2.1?
There was a problem hiding this comment.
Thank you for the info.
My environment is Scala 2.10 and Spark 1.6.3.
I'll check it on Spark 2.1.
There was a problem hiding this comment.
Checking on Scala 2.11/Spark 2.1/ES 5.1&5.3, I could not reproduce it yet...
My test script file is test_all.sh to build pio and run recommender template.
Is it reproduced in Docker tests? I'd like to know steps to reproduce it...
|
@marevol Would you like to add some integration test for this? https://github.com/jubos/fake-s3 or https://github.com/atlassian/localstack might be useful for mocking S3 locally within Travis. |
|
Added LocalStack for S3 testing. |
|
I would like to merge S3 support this week. |
|
Sorry for the delay. I was on vacation. This looks great to me, especially the use of |
No description provided.