Skip to content

CDAP-15492 Salesforce streaming integration tests fail, jar includes unnecessary Spark#38

Merged
albertshau merged 1 commit intodata-integrations:developfrom
aonischuk:feature/CDAP-15492-salesforce-integration-fix-and-jar-size-fix
Jun 10, 2019
Merged

CDAP-15492 Salesforce streaming integration tests fail, jar includes unnecessary Spark#38
albertshau merged 1 commit intodata-integrations:developfrom
aonischuk:feature/CDAP-15492-salesforce-integration-fix-and-jar-size-fix

Conversation

@aonischuk
Copy link
Copy Markdown
Contributor

@aonischuk aonischuk commented Jun 7, 2019

  1. Streaming integration tests fail with:
testValuesReturned(io.cdap.plugin.salesforce.etl.SalesforceStreamingSourceETLTest)  Time elapsed: 5.255 sec  <<< ERROR!
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError
     at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
     at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
     at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)
     at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
     at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
     at org.apache.spark.streaming.StreamingContext.withNamedScope(StreamingContext.scala:273)
     at org.apache.spark.streaming.StreamingContext.receiverStream(StreamingContext.scala:282)
     at org.apache.spark.streaming.api.java.JavaStreamingContext.receiverStream(JavaStreamingContext.scala:428)
     at io.cdap.plugin.salesforce.plugin.source.streaming.SalesforceStreamingSource.getStream(SalesforceStreamingSource.java:115)
     at io.cdap.cdap.etl.spark.plugin.WrappedStreamingSource$2.call(WrappedStreamingSource.java:59)
     ...

This happens because com.fasterxml.jackson.core/jackson-databind 2.6.7 is used by org.apache.spark/spark-core_2.11. While with the patch we use 2.9.9.
So I guess when we classload all the classes into a single scope wrong jackson-databind gets loaded, causing spark to fail.
2. Salesforce jar has size of 32Mb currently. Without spark core which is provided by CDAP, therefore unnecessary, it has size of 9Mb.
3. Fix a wrong default value for SFDC widget causing "unknown value" problems.

@googlebot
Copy link
Copy Markdown

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@aonischuk aonischuk force-pushed the feature/CDAP-15492-salesforce-integration-fix-and-jar-size-fix branch 2 times, most recently from 2cbad73 to 8ac9e2e Compare June 8, 2019 07:38
@aonischuk
Copy link
Copy Markdown
Contributor Author

@albertshau do you think we should keep the jira open after this is merged? I cannot see any viable solution for this though.

@aonischuk aonischuk force-pushed the feature/CDAP-15492-salesforce-integration-fix-and-jar-size-fix branch from 8ac9e2e to 529a374 Compare June 9, 2019 08:53
Copy link
Copy Markdown
Contributor

@albertshau albertshau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Yes let's keep the jira open so we don't lose it.

Comment thread pom.xml Outdated
<commons.csv.version>1.6</commons.csv.version>
<jackson.version>1.9.13</jackson.version>
<jackson2.version>2.9.9</jackson2.version>
<jackson2.version>2.6.7</jackson2.version>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't think we can package this version. Can you try updating spark to a version that uses a newer jackson? If that doesn't work, since you mentioned this only breaks the test and not the actual plugin, let's add a note about overriding this when running the test and keep an jira open for fixing this.

@albertshau albertshau merged commit 4cb9132 into data-integrations:develop Jun 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants