/mavenclasspath endpoint takes too long

[dsimansk]

When trying to do Twitter Mention -> Salesforce Create/Update Contact, I can see the following exception.

Could not load JAVA document. Document is not found: io.syndesis.connector.salesforce.Contact

[gashcrumb]

Underlying error response back is

{
  "timestamp": 1500314207172,
  "status": 500,
  "error": "Internal Server Error",
  "exception": "org.jboss.resteasy.spi.UnhandledException",
  "message": "java.lang.IllegalStateException: Invalid atlas uri io.syndesis.connector.salesforce.Contact does not begin with 'atlas:'",
  "path": "\/v2\/atlas\/mapping\/validate"
}

at least on staging. When running a local dev instance I get a 200 response but with a class not found:

{
  "ClassInspectionResponse" : {
    "jsonType" : "io.atlasmap.java.v2.ClassInspectionResponse",
    "javaClass" : {
      "jsonType" : "io.atlasmap.java.v2.JavaClass",
      "status" : "NOT_FOUND",
      "className" : "io.syndesis.connector.salesforce.Contact",
      "javaEnumFields" : {
        "javaEnumField" : [ ]
      },
      "javaFields" : {
        "javaField" : [ ]
      }
    },
    "executionTime" : 1
  }
}

[mattrpav]

We used to include these "Demo Dependencies" in the services build. Since the demo, we are starting to pull those back. For this to work, the iPaaS needs to provide a full classpath to use during inspection.

Does the original inspection request include a valid classpath entry that contains a jar file with the io.syndesis.connector.salesforce.Contact class?

Please paste the ClassInspectionRequest as well so I can help validate any other changes.

Thanks

[dsimansk]

@mattrpav classpath field is empty. I guess you pretty much identified the problem :)

{
  "ClassInspectionRequest": {
    "jsonType": "io.atlasmap.java.v2.ClassInspectionRequest",
    "classpath": "",
    "className": "io.syndesis.connector.salesforce.Contact",
    "disablePrivateOnlyFields": false,
    "disableProtectedOnlyFields": false,
    "disablePublicOnlyFields": false,
    "disablePublicGetterSetterFields": false
  }
}

[mattrpav]

@dsimansk the call before the ClassInspectionRequest should be the MavenClasspathRequest. Syndesis should pass in a pom (as a string) to the atlasmap-ui module and it will query the service to generate a classpath value at runtime

From: /atlas-java-parent/atlas-java-model/src/main/resources/atlas-java-model-v2.xsd

<complexType name="MavenClasspathRequest">
	<sequence>
		<element name="pomXmlData" type="string" minOccurs="0" maxOccurs="1" />
	</sequence>
	<attribute name="executeTimeout" type="long" use="optional" />
	<attribute name="workingDirectory" type="string" use="optional" />
</complexType>

In json format:

{ "MavenClasspathRequest": { "jsonType": "io.atlasmap.java.v2.MavenClasspathRequest", "pomXmlData": "<project xmlns=\"http://maven.apache.org/POM/4.0.0\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd\"><modelVersion>4.0.0</modelVersion><groupId>foo.bar</groupId><artifactId>test.model</artifactId><version>1.10.0</version><packaging>jar</packaging><name>Test :: Model</name><dependencies><dependency><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-annotations</artifactId><version>2.8.5</version></dependency><dependency><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-databind</artifactId><version>2.8.5</version></dependency><dependency><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-core</artifactId><version>2.8.5</version></dependency></dependencies></project>", "executeTimeout": 30000 } }

[gashcrumb]

Yep for me it's sending:

{
  "MavenClasspathRequest": {
    "jsonType": "io.atlasmap.java.v2.MavenClasspathRequest",
    "pomXmlData": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<project xmlns=\"http:\/\/maven.apache.org\/POM\/4.0.0\" xmlns:xsi=\"http:\/\/www.w3.org\/2001\/XMLSchema-instance\"\n         xsi:schemaLocation=\"http:\/\/maven.apache.org\/POM\/4.0.0 http:\/\/maven.apache.org\/maven-v4_0_0.xsd\">\n\n  <modelVersion>4.0.0<\/modelVersion>\n\n  <groupId>io.syndesis.integrations<\/groupId>\n  <artifactId>project<\/artifactId>\n  <version>0.1-SNAPSHOT<\/version>\n  <name>Syndesis Integrations :: Twitter to Salesforce Example<\/name>\n  <description>This is an example of a Twitter to Salesforce integration<\/description>\n  <packaging>jar<\/packaging>\n\n  <properties>\n    <spring-boot.version>1.5.2.RELEASE<\/spring-boot.version>\n    <camel.version>2.19.0<\/camel.version>\n    <funktion.version>1.1.55<\/funktion.version>\n    <atlasmap.version>1.10.0<\/atlasmap.version>\n  <\/properties>\n\n  <dependencyManagement>\n    <dependencies>\n      <dependency>\n        <groupId>org.springframework.boot<\/groupId>\n        <artifactId>spring-boot-dependencies<\/artifactId>\n        <version>1.5.2.RELEASE<\/version>\n        <type>pom<\/type>\n        <scope>import<\/scope>\n      <\/dependency>\n      <dependency>\n        <groupId>org.apache.camel<\/groupId>\n        <artifactId>camel-spring-boot-dependencies<\/artifactId>\n        <version>2.19.0<\/version>\n        <type>pom<\/type>\n        <scope>import<\/scope>\n      <\/dependency>\n    <\/dependencies>\n  <\/dependencyManagement>\n\n  <dependencies>\n    <!-- atlasmap runtime -->\n    <dependency>\n      <groupId>io.atlasmap<\/groupId>\n      <artifactId>camel-atlasmap<\/artifactId>\n      <version>1.10.0<\/version>\n    <\/dependency>\n\n    <!-- connectors used in this integration -->\n    <dependency>\n      <groupId>io.syndesis<\/groupId>\n      <artifactId>twitter-mention-connector<\/artifactId>\n      <version>0.4.4<\/version>\n    <\/dependency>\n    <dependency>\n      <groupId>io.syndesis<\/groupId>\n      <artifactId>salesforce-upsert-contact-connector<\/artifactId>\n      <version>0.4.4<\/version>\n    <\/dependency>\n\n    <!-- spring-boot as runtime -->\n    <dependency>\n      <groupId>org.apache.camel<\/groupId>\n      <artifactId>camel-spring-boot-starter<\/artifactId>\n    <\/dependency>\n    <dependency>\n      <groupId>org.springframework.boot<\/groupId>\n      <artifactId>spring-boot-starter-web<\/artifactId>\n    <\/dependency>\n    <dependency>\n      <groupId>org.springframework.boot<\/groupId>\n      <artifactId>spring-boot-starter-undertow<\/artifactId>\n    <\/dependency>\n    <dependency>\n      <groupId>org.springframework.boot<\/groupId>\n      <artifactId>spring-boot-starter-actuator<\/artifactId>\n    <\/dependency>\n\n    <!-- Funktion runtime -->\n    <dependency>\n      <groupId>io.fabric8.funktion<\/groupId>\n      <artifactId>funktion-model<\/artifactId>\n      <version>1.1.55<\/version>\n    <\/dependency>\n    <dependency>\n      <groupId>io.fabric8.funktion<\/groupId>\n      <artifactId>funktion-runtime<\/artifactId>\n      <version>1.1.55<\/version>\n    <\/dependency>\n\n    <!-- testing -->\n    <dependency>\n      <groupId>org.apache.camel<\/groupId>\n      <artifactId>camel-test<\/artifactId>\n      <version>2.19.0<\/version>\n      <scope>test<\/scope>\n    <\/dependency>\n\n  <\/dependencies>\n\n  <build>\n\n    <plugins>\n      <plugin>\n        <groupId>org.springframework.boot<\/groupId>\n        <artifactId>spring-boot-maven-plugin<\/artifactId>\n        <version>1.5.2.RELEASE<\/version>\n        <executions>\n          <execution>\n            <goals>\n              <goal>repackage<\/goal>\n            <\/goals>\n          <\/execution>\n        <\/executions>\n      <\/plugin>\n    <\/plugins>\n  <\/build>\n\n<\/project>\n",
    "executeTimeout": 3600000
  }
}

And getting back the same response as David. Will parse that pom in a few there's possibly a problem with it.

[mattrpav]

@gashcrumb are you able to capture the MavenClassPathResponse payload?

[gashcrumb]

Yep, here's the exact response:

{
  "MavenClasspathResponse" : {
    "jsonType" : "io.atlasmap.java.v2.MavenClasspathResponse",
    "classpath" : "",
    "executionTime" : 0
  }
}

[mattrpav]

executionTime: 0 is bad =) Check into the output from the service.. there must be something sideways w/ the env.

[rhuss]

btw in the summit demo there was some sort of 'hack' to always return an empty classpath: https://github.com/rhuss/syndesis-datamapper/blob/663ea8498fc615db7a4f7398fabd73a782e9a6e0/runtime/src/main/java/com/mediadriver/atlas/runtime/JavaServiceComponent.java#L44

Don't know whether this has been taken over to atlasmap ?

[rhuss]

What's about this --> https://github.com/atlasmap/atlasmap/blob/master/runtime/src/main/java/io/atlasmap/runtime/JavaServiceComponent.java#L45 ?

[mattrpav]

@rhuss that's it. The 'hack' needs to be rolled back.

[rhuss]

@chirino what was the reason to hide the "/mavenclasspath" endpoint ? Can we simply remove the JavaServiceComponent class ?

[rhuss]

@mattrpav when I look at the class request, when the class is not found, the response still returns with a 200 HTTP code instead of a 404. I think this is wrong.

[rhuss]

I have enabled it again, however the mavenclasspath call takes far too long, and as it goes over the OpenShift router from the outside we get a Gateway Timeout after 30s:

This raises the principal question whether it should be the duty of the UI to first request a pom.xml from syndesis-rest just to send it to atlasmap. Not only feels it architectural completely wrong that syndesis-ui has to deal with implementation details like a pom.xml (which it doesn't care of), it results in timing problems like this and a tight coupling to implementation details.

In fact, it should be syndesis-rest (the backend) which calls the atlasmap service and return to syndesis-ui only the information that the atlasmap-ui component needs to initialize its UI. This means also, that atlasmap-ui must not call to any backend but get the required information during initialization.

@mattrpav would this be something easy to change for the angular component?

With this in place we can do also our own caching to e.g. pre populate the meta data for the connectors we already know.

[chirino]

We had a call and discussed generating the introspections at product build time. the datamapper ui would need to be updated to support syndesis passing the pre-computed introspections via the component interface.

[mattrpav]

This is probably 'ok' to close, since syndesis now uses the atlas-maven-plugin to generate inspection definitions ahead of time.

[mattrpav]

Tomo working on syndesis.io UI ticket to complete the usage of the pre-generated inspection definitions

[mattrpav]

This needs to be discussed in order to support Stand Alone Data Mapper class inspection