Refactor build, deploy and run workflow #107

lburgazzoli · 2018-09-20T05:10:54Z

removed:

project build
intermediate copy of dependencies mvn dependency:copy-dependencies

improved:

maven update policy set to never

new things:

compute classpath with mvn dependency:build-classpath
classpath included in cr and runtime configmap
s2i/run now uses classpath from a file (from the configmap)

notes:

some workarounds are needed as run-java.sh appears to have troubles in case JAVA_APP_JAR is not set
artifacts location is different from build time /tmp/artifacts/m2 to runtime /runtime/m2

… computed classpath file

nicolaferraro

Generally it seems ok. If it works, I can start from this to separate classpath computation from image publishing and to define different strategies for the latter..

nicolaferraro · 2018-09-20T05:41:15Z

deploy/platform-integration-context-core.yaml

@@ -1,9 +1,12 @@
 apiVersion: camel.apache.org/v1alpha1
 kind: IntegrationContext
 metadata:
-  name: root.integrationcontexts.camel.apache.org
+  name: core.integrationcontexts.camel.apache.org


Why not just "core"?

because it's too simple :)

lburgazzoli · 2018-09-20T08:42:05Z

let me do some cleanup

lburgazzoli · 2018-09-20T15:40:33Z

context with dependencies:

$ oc get ictx groovy -o yaml
apiVersion: camel.apache.org/v1alpha1
kind: IntegrationContext
metadata:
  clusterName: ""
  creationTimestamp: 2018-09-20T15:36:35Z
  generation: 0
  labels:
    app: camel-k
    camel.apache.org/context.created.by.kind: Operator
    camel.apache.org/context.created.by.name: core
    camel.apache.org/context.type: platform
  name: groovy
  namespace: camel-k
  resourceVersion: "179138"
  selfLink: /apis/camel.apache.org/v1alpha1/namespaces/camel-k/integrationcontexts/groovy
  uid: f4cc58fc-bcea-11e8-97bd-525400eed8b1
spec:
  classpath:
  - id: org.apache.camel.k:camel-k-runtime-jvm:jar:0.0.3-SNAPSHOT
    location: /tmp/artifacts/m2/org/apache/camel/k/camel-k-runtime-jvm/0.0.3-SNAPSHOT/camel-k-runtime-jvm-0.0.3-SNAPSHOT.jar
  - id: org.slf4j:slf4j-api:jar:1.7.25
    location: /tmp/artifacts/m2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar
  - id: org.apache.logging.log4j:log4j-slf4j-impl:jar:2.11.0
    location: /tmp/artifacts/m2/org/apache/logging/log4j/log4j-slf4j-impl/2.11.0/log4j-slf4j-impl-2.11.0.jar
  - id: org.apache.logging.log4j:log4j-api:jar:2.11.0
    location: /tmp/artifacts/m2/org/apache/logging/log4j/log4j-api/2.11.0/log4j-api-2.11.0.jar
  - id: org.apache.logging.log4j:log4j-core:jar:2.11.0
    location: /tmp/artifacts/m2/org/apache/logging/log4j/log4j-core/2.11.0/log4j-core-2.11.0.jar
  - id: org.jooq:joor-java-8:jar:0.9.9
    location: /tmp/artifacts/m2/org/jooq/joor-java-8/0.9.9/joor-java-8-0.9.9.jar
  - id: commons-io:commons-io:jar:2.6
    location: /tmp/artifacts/m2/commons-io/commons-io/2.6/commons-io-2.6.jar
  - id: org.apache.commons:commons-lang3:jar:3.8
    location: /tmp/artifacts/m2/org/apache/commons/commons-lang3/3.8/commons-lang3-3.8.jar
  - id: org.apache.camel:camel-core:jar:2.22.1
    location: /tmp/artifacts/m2/org/apache/camel/camel-core/2.22.1/camel-core-2.22.1.jar
  - id: com.sun.xml.bind:jaxb-core:jar:2.3.0.1
    location: /tmp/artifacts/m2/com/sun/xml/bind/jaxb-core/2.3.0.1/jaxb-core-2.3.0.1.jar
  - id: com.sun.xml.bind:jaxb-impl:jar:2.3.0.1
    location: /tmp/artifacts/m2/com/sun/xml/bind/jaxb-impl/2.3.0.1/jaxb-impl-2.3.0.1.jar
  - id: org.apache.camel:camel-groovy:jar:2.22.1
    location: /tmp/artifacts/m2/org/apache/camel/camel-groovy/2.22.1/camel-groovy-2.22.1.jar
  - id: org.codehaus.groovy:groovy-all:jar:2.4.15
    location: /tmp/artifacts/m2/org/codehaus/groovy/groovy-all/2.4.15/groovy-all-2.4.15.jar
  dependencies:
  - camel:core
  - camel:groovy
status:
  digest: vTA1Ab9PkI7TjboutWhUsury6fw-NYeOzlzHNUfydMfw
  image: 172.30.1.1:5000/camel-k/camel-k-context-groovy:179015
  phase: Ready

lburgazzoli · 2018-09-20T15:41:36Z

classpath file content:

[jboss@3787dd263f1e deployments]$ cat classpath 
dependencies/org.apache.camel.k/camel-k-runtime-jvm-0.0.3-SNAPSHOT.jar
dependencies/org.slf4j/slf4j-api-1.7.25.jar
dependencies/org.apache.logging.log4j/log4j-slf4j-impl-2.11.0.jar
dependencies/org.apache.logging.log4j/log4j-api-2.11.0.jar
dependencies/org.apache.logging.log4j/log4j-core-2.11.0.jar
dependencies/org.jooq/joor-java-8-0.9.9.jar
dependencies/commons-io/commons-io-2.6.jar
dependencies/org.apache.commons/commons-lang3-3.8.jar
dependencies/org.apache.camel/camel-core-2.22.1.jar
dependencies/com.sun.xml.bind/jaxb-core-2.3.0.1.jar
dependencies/com.sun.xml.bind/jaxb-impl-2.3.0.1.jar
dependencies/org.apache.camel/camel-groovy-2.22.1.jar
dependencies/org.codehaus.groovy/groovy-all-2.4.15.jar

nicolaferraro

Really good! Just few comments, mainly for clarification..

nicolaferraro · 2018-09-20T15:39:59Z

pkg/apis/camel/v1alpha1/types.go

@@ -125,6 +125,7 @@ type IntegrationContext struct {
 // IntegrationContextSpec --
 type IntegrationContextSpec struct {
 	Dependencies  []string            `json:"dependencies,omitempty"`
+	Classpath     []ClasspathEntry    `json:"classpath,omitempty"`


Should we move this to "status"? I don't think it's something the user should set. It's computed from the dependencies.

nicolaferraro · 2018-09-20T15:44:33Z

pkg/apis/camel/v1alpha1/types.go

+// ClasspathEntry --
+type ClasspathEntry struct {
+	ID       string `json:"id" yaml:"id"`
+	Location string `json:"location,omitempty" yaml:"location,omitempty"`


We store both the ID and the relative location?

Or that is intended to be the location in the m2 repo in the builder pod? (or both?)

nicolaferraro · 2018-09-20T15:53:44Z

runtime/dependency-lister/pom.xml

+  <modelVersion>4.0.0</modelVersion>
+  <packaging>maven-plugin</packaging>
+
+  <artifactId>camel-k-runtime-dependency-lister</artifactId>


I think we can give it a more generic name and reuse it for other tasks (in the future, not necessarily now). E.g. "camel-k-runtime-meta-plugin".
I'm thinking e.g. to have a mojo that extract info on the route (to be called out-of-band respect to image build), in order to have full info about the route and then apply the optimizations we talked about in the dev mailing list. Just thinking..

agree, feel free to rename it :)

oscerd

Awesome

lburgazzoli · 2018-09-20T16:19:12Z

@nicolaferraro here the new context:

apiVersion: camel.apache.org/v1alpha1
kind: IntegrationContext
metadata:
  clusterName: ""
  creationTimestamp: 2018-09-20T16:17:00Z
  generation: 0
  labels:
    app: camel-k
    camel.apache.org/context.created.by.kind: Operator
    camel.apache.org/context.created.by.name: core
    camel.apache.org/context.type: platform
  name: groovy
  namespace: camel-k
  resourceVersion: "182724"
  selfLink: /apis/camel.apache.org/v1alpha1/namespaces/camel-k/integrationcontexts/groovy
  uid: 9a9901a5-bcf0-11e8-97bd-525400eed8b1
spec:
  dependencies:
  - camel:core
  - camel:groovy
status:
  classpath:
  - org.apache.camel.k:camel-k-runtime-jvm:jar:0.0.3-SNAPSHOT
  - org.slf4j:slf4j-api:jar:1.7.25
  - org.apache.logging.log4j:log4j-slf4j-impl:jar:2.11.0
  - org.apache.logging.log4j:log4j-api:jar:2.11.0
  - org.apache.logging.log4j:log4j-core:jar:2.11.0
  - org.jooq:joor-java-8:jar:0.9.9
  - commons-io:commons-io:jar:2.6
  - org.apache.commons:commons-lang3:jar:3.8
  - org.apache.camel:camel-core:jar:2.22.1
  - com.sun.xml.bind:jaxb-core:jar:2.3.0.1
  - com.sun.xml.bind:jaxb-impl:jar:2.3.0.1
  - org.apache.camel:camel-groovy:jar:2.22.1
  - org.codehaus.groovy:groovy-all:jar:2.4.15
  digest: vTA1Ab9PkI7TjboutWhUsury6fw-NYeOzlzHNUfydMfw
  image: 172.30.1.1:5000/camel-k/camel-k-context-groovy:182662
  phase: Ready

nicolaferraro

Very nice!

lburgazzoli added 2 commits September 20, 2018 06:58

maven: remove build step, set updatePolicy

f3de89a

maven: use maven like structure to store artifacts at runtime and pre…

f353c16

… computed classpath file

lburgazzoli added the status/wip Work in progress label Sep 20, 2018

lburgazzoli requested review from nicolaferraro and oscerd September 20, 2018 05:10

oscerd approved these changes Sep 20, 2018

View reviewed changes

nicolaferraro reviewed Sep 20, 2018

View reviewed changes

lburgazzoli added 4 commits September 20, 2018 12:30

maven: cleanup build process

03e2b50

maven: plugin to generate detailed project dependencies

8f21cb8

maven: refactor how dependencies are used

673aed4

chore(context): add dependencies to provided groovy context

4238889

lburgazzoli force-pushed the maven-classpath branch from 23fb6d5 to 4238889 Compare September 20, 2018 15:38

nicolaferraro reviewed Sep 20, 2018

View reviewed changes

oscerd approved these changes Sep 20, 2018

View reviewed changes

fix findings

750e8e5

nicolaferraro approved these changes Sep 20, 2018

View reviewed changes

nicolaferraro merged commit 8ffe7b5 into apache:master Sep 20, 2018

lburgazzoli removed the status/wip Work in progress label Sep 21, 2018

lburgazzoli deleted the maven-classpath branch September 21, 2018 04:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor build, deploy and run workflow #107

Refactor build, deploy and run workflow #107

lburgazzoli commented Sep 20, 2018

nicolaferraro left a comment

nicolaferraro Sep 20, 2018

lburgazzoli Sep 20, 2018

lburgazzoli commented Sep 20, 2018

lburgazzoli commented Sep 20, 2018

lburgazzoli commented Sep 20, 2018

nicolaferraro left a comment

nicolaferraro Sep 20, 2018

nicolaferraro Sep 20, 2018

nicolaferraro Sep 20, 2018

nicolaferraro Sep 20, 2018

lburgazzoli Sep 20, 2018

oscerd left a comment

lburgazzoli commented Sep 20, 2018

nicolaferraro left a comment

Refactor build, deploy and run workflow #107

Refactor build, deploy and run workflow #107

Conversation

lburgazzoli commented Sep 20, 2018

nicolaferraro left a comment

Choose a reason for hiding this comment

nicolaferraro Sep 20, 2018

Choose a reason for hiding this comment

lburgazzoli Sep 20, 2018

Choose a reason for hiding this comment

lburgazzoli commented Sep 20, 2018

lburgazzoli commented Sep 20, 2018

lburgazzoli commented Sep 20, 2018

nicolaferraro left a comment

Choose a reason for hiding this comment

nicolaferraro Sep 20, 2018

Choose a reason for hiding this comment

nicolaferraro Sep 20, 2018

Choose a reason for hiding this comment

nicolaferraro Sep 20, 2018

Choose a reason for hiding this comment

nicolaferraro Sep 20, 2018

Choose a reason for hiding this comment

lburgazzoli Sep 20, 2018

Choose a reason for hiding this comment

oscerd left a comment

Choose a reason for hiding this comment

lburgazzoli commented Sep 20, 2018

nicolaferraro left a comment

Choose a reason for hiding this comment