Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jitpack dependency is not resolved #2589

Closed
mmacphail opened this issue Aug 24, 2021 · 9 comments
Closed

jitpack dependency is not resolved #2589

mmacphail opened this issue Aug 24, 2021 · 9 comments
Labels
kind/bug Something isn't working

Comments

@mmacphail
Copy link
Contributor

mmacphail commented Aug 24, 2021

Versions
kubernetes: docker for dekstop 1.19
camelk client : Camel K Client 1.5.0

Given the following kamel installation:

kamel install --registry $REGISTRY --organization mmacphail --registry-secret regcred --maven-repository https://packages.confluent.io/maven/@id=confluent --maven-repository https://jitpack.io/@id=jitpack

And the following helloworld.groovy file:

def bootstrap = 'my-cluster-kafka-bootstrap.kafka:9092'
def topic = 'sensor'
def groupId = 'test-camel-k'
def keyDeserializer = 'org.apache.kafka.common.serialization.StringDeserializer'
def valueDeserializer = 'io.confluent.kafka.serializers.KafkaAvroDeserializer'
def schemaRegistryURL = 'http://schema-registry-service.kafka:8081'

def kafka = "kafka:${topic}?brokers=${bootstrap}&groupId=${groupId}&autoOffsetReset=earliest&keyDeserializer=${keyDeserializer}&valueDeserializer=${valueDeserializer}&schemaRegistryURL=${schemaRegistryURL}"

from(kafka)
  .process({e -> println e.getIn().getBody().getClass()})
  .to('log:info?showBody=true&showHeaders=true')

Running the integration using jitpack dependency fails using this kamel run command:

kamel run -d camel-kafka -d mvn:io.confluent:kafka-avro-serializer:6.0.0 -d mvn:com.github.mmacphail:temp-schemas:1.0 helloworld.groovy

This fails although https://jitpack.io/#mmacphail/temp-schemas/1.0 exists.

The failure happens during the building kit phase.
A describe such as kubectl describe integration helloworld shows the following information:

  Normal  ReasonRelatedObjectChanged   2m6s                 camel-k-build-controller            Integration helloworld subresource kit-c4ia81qmvrereqhrjk50 (Build) changed phase to "Running" (recovery 1 of 5)
  Normal  ReasonRelatedObjectChanged   2m3s                 camel-k-build-controller            Integration helloworld subresource kit-c4ia81qmvrereqhrjk50 (Build) changed phase to "Failed" (recovery 1 of 5)
  Normal  ReasonRelatedObjectChanged   54s (x15 over 116s)  camel-k-build-controller            (combined from similar events): Integration helloworld subresource kit-c4ia81qmvrereqhrjk50 (Build) changed phase to "Failed" (recovery 4 of 5)
  Normal  ReasonRelatedObjectChanged   54s (x15 over 116s)  camel-k-build-controller            (combined from similar events): Integration helloworld subresource kit-c4ia81qmvrereqhrjk50 (Build) changed phase to "Failed" (recovery 4 of 5)

Logs of the camel-k-operator pod show the following:

{"level":"error","ts":1629791431.6187263,"logger":"camel-k.maven.build","msg":"Failed to execute goal on project camel-k-integration: Could not resolve dependencies for project org.apache.camel.k.integration:camel-k-integration:jar:1.5.0: Could not find artifact com.github.mmacphail.temp-schemas:temp-schemas:jar:1.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]","stacktrace":"github.com/apache/camel-k/pkg/util/log.Logger.Errorf\n\tgithub.com/apache/camel-k/pkg/util/log/log.go:60\ngithub.com/apache/camel-k/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/pkg/util/maven/maven_log.go:62\ngithub.com/apache/camel-k/pkg/util/maven.(*Command).Do\n\tgithub.com/apache/camel-k/pkg/util/maven/maven_command.go:140\ngithub.com/apache/camel-k/pkg/builder.BuildQuarkusRunnerCommon\n\tgithub.com/apache/camel-k/pkg/builder/quarkus.go:189\ngithub.com/apache/camel-k/pkg/builder.buildQuarkusRunner\n\tgithub.com/apache/camel-k/pkg/builder/quarkus.go:163\ngithub.com/apache/camel-k/pkg/builder.(*builderStep).execute\n\tgithub.com/apache/camel-k/pkg/builder/steps.go:46\ngithub.com/apache/camel-k/pkg/builder.(*builderTask).Do\n\tgithub.com/apache/camel-k/pkg/builder/builder.go:125\ngithub.com/apache/camel-k/pkg/controller/build.(*monitorRoutineAction).runBuild\n\tgithub.com/apache/camel-k/pkg/controller/build/monitor_routine.go:146"}

I don't understand why this dependency is not found since it's the jitpack repository is correctly added to the settings.xml. The command kubectl describe configmap camel-k-maven-settings returns the following:

Name:         camel-k-maven-settings
Namespace:    default
Labels:       app=camel-k
Annotations:  <none>

Data
====
settings.xml:
----
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">
  <localRepository></localRepository>
  <profiles>
    <profile>
      <id>maven-settings</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>central</id>
          <url>https://repo.maven.apache.org/maven2</url>
          <snapshots>
            <enabled>false</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <releases>
            <enabled>true</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </releases>
        </repository>
        <repository>
          <id>confluent</id>
          <url>https://packages.confluent.io/maven/</url>
          <snapshots>
            <enabled>false</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <releases>
            <enabled>true</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </releases>
        </repository>
        <repository>
          <id>jitpack</id>
          <url>https://jitpack.io/</url>
          <snapshots>
            <enabled>false</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <releases>
            <enabled>true</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </releases>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>central</id>
          <url>https://repo.maven.apache.org/maven2</url>
          <snapshots>
            <enabled>false</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <releases>
            <enabled>true</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </releases>
        </pluginRepository>
        <pluginRepository>
          <id>confluent</id>
          <url>https://packages.confluent.io/maven/</url>
          <snapshots>
            <enabled>false</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <releases>
            <enabled>true</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </releases>
        </pluginRepository>
        <pluginRepository>
          <id>jitpack</id>
          <url>https://jitpack.io/</url>
          <snapshots>
            <enabled>false</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <releases>
            <enabled>true</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </releases>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
  <mirrors></mirrors>
</settings>
Events:  <none>
@mmacphail mmacphail changed the title Can't run integration with jitpack dependencies Can't run integration with jitpack dependency Aug 24, 2021
@squakez squakez added the kind/bug Something isn't working label Aug 24, 2021
@hadoken79
Copy link
Contributor

I have similar issue

Azure Kubernetes version 1.19.11
Camel K Client 1.3.1

i run a simple test integration which references a jitpack dependency containing a processor and some test-classes

kubectl describe integration test
Integration:

Name:         test
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  camel.apache.org/v1
Kind:         Integration
Metadata:
  Creation Timestamp:  2021-08-28T07:52:13Z
  Generation:          1
  Managed Fields:
    API Version:  camel.apache.org/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:dependencies:
        f:digest:
        f:kit:
        f:lastInitTimestamp:
        f:phase:
        f:platform:
        f:profile:
        f:runtimeProvider:
        f:runtimeVersion:
        f:version:
    Manager:      kamel
    Operation:    Update
    Time:         2021-08-28T07:52:13Z
    API Version:  camel.apache.org/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:dependencies:
        f:sources:
    Manager:         kamel.exe
    Operation:       Update
    Time:            2021-08-28T07:52:13Z
  Resource Version:  25481282
  Self Link:         /apis/camel.apache.org/v1/namespaces/default/integrations/test
  UID:               283f02cb-f82c-4fe5-b592-9152b32b59fc
Spec:
  Dependencies:
    jitpack:com.github.hadoken79/jitPackCamelTest/v1.1
  Sources:
    Content:  // camel-k: dependency=jitpack:com.github.hadoken79/jitPackCamelTest/v1.1
import ch.tralala.testProcessor;
import org.apache.camel.builder.RouteBuilder;

/**
     * A Camel Java DSL Router
     */
    public class Test extends RouteBuilder {

        /**
         * Let's configure the Camel routing rules using Java code...
         */
        public void configure() {

            //getContext().setTracing(true);

            from("timer://foo?fixedRate=true&period=10000")
                    .setProperty("key", constant("val"))
                    .setHeader("key", constant("val"))
                    .setBody(constant("foo"))
                    .process(new testProcessor());
        }
    }

    Name:  Test.java
Status:
  Conditions:
    First Truthy Time:     2021-08-28T07:52:13Z
    Last Transition Time:  2021-08-28T07:52:13Z
    Last Update Time:      2021-08-28T07:52:13Z
    Message:               camel-k
    Reason:                IntegrationPlatformAvailable
    Status:                True
    Type:                  IntegrationPlatformAvailable
    Last Transition Time:  2021-08-28T07:52:13Z
    Last Update Time:      2021-08-28T07:52:13Z
    Message:               creating a new integration kit
    Reason:                IntegrationKitAvailable
    Status:                False
    Type:                  IntegrationKitAvailable
  Dependencies:
    camel-quarkus:timer
    jitpack:com.github.hadoken79/jitPackCamelTest/v1.1
    mvn:org.apache.camel.k/camel-k-loader-java
    mvn:org.apache.camel.k/camel-k-runtime
  Digest:               vtgnAVk8BH8fmI5xvYHAVh-pS6fau6ggrJluJqoDSP5o
  Kit:                  kit-c4kum0bj3qfqnm9cbrl0
  Last Init Timestamp:  2021-08-28T07:52:13Z
  Phase:                Error
  Platform:             camel-k
  Profile:              Kubernetes
  Runtime Provider:     quarkus
  Runtime Version:      1.6.0
  Version:              1.3.1
Events:
  Type    Reason                       Age                    From                                Message
  ----    ------                       ----                   ----                                -------
  Normal  IntegrationConditionChanged  4m58s                  camel-k-integration-controller      Condition "IntegrationPlatformAvailable" is "True" for Integration test: camel-k
  Normal  IntegrationPhaseUpdated      4m58s                  camel-k-integration-controller      Integration test in phase "Initialization"
  Normal  IntegrationConditionChanged  4m58s                  camel-k-integration-controller      Condition "IntegrationKitAvailable" is "False" for Integration test: creating a new integration kit
  Normal  IntegrationPhaseUpdated      4m58s                  camel-k-integration-controller      Integration test in phase "Building Kit"
  Normal  ReasonRelatedObjectChanged   4m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Initialization" (recovery 4 of 5)
  Normal  ReasonRelatedObjectChanged   4m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Scheduling" (recovery 4 of 5)
  Normal  ReasonRelatedObjectChanged   4m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Pending" (recovery 4 of 5)
  Normal  ReasonRelatedObjectChanged   4m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Running" (recovery 4 of 5)
  Normal  ReasonRelatedObjectChanged   4m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Failed" (recovery 4 of 5)
  Normal  ReasonRelatedObjectChanged   3m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Initialization" (recovery 5 of 5)
  Normal  ReasonRelatedObjectChanged   3m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Scheduling" (recovery 5 of 5)
  Normal  ReasonRelatedObjectChanged   3m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Pending" (recovery 5 of 5)
  Normal  ReasonRelatedObjectChanged   3m21s                  camel-k-build-controller            Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Running" (recovery 5 of 5)
  Normal  ReasonRelatedObjectChanged   3m21s (x2 over 3m21s)  camel-k-build-controller            (combined from similar events): Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Error" (recovery 5 of 5)
  Normal  ReasonRelatedObjectChanged   3m21s (x2 over 3m21s)  camel-k-build-controller            (combined from similar events): Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Build) changed phase to "Error" (recovery 5 of 5)
  Normal  ReasonRelatedObjectChanged   3m21s                  camel-k-integration-kit-controller  Integration test subresource kit-c4kum0bj3qfqnm9cbrl0 (Integration Kit) changed phase to "Error"
  Normal  IntegrationPhaseUpdated      3m21s                  camel-k-integration-controller      Integration test in phase "Error"

on jitpack jar is available, also in a local mvn project route works in camel

https://jitpack.io/#hadoken79/jitPackCamelTest/v1.1

In kamel-operator logs:

{"level":"info","ts":1630139324.0974753,"logger":"camel-k.builder","msg":"step failed with error: unknown dependency type: jitpack:com.github.hadoken79/jitPackCamelTest/v1.1","step":"github.com/apache/camel-k/pkg/builder/InjectDependencies","phase":12,"name":"kit-c4kv763j3qfqnm9cbrm0","task":"builder"}

kubectl describe configmap camel-k-maven-settings

Name:         camel-k-maven-settings
Namespace:    default
Labels:       app=camel-k
Annotations:  <none>

Data
====
settings.xml:
----
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">
  <localRepository></localRepository>
  <profiles>
    <profile>
      <id>maven-settings</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>central</id>
          <url>https://repo.maven.apache.org/maven2</url>
          <snapshots>
            <enabled>false</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <releases>
            <enabled>true</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </releases>
        </repository>
        <repository>
          <id>jitpack.io</id>
          <url>https://jitpack.io</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>central</id>
          <url>https://repo.maven.apache.org/maven2</url>
          <snapshots>
            <enabled>false</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <releases>
            <enabled>true</enabled>
            <checksumPolicy>fail</checksumPolicy>
          </releases>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
</settings>
Events:  <none>

@hadoken79
Copy link
Contributor

Update:
I made a mistake
In my route i used a modeline dependency in the wrong format.
i used

// camel-k: dependency=jitpack:github/user/repo/version
i just had to use
// camel-k: dependency=github:user/repo/version

Then there is no need to add the jitpack repo in maven-settings configmap or xml. It then just works out of the box.

@mmacphail
Copy link
Contributor Author

mmacphail commented Aug 29, 2021

Thanks to your example @hadoken79 I was able to pinpoint the issue.

External dependencies in jitpack don't work when a dash - is present in their name.
I used the modeline // camel-k: dependency=github:mmacphail/temps-chemas/1.0 which did not work, error messages indicated that the artifact that was downloaded from github was com.github.mmacphail.temp-schemas:temp-schemas and not com.github.mmacphail:temp-schemas.

The error message when using the modeline // camel-k: dependency=github:mmacphail/temp-schemas/1.0:

Failure to find com.github.mmacphail.temp-schemas:temp-schemas:jar:1.0

I cloned temp-schemas and named it tempschemas, using the modline // camel-k: dependency=github:mmacphail/tempschemas/1.0, it worked.

@mmacphail
Copy link
Contributor Author

So I can currently pinpoint two problems:

  • Documentation should indicate that jitpack dependency can be defined using the github prefix instead of the mvn prefix and jitpack repository.
  • There is a bug that cause jitpack url dependencies to duplicate the artifact name in the group id if the artifact name contains a dash.

@mmacphail mmacphail changed the title Can't run integration with jitpack dependency jitpack dependencies containing a dash in their name are not resolved Aug 29, 2021
@tadayosi
Copy link
Member

Regarding documentation, contributions are always welcomed. Good points to enhance would be:
https://camel.apache.org/camel-k/latest/configuration/dependencies.html
https://camel.apache.org/camel-k/latest/cli/modeline.html

@mmacphail
Copy link
Contributor Author

So I tried to debug this yesterday, without any success so far.
I believe the dash - in the dependency name is not the problem, because I was able to run my integration using a new version of the same dependency (version 1.2)... And it works.
So we have one version, version 1.0, where the dependency is incorrectly queried in the repository: Could not find artifact com.github.mmacphail.temp-schemas:temp-schemas:jar:1.0, and one version, 1.2, which works fine.

It beats me as to why. I think I'll close the issue if nobody has any idea.

I made the following tests in the codebase:

  • camel.ManageIntegrationDependencies
  • maven.ParseGAV

But these worked fine.

@mmacphail mmacphail changed the title jitpack dependencies containing a dash in their name are not resolved jitpack dependency is not resolved Aug 31, 2021
@lburgazzoli
Copy link
Contributor

you can try to grab the generated pom and run it locally so you can troubleshoot it.
it may be a glitch in jitpack, like for some reason it processes the github repo as a multi module (the syntax as per the syntax com.github.mmacphail.temp-schemas:temp-schemas)

@lburgazzoli
Copy link
Contributor

lburgazzoli commented Aug 31, 2021

I've create a simple project like:

<?xml version="1.0"?>
<project 
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" 
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <modelVersion>4.0.0</modelVersion>
  <groupId>org.acme</groupId>
  <artifactId>getting-started</artifactId>
  <version>1.0.0-SNAPSHOT</version>

  <properties>
    <compiler-plugin.version>3.8.1</compiler-plugin.version>
    <maven.compiler.parameters>true</maven.compiler.parameters>
    <maven.compiler.source>11</maven.compiler.source>
    <maven.compiler.target>11</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>com.github.mmacphail</groupId>
      <artifactId>temp-schemas</artifactId>
      <version>1.0</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${compiler-plugin.version}</version>
      </plugin>
    </plugins>
  </build>

	<repositories>
		<repository>
		    <id>jitpack.io</id>
		    <url>https://jitpack.io</url>
		</repository>
	</repositories>
  
</project>

then when building it fails with:

INFO] Scanning for projects...
[INFO] 
[INFO] ----------------------< org.acme:getting-started >----------------------
[INFO] Building getting-started 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from jitpack.io: https://jitpack.io/com/github/mmacphail/temp-schemas/temp-schemas/1.0/temp-schemas-1.0.pom
Downloading from central: https://repo.maven.apache.org/maven2/com/github/mmacphail/temp-schemas/temp-schemas/1.0/temp-schemas-1.0.pom
[WARNING] The POM for com.github.mmacphail.temp-schemas:temp-schemas:jar:1.0 is missing, no dependency information available
Downloading from jitpack.io: https://jitpack.io/com/github/mmacphail/temp-schemas/temp-schemas/1.0/temp-schemas-1.0.jar
Downloading from central: https://repo.maven.apache.org/maven2/com/github/mmacphail/temp-schemas/temp-schemas/1.0/temp-schemas-1.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.804 s
[INFO] Finished at: 2021-08-31T09:26:40+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project getting-started: Could not resolve dependencies for project org.acme:getting-started:jar:1.0.0-SNAPSHOT: Could not find artifact com.github.mmacphail.temp-schemas:temp-schemas:jar:1.0 in jitpack.io (https://jitpack.io) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

So it looks like a set-up error in the project or a glitch in jitpack

@mmacphail
Copy link
Contributor Author

Thank you for your investigation! This glitch is interresting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants