Documentation. Can Kamelet pull dependencies from private artifactory?

[mkhudyakov]

Documentation provides an example of dependencies declaration

# ...
spec:
  # ...
  dependencies:
  - camel:telegram
  - mvn:org.apache.commons:commons-vfs2:2.7.0
  - github:apache/camel-kamelets
  template:
    # ...

Couple of questions though,

• can private artifactory be targeted? Any documentation/example please?
• can such dependency include bean or processor, so that Kamelet can use it?

[oscerd]

private artifactory is not supported as of today.

Depedencies could include beans and processors

[mkhudyakov]

Thank you @oscerd

Could you provide an example of dependency defined through the Kamelet, so that beans and processors are automatically bound to the registry?

1. I have an example of an application

        final KameletMain main = new KameletMain();
        main.configure().setBasePackageScan("com.apache.kamelet.poc");

and a application.properties

camel.component.kamelet.location = classpath:/kamelets,github:apache:camel-kamelets/kamelets
camel.main.routes-include-pattern = classpath:camel/*
camel.main.routes-reload-enabled = true
camel.main.routes-reload-directory = src/main/resources
camel.main.routes-reload-pattern = camel/*.yaml
camel.main.shutdown-timeout = 5
camel.main.lightweight = false

camel.main.autoConfigurationEnabled=true
camel.main.autoConfigurationEnvironmentVariablesEnabled=true
camel.main.autowiredEnabled=true
camel.main.basePackageScanEnabled=true

The application defines the route- camel/application-route.yaml

- route:
    id: "example"
    from:
      uri: "kamelet:example-source?topic=kafka_topic"
      steps:
        - log: "${body}"

2. The kamelet:example-source is defined in separate dependency (classpath-based lookup above allows application to see it)

apiVersion: camel.apache.org/v1alpha1
kind: Kamelet
metadata:
  name: example-source
  annotations:
    camel.apache.org/kamelet.support.level: "Stable"
    ...
spec:
  definition:
     ...
  types:
    out:
      mediaType: text/plain
  dependencies:
    - "camel:kamelet"
    - "camel:core"
    - "camel:kafka"
    - "mvn:com.apache.kamelet.poc:kamelet-poc-processors:1.0.0-SNAPSHOT"
  template:
    from:
      uri: "kafka:{{topic}}"
      parameters:
        brokers: "{{?bootstrapServers}}"
        groupId: "{{?consumerGroup}}"
        autoCommitEnable: "{{autoCommitEnable}}"
      steps:
        - bean: "customProcessor"
        - to: "kamelet:sink"

3. kamelet-poc-processors module defines bean & configuration

@Configuration
public class CustomConfiguration implements CamelConfiguration {

    @BindToRegistry
    public CustomProcessor customProcessor() {
        return new CustomProcessor();
    }
...
public class CustomProcessor {
    public void handle(Exchange exchange) throws Exception {
    ...

KameletMain pulls dependencies indeed, but then fails with

Caused by: org.apache.camel.NoSuchBeanException: No bean could be found in the registry for: customProcessor
	at org.apache.camel.component.bean.RegistryBean.doGetBean(RegistryBean.java:134)
	at org.apache.camel.component.bean.RegistryBean.getBean(RegistryBean.java:102)
	at org.apache.camel.component.bean.RegistryBean.createCacheHolder(RegistryBean.java:95)

However, once we add com.apache.kamelet.poc:kamelet-poc-processors:1.0.0-SNAPSHOT dependency to application's classpath, the error goes away.

How can such a problem be fixed, so that dependency is defined through the Kamelet? I will be glad to create a pull request in case such an issue can be resolved

[oscerd]

I think you should add the dependency as modeling. Cc @davsclaus

[davsclaus]

What Camel version do you use? And this is running via camel-jbang / camel-kamelet-main and NOT camel-k ?

Adding to the kamelet as you have via

• "mvn:com.apache.kamelet.poc:kamelet-poc-processors:1.0.0-SNAPSHOT"

should allow camel-kamelet-main to detect this and download it, however we have switched to a better maven downloader for the upcoming 3.20.0 release. So you can try using a -SNAPSHOT for that version

[davsclaus]

And in 3.20, then camel download standard maven approach via your .m2/settings.xml file (and what else maven has). You can also set explicit repo
https://camel.apache.org/manual/camel-jbang.html#_configuring_exporting

On kamelet main you can call the setRepos method. Its a URL for custom maven repo.

[mkhudyakov]

Thank you @davsclaus
I run on plain JRE (no camel-jbang or camel-k)

public class Application {

    public static void main(String[] args) throws Exception {
        final KameletMain main = new KameletMain();
        main.configure().setBasePackageScan("com.apache.kamelet.poc");

        main.setDownloadListener(new DownloadListener() {
            @Override
            public void onDownloadDependency(String groupId, String artifactId, String version) {
                System.out.println("Downloading " + groupId + ":" + artifactId + ":" + version);
            }

Just to clarify, there is no issue for KameletMain to download the dependency in such a case

09:13:16.453 [main] INFO  o.apache.camel.main.BaseMainSupport -     [application.properties]       camel.main.basePackageScan=com.apache.kamelet.poc
09:13:16.453 [main] INFO  o.apache.camel.main.BaseMainSupport -     [application.properties]       camel.component.kamelet.location=classpath:/kamelets,github:apache:camel-kamelets/kamelets
Downloading org.apache.camel:camel-core-languages:3.20.0-SNAPSHOT
Downloading org.apache.camel.kamelets:camel-kamelets-utils:0.9.2
Downloading org.apache.camel:camel-kafka:3.20.0-SNAPSHOT
**Downloading com.apache.kamelet.poc:kamelet-poc-processors:1.0.0-SNAPSHOT**
Downloading org.apache.camel:camel-yaml-dsl:3.20.0-SNAPSHOT
Downloading org.apache.camel:camel-kafka:3.20.0-SNAPSHOT

The problem is for KameletMain to detect @Configuration / CamelConfiguration in downloaded dependency and bind beans to the registry.
As a result I get

Caused by: org.apache.camel.NoSuchBeanException: No bean could be found in the registry for: customProcessor
	at org.apache.camel.component.bean.RegistryBean.doGetBean(RegistryBean.java:134)

The above issue goes away though in case dependency is explicitly added to the classpath (so no download happens for it)

I've tried the version 3.20.0-SNAPSHOT, but the result is the same.

[davsclaus]

Okay so scanning kamelet-main downloaded JARs via basePackageScanEnabled is not supported.

Its potentially a little bit difficult as downloads can happen later than the first scanning, and maybe its a bit "too magic".
As what should Camel do if there are similar annotations for spring boot / quarkus in the JAR with their style of annotations.

[mkhudyakov]

I can imagine the mechanism like

    public static void main(String[] args) throws Exception {
        final KameletMain main = new KameletMain();
        main.configure().setBasePackageScan("com.apache.kamelet.poc");

        Map<String, Class> configurations = new HashMap<>();
        main.setDownloadListener(new DownloadListener() {

            @Override
            public void onDownloadDependency(String groupId, String artifactId, String version) {
                System.out.println("Downloaded, " + groupId + ":" + artifactId + ":" + version);

                Reflections reflections = new Reflections("com.apache.kamelet.poc");
                Set<Class<?>> classes = reflections.getTypesAnnotatedWith(Configuration.class);

                for (Class clazz: classes) {
                    addConfiguration(configurations, clazz, main);
                }
            }

            @Override
            public void onAlreadyDownloadedDependency(String groupId, String artifactId, String version) {
                System.out.println("Already downloaded, " + groupId + ":" + artifactId + ":" + version);

                Reflections reflections = new Reflections("com.apache.kamelet.poc");
                Set<Class<?>> classes = reflections.getTypesAnnotatedWith(Configuration.class);

                for (Class clazz: classes) {
                    addConfiguration(configurations, clazz, main);
                }
            }
        });

        main.run(args);
    }

    private static void addConfiguration(Map<String, Class> configurations, Class clazz, KameletMain main) {
        if (configurations.get(clazz.getCanonicalName()) != null) {
            System.out.println(clazz.getCanonicalName() + " already registered");
            return;
        } else {
            configurations.put(clazz.getCanonicalName(), clazz);
        }

        try {
            main.configure().addConfiguration((CamelConfiguration) clazz.newInstance());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

as the downloading happens before the Kametet runs, but the question is how to force KameletMain start accounting for configurations?

The line

            main.configure().addConfiguration((CamelConfiguration) clazz.newInstance());

doesn't have an effect

[davsclaus]

you need to create the clazz instance via camel and not plain java - see injector on camel context

[davsclaus]

I created a ticket about scanning downloaded JARs
https://issues.apache.org/jira/browse/CAMEL-18815

[mkhudyakov]

Thank you a lot, @davsclaus, the injector helped

            CamelContext camelContext = main.getCamelContext();
            if (camelContext != null) {
                Object clazzInstance = main.getCamelContext().getRegistry().findSingleByType(clazz);
                if (clazzInstance != null) {
                    System.out.println(clazz.getCanonicalName() + " already registered");
                    return;
                }

                Injector injector = camelContext.getInjector();
                main.configure().addConfiguration((CamelConfiguration) injector.newInstance(clazz));
            } else {
                System.err.println("CamelContext is not defined");
            }

Could you also clarify, is such approach, when dependency is defined through the Kamelet, actually the way how it has to work once deployed to K8S / Camel K ?
Or whether the fat jars can be deployed to Camel K?

My goal with KameletMain is a transition from Spring Boot with Camel and camel route resource being deployed to K8S.
Is there any 'executive summary' documentation, which explain the benefits of Camel K as opposed to Spring Boot + Camel?

[davsclaus]

Camel K is not able to do all of this that camel-kamelet-main can do.

You can use camel-jbang and its 'export' command to export to camel-quarkus (what camel-k runs on top of) as a regular maven based project and deploy to k8s as a standard quarkus project.

For Camel K "benefits" then there is its website: https://camel.apache.org/camel-k/1.11.x/. And its better to ask on camel-k zulip chat room or user mailing list instead of issue tracker.

Closing this ticket