Cloud Native Integration with Apache Camel: Building Agile and Scalable Integrations for Kubernetes Platforms
@Singleton @Unremovable public class MyObjectConverter implements TypeConverters { (link)
Vikram: [A type converter]
.choice()
.when(header("preferred_title").isEqualToIgnoreCase("mrs"))
.setBody(simple("Hi Mrs.
Vikram: [Use a choice to select among alternatives]
exchange.getMessage().setBody(fileList); (link)
Vikram: [Get the message and set the payload to file list]
bean(FileReaderBean.class, "listFile" (link)
Vikram: [Call the bean's listFile method]
MyObjectConverter is a bean because it is annotated with @Singleton, which means that a single instance of this object will be created and maintained by the bean registry (link)
In this example, you are marshalling the data, which means transforming a Java object structure into a binary or textual format. (link)
Vikram: [This is similar to converting XML/JSON tree to bitstream (or BLOB)]
Using libraries as JAXB or JSON-B you can transform Java objects into XML/JSON or transform the Java object into an XML/JSON document. (link)
You can incorporate conditional steps in your routing logic using predicates. (link)
Beans can be accessed using the beans() fluent builder or using the bean component, in that case making reference to bean: endpoint. (link)
even though each exchange produces logs in different routes, the path-router and the logger- router, they are executed in the same thread because of the direct component. (link)
Direct allows you to aggregate a routing logic to another route. This way you can reuse a routing logic in more than one route. (link)
The direct component allows you to link, synchronously, different routes in the same Camel context. (link)
In 2016, the tooling and the specification were split, and the specification was renamed to OpenAPI. (link)
Created in 2011, Swagger is an open-source project that created a JSON/YAML representation for RESTful applications, taking many of the features built for the SOAP protocol. (link)
${body} is an example of an EL, in this case the Simple EL. (link)
Camel extensions already declare the other extensions they depend on. For example, the camel-quarkus-core dependency is already declared by camel-quarkus-rest. (link)
The JAX-RPC specification was a big step in standardizing HTTP-based communication using SOAP (Simple Object Access Protocol), an XML-based messaging protocol. JAX-RPC was eventually succeeded by the JAX-WS (Web Service) specification. (link)
tokens = client.refreshTokens(tokens.getRefreshToken()) .await().indefinitely(); currentTokens = tokens; (link)
Vikram: [Does this keep refreshing asynchronously? Why is it done on demand if it's done in init?
Possibly: Whenever called, the token is checked to see if it's expired, and then this call is made. As a getToken call was already made during initialization, now only refreshToken() is sufficient. The await() possibly means this is done asynchronously and indefinitely() means that there is no time limit.]
currentTokens = client.getTokens().await().indefinitely(); (link)
Vikram: [Done at init. Does this keep refreshing asynchronously?
Possibly doing this asynchronously with no time limit.]
.bean("tokenHandlerBean") (link)
Vikram: [All Token handling is done in the bean]
You are passing it as a header because HTTP clients transform message headers into HTTP headers. (link)
You also get a refresh token that allows the client to get new tokens and continue to access the resource server without a new authentication process. (link)
The only thing that is somewhat different is to implement the equals() and hashCode() methods . This is done to avoid duplicate entries in the contact list. (link)
By setting the bindingMode(RestBindingMode.json) you’re telling Camel that you want the incoming JSON data to be transformed into a POJO object, in this case the type(Contact.class) for the post() operation, and that the responses must be automatically converted to JSON. (link)
Keycloak implements two standards for authentication and authorization: SAML 2.0 and OpenID Connect. (link)
“OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol.”2 (link)
The OAuth 2.0 specification, which is the current version, describes six different grant types: Authorization Code
Client Credentials
Device Code
Refresh Token
Implicit Flow
Password Grant (link)
OpenID is responsible for the authentication and OAuth for authorization. (link)
SEDA. This component, based on a staged event-driven architecture, will create an in-memory queue so you can process messages in different threads. This way you are asynchronously sending a message copy to be processed by another thread. (link)
Vikram: [#TryOut Can I use this to implement #StatefulFilter that I had created for YNAP Gigya flow?]
Wire Tap is an integration pattern that allows an exchange to be copied or a new one to be generated using data from the original exchange, and sends it asynchronously to another endpoint. (link)
Camel comes with predefined strategies to handle exceptions called error handlers . (link)
As you can see, try/catch/finally works in a very similar way as in the Java language. You could have multiple doCatch() clauses, you could nest a try/catch/finally inside a doTry(), and so on. (link)
You are sending the parameters dynamically using the JPA_PARAMETERS_HEADER header. (link)
You are using CDI to register a named bean containing the parameters you need for that query. You use the JPQL named parameter as the key for the map and you can set any object as the value (link)
Instead of declaring the route logic nested in the REST DSL, I decided to separate each route and call them using direct (link)
Vikram: [Makes it easier to follow. It is like using a route label, but here each route is equivalent to a new flow]
.to("jpa:" + Contact.class.getName()+ "?findEntity=true"); (link)
Vikram: [New JPA component findEntity]
A mebibyte is just a different way to represent memory allocation, where 1 MiB = 2ˆ20 bytes or 1,048,576 bytes. Meanwhile 1MB is 10ˆ6 or 1,000,000 bytes. If you want to know how much 73Mi means in MB, just multiply the bytes for 1.04858, which is 76.546MB. (link)
If an application tries to use more memory than its limit, Kubelet will kill the application, causing it to restart to fix the problem. (link)
If a new pod has its requirements specified, the scheduler will try to find a node that matches the amount of CPU and RAM requested by the pod. (link)
Requests and Limits are ways to tell Kubernetes how much resources you believe the application should consume. (link)
In order to identify the application state, Kubernetes defines test routines called probes. There are three: Readiness, Liveness, and Startup. (link)
Quarkus allows you to use the same application.properties files to declare properties for different profiles (link)
ConfigMaps and Secrets are Kubernetes resources that allow users to separate images from the configuration. (link)
The code will be packed in the quarkus-app/app directory and the dependencies you use to work with Camel will be in the quarkus-app/lib/main directory. This process guarantees that the foundational classes are loaded first, in this case Quarkus classes, and then your code will be loaded, making the startup process smarter and, therefore, faster. (link)
The class-path points to Quarkus’ dependencies only and the main-class attribute points to a Quarkus class. (link)
Vikram: [This manifest file of quarkus-run.jar shows this.]
With the plugin goal quarkus:add-extension you can manipulate your pom structure in a simplified way. This command will add the dependencies you need using the version you have mapped in the Quarkus bill of materials, so you don’t need to worry about compatibility. (link)
Established in June 2015, the OCI is a Linux Foundation project, as is the CNCF, which was designed to create open industry standards around container formats and runtimes. (link)
Well, now you have the understanding of how to pack Java code using quarkus-maven-plugin. This is going to help you to distribute your code, especially in standalone applications, that you can configure as a service in the OS. (link)
Native Image is one running mode of GraalVM, a JDK project developed by Oracle to improve the code execution of Java and other JVM languages. In this mode, a native executable is created by the compiler (link)
In Quarkus, there are two ways of doing it: traditional JVM or native compilation. (link)
The MicroProfile specification is the equivalent of the Jakarta EE (formerly known as Java Platform, Enterprise Edition – Java EE) for the microservices frameworks. (link)
Vikram: [Microprofile is the J2EE subset for microservices.]
Quarkus provides a “bill of materials” dependency called quarkus-universe-bom, where every component of the framework is declared. This way you won’t need to worry about every dependency version and the compatibility between them. (link)
JDK 11 installed with JAVA_HOME configured appropriately
Maven 3.6.3 with M2_HOME configured (link)
Having many microservices running in Java is now very resource expensive. (link)
Now instead of deploying ten war files to an application server, these ten new services are packed into ten different container images, creating ten JVM process running as containers, which are orchestrated by your Kubernetes cluster. (link)
Vikram: [These services could be built with fat jars that run on jre.]
They created isolation mechanisms so every application could have its own class loading process, or could specify exactly which dependencies it would use, using for example the OSGi framework, but this didn’t solve the risk of putting all of the eggs in the same basket (link)
Application servers provided centralized capabilities such as security, resource sharing, configuration management, scalability, logging, and so on, with a very small price to pay: a few megabytes for the virtual machine and a few megabytes for the application server code itself, besides additional CPU usage. (link)
Vikram: [These are websphere, web logic and jboss. This was ok as all applications were deployed on the same server. Scaling these was a problem.]
Quarkus is an open source project released in 2018. It was made especially for the Kubernetes world, creating a mechanism to make Java development for Kubernetes a lot easier, and also dealing with Java “old” problems (link)
The setBody() method receives as an argument an object of the type Expression (link)
This means that every time the timer triggers, a new Exchange will be created, and that object will be available until that execution is finished (link)
Vikram: [Analogous to the code is in the compute node being activated for the duration of the message being processed. Exchange has the properties message, properties, exception which are similar to input root, input properties, and exception list. And it also has from endpoint, exchange id, pattern. The first two are part of the message headers and pattern must be responsible for parsing. If so it’s similar to the parsers set before the computer node.]
This execution of an incoming event is called an Exchange . (link)
By executing the configure() method, the outcome of these stacked calls is a route definition and it is used to instantiated many objects in memory. These objects in memory will react to events being triggered to or by the from (consumer) endpoint (link)
This is because Camel utilizes a fluent way to write routes where you can append definitions on how your route should behave or simply set attributes to your route (link)
Every integration built with Camel uses a concept called a route . The idea is that an integration always starts from an endpoint and then goes to one or multiples endpoints (link)
The CNCF’s objective was to maintain the Kubernetes project and also to serve as an umbrella for other projects that Kubernetes is based on or that would compose the ecosystem. In this context, cloud native means “made for the Kubernetes ecosystem.” (link)
Container: “A way of packing and distributing applications and their dependencies, from libraries to runtimes. From an execution standpoint, it is also a way to isolate OS (operating system) processes, creating a sandbox for each container, similar to a virtual machine idea.” (link)
Removing the integration layer wouldn’t mean that business information or data would get lost; it would only impact the integration of those services (link)
docker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic myTestTopic (link)
camel-kafka-consumer-v2/ $ mvn clean quarkus:dev -Ddebug=5006 -Dkafka.clientid=test -Dkafka.groupid=testGroup -Dkafka.consumers.count=2 -Dkafka.consumers.stream=1 (link)
Vikram: [Two consumers and one thread only. So only message is processed at a time.]
camel-kafka-consumer-v2/ $ mvn clean quarkus:dev -Ddebug=5006 -Dkafka.clientid=test -Dkafka.groupid=testGroup -Dkafka.consumers.count=1 -Dkafka.consumers.stream=10 (link)
Vikram: [Starting a consumer with one consumer and 10 threads.]
The consumerStreams parameter is responsible for setting the number of threads for the component’s thread pool and the consumersCount is responsible for setting the number of Kafka consumers in the application. (link)
Have in mind that the number of active consumers will depend on the number of partitions currently available for that topic. (link)
If you look at the consumers’ logs, you will see that the consumer without a partition assigned is not receiving messages. This is called a starving consumer. (link)
i=0; while [ $i -lt 10 ]; do ((i++)); curl -w "n" -X POST 'http://localhost:8080/kafka' -H 'Content-Type: text/plain' -d "Message number: $i" ; done (link)
from("{{kafka.uri}}") (link)
kafka.uri=kafka:{{topic}}?brokers={{brokers}}&clientId={{id}} (link)
Right before the Kafka component step, you are removing the message headers using a catch-all pattern. (link)
.removeHeaders("") .to("{{kafka.uri}}") .removeHeaders("") .setBody(constant("message sent.")) (link)
If you look at the logs on the first consumer, you will see that now only one partition is assigned to it. (link)
For this example, you are using Docker compose because Kafka needs a Zookeeper instance to connect to, so you define two services in the document, zookeeper and kafka (link)
topics in Kafka are essentially a group of partitions. (link)
Consumer group allows different instances of the same consumer application to parallelize readings without having duplication. (link)
Kafka keeps message indexes using a structure called offset (link)
In Kafka, messages are not consumed, they are rotated based on storage utilization or based on how long the message is persisted. (link)
Kafka only offers persistent topics (link)
Vikram: [high volumes and low latency]
In the words of the project web page, “Apache Kafka is an open-source distributed event streaming platform” (link)