-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
App crashes the first time it tries to subscribe to a kafka topic #426
Comments
This looks like a Kafka configuration issue. Subscribing to a nonexistent topic is supposed to fail in Kafka by default. However, like you said, you should be able to change that default behavior by enabling topic auto-creation. Can you make sure that the variable is set correctly? Depending on what Kafka Docker image you're using, it may have a different name. |
I 'll try some more then and report back. Thanks |
Well no luck...
As you can see from the logs the request is made but asynchronously. Anyway as a workaround I solved it by setting a If you could provide me a working combination of kafka image and config (like the one you use to test) I will be able to investigate this further. Otherwise go ahead and close this. :) Thanks for your time |
Here's the version: "3.7"
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 2181:2181
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_LISTENERS: PLAINTEXT://:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 Here's a tiny app I use for testing: import { KafkaConfig, KafkaMessage} from "kafkajs";
import { Workflow, WorkflowContext, Kafka, KafkaConsume } from '@dbos-inc/dbos-sdk';
const kafkaConfig: KafkaConfig = {
brokers: [`localhost:9092`]
}
@Kafka(kafkaConfig)
export class KafkaExample {
@KafkaConsume("dbos-topic")
@Workflow()
static async kafkaWorkflow(ctxt: WorkflowContext, topic: string, partition: number, message: KafkaMessage) {
ctxt.logger.info(`Message received: ${message.value?.toString()}`)
}
} |
Thanks. I had an app already. Just for reference here is my config that produced the error mentioned.
I tried the latest version and probably it all starts there. I will live with the zookeeper for a while... |
After some more digging this is what I found. I believe that a better approach is to let the developer declare the topics in some config and the app on init, before starting any consumers, should check the existence of the declared topics and create the missing ones. This can be easily achieved using |
After setting AUTO_CREATE_TOPICS to false the app keeps crashing in a loop. |
That's a good point! We can automatically create all decorated topics during startup if they don't already exist. |
If it is going to be implemented this way (reading from the decorators instead of a config) all the topic creation parameters should be available at the decorator level (maybe they are there and I missed them). As you have probably figured out already I have some time to spare these days so if you wish I could create a PR on this. |
I have to say that I would prefer the file based configuration solution. |
On second thought, I want to hold off on this for now. I think that:
DBOS works excellently as a Kafka consumer, but right now I don't think we should do Kafka cluster management. |
Great point there. I was thinking about it too. |
I just found out that there is an open PR at kafkajs that when merged will fix the issue that started this whole conversation. |
Nice find! |
Actually is more important tulios / kafkajs / issues / 1603 . |
docker-compose.txt You should see this. KafkaJSProtocolError: This server does not host this topic-partition |
Workaround for this particular issue until KafkaJS solves it: #434 |
Addressed in #434 until KafkaJS fixes the bug |
After adding the kafka integration to dbos-docker-boilerplate
When the app starts with newly created volumes (
docker compose down -v; docker compose up
)It encounters the following error
Debugging showed that the error is thrown at kafka.ts#L88 the first time it tries to subscribe
and the app exits runtime.ts#L54
Running
docker compose down; docker compose up
works fine after the first time.Setting the kafka param AUTO_CREATE_TOPIC=true did not change anything.
What did was adding this hack(?)
before initiating the consumer in
kafka.ts
I am no expert configuring kafka so it might just be a misconfiguration on my part.
The text was updated successfully, but these errors were encountered: