Problems running a producer / consumer setup using cppkafka #103
Assignees
Comments
|
hey @mcraveiro thanks for the detailed report. this does look like some issue with dns / ipv6 and there definitely are some lingering usability issues here. it looks like what might be happening is that an ipv6 address is being cached some where. could you post your redpanda config, and are you using a single node setup? also, please try using a new group and topic when switching over to the explicit 127.0.0.1 -- it might something ipv6-ish getting cached in the metadata. |
|
looks related to #99 |
|
@mcraveiro - can you run this tomorrow. We are hoping to update the rpk container in a few or early tom. the alternatively, you can remove this from all the configurations |
|
@mcraveiro fyi #100 just landed. it will be in the packages later this week |
|
Thanks very much for your prompt reply. I'll go through the different suggestions. ConfigMy config is unchanged from the package: Suggestion 1: Different topic and different groupThis does not seem to have fixed the issue. Producer: Consumer: Broker: No new messages in the broker. Suggestion 2: Remove API entries from configUpdates to config: root@lovelace:/etc/redpanda# diff -u redpanda.yaml redpanda.yaml.bak
--- redpanda.yaml.bak 2020-11-17 08:28:16.591878991 +0000
+++ redpanda.yaml 2020-11-17 08:28:59.083592953 +0000
@@ -10,15 +10,15 @@
advertised_kafka_api:
address: ""
port: 0
- advertised_rpc_api:
- address: ""
- port: 0
+# advertised_rpc_api:
+# address: ""
+# port: 0
auto_create_topics_enabled: true
data_directory: /var/lib/redpanda/data
developer_mode: false
- kafka_api:
- address: 0.0.0.0
- port: 9092
+# kafka_api:
+# address: 0.0.0.0
+# port: 9092
kafka_api_tls:
cert_file: ""
enabled: falseStopped and restarted server, but failed to restart: Server logs: Tried a couple of restarts but seems to fail with the same error. Reverting the changes resulted in a happy broker again, but I still get the same error. Many thanks for your time. |
|
The |
|
Thanks @0x5d, I will wait for this to hit the DEBs and then try again. |
|
@mcraveiro - yeah, the configuration for IP's must be correct once we bootstrap the raft quorum you'd have to perform a metadata update, etc. if it's just for testing remove but also we just released fixes for this. https://github.com/vectorizedio/redpanda/releases/latest can you test again and close if it fixes your problem. |
|
@mcraveiro if you already have it installed on your computer you can just do |
|
Thanks very much for all your tips. So, I trashed my data directory as mentioned: I did an upgrade - couldn't quite get the command @senior7515 supplied to work, but the following incantation did the trick: I then restarted the producer: And the consumer does look happier but not quite all sorted I don't think: I tried a few groups and different topics but always with the same result. The server logs: |
|
@mcraveiro did you create the topic or enable auto create ? https://vectorized.io/documentation/advanced-configuration/ see |
|
It appears the config that comes with the DEB sets it to true: |
|
But now you say that, it seems that is precisely the problem: Producer: Consumer: So that looks good but then it goes quiet. Now, I would have expected a lot of data to start flooding in :-) but perhaps these test applications are not very verbose. I shall look into the source code and see if it should indeed be this quiet. At any rate, thanks very much, it seems I am almost there now. |
|
Ah, silly me, it listens on Consumer: So I guess the only remaining question then is the automatic topic creation. |
|
@mcraveiro thanks for the update, this is looking good. one reason topic auto creation may not be working is that the client is setting of course, we may also have a bug related to topic creation. checking your client might be the easiest first step to debugging this before digging further. |
|
@mcraveiro resolving issue. if metadata is a bug, let's open another one. thank you. |
|
Thanks a lot guys, I'll look at the client for that option. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi Red Panda developers,
Thanks very much for your amazing contribution to FOSS. Following on from the HN announcement [1], I am trying to play around with Red Panda. First, I apologise in advance as I am not an experienced Kafka user so I am certain I am doing something wrong, but not quite sure what. I am using Debian Testing and I've installed your DEB packages as well as the distro-supplied
librdkafka[2] (version 1.51 on Debian). I then checked out and compiledcppkafka[3]. The Red Panda logs appear OK to the untrained eye:I then started the producer. It errored first but then succeeded:
However, when I try to start the consumer I get an error:
I tried a few variations such as using
127.0.0.1instead oflocalhostbecause I thought it could be an IPv6 issue, but to no avail:I also tried variations of groups such as 1, etc. I'll keep playing with it, but any pointers as to what I am doing wrong would be greatly appreciated.
Cheers
Marco
[1] https://news.ycombinator.com/item?id=25075739
[2] https://github.com/edenhill/librdkafka
[3] https://github.com/mfontanini/cppkafka
The text was updated successfully, but these errors were encountered: