Multiple listeners support : Confluent Schema-Registry with redpanda #98
Comments
raphaelauv
changed the title
|
thanks @raphaelauv for the report. We'll take a look. cc: @dotnwat |
|
@raphaelauv thanks for the detailed report. the good news is that this appears to be a networking issue. i was able to at least start the schema registry and get it connected to a redpanda instance not running inside docker: there are two issues at play here
I'll update this ticket when we've resolved these issues. In the mean time, I was able to get things working by running redpanda outside of docker, and then pointing the schema registery container at localhost via network_mode: host. |
|
@raphaelauv #100 just landed which fixes this issue. |
|
@raphaelauv fixed in the latest release https://github.com/vectorizedio/redpanda/releases/latest let us know if you run into issues again. |
|
Hi thanks for your reply. I just tried by pulling the latest docker image and it's still not working , did you try my docker-compose example ? Thank you again I tried with all theses different possibilities : SCHEMA_REGISTRY_BOOTSTRAP_SERVERS: PLAINTEXT://kafka-broker:29092
SCHEMA_REGISTRY_BOOTSTRAP_SERVERS: kafka-broker:29092
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka-broker:29092
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka-broker:29092 |
|
@raphaelauv thanks for testing again! sorry, this ticket was closed too early. there are a couple issues that we are working on to resolve this. first, we have a networking issue that is preventing the advertised server from being properly configured in this docker-compose scenario. follow #125 should be a pretty quick fix. i was able to circumvent this with host-based networking as a quick test. unfortunately the schema registry unearthed an issue with our fetch request compatibility. we have two pull request being tested right now that addresses this which you can follow here #25 and #126. all these fixes should drop in the next day or two. i'll update this ticket then. |
|
I pulled the latest container for redpanda still the same error |
|
@raphaelauv can you use host networking as above? We're still working on configuration via env vars. I've raised an issue for supporting multiple listeners. |
|
Thank for the hack ( host networking ) but multiple listeners is a necessary feature for my kafka stack. I will wait until it's ready. |
|
thanks @raphaelauv we'll update when its done. turned out to be more work than we anticipated, but i'd say early next week. |
|
Multiple listeners went in on #186 . Will tag here w/ release. |
|
Hi, I just tried my example ( from first post ) with the release v21.2.2 , it's not working |
raphaelauv
changed the title
|
We don't support env vars yet, so when you start Redpanda, you'll need to pass some command line flags: You might be able to get away with: |
|
@raphaelauv what @BenPope said :) We have a few users using confluent schema registry successfully now: |
|
@raphaelauv this works |
|
@senior7515 thank you but I just copy paste your example on a new docker-compose.yml file and then "docker-compose up -d" schema-registry give me : are you sure this work on your side ? Thanks |
|
Do you have the logs for Redpanda? |
|
|
Looks like the configuraton didn't work: kafka_api:
address: 0.0.0.0
port: 9092Needs to be: kafka_api:
- address: 0.0.0.0
name: plaintext
port: 29092
- address: 0.0.0.0
name: outside
port: 9092 |
|
OK, so the problem is: It looks like it tries to update the |
|
I've reopened #524 The easiest fix for now is to mount a configuration file that doesn't have kafka_api set: ./redpanda.yaml: redpanda:
auto_create_topics_enabled: true
node_id: 1
rpk:
enable_usage_stats: true |
|
Just stumbled upon this issue while I was trying to set up containerized redpanda to be accessible to consumers on the local machine. For anyone who needs a temporary fix in docker-compose form, here it is (assuming you have the kafka:
image: 'vectorized/redpanda:latest'
command: [redpanda, start, --kafka-addr, "PLAINTEXT://0.0.0.0:9092,OUTSIDE://0.0.0.0:29092", --advertise-kafka-addr, "PLAINTEXT://kafka:9092,OUTSIDE://localhost:29092"]
ports:
- 9092:9092
- 29092:29092
volumes:
- type: bind
source: ./redpanda.yml
target: /etc/redpanda/redpanda.yaml |
Looks great for a single broker, thanks for this. One caveat: Looks like host networking is required for OUTSIDE, so won't work on MacOS. |
|
Posting this here as progress - i tested w/ 21.3.4: version: '3.8'
services:
kafka-broker:
image: vectorized/redpanda:v21.3.4
ports:
- 9092:9092
- 29092:29092
entrypoint: [/usr/bin/rpk, redpanda, start, --kafka-addr, "PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092", --advertise-kafka-addr, "PLAINTEXT://kafka-broker:29092,OUTSIDE://localhost:9092"]
schema-registry:
image: confluentinc/cp-schema-registry:6.1.0
hostname: schema-registry
depends_on:
- kafka-broker
ports:
- 8081:8081
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka-broker:29092
and got: # docker-compose up -d
Creating network "schemaregistry_default" with the default driver
Creating schemaregistry_kafka-broker_1 ... done
Creating schemaregistry_schema-registry_1 ... done
# curl -X GET http://localhost:8081/schemas/ids/1
{"error_code":40403,"message":"Schema 1 not found"}%
# curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\": \"string\"}"}' \
http://localhost:8081/subjects/Kafka-value/versions
{"id":1}%
# curl -X GET http://localhost:8081/schemas/ids/1
{"schema":"\"string\""}% |
|
@0x5d can you close this ticket nwothat multiple listeners is in |
|
I also added a ticket + example of docker compose in here #931 |
|
Hi There, Need small help on configuring these values in helm kafka_api:
The default values.yaml doesn't contain internal. Kindly help in configuring the above values in helm chart. |
I want to use the confluent schema-registry with redpanda , I'm trying with the confluent docker image and I don't know what to set for :
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL
From this blog post https://www.confluent.fr/blog/how-to-prepare-for-kip-500-kafka-zookeeper-removal-guide/ , we should replace
kafkastore.connection.url by kafkastore.bootstrap.servers
So SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL become SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
https://docs.confluent.io/current/installation/docker/config-reference.html#schema-registry-configuration
Docker-compose configuration :
The shema-registry container logs :
and Redpanda logs :
Thank you
The text was updated successfully, but these errors were encountered: