Test containers for redpanda #2122
Replies: 9 comments 31 replies
-
|
Nice! Happy to help, and also here are a few things that would help simplifying the implementation:
|
Beta Was this translation helpful? Give feedback.
-
|
Totally. cc: @twmb - we should add a --dev to rpk which bundles a bunch of flags. for 2) we' are working on changing up the configuration to a single global config with hot reloading. but honestly don't see why not. this is a good idea. |
Beta Was this translation helpful? Give feedback.
-
|
btw @bsideup does testcontainers have a bounty program. Would be great to give some oss dev incentive to contribute a redpanda container for all langs. |
Beta Was this translation helpful? Give feedback.
-
|
not yet, but we're seriously considering adding it 👍 |
Beta Was this translation helpful? Give feedback.
-
|
@bsideup re: 2, I see that you're using the alter configs script -- are you asking for incremental config altering for brokers to be added to rpk? |
Beta Was this translation helpful? Give feedback.
-
|
@twmb I guess either the hot reloading of config or a command would do - we just need some way to make Redpanda return passed value instead of |
Beta Was this translation helpful? Give feedback.
-
hey @senior7515 just curious. is there any update on this? |
Beta Was this translation helpful? Give feedback.
-
|
@yottabytt yup; lots of PRs for centralized config. when that lands, setting listening addr/etc after the fact will be easy. it is in active development. the fundamental issue is few broker-level dynamic configs, so we are fixing it at the root - look for the PRs related to centralized config. |
Beta Was this translation helpful? Give feedback.
-
|
btw @yottabytt - for completeness, central config landed. |
Beta Was this translation helpful? Give feedback.
-
|
An example working implementation that I have been using to build my own https://github.com/quarkusio/quarkus/blob/48f0d28fbf978f54ce757e3836a7731259ec26c0/extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/DevServicesKafkaProcessor.java#L342 |
Beta Was this translation helpful? Give feedback.
-
|
This is tracked in #4671, which we will get to in 1 to 2 mo (currently estimated) |
Beta Was this translation helpful? Give feedback.
-
|
I've put in place very simple take on redpanda testcontainers, its something along this line https://gist.github.com/milenkovicm/09786b8dd70722e71e19e678bca555d3 It is just great way of testing kafka application as you dont have to bring zk container. Two major problems I faced while implementing (and they can be due to my limited experience with redpanda)
It would be great if somebody can point me in the direction what would be the easiest way of changing advertised port without restarting rp. |
Beta Was this translation helpful? Give feedback.
-
|
@senior7515 last question, when freash instance is started, first metadata query (for committed offsets) fails, with error "no partition coordinator". The second one will be successful, after few seconds. Is this expected behaviour? |
Beta Was this translation helpful? Give feedback.
-
|
@milenkovicm - yup, the kafka client has retries mechanisms for these. This is a retryable error code. All is saying is that redpanda hasn't reached quorum. @milenkovicm how can we make this better for your integration? What is the ideal scenario we should expose. Maybe we come up w/ a new command line flag for container-CI like setups. |
Beta Was this translation helpful? Give feedback.
-
|
cc: @twmb and @r-vasquez - dev experience feedback for us. |
Beta Was this translation helpful? Give feedback.
-
|
@senior7515 if the case, then I guess, it would be good to have it documented somewhere. I just haven't expected that it will fail for a topic with single partition and no replication, especially after write to that topic has been successful. Just to summarise steps:
|
Beta Was this translation helpful? Give feedback.
-
|
@milenkovicm can you file a bug w/ these steps. thank you! 🙌🏽 |
Beta Was this translation helpful? Give feedback.
-
|
@jcsp - now that we have centralized configuration, I think that we can let test-containers drive redpanda, rather than have test-containers discover redpanda. as in, test containers can simply tell redpanda the IP/port reconfiguration rather than for redpanda to listen wdyt ? |
Beta Was this translation helpful? Give feedback.
-
|
with your help, red panda rust testcontainer https://github.com/milenkovicm/testcontainers-redpanda-rs |
Beta Was this translation helpful? Give feedback.
-
|
@senior7515 @twmb should we use https://gist.github.com/bsideup/21762b4089a244e6fbfcfc606e7f0394, for testing repanda via testcontainers? |
Beta Was this translation helpful? Give feedback.
-
|
@sparrowV what version are you using. try 22.1.3 or higher. this should be a warning which is fine. you are probalby runnin on ubuntu. |
Beta Was this translation helpful? Give feedback.
-
|
great. Btw, any plans to merge https://gist.github.com/bsideup/21762b4089a244e6fbfcfc606e7f0394 to testcontainers repo? Also some values(image,port) are hardocoded,maybe make them more dynamic? |
Beta Was this translation helpful? Give feedback.
-
|
@sparrowV we chatted with the redpanda team about the "dev" flag that we would be happy to use from a Testcontainers module. Also, the values you see there are "internal" ports and hosts. Testcontainers will instruct Docker to use random ports, so these are not "hardcoded". The image is of course will be a parameter, in the Gist it is just for the demonstration purpose 👍 |
Beta Was this translation helpful? Give feedback.
-
|
@r-vasquez - let's add an issue for |
Beta Was this translation helpful? Give feedback.
-
|
Created: #5208 |
Beta Was this translation helpful? Give feedback.
-
|
Created PR: #5628 for the --dev flag in redpanda start.
Cc: @bsideup |
Beta Was this translation helpful? Give feedback.
-
Adding some notes here from @bsideup
https://gist.github.com/bsideup/21762b4089a244e6fbfcfc606e7f0394
Beta Was this translation helpful? Give feedback.
All reactions