Add embedded Druid cluster tests #18147

kfaraz · 2025-06-16T15:16:21Z

Summary

⏳ Run integration tests without the need to mvn build, create a Docker image or even run startup scripts
🚀 Just write a JUnit @Test and run it from your IDE! 🎉 🎉
🔡 Run tests against a fully functioning Druid cluster running in a single JVM
🔧 Complete control over all aspects of the Druid services being tested
🎩 Load any combination of Druid extensions
🐞 Debug the test and the services using debug points and the web-console!
📦 Get embedded ZooKeeper + Derby metadata store by default (non embedded versions are supported too)

Explore

💥 Run the EmbeddedKafkaClusterMetricsTest to see a simulation in action:

Start a Druid cluster which uses the KafkaEmitter to publish cluster metrics to a topic
Post a KafkaSupervisor to ingest the metrics back into the same Druid cluster! 🐶 🍖
Put debug points in the test to keep the web-console running and explore the metrics!

Usage

✏️ Write a JUnit 5 test class that extends EmbeddedClusterTestBase
🔩 Define your cluster in the method createCluster()
🧪 Write a @Test method
🚀 Run it!

Define a cluster

EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
cluster.add(new EmbeddedOverlord());
cluster.add(new EmbeddedIndexer());

Add an extension module

cluster.addExtension(KafkaIndexTaskModule.class);

Requirements:

Module must extend DruidModule
Module (and corresponding META-INF/services file) must be on the classpath of the test
- EITHER add a dependency to the corresponding maven module containing (e.g. maven module for KafkaIndexTaskModule is druid-kafka-indexing-service)
- OR add your test in the same maven module

Example test:

druid/extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/simulate/EmbeddedKafkaSupervisorTest.java

Lines 63 to 78 in 5a8dcd5

    
           public EmbeddedDruidCluster createCluster() 
        
           { 
        
             final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper(); 
        
             kafkaServer = new EmbeddedKafkaServer(cluster.getZookeeper(), cluster.getTestFolder(), Map.of()); 
        
             cluster.addExtension(KafkaIndexTaskModule.class) 
        
                    .addResource(kafkaServer) 
        
                    .useLatchableEmitter() 
        
                    .addServer(new EmbeddedCoordinator()) 
        
                    .addServer(overlord) 
        
                    .addServer(indexer) 
        
                    .addServer(broker); 
        
             return cluster; 
        
           }

Add common properties

cluster.addCommonProperty("druid.monitoring.emissionPeriod", "PT1S");

These properties correspond to the file common.runtime.properties that are applied on all the Druid services.

Add service-specific properties

EmbeddedOverlord overlord = new EmbeddedOverlord();
overlord.addProperty("druid.manager.segments.useIncrementalCache", "always");

These properties override the common runtime properties and correspond to the service-specific runtime.properties files.

Add a cluster-managed resource

cluster.addResource(new EmbeddedZookeeper());

Resources are

anything outside Druid services used by the cluster during its lifecycle.
started and stopped with the cluster.
implementations of the interface EmbeddedResource.

e.g. EmbeddedKafkaServer, EmbeddedZookeeper, InMemoryDerbyResource

Motivation

Druid ITs can be cumbersome to write, run and debug. They are very resource intensive , require multiple steps even to verify a simple change and are difficult to get right.

Owing to this barrier in writing new tests, Druid IT coverage of certain areas has been low.

This patch aims to address this issue by introducing a new framework to run lightweight integration tests using a fully-functioning Druid cluster running in a single process.

Goals

Simulation tests aim to:

🦸🏻 Increase developer productivity
🏗️ Fill the test coverage gap between class-level unit tests and docker-based integration tests.
✅ Simplify testing out new features on a full Druid cluster.
🍰 Easily define different cluster setups in the test itself without needing to rely on a bunch of config files.
🔧 Provide complete control over all aspects of the service(s) being tested.
🧵 Test scalability and concurrency handling in critical pieces like TaskQueue, TaskLockbox, TaskRunner, etc.
🐞 Debug failures quickly (a typical cluster takes < 1s to start and can then be used for any number of tests).
🐎 Discover race conditions due to the fast nature of the test (already discovered some while putting this up 🙂 ).
😈 Use real Druid services and no mocked dependencies.

Non-goals

Embedded cluster tests are not meant to:

Replace docker-based ITs (standard or revised)
Test running a multi-process Druid cluster
Test Druid docker image

Design

Run required Druid services in an embedded mode, that internally launch the corresponding Cli* runnable
Each Druid service is sandboxed and has its own separate Guice injector
No override of default Guice dependencies provided by the Cli* runnables
No mocks
Write out task logs and segments to temp directory
Choose extensions to load via config druid.extensions.modulesForEmbeddedTest
Add custom resources to the Druid cluster (e.g. EmbeddedKafkaServer, EmbeddedZookeeper)

Changes to review in core Druid

Wire up JettyServerInitializer properly so that multiple Jetty servers may be bound correctly within the same JVM
- See JettyServerInitUtils.getGuiceFilterHolder
Add new test-only config druid.extensions.modulesForEmbeddedTest which takes in a list of module class names that
should be loaded in a simulation test
Minor fix in SeekableStreamTaskRunner to allow taskDuration to be less than 1 minute
Add method OverlordClient.postSupervisor to hit API POST /druid/indexer/v1/supervisor
Update methods in test classes TestDerbyConnector and CuratorTestBase to have public access

New classes to review

EmbeddedDruidCluster: defines a Druid cluster used in an embedded cluster test
EmbeddedResource: defines any resource with a lifecycle that can be used in an EmbeddedDruidCluster
EmbeddedDruidServer that can provide a DruidServerResource with a lifecycle
EmbeddedZookeeper: uses the existing CuratorTestBase to run an embedded zk server
EmbeddedKafkaServer: an embedded Kafka server mostly copying code from TestBroker utility class. We can merge these two in a follow up PR.
JUnit 5 base test class EmbeddedClusterTestBase
LatchableEmitter to wait for metric events using countdown latches rather than repeatedly polling Druid APIs

This PR has:

abhishekagarwal87 · 2025-06-17T09:02:35Z

Great work @kfaraz. How do extensions get enabled/disabled in these embedded tests? Are those supported yet?

kfaraz · 2025-06-17T09:17:52Z

Thanks, @abhishekagarwal87 !

Edit:
Using extensions is already supported.

We can load any combination of extension modules as long as they are on the class path. Since these are plain UTs, that simply means adding a maven dependency to the respective extension module.
Required modules should be specified in the cluster definition. These are then populated in druid.extensions.modulesForSimulation (as opposed to production which uses druid.extensions.loadList).
We can also add custom resources to an embedded cluster such as an EmbeddedKafkaServer

A test in this PR is already using the Kafka extension as a proof-of-concept.

The syntax would be something like:

EmbeddedKafkaServer kafkaServer = new EmbeddedKafkaServer();
EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withExtensions(List.of(KafkaIndexTaskModule.class))
                                                   .addServer(new EmbeddedCoordinator())
                                                   .addServer(new EmbeddedIndexer())
                                                   .addServer(new EmbeddedOverlord())
                                                   .addResource(kafkaServer);

To nicely wire up the dependencies and the SDK, I am currently evaluating two alternatives described below.

I am leaning towards B. What are your thoughts?

Option A: Keep the test framework in a separate module `simulation-tests`

Currently implemented in this PR
All simulation tests would live in this module itself
When we want to write a simulation test for an extension, say Kinesis indexing service,
simulation-tests module would need to depend on that extension.

Pros:

The framework, embedded resources for extensions and tests all live in one module.
Embedded clusters can use any mix of extensions (e.g. say MySQL metadata store + Kafka ingestion)

Option B: Keep the test framework in `services/test` package

This is highest level Druid core module containing the Cli* runnables
Write simulations for core Druid in services/test itself
Extensions can then depend on services/test and use the SDK to define their own cluster resources
e.g. Kafka server (embedded or otherwise) and run tests using these resources

Pros:

More maintainable in the long run
Keep the simulation tests in the same package as the code being tested
Simulations will be added organically as a part of feature development
Simulations count towards Jacoco coverage for extensions
Clean maven dependencies

Cons:

Embedded cluster would be able to use only one extension at a time since extensions don't seem
to be able to depend on each other
Possible remedy: Add a module that depends on multiple extensions. Simulation tests that use multiple extensions would have to live in this module. A module which may already serve this purpose is integration-tests.

...tests/src/test/java/org/apache/druid/testing/simulate/embedded/DruidServerJunitResource.java

simulation-tests/src/test/java/org/apache/druid/testing/simulate/embedded/EmbeddedOverlord.java

kgyrtkirk · 2025-06-17T10:11:06Z

...tion-tests/src/test/java/org/apache/druid/testing/simulate/embedded/EmbeddedDruidServer.java

+    @Inject
+    CoordinatorClient coordinator;


I don't really understand why this injection will happen - an instance of it is only present as a private field of this class - could you give a hint? :)

Guice takes care of since we have bound it using .toInstance() in the method getInitModules().

binder.bind(ServiceClientHolder.class).toInstance(clientHolder)

interesting; I wasn't expecting that during a toInstance call that will be interpreted; somehow I was thinking guice have to be the one creating the original instance...makes sense ; but I still find it a little bit odd :)
I won't forget this trick :D

simulation-tests/src/test/java/org/apache/druid/testing/simulate/embedded/EmbeddedBroker.java

kfaraz · 2025-06-17T10:45:52Z

@kgyrtkirk , thanks for the suggestions! Will try to address them.
Also, please let me know if you have any thoughts on #18147 (comment).

…tests

kgyrtkirk · 2025-06-18T10:23:08Z

regarding comment

I would really like the separate module - but placing these tests where they should belong will be more in front of someone working on that part...so I think its better to put it inside an existing module - it might be possible to create a separate source set for it; but that would only be a logical isolation (probably with no real benefits)

we could have a module which loads multiple (or all) extensions so that more complicated scenarios could be tested....actually that was the reason I created quidem-ut - maybe we could rename that module so that the multi-extension simulation tests could live there along with the qtest stuff?

kfaraz · 2025-06-18T10:43:44Z

I would really like the separate module - but placing these tests where they should belong will be more in front of someone working on that part...so I think its better to put it inside an existing module - it might be possible to create a separate source set for it; but that would only be a logical isolation (probably with no real benefits)

Thanks for the feedback, @kgyrtkirk ! I feel the same.

we could have a module which loads multiple (or all) extensions so that more complicated scenarios could be tested....actually that was the reason I created quidem-ut - maybe we could rename that module so that the multi-extension simulation tests could live there along with the qtest stuff?

Yeah, that would be nice! I originally thought we could put it into integration-tests module but the quidem UT module is more closely related. We could just call that module simulation-tests since quidem UTs are also essentially a type of simulation.

...n-tests/src/test/java/org/apache/druid/testing/simulate/indexing/batch/IndexTaskSimTest.java

...s/src/test/java/org/apache/druid/testing/simulate/indexing/kafka/KafkaSupervisorSimTest.java

...g-service/src/test/java/org/apache/druid/indexing/kafka/simulate/KafkaSupervisorSimTest.java

quidem-ut/src/test/java/org/apache/druid/indexing/IndexTaskSimTest.java

uds5501 · 2025-06-24T05:14:17Z

...g-service/src/test/java/org/apache/druid/indexing/kafka/simulate/KafkaSupervisorSimTest.java

+
+    // Get the task ID
+    List<TaskStatusPlus> taskStatuses = ImmutableList.copyOf(
+        getResult(cluster.leaderOverlord().taskStatuses(null, dataSource, 1))


Question - For this TC, their are 2 partitions, so number of tasks should be 2 as well right?

Or we are using a single task for demonstration purpose?

The requirement is Number of tasks <= Number of partitions.
A single task can read from any number of partitions.
We launch more tasks to increase ingestion throughput.

uds5501 · 2025-06-24T05:20:10Z

...g-service/src/test/java/org/apache/druid/indexing/kafka/simulate/KafkaSupervisorSimTest.java

+    );
+  }
+
+  private List<ProducerRecord<byte[], byte[]>> generateRecordsForTopic(


Suggestion: There should be a provision to produce records for each partition

…tests

kgyrtkirk · 2025-06-25T08:48:52Z

services/src/test/java/org/apache/druid/testing/simulate/EmbeddedDruidServer.java

+ * This class and most of its methods are kept package protected as they are used
+ * only by the specific server implementations in the same package.
+ */
+public abstract class EmbeddedDruidServer implements ServerReferencesProvider


since almost every class is called EmbeddedX ; I feel like it would probably be an option to call these "embedded tests" ?

Yeah, that could work too 👍🏻

I had originally set out to write a simulation framework (that only emulated the behaviour of a cluster and didn't need to be an actual Druid cluster) but it ended up being a single-process embedded cluster instead.

As for the name, the word "simulation" stuck and it also made for a small and ~~easy~~ catchy suffix for test names like XyzSimTest, so I kept it anyway. But I am not opposed to calling it something else either.

my 2¢: to me the word "simulation" is misleading given the final design- I don't believe anything is really being simulated. IMO using "embedded" across the board is better. I suggest searching for "simulate" or "simulation" and adjusting them so we can keep the wording consistent. As for XyzSimTest, could be XyzEmbedTest.

Sounds good, will make the change.

quidem-ut/src/test/java/org/apache/druid/simulate/indexing/KafkaIngestSelfMetricsSimTest.java

…tests

gianm

Great work. I've always wished for a testing system like this.

gianm · 2025-06-26T14:26:09Z

quidem-ut/src/test/java/org/apache/druid/simulate/indexing/IndexTaskSimTest.java

+/**
+ * Simulation tests for batch {@link IndexTask} using inline datasources.
+ */
+public class IndexTaskSimTest extends IndexingSimulationTestBase


quidem-ut is now not really an appropriate name for this module, as these tests aren't using quidem. I don't think you need to change it in this PR necessarily, as there's already a lot of other changes and I don't want to get into a debate over the name in this PR 😄. A follow up to change the name would be useful.

Sounds good. 👍🏻

I could just put these tests in a separate module, if that seems better?

Might as well be a separate module named embedded-tests. The main reason I say that is I am struggling to think of a good name for the combined quidem + embedded test module.

That works!

I guess quidem tests are also essentially embedded tests. It's just that they don't use a full cluster but only a Broker-like setup.

I am not sure how much value it would add, but we could try piping quidem queries through an embedded cluster itself in the future. If we ever do that, we could have just the one module, embedded-tests.

gianm · 2025-06-26T14:29:14Z

services/src/test/java/org/apache/druid/testing/simulate/junit5/IndexingSimulationTestBase.java

+import java.util.Map;
+
+/**
+ * Base class for simulation tests related to ingestion and indexing.


IMO would be better to have this be a collection of static utility methods. The reason is that if we ever add another kind of "test base" with different kinds of convenience functions, a single test class won't be able to use both sets of convenience functions, due to the impossibility of multiple inheritance. The presence of runSql in this class is a hint that something is off (that isn't an indexing API).

I was wondering if these methods shouldn't just go into the EmbeddedDruidCluster class itself, given that they are always used to interact with a specific cluster.

That would also work. It would make the calling code look nicer.

The main downside I can think of is that at some point, there may be a lot of functions and the EmbeddedDruidCluster class could get pretty long. But if that happens, there are ways to address it. For example we could split the functions out into default methods in interfaces that extend DruidClusterApi and have EmbeddedDruidCluster implement DruidClusterApi and multiple of those other interfaces. Please don't do that now though. I'm just thinking through how it might evolve in the future. For now I think adding the methods directly to EmbeddedDruidCluster is good.

Thanks for the suggestion! Ended up doing something similar.

Renamed the class IndexingSimulationTestBase to EmbeddedClusterApis

Kept some methods like createTaskFromPayload() and createTestDatasourceName() as static

Created an instance of EmbeddedClusterApis for each EmbeddedDruidCluster

Tests can now invoke methods as follows:

// general usage pattern cluster.callApi().onLeaderOverlord(o -> o.runTask(taskId, task)); cluster.callApi().waitForTaskToSucceed(taskId); // shorthand for SQL cluster.runSql(...);

gianm · 2025-06-26T15:56:11Z

...xing-service/src/main/java/org/apache/druid/indexing/overlord/duty/UnusedSegmentsKiller.java

+      );
+      return new DutySchedule(
+          killConfig.getDutyPeriod().toStandardDuration().getMillis(),
+          Duration.standardSeconds(10).getMillis()


Is it intentional that this was changed this from 1 minute to 10 seconds?

Yes, it helps with the tests finishing faster and doesn't really affect a prod setup.
I could also just pass this value as a config too for consistency and let the default be 1 minute.

If it doesn't really affect a prod setup then it's imo ok to leave it like this.

Ended up changing this slightly since I realized that it is possible that an Overlord has just come up and has been elected leader but the segment metadata cache has still not warmed up (if operating in mode ifSynced).

The unused segment killer performs better with the cache fully synced.

So using the following values which feel good enough for all cases. We can revisit them later if needed.

duty period = 1 hour,
initial delay = duty period / 4 = 15 mins

gianm · 2025-06-26T16:03:37Z

server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java

@@ -679,10 +679,10 @@ private void pollSegments()

    stopwatch.stop();
    emitMetric("segment/poll/time", stopwatch.millisElapsed());
-    log.info(
+    /*log.info(


This seems unintentional; please restore the original code if so.

gianm · 2025-06-26T16:08:44Z

server/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorReport.java

@@ -30,7 +31,12 @@ public class SupervisorReport<T>
  private final DateTime generationTime;
  private final T payload;

-  public SupervisorReport(String id, DateTime generationTime, T payload)
+  @JsonCreator


I haven't yet found the place where this deserializer is used or tested- where is that? I wanted to check something about its usage: when there's a totally-generic thing like T as a property, Jackson doesn't know what T to use unless a TypeReference is provided. So it's important that the deserialization be done in that specific way.

Ah, thanks for catching this. I had intended to use this but it is not being used anymore. Will revert this.

gianm · 2025-06-26T16:10:24Z

services/src/test/java/org/apache/druid/testing/simulate/EmbeddedMiddleManager.java

+  public EmbeddedMiddleManager()
+  {
+    // Don't sync lookups as cluster might not have a Coordinator
+    // addProperty("druid.lookup.enableLookupSyncOnStartup", "false");


Please don't include commented-out code.

gianm · 2025-06-26T16:18:19Z

services/src/test/java/org/apache/druid/testing/simulate/EmbeddedDruidServer.java

+ * This class and most of its methods are kept package protected as they are used
+ * only by the specific server implementations in the same package.
+ */
+public abstract class EmbeddedDruidServer implements ServerReferencesProvider


my 2¢: to me the word "simulation" is misleading given the final design- I don't believe anything is really being simulated. IMO using "embedded" across the board is better. I suggest searching for "simulate" or "simulation" and adjusting them so we can keep the wording consistent. As for XyzSimTest, could be XyzEmbedTest.

gianm · 2025-06-26T16:26:08Z

services/src/test/java/org/apache/druid/testing/simulate/EmbeddedZookeeper.java

+
+/**
+ * {@link EmbeddedResource} for an embedded zookeeper cluster that can be used
+ * as a JUnit Rule in simulation tests.


Is this comment about "JUnit Rule" correct?

No, will clean it up.

kfaraz · 2025-06-26T16:55:34Z

Thanks for the feedback, @gianm and for catching the leftover superfluous changes!
I will address your comments and take another pass to ensure that no extra changes have made their way into this PR.

...test/java/org/apache/druid/testing/embedded/indexing/KafkaIngestSelfMetricsEmbeddedTest.java

…tests

...rc/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedKafkaClusterMetricsTest.java

+    final int numSegments = Integer.parseInt(
+        cluster.runSql("SELECT COUNT(*) FROM sys.segments WHERE datasource = '%s'", dataSource)
+    );


...rc/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedKafkaClusterMetricsTest.java

+    final int numRows = Integer.parseInt(
+        cluster.runSql("SELECT COUNT(*) FROM %s", dataSource)
+    );


...rc/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedKafkaClusterMetricsTest.java

+        InlineSchemaDataSourceCompactionConfig
+            .builder()
+            .forDataSource(dataSource)
+            .withSkipOffsetFromLatest(Period.seconds(0))
+            .withMaxRowsPerSegment(compactedMaxRowsPerSegment)


...rc/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedKafkaClusterMetricsTest.java

+    final int expectedValueForSegmentsAssigned = (int) Double.parseDouble(
+        cluster.runSql(
+            "SELECT COUNT(*) FROM %s WHERE metric = '%s' AND host = '%s' AND service = '%s'",
+            dataSource, metricName, server.selfNode().getHostAndPort(), server.selfNode().getServiceName()
+        )
+    );


gianm

LGTM

kfaraz · 2025-06-28T01:17:16Z

Thanks a lot for the review, @gianm !

Add simulation tests

7715b92

github-actions bot added Area - Dependencies Area - Ingestion labels Jun 16, 2025

kfaraz added 3 commits June 16, 2025 21:00

Revert extra change to HttpRemoteTaskRunner

c8af0ae

Remove extra changes

3c0211d

Add OverlordClient test

bdbd0f6

kfaraz changed the title ~~[WIP] Add Druid cluster simulation tests~~ [WIP] Add Druid cluster simulations (docker-less mini integration tests) Jun 17, 2025

kfaraz requested a review from gianm June 17, 2025 05:24

kgyrtkirk reviewed Jun 17, 2025

View reviewed changes

simulation-tests/src/test/java/org/apache/druid/testing/simulate/embedded/EmbeddedBroker.java Outdated Show resolved Hide resolved

kfaraz mentioned this pull request Jun 18, 2025

Add kttm-nested dataset and some usage sessions of it #16867

Merged

kfaraz added 3 commits June 18, 2025 11:08

Clean up interfaces, run with JUnit 5

91a6951

Merge branch 'master' of github.com:apache/druid into add_simulation_…

682e32d

…tests

Fix compilation

bfa6d9a

Fix imports, forbidden APIs etc

6242955

github-advanced-security bot found potential problems Jun 18, 2025

View reviewed changes

Reorganize classes

1629235

github-actions bot added the Area - Streaming Ingestion label Jun 18, 2025

github-advanced-security bot found potential problems Jun 18, 2025

View reviewed changes

kfaraz added 2 commits June 18, 2025 19:56

Fix dependencies and compilation

4da28de

Add test extension for in-memory Derby, add test coverage

1c5349f

github-actions bot added Area - Batch Ingestion Area - Querying Area - MSQ labels Jun 19, 2025

Remove system.out

a53ab42

uds5501 reviewed Jun 24, 2025

View reviewed changes

kfaraz mentioned this pull request Jun 24, 2025

Route task logs on Indexers to dedicated task log files #18170

Merged

10 tasks

kfaraz added 2 commits June 25, 2025 08:44

Merge branch 'master' of github.com:apache/druid into add_simulation_…

fef9233

…tests

Speed up Kafka metrics test

f937b6e

kgyrtkirk reviewed Jun 25, 2025

View reviewed changes

kfaraz added 2 commits June 25, 2025 21:06

Add concurrent compaction and embedded kill simulation test

f0cd9c3

Add client tests for coverage

ce83512

github-advanced-security bot found potential problems Jun 25, 2025

View reviewed changes

quidem-ut/src/test/java/org/apache/druid/simulate/indexing/KafkaIngestSelfMetricsSimTest.java Fixed Show fixed Hide fixed

kfaraz added 3 commits June 25, 2025 23:12

Add an optional EmbeddedMiddleManager

6dfc69d

Fix failure

6952ffb

Merge branch 'master' of github.com:apache/druid into add_simulation_…

7a4ddd9

…tests

gianm reviewed Jun 26, 2025

View reviewed changes

kfaraz changed the title ~~Add Druid cluster simulations (docker-less mini integration tests)~~ Add embedded Druid cluster tests Jun 26, 2025

kfaraz added 2 commits June 27, 2025 13:00

Move new tests to module embedded-tests

8e7d407

Try to trigger the new tests

3fe47aa

github-advanced-security bot found potential problems Jun 27, 2025

View reviewed changes

kfaraz added 4 commits June 27, 2025 17:30

Merge branch 'master' of github.com:apache/druid into add_simulation_…

8c51de6

…tests

Remove unused dependencies

569416a

Merge branch 'master' of github.com:apache/druid into add_simulation_…

fbbfe1c

…tests

Rename tests to follow pattern Embedded*Test

5a8dcd5

github-advanced-security bot found potential problems Jun 27, 2025

View reviewed changes

kfaraz added 3 commits June 27, 2025 19:14

Remove unused test dependencies

6060f0b

Fix dependencies

7c15366

Minor cleanup, exclude new module from jacoco

894ec69

gianm approved these changes Jun 27, 2025

View reviewed changes

kfaraz merged commit b3cdf6b into apache:master Jun 28, 2025
206 of 209 checks passed

capistrant added this to the 34.0.0 milestone Jul 22, 2025

	public EmbeddedDruidCluster createCluster()
	{
	final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();

	kafkaServer = new EmbeddedKafkaServer(cluster.getZookeeper(), cluster.getTestFolder(), Map.of());

	cluster.addExtension(KafkaIndexTaskModule.class)
	.addResource(kafkaServer)
	.useLatchableEmitter()
	.addServer(new EmbeddedCoordinator())
	.addServer(overlord)
	.addServer(indexer)
	.addServer(broker);

	return cluster;
	}

Add embedded Druid cluster tests #18147

Add embedded Druid cluster tests #18147

Uh oh!

Conversation

kfaraz commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Explore

Usage

Define a cluster

Add an extension module

Add common properties

Add service-specific properties

Add a cluster-managed resource

Motivation

Goals

Non-goals

Design

Changes to review in core Druid

New classes to review

Uh oh!

abhishekagarwal87 commented Jun 17, 2025

Uh oh!

kfaraz commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Option A: Keep the test framework in a separate module simulation-tests

Option B: Keep the test framework in services/test package

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kfaraz commented Jun 17, 2025

Uh oh!

kgyrtkirk commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kfaraz commented Jun 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

gianm Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz commented Jun 16, 2025 •

edited

Loading

kfaraz commented Jun 17, 2025 •

edited

Loading

Option A: Keep the test framework in a separate module `simulation-tests`

Option B: Keep the test framework in `services/test` package

kfaraz Jun 17, 2025 •

edited

Loading

kgyrtkirk commented Jun 18, 2025 •

edited

Loading

kfaraz Jun 25, 2025 •

edited

Loading

gianm Jun 26, 2025 •

edited

Loading

kfaraz Jun 27, 2025 •

edited

Loading