Add Schema Registry option to the Kafka hosting package.#13090
Add Schema Registry option to the Kafka hosting package.#13090TheRubble wants to merge 7 commits intomicrosoft:mainfrom
Conversation
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 13090Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 13090" |
dotnet-policy-service
bot
added
the
community-contribution
There was a problem hiding this comment.
Pull request overview
This PR adds Kafka Schema Registry support to the Aspire.Hosting.Kafka package, allowing developers to configure and orchestrate a Confluent Schema Registry container alongside their Kafka resources. The Schema Registry provides centralized schema management for Kafka topics.
Key changes:
- New
KafkaSchemaRegistryResourceclass andWithKafkaSchemaRegistryextension method for adding/configuring the schema registry container - Support for custom host port binding via
WithHostPortextension method - Documentation updates in README with usage examples
- Comprehensive test coverage for the new functionality
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Aspire.Hosting.Kafka/KafkaSchemaRegistryResource.cs | New resource class representing the Schema Registry container with connection string support |
| src/Aspire.Hosting.Kafka/KafkaBuilderExtensions.cs | Extension methods for adding and configuring the Schema Registry |
| src/Aspire.Hosting.Kafka/KafkaContainerImageTags.cs | Constants for Schema Registry container image and tag |
| src/Aspire.Hosting.Kafka/README.md | Documentation with usage example for associating Schema Registry with Kafka |
| tests/Aspire.Hosting.Kafka.Tests/AddKafkaTests.cs | Test coverage for Schema Registry resource creation and configuration |
| playground/kafka/KafkaBasic.AppHost/Program.cs | Sample demonstrating Schema Registry usage |
…builder and resource classes
| /// <param name="configureContainer">Configuration callback for KafkaUI container resource.</param> | ||
| /// <param name="containerName">The name of the container (Optional).</param> | ||
| /// <returns>A reference to the <see cref="IResourceBuilder{KafkaSchemaRegistryResource}"/>.</returns> | ||
| public static IResourceBuilder<KafkaSchemaRegistryResource> WithKafkaSchemaRegistry(this IResourceBuilder<KafkaServerResource> builder, string? containerName = null, Action<IResourceBuilder<KafkaSchemaRegistryResource>>? configureContainer = null) |
There was a problem hiding this comment.
~None of the other resources have a containerName. Why is this needed?`
Is this shared across all kafka resoures?
There was a problem hiding this comment.
Hi @davidfowl, probably me misunderstanding but the other resources do have a containerName
It's just at the end, I've moved it to be at the of the method args now as per the UI resource.
I'd say you'd want one registry per Kafka group generally and sharing one across multiple instances would be a more advanced use case inside the usual dev loop.
Hope I've answered your query.
There was a problem hiding this comment.
I see it now! The other resouces that look like this are a singleton shared across all resources. If this is 1:1 with the kafka instance then you'd change this style a bit.
There was a problem hiding this comment.
Reviewing the schema usage stuff it does appear that 1:1 is the right way to go... I thought about specifying the registry as a seperate package / resource but it requires Kafka during it's startup via the "SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS" environment variable. Which means there's a dependency between the two.
I've removed the check for the registry so you get one per kafka cluster, happy to take your advice / guidance though as I'm not sure of the agreed best practice on this topic.
…nd consistency, Actioned the PR comments around using the endpoint and not hardcoding and replacing with the primary endpoint.
…ing and adding support for multiple schema registries in builder configuration
|
@TheRubble You can do an end to end test with #13090 (comment). Try it out and see if it works well. |
| /// <param name="configureContainer">Configuration callback for KafkaUI container resource.</param> | ||
| /// <param name="containerName">The name of the container (Optional).</param> | ||
| /// <returns>A reference to the <see cref="IResourceBuilder{KafkaSchemaRegistryResource}"/>.</returns> | ||
| public static IResourceBuilder<KafkaSchemaRegistryResource> WithKafkaSchemaRegistry(this IResourceBuilder<KafkaServerResource> builder, Action<IResourceBuilder<KafkaSchemaRegistryResource>>? configureContainer = null, string? containerName = null) |
There was a problem hiding this comment.
If kafka UI is registered, we should add a reference to it. I currently do it like this:
kafkaServer.WithKafkaUI(e =>
{
e.WithEnvironment(context =>
context.EnvironmentVariables["KAFKA_CLUSTERS_0_SCHEMAREGISTRY"] =
$"http://schema-registry:{schemaRegistryPort}");
});This is assuming container name is schema-registry. Can be easily replaced with an endpoint reference.
Only problem would be multiple schema registry registrations. I'm not using multiple schema registries, but we would probably need to keep track of the instances similarly to what is being done here:
There was a problem hiding this comment.
Looking at the KafkaUI implementation, I think it would make sense to do it in the WithKafkaUI method.
Description
This pull request introduces support for adding a Kafka Schema Registry resource alongside existing Kafka resources in the Aspire hosting framework. The main changes include new APIs for adding, configuring, and referencing a Schema Registry container, updates to documentation, and new tests to ensure correct behavior.
Schema Registry Support:
KafkaSchemaRegistryResourceclass to represent the schema registry container, including support for connection strings and endpoint management.WithKafkaSchemaRegistryextension method onIResourceBuilder<KafkaServerResource>to add and configure a schema registry container, including image/tag selection, environment variables, health checks, and host port binding.WithHostPortextension method forKafkaSchemaRegistryResourceto allow specifying a fixed host port for the schema registry container.KafkaContainerImageTags.Sample and Documentation Updates:
Program.cs) to demonstrate adding and referencing a schema registry resource, and updated the README with usage instructions and code samples for associating a schema registry with Kafka. [1] [2]Testing:
Checklist
<remarks />and<code />elements on your triple slash comments?doc-ideatemplatebreaking-changetemplatediagnostictemplate