NATS Aspire container #1175

mtmk · 2023-12-02T06:07:21Z

This PR is an effort to introduce NATS container and client support.

Implemented most of the features as much as I could gather from other implementations. There are a few open tasks as far as I can see which may be left for another PR:

NATS server configuration is very basic at the moment. I'm not sure how the manifest is going to reflect all the configuration options. Currently only option is to enable JetStream for local development. Also no clustering option is implemented.
Tracing is still in our backlog Support distributed tracing / OpenTelemetry nats-io/nats.net.v2#53
Same with metrics Add OTEL Metrics nats-io/nats.net.v2#316

src/Aspire.Hosting.Nats/Aspire.Hosting.Nats.csproj

oising · 2023-12-08T19:53:00Z

@mtmk I'm rebasing this on latest main to fix merge conflicts and update any broken integration points. Also updating to use centralized package management /cc @ReubenBond

stebet · 2024-01-13T09:53:59Z

Oooh exciting! Does it also have cluster/JetStream support?

mtmk · 2024-01-14T20:27:44Z

Oooh exciting! Does it also have cluster/JetStream support?

Not yet 😅 but I think you can add the argument yourself for now if you like:

var nats = builder.AddNatsContainer("nats-server").WithArgs("-js");

Edit: also added options to AddNatsContainer which essentially does the above, even though I'm not sure if that's the correct approach. (Really need to look at others to see how they've done it)

var nats = builder.AddNatsContainer("nats-server", enableJetStream = true);

# Conflicts: # Aspire.sln

# Conflicts: # Directory.Packages.props

mtmk · 2024-02-12T19:33:34Z

@oising do you think we're ready for review?

mtmk · 2024-02-12T19:38:22Z

@azure-pipelinesazure-pipelines
/ dotnet.aspire
src/Aspire.Hosting.Nats/Aspire.Hosting.Nats.csproj#L0
src/Aspire.Hosting.Nats/Aspire.Hosting.Nats.csproj(0,0): error NU1102: (NETCORE_ENGINEERING_TELEMETRY=Restore) Unable to find package NATS.Net with version (>= 2.1.0)

@mitchdenny, i'm not sure why ci is failing NATS.Net v2.1.0 in up on NuGet https://www.nuget.org/packages/NATS.Net/2.1.0

davidfowl · 2024-02-12T19:47:18Z

playground/nats/Nats.ApiService/Program.cs

@@ -0,0 +1,17 @@
+using NATS.Client.Core;


Can we make this app a little more real? Can it use pubsub as well?

sure, let me have a go!

I created a simple JetStream example since it's easier demonstrate persisted messages using Web API. As for pubsub I suppose we'd need a frontend using SignalR or Blazor. Maybe a simple chat or something?

playground/nats/Nats.ApiService/Program.cs

davidfowl · 2024-03-03T22:27:04Z

playground/nats/Nats.Backend/Program.cs

+        {
+            await foreach (var msg in nats.SubscribeAsync<AppEvent>("events.>", cancellationToken: _cts.Token).ConfigureAwait(false))
+            {
+                Console.WriteLine($"Processing event: {msg.Data}");


davidfowl · 2024-03-03T22:29:34Z

tests/Aspire.Hosting.Tests/Nats/AddNatsTests.cs

+
+        var manifest = await ManifestUtils.GetManifest(nats.Resource);
+
+        Assert.Equal("container.v0", manifest["type"]?.ToString());


@mtmk we made some changes here

aspire/tests/Aspire.Hosting.Tests/Kafka/AddKafkaTests.cs

Lines 75 to 95 in f6a2f9f

var manifest = await ManifestUtils.GetManifest(kafka.Resource);

var expectedManifest = """

{

"type": "container.v0",

"connectionString": "{kafka.bindings.tcp.host}:{kafka.bindings.tcp.port}",

"image": "confluentinc/confluent-local:7.6.0",

"env": {

"KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://localhost:29092,PLAINTEXT_HOST://localhost:9092"

},

"bindings": {

"tcp": {

"scheme": "tcp",

"protocol": "tcp",

"transport": "tcp",

"containerPort": 9092

}

}

}

""";

Assert.Equal(expectedManifest, manifest.ToString());

.

BTW I'm sorry that these changes must feel like a roller coaster 😄. FWIW, it's helping us land on solid patterns that seem to scale well 🚀

no worries, my fault really having the PR open so long. thanks for bearing with me 🙏 also loving to witness the evolution 😍

If you are interested in more evolution:

aspire/src/Aspire.Hosting/Redis/RedisBuilderExtensions.cs

Line 26 in 525c59a

.WithEndpoint(hostPort: port, containerPort: 6379, name: RedisResource.PrimaryEndpointName)

aspire/src/Aspire.Hosting/Redis/RedisResource.cs

Lines 16 to 19 in 525c59a

/// <summary>

/// Gets the primary endpoint for the Redis server.

/// </summary>

public EndpointReference PrimaryEndpoint => _primaryEndpoint ??= new(this, PrimaryEndpointName);

aspire/src/Aspire.Hosting/Redis/RedisResource.cs

Line 33 in 525c59a

return $"{PrimaryEndpoint.GetExpression(EndpointProperty.Host)}:{PrimaryEndpoint.GetExpression(EndpointProperty.Port)}";

aspire/src/Aspire.Hosting/Redis/RedisResource.cs

Line 63 in 525c59a

return $"{PrimaryEndpoint.Host}:{PrimaryEndpoint.Port}";

We're trying to get rid of all the manual string formatting and building up the object model to represent all kids of references. Makes it less error prone and brings consistency to the model.

eerhardt · 2024-03-04T21:24:00Z

src/Components/Aspire.Nats.Client/Aspire.Nats.Client.csproj

+    <PackageReference Include="Microsoft.Extensions.Configuration.Binder" />
+    <PackageReference Include="Microsoft.Extensions.Diagnostics.HealthChecks" />
+    <PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
+    <PackageReference Include="NATS.Net" />


Our naming rules are https://github.com/dotnet/aspire/tree/main/src/Components#naming.

The name of this component should be Aspire.NATS.Net to align with those rules.

eerhardt · 2024-03-04T21:24:32Z

playground/nats/Nats.AppHosting/Nats.AppHosting.csproj

@@ -0,0 +1,23 @@
+<Project Sdk="Microsoft.NET.Sdk">


This project and folder should be named Nats.AppHost.

eerhardt

Can you also update

tests/Aspire.Nats.Client.Tests/ConfigurationTests.cs

eerhardt · 2024-03-04T21:26:23Z

tests/Aspire.Nats.Client.Tests/ConformanceTests.cs

+    protected override void SetHealthCheck(NatsClientSettings options, bool enabled)
+        => throw new NotImplementedException();


We have health checks in the component. We should enable these tests.

eerhardt · 2024-03-04T21:28:00Z

src/Components/Aspire.Nats.Client/NatsHealthCheck.cs

+
+namespace Aspire.Nats.Client;
+
+public class NatsHealthCheck(INatsConnection connection) : IHealthCheck


This shouldn't be public.

eerhardt · 2024-03-04T21:31:16Z

src/Components/Aspire.Nats.Client/NatsHealthCheck.cs

+        return Task.FromResult(connection.ConnectionState == NatsConnectionState.Open
+            ? HealthCheckResult.Healthy()
+            : HealthCheckResult.Unhealthy());
+    }


https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/73abc7a094f9ef625b83455898a21d22d2fa6a82/src/HealthChecks.Nats/NatsHealthCheck.cs#L74-L83 has more complicated logic than this. Is this enough to be able to tell if the connection is healthy?

updated to be a bit more granular i.e. added degraded state. But description and stats only exist on the old client Nats.Client v1. One we're using here is v2 and it's a rewrite and most of that is not exposed yet.

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>

eerhardt · 2024-03-05T12:56:44Z

src/Components/Aspire.NATS.Net/ConfigurationSchema.json

+          "properties": {
+            "Net": {
+              "type": "object",
+              "properties": {


Are any of the underlying NatsOpts worth being able to be configured in the configuration? See some of the other components for an example on how we hook up the config binder.

Ex:

aspire/src/Components/Aspire.StackExchange.Redis/AspireRedisExtensions.cs

Lines 150 to 155 in 184b129

private static ConfigurationOptions BindToConfiguration(ConfigurationOptions options, IConfiguration configuration)

{

var configurationOptionsSection = configuration.GetSection("ConfigurationOptions");

configurationOptionsSection.Bind(options);

return options;

aspire/src/Components/Aspire.StackExchange.Redis/ConfigurationSchema.json

Lines 21 to 45 in 184b129

"ConfigurationOptions": {

"type": "object",

"properties": {

"AbortOnConnectFail": {

"type": "boolean",

"description": "Gets or sets whether connect/configuration timeouts should be explicitly notified via a TimeoutException."

},

"AllowAdmin": {

"type": "boolean",

"description": "Indicates whether admin operations should be allowed."

},

"AsyncTimeout": {

"type": "integer",

"description": "Specifies the time in milliseconds that the system should allow for asynchronous operations (defaults to SyncTimeout)."

},

"ChannelPrefix": {

"type": "object",

"properties": {

"UseImplicitAutoPattern": {

"type": "boolean",

"description": "Indicates whether channels should use 'StackExchange.Redis.RedisChannel.PatternMode.Auto' when no 'StackExchange.Redis.RedisChannel.PatternMode' is specified; this is enabled by default, but can be disabled to avoid unexpected wildcard scenarios."

}

},

"description": "Automatically encodes and decodes channels."

},

I shall look into this in another PR

# Conflicts: # Directory.Packages.props

sebastienros · 2024-03-07T01:35:01Z

Fixed conflicts, added manifest.json and renamed AddNats() to AddNatsClient to follow a recent change we did in other libraries to prevent conflict with the distributed app extension methods.

Tried the sample, worked for me, then using the Swagger I found an issue, it's not taking the custom json converter for TimeSpan properties in the config object, I believe it's an issue in SwaggerUI. (found domaindrivendev/Swashbuckle.AspNetCore#2500)

@mtmk if you are ok with my changes I'll merge it.

davidfowl · 2024-03-07T07:32:55Z

Thank you @mtmk ! I look forward to future contributions!

NATS Aspire container

a4dcc11

dotnet-issue-labeler bot added the area-components Issues pertaining to Aspire Component packages label Dec 2, 2023

danmoseley reviewed Dec 2, 2023

View reviewed changes

src/Aspire.Hosting.Nats/Aspire.Hosting.Nats.csproj Outdated Show resolved Hide resolved

Use central package management

135a213

joperezr added the community-contribution Indicates that the PR has been added by a community member label Dec 11, 2023

Merge branch 'main' into nats

2088f5b

mtmk mentioned this pull request Jan 8, 2024

Add OpenTelemetry Instrumentation nats-io/nats.net.v2#124

Merged

5 tasks

mitchdenny self-assigned this Jan 10, 2024

mtmk added 11 commits January 14, 2024 20:31

Add telemetry to defaults

7b9690f

Add JetStream option

732c8f1

Merge branch 'main' into nats

6c2d73c

# Conflicts: # Aspire.sln

Fixed build

56126f5

Merge branch 'main' into nats

35a8972

# Conflicts: # Directory.Packages.props

Fixed sln merge issue

35f22fe

Moved sample to playgrounds

7d94f67

sln fixes

27ec961

Updated playground nats

70987ad

Moved hosting bits

b5c2c37

Merge fix

28b2112

Fixed spelling

47d5268

davidfowl reviewed Feb 12, 2024

View reviewed changes

Slightly more involved NATS sample

4d2ca61

davidfowl reviewed Feb 12, 2024

View reviewed changes

playground/nats/Nats.ApiService/Program.cs Outdated Show resolved Hide resolved

mtmk added 2 commits February 12, 2024 21:19

NATS sample tweak

7916e6b

Rename to Aspire.Nats.Client and added config

27a8609

mtmk added 4 commits March 3, 2024 07:05

Merge remote-tracking branch 'upstream/main' into nats

a40a832

nats with image

1405623

nats image tweaks

8eb4d94

revert

4d488e2

davidfowl reviewed Mar 3, 2024

View reviewed changes

mtmk added 2 commits March 4, 2024 08:46

Merge remote-tracking branch 'upstream/main' into nats

4afdad8

nats manifest test

1a62a75

eerhardt reviewed Mar 4, 2024

View reviewed changes

mtmk and others added 8 commits March 5, 2024 05:26

Update tests/Aspire.Nats.Client.Tests/ConfigurationTests.cs

3606984

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>

rename nats AppHost

6eefa74

rename Aspire.NATS.Net

388bd85

nats health checks tweak

f31d49f

nats component test tweaks

2c10732

Merge remote-tracking branch 'upstream/main' into nats

f98cfa6

nats component docs

0eab699

nats resource string constants

1f773f9

eerhardt reviewed Mar 5, 2024

View reviewed changes

sebastienros added 3 commits March 6, 2024 17:03

Merge branch 'main' into nats

15ffa21

# Conflicts: # Directory.Packages.props

Add manifest

7836160

Rename AddNats to AddNatsClient

d005020

sebastienros approved these changes Mar 7, 2024

View reviewed changes

sebastienros merged commit 44cde5d into dotnet:main Mar 7, 2024
8 checks passed

mtmk deleted the nats branch March 7, 2024 07:24

eerhardt mentioned this pull request Mar 11, 2024

[Components] New NATS component article dotnet/docs-aspire#483

Closed

github-actions bot locked and limited conversation to collaborators Apr 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NATS Aspire container #1175

NATS Aspire container #1175

mtmk commented Dec 2, 2023 •

edited

oising commented Dec 8, 2023 •

edited

stebet commented Jan 13, 2024

mtmk commented Jan 14, 2024 •

edited

mtmk commented Feb 12, 2024

mtmk commented Feb 12, 2024

davidfowl Feb 12, 2024

mtmk Feb 12, 2024

mtmk Feb 13, 2024

davidfowl Mar 3, 2024

davidfowl Mar 3, 2024

mtmk Mar 4, 2024

davidfowl Mar 4, 2024

eerhardt Mar 4, 2024

eerhardt Mar 4, 2024

eerhardt left a comment

eerhardt Mar 4, 2024

eerhardt Mar 4, 2024

eerhardt Mar 4, 2024

mtmk Mar 5, 2024

eerhardt Mar 5, 2024

mtmk Mar 7, 2024

sebastienros commented Mar 7, 2024 •

edited

davidfowl commented Mar 7, 2024


		var manifest = await ManifestUtils.GetManifest(nats.Resource);

		Assert.Equal("container.v0", manifest["type"]?.ToString());

	var manifest = await ManifestUtils.GetManifest(kafka.Resource);

	var expectedManifest = """
	{
	"type": "container.v0",
	"connectionString": "{kafka.bindings.tcp.host}:{kafka.bindings.tcp.port}",
	"image": "confluentinc/confluent-local:7.6.0",
	"env": {
	"KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://localhost:29092,PLAINTEXT_HOST://localhost:9092"
	},
	"bindings": {
	"tcp": {
	"scheme": "tcp",
	"protocol": "tcp",
	"transport": "tcp",
	"containerPort": 9092
	}
	}
	}
	""";
	Assert.Equal(expectedManifest, manifest.ToString());

	/// <summary>
	/// Gets the primary endpoint for the Redis server.
	/// </summary>
	public EndpointReference PrimaryEndpoint => _primaryEndpoint ??= new(this, PrimaryEndpointName);

		protected override void SetHealthCheck(NatsClientSettings options, bool enabled)
		=> throw new NotImplementedException();


		namespace Aspire.Nats.Client;

		public class NatsHealthCheck(INatsConnection connection) : IHealthCheck

	private static ConfigurationOptions BindToConfiguration(ConfigurationOptions options, IConfiguration configuration)
	{
	var configurationOptionsSection = configuration.GetSection("ConfigurationOptions");
	configurationOptionsSection.Bind(options);

	return options;

	"ConfigurationOptions": {
	"type": "object",
	"properties": {
	"AbortOnConnectFail": {
	"type": "boolean",
	"description": "Gets or sets whether connect/configuration timeouts should be explicitly notified via a TimeoutException."
	},
	"AllowAdmin": {
	"type": "boolean",
	"description": "Indicates whether admin operations should be allowed."
	},
	"AsyncTimeout": {
	"type": "integer",
	"description": "Specifies the time in milliseconds that the system should allow for asynchronous operations (defaults to SyncTimeout)."
	},
	"ChannelPrefix": {
	"type": "object",
	"properties": {
	"UseImplicitAutoPattern": {
	"type": "boolean",
	"description": "Indicates whether channels should use 'StackExchange.Redis.RedisChannel.PatternMode.Auto' when no 'StackExchange.Redis.RedisChannel.PatternMode' is specified; this is enabled by default, but can be disabled to avoid unexpected wildcard scenarios."
	}
	},
	"description": "Automatically encodes and decodes channels."
	},

NATS Aspire container #1175

NATS Aspire container #1175

Conversation

mtmk commented Dec 2, 2023 • edited

oising commented Dec 8, 2023 • edited

stebet commented Jan 13, 2024

mtmk commented Jan 14, 2024 • edited

mtmk commented Feb 12, 2024

mtmk commented Feb 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eerhardt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebastienros commented Mar 7, 2024 • edited

davidfowl commented Mar 7, 2024

mtmk commented Dec 2, 2023 •

edited

oising commented Dec 8, 2023 •

edited

mtmk commented Jan 14, 2024 •

edited

sebastienros commented Mar 7, 2024 •

edited