A Data Plane SDK for .NET. This SDK provides components for creating .NET-based data planes that interface with Control Planes via the Data Plane Signaling API (DPS API). The SDK includes callbacks on API events, transactional persistence and mutual authentication and authorization scaffolding.
- Dataplane SDK .NET
This SDK is compiled against net9.0 so consuming applications must be upgraded to that as well.
To install the SDK, add the following packages to your .NET app:
- install the project's NuGet feed
https://nuget.pkg.github.com/metaform/index.json( see details) dotnet add package DataPlane.DataPlane.Sdk.Api --version X.Y.Zfor the API extensions sdotnet add package DataPlane.Sdk.Core --version X.Y.Zfor the SDK core, can be omitted ifDataPlane.Sdk.Apiis used
Note that while the DataPlane.Sdk.Api package is not strictly required, it handles all incoming DPS API communication,
so it
should only be omitted if a custom API implementation is used. See this chapter for details.
The SDK is currently hosted on GitHub's NuGet feed, which requires authorization!
The best way to see how the SDK used is to check out one of the Samples.
In a typical .NET/ASP.NET Core application that uses .NET's native dependency injection mechanism, the recommended way is to register all SDK-related services with the DI container, preferably in an extension method to keep things clean. Let's write a simple extension method:
using DataPlane.Sdk.Core;
using DataPlane.Sdk.Core.Data;
using DataPlane.Sdk.Core.Domain.Messages;
using DataPlane.Sdk.Core.Domain.Model;
using Void = DataPlane.Sdk.Core.Domain.Void;
namespace MyProject;
public static class MyExtensions
{
public static void AddDataPlaneSdk(this IServiceCollection services)
{
// initialize and configure the DataPlaneSdk
var config = configuration.GetSection("DataPlaneSdk").Get<DataPlaneSdkOptions>() ?? throw new ArgumentException("Configuration invalid!");
var sdk = new DataPlaneSdk
{
DataFlowStore = () => DataFlowContextFactory.CreatePostgres(configuration, config.RuntimeId),
RuntimeId = config.RuntimeId,
OnStart = f => StatusResult<DataFlowResponseMessage>.Success(new DataFlowResponseMessage { DataAddress = f.Destination }),
OnRecover = _ => StatusResult<Void>.Success(default),
OnTerminate = _ => StatusResult<Void>.Success(default),
OnSuspend = _ => StatusResult<Void>.Success(default),
OnPrepare = f =>
{
f.State = DataFlowState.Prepared;
return StatusResult<DataFlow>.Success(f);
}
};
}
//... more init code
}There are several noteworthy things going on here:
- Binding the application config (
appsettings[.*].json) to a configuration object (see this chapter for details) - Registering API callbacks: these are invoked when respective DPS API requests are received (see this chapter for details)
- Initialization of the PostgreSQL-based data storage (see this chapter for details)
The DataPlane SDK integrates well with .NET's dependency injection mechanism, and to make the most of that, its services
are registered with the DI container (the IHost):
// add SDK core services
var dataplaneConfig = configuration.GetSection("DataPlaneSdk");
services.AddSdkServices(sdk, dataplaneConfig);The SDK takes care of registering all configuration objects and services it needs internally.
The AddSdkServices extension method is provided by the SDK and registers SDK services like persistence, token
providers and API clients and configuration.
Next, we need to configure API authentication and authorization. The SDK does bring most of the scaffolding and glue code, but client apps still need to implement the following:
- API authentication logic: validating incoming auth tokens and their signatures
- decorating outgoing HTTP requests: this is needed when the data plane sends DPS or other HTTP requests to the control plane: an authorization token header must be added.
// wire up ASP.net authentication services
services.AddAuthentication(...)For a more comprehensive example that uses Keycloak as authentication provider, see the example project.
All resources that the DataPlane Signaling API are protected with access control. Please add the following line to your
Program.cs to enable authz:
services.AddSdkAuthorization();Omitting this will cause the DataPlane Signaling API to be unprotected!
This registers authorization handlers for all resource types, that reject any request, where the participantContextId
does not match the auth token's sub claim, for example:
/api/v1/participant123/dataflows/dataflowXYZandsub: participant123-> accepted, ifparticipant123ownsdataflowXYZ/api/v1/participant123/dataflows/dataflowXYZandsub: participant456-> rejected, ifparticipant123does not owndataflowXYZ
The data plane needs to send HTTP requests to the control plane on several occasions, for example, when sending asynchronous DPS messages, or to register and un-register the data plane with the control plane.
These requests must be authenticated, i.e. carry an Authorization: Bearer ey... header. Fortunately, the DataPlane SDK
handles this centrally using the ITokenProvider interface.
To configure this, add the following to your extension method or Program.cs:
services.AddSingleton<ITokenProvider, MyTokenProvider>();It is imperative to register the provider as a singleton so that the default (no-op) token provider from the SDK gets overwritten properly. The token provider's job is to get an access token from a third-party IdP such as KeyCloak. The specifics of that are beyond the scope of this document, but the following general sequence could be implemented:
public class MyTokenProvider(HttpClient httpClient) : ITokenProvider
{
public Task<string> GetTokenAsync()
{
var clientId = GetSecretFromVault("client_id");
var clientSecret = GetSecretFromVault("client_secret");
var tokenEndpoint = "http://identity.yourcompany.com/openid-connect/token";
var request = new HttpRequestMessage(HttpMethod.Post, _tokenEndpoint);
request.Content = new FormUrlEncodedContent(new Dictionary<string, string>
{
{ "grant_type", "client_credentials" },
{ "client_id", clientId },
{ "client_secret", clientSecret }
});
var response = await httpClient.SendAsync(request);
if (!response.IsSuccessStatusCode)
{
throw new Exception($"Token request failed: {response.StatusCode} - {await response.Content.ReadAsStringAsync()}");
}
var payload = await response.Content.ReadFromJsonAsync<TokenResponse>();
return payload?.AccessToken ?? throw new Exception("No access token returned");
}
private class TokenResponse
{
[JsonPropertyName("access_token")]
public string AccessToken { get; set; }
}
}To avoid conflicts and potential infinite loops during token generation, the token provider is only registered for a "
named" HttpClient (name = "SdkHttpClient"). As a general rule of thumb, client code should:
- use named
HttpClientobjects by usingIHttpClientFactory.CreateClient("SdkHttpClient")when making HTTP requests to the DataPlane Signaling Api, or other control plane APIs - use unnamed
HttpClientobjects when making arbitrary HTTP requests to external services, like an IdP or third-party APIs
In situations where the built-in API server for DataPlane Signaling cannot be used, it may be an option to use only the
DataPlane.Sdk.Core module. While this will forego all API controllers, authentication and authorization, it will still
provide
core services and persistence. To do that, add the DataPlane.Sdk.Core package to your .NET project:
dotnet add package DataPlane.Sdk.Core --version <VERSION>.
Depending on the type of project (console, webapi) an IHost may or may not be available. If it is, client code can
still utilize the dependency injection facilities built into the SDK by calling the AddSdkServices(sdk) extension
method.
The SDK should only be used in the "core-only" configuration in specific circumstances. In most cases the full SDK should be used.
The SDK makes use of .NET's configuration mechanism, specifically the appsettings.json that usually contains
application configuration.
We opted for combining all SDK-related configuration in one config object:
{
"DataPlaneSdk": {
"ControlApi": {
"BaseUrl": "http://localhost:8083/api/control"
},
"InstanceId": "test-dataplane-instance",
"RuntimeId": "example-lock-id",
"AllowedSourceTypes": [
"test-source-type"
],
"AllowedTransferTypes": [
"test-transfer-type"
]
}
}With the exception of the RuntimeId, which is optional, all entries are required, and omitting them will result in a
runtime exception.
ControlApi.BaseUrl: this is the base URL for the control plane's control API which is used to register and un-register this dataplaneInstanceId: this should be a unique ID which identifies this data plane. This is used during data plane registrationRuntimeId: an internal identifier that is used for various details such as database-level locking of entitiesAllowedSourceTypes: array of types of data sources that this data plane can handle. Influences the control plane's catalog.AllowedTransferTypes: array of types of transfer types that this data plane can handle. Influences the control plane's catalog.
If PostgreSQL persistence use used, the appsettings.json file must contain a connection string:
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=SdkApi;Username=postgres;Password=postgres"
}
}The Data Plane SDK defines several callbacks to intercept and influence DataPlane Signaling interactions. The callbacks should be registered when initializing the SDK.
When using SDK callbacks, users should keep in mind the following tenets:
- all SDK callbacks are invoked before objects are stored in persistence
- callbacks are always involved inside a transaction, i.e. before a call to
DbContext.SaveChanges[Async] - as a result, callbacks should not throw any exceptions, instead they should communicate any error using a
StatusResult
The Data Plane SDK uses the .NET EntityFramework (EF) for persistent storage, so switching between in-memory and actual database persistence is seamless.
In most .NET applications the DbContext is provided via dependency injection. While the SDK does use dependency
njection, it cannot require it because some applications might not use it. For this reason the DbContext is provided
via the factory pattern.
The entry point is the DataPlaneSdk class:
var sdk = new DataPlaneSdk
{
DataFlowStore = DataFlowContextFactory.CreatePostgres(configuration, config.RuntimeId),
// alternatively:
// DataFlowStore = DataFlowContextFactory.CreateInMem(config.RuntimeId)
// ...
}Note that the DbContext is still registered as a service in the DI container if the AddSdkServices(sdk) extension
method is invoked.
The Control API is a REST interface of the control plane, that can be used to register, un-register and delete data plane instances.
For convenience, the SDK offers the ControlApiService that encapsulates API
requests, authentication and authorization and
deserialization.
This service is intended to be used directly from client code, as the SDK does not invoke it on its own. It does, however, register it with the DI container.
For example:
DataPlaneSdkOptions config = ...;
var result = await controlService.RegisterDataPlane(new DataPlaneInstance(config.InstanceId)
{
Url = config.PublicUrl,
State = DataPlaneState.Available,
AllowedSourceTypes = config.AllowedSourceTypes,
AllowedTransferTypes = config.AllowedTransferTypes
});
if(result.IsFailed)
{
//handle error
}