Data Conveyer is toolkit and a lightweight transformation engine to facilitate real-time data migrations. It enables rapid implementation of data centric solutions and can apply to any .NET project targeting .NET Framework 4.5 or .NET Standard 2.0.
Data Conveyer can be added to a .NET project by simply installing its NuGet package.
Details on Data Conveyer features, usage samples, self-guided tutorial, etc. are available at Data Conveyer site.
A complete reference guide of the API exposed by Data Conveyer is available at Data Conveyer Help portal.
In order to add Data Conveyer to your .NET project, it is recommended use the "Manage NuGet Packages..." option in Visual Studio, (available upon right-clicking on the project in the Solution Explorer window).
Data Conveyer NuGet package can also be added using Package Manager:
Install-Package DataConveyer -Version n.n.nor .NET CLI:
dotnet add package DataConveyer --version n.n.n
, where n.n.n represents the version number (e.g. 3.0.1).
Alternatively, if you prefer to build the Data Conveyer yourself, you can fork this repository and clone it onto your local machine.
Once Data Conveyer is added to your project, its use typically involves these 3 steps:
- Configure Data Conveyer's orchestrator by defining input/output data types, transformation functions, etc.
- Launch Data Conveyer's process by calling the
ExecuteAsyncmethod. - Evaluate
ProcessResultreturned by Data Conveyer.
The following code snippet represents a simple Data Conveyer's usage:
using Mavidian.DataConveyer;
using Mavidian.DataConveyer.Orchestrators;
var config = new OrchestratorConfig()
{
InputDataKind = KindOfTextData.Delimited,
InputFileName = "input.csv",
HeadersInFirstInputRow = true,
OutputDataKind = KindOfTextData.Keyword,
OutputFileName = "output.kw"
};
ProcessResult rslt;
using (var orchtr = OrchestratorCreator.GetEtlOrchestrator(config))
{
rslt = orchtr.ExecuteAsync().Result;
}
if (rslt.CompletionStatus == CompletionStatus.IntakeDepleted)
{
Console.WriteLine($"Successfully converted {rslt.RowsWritten} records!");
}For additional examples on using Data Conveyer, please examine the Data Conveyer tutorial or the Data Conveyer Primer section of the Data Conveyer Help portal.
The DataConveyer solution consist of the following projects:
- DataConveyer A class library that implements all Data Conveyer's functionality.
- DataConveyer.Tests Tests using xUnit.NET framework.
- DataConveyer_doc Sandcastle documentation project (its output is Data Conveyer Help portal.
- DataConveyer_tests Tests using MSTest framework.
- TextDataHandler A simple console application that executes sample Data Conveyer's processes.
You will notice 2 separate test projects. This is for historical reasons. Initial tests were developed using the MSTest framework. Due to the need for additional functionality (e.g. test repeats), some of the tests were subsequently ported to xUnit.NET framework. The idea is that at some point in the future only xUnit.NET tests will remain, so that the DataConveyer_tests project can be removed.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
Copyright © 2019-2021 Mavidian Technologies Limited Liability Company. All Rights Reserved.