Skip to content

Latest commit

 

History

History
84 lines (66 loc) · 4.57 KB

installation.md

File metadata and controls

84 lines (66 loc) · 4.57 KB
Raw
sidebar_position
2

Installation & Execution

How it works

No actual installation is necessary. Just add a NuGet reference to the core package into a .NET project. On top of it, add any NuGet reference to the necessary extensions packages depending on what is needed.

The core package is Paillave.EtlNet.Core that must be referenced into anything that deals with ETL.NET. Other packages Paillave.EtlNet.* are extensions.

These are the types of extensions:

  • Operators for extra processes, input or output
  • Extra behavior related to the runtime

Once references are added, the ETL process can be described in .NET and run anywhere in the application.

ETL.NET packages

NuGet Package Type Purpose
Paillave.EtlNet.Core Core Contains the streaming mechanism, the runtime, and every essential operator that can be expected in any regular ETL
Paillave.EtlNet.Autofac Runtime extension Some operators like Entity Framework Core extension or Sql Server extension may need some resources to work like a connection. This works using Dependency Injection. ETL.NET offers a very primitive DI implementation out of the box that can be replaced by this autofac wrapper implementation
Paillave.EtlNet.Dropbox Input/Output Read or write file directly on dropbox
Paillave.EtlNet.EntityFrameworkCore Input/Output Bulkload, save, upsert, read, make lookups using Entity Framework Core context
Paillave.EtlNet.ExcelFile Input/Output Read or write Excel files tables
Paillave.EtlNet.ExecutionToolkit Runtime extension Visualization of processes and libraries to make a console application dedicated to ETL.NET executions
Paillave.EtlNet.FileSystem Input/Output Read or write files on the local file system
Paillave.EtlNet.FromConfigurationConnectors Runtime extension Provides connectors to the runtime using a configuration file
Paillave.EtlNet.Ftp Input/Output Read or write files on FTP or FTPS
Paillave.EtlNet.Mail Input/Output Send emails, or read emails attached files from SMTP folders
Paillave.EtlNet.Sftp Input/Output Read or write files on SFTP
Paillave.EtlNet.SqlServer Input/Output Upsert, read into Sql Server directly using drivers
Paillave.EtlNet.TextFile Input/Output Parse or create text files in csv (separated or fixed size columns)
Paillave.EtlNet.Bloomberg Input/Output Parse bloomberg response files. Note: creating a request file is not implemented at the moment
Paillave.EtlNet.XmlFile Input/Output Parse XML files. Note: writing XML is not implemented at the moment
Paillave.EtlNet.Zip Input/Output Read files from zipped files. Note: creating a zip file is not implemented at the moment
Paillave.EtlNet.Pdf Input/Output Read pdf files, by telling the structure of paragraphs.
Paillave.EtlNet.Scheduler Input/Output Submit ticks depending based on cron setups.

Definition of the ETL process and its execution

The definition will be done in a method that receives the trigger stream as a parameter.

The execution will be done by calling the runtime.

Traces can be handled by a process defined using the stream of events given by the runtime.

Here is the simplest way to define and trigger an ETL that starts using a string parameter from the command line:

using System;
using System.Threading.Tasks;
using Paillave.Etl.Core;

namespace MyEtlApplication
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var processRunner = StreamProcessRunner.Create<string>(DefineProcess);
            processRunner.DebugNodeStream += (sender, e) => { /* PLACE A CONDITIONAL BREAKPOINT HERE FOR DEBUG ex: e.NodeName == "parse file" */ };
            var executionOptions = new ExecutionOptions<string> { TraceProcessDefinition = DefineTraceProcess };
            var res = await processRunner.ExecuteAsync(args[0], executionOptions);
            Console.Write(res.Failed ? "Failed" : "Succeeded");
        }
        private static void DefineProcess(ISingleStream<string> contextStream)
        {
            // TODO: Define the ETL process here
        }
        private static void DefineTraceProcess(IStream<TraceEvent> traceStream, ISingleStream<string> contentStream)
        {
            // TODO: Define the ETL process to handle traces here
        }
    }
}

Learn more

Learn about the most essential features, and get it touch with common patterns and recipes.