LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer. The utility can pull source data from a local folder or from a storage container. LightIngest is most useful when you want to ingest a large amount of data, because there is no time constraint on ingestion duration. It's also useful when you want to later query records according to the time they were created, and not the time they were ingested.
Binaries are found under each release in release section. They are standalone and as such have no prerequisites, simply download the one tergeting your operating system and run:
LightIngest.exe "https://ingest-{Cluster name and region}.kusto.windows.net;Fed=True" -db:{Database} -table:{table} -source:"https://{Account}.blob.core.windows.net/{ROOT_CONTAINER};{StorageAccountKey}" -creationTimePattern:"'historicalvalues'yyyyMMdd'.parquet'" -pattern:"*.parquet" -format:parquet -limit:2 -cr:10.0LightIngest is published as a dotnet tool from this feed Dotnet tools require .Net SDK >= 6.0 installed and run installation command:
dotnet tool install --global Microsoft.Azure.Kusto.LightIngest --version 12.0.0-preview.1
To run the tool simply use its name:
LightIngest "https://ingest-{Cluster name and region}.kusto.windows.net;Fed=True" -db:{Database} -table:{table} -source:"https://{Account}.blob.core.windows.net/{ROOT_CONTAINER};{StorageAccountKey}" -creationTimePattern:"'historicalvalues'yyyyMMdd'.parquet'" -pattern:"*.parquet" -format:parquet -limit:2 -cr:10.0
See the full documentation here
This repo contains a command-line project for ingesting to Azure Data Explorer
This tool is also available as a package on nuget.org
The code is using the [ADX C# SDK](package on nuget.org)
There are many ways to contribute to the project.
- Submit bugs
- Review the source code changes.
- Microsoft Documentation
- Stack Overflow - Ask questions about how to use Kusto. Start posts with 'LightIngest'. This is monitored by Kusto team members.
- User Voice - Suggest new features or changes to existing features.
- Azure Data Explorer - Give feedback or report problems using the user feedback button (top-right near settings).
- Azure Support - Report problems with the Kusto service.
- Open an issue here - for problems specifically with this library.
This project has adopted the Microsoft Open Source Code of Conduct.
Resources:
- Microsoft Open Source Code of Conduct
- Microsoft Code of Conduct FAQ
- Contact opencode@microsoft.com with questions or concerns