Parquet.Net

Status

Fully managed .NET library to read and write Apache Parquet files. Supports:

• .NET 4.5.1 and up.
• .NET Standard 1.4 and up.

Runs on all flavors of Windows, Linux, and mobile devices (iOS, Android) via Xamarin

Why

Parquet library is mostly available for Java, C++ and Python, which somewhat limits .NET/C# platform in big data applications. Whereas C# is a beautiful language (C# is just Java done right) working on all platforms and devices, we still don't have anything good in this area.

This project is aimed to fix this problem. We support all the popular server and client operating systems, mobile devices, gaming consoles and everything that can run .NET which is quite a lot!

Index

• Reading Data
• Writing Data
• DataSet
• Declaring Schema
  • Supported Types
• Advanced Types
  • Structures
  • Maps
  • Lists
• Performance Tuning
• JSON Support

You can track the amount of features we have implemented so far.

Related Projects

• Apache Parquet viewer for Windows 10.
• Azure Data Lake Analytics Integration.

Download Parquet Viewer from Windows 10 store:

Getting started

Parquet.Net is redistributed as a NuGet package. All the code is managed and doesn't have any native dependencies, therefore you are ready to go after referencing the package. This also means the library works on Windows, Linux and MacOS X.

Reading files

In order to read a parquet file you need to open a stream first. Due to the fact that Parquet utilises file seeking extensively, the input stream must be readable and seekable. This somewhat limits the amount of streaming you can do, for instance you can't read a parquet file from a network stream as we need to jump around it, therefore you have to download it locally to disk and then open.

For instance, to read a file c:\test.parquet you woudl normally write the following code

using System.IO;
using Parquet;
using Parquet.Data;

using(Stream fs = File.OpenRead("c:\\test.parquet"))
{
	using(var reader = new ParquetReader(fs))
	{
		DataSet ds = reader.Read();
	}
}

this will read entire file in memory as a set of rows inside DataSet class.

Writing files

Parquet.Net operates on streams, therefore you need to create it first. The following example shows how to create a file on disk with two columns - id and city.

using System.IO;
using Parquet;
using Parquet.Data;

var ds = new DataSet(
	new DataField<int>("id"),
	new DataField<string>("city")
);

ds.Add(1, "London");
ds.Add(2, "Derby");

using(Stream fileStream = File.OpenWrite("c:\\test.parquet"))
{
	using(var writer = new ParquetWriter(fileStream))
	{
		writer.Write(ds);
	}
}

Tools

parq

Parq is a .NET runtime (written for Windows but will run elsewhere) that brings tooling for inspecting Parquet files to developers.

It is a command line utility held in a different repository, for which you can find out more by reading this guide.

License

Parquet.Net is licensed under the MIT license.

Contributing

We are desparately looking for new contributors to this projects. It's getting a lot of good use in small to large organisations, however parquet format is complicated and we're out of resources to fix all the issues.

For details on how to start see this guide. If you are a developer who is interested in Parquet development please read this guide

If you need to reference the latest preview version (published on every commit) please use the following NuGet Feed: https://ci.appveyor.com/nuget/parquet-dotnet-nvq9lkymhrek