Apache Parquet for modern .NET
Clone or download
klukasz and aloneguid Update ProquetConvert to take IEnumerable (#328)
* ProquetConvert takes IEnumerable

Fixes
Changing parameters to be IEnumerable.

Issue #

Description
It makes ProquetConvert more flexible when input data comes from big db table, or file

* Update ProquetConvert to take IEnumerable - tests

I've udpated test to reflect recent change. I aslo set test batch size small to have few batches.
Latest commit c144190 Sep 12, 2018

README.md

Apache Parquet for .Net Platform

Icon

This documenation is for a Release Candiate of Parquet.Net v3 which is a complete redesign of Parquet library and is not compatible with V2. To see documentation for the latest stable version please follow this link.

Note that Elastacloud provides commercial support for Parquet.Net, therefore if you need any professional advise or speedy development of new features and bugfixes please write to parquetsupport@elastacloud.com.

Status

NuGet Build status

Fully managed .NET library to read and write Apache Parquet files. Supports:

  • .NET 4.5 and up.
  • .NET Standard 1.4 and up (for those who are in a tank that means it supports .NET Core (all versions) implicitly)

Runs on all flavors of Windows, Linux, and mobile devices (iOS, Android) via Xamarin

Why

Parquet library is mostly available for Java, C++ and Python, which somewhat limits .NET/C# platform in big data applications. Whereas C# is a beautiful language (C# is just Java done right) working on all platforms and devices, we still don't have anything good in this area.

This project is aimed to fix this problem. We support all the popular server and client operating systems, mobile devices, gaming consoles and everything that can run .NET which is quite a lot!

Index

You can track the amount of features we have implemented so far.

Related Projects

Download Parquet Viewer from Windows 10 store:

Get it on Windows 10

Getting started

Parquet.Net is redistributed as a NuGet package. All the code is managed and doesn't have any native dependencies, therefore you are ready to go after referencing the package. This also means the library works on Windows, Linux and MacOS X.

General

This intro is covering only basic use cases. Parquet format is more complicated when it comes to complex types like structures, lists, maps and arrays, therefore you should read this page if you are planning to use them.

Reading files

In order to read a parquet file you need to open a stream first. Due to the fact that Parquet utilises file seeking extensively, the input stream must be readable and seekable. You cannot stream parquet data! This somewhat limits the amount of streaming you can do, for instance you can't read a parquet file from a network stream as we need to jump around it, therefore you have to download it locally to disk and then open.

For instance, to read a file c:\test.parquet you would normally write the following code:

using System.Collections.Generic;
using System.IO;
using System.Linq;
using Parquet.Data;

// open file stream
using (Stream fileStream = System.IO.File.OpenRead("c:\\test.parquet"))
{
   // open parquet file reader
   using (var parquetReader = new ParquetReader(fileStream))
   {
      // get file schema (available straight after opening parquet reader)
      // however, get only data fields as only they contain data values
      DataField[] dataFields = parquetReader.Schema.GetDataFields();

      // enumerate through row groups in this file
      for(int i = 0; i < parquetReader.RowGroupCount; i++)
      {
         // create row group reader
         using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(i))
         {
            // read all columns inside each row group (you have an option to read only
            // required columns if you need to.
            DataColumn[] columns = dataFields.Select(groupReader.ReadColumn).ToArray();

            // get first column, for instance
            DataColumn firstColumn = columns[0];

            // .Data member contains a typed array of column data you can cast to the type of the column
            Array data = firstColumn.Data;
            int[] ids = (int[])data;
         }
      }
   }
}

Writing files

Parquet.Net operates on streams, therefore you need to create it first. The following example shows how to create a file on disk with two columns - id and city.

//create data columns with schema metadata and the data you need
var idColumn = new DataColumn(
   new DataField<int>("id"),
   new int[] { 1, 2 });

var cityColumn = new DataColumn(
   new DataField<string>("city"),
   new string[] { "London", "Derby" });

// create file schema
var schema = new Schema(idColumn.Field, cityColumn.Field);

using (Stream fileStream = System.IO.File.OpenWrite("c:\\test.parquet"))
{
   using (var parquetWriter = new ParquetWriter(schema, fileStream))
   {
      // create a new row group in the file
      using (ParquetRowGroupWriter groupWriter = parquetWriter.CreateRowGroup(2))
      {
         groupWriter.WriteColumn(idColumn);
         groupWriter.WriteColumn(cityColumn);
      }
   }
}

License

Parquet.Net is licensed under the MIT license.

Contributing

We are desparately looking for new contributors to this projects. It's getting a lot of good use in small to large organisations, however parquet format is complicated and we're out of resources to fix all the issues.

For details on how to start see this guide. If you are a developer who is interested in Parquet development please read this guide