Python

lance_nd stores xarray datasets in a Lance table, one dataset per row. Open your NetCDF files with xarray and write each dataset to the table. You can then read, delete and replace datasets, and the table keeps each earlier version.

pip install lance_nd

from pathlib import Path

import xarray as xr
import lance_nd

# Write each NetCDF file as one row.
with lance_nd.create("profiles.lance", stats=True) as writer:
    for path in sorted(Path("profiles").glob("*.nc")):
        with xr.open_dataset(path) as ds:
            writer.write(ds)

# Read the datasets with a temperature above 20.
table = lance_nd.open("profiles.lance")
for batch in table.scan(prune="TEMP.values > 20"):
    ds = batch.to_xarray()

Getting started

Getting started explains what you can write, makes your first table from NetCDF files and reads the datasets back, in a few minutes.

User guide

Each chapter explains one task, with examples that you can run.

Chapter	Contents
Create a table	A new table, stats, compression, overwrite and fragment size.
Add datasets	Append files, many files, errors, new variables, type casts and dask chunks.
Read datasets	Scan, select variables, filter on attributes and values, prune with stats.
Delete datasets	Delete one dataset or many datasets with one filter.
Replace datasets	Change datasets: delete the old datasets and add the new datasets.
Compact a table	Merge small fragments and remove deleted rows.
Versions and disk space	Read earlier versions, undo a change and free disk space.
Maintain a collection	A routine for a collection that gets new files each day.

Reference

• API reference: each function, option, error and encoding rule.
• Object stores: S3, Google Cloud Storage and Azure.
• Table layout: how a table stores the datasets.