Python API Reference

The reference of the lance_nd package. The user guide shows each task with examples.

Functions

Function	Returns	Purpose
create(uri, mode="create", column_defaults=None, column_options=None, stats=False, target_fragment_bytes=None, write_parallelism=None, casting="no", storage_options=None)	Writer	A writer for a new table. mode="overwrite" replaces a table at finish().
append(uri, column_options=None, target_fragment_bytes=None, write_parallelism=None, casting="no", storage_options=None)	Writer	A writer for a table that exists.
open(uri, storage_options=None)	Table	The table at its latest version.

Argument	Default	Meaning
mode	"create"	"create" raises TableExistsError for a table that exists. "overwrite" replaces it.
column_defaults	None	The encoding options of all columns.
column_options	None	A dict from variable name to encoding options.
stats	False	Store the minimum and the maximum of each variable for each dataset. See Stats.
target_fragment_bytes	128 MiB	The size of a fragment on disk. See Fragments.
write_parallelism	4	The number of fragments that are written at the same time.
casting	"no"	What happens when a variable has another type than the table. See Casting.
storage_options	None	The Lance options of an object store. See Object stores.

Writer

Member	Purpose
write(data, chunks=None, on_drop="warn")	Write datasets without a commit. data is an xarray.Dataset, an NdRecordBatch, a list of these, or Arrow data in the nd-arrow format. See Chunked datasets for chunks and on_drop.
finish()	Commit all written datasets in one version. Returns the Table.
abort()	Drop all written datasets.
table	The committed Table after finish(), else None.

Rules:

• A writer that you abort, or that you do not finish, commits nothing.
• As a context manager, the writer finishes at the end of the block. When the block raises, the writer aborts. After a finish() in the block, the block does not finish again.
• finish() on a new table without datasets raises an error. On an append without datasets, it makes no commit.
• A write() that fails a check, for example a variable or an attribute with another type than the table, raises LanceNdError and writes none of its datasets. The writer stays usable, and finish() commits the other datasets.
• A failed fragment write makes the writer fail. finish() then commits nothing.
• When another writer adds variables before your finish(), and your writer also adds variables, finish() raises CommitConflictError.

Table

Member	Purpose
uri	The uri of the table.
version	The version of the table.
has_stats	True when the table stores stats.
num_fragments	The number of fragments.
count_rows()	The number of datasets.
variables()	The names of all variables.
scan(project=None, filter=None, prune=None, limit=None, batch_size=None, batch_size_bytes=None)	A Scanner of the datasets. See Scan.
delete(predicate)	Delete the datasets that match the Lance SQL filter predicate, in one commit. Returns the number of deleted datasets.
compact(target_fragment_bytes=None, deleted_fraction=None)	Merge small fragments, split large fragments and remove deleted rows. Returns fragments_removed, fragments_added, files_removed and files_added.
cleanup_old_versions(older_than=timedelta(days=14))	Remove the files of older versions. Returns old_versions and bytes_removed.
versions()	(version, timestamp) for each version that has its files, oldest first.
checkout(version)	The table at version, to read.
restore(version)	Make the content of version the latest version, in a new commit.
widen(variable, to)	Change the value type of variable to the wider pyarrow type to.

delete(), compact(), restore() and widen() change the table object to the new version.

Scan

Table.scan() gives a Scanner, an iterator of NdRecordBatch objects with one dataset each. NdRecordBatch.to_xarray() gives the xarray.Dataset.

Argument	Meaning
project	The variable names to read. [] reads only the global attributes. A nested field raises an error.
filter	An exact Lance SQL filter, for example __global_attrs__.cruise = 'a' or TEMP.attrs.units = 'K'.
prune	A filter on values, for example TEMP.values > 20. It uses the stats to skip datasets that cannot match. A kept dataset comes back whole.
limit	The maximum number of datasets.
batch_size, batch_size_bytes	The size of the Lance batches that the scan reads.

The columns of a filter:

Column	Contents
__global_attrs__.<name>	A global attribute.
<variable>.attrs.<name>	An attribute of a variable.
<variable>.values	The values, as a flat list. The first value is values[1].
<variable>.dim_names, <variable>.dim_sizes	The dimension names and sizes.
<variable> IS NULL	True for a dataset without the variable.

Errors

All errors from lance-nd are lance_nd.LanceNdError or a subclass:

Error	When
TableNotFoundError	No table exists at the uri.
TableExistsError	create() finds a table at the uri.
InvalidTableError	The table is not a lance-nd table of this format version.
SchemaMismatchError	A variable or an attribute does not fit the table.
CommitConflictError	Another commit changed the table first.

Encoding options

create() takes column_defaults and column_options, and append() takes column_options. Each option set is a dict:

Key	Values
compression	"none", "lz4", "zstd"
compression_level	an integer, with "zstd" only
byte_stream_split	"off", "on", "auto"
rle_threshold	a float
structural_encoding	"miniblock", "fullzip"

lance_nd.create(
    "data.lance",
    column_defaults={"compression": "zstd", "compression_level": 3},
    column_options={"sst": {"byte_stream_split": "on"}},
)

Stats

create(uri, stats=True) stores the minimum and the maximum of each variable for each dataset, in one column __stats__:<variable> for each variable. A scan with prune uses them to skip datasets. An append follows the table. Each variable name that starts with __stats__ is reserved.

Casting

create() and append() take casting, as the numpy casting argument:

• "no" (default): the write raises SchemaMismatchError.
• "safe": the writer casts the new values when no value can change, for example float32 to float64, or int16 to float64.
• "same_kind": the writer also casts within a kind, for example float64 to float32. An integer that does not fit raises LanceNdError.
• "promote": as numpy type promotion. When the new values do not fit the table type, the writer widens the variable in the table to the smallest type that holds both. A writer first commits the datasets that it wrote before such a widening.

With "no", "safe" and "same_kind", the table type always wins, so the first dataset with a variable sets its type.

Chunked datasets

Writer.write(ds, chunks=[...], on_drop="warn") writes one row for each dask chunk:

• chunks names the chunk dimensions. Each row is one dask chunk over these dimensions, in C order with the first name as the outermost.
• A variable is kept when all its dimensions are chunk dimensions, for example (time), (time, depth) or a scalar. A variable is also kept when it has all chunk dimensions, for example (time, depth, position). Its other dimensions keep their full size in each row.
• All other variables are dropped, also coordinates such as position (position). on_drop sets what happens: "warn" gives a UserWarning, "raise" gives a ValueError, and "ignore" drops them silently.
• Kept variables keep their attributes. Each row keeps the global attributes.
• The writer computes one chunk at a time.

Fragments

A writer buffers datasets and writes them as Lance fragments of about target_fragment_bytes (128 MiB) on disk, with write_parallelism (4) fragment writes at the same time. The writer estimates the size on disk from the fragments that it wrote, so the sizes are not exact. Table.compact() gives exact sizes.

Encoding rules

The nd-arrow-array package converts xarray datasets, see its documentation for the value types, missing values, coordinates and calendars.

xarray	nd-arrow
One Dataset	One row
Each data variable and coordinate	One nd_arrow.array column
Variable.attrs	The attrs of the array
Dataset.attrs	The __global_attrs__ column
Variable.encoding, Dataset.encoding	Not written

lance-nd adds these rules:

• Byte strings become text: a variable of fixed-width bytes (|S), a bytes attribute and a byte attribute array are decoded as UTF-8.
• cftime dates, for example of the Julian calendar, become timestamps of the same instant.
• An attribute that is an empty array, such as [], holds no value, so the writer leaves it out.
• The variable name __global_attrs__ and each name that starts with __stats__ are reserved. They raise SchemaMismatchError.
• Datasets can have different variables. A dataset without a variable has a null in its column. One variable has one value type, see Casting.
• Dask arrays are loaded into memory, one dataset or one chunk at a time.

Object stores

create, append and open take storage_options, the Lance storage options of S3, Google Cloud Storage or Azure:

options = {"aws_region": "eu-west-1"}
with lance_nd.create("s3://bucket/data.lance", storage_options=options) as writer:
    writer.write(ds)
table = lance_nd.open("s3://bucket/data.lance", storage_options=options)

Without options, Lance reads the credentials from the environment. See Object stores.

Type hints

The package has type hints and a py.typed marker.

Type	Values
lance_nd.ColumnOptions	The option set of one column, a TypedDict.
lance_nd.Mode	"create", "overwrite"
lance_nd.Casting	"no", "safe", "same_kind", "promote"
lance_nd.OnDrop	"warn", "raise", "ignore"
lance_nd.ArrowData	Arrow data with the Arrow C stream or array interface.

Slow NetCDF-4 files on Windows

xarray asks netCDF4 for the compression filters of each variable of a NetCDF-4 file. netCDF4 then searches the HDF5 filter plugins on disk, and on Windows this can take about 30 ms per variable. When your files use only the filters that HDF5 has built in, such as zlib, set HDF5_PLUGIN_PATH to an empty folder before you import xarray:

import os

os.environ["HDF5_PLUGIN_PATH"] = r"C:\empty-folder"

import xarray as xr

A file that needs a plugin filter, such as zstd or blosc, can then fail to read. Classic NetCDF-3 files are not affected.