Python Create a Table

lance_nd.create() makes a new table and gives a writer for its first datasets. The table exists only after the first commit.

Example data for this page

The examples use six CTD profiles from two cruises, see Getting started. Run this block first to make the table profiles.lance.

import numpy as np
import pandas as pd
import xarray as xr

import lance_nd


def profile(station, cruise, lat, lon, levels, time):
    depth = np.arange(levels, dtype="float64") * 10.0
    return xr.Dataset(
        {
            "TEMP": ("depth", 15.0 - depth / 100.0, {"units": "degree_Celsius"}),
            "PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
            "LATITUDE": ((), lat),
            "LONGITUDE": ((), lon),
            "TIME": ((), pd.Timestamp(time)),
        },
        coords={"depth": ("depth", depth, {"units": "m"})},
        attrs={"station": station, "cruise": cruise},
    )


def profiles():
    return [
        profile("A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01"),
        profile("A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02"),
        profile("A3", "cruise-1", 60.0, 2.0, 120, "2026-06-03"),
        profile("B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01"),
        profile("B2", "cruise-2", 55.5, 3.5, 70, "2026-07-02"),
        profile("B3", "cruise-2", 57.0, 5.0, 90, "2026-07-03"),
    ]


def stations(table):
    """The station of each dataset in the table, in table order."""
    return [batch.to_xarray().attrs["station"] for batch in table.scan(project=[])]


with lance_nd.create("profiles.lance", mode="overwrite", stats=True) as writer:
    writer.write(profiles())
table = lance_nd.open("profiles.lance")

Create a table

with lance_nd.create("new.lance") as writer:
    writer.write(profiles())

table = writer.table
print(table.uri.endswith("new.lance"), table.version, table.count_rows())

True 1 6

• The writer commits all datasets at the end of the with block, in one version.
• writer.table is the table after the commit.
• When the block raises an error, the writer commits nothing and the table does not exist.

Without a with block

Call finish() to commit, or abort() to drop the datasets.

writer = lance_nd.create("manual.lance")
writer.write(profiles()[:2])
table = writer.finish()
print(table.count_rows())

2

A writer that you do not finish commits nothing.

Store stats

stats=True stores the minimum and the maximum of each variable for each dataset. A scan with prune uses them to skip datasets, see Read datasets.

with lance_nd.create("stats.lance", stats=True) as writer:
    writer.write(profiles())
print(writer.table.has_stats)

True

• Set stats when you create the table. Each append then stores stats too.
• Stats need a little disk space: one small column for each variable.

Compression

column_defaults sets the encoding of all columns. column_options sets the encoding of one variable and overrides the defaults.

with lance_nd.create(
    "zstd.lance",
    column_defaults={"compression": "zstd", "compression_level": 3},
    column_options={"TEMP": {"byte_stream_split": "on"}},
) as writer:
    writer.write(profiles())
print(writer.table.count_rows())

6

Key	Values
compression	"none", "lz4", "zstd"
compression_level	an integer, with "zstd" only
byte_stream_split	"off", "on", "auto"
rle_threshold	a float
structural_encoding	"miniblock", "fullzip"

"zstd" gives small files for most scientific data. byte_stream_split often makes float values smaller.

Replace a table

create() raises TableExistsError when the table exists. mode="overwrite" replaces the table at the commit.

try:
    lance_nd.create("new.lance")
except lance_nd.TableExistsError as error:
    print(type(error).__name__)

with lance_nd.create("new.lance", mode="overwrite") as writer:
    writer.write(profiles()[:1])
print(writer.table.count_rows(), writer.table.version)

TableExistsError
1 2

• The overwrite is a new version, so you can still read the old content, see Versions.
• Readers see the old table until the commit.

Fragment size

A writer collects datasets in memory, then writes them as one fragment of about 128 MiB on disk. Up to 4 fragments are written at the same time.

lance_nd.create("big.lance", target_fragment_bytes=512 << 20, write_parallelism=8)

• A larger target_fragment_bytes gives fewer, larger files. A scan opens fewer files, but the writer uses more memory.
• The writer estimates the size on disk, so the sizes are not exact. Compact gives exact sizes later.

Other options

• casting decides what happens when a variable gets another type, see Add datasets.
• storage_options writes to S3, Google Cloud Storage or Azure, see Object stores.
• The API reference lists all arguments.