Repository navigation
Python Read Datasets
Table.scan() reads the datasets of a table as a stream. You can select
variables, filter on attributes and values, and skip datasets with the
stats.
Example files and table for this page
The examples use small NetCDF files of CTD profiles, see
Getting started. Run this
block first. It writes the files and makes the table profiles.lance
from the six files in profiles/.
from pathlib import Path
import numpy as np
import pandas as pd
import xarray as xr
def write_profile(path, station, cruise, lat, lon, levels, time, temp_type="float64", extra=None):
"""Write one CTD profile as a NetCDF file."""
depth = np.arange(levels, dtype="float64") * 10.0
ds = xr.Dataset(
{
"TEMP": ("depth", (15.0 - depth / 100.0).astype(temp_type), {"units": "degree_Celsius"}),
"PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
"LATITUDE": ((), lat),
"LONGITUDE": ((), lon),
"TIME": ((), pd.Timestamp(time)),
},
coords={"depth": ("depth", depth, {"units": "m"})},
attrs={"station": station, "cruise": cruise},
)
if extra:
ds["DOXY"] = ("depth", np.full(levels, 250.0), {"units": "umol/kg"})
Path(path).parent.mkdir(exist_ok=True)
ds.to_netcdf(path)
# Six profiles of two cruises.
write_profile("profiles/A1.nc", "A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01")
write_profile("profiles/A2.nc", "A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02")
write_profile("profiles/A3.nc", "A3", "cruise-1", 60.0, 2.0, 120, "2026-06-03")
write_profile("profiles/B1.nc", "B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01")
write_profile("profiles/B2.nc", "B2", "cruise-2", 55.5, 3.5, 70, "2026-07-02")
write_profile("profiles/B3.nc", "B3", "cruise-2", 57.0, 5.0, 90, "2026-07-03")
# Three new profiles of a third cruise.
write_profile("new/C1.nc", "C1", "cruise-3", 58.0, 6.0, 40, "2026-08-01")
write_profile("new/C2.nc", "C2", "cruise-3", 58.5, 6.5, 45, "2026-08-02")
write_profile("new/C3.nc", "C3", "cruise-3", 59.0, 7.0, 50, "2026-08-03")
# Corrected profiles, with other depth levels.
write_profile("corrected/A1.nc", "A1", "cruise-1", 52.5, 4.0, 55, "2026-06-01")
write_profile("corrected/A2.nc", "A2", "cruise-1", 53.0, 4.5, 90, "2026-06-02")
write_profile("corrected/B1.nc", "B1", "cruise-2", 54.0, 3.0, 65, "2026-07-01")
write_profile("corrected/B2.nc", "B2", "cruise-2", 55.5, 3.5, 75, "2026-07-02")
# Files that differ from the others.
write_profile("other/int_temp.nc", "D1", "cruise-4", 50.0, 1.0, 10, "2026-09-01", temp_type="int32")
write_profile("other/with_doxy.nc", "E1", "cruise-5", 51.0, 2.0, 20, "2026-10-01", extra=True)
write_profile("other/float32_temp.nc", "F1", "cruise-6", 52.0, 2.0, 10, "2026-11-01", temp_type="float32")
import lance_nd
def stations(table):
"""The station of each dataset in the table, in table order."""
return [batch.to_xarray().attrs["station"] for batch in table.scan(project=[])]
with lance_nd.create("profiles.lance", mode="overwrite", stats=True) as writer:
with xr.open_dataset("profiles/A1.nc") as ds:
writer.write(ds)
with xr.open_dataset("profiles/A2.nc") as ds:
writer.write(ds)
with xr.open_dataset("profiles/A3.nc") as ds:
writer.write(ds)
with xr.open_dataset("profiles/B1.nc") as ds:
writer.write(ds)
with xr.open_dataset("profiles/B2.nc") as ds:
writer.write(ds)
with xr.open_dataset("profiles/B3.nc") as ds:
writer.write(ds)
table = lance_nd.open("profiles.lance")Each item of a scan is an nd_arrow_array.NdRecordBatch with one dataset.
to_xarray() gives the xarray.Dataset.
for batch in table.scan():
ds = batch.to_xarray()
print(ds.attrs["station"], ds.sizes["depth"], str(ds.TIME.values))A1 50 2026-06-01T00:00:00.000000000
A2 80 2026-06-02T00:00:00.000000000
A3 120 2026-06-03T00:00:00.000000000
B1 60 2026-07-01T00:00:00.000000000
B2 70 2026-07-02T00:00:00.000000000
B3 90 2026-07-03T00:00:00.000000000
- A scan gives the datasets in the order of the table.
- A scan reads only a few batches ahead, so it uses little memory, also on a table of many gigabytes.
-
table.count_rows()gives the number of datasets, andtable.variables()the names of all variables.
project reads only the variables that you name. Each dataset keeps its
global attributes.
ds = next(iter(table.scan(project=["TEMP", "depth"]))).to_xarray()
print(list(ds.variables), ds.attrs)['TEMP', 'depth'] {'station': 'A1', 'cruise': 'cruise-1'}
- A coordinate is a variable too.
project=["TEMP"]reads nodepth, so name the coordinates that you need. -
project=[]reads only the global attributes. This is fast. - A scan with few variables reads less data from disk. Use
projectwhen you need a part of a large table. - A name of a nested field, such as
TEMP.attrs, raisesLanceNdError.
filter is a Lance SQL filter. It keeps the datasets that match exactly.
The global attributes are the fields of __global_attrs__.
cruise_2 = table.scan(project=[], filter="__global_attrs__.cruise = 'cruise-2'")
print([b.to_xarray().attrs["station"] for b in cruise_2])['B1', 'B2', 'B3']
The attributes of a variable are the fields of <variable>.attrs:
celsius = table.scan(project=[], filter="TEMP.attrs.units = 'degree_Celsius'")
print(len(list(celsius)))6
Combine filters with AND, OR, NOT, IN and LIKE:
f = "__global_attrs__.station IN ('A1', 'B3') OR __global_attrs__.station LIKE 'A3%'"
print([b.to_xarray().attrs["station"] for b in table.scan(project=[], filter=f)])['A1', 'A3', 'B3']
Text in a filter goes in single quotes. Write a single quote in the text
as two single quotes, for example 'Kapitan O''Brien'.
The values of a variable are the list <variable>.values. The first value
is values[1]. A scalar, such as LATITUDE, has one value.
north = table.scan(project=[], filter="LATITUDE.values[1] > 55")
print([b.to_xarray().attrs["station"] for b in north])
july = table.scan(project=[], filter="TIME.values[1] >= TIMESTAMP '2026-07-01'")
print([b.to_xarray().attrs["station"] for b in july])
deep = table.scan(project=[], filter="TEMP.dim_sizes[1] > 80")
print([b.to_xarray().attrs["station"] for b in deep])['A3', 'B2', 'B3']
['B1', 'B2', 'B3']
['A3', 'B3']
| Filter | Keeps the datasets where |
|---|---|
LATITUDE.values[1] > 55 |
the first value is larger than 55 |
array_max(TEMP.values) > 14 |
the largest value is larger than 14 |
array_has(TEMP.values, 15.0) |
one value is 15.0 |
TEMP.dim_sizes[1] > 80 |
the first dimension is longer than 80 |
TEMP.dim_names[1] = 'depth' |
the first dimension is depth
|
TEMP IS NOT NULL |
the dataset has the variable TEMP
|
A filter on values reads the values of each dataset. On a large table,
use prune too, so that the scan skips most datasets first.
prune is a filter on values that uses the stats of the table. It skips
each dataset that cannot match, without a read of its values. The table
needs stats, see Create a table.
warm = table.scan(project=[], prune="LATITUDE.values > 55")
print([b.to_xarray().attrs["station"] for b in warm])['A3', 'B2', 'B3']
-
prunekeeps a dataset when its minimum and maximum allow a match. The dataset comes back whole, so it can hold values outside the filter. For example,prune="TEMP.values > 14"keeps a profile with values from 10 to 15. -
prunetakes comparisons (>,>=,<,<=,=,BETWEEN) on<variable>.values, withANDandOR. - Without stats,
prunekeeps all datasets. - Use
pruneandfiltertogether:pruneskips most datasets fast, andfilterthen keeps the exact matches.
A region query uses prune on the position:
region = "LATITUDE.values BETWEEN 52 AND 54 AND LONGITUDE.values BETWEEN 3 AND 5"
print([b.to_xarray().attrs["station"] for b in table.scan(project=[], prune=region)])['A1', 'A2', 'B1']
limit stops the scan after a number of datasets.
print(len(list(table.scan(limit=2))))2
batch_size and batch_size_bytes limit the size of the Lance batches
that the scan reads. Use them when the datasets are very large.
table.checkout(version) gives the table at an earlier version. Scan it
as any other table, see Versions.
Concepts
Rust
Python