-
Notifications
You must be signed in to change notification settings - Fork 0
API Reference
Most of what the viewer does is available as ordinary Python. The analysis
functions take a plain AnnData and return plain results, so a notebook can call
them directly — useful for batching across datasets, for re-running one step with
parameters the GUI does not expose, or for picking up where an exported notebook
left off.
import sys
sys.path.insert(0, "/path/to/palms/src") # or: pip install -e .
from palms import loader
from palms.utils import gene_analysis, spatial_analysisInstalling the package (pip install -e ., which environment.yml does for you)
makes the sys.path line unnecessary.
Everything on this page takes ordinary arguments and works without a GUI. The
codebase contains a great deal more that is public in the Python sense but is
really viewer internals — it takes a ViewerContext, mutates napari layers, or
assumes a Qt event loop. Those are listed under
Reachable, but needs a live viewer rather
than documented as API, because depending on them from a notebook will not go
well.
Signatures below are extracted from the live objects when this page is generated, so they cannot describe a function that no longer exists.
load_sdatawrites to disk. On a cache miss it buildssdata_cached.zarrnext to the dataset, which takes minutes and tens of GB of disk, and it may prompt about a stale cache. Passuse_cache=Falsefor a read-only load that touches nothing.
A few defaults below are Paths resolved against the working directory at import
time; they are shown as .../name, since the absolute value depends on where the
interpreter was started and is not something to rely on. Pass those explicitly.
- Loading a dataset
- Expression analysis
- Spatial statistics
- Copy number
- Image registration
- Cache, storage and persistence
- Provenance and notebooks
- The template registry
- Memory and raster helpers
- Reachable, but needs a live viewer
The same entry points the viewer uses at startup. load_sdata returns a SpatialData; the table is sdata.tables['table'], an ordinary AnnData, which is what every analysis function below expects.
from palms.loader import load_sdata
load_sdata(path: pathlib.Path, build_pyramid: bool = True, n_jobs: int = 8, use_cache: bool = True, on_stale: Optional[str] = None)Load a Xenium output directory, through the zarr cache when there is one. With a cache this takes seconds rather than the minutes a cold spatialdata_io.xenium() needs, and the pyramid levels come back read from disk rather than as a lazy coarsen() chain — which is the difference between a 1.7 GB and a 23 GB peak the moment you touch a low-resolution level. on_stale answers the stale-cache question in advance ('keep', 'rebuild' or 'restore') instead of prompting, which is what makes the load usable with no GUI attached; the palms-build-cache console script is this function with that flag exposed.
Load the Xenium 3.x output as a SpatialData object.
Caveat: Builds and writes the cache on a miss; use_cache=False avoids that.
from pathlib import Path
from palms import loader
sdata = loader.load_sdata(Path('/data/xenium_run'))
adata = sdata.tables['table']
print(adata)from palms.loader import load_clusterings
load_clusterings(path: pathlib.Path)The clusterings 10x shipped with the run, read from the output's analysis/clustering/ directory — graphclust, the kmeans_* series — as a dict of name to a Series indexed by cell barcode. Takes the dataset path, not an AnnData, and returns {} when there is no analysis/ folder, which is the case for a Crop Dataset export. Clusterings you computed live in adata.obs, not here.
Load all cluster assignments from analysis/clustering/.
clusterings = loader.load_clusterings(Path('/data/xenium_run'))
list(clusterings) # [] for a cropped export
# Your own clusterings are obs columns on the table:
[c for c in adata.obs.columns if c.startswith('clustering_')]from palms.loader import load_umap
load_umap(path: pathlib.Path)The UMAP embedding from the Xenium output's own analysis/ directory, if the run produced one.
Load precomputed UMAP coordinates.
from palms.loader import get_label_to_obs_mapping
get_label_to_obs_mapping(sdata)Map label-raster values to row positions in adata.obs. This is what turns a pixel you clicked into a cell, and it is needed whenever you colour the segmentation raster by a per-cell value.
Build a mapping from integer label value -> AnnData obs index position.
from palms.utils.transcript_index import TranscriptLoader
TranscriptLoader(cache_dir: 'Path' = Path('.../transcript_cache'), parquet_path: 'Path' = Path('.../transcripts.parquet'), min_qv: 'int' = 20, pixel_size: 'float' = 0.2125)Per-gene transcript access. With the feather index built by palms-preprocess a gene loads in ~100 ms; without it, each query falls back to scanning transcripts.parquet and takes seconds. cached_genes is a property, not a method, and is empty when the index has not been built.
Loads per-gene transcript locations.
from pathlib import Path
from palms.utils.transcript_index import TranscriptLoader
run = Path('/data/xenium_run')
tl = TranscriptLoader(cache_dir=run / 'transcript_cache',
parquet_path=run / 'transcripts.parquet')
print(len(tl.cached_genes)) # 0 if palms-preprocess never ran
df = tl.load_gene('EPCAM') # x, y, and quality columnsThese take the normalised AnnData. Build it the way the templates do — sc.pp.normalize_total(target_sum=1e4), sc.pp.log1p, sc.pp.pca on a copy — or read the normalize template for the exact three lines.
from palms.utils.gene_analysis import run_rank_genes
run_rank_genes(adata_norm: 'sc.AnnData', groupby: 'str', method: 'str' = 'wilcoxon', n_genes: 'int' = 25) -> 'pd.DataFrame'Rank marker genes per cluster and return them as a tidy DataFrame, one row per gene per group.
Run rank_genes_groups and return full results DataFrame.
import scanpy as sc
from palms.utils import gene_analysis
adata_norm = adata.copy()
sc.pp.normalize_total(adata_norm, target_sum=1e4)
sc.pp.log1p(adata_norm)
sc.pp.pca(adata_norm)
# Clusterings the viewer saved are obs columns named clustering_*
groupby = next(c for c in adata.obs.columns
if c.startswith('clustering_'))
df = gene_analysis.run_rank_genes(adata_norm, groupby=groupby)
df.head()from palms.utils.gene_analysis import rank_genes_key
rank_genes_key(groupby: 'str') -> 'str'The uns key a ranking is stored under for a given clustering. Rankings are keyed per clustering so a second one does not overwrite the first — use this rather than assuming 'rank_genes_groups'.
The
unsslot holding the ranking for clustering groupby.
from palms.utils.gene_analysis import resolve_rank_key
resolve_rank_key(adata, groupby: 'Optional[str]') -> 'str'Find the ranking actually present for a clustering, falling back to the unkeyed default for results saved before keying existed.
The keyed slot if adata has one, else scanpy's default.
from palms.utils.gene_analysis import add_clustering_to_obs
add_clustering_to_obs(adata_norm: 'sc.AnnData', adata_orig: 'sc.AnnData', clustering_series: 'pd.Series', key_name: 'str') -> 'None'Attach a clustering to adata.obs under the viewer's naming conventions, so the GUI and the session store recognise it.
Align clustering_series to adata_norm.obs via cell_id and store as categorical.
from palms.utils.gene_analysis import make_rank_genes_dotplot
make_rank_genes_dotplot(adata_norm: 'sc.AnnData', groupby: 'str', n_genes: 'int' = 5, cluster_labels: 'Optional[dict]' = None, dendrogram: 'bool' = True, key: 'Optional[str]' = None)The dotplot the Markers tab draws, as a matplotlib figure.
Create a dotplot of top marker genes per cluster.
from palms.utils.gene_analysis import run_celltypist_annotation
run_celltypist_annotation(adata, model_name)Annotate cell types with CellTypist against a downloaded reference model.
Run CellTypist annotation and return per-cell predictions with confidence.
Thin, honest wrappers over squidpy: they build the neighbour graph the way the viewer does and return the results rather than plotting them. Call compute_spatial_neighbors first — the others need the graph.
from palms.utils.spatial_analysis import compute_spatial_neighbors
compute_spatial_neighbors(adata_norm: 'sc.AnnData', n_neighs: 'int' = 6) -> 'None'Build the spatial neighbour graph on the normalised AnnData. Mutates adata_norm in place, as squidpy does.
Compute spatial neighbor graph (modifies adata_norm in-place).
from palms.utils import spatial_analysis as sa
sa.compute_spatial_neighbors(adata_norm, n_neighs=6)
res = sa.run_nhood_enrichment(adata_norm, cluster_key=groupby,
n_perms=100)
res['zscore'].shapefrom palms.utils.spatial_analysis import run_nhood_enrichment
run_nhood_enrichment(adata_norm: 'sc.AnnData', cluster_key: 'str', n_perms: 'int' = 1000, seed: 'int' = 42) -> 'dict'Neighbourhood-enrichment permutation test: which cluster pairs are adjacent more or less often than chance.
Run neighborhood enrichment analysis.
from palms.utils.spatial_analysis import make_nhood_enrichment_plot
make_nhood_enrichment_plot(result: 'dict', mode: 'str' = 'zscore', cluster_filter: 'list[str] | None' = None, cluster_labels: 'dict | None' = None, annotate: 'bool' = False) -> 'plt.Figure'The enrichment heatmap, as a figure.
Create a neighborhood enrichment heatmap matching squidpy native style.
from palms.utils.spatial_analysis import make_co_occurrence_plot
make_co_occurrence_plot(result: 'dict', clusters_to_plot: 'list[str] | None' = None, target_clusters: 'list[str] | None' = None, cluster_colors: 'dict | None' = None, cluster_labels: 'dict | None' = None) -> 'plt.Figure'The co-occurrence-versus-radius plot, as a figure.
Create co-occurrence line plots. Returns matplotlib Figure.
from palms.utils.spatial_analysis import make_ligrec_plot
make_ligrec_plot(result: 'dict', pvalue_threshold: 'float' = 0.05, source_groups: 'list[str] | None' = None, target_groups: 'list[str] | None' = None, cluster_labels: 'dict | None' = None) -> 'plt.Figure'The ligand-receptor dotplot, as a figure.
Create a ligand-receptor interaction dot plot. Returns matplotlib Figure.
inferCNV runs in the main environment. CopyKAT does not — it needs rpy2 and R 4.3, which pin python 3.11, so it runs in the separate palms_copykat environment via a detached worker. From a notebook in the main environment, use backend='infercnv'.
from palms.utils.cnv_analysis import run_cnv_pipeline
run_cnv_pipeline(adata, reference_series: 'pd.Series', reference_categories: 'list[str]', reference_clustering_name: 'str' = '', n_neighbors: 'int' = 15, smoothing_neighbors: 'int' = 20, window_size: 'int' = 60, step: 'int' = 10, lfc_clip: 'float' = 4.0, resolution: 'float' = 0.2, n_cores: 'int' = 1, analyze_categories: 'list[str] | None' = None, backend: 'str' = 'infercnv', copykat_output_dir: 'str | None' = None) -> 'dict'Infer CNV against a reference population and cluster the result.
Run the InSituCNV pipeline on
adata(raw counts expected in.X).
from palms.utils import cnv_analysis
result = cnv_analysis.run_cnv_pipeline(
adata,
reference_series=adata.obs['leiden'],
reference_categories=['0', '3'], # the normal clusters
backend='infercnv',
)from palms.utils.cnv_analysis import check_gene_mapping
check_gene_mapping(n_mapped: 'int', n_total: 'int', var_names=None) -> 'None'How many panel genes map to genomic positions — worth checking before a run, since a Xenium panel covers a few hundred genes and coverage per chromosome arm drives whether the result means anything.
Fail loudly when the panel barely matches the gene-position reference.
from palms.utils.cnv_analysis import make_cnv_heatmap
make_cnv_heatmap(adata_cnv, groupby: 'str')The chromosome-ordered CNV heatmap, as a figure.
Build an infercnvpy chromosome heatmap figure for
groupby.
The maths behind the H&E and ARMS overlays. Landmark pairs in, similarity affine out — usable on any two images, not just the ones the viewer loads.
from palms.utils.registration import compute_landmark_affine
compute_landmark_affine(xenium_pts_yx, he_pts_yx)Least-squares similarity transform (rotation, uniform scale, translation) from matched landmark pairs.
Estimate a similarity affine from paired landmark points.
import numpy as np
from palms.utils import registration
fixed = np.array([[100, 200], [800, 250], [450, 900]])
moving = np.array([[110, 190], [810, 260], [460, 880]])
affine = registration.compute_landmark_affine(moving, fixed)from palms.utils.registration import load_he_pyramid
load_he_pyramid(path)Read an H&E slide as a multiscale pyramid, lazily.
Load an H&E OME-TIFF/SVS as a list of dask arrays (one per pyramid level).
from palms.utils.registration import load_multichannel_pyramid
load_multichannel_pyramid(path)The same for a multi-channel fluorescence image.
Load a possibly-multichannel OME-TIFF as a dask pyramid with explicit channel axis.
from palms.utils.registration import save_landmarks
save_landmarks(path, xenium_yx, he_yx, affine=None, he_filename=None)Persist landmark pairs beside the dataset.
Save landmark points and optional affine to a JSON file.
from palms.utils.registration import load_landmarks
load_landmarks(path)Read them back.
Load landmark points and optional affine from a JSON file.
from palms.utils.registration import extract_tissue_mask
extract_tissue_mask(image_gray, blur_ksize=5, open_ksize=5, close_ksize=5, min_area_ratio=0.01)Segment tissue from background, for a coarse initial alignment.
Extract a binary tissue mask via Otsu thresholding + morphological cleanup.
Never write to the zarr store directly. Every element write goes through the staging-and-rename path in zarr_safe, which journals the operation and moves the previous copy to .xv_trash/ rather than deleting it. Writing straight to the store leaves it invalid if anything interrupts.
from palms.utils.cache_repair import verify
verify(cache_path: 'Path') -> 'HealthReport'Read-only health check of a cache. Parses the root zarr.json with json.loads rather than opening the store, so it works on one too broken to open.
Inspect cache_path without modifying anything.
from pathlib import Path
from palms.utils import cache_repair
report = cache_repair.verify(Path('/data/xenium_run/sdata_cached.zarr'))
print(report.ok, report.missing_on_disk)from palms.utils.cache_repair import repair
repair(cache_path: 'Path', report: 'Optional[HealthReport]' = None, *, level: 'str' = 'auto') -> 'RepairResult'Fix what verify found. Only renames, clears debris and re-consolidates — it never deletes a cache.
Fix what report found. Idempotent; safe to run on a healthy cache.
from palms.utils.cache_repair import describe_store
describe_store(cache_path: 'Path') -> 'dict'A human-readable summary of what is in a store.
Size and free-space figures for the health panel.
from palms.utils.store_inventory import build_inventory
build_inventory(data_path, cache_path=None) -> 'list[Section]'Everything on disk for a dataset — raw output, cache elements, session state, derived caches, backups — with sizes and an honest recoverable flag per node. Filesystem-only and read-only.
Every viewer-visible thing on disk for this dataset, in five sections.
from palms.utils import store_inventory
for section in store_inventory.build_inventory(Path('/data/xenium_run')):
total = sum(n.size_bytes for n in section.nodes)
print(f'{section.title:24s} {total / 2**30:6.2f} GiB')from palms.utils.sdata_write import write_sdata
write_sdata(sdata: 'SpatialData', path: 'str | Path', progress_cb: 'Optional[ProgressCb]' = None, log: 'Optional[Callable[[str], None]]' = None, start_pct: 'int' = 0, end_pct: 'int' = 100) -> 'None'Write a SpatialData one element at a time, capping dask's concurrency where the intermediates are large and releasing memory between elements. Interchangeable with SpatialData.write() — a test asserts the stores match — but it reports progress and holds a far lower peak.
Write sdata to path, one element at a time.
from palms.utils import sdata_write
sdata_write.write_sdata(sdata, Path('/tmp/out.zarr'),
progress_cb=lambda pct, msg: print(pct, msg))from palms.utils.adata_persistence import sidecar_write_path
sidecar_write_path(ctx_or_sdata, name: 'str') -> 'Path'Where a derived result belongs: <data_path>/viewer_cache/, not the zarr store root. Files in the store root make zarr's hierarchy walk warn on every consolidation, and a rebuild deletes them.
Where a sidecar should be written. Always the new location.
from palms.utils.adata_persistence import find_sidecar
find_sidecar(sdata, name: 'str') -> "'Path | None'"Read one back, falling back to the legacy in-store location for datasets that predate the move.
from palms.utils.session import load_session
load_session(zarr_path: 'Path') -> 'Optional[dict]'Read the viewer session — ROIs, registrations, clusterings, DEG results, the provenance graph — out of a cache without starting the GUI.
Load viewer session state from zarr store.
The provenance graph is the single source of truth for reproducible code: a DAG of steps, each with a stable id, its rendered code and its dependencies. The notebook is derived from it by topological sort, so it respects dependencies regardless of the order you clicked things — even across sessions.
from palms.utils.prov_graph import ProvGraph
ProvGraph() -> 'None'The DAG itself: upsert(node_id, code, deps=…) adds or revises a step and flags its descendants stale; a missing dependency errors at record time rather than at replay. topo_sort() returns node ids, in dependency order; get(node_id) gives the node.
A mutable DAG of :class:
ProvNode, keyed by node id.
import json
from pathlib import Path
from palms.utils.prov_graph import ProvGraph
sidecar = Path('/data/xenium_run/viewer_cache/prov_graph.json')
graph = ProvGraph.from_list(json.loads(sidecar.read_text()))
for node_id in graph.topo_sort():
print(node_id, '<-', graph.get(node_id).deps)from palms.utils.prov_graph import graph_to_script
graph_to_script(graph: 'ProvGraph', include_terminals: 'bool' = True) -> 'str'Render the graph as a flat analysis.py.
Flat
.pyrendering: the code cells joined in topological order.
from palms.utils.prov_graph import graph_to_mermaid
graph_to_mermaid(graph: 'ProvGraph') -> 'str'Render the DAG as mermaid diagram text.
Render the DAG as a Mermaid flowchart (paste into any Mermaid viewer).
from palms.utils.notebook_export import write_graph_notebook
write_graph_notebook(graph, path: 'str | Path', include_terminals: 'bool' = True) -> 'None'Write the graph out as a real .ipynb. The notebook is code-only and replays from the raw Xenium output.
Convenience: derive cells from graph and write them to path.
from palms.utils import notebook_export
notebook_export.write_graph_notebook(graph, Path('analysis.ipynb'))from palms.utils.notebook_export import execute_notebook
execute_notebook(path: 'str | Path', cwd: 'Optional[str | Path]' = None, timeout: 'int' = 1800, on_cell_start: 'Optional[Callable]' = None, on_cell_executed: 'Optional[Callable]' = None, on_cell_error: 'Optional[Callable]' = None)Execute a notebook in a throwaway kernelspec pointing at sys.executable — deliberately not the installed python3 kernel, which on a conda box belongs to whichever environment registered it last.
Execute the notebook at path in a fresh kernel and return it.
from palms.utils.notebook_export import customisation_banner
customisation_banner(graph) -> 'str | None'The markdown cell prepended when any step ran from a non-shipped template.
A markdown note naming steps that did not use the shipped template.
Read the shipped templates, or resolve them the way the viewer does with user overrides applied. See Analysis Templates for what each one contains.
from palms.utils.step_templates.loader import builtin_ids
builtin_ids() -> 'list[str]'Every shipped template id.
Every registered builtin template id, sorted.
from palms.utils.step_templates import loader as registry
for tid in registry.builtin_ids():
spec = registry.builtin_spec(tid)
print(tid, '->', spec.doc)from palms.utils.step_templates.loader import builtin_spec
builtin_spec(template_id: 'str') -> 'TemplateSpec'The parsed contract: params, requires, outputs, blocks, assemblies.
The shipped spec for template_id. Never consults an override path.
from palms.utils.step_templates.loader import builtin_text
builtin_text(template_id: 'str') -> 'str'The shipped body, verbatim. Ignores user overrides — which is what you want for reading the default, and not what a run site wants.
The shipped text of a single-block template, in file order.
from palms.utils.step_templates.loader import resolve
resolve(template_id: 'str') -> 'ResolvedTemplate'The template as it would actually run, with per-block user overrides merged. Never raises and never returns nothing: an invalid override is skipped and the problems ride along on the result.
The spec actually used for template_id, plus why.
from palms.utils.step_templates.loader import step_template
step_template(template_id: 'str', block_names: 'Iterable[str]') -> 'dict'Resolved text and its provenance stamp together — a stamp fetched separately could describe a different resolution than the text it labels.
Step kwargs for template_id: the resolved text plus its provenance stamp.
Useful when working with the full-resolution images, which are large enough that how you read them decides whether the process survives.
from palms.utils.raster_io import open_ome_tiff_pyramid
open_ome_tiff_pyramid(tif_path: 'Path') -> 'list'Open an OME-TIFF through its own tiles as a list of dask arrays, one per resolution level. dask_image.imread gives one chunk per full channel page instead — 5.93 GB each on a full slide, which must be decoded whole before a single tile comes out.
All resolution levels of tif_path as dask arrays, chunked as on disk.
from palms.utils import raster_io
# find_morphology_tiff handles both output layouts — a
# morphology_focus/ directory of per-channel files, and the
# single morphology.ome.tif older runs produced.
tif = raster_io.find_morphology_tiff(Path('/data/xenium_run'))
levels = raster_io.open_ome_tiff_pyramid(tif)
[lvl.shape for lvl in levels]from palms.utils.raster_io import find_morphology_tiff
find_morphology_tiff(data_path: 'Path') -> 'Optional[Path]'Locate the OME-TIFF holding morphology_focus across Xenium output layouts — a morphology_focus/ directory of per-channel files on 3.x, a single morphology.ome.tif on older runs. Returns None rather than raising when there is neither.
The OME-TIFF holding morphology_focus, across Xenium output layouts.
from palms.utils.mem_probe import format_memory
format_memory(tag: 'str' = '') -> 'str'One line of RSS, peak RSS and napari's dask-cache occupancy — for logging inside a long loop.
One-line memory summary: current RSS, peak RSS, and the dask cache.
from palms.utils import mem_probe
print(mem_probe.format_memory('after load'))from palms.utils.mem_probe import release
release(collect: 'bool' = True) -> 'None'gc.collect() plus malloc_trim(0). glibc keeps freed blocks in per-thread arenas, so without the trim RSS ratchets to the high-water mark and stays there — which looks exactly like a leak.
Drop what Python can drop, then hand the freed pages back to the OS.
These are public in the Python sense and you will find them by reading the
source, but they take a ViewerContext, touch napari layers, or assume a Qt
event loop. They are listed so you know what they are, not so you call them from
a notebook:
| Function | Why it is not API |
|---|---|
utils.crop_export.crop_and_export(ctx, …) |
Takes a ViewerContext; reads the crop polygon from a napari shapes layer and rebuilds the transcript cache. Use the Crop Dataset tab. |
utils.steps.StepExecutor / ctx.run_step
|
Executes a rendered template and records it in the provenance graph. Outside the viewer there is no graph to record into. |
utils.coloring.CellColorManager |
Drives a napari DirectLabelColormap. |
utils.umap_widget.UMAPViewer, utils.minimap_widget
|
Qt windows. |
tabs.*.build_tab(ctx) |
Every tab module. Builds widgets. |
utils.reporting.* |
Routes messages to napari's notification system. |
If you want what one of these does without the GUI, the corresponding template in Analysis Templates is the code it runs, and it is plain scverse source you can paste into a notebook.
Reference
Cells
Genes
Spatial
- ROI Analysis
- Ligand-Receptor
- Neighborhood Enrichment
- Co-occurrence
- Spatial Domains
- Annot Nhood
- Annot Distance
Images
Tools
Tutorials
- Getting Started
- Clustering and DEG
- H&E Registration
- ARMS Overlay
- ROI Analysis
- Annotations
- Recovering a Cache