cengen-reference-data

Two related data products for CeNGEN's reference expression data, one per developmental stage / sex combination (e.g. adult hermaphrodite, L4 male):

1. Aggregate gene x cell-type tables — small (tens of MB), versioned, hosted as a pins board directly in this git repo. This is what cengenAnnotate consumes for automated cluster annotation, read over plain HTTPS via pins::board_url() — no GitHub account or token required.
2. Per-cell data (raw count matrix + per-cell annotation/metadata) — much larger (tens to ~100 MB per dataset), hosted as GitHub Release assets instead of git-tracked pins, since these routinely exceed git's practical size limits. See "Per-cell data" below.

Both are designed to be useful beyond cengenAnnotate — for any other tool or analysis that wants CeNGEN reference data.

New datasets can be published here at any time without requiring any changes to consumers such as cengenAnnotate — they just call list_cengen_datasets() / load_cengen_reference() and see the new dataset appear.

Reading the aggregate tables (any R user)

board <- pins::board_url("https://raw.githubusercontent.com/cengenproject/cengen-reference-data/main/")
pins::pin_list(board)
pins::pin_meta(board, "adult_herm")
data <- pins::pin_read(board, "adult_herm")

(cengenAnnotate::cengen_board() / list_cengen_datasets() / load_cengen_reference() wrap exactly this.)

Dataset schema

Each pin is a single tidy long table, unthresholded, with every (gene, cell_type) pair present (including zeros):

column	type	meaning
gene	character	gene symbol / WormBase ID (consistent per dataset; document which in the pin's notes)
cell_type	character	CeNGEN cell/tissue type — mostly neuron classes (e.g. "AWA", "AVA", "ASEL"), plus non-neuronal types (e.g. "Body_wall_muscle", "Intestine", "Germline") where the source dataset includes them
avg_expr	double	unthresholded average expression level for that gene in that cell type
pct_expr	double	percent of cells of that cell type expressing the gene, 0-100

These are the same two quantities a CeNGEN dot plot encodes as color and dot size, kept unthresholded so consumers can apply their own coverage/specificity logic (see cengenAnnotate's coherence score) rather than losing information to a pre-binned threshold category.

Pin naming

One pin per stage + sex variant, lowercase snake_case, no version in the name (pins itself tracks versions on re-publish):

adult_herm, adult_male, L4_herm, L4_male, extensible to L1_herm, dauer_herm, etc. as more variants become available. Keep this vocabulary consistent — it's what list_cengen_datasets()'s stage/sex columns are built from.

Pin metadata

Every pin should be written with this metadata (via pin_write(..., metadata = list(...))), so list_cengen_datasets() can show useful info without downloading the full table:

list(
  stage          = "adult",           # "L1" | "L4" | "adult" | "dauer" | ...
  sex            = "hermaphrodite",   # "hermaphrodite" | "male"
  dataset_label  = "adult hermaphrodite (CeNGEN)",
  source_version = "...",             # which upstream analysis/paper/run this traces to
  date_prepared  = "YYYY-MM-DD",
  n_cell_types = 128L,
  n_genes        = 20191L,
  prepared_by    = "...",
  notes          = "gene IDs are WormBase Gene IDs; avg_expr is ..."
)

Per-cell data

For each dataset, {name}_cells.rds (e.g. L4_herm_cells.rds) is a GitHub Release asset — see the percell-data-v1 release for the current download URLs. Each file is a single readRDS()-loadable list:

element	contents
$counts	dgCMatrix, genes (WormBase Gene IDs) x cells (barcodes)
$cell_metadata	data frame, one row per cell (rownames match $counts colnames): cell type/tissue annotation, barcode, cluster assignment, UMAP coordinates, QC metrics — exact columns vary by dataset, see colnames()
$gene_map	data frame: gene_id (WormBase) <-> gene_name (symbol) <-> seqnames lookup

Load one directly over HTTPS, no pins or GitHub auth required:

url <- "https://github.com/cengenproject/cengen-reference-data/releases/download/percell-data-v1/L4_herm_cells.rds"
tmp <- tempfile(fileext = ".rds")
download.file(url, tmp, mode = "wb")
x <- readRDS(tmp)
x$counts[1:5, 1:5]
table(x$cell_metadata$Cell.type)

To publish a new or updated per-cell dataset: build the same three-element list, saveRDS(x, "name_cells.rds"), then either upload it to the existing percell-data-v1 release (gh release upload percell-data-v1 name_cells.rds --repo cengenproject/cengen-reference-data --clobber to replace, or without --clobber to add) or cut a new release (e.g. percell-data-v2) if the schema changes in a way that breaks old consumers, and update the URLs above.

Publishing a new (or updated) aggregate dataset

Publishing is a local, git-based step — not done through this README's read-only board_url() path:

# 1. Clone this repo locally, then from inside it:
board <- pins::board_folder(".", versioned = TRUE)

# 2. Build `data` as the tidy long table described above, then:
pins::pin_write(
  board, data,
  name = "adult_herm", type = "rds",
  title = "CeNGEN adult hermaphrodite reference (unthresholded)",
  metadata = list(
    stage = "adult", sex = "hermaphrodite",
    dataset_label = "adult hermaphrodite (CeNGEN)",
    source_version = "...", date_prepared = Sys.Date() |> as.character(),
    n_cell_types = length(unique(data$cell_type)),
    n_genes = length(unique(data$gene)),
    prepared_by = "...", notes = "..."
  )
)

# 3. Regenerate the manifest that board_url() readers rely on:
pins::write_board_manifest(board)

# 4. Commit and push
git add -A
git commit -m "Add/update adult_herm dataset"
git push

Verify the round trip before considering it done:

board <- pins::board_url("https://raw.githubusercontent.com/cengenproject/cengen-reference-data/main/")
pins::pin_list(board)          # new/updated dataset should appear
pins::pin_read(board, "adult_herm")