CRISPRme v2.3.2

Fixed

• Self-built dict-less indexes are now self-complete for samplesIDs.
  build-index-only (with --samplesID) writes the per-dataset and the combined
  <vcf>.samplesID.txt into the install (reusing the one union implementation), and
  publish-index bundles them into the index tarball — so a fresh download --index-name <X> + search works with no separate --what samples/--what all
  dependency (the download-side synthesis added in 2.3.1 becomes a pure fallback).
  No-op for single-dataset, reference, and dict-based indexes.

Documentation

• Rewrote the index-build docs for the dict-less flow. docs/PRECOMPUTED_INDEXES.md
  now covers build-index-only --vcf --samplesID (required to emit the Tier-0
  registry + Tier-1 genotype tiers — without it you get a dicts-only index),
  publish-index --dictless, the separate genotypes_<vcf>.tar.gz, download --no-genotypes, the -dictless canonical install-name behavior, and the updated
  manifest.json fields (with a caveat that name-parsed fields are reference-index
  only). Added the variant-index build path to README §2.2.9, crisprme_data_setup
  §3.5, and SETTINGS_DATA_MANAGER.md; fixed an NGG→NRG example nit.

CRISPRme v2.3.1

Fixed

• Dict-less download --what index is now self-sufficient. A standalone
  download --what index for a merged variant index (e.g.
  NRG_3_hg38-dictless+hg38_1000G_HGDP) now fetches the per-dataset samplesID
  lists it needs before synthesizing the combined <vcf>.samplesID.txt, so it no
  longer requires a prior download --what all/--what samples. No-op when those
  files are already present (the --what all → --what index path is unchanged).
• Restored the unit-tests CI gate. A colon in a workflow step name
  ((#172: fakechrom match)) made GitHub Actions reject unit-tests.yml at startup
  (a 0 s "workflow file issue"), so the unit-tests job had silently not run since the
  v2.3.0 merge (the tests themselves pass locally). Quoting the name fixes the parse;
  the full 22-step unit suite runs in CI again.

CRISPRme v2.3.0

The dict-less variant-analysis engine: replaces the ~152 GB per-sample SNP
dictionaries with compact, memory-mapped tiers, adds population-level off-target
summaries, and corrects the allele-frequency column. Backward-compatible — a
dict-based install still works unchanged.

Added

• Compact dict-less variant post-analysis. A tiny, always-shipped Tier-0
  registry (per-(pos,alt) AC/AN + per-(db×subpop)/global allele counts + rsID,
  mmap+bisect) powers off-target detection and corrected allele frequencies out of
  the box; an optional Tier-1 genotype store reconstructs the exact per-sample
  Samples column. Together they replace the 152 GB per-sample dicts with ~7 GB
  (registry) + ~22 GB (genotype tier), random-access — SNP post-analysis no longer
  streams 152 GB of per-sample JSON. build-index-only emits the tiers; the search
  auto-detects and uses them (falling back to dicts when present).
• Combination-aware population-summary output — a <output>.population_summary.tsv
  companion per variant off-target with per-database, per-superpopulation and global
  allele/carrier frequencies, max-subpopulation AF (+label), homozygote counts and
  absolute Ns, with dataset provenance preserved (never conflates 1000G vs HGDP).
  Phased datasets use exact cis co-occurrence; unphased report an assume-cis upper
  bound + a labeled lower bound.
• Dict-less HuggingFace distribution. publish-index --dictless ships the
  registry in the main index tarball + the genotype tier as a separate optional
  genotypes_<vcf>.tar.gz; download fetches the genotype tier by default
  (--no-genotypes to skip). Published NRG_3_hg38-dictless+hg38_1000G_HGDP on
  lucapinello/crisprme-data (the dict-based NRG_3/NGG_3 indexes remain the
  default). Fetch it with download --what all first, then download --what index,
  so the per-dataset samplesID lists are on disk when the combined
  <vcf>.samplesID.txt is auto-synthesized (a standalone --what index without a
  prior samples fetch leaves the combined file absent — hardening tracked as a
  follow-up).

Fixed

• Corrected allele frequencies. The AF column — empty/mis-polarized for ~95 % of
  variants in the dict format (a documented 2.2.0 limitation) — is re-derived from
  AC/AN over the full panel.
• Out-of-the-box dict-less install. The skip-enrichment gates
  (submit_job_automated_new_multiple_vcfs.sh, validate_inputs.py) now accept a
  registry_<vcf>/ tier in place of dictionaries_<vcf>/; download installs a
  -dictless-named index under its search-resolvable canonical name and synthesizes
  the combined <vcf>.samplesID.txt from the per-db files. Verified end-to-end: a
  fresh dict-less download finds the CPS1 off-target (chr2:210530658, CFD 0.947,
  rs114518452) with Samples reconstructed from the genotype tier and a populated
  population summary.

CRISPRme v2.2.0

First stable release of the CRISPRme+ 2.2.0 line (Python 3.11 / Dash 2.x). It
consolidates the 2.2.0-alpha.27–alpha.30 pre-releases (see those sections
below for the full history); the entries here are the changes since alpha.30.

Fixed

• Phased multi-SNP haplotype off-targets are no longer under-reported. In the
  phased post-analysis path — the default for phased datasets such as 1000G/HGDP —
  iupac_decomposition failed to assemble the full-haplotype off-target when one
  sample carried ≥4 co-occurring variants in a single protospacer window: the
  true worst target (fully substituted, lowest-mismatch/highest-CFD) was silently
  under-reported or dropped. The in-loop level-0 subtraction (a dedup device that
  starved the deeper combination layers) is replaced with a deferred peel that
  runs after the full lattice is built, so the maximal cis combination forms while
  per-haplotype attribution stays deduplicated. Validated byte-identical for every
  previously-correct case (unphased, phased ≤3-variant, single-SNP, >cap greedy)
  plus a real-data genome-wide no-regression diff. (#41; a long-standing defect
  inherited from classic CRISPRme, tracked there at pinellolab/CRISPRme#175.)

Changed

• environment.yml now includes ijson, so from-source installs get the streaming
  low-RAM SNP-dict reader that the Docker image already had (without it, SNP
  post-analysis falls back to a whole-file json.load; results are identical, but
  RAM use and the OOM-guard estimate are higher).

Added

• Graphical Settings / Data Manager page in the web interface: add reference
  genomes (UCSC by assembly name — e.g. the pig susScr11 — HuggingFace, or a
  direct URL), precomputed indexes (download from HuggingFace or build locally
  from an installed genome + PAM), VCF datasets (HuggingFace or register an
  existing server folder), annotations (BED upload), and nucleases/PAMs (a small
  form). New data lands in the local data folder and is auto-discovered by the
  search form. Long operations run as detached jobs on a dedicated executor with
  live progress, so they never starve the search slots. Mutations are local-mode
  only; publishing an index to the shared HuggingFace repo is maintainer-only.
  download --what genome gained --source {hf,ucsc,url} (+ --url) so the CLI
  and web share one non-human-genome download path.
• Python 3.11 modernization: pipeline fixes for pandas 2.x / matplotlib 3.x and
  a Dash 1.x → 2.x web-app migration, plus a Python-3.11 Docker image built from
  source (CRISPRitz 2.8.1) (#131).
• assembly-search subcommand: off-target search on a personal diploid genome
  assembly (two haplotypes, no VCF), reconciled to hg38 via liftOver (#113).
• Reference-index UX: build-index-only pre-builds the reusable CRISPRitz
  reference index without running a search, and complete-search --index-path
  reuses a prebuilt/staged index library (a missing index is a hard error rather
  than a silent rebuild).
• HuggingFace data distribution: download fetches reference data (genome,
  annotations, PAMs, sample IDs, VCFs, precomputed indexes) from a HuggingFace
  dataset repository over its CDN, and publish-index uploads a locally built
  index for reuse. Default repo lucapinello/crisprme-data, overridable via
  --hf-repo / CRISPRME_HF_REPO. setup/complete-test also try HuggingFace
  first and fall back transparently to the original UCSC/EBI/Sanger sources
  (#140, #141).
• complete-search --max-total-edits N: cap the total edits (mismatches +
  bulges) per reported alignment; over-cap targets are dropped right after the
  search, shrinking intermediate files and post-analysis time (#107).

CI

• New unit tests workflow: fast, hermetic byte-compile + network-free HF/index
  unit tests on every code PR.
• New web e2e (playwright) workflow: builds the py3.11 image, serves the web
  app, and drives Chromium to assert every Dash 2.x page renders (no blank pages
  / JS errors).
• validate-benchmarks gained a new-subcommand dispatch + unit-test smoke step.

Changed

• Clearer failure reporting: when a search fails, CRISPRme now prints which
  stage failed (from the per-stage log) and the last lines of the error log,
  instead of only "run failed — see log_error.txt". Makes failures actionable
  for non-expert users.

Fixed

• Web interface (Dash 2.x) hardening, from a full Playwright stress test of the
  running app:
  • The web server no longer crashes on a from-source install. Dash 2.x's
    app.run() lets the HOST environment variable override the host argument,
    and the from-source conda env sets HOST to a non-bindable compiler build
    triple (x86_64-conda-linux-gnu); the server now forces the intended host/
    port so it binds correctly.
  • The Query Genomic Region and Personal Risk Cards result tabs no
    longer return HTTP 500. Both callbacks type-checked their inputs before the
    "no click yet" guard, so Dash's initial (empty) render raised a TypeError;
    the guard now runs first, and Filter/Generate with nothing selected is a
    graceful no-op.
  • Removed the dead cross-origin "skeleton" stylesheet (blocked by browsers on
    every page); the layout already uses the Bootstrap grid.
  • The nuclease dropdown collapses case-variant duplicate PAM files so each
    nuclease is listed once.
• Zero-hit searches now complete cleanly with an empty result instead of
  aborting. A search that finds no off-targets (e.g. a very stringent
  guide/parameter combination) previously failed part-way through post-analysis
  ("off-targets post-analysis (reference) failed", then a cascade through the
  rsID / summary / integration steps, all of which assumed at least one target).
  Added a zero-target guard to the reference SNP post-analysis (mirroring the
  INDELs one), made remove_n_and_dots.py tolerate a header-only report, and
  added a high-level short-circuit that emits an empty-but-valid result and exits
  0 when no off-targets are found. Verified end-to-end on ml007 (empty result
  exits 0; a normal with-hits search is unaffected).

Documentation

• New docs/DOCKER_QUICKSTART.md + a README quickstart callout: a few-command,
  no-conda / no-410 GB path to the web interface for non-experts — fast HF data
  download, a prebuilt index, then web-interface in the browser, with
  troubleshooting and "install more indexes as needed".
• Data-setup guide: documented what setup produces (including the combined
  1000G+HGDP config files), the HuggingFace fast-download path, and a new
  "Prebuild, reuse, and share the reference index" section.
• README: added a from-source install path and a reference-index /
  data-distribution commands section.
• Documented the variant PAM behaviour in docs/INPUT_FORMATS.md: PAM creation
  (variants that add an off-target) is reported; PAM disruption (variants that
  remove a reference PAM for carriers) is a known, intentional gap — with the
  rationale (disruption lowers rather than raises predicted risk and would need
  sample-specific, non-scorable semantics).

CRISPRme v2.2.0-alpha.30

Fixed

• Indel off-targets were silently dropped since v2.1.9. post_analisi_indel.sh
  subset the per-chromosome indel targets with grep -F -w $chrom, but the indel
  search runs on a per-chromosome fake genome (pool_search_indels.py searches and
  names its targets fake<chrom>), so every indel target row's Chromosome column is
  fake<chrom> (e.g. fakechr22) — which grep -F -w chr22 can never match (-w's
  left word boundary fails because chr22 is preceded by the word character e). The
  per-chromosome subsets came back empty, the indel post-analysis processed nothing,
  and all indel off-targets were dropped with a clean exit. Both grep lines now
  match on "$fakechrom" (-w still prevents fakechr2 matching fakechr22); the
  NF >= 10 malformed-line guard that the same refactor had dropped is restored; and
  an empty subset now prints a WARNING to stdout (stderr is fatal in this pipeline) so
  the failure can never be silent again. A hermetic regression test
  (test_indel_chrom_subset.py) is added and wired into CI. SNP post-analysis is
  unaffected (SNP targets use real chromosome names). Thanks to @munchr-gene1 for the
  report and diagnosis (#172).

CRISPRme+ 2.2.0-alpha.9

Alpha release of the Python 3.11 / Dash 2.x CRISPRme+ line for beta users.

Docker: docker pull pinellolab/crisprme:v2.2.0-alpha.9 (multi-arch: amd64 + arm64)

Added

• Annotation manager (Settings → Data Manager): per-genome Manage annotations
  (enable / disable) checklist backed by a persisted manifest, plus format
  validation on every uploaded annotation BED (≥4 tab columns, integer
  start ≤ end, whitespace-free label, size cap — malformed files are rejected
  with a clear message). Enabled tracks are merged on demand and applied
  automatically to every search.

Changed

• Annotations are now applied automatically from the enabled set rather than chosen
  per search. The annotation dropdown was removed from the search form (Step 3 is
  now email + job name); the built-in ENCODE cCREs (SCREEN) + DHS + GENCODE bundle
  is enabled by default, so annotations work out of the box.
• The Maximum edits slider is floored at 1 (the 0-edit on-target is always
  reported), avoiding an empty-result edge case on large precomputed indexes.
• Search-submission form: consistent left indentation for the PAM and
  variant-dataset dropdowns and the submit/example buttons.

CI

• test/web/test_annotation_manager.py (15 checks) wired into web-e2e.

CRISPRme+ 2.2.0-alpha.8

CRISPRme+ 2.2.0 open alpha — variant-aware CRISPR off-target nomination, batteries included. For production/clinical work use the frozen CRISPRme 2.1.14.

Quickstart

docker pull pinellolab/crisprme:v2.2.0-alpha.8

Then follow the batteries-included quickstart. Note: the batteries install needs ≈250 GB free disk (~44 GB download that expands to ~190 GB — the per-sample variant dictionaries are large) and 32 GB RAM (64 GB recommended) for the genome-wide variant search.

Changes since alpha.6

• Annotations ON by default (alpha.7): the built-in ENCODE cCREs (SCREEN) + DHS + GENCODE bundle now surfaces on a fresh install (the download ships it bgzipped; the option-check accepted only the unzipped form, so searches ran unannotated). It's now selected by default.
• Maximum-edits slider floored at 1 (alpha.8): 0 was a niche exact-matches-only mode that hit a crispritz --max-edits 0 bug (empty result on large indexes). The on-target (0 edits) is still reported at any setting (Total 0 ≤ 1), so nothing useful is lost.
• (alpha.7 also bundled pigz and shipped crispritz v2.8.2, whose -index search ignores hidden metadata sidecars — fixing the empty SNP/variant result on the batteries index.)

Upgrading

If you installed the combined index before this line, re-download it so it includes the per-sample dictionaries:

rm -rf <data>/genome_library/NGG_3_hg38+hg38_1000G_HGDP*  <data>/Dictionaries/*hg38_1000G_HGDP
docker run --rm -v "<data>:/DATA" -w /DATA pinellolab/crisprme:v2.2.0-alpha.8 \
  crisprme.py download --what index --index-name NGG_3_hg38+hg38_1000G_HGDP --path /DATA

🤖 Generated with Claude Code

CRISPRme+ 2.2.0-alpha.6

CRISPRme+ is the next major version of CRISPRme (2.2.0) — variant-aware CRISPR off-target nomination, batteries included. Open alpha for beta testers. For production/clinical work, stay on the frozen CRISPRme 2.1.14.

Quickstart

docker pull pinellolab/crisprme:v2.2.0-alpha.6

Then follow the batteries-included quickstart — one download fetches the reference data and a ready-made variant-aware SpCas9 index (1000 Genomes + HGDP), so you can run a real off-target search immediately (no index build).

What's fixed in this release

This is the first alpha where a variant search on a batteries-included install works end-to-end (index-only download, no raw VCFs):

• Batteries variant search (#7): the precomputed variant index now bundles its per-sample dictionaries, submit_job skips the redundant genome enrichment when a precomputed index is present, and the indel pipeline derives chromosomes from the bundled logs. A variant search no longer requires the multi-GB source VCFs.
• crispritz v2.8.2 (#8, CRISPRitz#42): the -index search now ignores hidden metadata sidecars, so the SNP/variant search returns results instead of erroring with "only .bin files". Also bundles pigz for fast index publishing.
• The hosted combined NGG_3_hg38+hg38_1000G_HGDP index was re-published with its dictionaries.

Validation

End-to-end genome-wide search on a real batteries install (downloaded index+dicts, no VCFs): enrichment skipped, precomputed index reused, SNP + variant + indel searches ran, post-analysis loaded the 152 GB dictionaries, full report + database built — 1,735 off-target sites produced. Also verified identical to a with-VCFs baseline on a chr22 harness.

Memory

16 GB is enough for a first run; the default genome-wide 1000G+HGDP variant search is memory-intensive — give Docker at least 32 GB (64 GB recommended).

🤖 Generated with Claude Code

CRISPRme+ 2.2.0-alpha.14

Alpha release for beta users. Docker: docker pull pinellolab/crisprme:v2.2.0-alpha.14 (multi-arch amd64 + arm64)

Fixed

• A resubmit no longer resurfaces a previously-FAILED job. Results are deduplicated by
  search parameters; the dedup reused any matching prior job, so re-running a search whose
  earlier attempt had failed (e.g. left by a since-fixed bug) returned that stale failure —
  shown as "The selected result encountered some errors, please remove it and try to submit
  again." Dedup now skips prior results that didn't finish cleanly (non-empty log_error.txt
  or no Job Done) and runs fresh. (Workaround on older images: delete the stale
  Results/<id>/ folder and resubmit.)

Changed

• Docs: removed the now-wrong "peak memory is in post-processing / can exceed 32 GB /
  ~64–100 GB" guidance (the alpha.12 streamed SNP-dict load eliminated that post-analysis
  spike); the 16 / 32 / 64 GB tiers stay.

CRISPRme+ 2.2.0-alpha.13

Alpha release for beta users. Docker: docker pull pinellolab/crisprme:v2.2.0-alpha.13 (multi-arch amd64 + arm64)

Performance follow-up to alpha.12's genome-wide post-analysis memory fix (no behavior change):

Changed

• ijson now uses the fast yajl2_cffi C backend (added yajl + cffi) instead of the
  pure-python fallback — much faster streaming of the per-chromosome SNP dictionary.
• Post-analysis runs chromosomes in parallel again. With streaming, per-worker peak RAM is
  small (chromosome genome string + queried subset), so the worker cap uses a modest fixed
  estimate when ijson is available instead of sizing from the (large) dictionary file — which
  had been serializing to a single worker. The INDEL pool likewise (it reads the small
  log_indels, never the big dictionary).

(alpha.12 remains the crash-fix release: genome-wide variant post-analysis OOM + the on-demand
INDELS double-prefix regression. alpha.13 only makes that post-analysis fast.)