RFC: GeneBe Database Format for Genetic Variant Annotations

[pstawinski]

RFC: GeneBe Database Format for Genetic Variant Annotations

Introduction

GeneBe Hub Database format is a repository format for genetic variant annotations, designed to store and query genomic data efficiently. The format leverages the Parquet file format, known for its wide support and efficient columnar data storage. This RFC discusses the structure of the GeneBe database format, its usage, and its limitations.

As we have been working with variant analysis and annotation for a long time, we typically annotate our variants based on:

1. The variant itself, where the chromosome, position, and change must match.
2. The position of the variant, where the chromosome and position must match.
3. The range, where the chromosome and position must intersect with the annotation range.
4. The gene in which the variant is located.

Therefore, we currently distinguish four types of annotations in GeneBe:

• Variant annotations
• Position annotations
• Range annotations
• Gene annotations (not yet implemented)

Format Specification

GeneBe uses Parquet for data storage. It is a columnar storage format. Columns with names starting with "_" are special columns (key fields) used for indexing annotations by variant, position, or range.

Key Field Details

• _seq (String): Chromosome name, without the "chr" prefix, e.g., 1, 2, X, Y, M.
• _pos (Integer, 32-bit): 0-based position.
• _del (Integer, 32-bit): Number of bases deleted.
• _ins (String): Insertion sequence (ACTG).
• _start (Integer, 32-bit): 0-based start of the range, inclusive.
• _end (Integer, 32-bit): 0-based end of the range, exclusive.

GeneBe encodes variants following the SPDI (Sequence, Position, Deletion, Insertion) format but with several modifications:

• Sequence (_seq): Chromosome name (e.g., 1, 2, ..., X, M for mitochondrion).
• Position (_pos): 0-based position, reflecting the base pair position in the sequence.
• Deletion (_del): The number of deleted bases in the reference.
• Insertion (_ins): The insertion sequence (composed of ACTG bases only, in uppercase).

For example, the VCF variant chr1-1000-AG-TT would be encoded as 1:999:2:TT.

Key Fields by Database Type

• VARIANT databases: Must contain the columns _seq, _pos, _del, and _ins. The combination of _seq, _pos, _del, and _ins must be unique. Used for annotating the exact same SNVs, MNVs, and small indels as seen before. Example database: ClinVar.
• POSITION databases: Must contain the columns _seq and _pos. The combination of _seq and _pos must be unique. Used for representing annotations of genomic positions. Example databases: phyloP100way, coverage depth in GnomAD.
• RANGE databases: Must contain the columns _seq, _start, and _end. Example databases: CCRS.
• GENE databases: NOT YET DESIGNED. It will likely require a gene identifier such as Ensembl, RefSeq, or HUGO ID. Gene symbols, while human-friendly, change over time, which could lead to hard-to-detect errors in the future.

Justification

• Chromosome names should be simple, short, and human-readable, hence 1 is preferred over chr1, and M over MT.
• Encoding variants with deletion length and insertion sequence instead of the VCF-style reference:alternate format is:
  1. More storage-efficient, reducing data size.
  2. More resistant to local specifications, as some reference sequences use IUPAC codes while others do not.
• We use 0-based position encoding in variant notation to align with SPDI notation.
• We use 0-based start and end encoding in ranges to align with the BED format.

File Format and Partitioning

GeneBe databases are stored in Parquet format and partitioned using the _seq column. The Parquet files are sorted by:

• VARIANT databases: _seq, _pos, _del, _ins.
• POSITION databases: _seq, _pos.
• RANGE databases: _seq, _start, _end.

Justification

• Parquet is widely supported across many languages, efficiently compressed, and enables fast point lookups when sorted by query column.
• Hive partitioning optimizes point lookups, facilitates sorting by position, and enables memory-efficient whole-genome processing.
• Alternative formats considered:
  • TSV, CSV, JSON – Readable but inefficient when compressed and slow to query.
  • SQLite – Fast for point queries but unsuitable for large whole-genome annotations like CADD due to file size constraints.
  • VCF – Supports tabix for fast lookups and BGZIP for compression but lacks flexibility.

Partitioning and File Naming

• Databases must be Hive-partitioned on _seq.
• Parquet files must use the .parquet extension and be compressed with zstd.
• File names must follow the pattern [a-zA-Z0-9.@-]+ and avoid reserved substrings like ".." and symlinks.

Database Versioning

GeneBe databases use Semantic Versioning (SemVer):

• Major version increments for structural changes (e.g., field removals).
• Minor version increments for new columns.
• Patch version increments for content updates (e.g., new annotations).

If a database follows another versioning system (e.g., ClinVar's date-based versioning), use the pre-release version in SemVer, e.g., 1.0.0-20241201.

Justification

Semantic Versioning is widely adopted. While we could adopt original database versioning (e.g., dbSNP 156, GnomAD 4.1.0), this wouldn't convey backward compatibility of column names. SemVer with pre-release tags ensures safe updates.

Database Metadata

Metadata is stored in local database storage alongside Parquet data:

• README.md – Human-friendly database description, also displayed on GeneBe Hub.
• LICENSE.txt – License information for the database.
• description.toml – Human- and machine-readable metadata including genome version, species, database type, and column descriptions.

Justification

• Some databases will have custom licenses, requiring LICENSE.txt.
• TOML is preferred over YAML/JSON for its readability and structured format.

Discussion

Several aspects need further consideration:

• Annotation types: Gene/Transcript annotations are not yet implemented. Key questions include index selection (e.g., Ensembl vs. RefSeq) and whether gene symbols should be used despite their instability.
• Performance and Optimization: Parquet is not optimal for high-speed point lookups. It may be beneficial to convert the columnar representation to a row-based format with a hash index using engines like SQLite or RocksDB. This conversion could be client-side, allowing Parquet to coexist with an optimized local format. However, is this necessary?

Try it yourself

To see format in action follow the steps from GeneBe Hub getting started guide, download the client and the annotations. You can also read more about the format and opening the downloaded annotations on dedicated GeneBe Hub Format page.

Call for Feedback

This is an early draft of the GeneBe database format. We encourage feedback, suggestions, and concerns to refine the format before moving the implementation from alpha to beta 😊.

[pstawinski]

First Round of the RFC is Over – Thank You!

The first round of the RFC has officially concluded. A huge thank you to everyone who shared their ideas via private messages! Your input has been greatly appreciated.

While this round is over, the discussion remains open—if you still have suggestions, feel free to send them.

A summary of the first round will be shared soon. Stay tuned!