Relational Transformers: Prediction and Fine-Tuning over Related Data

This framework provides an easy method to run and train relational transformer models over user-provided embeddings. It can be used to make predictions from sets of related cells (quickstart), measure which parts of a context affect those predictions (ablation), fit lightweight task heads (training overview), or fine-tune a complete model (training overview). This supports binary and multiclass classification, regression, forecasting, multilabel ranking, and other prediction tasks over related data.

Unlike frameworks that start with raw text or tables, Relational Transformers starts with embeddings that you have already created. You choose how strings, numbers, timestamps, images, categories, and domain objects become vectors. The framework handles typed relations, batching, relational attention, training, evaluation, and model checkpoints. It never silently downloads an encoding model or couples your model to a particular database.

your data → your encoders → embeddings + relations → RelationalTransformer → predictions

Pretrained models, fitted task heads, and fine-tuned checkpoints can be shared through the Hugging Face Hub. Each model declares its required embedding space, input dimension, and relation vocabulary in its model card. You can use a model as published, fit a small head over its frozen cell states, or fine-tune the complete relational transformer for your own feature pipeline.

For the full documentation, see Relational Transformers Documentation.

Installation

We recommend Python 3.10+ and PyTorch 2.2+.

pip install -U relational-transformers sentence-transformers

See Installation in the docs for source and editable installs and the ONNX, Triton, documentation, and development extras.

Getting Started

See Quickstart in our documentation.

Relational Prediction Models

First load a pretrained Relational Transformer and the encoder your application uses for text-valued cells.

from sentence_transformers import SentenceTransformer
from relational_transformers import RelationalTransformer

text_encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L12-v2")
model = RelationalTransformer()

Suppose your application needs to classify whether a GitHub issue is a bug. RT-J text cells have two separately encoded channels: the column and its value. Concatenate those two embeddings to build each model-ready cell vector. The masked target has a column embedding and a zero value channel.

import numpy as np

issue = {
    "title": "Database connections time out after 30 seconds",
    "body": "The pool stops returning connections after the service has been idle.",
    "latest_comment": "Restarting the process temporarily fixes it.",
}

def text_cell_vector(column, value):
    column_vector = text_encoder.encode(column)
    value_vector = text_encoder.encode(value)
    return np.concatenate([column_vector, value_vector])

def target_cell_vector(column):
    column_vector = text_encoder.encode(column)
    masked_value_vector = np.zeros(384, dtype=np.float32)
    return np.concatenate([column_vector, masked_value_vector])

bug_target_vector = target_cell_vector("is bug")
title_vector = text_cell_vector("title", issue["title"])
body_vector = text_cell_vector("body", issue["body"])
comment_vector = text_cell_vector("latest comment", issue["latest_comment"])

cell_vectors = np.stack([
    bug_target_vector,
    title_vector,
    body_vector,
    comment_vector,
])

probability = float(model.predict(cell_vectors, target=0))
print(f"P(issue is a bug) = {probability:.1%}")

And that's already it. RT-J receives only the [column_embedding, value_embedding] vectors; it never receives the issue dictionary or its strings. target=0 marks bug_target_vector as the masked prediction target. RelativeDB will construct this same model-ready representation from its schema and retrieved context before calling this library. Pass a list of vector arrays to batch several issues. See Encoding Cells for the full typed-cell contract.

Ablation

Ablation is just the same prediction with context deliberately removed. Here we remove the comment vector, then run the full and ablated contexts together.

without_comment = np.delete(cell_vectors, 3, axis=0)
full, ablated = model.predict(
    [cell_vectors, without_comment],
    target=0,
)

print(f"with latest comment:    {full:.1%}")
print(f"without latest comment: {ablated:.1%}")
print(f"change:                 {ablated - full:+.1%}")

A large change means the removed cell was load-bearing context; a change near zero means it was not affecting this prediction. Nothing automatically decides what to remove—you define the ablation that answers your question and compare its prediction with the original.

Pre-Trained Models

Pretrained RT-J models and deployment artifacts are available from RelativeDB on the Hugging Face Hub. Each weight repository contains both classification/ and regression/ checkpoints. Classification is loaded by default; select the regression checkpoint with RelationalTransformer(..., task="regression").

• RelativeDB/rt-j-fp16 — half-precision weights
• RelativeDB/rt-j-fp8 — native E4M3 FP8 matrix weights
• RelativeDB/rt-j-int8 — 8-bit quantized weights
• RelativeDB/rt-j-int4 — 4-bit quantized weights
• RelativeDB/rt-j-onnx — full RT-J export for ONNX Runtime
• Prediction
• Batches and the Model Input Contract
• Custom and Local Models

The published configs specify RT-J's 384-wide text input, 512-wide hidden states, 12 transformer blocks, 8 attention heads, and expected all-MiniLM-L12-v2 embedding space. Matching d_text=384 alone is not an interoperability guarantee: inputs must use the embedding model, normalization, semantic conventions, and relational structure documented by the checkpoint. For a different embedding space, train an input adapter or fine-tune a checkpoint with appropriate data.

Backends

The same constructor selects portable PyTorch, optimized Triton CUDA, ONNX Runtime, or a zero-allocation meta model.

# CPU, MPS, or CUDA; supports inference and training
model = RelationalTransformer("RelativeDB/rt-j-fp16", backend="torch")

# CUDA inference through the optimized relational-attention kernels
model = RelationalTransformer("RelativeDB/rt-j-fp16", backend="triton")

# Inspect dimensions and modules without allocating 85 million parameters
model = RelationalTransformer("RelativeDB/rt-j-fp16", backend="meta")
print(model.get_model_kwargs())

The published ONNX model downloads automatically from Hugging Face:

onnx_model = RelationalTransformer("RelativeDB/rt-j-onnx", backend="onnx")
predictions = onnx_model.predict(batch)

RelativeDB/rt-j-onnx is also the default when you omit the model name and select backend="onnx".

You can also export a loaded PyTorch checkpoint and open the local result:

torch_model = RelationalTransformer(device="cpu")
torch_model.export_onnx("rt-j.onnx", example_batch)
onnx_model = RelationalTransformer("rt-j.onnx", backend="onnx")

The release pipeline exports the published RelativeDB/rt-j-fp16 checkpoint—not a reduced test model—and checks dynamic batch and context lengths for numerical parity before attaching the ONNX file to the GitHub release. The fast test suite separately exercises the same path with a small deterministic checkpoint.

See Backends for supported devices, ONNX dynamic axes, and Triton limitations.

Training

This framework allows you to adapt relational transformer models to your own feature pipeline and task. You can fit a small multiclass or multilabel-ranking head over a frozen backbone, fine-tune the complete model for scalar binary or regression tasks with RelationalTrainer, or use the model in an ordinary PyTorch loop.

• Task-Head Tuning
  • Frozen-Backbone Head Tuning
• Full-Model Fine-Tuning
  • Full-Model Fine-Tuning
• Custom Models
  • Custom Models and Local Checkpoints

A frozen-backbone head is the fastest adaptation path. Each training input is encoded once, then only the selected task head is optimized.

from relational_transformers import RelationalExample

head_dataset = [
    RelationalExample(input=issue_a_batch, label=2),
    RelationalExample(input=issue_b_batch, label=0),
]

model = RelationalTransformer("RelativeDB/rt-j-fp16")
head = model.fit_head(
    head_dataset,
    task="issue_label",
    num_labels=5,
    problem_type="multiclass",
    epochs=100,
    learning_rate=1e-3,
)
head.save_pretrained("models/issue-label-head")

Use full-model fine-tuning when the relational backbone itself must adapt:

from relational_transformers import (
    RelationalExample,
    RelationalTrainer,
    RelationalTrainingArguments,
)

train_dataset = [
    RelationalExample(input=customer_a_batch, label=1.0),
    RelationalExample(input=customer_b_batch, label=0.0),
]

model = RelationalTransformer("RelativeDB/rt-j-fp16")
args = RelationalTrainingArguments(
    output_dir="models/customer-churn",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    learning_rate=2e-5,
)

trainer = RelationalTrainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    task="churn",
)
trainer.train()

On CUDA, set training_backend="triton" in RelationalTrainingArguments to compile the trainable PyTorch graph through TorchInductor's Triton code generation. The optimized backend="triton" constructor remains the lower-latency inference path.

Some highlights across the different types of training are:

• User-provided embeddings for text, numbers, categories, images, and other modalities
• Typed sparse relations and variable-length relational inputs
• Frozen-backbone multiclass and multilabel-ranking head tuning
• Full-model fine-tuning for scalar binary and regression tasks
• Binary, multiclass, multilabel, regression, forecasting, and ranking objectives
• Multi-task adaptation through named prediction heads
• Ordinary PyTorch modules and optimizers for custom training loops

Application Examples

The examples directory contains complete, runnable workflows:

• Issue prediction builds externally encoded column/value vectors and predicts a real label.
• Batched prediction scores variable-length contexts together.
• Typed customer churn constructs scalar, text, table, node, and foreign-key tensors explicitly.
• Support-history ablation compares caller-defined contexts.
• Evaluation combines classification and ablation metrics.
• Task-head tuning trains a multiclass issue head over a frozen backbone.
• Full fine-tuning adapts the complete model with mini-batches.
• ONNX export, meta inspection, and Triton FP8 inference cover deployment workflows.

RelativeDB is the first real-world integration: it retrieves related rows, constructs typed RelationalBatch inputs, and selects a supported serving path.

Companion Resources

• RelativeDB models on Hugging Face
• RelativeDB, the database retrieval and context-construction integration

Development setup

After cloning the repository (or a fork), install it in editable mode with the development dependencies:

python -m pip install -e ".[dev]"

To test your changes, run:

pytest

This runs deterministic, offline tests over typed customer, order, and support contexts. To validate the published Hugging Face checkpoints or compare Triton with PyTorch on CUDA, see the Testing guide.

To build the documentation, run:

make docs

Citing & Authors

If you find Relational Transformers useful in your research or application, you can cite the software:

@software{relational_transformers_2026,
    title = {Relational Transformers: Prediction and Fine-Tuning over Related Data},
    author = {{RelativeDB}},
    year = {2026},
    url = {https://github.com/RelativeDB/relational-transformers},
}

Don't hesitate to open an issue if something is broken or if you have questions about using your own embedding pipeline.

Maintainers

Relational Transformers is maintained by RelativeDB.

License

Relational Transformers is licensed under the Apache License 2.0.