Introduce new token-level architectures

[Ingvarstep]

Add Token-Level Relation Extraction Architecture & Span Representation for Token Models

Summary

Introduces UniEncoderTokenRelexGLiNER - a new architecture combining token-level NER with relation extraction. Also adds support for span representations for token-level models and a new token decoder architecture.

Key Changes

Token-Level Relation Extraction:

• UniEncoderTokenRelexConfig - Configuration for token-level relation extraction
• UniEncoderTokenRelexModel - Core model with BIO-style entity tagging + relation prediction
• UniEncoderTokenRelexGLiNER - High-level API for joint entity and relation extraction
• TokenRelexDecoder - Decodes both entity spans (via BIO tags) and relations between them
• RelationExtractionTokenProcessor / RelationExtractionTokenDataCollator - Data processing
• UniEncoderTokenRelexORTModel - ONNX Runtime support

Span Representation for Token Models:

• New represent_spans method extracts span representations from token-level predictions
• Supported by both UniEncoderTokenModel and BiEncoderTokenModel
• Enables downstream tasks (e.g., relation extraction) on top of token-level NER

Token-Level Decoder Model:

• UniEncoderTokenDecoderConfig - Token-level NER with label generation decoder
• Combines BIO tagging with generative label decoding

Usage

from gliner import GLiNER

# Token-level relation extraction
model = GLiNER.from_pretrained("path/to/token-relex-model")

# Or via config
config = UniEncoderTokenRelexConfig(
    model_name="microsoft/deberta-v3-small",
    relations_layer="biaffine"
)
model = UniEncoderTokenRelexGLiNER(config)