Analyser Overview

A list of Analysers implemented in vitrivr-engine. This document is both, an overview for users of vitrivr-engine, but also a good starting point for potential contributors.

All descriptors, regardless of analyser do have two sub-fields (in this order): id (DescriptorId) and retrievableId (RetrievableId). The former is the unique id per descriptor, the latter the foreign key of the retrievable.

Core Module

Analysers from the core module.

Exif Metadata (StructDescriptor)

Analyser Name: ExifMetadata

Base Descriptor: StructDescriptor

The Exif Metadata analyser is capable of reading exif metadata.

Value:

A StructDescriptor has subfields and no value.

Sub-Fields:

The subfields of this analyser are configurable

Sub-Fields:

Sub-Field	Description	Type
startNs	The start point, relative to the beginning and in nanoseconds	Long
endNs	The end point, relative to the beginning and in nanoseconds	Long

Local Context Properties:

none

For exif metadata, the ExifMetadata Analyser can be used. For each exif tag that should be included, a parameter with the name "{EXIF_DIRECTORY_NAME}_{TAG_NAME}" must be set to a type. Keys that do not match an exif tag via the aforementioned pattern are interpreted to be custom metadata tags that are stored in the exif UserComment tag in JSON format. Each parameter corresponds to a sub-field. Here is an example with custom "time_zone" metadata:

        "exif": {
          "factory": "ExifMetadata",
          "parameters": {
            "ExifSubIFD_FocalLength": "INT",
            "ExifSubIFD_ApertureValue": "FLOAT",
            "ExifSubIFD_DateTimeOriginal": "DATETIME",
            "ExifSubIFD_MeteringMode": "STRING",
            "time_zone": "STRING"
          }
        }

For extraction, the exif UserComment of images might look like this:

{"time_zone": "Europe/Berlin", "hours_awake": 12}

File Source Metadata

Analyser Name: FileSourceMetadata

Base Descriptor: StructDescriptor

Provides the path of the source file, as well as its size.

Value:

A StructDescriptor has subfields and no value.

Sub-Fields:

Sub-Field	Description	Type
path	The absolut path of the file	String
size	The file's size in bytes	Long

**

Local Context Properties:

none

Video Source Metadata

Analyser Name: VideoSourceMetadata

Provides basic metadata of the video file, such as width and height (in pixels), frames per seconds and some.

Value:

A StructDescriptor has subfields and no value.

Sub-Fields:

Sub-Field	Description	Type
width	The width in pixels	Int
height	The height in pixels	Int
fps	The video's frames per second	Double
channels	The video's channels	Int
sampleRate	The sample rate	Int
sampleSize	The sample size	Int

Local Context Properties:

none

Temporal Metadata

Analyser Name: TemporalMetadata

Base Descriptor: StructDescriptor

Provides temporal information (e.g. of the segment)

Value:

A StructDescriptor has subfields and no value.

Sub-Fields:

Sub-Field	Description	Type
startNs	The start point, relative to the beginning and in nanoseconds	Long
endNs	The end point, relative to the beginning and in nanoseconds	Long

Local Context Properties:

none

Features Module

Analysers from the features module.

Feature Extraction Server Module

Analyser Name: one of: ASR, OCR, DenseEmbedding

Base Descriptor: one of: StringDescriptor (base: ScalarDescriptor), FloatVectorDescriptor (base: VectorDescriptor)

Analysers which wrap the Feature Extraction Server. Requires a running FES instance.

Value:

Value depends on concrete analyser

Sub-Fields:

none

Local Context Properties:

Property	Description
host	The host of the Feature Extraction Server microservice, including the protocol (which is either http:// or https://)
model	The name of the model to be used (optional, as there are default models defined).
length	For vector based analysers mandatory, specifying their length.

TorchServe Module

Analyser Name: TSEmbedding

Base Descriptor: FloatVectorDescriptor

Analyser for wrapping embeddings from a running TorchServe instance.

Value:

Embedding vector from the selected model.

Sub-Fields:

none

Local Context Properties:

Property	Description
host	The host running the TorchServe instance
port	The corresponding port for the TorchServe instance (inference_address)
model	The name of the model to be used (listed in e.g. http://127.0.0.1:8081/models)
dimensions	The length of vectors returned by the selected model

Example:

"clip": {
          "factory": "TSEmbedding",
          "parameters": {
            "host": "127.0.0.1",
            "port": "8080",
            "model": "clip_model",
            "dimensions": "512"
          }
        }

M3D Module