Data Contracts in OpenMetadata: JSON Schema Design

[harshach]

Introduction

Data contracts formalize an agreement between data producers and consumers about what to expect from a data asset’s data. They capture the structure, semantics, quality, and SLAs of data in a machine-readable way, similar to an API contract but for data. In essence, an DataContract is enforceable in the data ecosystem to bring standardization, control, and reliability

OpenMetadata, as a metadata platform, can integrate this concept by introducing a DataContract entity defined via JSON Schema. This allows OpenMetadata admins and data product owners to attach a contract to any data asset (Table, Topic, ML Model, Dashboard, Data Product, etc.), codifying expectations in a structured format. The contract can then be enforced or validated using OpenMetadata’s existing metadata and data quality frameworks. The goal is to achieve “contextually rich, high-quality, well-governed data that is trustworthy”
by making data expectations explicit and automating their enforcement.

DataContract Entity Schema Design

We propose a JSON Schema definition for a new DataContract entity in OpenMetadata. This schema will define the contract’s structure and allowed fields.
The contract covers four main categories of expectations

1. Schema
2. Semantics
3. Security
4. Business Assertions (data quality).

We also include an SLA section for service-level agreements and an Ownership field for accountability. Each DataContract is designed to represent one single data asset (dataset, topic, model, etc.) in a well-structured, templated format

Below is an outline of the JSON Schema for the DataContract entity

 {
  "$id": "https://openmetadata.example/schema/datacontract/datacontract.json",
  "type": "object",
  "title": "DataContract",
  "description": "A contract specifying expectations for a data asset.",
  "properties": {
    "id": { "type": "string", "format": "uuid" },
    "name": { "type": "string" },
    "description": { "type": "string" },
     "incidentManagement": { "type": "boolean" },
     "owners": {
          "description": "Owners of this Table.",
          "$ref": "../../type/entityReferenceList.json",
          "default": null
     },
    "targetEntity": {
      "description": "The data asset this contract applies to",
       "$ref": "../../type/entityReference.json"
    },
    "schema": {
      "type": "object",
      "description": "Expected schema structure of the data asset",
      "properties": {
        "fields": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name":   { "type": "string" },
              "type":   { "type": "string" },
              "nullable": { "type": "boolean", "default": true },
              "description": { "type": "string" },
              "constraints": {
                "type": "array",
                "items": { "type": "string" },
                "description": "Integrity constraints (e.g. UNIQUE, FOREIGN_KEY to another asset)"
              }
            },
            "required": ["name", "type"]
          }
        },
        "strict": {
          "type": "boolean",
          "description": "If true, the data asset’s schema must match exactly these fields (no extra columns).",
          "default": false
        }
      },
      "required": ["fields"]
    },
    "semantics": {
      "type": "object",
      "description": "Business semantics and documentation requirements",
      "properties": {
        "entityDescriptionRequired": { "type": "boolean", "default": true },
        "columnDescriptionRequired": { "type": "boolean", "default": false },
        "tags": {
          "type": "array",
          "items": { "type": "string" },
          "description": "List of tags or glossary terms that must be applied to this asset (or its columns)."
        },
        "customIntegrityRules": {
          "type": "array",
          "items": { "type": "string" },
          "description": "Additional business logic rules (expressed in plain language or as expressions) that must hold true."
        }
      }
    },
    "security": {
      "type": "object",
      "description": "Security and access policy expectations",
      "properties": {
        "accessPolicy": { "type": "string", "description": "Reference to an access policy ID or name that should govern this data" },
        "dataClassification": { "type": "string", "description": "Expected data classification (e.g. Confidential, PII, etc.)" }
      }
    },
    "quality": {
      "type": "object",
      "description": "Data quality tests and assertions",
      "properties": {
        "tests": {
          "type": "array",
          "items": {
            "$ref": "../type/entityReference.json",
            "description": "Reference to a TestCase or TestDefinition that should be enforced"
          }
        },
        "expectations": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "expression": { "type": "string", "description": "A business-friendly description of the quality expectation (could be translated to a test)." },
              "raiseIncident": { "type": "boolean" }
            },
            "required": ["name", "expression"]
          },
          "description": "High-level quality expectations (may correspond to one or more concrete tests)."
        }
      }
    },
    "sla": {
      "type": "object",
      "description": "Service Level Agreement expectations (timeliness, availability, etc.)",
      "properties": {
        "refreshFrequency": {
          "type": "object",
          "properties": {
            "interval": { "type": "integer" },
            "unit": { "type": "string", "enum": ["hour", "day", "week", "month"] }
          },
          "description": "Expected frequency of data updates (e.g. every 1 day)",
          "required": ["interval", "unit"]
        },
        "maxLatency": {
          "type": "object",
          "properties": {
            "value": { "type": "integer" },
            "unit": { "type": "string", "enum": ["minute", "hour", "day"] }
          },
          "description": "Maximum acceptable latency between data generation and availability (e.g. 4 hours)",
          "required": ["value", "unit"]
        },
        "availabilityTime": {
          "type": "string",
          "description": "Time of day by which data is expected to be available (e.g. \"09:00 UTC\")"
        },
        "retention": {
          "type": "object",
          "properties": {
            "period": { "type": "integer" },
            "unit": { "type": "string", "enum": ["day", "year"] }
          },
          "description": "How long the data is retained (if relevant)",
          "required": ["period", "unit"]
        }
      }
    },
    "version": {
      "description": "Version of the Data Contract.",
      "$ref": "../../type/entityHistory.json#/definitions/entityVersion"
    },
    "status": {
      "type": "string",
      "enum": ["DRAFT", "ACTIVE", "VIOLATED"],
      "description": "Lifecycle status of the contract (e.g., PENDING implementation, ACTIVE, or VIOLATED if broken)"
    }
  },
  "required": ["name", "owner", "targetEntity", "schema"]
}

Schema Section: schema defines the expected structural schema of the data asset. It includes a list of fields (for a table, these are columns; for a topic, message schema fields; for a model, input/output fields) each with name and data type. This captures the contractual schema that producers and consumers agreed on. You can also specify if fields are allowed to be null and provide each field’s description and any integrity constraints (like unique, foreign key reference, etc.). A boolean strict indicates if the schema is exact – e.g., if strict=true, no additional columns beyond those listed can appear without violating the contract (useful for enforcing strict schema evolution control). If strict=false, the contract defines a minimum expected schema (new columns could be added as long as they don’t break existing fields).

Semantics Section: semantics captures business meaning and documentation requirements. For example, one can enforce that the data asset must have an description (entityDescriptionRequired=true) and optionally that every column/field has a description (columnDescriptionRequired). It can also list required tags or glossary terms that must be associated (ensuring proper classification and taxonomy linkage. Additionally, customIntegrityRules allow business users to specify any high-level rules about the data (e.g. “Account status must be one of ACTIVE/INACTIVE” or “All records should have a corresponding entry in master table”). These rules complement the formal tests in the quality section, acting as documentation of business expectations (they could later be mapped to actual test cases or checks). This semantics section ensures the contract isn’t just about technical schema, but also carries business context.

Security Section: security defines data security and access expectations. This can reference an accessPolicy (for example, the name/ID of a policy in OpenMetadata’s Access Control that should apply to this dataset) or a required dataClassification label. In practice, this means the contract might require the data asset to be tagged as “PII” or “Confidential” if appropriate, and that only certain roles can access it (through an associated policy). While enforcement of security in OpenMetadata is typically done via role-based policies, including it in the contract ensures producers/owners declare the intended security level as part of the contract.

Quality (Assertions) Section: quality holds the data quality tests and assertions required by the contract. This can be represented in two ways (which are not mutually exclusive):
A list of explicit tests referencing OpenMetadata’s Test Case entities. Each TestCase in OpenMetadata corresponds to a specific data quality check (for example, a “non-null constraint on column X” or “freshness under 24h” test). By linking to TestCase or TestDefinition entries, the contract directly ties into OpenMetadata’s testing framework. (For instance, if a Great Expectations suite or a custom SQL check exists for this dataset, it can be referenced here.)
A list of high-level expectations described in natural language or simple expressions. These are akin to the “business assertions”
. For example, “No more than 5% of records can be incomplete” or “Column X should be unique”. Each expectation has a name and an expression/description. These may correspond to one or multiple TestCases; they serve as a human-readable summary of the quality guarantees. In essence, the quality section enumerates all the data quality conditions that the data must meet to satisfy the contract.

SLA Section: sla captures Service-Level Agreements related to the data’s timeliness and lifecycle.
This includes:
refreshFrequency: how often the data is expected to be updated or refreshed (e.g., {"interval": 1, "unit": "day"} for daily updates).
maxLatency: the maximum allowed delay between data generation and when it’s available to consumers (e.g., {"value": 4, "unit": "hour"} meaning data may be up to 4 hours old at most).
availabilityTime: the time by which daily or periodic data should be available (e.g., “09:00 UTC” daily data drop).
retention: how long the data is kept accessible (if applicable).

Including them in the contract means producers commit to certain delivery timelines, and consumers know what availability to expect. In OpenMetadata, these can be linked to freshness tests or used for monitoring (for example, a Table Data to Be Fresh test can ensure the refreshFrequency is respected).

Status: A status field indicating whether the contract is active, draft, or currently violated. For instance, when first created, a contract might be DRAFT (not yet enforced or not fully implemented by the data producer). Once the data contract is published, it becomes ACTIVE. If a violation occurs (e.g., a test fails or schema deviates), the status could be marked VIOLATED automatically, signaling attention needed.

Applying Contracts to Different Asset Types

Let’s illustrate how these contracts would look for various data assets:

Data Contract for Table warehouse.sales.orders

This data contract defines the schema and quality expectations for a database table warehouse.sales.orders (e.g., an orders table in the sales schema of a warehouse). We enable incident management globally (incidentManagement: true) to ensure any critical data quality failures raise an incident in OpenMetadata’s Incident Manager. All quality checks use OpenMetadata’s native engine (specified for clarity via engine: OpenMetadata), and critical expectations have raiseIncident: true so that their violations will trigger incidents. Less critical checks can be set with raiseIncident: false to avoid incident noise.

name: warehouse.sales.orders
type: table
description: >
  Data contract for the orders table, defining expected schema and data quality constraints.
owner: Analytics Team
incidentManagement: true  # Enable auto-logging of violations as incidents
schema:
  columns:
    - name: order_id
      type: INTEGER
      description: Primary key for orders
      nullable: false
      unique: true
    - name: order_timestamp
      type: TIMESTAMP
      description: UTC timestamp when the order was placed
      nullable: false
    - name: order_total
      type: DECIMAL(10,2)
      description: Total amount of the order in USD
      nullable: false
    - name: customer_id
      type: INTEGER
      description: Reference to the customer placing the order
      nullable: false
    - name: customer_email
      type: STRING
      description: Email address of the customer (may be null if not provided)
      nullable: true
quality:
  expectations:
    - name: NonNull_OrderID
      description: "Order ID must never be null."
      field: order_id
      condition: "IS NOT NULL"        # order_id should be present in every record
      engine: OpenMetadata           # using OpenMetadata's DQ engine
      raiseIncident: true            # critical - trigger incident if violated
    - name: Valid_OrderTotal
      description: "Order total should be non-negative."
      field: order_total
      condition: ">= 0"              # order_total must be 0 or greater
      engine: OpenMetadata
      raiseIncident: true            # critical check
    - name: RowCount_Minimum
      description: "At least 5000 orders are expected in the table."
      condition: "row_count >= 5000" # table-level check on minimum row count
      engine: OpenMetadata
      raiseIncident: true            # raise incident if row count drops below threshold
    - name: Optional_CustomerEmail
      description: "Customer email, if present, should not be null."
      field: customer_email
      condition: "IS NOT NULL IF PRESENT"  # if email field exists, it should have a value
      engine: OpenMetadata
      raiseIncident: false           # not critical (no incident for missing emails)

Kafka Topic: prod.kafka.customer_signups

This example is a data contract for a Kafka topic prod.kafka.customer_signups. It outlines the expected message schema (key and value structure) and quality expectations for the streaming data. Here we set incidentManagement: false to demonstrate the global toggle – in this case, contract violations will not automatically create incidents (we might choose to monitor this topic without raising immediate incidents). Nonetheless, each expectation still indicates its criticality via raiseIncident.

apiVersion: v1
kind: DataContract
name: prod.kafka.customer_signups
domain: customer
dataProduct: signups
description:
  purpose: "Capture user signup events from production systems for analytics."
  limitations: "Contains PII (emails); ensure compliance with privacy regulations."
  usage: "Feeds downstream analytics and user onboarding metrics."
owners:
  - name: "Data Streaming Team"
    type: team
  - name: "john.smith"
    type: user
tags: ["customer", "signups", "PII"]

schema:
  fields:
    - name: event_id
      type: string
      description: "Unique identifier for the signup event."
      constraints:
        not_null: true
    - name: customer_id
      type: string
      description: "Unique identifier of the customer who signed up."
      constraints:
        not_null: true
    - name: signup_time
      type: timestamp
      description: "Timestamp when the signup occurred."
      constraints:
        not_null: true
        no_future_dates: true
    - name: signup_source
      type: string
      description: "Channel through which the signup occurred (e.g., web or mobile)."
      constraints:
        enum: ["web", "mobile", "partner"]
    - name: email
      type: string
      description: "Email address provided by the customer at signup."
      constraints:
        not_null: true
        check_pii: true

semantics:
  - "Each record represents one customer signup event."
  - "A given customer_id should have at most one signup event (no duplicate signups for the same customer)."

security:
  classification: "Sensitive"
  compliance: ["GDPR", "CCPA"]
  encryption: "TLS encryption in transit"
  access: "Limited to authorized internal consumers due to PII."

quality:
  - name: "Unique Event ID"
    description: "No duplicate event_id across messages."
    type: custom
    engine: openmetadata
    implementation: |
      type: duplicate_count
      columns:
        - event_id
      must_be_equal_to: 0
  - name: "Valid Email Format"
    description: "Email field should match standard email pattern."
    type: custom
    engine: greatexpectations
    implementation: |
      expectation_type: expect_column_values_to_match_regex
      kwargs:
        column: email
        regex: "^[A-Za-z0-9+_.-]+@[A-Za-z0-9.-]+$"
  - name: "Daily Volume Threshold"
    description: "At least 50 signup events should arrive per day."
    type: sql
    query: "SELECT COUNT(*) FROM prod.kafka.customer_signups WHERE DATE(signup_time) = CURRENT_DATE"
    mustBeGreaterOrEqualTo: 50

sla:
  dataRetention: "7 days"
  maxLatency: "5 seconds"
  throughput: ">= 500 events/minute"
  availability: "99.9%"
  producers:
    - name: "Auth Service"
      type: application
      description: "Microservice producing signup events to Kafka."
  consumers:
    - name: "Ingestion Pipeline"
      type: pipeline
      description: "Pipeline consuming events into the data warehouse."
    - name: "Realtime Dashboard"
      type: application
      description: "Application consuming events for live signup metrics.”

Data Contract for MLModel churn_prediction_model_v2

apiVersion: v1
kind: DataContract
name: churn_prediction_model_v2
domain: customer
dataProduct: churn_prediction
description:
  purpose: "Provides churn probability predictions for customers (version 2 of the model)."
  limitations: "May not account for new behaviors not present in training data."
  usage: "Used by retention team to identify high-risk customers for outreach."
owners:
  - name: "Data Science Team"
    type: team
  - name: "alice.brown"
    type: user
tags: ["machine_learning", "customer_retention"]

schema:
  inputs:
    - name: tenure_months
      type: integer
      description: "Number of months since the customer's signup."
      constraints:
        not_null: true
        not_negative: true
    - name: avg_monthly_sessions
      type: integer
      description: "Average number of sessions per month."
      constraints:
        not_null: true
        not_negative: true
    - name: support_tickets_last_3m
      type: integer
      description: "Number of support tickets in the last 3 months."
      constraints:
        not_null: true
        not_negative: true
    - name: is_premium_member
      type: boolean
      description: "Whether the customer is a premium (paid) member."
      constraints:
        not_null: true
  outputs:
    - name: churn_probability
      type: float
      description: "Predicted probability (0.0 to 1.0) that the customer will churn."
      constraints:
        not_null: true
        min_value: 0.0
        max_value: 1.0
    - name: churn_prediction
      type: boolean
      description: "Predicted churn outcome (True if churn likely, False otherwise)."
      constraints:
        not_null: true

semantics:
  - "Churn is defined as a customer who cancels or becomes inactive within 30 days."
  - "High churn_probability (above 0.5) corresponds to churn_prediction = True."
  - "Model v2 was trained on customer behavior data up to 2025-04."

security:
  classification: "Internal"
  compliance: ["GDPR"]
  access: "Internal use only (predictions contain customer-related insights)."

quality:
  - name: "Probability Range"
    description: "All churn_probability values should be between 0.0 and 1.0."
    type: library
    rule: value_range
    field: churn_probability
    range: [0.0, 1.0]
  - name: "Minimum Model AUC"
    description: "Model AUC on validation data is at least 0.80."
    type: custom
    engine: custom
    implementation: "AUC >= 0.80 on validation dataset"
  - name: "Max Predicted Churn Rate"
    description: "No more than 30% of customers should be predicted as churn (to limit false positives)."
    type: sql
    query: >
      SELECT CAST(SUM(CASE WHEN churn_prediction = TRUE THEN 1 ELSE 0 END) AS FLOAT) / COUNT(*)
      FROM model_predictions
    mustBeLessThan: 0.30

sla:
  predictionLatency: "100 ms per prediction"
  retrainingFrequency: "Monthly"
  availability: "99.5%"
  producers:
    - name: "Model Training Pipeline"
      type: pipeline
      description: "Pipeline retraining and deploying the model monthly."
  consumers:
    - name: "Churn Detection Service"
      type: application
      description: "Service using the model to score customers in production."
    - name: "Retention Analytics"
      type: team
      description: "Team analyzing model outputs for business insights.”

How to use these Data Contracts

Store them under source control (e.g., git) so contract changes are peer-reviewed.
Upload via the OpenMetadata REST API (POST /api/v1/dataContracts) or create through the UI (paste YAML).
Which will allow you to go through the review process, collaborate and improve the data contract. Once approved it becomes active.

Attach each contract automatically by matching targetEntity.fullyQualifiedName, or manually link it in the asset’s “Contracts” tab.

Schedule data-quality workflows; if any test fails, OpenMetadata will mark the contract as VIOLATED and alert the owner.

Related issues

• ...

[TeddyCr]

"security": {
  "type": "object",
  "description": "Security and access policy expectations",
  "properties": {
    "accessPolicy": { "type": "string", "description": "Reference to an access policy ID or name that should govern this data" },
    "dataClassification": { "type": "string", "description": "Expected data classification (e.g. Confidential, PII, etc.)" }
  }
},

Would this would be the target service policy (not OM)?

[harshach]

"security": {
  "type": "object",
  "description": "Security and access policy expectations",
  "properties": {
    "accessPolicy": { "type": "string", "description": "Reference to an access policy ID or name that should govern this data" },
    "dataClassification": { "type": "string", "description": "Expected data classification (e.g. Confidential, PII, etc.)" }
  }
},

Would this would be the target service policy (not OM)?

Yes thats right target service policy.

[pmbrull]

Some further thoughts on the implementation side.

Approval Workflows

Any data contract creation/change needs to follow an approval workflow, similar to how we handle the glossary approval via the governance workflows.

Here we need to decide who should be the reviewer in this case: is this a platform level setting where folks can just add stewards? Or is it something we want to involve just the Owner as reviewer? Or is it both with default reviewers + owner for each asset?

UI Editing

We should keep a closed form on managing the Data Contracts from the UI, since we want non-technical users to be able to handle them on their own.

However, it's also true that enabling import/export in YAML format can very easily help people to handle them in bulk once they understand the shape.

The UI should allow to "view" a data contract in YAML, to download it and to import it back, but we can keep the UI experience simple without YAML edition directly (at least for now).

Data Contract Status

1. Draft: some required details pending to be included (same approach as glossary approval -> which pieces of the contract we want to force to inform as a company)
2. In Review: task is opened
3. Compliant: All checks are OK
4. Partially Violated: Some checks are failing
5. Violated: All checks are failing

We can then show the data contract as a "Data Contract Card" with the different sections/topics of validation:

1. Data Quality (DQ tests)
2. Semantics / Metadata (owner is informed, description is informed,...)
3. SLAs (freshness & anomaly det.)

This card could be similar to our current Collate's DQ dashboard, showing the cards of the tests passing/failing.

We can also compute a score based on how many rules are passing/failing,

Execution

Data Quality

Creating DQ tests automatically means we need to make certain decisions/validations

1. First, if a test is already created, we should not create it again
2. However, how do we handle a TestCase already present in an entity but with different params? E.g., "values to be between 10-100 vs. 100-1000"? Should the Data Contract win or the existing test?
   1. IMO, a first approach would be to let the Data Contract win and update, since Data Contract will have approavls whereas DQ tests might be more open to external contribution
3. In which Test Suite are Tests created?
   1. I'd default to the Base Test Suite of the entity just keep things simple. We should want to get them executed normally

SLAs

Related to this there's SLAs on data freshness, e.g., hourly, daily, etc.

Here the discussion could be:

1. Do we want SLAs living on a separate test suite (that could execute hourly as the smallest unit) running only freshness / anomaly detection based on SLAs
2. Or do we want to "force" the scheduling of the Base Test Suite to run all the checks on every "SLA check" since new data means new possibilities of the contract being broken?

I might be voting towards 2) to keep things first, simpler to manage, and second, ensuring no gaps between new data and possible contract breaches.

Metadata Checks

DQ/SLA validations are "easy" to manage: We have the DQ framework taht knows how to work.

How do we handle then semantics/metadata validations? E.g., owner cannot be null,

In this case, we can reuse similar logic already present in the Governance Workflows, where we are letting users control the certification process or the validation process for glossary terms. In this scenario, the UI shows a form with the entity fields and parses that result as JSON Logic that the server can very easily "check" against an entity.

Reference example

We can then add this as part of the JSON Schema in the same way that we have managed to get it for Gov. Workflows.

How do checks results / metadata changes get reflected in the contract

1. We can add a "dataContract" property to every entity
2. If an entity has a Data Contract informed, any change to its metadata or test results should "push" the change into that entity's Data Contract
3. This means we could have a central Data Contract logic in the EntityRepository controlling how it behaves after every update on its metadata or DQ results

The good thing about this is that

1. Server is not that overloaded, since the costly pieces to compute are DQ/SLAs and are already happening in the Ingestion Framework
2. We can get live updates of any contract breaches
3. Leaves us room for the future to enable "locking" changes that could break a contract
   4. E.g., if a contract requires owner to be set and you try to PATCH to remove the owner, the EntityRepository should raise an exception and block the transaction
   5. For a first implementation, we can keep these things as "soft" validations and even run them async, just passing the "this change happened" to the contract but letting the PUT/PATCH go through

TLDR:

1. DQ/SLAs run on the Ingestion Framework at the required schedule. Results are synced with the contract in the Entity Repository.
2. Metadata/Semantic checks are validated live in the server, synced with the contract in the Entity Repository
3. We show a card in the UI with the different aspects of the contract as OK/KO with a compliance %

[hetalghelani]

Hi.

We’re currently in the process of implementing data contracts in our organization, and this feature would help us a lot.
Could you please share if there’s any rough timeline or plan for when this might be available?

We’d really appreciate any update, as it would help us plan accordingly.

Thanks in advance for your time and efforts! 🙏

[pmbrull]

hi @hetalghelani we'll be getting it out with UI support on 1.9 release, early August

[harshach]

Shipped in 1.9