Support read/write of object data

[andriyko]

This PR introduces object data support to SpiceDB, allowing storage and retrieval of arbitrary object metadata alongside relationships. This feature aims to solve the dual-write problem of maintaining consistency between SpiceDB and application databases.

Related PRs:
authzed/api#136
authzed/authzed-go#311

Background

Several feature requests in SpiceDB relate to the dual-write problem:

• Return operation statuses on WriteRelationships
• Need for relationship rollbacks
• Maintaining consistency between SpiceDB and application data

Features

1. Storage Layer (Note: PostgreSQL only)

   • Added object_data table using PostgreSQL JSONB type
   • Implemented schema migration with appropriate indexes and constraints
   • Support for both resource and subject object metadata
   • Bulk operations with batch processing

2. API Extensions

   • Added object data support to key PermissionsService APIs:
     • ReadRelationships
     • WriteRelationships
     • LookupResources
     • LookupSubjects
     • ImportBulkRelationships
     • ExportBulkRelationships
   • Maintained backward compatibility by making object_data optional
   • Uses protobuf Struct format for flexible data storage

Implementation Details

• Object data is stored using PostgreSQL's JSONB type
• Added indexes for query performance
• Test coverage for new functionality

Migration

The changes are backward compatible:

• Existing relationships remain unaffected
• Object data is optional in all APIs
• New schema migrations handle the addition of object_data table

Example Usage

Write schema

grpcurl -plaintext -H 'Authorization: Bearer somerandomkeyhere' -d '{
  "schema": "definition user {}\ndefinition document {\nrelation writer: user\nrelation viewer: user\npermission write = writer\npermission view = viewer + writer\n }"
}' localhost:50051 authzed.api.v1.SchemaService/WriteSchema

Write relationships

grpcurl -plaintext -H 'Authorization: Bearer somerandomkeyhere' -d '{
  "includeObjectData": true,
  "updates": [{
    "operation": "OPERATION_CREATE",
    "relationship": {
      "resource": {
        "objectType": "document",
        "objectId": "doc2",
        "objectData": {
          "metadata": {
            "size": 100240,
            "parent": {"folder": "folder1"}
          }
        }
      },
      "relation": "viewer",
      "subject": {
        "object": {
          "objectType": "user",
          "objectId": "bob",
          "objectData": {
            "name": "Bob",
            "email": "bob@user.com"
          }
        }
      }
    }
  }]
}' localhost:50051 authzed.api.v1.PermissionsService/WriteRelationships

Read relationships

grpcurl -plaintext -H 'Authorization: Bearer somerandomkeyhere' -d '{
  "includeObjectData": true,
  "relationshipFilter": {
    "resourceType": "document"
  }
}' localhost:50051 authzed.api.v1.PermissionsService/ReadRelationships

Lookup operations

grpcurl -plaintext -H 'Authorization: Bearer somerandomkeyhere' -d '{
  "includeObjectData": true,
  "resource": {
    "object_type": "document",
    "object_id": "doc1"
  },
  "permission": "view",
  "subject_object_type": "user"
}' localhost:50051 authzed.api.v1.PermissionsService/LookupSubjects

Export relationships

grpcurl -plaintext -H 'Authorization: Bearer somerandomkeyhere' -d '{
  "includeObjectData": true
}' localhost:50051 authzed.api.v1.PermissionsService/ExportBulkRelationships

Import relationships

grpcurl -plaintext -H 'Authorization: Bearer somerandomkeyhere' -d '{
  "relationships": [
    {
      "resource": {
        "objectType": "document",
        "objectId": "doc102",
        "objectData": {"name": "doc102"}
      },
      "relation": "viewer",
      "subject": {
        "object": {
          "objectType": "user",
          "objectId": "bob102",
          "objectData": {
            "email": "bob102@user.com",
            "name": "Bob102"
          }
        }
      }
    }
  ]
}' localhost:50051 authzed.api.v1.PermissionsService/ImportBulkRelationships

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[andriyko]

I have read the CLA Document and I hereby sign the CLA

[tstirrat15]

How does this solve the dual-write problem? SpiceDB is not and will not be a good key/value store - it's optimized around the access patterns of permission checks. It won't make a good application database and we don't necessarily want to encourage folks using it that way.

[tstirrat15]

@andriyko I wanted to reopen this and say that we appreciate the work that you put into this and that my question was genuine - I don't want to dismiss what you've put forward out of hand.

[tstirrat15]

I chatted with Andriy out of band - I do think this solves an interesting problem that SpiceDB doesn't otherwise have a solution for, which is using SpiceDB without dealing with the dual-write problem. Perhaps this idea could be turned into an optional extension of SpiceDB? I'm going to re-close in the meantime, though.