Skip to content

Critical Decision: API Compatibility Strategy for Nested Resources #3

New issue
New issue
Open
Open
Critical Decision: API Compatibility Strategy for Nested Resources#3
@AlexMikhalev

Description

@AlexMikhalev
AlexMikhalev
opened on Dec 23, 2025

API Compatibility Strategy: Nested Resources & Resource Arrays

Status: Critical Decision Point
Date: 2025-01-23
Related: PR #2 Evaluation, Parity Roadmap

Executive Summary

The terraphim/atomic-server fork currently has different behavior than upstream regarding nested resources:

Feature Fork Current Upstream Current Breaking Change
Value::Resource ✅ Present ❌ Removed YES
SubResource::Resource ✅ Present ❌ Removed YES
Value::ResourceArray ✅ Present ✅ Present NO
SubResource::Nested ✅ Present ✅ Present NO

Critical Decision: The fork must choose between:

  1. Full API Compatibility - Keep nested resources for clients that depend on them
  2. Full Upstream Parity - Remove nested resources, break API for existing clients

Analysis

Current State

Fork (terraphim/atomic-server):

pub enum Value {
    AtomicUrl(String),
    // ...
    ResourceArray(Vec<SubResource>),
    NestedResource(SubResource),
    Resource(Box<Resource>),  // ✅ PRESENT
    // ...
}

pub enum SubResource {
    Resource(Box<Resource>),   // ✅ PRESENT
    Nested(PropVals),
    Subject(String),
}

// Implementations exist:
impl From<Resource> for Value { ... }
impl From<Resource> for SubResource { ... }

Upstream (atomicdata-dev/atomic-server):

pub enum Value {
    AtomicUrl(String),
    // ...
    ResourceArray(Vec<SubResource>),
    NestedResource(SubResource),
    // Resource(Box<Resource>)  ❌ REMOVED
    Uri(String),               // ✅ NEW
    JSON(serde_json::Value),   // ✅ NEW
    YDoc(Vec<u8>),           // ✅ NEW
    // ...
}

pub enum SubResource {
    // Resource(Box<Resource>)  ❌ REMOVED
    Nested(PropVals),
    Subject(String),
}

// Implementations removed:
// impl From<Resource> for Value { ... }  ❌ REMOVED
// impl From<Resource> for SubResource { ... }  ❌ REMOVED

Usage Analysis

Search results show the following usage in the fork:

lib/src/values.rs:

  • impl From<Resource> for Value - Creates Value::Resource(Box::new(val))
  • impl From<Resource> for SubResource - Creates SubResource::Resource(Box::new(val))
  • impl From<Vec<Resource>> for Value - Creates Value::ResourceArray with SubResource::Resource items

server/src/handlers/export.rs:

  • Value::Resource(resource) pattern match for export handling
  • Uses .get_propvals() from nested resource

lib/src/schema.rs, lib/src/collections.rs, lib/src/commit.rs:

  • Uses Value::ResourceArray for standard arrays (not Value::Resource)

Upstream's Reasoning

From commit 37a4e53e (Remove named nested resources from value type and migrate to messagepack atomicdata-dev#1107):

Why removed:

  • Named nested resources created inconsistencies and complexity
  • Arrays should be used instead for collections
  • Storing resources inline caused duplication and sync issues
  • Migration path: convert to subject references

Upstream's new approach:

// Instead of:
Value::Resource(Box<Resource>)

// Use subject references:
Value::ResourceArray(vec![
    SubResource::Subject("https://example.com/resource/1".to_string())
])
// Or nested properties:
Value::ResourceArray(vec![
    SubResource::Nested(vec![("property", "value")])
])

API Compatibility Options

Option 1: Full Upstream Parity (BREAKING CHANGE)

Description: Merge upstream completely, remove Value::Resource and SubResource::Resource.

Pros:

  • ✅ Full parity with upstream
  • ✅ Same behavior as original atomic-server
  • ✅ No divergence in core data model
  • ✅ Easier to merge future upstream changes
  • ✅ Cleaner codebase (less complexity)

Cons:

  • ❌ Breaking API change for any clients expecting nested resources
  • ❌ Requires migration of existing data
  • ❌ May break fork-specific functionality
  • ❌ Clients must be updated to use new API

Migration Path:

// Convert old to new:
Value::Resource(Box<Resource>) ->
    Value::ResourceArray(vec![SubResource::Subject(resource.get_subject())])

// Database migration required (automatic in upstream)

Impact Assessment:

Impact Severity Affected Users
API Clients High Any code using nested resources in responses
Database Critical All data with nested resources must migrate
Tests Medium Need to update tests to use new format

Option 2: Keep Backward Compatibility (FORK DIVERGENCE)

Description: Maintain Value::Resource and SubResource::Resource for backward compatibility.

Pros:

  • ✅ No breaking API changes
  • ✅ Existing clients continue working
  • ✅ Fork can serve old and new clients
  • ✅ Gradual migration possible

Cons:

  • ❌ Permanent divergence from upstream
  • ❌ Harder to merge future upstream changes
  • ❌ Confusing API (two ways to represent the same thing)
  • ❌ Maintenance burden (need to support both)
  • ❌ Doesn't address the root problems (inconsistency, complexity)

Implementation:

// Keep old variant for backward compatibility:
pub enum Value {
    // ...
    Resource(Box<Resource>),  // Keep for backward compat
    ResourceArray(Vec<SubResource>),
    // ...
}

// Add deprecation warning:
#[deprecated(since = "0.1.0", note = "Use ResourceArray with Subject instead")]
pub enum SubResource {
    Resource(Box<Resource>),  // Deprecated
    Nested(PropVals),
    Subject(String),
}

// Auto-migrate on read/write:
fn normalize_value(val: Value) -> Value {
    match val {
        Value::Resource(res) => Value::ResourceArray(vec![
            SubResource::Subject(res.get_subject().into())
        ]),
        other => other,
    }
}

Migration Path:

  • Keep support indefinitely
  • Recommend clients to migrate to new API
  • Add feature flag to disable old API

Option 3: Hybrid Approach (Recommended)

Description: Support both formats with automatic migration and deprecation.

Pros:

  • ✅ Backward compatible during transition period
  • ✅ Clear deprecation path
  • ✅ Eventual parity with upstream
  • ✅ Gradual client migration
  • ✅ Can be feature-gated

Cons:

  • ⚠️ Temporary divergence from upstream
  • ⚠️ More complex implementation
  • ⚠️ Requires migration timeline

Implementation Plan:

Phase 1: Dual Support (Weeks 1-2)

// lib/src/values.rs
pub enum Value {
    // ... existing types
    #[deprecated(since = "0.15.0", note = "Use ResourceArray with Subject instead")]
    Resource(Box<Resource>),
    ResourceArray(Vec<SubResource>),
    // ... new upstream types: Uri, JSON, YDoc
}

pub enum SubResource {
    #[deprecated(since = "0.15.0", note = "Use Subject instead")]
    Resource(Box<Resource>),
    Nested(PropVals),
    Subject(String),
}

// Add normalization function:
pub fn normalize_value(val: Value) -> Value {
    match val {
        Value::Resource(res) => {
            warn!("Value::Resource is deprecated. Migrating to ResourceArray");
            Value::ResourceArray(vec![SubResource::Subject(res.get_subject().into())])
        },
        other => other,
    }
}

// Add to SubResource:
pub fn normalize_subresource(sub: SubResource) -> SubResource {
    match sub {
        SubResource::Resource(res) => {
            warn!("SubResource::Resource is deprecated. Migrating to Subject");
            SubResource::Subject(res.get_subject().into())
        },
        other => other,
    }
}

Phase 2: Write-Time Migration (Weeks 3-4)

// Automatically convert deprecated format on write:
impl CommitBuilder {
    pub fn set_property(&mut self, property: String, value: Value) -> Result<()> {
        let normalized = normalize_value(value);
        self.set.push((property, normalized));
        Ok(())
    }
}

// Add feature flag:
#[cfg(feature = "backward-compat")]
pub const ENABLE_DEPRECATED_NESTED_RESOURCES: bool = true;

#[cfg(not(feature = "backward-compat"))]
pub const ENABLE_DEPRECATED_NESTED_RESOURCES: bool = false;

Phase 3: Read-Time Conversion (Weeks 5-6)

// In API handlers:
pub async fn handle_get_resource(subject: String) -> Result<HttpResponse> {
    let resource = store.get_resource(&subject)?;

    // Convert if using backward compat mode
    let normalized = if !ENABLE_DEPRECATED_NESTED_RESOURCES {
        normalize_resource(resource)?
    } else {
        resource
    };

    Ok(HttpResponse::Ok().json(normalized))
}

// Normalize resource properties:
fn normalize_resource(mut resource: Resource) -> Result<Resource> {
    for (prop, val) in resource.propvals.iter() {
        let normalized = normalize_value(val.clone());
        resource.set_unsafe(prop.clone(), normalized);
    }
    Ok(resource)
}

Phase 4: Deprecation Timeline

Version Action Timeline
v0.15.0 Add deprecation warnings Immediately
v0.16.0 Enable write migration by default 1 month
v0.17.0 Feature flag to disable old API 2 months
v0.18.0 Remove deprecated code 6 months

Migration Tools:

# CLI tool to migrate resources:
atomic-server migrate-resources --from old-format --to new-format

# Automatic migration on server startup:
atomic-server --auto-migrate-deprecated

API Compatibility Matrix

Read Operations (GET /{resource})

Format Fork Behavior Upstream Behavior Hybrid Approach
Old: nested resource Returns inline Error Convert to subject reference
New: resource array Returns array of subjects Returns array of subjects Returns array of subjects
New: nested properties Returns nested properties Returns nested properties Returns nested properties

Write Operations (POST /commit)

Format Fork Behavior Upstream Behavior Hybrid Approach
Old: nested resource Accepts and stores Rejects Migrate to new format
New: resource array Accepts and stores Accepts and stores Accepts and stores
New: nested properties Accepts and stores Accepts and stores Accepts and stores

Recommended Decision

Recommendation: Option 3 - Hybrid Approach

Rationale

  1. Backward Compatibility First

    • Existing clients continue working
    • No immediate breaking changes
    • Controlled migration timeline

  2. Clear Path to Parity

    • Deprecation warnings guide migration
    • Feature flag for gradual rollout
    • Planned removal of deprecated code

  3. Business Continuity

    • Avoid breaking production deployments
    • Allow client teams to update at their own pace
    • Reduce migration risk

  4. Future Proof

    • Eventually achieves full parity
    • Easier to merge future upstream changes
    • Cleaner long-term codebase

Updated Parity Roadmap

Phase 1: Hybrid Compatibility Layer (Weeks 1-2)

Tasks

1.1 Add Upstream Missing Features

  • Merge upstream Value::Uri datatype
  • Merge upstream Value::JSON datatype
  • Merge upstream Value::YDoc datatype
  • Update DataType enum

1.2 Implement Hybrid Support

  • Add normalize_value() function
  • Add normalize_subresource() function
  • Add deprecation warnings to Value::Resource
  • Add deprecation warnings to SubResource::Resource

1.3 Configuration

  • Add ENABLE_DEPRECATED_NESTED_RESOURCES feature flag
  • Add config option for migration mode
  • Document deprecation timeline

1.4 Testing

  • Add tests for old format handling
  • Add tests for new format handling
  • Add tests for conversion
  • Add migration tests

Phase 2: Write-Time Migration (Weeks 3-4)

Tasks

2.1 Commit Handling

  • Auto-convert deprecated format in CommitBuilder
  • Log migrations for monitoring
  • Add metrics for migration tracking

2.2 API Handlers

  • Update POST handlers to normalize values
  • Update PUT handlers to normalize values
  • Add migration logging

2.3 Client Library Updates

  • Update browser/lib to send new format by default
  • Add migration helper functions
  • Update documentation

Phase 3: Read-Time Conversion (Weeks 5-6)

Tasks

3.1 GET Handlers

  • Update GET handlers to normalize resources
  • Update SEARCH handlers to normalize results
  • Add configuration for conversion mode

3.2 Serialization

  • Update JSON-AD serialization
  • Update JSON-LD serialization
  • Update other formats

Phase 4: Migration Tools (Weeks 7-8)

Tasks

4.1 CLI Migration Tool

  • Add migrate-resources command
  • Add batch migration support
  • Add validation tools

4.2 Server-Side Migration

  • Add --auto-migrate-deprecated flag
  • Add automatic migration on startup
  • Add progress reporting

Phase 5: Deprecation and Removal (Months 6-12)

Tasks

5.1 Monitoring

  • Track usage of deprecated formats
  • Monitor client adoption of new format
  • Send deprecation notices

5.2 Removal

  • v0.18.0: Remove Value::Resource variant
  • v0.18.0: Remove SubResource::Resource variant
  • Update all tests
  • Update documentation

Testing Strategy

Compatibility Tests

#[test]
fn test_backward_compat_old_to_new() {
    let mut resource = Resource::new("https://example.com/test");

    // Old format:
    let nested = Resource::new("https://example.com/nested");
    resource.set("https://example.com/properties/related", Value::Resource(Box::new(nested))).unwrap();

    // Normalize:
    let normalized = normalize_resource(resource).unwrap();

    // Should convert to array of subjects:
    match normalized.get("https://example.com/properties/related") {
        Some(Value::ResourceArray(vec)) => {
            assert_eq!(vec.len(), 1);
            match &vec[0] {
                SubResource::Subject(s) => {
                    assert_eq!(s, "https://example.com/nested");
                }
                _ => panic!("Expected Subject"),
            }
        }
        _ => panic!("Expected ResourceArray"),
    }
}

#[test]
fn test_new_format_passthrough() {
    let mut resource = Resource::new("https://example.com/test");

    // New format:
    resource.set(
        "https://example.com/properties/related",
        Value::ResourceArray(vec![SubResource::Subject("https://example.com/nested".to_string())])
    ).unwrap();

    // Normalize:
    let normalized = normalize_resource(resource).unwrap();

    // Should remain unchanged:
    match normalized.get("https://example.com/properties/related") {
        Some(Value::ResourceArray(vec)) => {
            assert_eq!(vec.len(), 1);
            match &vec[0] {
                SubResource::Subject(s) => {
                    assert_eq!(s, "https://example.com/nested");
                }
                _ => panic!("Expected Subject"),
            }
        }
        _ => panic!("Expected ResourceArray"),
    }
}

Migration Checklist

Before Enabling Hybrid Mode

  • Test old client compatibility

    • Verify existing clients still work
    • Test GET operations
    • Test POST/PUT operations
    • Test error handling

  • Test new client behavior

    • Verify new clients use correct format
    • Test write operations
    • Test read operations
    • Test error handling

  • Add monitoring

    • Track deprecated format usage
    • Monitor conversion performance
    • Log migration events

  • Documentation

    • Update API documentation
    • Document deprecation timeline
    • Add migration guide
    • Update examples

Before Removing Deprecated Code

  • Verify no usage

    • Check production logs
    • Check client codebases
    • Confirm zero usage for 30 days

  • Final migration

    • Run final migration script
    • Verify all data converted
    • Backup database

  • Update all tests

    • Remove old format tests
    • Update integration tests
    • Update E2E tests

Decision Matrix

Factor Option 1 (Break) Option 2 (Keep) Option 3 (Hybrid)
Backward Compatibility ❌ No ✅ Yes ✅ Yes
Upstream Parity ✅ Yes ❌ No ✅ Yes (eventual)
Migration Complexity ⚠️ High ✅ Low ⚠️ Medium
Maintenance Burden ✅ Low ❌ High ⚠️ Medium-Term
Client Impact ❌ High ✅ None ⚠️ Low
Future Merges ✅ Easy ❌ Hard ✅ Easy (eventual)
Timeline Immediate Indefinite 6-12 months

Conclusion

Recommended Approach: Option 3 - Hybrid Compatibility

This approach provides the best balance between:

  1. Maintaining API compatibility for existing clients
  2. Achieving parity with upstream
  3. Enabling gradual migration without breaking changes
  4. Providing clear deprecation path for cleanup

The 6-12 month timeline allows teams to update at their own pace while ensuring the fork eventually achieves full parity with upstream.

Next Steps:

  1. Review and approve this strategy
  2. Update PARITY_ROADMAP.md with hybrid approach
  3. Begin Phase 1 implementation
  4. Communicate deprecation timeline to all stakeholders

Document Version: 1.0
Last Updated: 2025-01-23
Author: Generated from PR #2 Analysis + API Compatibility Requirements

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions