Skip to content

Fix: handle proto type instead of schema on Lineage API - #57

Merged
luthfifahlevi merged 2 commits into
mainfrom
fix/lineage-api-handle-proto-type
May 13, 2026
Merged

Fix: handle proto type instead of schema on Lineage API#57
luthfifahlevi merged 2 commits into
mainfrom
fix/lineage-api-handle-proto-type

Conversation

@luthfifahlevi

@luthfifahlevi luthfifahlevi commented May 12, 2026

Copy link
Copy Markdown

Summary

The lineage API previously mapped {schema_name} in the URL directly to both a schema container (stored artifact in the DB) and the proto message name to trace. This worked only in the early one-schema-one-message model but breaks completely for real usage where one schema artifact (e.g. esb-log-entities) bundles hundreds of proto messages.

This MR redesigns the lineage API endpoint so that:

  1. The URL path identifies both the schema container (schema_id) and the proto message root (type_name) explicitly.
  2. Response nodes return fully-qualified proto names (FQNs) everywhere, not short last-segment names.
  3. type_name is a required path parameter — missing it returns 400.

Problem

Before

GET /v1beta1/namespaces/{namespace_id}/schemas/{schema_name}/lineage
  • schema_name served dual purpose: DB lookup key and proto message root.
  • A call like GET .../schemas/esb-log-entities/lineage would try to find a proto message named esb-log-entities in the descriptor — and always return empty results for real schema containers.
  • Callers had no way to specify which message inside a bundle they wanted lineage for.
  • Response nodes returned short names like Item, Order — ambiguous without package context.

After

GET /v1beta1/namespaces/{namespace_id}/schemas/{schema_id}/types/{type_name}/lineage
  • schema_id = DB artifact key (esb-log-entities)
  • type_name = fully-qualified proto message (gojek.esb.types.Location)
  • Missing type_name400 Bad Request
  • Response nodes carry full FQNs: gotocompany.events.Order.Item not Item

Changes

API path

Before After
Path /schemas/{schema_name}/lineage /schemas/{schema_id}/types/{type_name}/lineage
schema_name dual-use ✅ (DB key + message root) removed
schema_id DB artifact key only
type_name optional query ?type_name= required path segment

Response contract

Field Before After
root_schema.schema_name "User" removed
root_schema.schema_id "esb-log-entities"
root_schema.type_name "gojek.esb.types.Location"
node schema_name "Item" removed
node schema_id "esb-log-entities"
node type_name "Item" "gotocompany.events.Order.Item"
node path[i] ["User","Item"] ["gotocompany.events.User","gotocompany.events.Order.Item"]

Files changed

File What changed
internal/api/api.go Route updated to /schemas/{schema_id}/types/{type_name}/lineage
internal/api/schema.go Handler reads schema_id + type_name from path params; 400 when type_name empty
internal/api/mocks/schema_service.go Updated mock signature for new GetLineage(ctx, namespaceID, schemaID, rootType, level, direction)
internal/api/lineage_test.go Tests updated to use new path, new response fields; added missing-type_name 400 test
core/schema/lineage.go convertLineageNodes now emits full FQN in TypeName and Path; RootSchemaRef and LineageSchema use schema_id + type_name
core/schema/service.go GetLineage receives schemaID + rootType separately; descriptor bytes fetched by schemaID; lineage computed from rootType
core/schema/lineage_test.go Updated to expect FQN values in assertions
proto/gotocompany/apidocs.swagger.json Path updated; type_name marked required: true (in path); schema_id/type_name added to response definitions
test_helper/seed_lineage_data.go Reworked into a proper E2E harness: seeds one esb-log-entities container, asserts FQN-based lineage per test case

Example

# Find everything that depends on gojek.esb.types.Location
# inside the esb-log-entities schema container:
curl -s 'https://<host>/v1beta1/namespaces/gojek/schemas/esb-log-entities/types/gojek.esb.types.Location/lineage?direction=downstream'
{
  "root_schema": {
    "namespace_id": "gojek",
    "schema_id": "esb-log-entities",
    "type_name": "gojek.esb.types.Location"
  },
  "direction": "downstream",
  "downstream": [
    {
      "namespace_id": "gojek",
      "schema_id": "esb-log-entities",
      "type_name": "gojek.esb.booking.GoKilatBookingLogMessage",
      "level": 1,
      "path": [
        "gojek.esb.types.Location",
        "gojek.esb.booking.GoKilatBookingLogMessage"
      ]
    }
  ],
  "summary": {
    "downstream_count": 1,
    "upstream_count": 0,
    "total_count": 1
  }
}

Testing

Unit tests

go test ./internal/api ./core/schema

End-to-end

# start local server first:
./stencil_bin server start -c config.local.yaml

# seed + assert:
go run ./test_helper/seed_lineage_data.go

E2E harness covers:

  • type_name FQN downstream traversal (top-level message)
  • type_name FQN downstream traversal (nested message, e.g. Payment.Receipt)
  • type_name FQN upstream traversal
  • 400 on invalid direction

Breaking changes

Breaking Detail
URL path changed /schemas/{schema_name}/lineage/schemas/{schema_id}/types/{type_name}/lineage
type_name is now required omitting it returns 400 Bad Request
schema_name field removed replaced by schema_id + type_name in both root and lineage nodes
path values changed now FQNs instead of short names

Since the lineage API was new and had no production consumers at time of this change, this is a clean break with no migration needed.

@luthfifahlevi luthfifahlevi self-assigned this May 12, 2026
@luthfifahlevi
luthfifahlevi requested a review from mabdh May 12, 2026 10:33
@luthfifahlevi
luthfifahlevi merged commit aa0ab3c into main May 13, 2026
9 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants