Optionally skip writing a sourced_from source type to its own index

[ellisandrews-toast]

Summary

Every indexed type in ElasticGraph gets a __self update target that writes its own events to its own index. When a type is used purely as a source for sourced_from fields on another indexed type, this means its data is stored twice: once in its own index, and again in the destination type's index via the sourced-from update targets. For source types that are never queried on their own, the copy in their own index is pure overhead. This proposes an opt-in way to suppress a source type's __self write so its data flows only to its other update targets.

Proposed behavior

Add an opt-in flag (exact spelling TBD) that, when set on a source type, suppresses its __self update target: events of that type are no longer written to its own index, but the sourced_from update targets are still generated and applied, so the data continues to flow into the destination type(s)' indices exactly as before. Only the redundant standalone copy is dropped.

This is independent of whether the source type feeds top-level or nested sourced_from fields — in both cases the __self target is assembled separately from the sourced-from targets, so suppressing it leaves the sourcing flow intact.

Considerations

A couple of considerations when designing this:

• The type must not be queryable. If the type is exposed via root query fields, suppressing its own index would make those queries return nothing. This should be a schema-definition error.
• Source types carrying their own (non-sourced) data. If the type has fields that aren't flowed out to a destination, suppressing the self-write silently loses that data. This should be clear to end users.

Open questions

• Exact DSL surface (a flag on index, a separate DSL method, or a type-level setting).
• Whether to also omit the datastore index/mapping config for the suppressed type (no unused index created at all), or just suppress the writes while leaving the index in place.

[myronmarston]

Add an opt-in flag (exact spelling TBD) that, when set on a source type, suppresses its __self update target: events of that type are no longer written to its own index, but the sourced_from update targets are still generated and applied, so the data continues to flow into the destination type(s)' indices exactly as before. Only the redundant standalone copy is dropped.

I think the option should do more than that. It should also omit the index definition from datastore_config.yaml.

I have a couple ideas for how to implement this. Here's one:

schema.object_type "CoachProfile" do |t|
  t.field "id", "ID!"
  t.field "team_id", "ID"
  t.field "coach_id", "ID"
  t.field "annual_salary", "Int"
  t.field "team_league", "String"
  t.field "team_formed_on", "Date"
  t.index disabled: true
end

Essential, explicitly disable the index definition for the type. However, it occurs to me that the natural state of an object_type is to have no index definition, and it feels odd to have to call t.index disabled: true here. That makes me wonder--can we make it work without any opt-in flag at all?

schema.object_type "CoachProfile" do |t|
  t.field "id", "ID!"
  t.field "team_id", "ID"
  t.field "coach_id", "ID"
  t.field "annual_salary", "Int"
  t.field "team_league", "String"
  t.field "team_formed_on", "Date"
end

Right now, the only types that are ingestable as JSON schema events are root document types:

https://github.com/block/elasticgraph/blob/c845040575d4ad23a08fe6bcd4d5657d1cec610c/elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/json_schema_builder.rb#L36C82-L52

We can introduce the concept of "ingestable" types. Root document types would be ingestable, and also types which are used as the source for a sourced_from field.

If we can get that to work, it would be really simple to use.