Skip to content

Support retrieved_from: :doc_values for direct leaf fields#1110

Open
jwils wants to merge 1 commit intomainfrom
joshuaw/docvalue-leaf-fetching
Open

Support retrieved_from: :doc_values for direct leaf fields#1110
jwils wants to merge 1 commit intomainfrom
joshuaw/docvalue-leaf-fetching

Conversation

@jwils
Copy link
Copy Markdown
Collaborator

@jwils jwils commented Apr 6, 2026

Summary

This PR adds a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored _source.

t.field "internal_code", "String", retrieved_from: :doc_values

When a field is marked retrieved_from: :doc_values, ElasticGraph:

  • keeps the field in GraphQL output types
  • excludes the field from datastore _source via _source.excludes
  • requests the field through datastore docvalue_fields
  • resolves the field from fields in datastore hits when _source omits it

Implementation

The change is intentionally narrow:

  • only direct fields on indexed root document types are supported
  • only GraphQL leaf fields are supported
  • dotted name_in_index paths are rejected
  • text fields are rejected because they do not support datastore doc values
  • query planning only uses docvalue_fields when every participating index definition agrees on the retrieval method

The runtime metadata now records how an index field is retrieved, the GraphQL datastore query can split requested fields between _source and docvalue_fields, and datastore response helpers can read direct leaf values back from hit fields when needed.

Tests

Updated unit specs cover:

  • keeping retrieved_from: :doc_values fields in GraphQL output types
  • emitting _source.excludes and runtime metadata for those fields
  • requesting docvalue_fields and falling back to _source when index definitions disagree
  • resolving direct scalar and list field values from hit fields
  • rejecting unsupported retrieved_from usages

I was not able to run the RSpec files in this local environment because Bundler is missing required gems/native extensions.

Stack Info: Stacked on #1108

@jwils jwils changed the title Support for direct leaf fields Support retrieved_from: :doc_values for direct leaf fields Apr 6, 2026
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch 3 times, most recently from ad363a5 to a2b2407 Compare April 7, 2026 19:38
@jwils jwils force-pushed the joshauw/fetchable-false-prototype branch from 8e9b12f to bde522d Compare April 7, 2026 19:42
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from a2b2407 to fdf2582 Compare April 7, 2026 19:46
@jwils jwils force-pushed the joshauw/fetchable-false-prototype branch 4 times, most recently from caabea3 to fd1d61a Compare April 14, 2026 01:28
Base automatically changed from joshauw/fetchable-false-prototype to main April 14, 2026 01:49
jwils added a commit that referenced this pull request Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay
returnable in GraphQL while being excluded from stored `_source`.

When a field is marked `retrieved_from: :doc_values`, ElasticGraph:

- keeps the field in GraphQL output types
- excludes the field from datastore `_source` via `_source.excludes`
- requests the field through datastore `docvalue_fields`
- resolves the field from `fields` in datastore hits when `_source` omits it

The change is intentionally narrow: only direct, non-list, non-text
GraphQL leaf fields on indexed root document types are supported.
Query planning only uses `docvalue_fields` when every participating
index definition agrees on the retrieval method.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from fdf2582 to 83f0f4b Compare April 14, 2026 02:12
jwils added a commit that referenced this pull request Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay
returnable in GraphQL while being excluded from stored `_source`.

When a field is marked `retrieved_from: :doc_values`, ElasticGraph:

- keeps the field in GraphQL output types
- excludes the field from datastore `_source` via `_source.excludes`
- requests the field through datastore `docvalue_fields`
- resolves the field from `fields` in datastore hits when `_source` omits it

The change is intentionally narrow: only direct, non-list, non-text
GraphQL leaf fields on indexed root document types are supported.
Query planning only uses `docvalue_fields` when every participating
index definition agrees on the retrieval method.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from 83f0f4b to 86787f5 Compare April 14, 2026 02:21
jwils added a commit that referenced this pull request Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay
returnable in GraphQL while being excluded from stored `_source`.

When a field is marked `retrieved_from: :doc_values`, ElasticGraph:

- keeps the field in GraphQL output types
- excludes the field from datastore `_source` via `_source.excludes`
- requests the field through datastore `docvalue_fields`
- resolves the field from `fields` in datastore hits when `_source` omits it

The change is intentionally narrow: only direct, non-list, non-text
GraphQL leaf fields on indexed root document types are supported.
Query planning only uses `docvalue_fields` when every participating
index definition agrees on the retrieval method.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from 86787f5 to 8e29a0b Compare April 14, 2026 12:59
jwils added a commit that referenced this pull request Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay
returnable in GraphQL while being excluded from stored `_source`.

When a field is marked `retrieved_from: :doc_values`, ElasticGraph:

- keeps the field in GraphQL output types
- excludes the field from datastore `_source` via `_source.excludes`
- requests the field through datastore `docvalue_fields`
- resolves the field from `fields` in datastore hits when `_source` omits it

The change is intentionally narrow: only direct, non-list, non-text
GraphQL leaf fields on indexed root document types are supported.
Query planning only uses `docvalue_fields` when every participating
index definition agrees on the retrieval method.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from 8e29a0b to f9afdfc Compare April 14, 2026 13:33
jwils added a commit that referenced this pull request Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay
returnable in GraphQL while being excluded from stored `_source`.

When a field is marked `retrieved_from: :doc_values`, ElasticGraph:

- keeps the field in GraphQL output types
- excludes the field from datastore `_source` via `_source.excludes`
- requests the field through datastore `docvalue_fields`
- resolves the field from `fields` in datastore hits when `_source` omits it

The change is intentionally narrow: only direct, non-list, non-text
GraphQL leaf fields on indexed root document types are supported.
Query planning only uses `docvalue_fields` when every participating
index definition agrees on the retrieval method.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from f9afdfc to 6e0529f Compare April 14, 2026 13:54
jwils added a commit that referenced this pull request Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay
returnable in GraphQL while being excluded from stored `_source`.

When a field is marked `retrieved_from: :doc_values`, ElasticGraph:

- keeps the field in GraphQL output types
- excludes the field from datastore `_source` via `_source.excludes`
- requests the field through datastore `docvalue_fields`
- resolves the field from `fields` in datastore hits when `_source` omits it

The change is intentionally narrow: only direct, non-list, non-text
GraphQL leaf fields on indexed root document types are supported.
Query planning only uses `docvalue_fields` when every participating
index definition agrees on the retrieval method.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from 6e0529f to a006605 Compare April 14, 2026 14:13
jwils added a commit that referenced this pull request Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay
returnable in GraphQL while being excluded from stored `_source`.

When a field is marked `retrieved_from: :doc_values`, ElasticGraph:

- keeps the field in GraphQL output types
- excludes the field from datastore `_source` via `_source.excludes`
- requests the field through datastore `docvalue_fields`
- resolves the field from `fields` in datastore hits when `_source` omits it

The change is intentionally narrow: only direct, non-list, non-text
GraphQL leaf fields on indexed root document types are supported.
Query planning only uses `docvalue_fields` when every participating
index definition agrees on the retrieval method.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from a006605 to 4fe1200 Compare April 14, 2026 14:14
Add a narrow secondary retrieval path for fields that should stay
returnable in GraphQL while being excluded from stored `_source`.

When a field is marked `retrieved_from: :doc_values`, ElasticGraph:

- keeps the field in GraphQL output types
- excludes the field from datastore `_source` via `_source.excludes`
- requests the field through datastore `docvalue_fields`
- resolves the field from `fields` in datastore hits when `_source` omits it

The change is intentionally narrow: only direct, non-list, non-text
GraphQL leaf fields on indexed root document types are supported.
Query planning only uses `docvalue_fields` when every participating
index definition agrees on the retrieval method.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jwils jwils force-pushed the joshuaw/docvalue-leaf-fetching branch from 4fe1200 to a6256d3 Compare April 14, 2026 14:32
@jwils jwils marked this pull request as ready for review April 14, 2026 14:35
Copy link
Copy Markdown
Collaborator

@myronmarston myronmarston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

retrieved_from: :doc_values exposes a level of detail that, until now, we haven't exposed to ElasticGraph users. ElasticGraph does many things to query the data as efficiently as possible, without asking users to configure those things.

Can we do something similar here? For example, can we determine in which situations it's more efficient to use doc_values and automatically do so without requiring users to configure it at the field level?

Also, if we're going to change how fields are retrieved, I'd like to take a step back and review a full design that takes into account all the options Elasticsearch/OpenSearch offer us here. Besides _source and docvalue_fields there's also fields and stored_fields. I'd like to consider a unified design rather than just tacking this on as a one-off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants