Feature Request: Expanding References

[refactorized]

It would be amazing if the client could take care of expanding (some/all) references and returning properly typed, nested data. This would be especially useful for page-building apps, or any app in which the structure of a query will depend on the shape of the data.

In our case we just get all the sections that make up a page, each of which may have one or more images, assets or referenced sub-documents, which need to be resolved.

EDIT: I am realizing a subtlety here, in that query-time data is essentially runtime data and therefore typeless. But If I have a schema type that includes things like images which I will want expanded, it should still be possible to generate a type where those images are typed as images and not references to images. That type could then be used in the client and could indicate that images are to be expanded.

In our particular use case, a page would contain a list of sections, specified as an array of some large union of section types. These section types themselves would be types generated with reference expansion enabled. So any component code we write to render the section would be against the expanded type. (mapping sanity data to components happens at runtime and uses the sanity _type string, and basically promises all our other typed code that it knows what it's doing)

---

More background:

So far this is an excellent tool, but it clashes with a script we have that automatically expands references as they occur in results. We expand the references but then have no correct type definition for the results. Right now my options are:

1. write out custom typedefs based on the codegen types but with expanded references
2. modify the script we are using to inject those types as they go
3. scrap it, and just write out the types by hand

Option 1 results is less readable type defs and no less code. Option 2 requires introducing type mapping into an otherwise simple and ambivalent script, though using your experimental client to do the work could help a fair bit. Option 3 is what I probably need to go with for this project.

The script we use (sanity-io/GROQ#21 (comment)) expands the references by kicking off new fetches which isn't ideal, but works well enough for us currently, and is probably the best we can hope for currently. Hopefully Sanity will eventually support some kind of query syntax that can specify projection/resolution strategies for references of a given type (or other predicate) allowing resolution during the initial fetch.

Either way, a client that has a deep understanding of the types could both expand eventual references and ensure they were properly typed. This feature would be game changing, and eliminate tons of code for projects like ours.

(The 1.0 Roadmap hints that this might be under development already with its notes on weak references but I can't find anything explicit)

[ricokahler]

hi @refactorized 👋

thanks for opening this issue. it's definitely something i've wanted myself and i think it could be a really great value add.

(The 1.0 Roadmap hints that this might be under development already with its notes on weak references but I can't find anything explicit)

for more context on this project, i would want to develop a feature like this in the newer version due to the size of the feature and the potential breaking changes. weak references item you are talking about though is regarding sanity's _weak references and is unrelated to this proposed feature.

before we dive into solutions, i do want to ask if the upcoming GROQ codegen feature could help you accomplish what you need to a practical extent. this upcoming feature parses GROQ queries that are inline in your source code and generates typescript types from your query. GROQ allows for limited expansion/dereferencing so manually writing a deep enough nested expansions could practically do the job (though admittedly less ideal)

in theory, would that work for your use case? if so, i'm happy to help you set up this next version. feedback from early adopters would greatly help in the development.

---

regardless, i think have a solution for you that should work in the stable version of this lib and for your use case:

instead of writing all the types by hand, you could write a recursive type helper that uses typescript's conditional types to match and return different sub types of your type that you want resolved.

this lib explicitly add type parameters to sanity reference field types (even though the type parameter is unused) specifically for this type of metaprogramming.

that type helper could look something like this:

Edit: see updated code here: #178 (reply in thread)

type ResolvedReferences<T> =
    // match `SanityKeyedReference` and unwrap via `infer U`
    T extends SanityKeyedReference<infer U>
    ? U

    // match `SanityReference` and unwrap via `infer U`
    : T extends SanityReference<infer U>
    ? U

    // match arrays, unwrap with `T[number]`,
    // recursively run through `ResolvedReferences`
    // then re-wrap in an another array
    : T extends any[]
    ? Array<ResolvedReferences<T[number]>>

    // match objects, then utilize map types to
    // recursively run children through `ResolvedReferences`
    : T extends Record<string, unknown>
    ? { [P in keyof T]: ResolvedReferences<T[P]> } : T

type FooDoc = { _type: 'foo' }
type BarDoc = { _type: 'bar' }

type SomethingWithReferences = {
    objWithReference: {
        myReference: SanityReference<FooDoc>
    }
    arrayOfReferences: Array<SanityKeyedReference<BarDoc>>
}

type SomethingWithResolvedReferences = ResolvedReferences<SomethingWithReferences>

// this 👆 results in the following 👇

// type SomethingWithResolvedReferences = {
//     objWithReference: {
//         myReference: FooDoc;
//     };
//     arrayOfReferences: BarDoc[];
// }

ts playground link

the idea here is to then use this type helper in the asserted return type of this script you mention:

import * as Schema from '../your-generated-types';

async function getResolvedSections(sectionId: string): ResolvedReferences<Schema.Section> {
  // ...
  // your implementation here
  // ...
  //
  // note: you would still have to write the code to recursively resolve your query.
  // for it to truly deference infinitely deep, you would have to query more than once
  // and query for each nested reference yourself.
  //
  // the type helper is an idea you can use utilize the current types in a more generic way.
}

[refactorized]

Firstly, thank you so much for such a speedy and thorough reply. I am very excited about this project and all the quality work that has gone into it so far. I imagine further iterations of this tooling could become an essential part of our stack.

instead of writing all the types by hand, you could write a recursive type helper that uses typescript's conditional types to match and return different sub types of your type that you want resolved.

I think this is what I might be after, but I will have to investigate a little further. I have been jumping into typescript with both feet but didn't suspect that I could recursively type things. If that's the case, then it should allow me to apply just a small amount of code against all of the generated code, and would definitely put us in a better place!

I don't think generating types from groq is going to help me, in that I want my groq query to basically remain "get me all the items that live on this page" and even If was up for writing a huge query I don't think groq gives me a way to build a query that would describe how to resolve each item (which can be any one of a list of types) based on the type. In my use case it's not even an issue of recursion, I just want to be able to query for a flat list of items of a unioned type, but then treat each item as its more specific type (keyed on the _type string, which I have had success in combining with discriminated unions already).

[ricokahler]

I have been jumping into typescript with both feet but didn't suspect that I could recursively type things. If that's the case, then it should allow me to apply just a small amount of code against all of the generated code, and would definitely put us in a better place!

hey learning typescript is hard. people say it's "just javascript with types" but that's not really the case. the type system is its own turing complete programming language. it's the most expressive type system i've used and i'm still learning things about it today.

even If was up for writing a huge query I don't think groq gives me a way to build a query that would describe how to resolve each item

this should be possible with array unions given the select() operator in GROQ. it still pretty manual but select would allow you to split the union and write sub-queries for each type:

*[_type == 'section'] {
  ...,
  'content': content[] {...select(
    _type == 'movieSection' => {..., moviePosterReference->},
    _type == 'bookSection' => {...,bookCoverReference->},
    _type == 'articleSection' => {
      ...,
      'someObj': {
        ...,
        nestedRef->
      },
    }
  )}
}

in the query above, content is an array with multiple types configured in its of property. select is then used to switch based on the _type and splats ... are used to pull the rest of the data as-is down. the projection you'd write could replace the existing, unexpanded content key with a new transformation that resolves the wanted references

---

to be more transparent, i'm currently hesitant on adding more features to the sanity-codegen client. i started this project before I had a real grasp on GROQ and i never knew how valuable it was. this realization has lead to a shift in philosophy for this lib where i'd rather meet people where they're at and go along with sanity's ecosystem vs introducing new concepts

so with respect to that i would rather have GROQ itself support recursively expanding references first and then add support for it in groq-codegen. i can ask internally to see if that's a common request and if there are any plans for it (no promises however)

for now though it should still be possible to write an async function that recursively and generically resolves all references that will comply with a recursively transformed type. you would have to traverse the document yourself, find all references, download the referenced doc, and repeat until finished

[refactorized]

so with respect to that i would rather have GROQ itself support recursively expanding references first and then add support for it in groq-codegen. i can ask internally to see if that's a common request and if there are any plans for it (no promises however)

That is 100% what I would like to see long term, and what has the best chance of being widely applicable.

Short term, I think you have shown me some really important things that I have been searching for, and those things can probably fill in the gaps. It's been difficult searching for the solutions to this set of problems, and you have been 1000x more helpful than google, zeroing-in on the exact language features I am looking for in both typeScript and groq. I am sure this information exists in the respective documentation, but finding the right questions to get me there has been frustrating.

Thanks again for your help.

[ricokahler]

hey anytime. these features request/questions help more than you know. i think a good action item is to follow up with some recipes in the docs. i'll write this feedback down.

for now i'll close. if you have any more issues, don't hesitate to open more. i also just opened up discussions in this repo if you have any other questions that aren't related to issues or feature requests etc

[hacknug]

In our case we just get all the sections that make up a page, each of which may have one or more images, assets or referenced sub-documents, which need to be resolved.

so with respect to that i would rather have GROQ itself support recursively expanding references first and then add support for it in groq-codegen. i can ask internally to see if that's a common request and if there are any plans for it (no promises however)

This is something I would also love to see in GROQ. My use-case is pretty much the same: I have documents with content on Sanity whose structure changes from time to time. I'd love to be able to query the whole document in one line without the need to expand references manually via explicit queries or filter methods.